FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic
Authors: Kanghyun Choi
Hyeyoon Lee
SunJong Park
Dain Kwon
Jinho Lee
Venue: NeurIPS 2025 poster ()
Keywords: FP8
Quantization
LLM
LoRA
Quantized training
Primary area: deep_learning
TL;DR: FALQON accelerates LoRA fine-tuning by up to 3$\times$ through merging adapters into an FP8-quantized backbone, removing redundant quantization overhead from small matrices.

Abstract: Low-bit floating-point (FP) formats, such as FP8, provide significant acceleration and memory savings in model training thanks to native hardware support on modern GPUs and NPUs. However, we analyze that FP8 quantization offers speedup primarily for large-dimensional matrix multiplications, while inherent quantization overheads diminish speedup when applied to low-rank adaptation (LoRA), which uses small-dimensional matrices for efficient fine-tuning of large language models (LLMs). To address this limitation, we propose FALQON, a novel framework that eliminates the quantization overhead from separate LoRA computational paths by directly merging LoRA adapters into an FP8-quantized backbone during fine-tuning. Furthermore, we reformulate the forward and backward computations for merged adapters to significantly reduce quantization overhead, and introduce a row-wise proxy update mechanism that efficiently integrates substantial updates into the quantized backbone. Experimental evaluations demonstrate that FALQON achieves approximately a 3$\times$ training speedup over existing quantized LoRA methods with a similar level of accuracy, providing a practical solution for efficient large-scale model fine-tuning. Moreover, FALQON’s end-to-end FP8 workflow removes the need for post-training quantization, facilitating efficient deployment. Code is available at https://github.com/iamkanghyunchoi/falqon.

FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic
Kanghyun Choi
Hyeyoon Lee
SunJong Park
Dain Kwon
Jinho Lee
Department of Electrical and Computer Engineering Scoul National University
(kanghyun.chop) choi, hylee817, ryan0507, dain. 1en ac. kr
Abstract
Low-bit floating-point (FP) formats, such FP8, provide significant acceleration and memory savings in model training thanks to native hardware support on modern GPUs and NPUs. However, we analyze that FP8 quantization offers speedup primarily for large-dimensional matrix multiplications, while inherent quantization overheads diminish speedup when applied low-rank adaptation (LoRA), which uses small-dimensional matrices for efficient fine-tuning of large language models (LLMs). To address this limitation, we propose FALQON, a novel framework that eliminates the quantization overhead from separate LoRA computational paths by directly merging LoRA adapters into an EP8-quantized backbone during fine- tuming. Furthermore, reformulate the forward and backward computations for merged adapters to significantly reduce quantization overhead, and introduce a row-wise proxy update mechanism that efficiently integrates substantial updates into the quantized backbone. Experimental evaluations demonstrate that FALQON achieves approximately 3x training speedup over existing quantized LoRA methods with similar level of accuracy, providing a practical solution for efficient large-scale model fine-tuning. Moreover, FALQON's end-to-end FP8 workflow removes the need for post-training quantization. facilitating efficient deployment. Code available at htltttss:/githhin com com/iammeeae htomm//imaaagglyuccoo////gggg
Introduction
Recent advancements in large language models (LLMs) have demonstrated remarkable progress across a wide range of natural language understanding and generation tasks. Despite these achieve- ments, the massive parameter counts in LLMS require prohibitively large computational and memory resources, posing significant challenges for both training and deployment. In many real-world settings, those with reasonable amount of computational resources, training fine-tuning such massive
models becomes especially challenging 66 13 40).
One promising way reduce the computational burden of LLM fine-tuning is through low-precision Boating-poine (FP) 29 33 34), such as FP8, which can theoretically double the throughput FP16 matrix multiplication thanks to the hardware support of modern GPUs and NPUs. However, it is known that FP8 quantization primarily accelerates targe-dimensionee matrix multiplication, while computations suffer from unexpected slowdown. This is due to quantization overhead, which requires a reduction operation followed by element-wise scaling of a given tensor. Therefore, FP8 quantization becomes effective when computational gains surpass the overhead, which empirically occurs when each dimension of the matrices exceeds approximately 4K elements (33
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

These quantization overheads are especially critical when it comes to fine-tuning with low-rank adaptation (LoRA) [20]. LoRA inserts mmals-nimenioonal trainable low-rank matrices (adapters) to capture task-specific knowledge, significantly reducing the memory cost by using fewer trainable parameters. However, for matrices with small dimensions, such as LoRA adapters, the overhead incurred by FP8 quantization can outweigh the benefits from FP8 multiplications. Also, separate for- ward and backward paths for LoRA introduce a larger number of quantization operations, worsening the overall overhead. In our preliminary analyses (Section[4) 4) we show that applying FP8 quantization to LoRA introduces significant quantization overhead, limiting speedup. This slowdown in LoRA poses critical challenges in practical scenarios, where numerous adapters must be trained to support personalization [59], multi-task learning [28). and rapid updates in dynamic, user-specific environ- ments (see Section B.I.. Thus, efficient acceleration of LoRA fine-tuning is essential to enable timely, scalable, and cost-effective deployment of L.LMs under practical computational constraints.
To address this, we propose FALQON (FP8-Accelerated LoRA Quantization), a novel framework designed specifically to accelerate FP8-based quantized LoRA fine-tuning by reducing quantization overheads. Instead of separate LoRA adapters. FALQON merges adapters directly into the FP8 backbone d fine-tuning, leveraging the initial quantization error as an implicit LoRA initialization (melded LoRA) to eliminate extra quantization steps. Additionally, we reformulate both forward and backward computational paths for efficient gradient calculation of the merged adapters. A row-wise proxy update mechanism selectively applies substantial weight updates to the backbone, avoiding ineffective updates that vanish under low-bit quantization and further enhancing overall efficiency.
Through extensive experiments on various tasks, we demonstrate that F achieves up to 3x faster fine-tuning compared to quantized LoRA baselines, while maintaining comparable accu- racy. Moreover, the end-to-end FP8 workflow of FALQON eliminates the need for post-training quantization, facilitating efficient deployment. Our key contributions are summarized as follows:
We analyze FP8 quantization overhead and show that existing FP8 quantization methods primarily target dineeddimniio matrix multiplications, resulting in substantial overhead and limited speedups when directly applied to LoRA's small-dimensional adapters.
We propose FALQON, a novel framework that merges LoRA adapters into an FP8-quantized backbone during fine-tuning, significantly reducing quantization overhead.
We reformulate forward and backward paths for efficient gradient computation of merged adapters and introduce row-wise proxy update mechanism that selectively integrates substantial updates, avoiding unnecessary weight modifications under low-bit quantization.
We empirically demonstrate that FALQON achieves up to 3x faster fine-tuning compared to existing methods, providing detailed breakdown analyses to identify sources of acceleration, while maintaining comparable accuracy across comprehensive evaluations.
2 Background
2.1 Low-rank adaptation
Low-rank adaptation (LoRA) aims to reduce trainable parameters during fine-tuning by decomposing a parameter update into low-rank matrices. Specifically, let W Rmxx be the weight matrix and a E IR R be the activation. LoRA represents the weight update AW via a product of low-rank matrices B E Rmxrr and A E B' where r < min(m, n). Then, the forward pass becomes:
(W + AW)= (W + BA)= Wx + BAz.
(1)
The low-rank product BAr serves an efficient approximation of AWx, reducing trainable param- eters to mr + nr (compared to mn), which essential for fine-tuning large models under resource constraints. During backpropagation, the gradients of the loss L with respect to A and B are:
OC = 04
OC = B
OL OL
80 x TAT
80
OB
where 0 = (W- + BA)x is the layer's output. By keeping r & min(m, n), LoRA enables efficient fine-tuning of large models without requiring full parameter updates.
Quantized LoRA approaches 112 26 35 52) further reduce memory usage by applying weight-only
quantization to backbone weights. As the weight parameters of LLMs often span tens to hundreds of

---

gigabytes (e.g., approximately 14GB for 7B parameters and 140GB for 70B parameters in FP16), these methods quantize the backbone weights to low- oow-precision formats, follows:
Wr + BA R DQ(Q(W)) + BAr,
(3)
Q(W W, DQ(W) w,,
where Q(-) quantizes weights to low-precision representations (WQ), and DQ(-) reconstructs the high-precision approximation of quantized weight W. These additional quantization and dequanti- zation operations inherently introduce computational overhead, slowing down the fine-tuning. Our proposed approach is designed to mitigate quantization overhead and accelerate LoRA fine-tuning through low- low-precision arithmetic. By quantizing both weights and activations with minimal overhead, FALQON enables efficient matrix multiplication in contrast weight-only quantization.
2.2 Low-precision floating-poit quantization
Modern hardware accelerators (e.g., NVIDIA Hopper [2] and Blackwell (3T). AMD CDNA [T. Google Ironwood TPU (15) increasingly support low-precision FP arithmetic, significantly boosting efficiency in training and inference. In general, FP datatypes are often denoted by an EaM 3 format, which consists of single sign bit, exponent bits, and / mantissa (fraction) bits.
Low-precision FP8 formats (e.g., E4M3 E5M2) have a narrow representable range. Therefore, higher-precision tensor X (e.g.. FP32 or FP16) must be scaled by a per-tensor scaling factor 8x. The scaled tensor s X is then rounded to FP8, as follows:
X = Q = X sx], 8x max(FP8 dtype)'
(5)
where X the quantized tensor, (.ps()) is the 8-bit floating-point quantization function, [-1 is round-to-nearest operation, and maP( dtype) is the maximum representable value for the chosen datatype, i.e., 448 for E4M3 and 57344 for E5M2. The low- oowpprccision multiplication can be performed natively on hardware optimized for FP8, after which the result rescaled by 1/ 1/(sw8)):
matmul
Wx R W
8w82. Rescale
(6)
1: W. DQ2 p8 (W)
wwwa x W/sw,
fps(()) 3/S3.
Although FP8 computations can be significantly faster, the overhead of computing the per-tensor scale 8x (which involves reduction operation max([Xl)) and element-wise scale operation (Equation (5) may overshadow the speedup. The FP8 quantization overhead (amax reduction and element-wise scaling Equation (5) scales quadratically (O(n²) with matrix size This overhead is memory- bound, requiring repeated reads and writes for each tensor element, dominating execution time for smaller matrices. Therefore, FPS arithmetic only becomes beneficial at larger matrix sizes (empirically around n 2 4096), where the cubic complexity (O(n²)) of matrix multiplication outweighs the quantization overhead. Detailed analysis of these trade-offs provided in Section 4
3 Related Work
3.1
Low-rank adaptation
LoRA 20 significantly reduces resource usage by updating only low-rank adapters during fine-
tuning 8 39 46) or even training from scratch [27]. Recently, there has been an increased interest
in scenarios involving the training numerous LoRA adapters. Personalization: RecLoRA (59)
and PLoRA (55) train an independent LoRA for different users for recommendation. Also, Liu
et al. (28) trains multiple LoRA for personalized RLHF training. Multitask Learning: Mixture of
LoRA Experts [49 trains multiple task task-specific LoRA adapters and arranges these adapters. Domain
Adaptation: LoRA adapters trained with multiple domain are used handle various contexts (TT). Multilingual Summarization: Whitehouse el al. (47) demonstrate the application of LoRA for multilingual summarization. Given these diverse and practical applications requiring numerous adapters, accelerating LoRA fine-tuning holds significant practical value.
3

---

Forward MM
Ingur Cust
Gradiens Cust
Self-Att
Proj
Dewn-
Tosal
Weight
Theceitiol Massmmn Speod-up (3,00)
400
Spoed-up Factor
300
1.5
FPS Pre-trining Speeisup 58x)
Time
200
1.0
IP Bascline (.0g)
100
0
Ath
Up
Domn
Atti
Up
Down
256
FP16
FP8
LoRA Rank
(a) Breakdown analysis of LoRA computation.
(b) Speedup comparison of various LoRA ranks
Figure I: Preliminary experiments quantization overhead for LoRA (r 64) using linear layers from LLaMA-7B (self-attention, up-projection, down-projection). (a) FP8 reduces computation time. However, the quantization overhead dominates the computation and results in nearly fourfold higher latency. (b) FP8 shows consistently lower throughput than FP16 across ranks. These results demonstrate that FP8 quantization overhead cancels the expected acceleration in LoRA fine-tuning.
Several quantized LoRA approaches have recently been proposed for memory efficiency of LoRA fine- tuning. QLoRA [2] uses weight-only NF4 quantization to significantly reduce memory usage, while QA-LoRA (52 leverages group-aware INT4 quantization for efficient deployment. The following approaches improve quantization by employing integer linear programming [T6], SVD-based adapter initialization [26), dynamic rank allocation [21] 25], integer-based LoRA quantization [17]. and information recovery strategies [35] However, these methods primarily focus on memory efficiency while overlooking fine-tuning speed. Consequently, they often exhibit slower training than standard LoRA due to substantial quantization overhead. Moreover, since these methods rely weight-only quantization, they do not leverage hardware acceleration from quantization. In contrast, our proposed method successfully reduces memory consumption while significantly accelerating LoRA fine-tuning demonstrating superior efficiency compared to previous quantized LoRA approaches.
3.2 Quantized training with FP datatypes
Recently, FP quantization has gained attention due to native hardware support on GPUs and NPUs, providing wider dynamic range and improved precision than integers (45). Recognizing different numerical ranges of gradients and activations, Sun et al. [41) proposed hybrid FP8 training, using E5M2 for gradients and E4M3 for activations and weights. Other studies have addressed the limited dynamic range of FP8 by adjusting exponents [24] or modifying special-value representations [29].
Early quantized training methods primarily utilize low-bit integers 48 50 56 58
Following the release of NVIDIA's Hopper architecture with native FP8 support 33 recent studies have actively pursued practical acceleration of FP8-based training. FP8-LM (34) introduced mixed-
precision training, keeping only master weights and second-ordes momentum at higher precision. Fishman ef al. (14) further explored large-scale FP8 training stability and outliers, proposing modified SwiGLU. Meanwhile, unit-scaling methods 4 5 simplified FP8 training by determining optimal scaling at initialization. Despite these advances, they target large-dimensional matrices of L.MM train- ing. contrast, our method specifically accelerates the smaller-dimensional matrix multiplications in LoRA fine-tuning, achieving significant acceleration for the LoRA adapters.
4 Preliminary analysis
LoRA fine-tuning commonly involves small-dimensional matrices. In such cases, the computational overhead of quantization outweighs the speedup. This is because the separate LoRA path requires additional quantization for small matrices. As illustrated in Figure the LoRA fine-tuning introduces three additional quantization processes for two small-tensor multiplications per iteration compared to the direct training of the backbone. From the FP8-quantized X, LoRA fine-tuning requires quantizations of and from their newly updated weights, and another quantization for O the intermediate activation. Because À and B are both small, their speedup from using FP8 arithmetic cannot offset the quantization overheads.

---

In Figure la we investigate the impact of quantization overhead on mmal--dimensional LoRA (r=64) by breaking down the computation time of FP8 and FP16 arithmetic using an RTX 4090 GPU. Although FP8 arithmetic significantly reduces the computation time, the quantization overhead substantially outweighs these benefits, exhibiting approximately three four times higher latency. This issue persists even when employing adapters with larger ranks. In Figure 1b we compare the computation time of FP8 and FP16 LoRA adapters across various ranks. For the reference, we show empirical speedup of pretraining of LLaMA-7B model with TorchAO [43]. The results indicate a consistent slowdown FP8 compared to FPI6, showing roughly half the throughput across typical LoRA ranks (16-128). Even significantly higher ranks (e.g., 256 512), FP8 adapters continue exhibiting notable degradation. Please refer to Section in the Appendix for the results from the larger ranks. These preliminary results indicate that the inherent quantization overhead in FP8 arithmetic outweighs its computational advantages, emphasizing the need to significantly reduce quantization overhead to fully leverage FP8 arithmetic accelerate LoRA fine-tuning.
5 Proposed methods
In this section, we introduce FALQON, a novel framework for LoRA fine-tuning using FP8 arithmetic with significant speedups that were previously unavailable due to the quantization overheads. Moti- vated by inefficiencies arising from separate LoRA computations, our core idea is to directly meld LoRA adapters into the quantized backbone weights. This eliminates separate small-tensor LoRA op- crations in the forward pass. Because we let the weight updates still be done only through the LoRA, maintain the memory benefits of LoRA fine-tuning. In
even though it is melded into the backbone, the remainder of this section, we describe how we initialize and quantize the melded LoRA adapters (Section 5.I). how to compute gradients without explicitly storing the adapter weights (Section 5.2) and propagate the gradient of the adapters to the quantized melded backbone (Section 5.3)
5.1 Melded LoRA: Merging backbone and LoRA from the start
We propose melded LoRA, method that merges LoRA adapters directly into the quantized backbone weights to eliminate separate LoRA adapter paths. Motivated by Section 4 melded LoRA merges the LoRA adapter into the backbone weights during fine-tuning, removing the separate paths.
Algorithm 1 FALQON: Initialization
1: Input: rank r, pretrained backbone
W. quantization function Q(-)
2: for (= L, ...... do
3:
Wi, sw, + Q(Wi)
The key idea of melded LoRA is interpreting quantization error AQ W as additive tensor to the original high-
4:
AW, + W WW//w;
5:
Bi, -SVD(AW,,7
precision weight W. Quantized backbone weights W inherently have quantization error Ag W compared to the original high-precision backbone weights W, as follows:
6:
A + |Ar-sw;
7:
W" <- [WT] TTT
8:
A Buffer; <- [0) Coue
pQ (W) + AqW W,
(8)
9:
end for
where DQ denotes the FP8 dequantization function.
Then, reformulate Equation (8) follows:
DQ fps (W) = W - AQW & W + WWSSD((-AWWW)) W + BA,
where Ā, B are low-rank matrices from the rank-r SVD of W. This reformulation interprets the quantized backbone pQsss(W) as implicitly including the LoRA adapter BA, allowing a single computation of pQ (W): jar to produce the LoRA-adapted output directly (i.e., Wx B Arx) thus allowing efficient forward passes without additional adapter computations. Note that although existing works [16] 266] similarly leverage quantization error, they require separate LoRA to restore the original weights W, thus they do not address computational overhead. In contrast, our method effectively removes overhead, significantly enhancing efficiency (Figure
5.2 Efficient gradient computation for Melded LoRA
Unlike conventional LoRA, melded LoRA integrates adapters implicitly into backbone weights. Thus, adapter gradients are not explicitly computed by standard autograd operations. We address this by
5

---

Baseline
FALQON (Ours)
W
W
Procomputeed
LoRA path
Quant
Forward ($5.1, $5.2)
3. Update ($5.3)
Backward (65.2)
B
Forward
optimizer step
Backward
apply
top-k
updatess
Figure 2: Comparison between conventional LoRA fine-tuning and the proposed FALQON frame- work. Left: LoRA fine-tum tie-tunnng introduces redundant quantization for mmall-dimensional matrices (A, B. and OA), where the quantization overhead outweighs the computational benefit of FP8 arithmetic. Right: FALQON eliminates these overheads by directly melding LoRA into the quantized backbone, allowing the forward pass to be executed without separate small-tensor operations.
proposing an effective gradient computation method, specifically designed for the merged structure of melded LoRA, for further reducing overhead.
Algorithm 2 FALQON: Gradient Computation
To naively compute LoRA gradients, one could re- construct low-rank matrices A and B each iteration from the quantization error Aq W using SVD:
1: Input: Training data {d rank Quantized backbone W. quantization func- tion Q
A, 8 = SV D(W DQ r))
(10)
2: for = 1, 1,... N do
With these approximations, compute gradients following Equation (2).
3:
f di
4:
for 1, ... do
5:
11- SxL- t (11s(2111))
However, conventional gradient computations of LoRA involve mmll--limenioonal matrix multipli- cations, which incur significant quantization over- head. To mitigate this, we reformulate the gradient calculation of B in Equation
6:
mnerged +- W/ 82
7:
0, + Omerged
8:
9: ctx.save_for
(2). follows:
10: end for
OC Oc x AT
a0 _ (Ax)).
ac
11: Backward and Update (Algorithm
(11)
0B
30
12:
end for
Then, as the computation Wz x and Ar shares the input æ, we integrate the computation of Ar into the quantized forward pass by concatenating À to the quantized backbone weights W:
W
R (m+)))n
Ā= 1ps (A: = [Â sw s...
(12)
This enables simultaneous computation of both backbone output O and intermediate output O = Ar for gradient computation (Equation (Th) in a single forward without additional quantization:
O mmeggdd W WV'3/sw52
O O
E RR (m)+r)xx
(13)
where O E R.xxd O E Rexd By embedding and freezing À in the quantized backbone, we eliminate any additional gradient computations for the LoRA path. We only compute gradients of B, which is empirically sufficient for effective fine-tuning (54). Although concatenating A to the forward pass slightly increases the computation, the additional cost is negligible due to its small dimension (r). See lines 3-7 of Algorithm for initialization and Algorithm for computations.
5.3 Row-wise updates of quantized weights using proxy buffer
Given the directly applied to the merged backbone weights, because of the absence separate LoRA path. Formally, the weight updates from B matrix are:
merged structure of melded LoRA, gradients computed for the approximated Ô must be
W + (B + AB)A = (W + BA) + ABA W ABA.
(14)

---

Quantized LoRA
LLaMA-7B
Quantized
LLaMA-13B
Time per iteration (s)
10
9.38
20
18.25
8.27
Low-hit Quank.
Low-bit
5,45
10-
9.35
10.25
2,72
218
2.29
-00M
3:24
QLoRA
QA-LoRA
IRRQL
FP6-LLM
TarcAAO
Fishman atal.
FALQON (Ours)
QLoRA
QA-LoRA
IR-QLORA
FP6-LLM
TorohAO
Fishman al.
FALOON (Ours)
Figure 3: Overall computational cost comparison of baselines and FALQON (lower is better).
This indicates that the merged backbone weights W can be updated only from updates AB and A. without the need to store full-precision W and B. From this, we introduce a proxy update ma- trix Bufferr which stores only changes of the B matrix. Specifically, instead of updating B, we store gradient-driven changes in the proxy matrix
Algorithm 3 FALQON: Backward and Update
1: Input: learning rate v, Quantized backbone
W. Optimizer, gradient 0L/0rL
2:
for [ L, .... 1 do BC
3:
> Equation 0C Buffer Oppimizerr .
8n, + ac OT 8m
4:
Bufffr by recording changes of B from the opti- mizer step. Then, by storing AB, we can apply the updates of melded LoRA into merged backbone.
5:
k + topk(E,) AB k),
6: Wik = Wik + A Lanffer]]/4
7: end for
To efficiently incorporate impactful updates of A Buffer into the backbone weights, we selectively propagate large updates. Given that W is represented in low-bit, minor changes often result in no effective update. Therefore, we identify the indices with the top-k largest updates in Bufffer and selectively apply only significant changes:
k topk( > A |
W[K
W [ + Bufferr
(15)
Since k < m (output channels), this selective update method (Algorithm 3
efficiently integrates sub- stantial updates into W, while minimizing redundant computations and improving overall efficiency.
Evaluation
6.1 Experimental settings
We fine-tune ILaMA-7B and 13B on the Alpaca
42 and OASSTI
23
datasets, then evaluate
on the Massively Multitask Language Understanding (MMLU (19 benchmark, which measures knowledge and reasoning across a diverse set of domains. addition, we assess our models on
HellaSwag [53]. PIQA (3) WinoGrande [36], ARC (10), BoolQ (9.. and OpenBookQA (30) for
commonsense QA, following QA-LoRA's five-shot evaluation protocol via the
framework. Unless otherwise noted, all experiments are conducted on single 24GB RTX 4090
GPU. We compare our method (FALQON) against quantized LoRA baselines (QLoRA 12] QA-
LoRA [52], IR-QLoRA 35 ) and FP quantization methods (FP6-LLM [5T], Fishman al. [14] and
TorchAO [43).. J. We follow the settings of QLoRA: a Paged Adam W optimizer with a batch size of
16, learning rate of 2c-5, and 1,875 training steps. Refer to the Appendix for the detailed settings.
6.2
Comparison of fine-tuning efficiency and scalability
We evaluate the fine-tuning efficiency of FALQON against quantized LoRA and low-bit FP baselines (Figure 3 Quantized LoRA methods primarily reduce memory usage but exhibit higher computa- tional overhead, enabling larger (13B) models to fit into GPU memory but at significantly slower speeds. Low-bit FP methods achieve faster computation due to accelerated matrix multiplication. However, methods like TorchAO and Fishman el al. [14) maintain high-precision tensors, thus failing to fine-tune the 13B model due to out-of-memory (OOM) emror. In contrast, FAL.QN simultaneously accelerates FP8 computation and reduces memory cost, resulting in the lowest iteration time for both 7B and 13B models, achieving superior efficiency. See Section G for comparisons other GPUs.
7

---

Table 1: Comparison of time per step (lower better) and MMLU (higher is better)
(a) LLaMA-7B and 13B on Alpaca dataset
(b) LLaMA-7B and 13B on OASST1 dataset
Metric
QLoRA
QA-LoRA
IR RR-QL
FALQONN (Ours)
Metric
QLeRA QA-LoRA IR-QLoRA
FALQQN
(Ours)
Time Step 69)
5.45
(3.02%)
Time Step (5)
5.45
(3.4x)
#1. Panims,
160M
89M
89M
80M
#1. Params.
160M
89M
89M
80M
Humanities
0.3095
0.3413
0.3234
0.3322
Humanities
0.3367
0.3439
0.3362
0.3373
0.3137
0.2997
0.3143
0.3159
Social
0.3507
0.3711
0.3659
0,3858
Social
0.3894
0.3890
0.3949
0.3776
Other
0,3685
0,4007
0.3762
0.3795
Other
0.3972
0.4016
0,4010
0.3708
Avenge
0.3272
Average
0.3564
0.3609
0.3605
Time / Step (6)
9.37
18.02
14,46
3.26 (2.87
Time / Step (5)
9.35
18.25
15.22
3.24 (2.89
#1. Params.
250M
140M
#T. Params.
250M
140M
140M
Humanities
0.4253
0.443]
0.4157
0,4408
Humanities
0.4355
0.4387
0.4321
0.4436
13B
STEM 0,3438
0.3834
0.3356
0,3638
13B
STEM
0,3717
0.3920
0.3765
0.3638
Social 0.5096
0.5398
Social
0.5226
0.5499
0.5193
Other 0.5105
0.5426
0.5092
0.5259
Other
0.5272
0.5488
0.5375
0.5288
Average
0.4443
0,4729
0.4349
0,4644
Average
0.4605
0,4769
0.4620
0,4645
Table 2: Comparison of low-precision FP quantization methods on Alpaca and OASSTI dataset
Time Step (8)
Trainable Params
Alpaca (MMLU)
OASST1 (MMLU)
Method
Type
Hum.
STEM
Social
Other
Avg.
Hum.
STEM
Social
Other
Avg.
LoRA
FP16
2.87
160M
0.3295
0.3031 0.3717 0.3873 0.3456 0.3401 0.3258 0,4006 0.4102
0,3656
TorchAO
FP8
2.18
160M
0.3231
0.2969 0.3679 0.3785 0.3393 0.3273 0,3092 0,3672 0.3869 0.3452
FP6-LLM
E2M3
2.72
160M
0.2421
0.2125
0.2171 0.2398 0.2295 0.2448 0.2125 0.2177 0.2411 0.2308
FP6-LLM
2.72
0.2487
0.2693
0.2532 0.2333 0.2509 0.2423 0.2249
0,2190 0.2411 0,2330
Fishman at.
FP8
2.29
160M
0,3337
0,3108
0.3893
0,3923
0.3537
0.3241 0.2969
0,3773 0.3714 0.3401
FALQON (Ours)
FP8
1.79
80M
0.3322
0.3086
0.3858
0.3795
0.3491
0.3373
0.3130
0.3776
0.3708
0.3481
6.3
Comparison with quantized LoRA frameworks
We compare the fine-tume tinittnnng speed and accuracy of our method (FALQON) against quantized LoRA baselines on the Alpaca (Table Tab and OASSTI (Table Ib) datasets. For LLaMA-7B, FALQON achieves roughly three-fold speedup over QA-LoRA (c.g.. 1.80s vs. 9.44s per step on Alpaca), while maintaining competitive MMLU scores. Similarly. when scaling LLaMA-13B, FALQON significantly outperforms baselines computational efficiency (3.26s vs. 18.02s per step for QA- LoRA on Alpaca), without meaningful degradation in accuracy. Furthermore, FALQON consistently demonstrates stable performance, avoiding accuracy fluctuations across various categories observed in baseline methods. These results highlight the robustness and practicality of FALQON an effective and efficient solution for quantized LoRA fine-tuning.
6.4 Comparison with FP quantization methods
Table 2 compares the MMLU scores and training speeds of Baseline FP16 LoRA and low-bit FP quantization methods (TorchAO, FP6-LLM, Fishman et al. [14) and FALQON). We quantize all linear layers for FP8 methods and quantize the backbone for FP6-LLM similar to QLoRA. FALQON achieves the fastest runtime (1.79s/step) while showing comparable MMLU scores. In particular, FALQON average MMLU score on Alpaca (0.3491) surpasses those of TorchAO (0.3393) and FP6-LLM, and a similar trend is observed on OASSTI, achieving the highest accuracy (0.3481). These results underscore the efficiency and effectiveness of FALQON.
Table 3: Comparison common sense reasoning benchmarks
Method
ARC-C
ARC-E
BoolQ
HellaSwag
OBQA
PIQA
Winogrande
Average
LoRA (FP16)
0.4706
0.7844
0.8025
0.5849
0.3684
0.7984
0.7154
0.6464
QLoRA (NF4)
0.4735
0.7891
0.7856
0.5787
0.3660
0.7976
0.7159
0.6438
QA-LoRA (INT4)
0.4735
0.7723
0.7511
0.5618
0.3620
0.7867
0.7230
0.6329
IR-QLORA (NF4)
0.4812
0.7786
0.7902
0.5819
0.3680
0.8009
0.7230
0.6463
FP6-LLM (E2M3)
0.2125
0.2681
0.3783
0.2616
0.1400
0.5229
0.5075
0.3273
FP6-LLMM (E3M2)
0.2073
0.2647
0.3783
0.2600
0.1020
0.5239
0.4957
0.3188
TorchAO (FP8)
0.4753
0.7870
0.7933
0.5824
0.3640
0.7960
0.7206
0,6455
Fishman et al. (FP8)
0.4804
0.7896
0.7792
0.5826
0.3640
0.7949
0.7222
0,6447
FALQON (Ours)
0.4608
0.7786
0.7676
0.5711
0.3420
0.7905
0.7135
0.6320

---

Forward MM
Marlin Kemel Fwd.
Input Cast
Gradient Cast
Backward MM
Marlin Kemel Bwd.
Weight Cast
2500
Time (is))
2000
1500
1000
500
0
Attn UpDown
Attn UpDown QLoRA
Attn UnDown FP6-LLM
Attn UpDown TorchAO
Attn UpDown FALQON (Ours)
FP16 LoRA
Figure 4: Breakdown analysis of LoRA fine-tuning. Compared quantized LoRA baselines, which suffer from quantization overheads, FALQON minimizes redundant operations for superior efficiency.
6.5 Commonsense reasoning benchmark
Table reports performance on commonsense reasoning tasks. FALQON substantially outperforms the FP6-LLM, reflecting the benefits of its quantization strategy. By contrast, FP6-LLM struggles on tasks such as ARC-C and OBQA, highlighting the difficulties of maintaining accuracy with lower-precision formats. Although some baselines achieve slightly higher averages compared to FALQON, the gap varies by task. For instance, differences on ARC-E and PIQA are within 1-2%, whereas OBQA exhibits more pronounced gap. Despite not leading on every benchmark, FAIQQNN provides stable results across all tasks and balances accuracy with efficiency.
Analysis
7.1
Breakdown analysis of quantized LoRA fine-funing
We perform detailed breakdown analysis to understand computational bottlenecks in quantized LoRA fine-tuning methods, including QLoRA, FP6-LLM, TorchAO, and FALQON. Specifically, we measure the time of forward and backward matrix multiplication, along with the quantization overhead. For FP6-LLM, which uses the Marlin kernel to fuse quantization and computation, we separately measure fused kernel time for forward and backward. The results are shown in Figure
QLoRA introduces significant overhead from welg quantization, frequent dequantization results in nearly double the training time compared to FP1 FP6-LLM partially alleviates quantization overhead through specialized kernels but constrained by FP16 arithmetic. Similarly, despite TorchAO employing optimized FP8 arithmetic, substantial quantization overhead diminishes overall performance improvements. In contrast, FALQON effectively minimizes quantization overhead by reformulating computational paths and merging adapter computations. This strategy enables FALQON to fully exploit accelerated FP8 matrix multiplication, achieving significantly reduced computation times and superior overall efficiency compared to baseline methods.
7.2 Analysis of top-k selection overhead and efficiency
To evaluate the computational cost of the proposed top-k selective update in Algorithm 3 we measure its contribution to the overall training time. Table 4 summarizes the measured latency of each configuration, including the top-k overhead and the resulting reduction in total step time.
The results show that the top-k selection introduces marginal overhead-less than 0.6% of the total step time-while sub- stantially improving computational efficiency. Since FALQON performs updates only on the most significant components, the arithmetic cost in the update step considerably reduced. This results in faster iteration times and higher throughput. In practice, the minor additional cost well compensated by the overall gain in training efficiency, demonstrating that the in- clusion of the top-k selection computationally advantageous. For the effect of top-k on accuracy, please refer to Table 6
Table 4: Measured overhead and latency from top-k selection.
Time (ms)
7B
13B
Top-& overhead
+9.97
+13.85
Step w/o top-k Step w/ top-k
1814.98
3307. .48
1769.09
3210.36
Time reduction
-45.89
-97.12

---

Table 5: LLaMA-7B Training time, cost, and reduction for LoRA fine-tuning, Financial estimates are based on observed cloud GPU pricing.
Training Time (days, 8 GPUs)
Training Cost ($ USD)
Cost Reduction ($ USD)
Device
QLoRA
QA-LoRA
FALQON
QLoRA
QA-LoRA
FALQON
QLORA
QA-LoRA
RTX 4090
89.3
153.7
35.7
6,001
10,328
1,971
|140300
8,337
L40S
98.3
164.0
37.7
35,126
58,603
10,070
25,005
448.533
H100
31.1
25.1
13.3
41,122
33.114
13,419
427.703
4 41,6695
7.3 Scalability and efficiency of FALQON
We evaluate the scalability of FALQON under multi-adapter workloads representative of large-scale
personalization and adaptation tasks [22 55). Using the Moviel IM dataset (18) with 6,040 users, we estimate training time and monetary cost based on observed rates from cloud GPU platforms: Vast.ai (RTX 4090) [44,, AWS G6e (L40S) [37], and AWS P5 (H100) [38. As shown in Table FALQON achieves substantial cost reductions compared baselines, owing up 3x faster training throughput. These results indicate that FALQON delivers cost-efficient fine-tuning across diverse GPU cards and deployment environments, improving resource efficiency in large-scale adaptation scenarios.
7.4 Sensitivity study
We analyze the sensitivity of the LLaMA-7B model to hyperparameter choices, using MMLU
Table 6: Performance across different learning
rates (1) and top-k rows
accuracy as the evaluation metric. Table 6 illus- trates how different learning rates and the number of top-k rows affect performance. Table 6 we see that reducing the learning rate generally yields slightly improved results. Despite the highest set-
Number of Top-k Rows
Ir
10
20
30
50
20-1
0,2465
0.2347
0.2413
0.2295
0,2302
0.2295
2e-2
0.3209
0.3052
0.2971
0.2933
0.2849
0.3015
tings (e.g., 2e-4 with 1=))), the overall differences
0.3410
0.3488
0.3310
0.3460
0.3363
0.3330
across the tested ranges remain moderate, indicat- ing that our approach is robust to variations in both learning rate and k:
2e-4
0.3460
0.3462
0.3491
0.3470
0.3468
0.3426
2e-5
0.3454
0.3436
0.3460
0.3465
0.3440
0.3469
In addition, we conduct sensitivity analysis on batch size and LoRA rank to examine their effects on perfor- mance. As shown in the Table 7 performance remains steady across batch sizes (2-16) and ranks (16-128), with metrics ranging narrowly between 0 and 0.3494. Ad- ditionally, we analyze the sensitivity of fine-tuning speed to varying batch sizes, with detailed results provided in the Appendix. These results indicate that our method is robust to variations in various hyperparameter settings, demonstrating stable performance without significant tuning requirements. Refer to Section Hor further sensitivity studies the LLaMA- 13B model.
Table Sensitivity study on batch size
and rank
Rank (r)
Batch
32
64
128
2
0.3465
0.3457
0.3473
0.3484
0.3431
0.3494
0.3428
0.3462
0.3418
0.3456
0.3463
0.3482
16
0.3458
0.3486
0.3491
0.3462
8 Conclusion
We proposed FALQON, novel FPS quantization framework for accelerating LoRA fine-tuning. We showed that existing FP8 methods primarily target lange-dimensional matrix multiplications and thus experience substantial quantization overhead with dmall--dimenioonl LoRA adapters, limiting achievable speedups. To overcome this, FALQON directly merges LoRA adapters into the FP8- quantized backbone and employs a row-wise selective update mechanism, significantly reducing overhead and improving training efficiency. Experimental results demonstrate that FALQON achieves up to 3x speedup over existing quantized LoRA methods with similar accuracy, providing a practical and efficient solution for fine-tuning large-scale models.
10

---

Acknowledgements
The authors would like thank Morch, Inc. for funding this research and inspiring the initial concept.
Jinho Lee is the corresponding author, and he is funded by the following programs: (RS-2024- 00395134, RS-2024-00347394, RS-2023-00256081, 22022022122222 RS-2025-00564840)
References
[]] AMD. AMD CDNATM Architecture White Paper. Tech, rep. Advanced Micro Devices, Inc..
2024. URL: https: / amd com content /amm amd/ en /etdddouutt///////// ceccumentsiinsinnct tech-docs/ white- http:/hhttttppppppee aaappers//mddddaaeee amd cdna 3- white- pee-ppape.... sopeer-drf pdf]
[2] Ron Banner et al. "Scalable methods for 8-bit training of neural networks". In: Advances in
neural information processing systems (2018).
[3] Yonatan Bisk et al. "Piqa: Reasoning about physical commonsense in natural language". In: Proceedings of the AAAI conference on artificial intelligence. 2020.
[4] Charlie Blake, Douglas Orr, and Carlo Luschi. "Unit scaling: Out-of-the-box low-precision training". In: International Conference Machine Learning. 2023.
[5] Charlie Blake al. "u-P: The Unit-Scaled Maximal Update Parametrization". In: Proceedings
of the ICML 2024 Workshop on WANT. 2024.
[6] Arnav Chavan et al. "Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward". In: arXiv preprint arXiv:2402 01799 (2024).
[7] Arnav Chavan et al. One-for-All: Generalized LoRA for Purameter-tificient Fine-tuning. 2023.
arXiv: 2306 07967 [cs.LG]]
[8] Yukang Chen al. "LongLoRA: Efficient Fine-tuning of Long- Long-Context Large Language Models". In: The Twelfth International Conference on Learning Representations, 2024.
[9] Christopher Clark al. "BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions". In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019.
[10] Peter Clark al. "Think you have solved question answering? try arc, the ai2 reasoning challenge". In: arXiv preprint arXiv: 05457 (2018).
[11] Eric Nuertey Coleman et al. "Adaptive LoRA Merging for Efficient Domain Incremental Learning". In: NeurlPS Workshop: Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning. 2024.
[12] Tim Dettmers al. "Qlora: Efficient finetuning of quantized llms". In: Advances in neural information processing systems (2023).
[13] Jiangfei Duan et al. "Efficient training of large language models on distributed infrastructures: (2024).
a survey". arXiv preprint arXiv:2407.2000
[14] Maxim Fishman et al. "Scaling FP8 training to trillion-token LLMs". In: International Confer- ence Learning Representations. 2025.
[15] Google Cloud. Ironwood: The First Google TPU for the Age of Inference. httpss:/blog
www.d/drrowwooo.oooooooo tpu- age- oof-inffrennce 2025. [16] Han Guo et al. "LQ-LLRA A: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning". In: The Twelfth International Conference on Learning Repre- sentations. 2024.
ppoggee/pooduct//goggg.
[17] Hang Guo et al. "IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models". In: arXiv preprint arXiv:2410.2I1
(2024).
[18] F. Maxwell Harper and Joseph A. Konstan. "The MovieLens Datasets: History and Context".
In: ACM Trans. Interact. Intell. Syst. (2015).
[19] Dan Hendrycks et al. "Measuring Massive Multitask Language Understanding". In: The Ninth International Conference on Learning Representattions. 2021.
[20] Edward Hu et "LoRA: Low-Rant Adaptation of Large Language Models". In: Interna-
tional Conference on Learning Representations. 2022.
[21] Minsoo Kim et al. "RA-LoRA: Rank-Adaptive Parameter-Efficien Fine-Tuning for Accurate
2-bit Quantized Large Language Models". In: Findings of the Association for Computational Linguistics. 2024.

---

[22] Xiaoyu Kong et al. "Customizing language models with instance-wise lora for sequential recommendation" In: Advances Neural Information Processing Systems (2024). [23] Andreas Köpf et al. "Openassistant conversation--demorratiiii large language model align- ment". In: Advances in Neural Information Processing Systems (2024).
[24]
Andrey Kuzmin al. "FP8 quantization: The power of the exponent". In: Advances in Neural Information Processing Systems (2022).
[25] Geonho Lee et al. "RILQ: Bank-Insensitive LoRA-based Quantization Error Compensation for Boosting 2-bit Large Language Model Accuracy". In: Proceedings of the AAAI Conference Artificial Intelligence. 2025.
[26] Yixiao Li et al. "LoftQ: LoRA-Fine-Tuuing-wwaa Tuning-aware Quantization for Large Language Models".
In: The Twelfth International Conference on Learning Representations. 2024.
[27] Vladislav Lialin et al. "Relora: High-rank training through low-rank updates". In: The Twelfth International Conference on Learning Representations. 2023.
[28] Renpu Liu et al. "A Shared Low-Rank Adaptation Approach to Personalized RLHF". In: The 28th International Conference Artificial Intelligence and Statistics. 2025.
[29] Paulius Micikevicius
al. "Fp8 formats for deep learning". In: arXiv preprint
arXiv:2209.0544 (2022).
[30] Todor Mihaylov et al. "Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering" In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.
[31]
IVDDIANVVDDL Blackwell GPU Architecture Technical Overview.
nvidia. en- www.-s-bblaaeellllarrriieeeeeeee
2025.
[32]
NVIDIA, NVIDIA Hopper Architecture. https: / www. nvidia. com/ en us data
cetterr/bcchnollggee//ppooo----e http:/hooppeeraaacttttttueueee www.itecttteeeeeeeeeeeeeeeeeeeet 2022.
[33]
NVIDIA. Transformer Engine. hntpss: /t github........gg cww/NVIDIA///////. TWW.DD///TraasfommrrBniggg
Accessed: 2025-05-11. 2023.
[34] Houwen Peng et al. "Fp8-Im: Training fp8 large language models". In: arXiv preprint
arXiv:2310. /83 (2023).
[35] Haotong Qin et al. "Accurate LoRA-finetuning quantization of LLMs via information reten-
tion". In: Proceedings of the International Conference on Machine Learning. 2024.
[36] Keisuke Sakaguchi et al. "Winogrande: An adversarial winograd schema challenge at scale".
In: Communications of the ACM (2021).
[37]
Amazon Web Services. Amazon EC2 G6e Instances (L40S GPUs). https://asa amazon
com/ wtt///nstannceo http://stttncceeeeeeeeeeeaeeeeee g Accessed August 2025.
[38]
Amazon Web Services. Amazon EC2 P5 Instances (H100 GPUs). httph:tt/////// amazon
com/ httm///c/nnstaacetteee/e///// typps///eetppppp////pp Accessed August 2025.
[39] Ying Sheng et al. "S-lora: Serving thousands of concurrent lora adapters". In: arXiv preprint
arXiv:2311.033:
(2023).
[40] Venkat Srinivasan al. "Training large language models efficiently with sparsity and dataflow".
In: arXiv preprini arXiv:2304.055// (2023).
[41] Xiao Sun et al. "Hybrid 8-bit floating point (HFP8) training and inference for deep neural networks". In: Advances neural information processing systems (2019).
[42] Rohan Taori et al. Stanford Alpaca: An cnstruction-ooloviin LLaMA model. https: //
github. www.co//ttttoe htt///ttsuuuae st//sfsaoorrrrr _alpddaapppaaa__ele 2023.
[43] TorchAO Maintainers and Contributors. torchao: PyTorch Native Quantization and Sparsity
for Training and Inference. GitHub repository. htt.::.tttttt/tttt httt::///iiithbb..o/mpttrrrrrrrt
2024.
[44] Vast.ai. Pricing- Rent RTX 4090 GPUs on Vost.ai http:://rasttaaapprrcccc////
4090 Accessed August 2025.
[45] Naigang Wang al. "Training deep neural networks with 8-bit floating point numbers". In: Advances in neural information processing systems (2018).
[46] Yiming Wang et al. "Multilora: Democratizing lora for better multi-task learning". In: arXiv preprint arXiv:231 1550 (2023).
[47] Chenxi Whitehouse et al. "Low-Rank Adaptation for Multilingual Summarization: An Empiri-
cal Study". In: Findings of the Association for Computational Linguistics: NAACL. 2024.
12

---

[48] Mitchell Wortsman et al. "Stable and low- wew-precision training for large-scale vision- lsionnlanguuae models". In: Advances in Neural Information Processing Systems (2023).
[49] Xun Wu, Shaohan Huang, and Furu Wei. "Mixture of LoRA Experts". In: The Twelfth Interna- tional Conference Learning Representations. 2024.
[50] Haocheng Xi al. "Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Ber-Block Quantization", In: International Conference on Machine Learning. 2024.
[51] Haojun Xia et al. "Quant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric Algorithm-Systemt Co-Design on Modern GPUs". In: 2024 USENIX Annual Technical Conference. 2024.
[52] Yuhui Xu al. "QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models". In: The Twelfth International Conference on Learning Representationss 2024.
[53] Rowan Zellers et al. "HellaSwag: Can a Machine Really Finish Your Sentence?" In: Proceed- ings of the 57th Annual Meeting of the Association for Computationnal Linguistics. 2019.
[54] Longteng Zhang et al. "Lora-fa: Memory-efficiet low-rank adaptation for large language models fine-tuning" In: arXiv preprint arXiv:2308 033003 (2023).
[55] You Zhang al. "Personalized LoRA for human-centered text understanding". In: Proceedings
of the AAAI Conference on Artificial Intelligence. 2024.
[56] Kang Zhao et al. "Distribution adaptive int8 quantization for training cnns". In: Proceedings of the AAAI Conference on Artificial Intelligence. 2021.
[57] Qihua Zhou et al. "Octo:INTS training with loss-aware compensation and backward quantiza- tion for tiny on-device learning". In: 202/ USENIX Annual Technical Conference. 2021. [58] Feng Zhu al. "Towards unified int8 training for convolutional neural network". In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[59] Jiachen Zhu et al. "Lifelong personalized low-rank adaptation of large language models for recommendation" " arXiv preprint arXiv:2408.0355 03533 (2024).
13

---

A Code
We provide the implementation of FALQON in public GitHub repository: https://github com/ wtw.aaaannngynnoootaaaall
Our code is implemented based on the official repository of QLoRA [12]. The repository includes the complete training environment setup, evaluation scripts, and configuration details to ensure reproducibility. The code is under the terms of the MIT license.
B Full algorithm of FALQON
This section provides the complete algorithm of the proposed FALQON method. The algorithm integrates LoRA adapters directly into an FP8 FP8-quantized backbone from initialization, removing separate adapter computations in forward and backward passes. It also implements row-wise proxy update mechanism to efficiently propagate significant updates, achieving both computational efficiency and robust training performance. The detailed procedure, including initialization, forward pass computation, gradient calculation, and selective backbone updates, is presented Algorithm
Algorithm 4 Overall Framework of FALQON
1: Input: Training data {d H/ learning rate 1), rank pretrained backbone W, quantization function Q fps. update hyperparameter 2: Initialization Phase (Section 5.1)
3: for 1= 1,.... do
4: Wi,sw, + 1 (Wi)
5: AW W: Wi/sW ssw,
6: Br, A₁ + SV SVV(AW;,.)
Melded LoRA (Section
5.1
Equation (0
7: A < [A sw.
8: W' - [WT] AT)
A integration (Section 5.2
Equation 12
9: ABuffrr + {0] Cou
D A Buffer (Section 5.3
10: end for
Training Phase (Sections 5.2 and 5.3
12: for 1,.... N do
13: 20 +- d2
14: for = 1,..., L do
15:
81 5 1pps (21-)111)
Omerged W/ ss, Sx1-1 O,OA, +0 I1 0
17:
> Equation 13
19:
ctt.sQve for baar_baccwar(000 end for
21:
for L, .... do
Da ctx. ctv.saved tensors at < al O 8B,
Equation 2
Sw:
24:
Buffer, + Oplimizzer( 0L
:
25:
K +- topk((), A k),
26:
W|K] = W W[K] + Bufferr[K]
b Equation 15
27:
end for
28:
end for
Lines 2-10 describe the initialization stage of the FALQON framework. For each linear layer (line 3), weights are first quantized into FP8 format. To get the implicitly embedded LoRA adapters in the FP8-quantized weights, compute the quantization enoo (line 5), followed by singular-value
decomposition to obtain the low-rank matrices Á and B (line 6). Matrix Á is quantized (line 7) and concatenated directly into the quantized backbone (line 8), and the update buffer A Buffer initialized to zero (line 9).
14

---

Table 8: Efficiency analysis across various LoRA ranks. Higher ranks yield improved speed but rapidly increase memory consumption.
LoRA Rank
Model
Metric
16
32
64
128
256
512
1024
2048
4096
8192
16384
Speed-up (x)
0.50
0.55
0.44
0.48
0.43
0.62
0.69
0.60
0.80
1.18
1.22
LLaMA-7B
LoRA Mem. (GB)
0.67
0.78
0.98
1.39
2.20
3.83
7.09
13,60
26.63
52.70
104.82
Speed-up (x)
0.55
0.59
0.50
0.52
0.46
0.48
0.64
0.61
0.88
1.21
1.20
LLMMA-13B
LoRA Mem. (GB)
1.31
1.51
1.91
2.70
4.29
7.46
13.81
26.50
$1.89
102.66
204.21
Lines 11-28 show the training stage, comprising accelerated forward (lines 14-20) and backward computations (lines 2/-27). In the forward path, inputs are quantized into FP8 (line 15), then passed through the merged linear layer obtain both the output O and intermediate result O needed for gradient calculations (line 17). The intermediate results are saved for use in backpropagation (line 19). During the backward phase, stored intermediate outputs for matrix B (line 23). An optimizer updates the update buffer Buffer using computed gradients (line 24). Finally, the top-k largest updates in A Buffer are selectively applied the UP8-quantized weights (lines 25-26), ensuring efficient gradient propagation and maintaining training efficiency.
loaded (line 22) compute the gradient
C Extended results of speedup comparison across LoRA ranks
Figure iin the main text illustrates the efficiency trend of LoRA fine-tuning with varying adapter ranks. While the efficiency appears stagnant in the low-rank region (up to 512), it begins to improve beyond rank 1024, as summarized Table X The results indicate that higher-rank LoRA configurations achieve better computational efficiency but incur substantial memory costs, eventually exceeding those of full fine-tuning.
The observed trend is not counter-intuitive when considering the scaling behavior of quantization and matrix operations. The quantization overhead scales with O(n²), whereas the matrix multiplication cost scales with O(n"). Thus, for amall-dimensional matrices (common ranks of 16-512). the quantization overhead dominates, masking potential speedups. As the rank increases, the cubic growth of computation amortizes the quadratic quantization cost, leading to improved efficiency beyond rank 1024. However, the accompanying memory footprint increases sharply, making very large ranks (8192-16384) impractical for most fine-tuning setups.
D Ablation on matrix A update
In FALQON, updating the matrix A contributes marginally to model quality while incurring additional computational cost. Earlier work, LoRA-FA (54), empirically demonstrated that freezing matrix has negligible effect on the final fine-tuning performance, while significantly improving memory efficiency and throughput. Building on this observation, we conduct an ablation study to further verify the validity of freezing A within our framework.
Table/]preseen the training time and evaluation accuracy when matrix A is updated kept frozen. The results confirm that computing the gradient of A not only increases the computational bur- den-resulting in approximately a 16% slowdown on the 13B model-but also provides minimal gain in model quality. This overhead arises because the gradient of A, aı be precomputed. Consequently, freezing A during fine-tuning yields a more efficient optimization process without noticeable degradation in downstream task performance.
ac B ao xT cannot
E Detailed experimental settings
We fine-tune LLaMA-7B and 13B on the Alpaca 42 and OASSTI (3 datasets, then evaluate on the
Massively Multitask Language Understanding (MMLU 199 benchmark, which measures inowledge
and reasoning across diverse set of domains. In addition, we assess our models on HellaSwag (53
PIQA (3), WinoGrande (36). ARC (10), BoolQ [9, and OpenBookQA (30) for commonsense QA,
following QA-LoRA five- shot evaluation protocol via the lm-eval -arness framework.
15

---

Table Effect of updating matrix on training speed and model quality.
Model
Method
Time per Step (s)
Alpaca (MMLU)
OASSTI (MMLU)
LLaMA-7B
FALQON
1.80
0.3491
0.3481
Update matrix
1.94
0.3469
0.3470
LLaMA-13B
FALQON
3.26
0.4644
0.4645
Update matrix A
3.89
0.4634
0.4656
10.01
Time per iteration (s)
10.0
9.38
QLoRA
QA-LoRA
7.5
TorchAO
6,00
FALQON
5,45
(Ours)
5.0
2.18 1.18
1.79
2.30
1.72
1.90
1.53
0.810
0.8
0.0
RTX 4090
L40S
H100
Figure 5: Comparison of training speed on various GPUs.
All computational cost evaluations (Figure and Tables la 1b and 2 and detailed breakdown analyses (Figures Ia Tb and 4) are conducted on dedicated node of a local computing cluster. The computing node contains two Intel Xeon Gold 6442Y CPUs (total of 48 physical cores, 96 threads) running at 2.60 GHz, equipped with ITB DDRS ECC memory. The node has NVIDIA GeForce RTX 4090 GPUs (24GB VRAM) with NVIDIA driver version 550.54.15 and CUDA 12.4, running on Ubuntu 22.04.4 4 LTS with Linux kernel utilize a single RTX 4090 GPU from this node.
Unless otherwise noted, all experiments
We compare our method (FALQON) against quantized I LoRA baselines (QLoRA 12 QA-LoRA 52 IR-QLoRA
(35) and FP quantization methods (FP6-LLM (5T), Fishman et al. [14] and Tor- chAO 43 For each baseline we either use its official implementation or replicate its original experimental setup. In the quantized-LoRA baselines, we adopt weight-only quantization: NF4 for QLoRA and IR-QLoRA, and INT4 for QA-LoRA, For FP6-LLM, we use weight-only FP6-E3M2 and FP6-E2M3 quantization. For FPS-based methods (Fishman et al. TorchAO, and FALQON), use FP8-E4M3 for weights and activations, and FP8-E5M2 for gradients. We follow the settings of QLoRA: a Paged Adam W optimizer with batch size of 16, max gradient norm 0.3. learning rate of 2e-4, 1=10, and 1,875 training steps. The evaluation is conducted with the trained weight from the end of the training (i.e. 1.875-th step).
F Limitations
In this section, we discuss several limitations of our current approach. While our framework is generally applicable to any neural network layer involving linear operations, its practical effectiveness has thus far only been validated on Decoder-based Transformers such as L.LMMA models. Extending our method to additional architectures, including Encoder-based Transformers and Text-to-Image diffusion models, could further demonstrate its broader applicability. However, extending approach to these additional architectures is beyond the scope this work, and we leave it as an important direction for future research.
Also, our experimental evaluation primarily considers LLaMA-7B and 13B models, which are sufficiently large to highlight efficiency gains from FP8 quantization, while fitting within typical consumer-level GPUs (VRAM 24GB). Although evaluating even larger-scale models (e.g., over 70B parameters) would further strengthen our claims regarding scalability and efficiency, this was impractical within our computational constraints. Thus, such assessments remain important directions for future studies when additional computational resources available.
16

---

G Speed comparison on various GPUs
We evaluate the fine-tuning speed of quantized LoRA methods (QLoRA, QA-LoRA, TorchAO, and our proposed FALQON) across RTX4090, L40S, and H100 GPUs. The experimental results are shown in Figure 5 The RTX4090 and L40S share NVIDIA's Ada Lovelace architecture, resulting in similar training performance, while the Hopper-based H100 GPU exhibits distinct throughput characteristics. Our proposed method, FALQON, consistently achieves the fastest iteration times across all GPU platforms. Despite the varying relative performances of baseline methods, especially on H100, which designed for datacenter workload and has a different architecture and high memory bandwidth, FALQON maintains clear superiority, demonstrating its efficiency and adaptability across different hardware environments.
H Further experimental results
We present additional MMLU evaluation results in Table TO extending the analysis in the main body to include both LLaMA-7B and 13B models across Alpaca and OASST1 datasets. For the 13B model, TorchAO and Fishman al. cannot fine-tune due to memory constraints GPU (marked as OOM). Overall, FALQON consistently achieves the fastest training speeds while maintaining competitive or better MMLU scores. These results confirm the effectiveness of FALQON in balancing efficiency, accuracy, and scalability compared to existing FP-quantized methods.
single RTX4090
Also, we provide additional experimental results for commonsense reasoning benchmarks in Table While results the main paper (Table 3 focus on the Alpaca dataset and LLaMA-7B due to space limitations, here report comprehensive results both Alpaca and OASST1 datasets across LLaMA-7B and 13B models. Overall, we observe consistent performance trends across datasets and model sizes. Notably, unlike the 7B setting, TorchAO and Fishman al. cannot fine-tune the 13B model on a single RTX4090 GPU due to memory constraints, and thus these cases are noted as out-of-memory (0OM).
In addition, we provide an extended sensitivity study including the LLaMA-13B model (Table 12p. The trends observed for the 13B model align closely with those of the 7B counterpart in the main body, with moderate learning rates (around 2 x 10 to 2 x 10-3) generally producing optimal results. However, notice slightly increased sensitivity at the extreme ends of the tested learning rates, reflecting mild performance drop at very high (2e - 1) or very low (2e - 5) settings. Overall, the consistency in results the moderate learning rate region 7B and 13B models supports the robustness of our method across varied hyperparameter choices and different model scales.
Lastly, we extend the sensitivity analysis on batch size and LoRA rank, previously discussed for the LLaMA-7B on Alpaca dataset, to additional configurations, datasets, and model scales (Table [3). Results from the OASSTI dataset using the 7B model reveal similar robustness as the Alpaca dataset, with minor fluctuations in performance metrics. For the larger LLaMA-13B models, performance remains consistently stable across varied batch sizes and ranks, suggesting that our method maintains reliability even at larger scales. Collectively, these additional analyses reinforce the general robustness and stability of our method under different hyperparameter choices.
I Batch size sensitivity of training speed
We measure training throughput across different batch sizes (Figure (6) and observe that FALQQN consistently achieves significantly higher throughput compared to QLoRA and QA-LoRA. Although increasing the batch size naturally reduces throughput due to greater per-step computational load, FALQON demonstrates substantially smaller decline, showing its superior efficiency and lower computational overhead. This empirically validates that FALQON is especially beneficial for practical scenarios with constrained GPU resources or varying batch size requirements, offering meaningful reductions in overall fine-tuning time.
17

---

Table 10: Comparison of low-precision FP quantization methods on Alpaca and OASSTI dataset
Time/ Step
Trainable Params
Alpaca (MMLU)
OASSTI (MMLU)
Model Method
Type
STEM
Social
Other
Avg.
STEM
Social
Other
Avg.
TorchAQ
FP8
2.18
160M
0.3231
0.2969 0.3679 0,3785 0.3393 0,3273
0.3092 0.3672 0.3869
0.3452
FP6-1.1M
2.72
160M
0.2421
0.2125
0.2171 0.2398 0,2295 0.2448
0.2125 0.2177 0.2411
7B
1P6-LLMM
E3M2
2.72
160M
0.2487
0.2693
0.2532 0.2333
0,2509
0.2423
0.2249 0,2190 0.2411 0.2330
Fishman al.
2.29
160M
0.3108
0.3893 0.3923
0.3537
0.2999
FALQON (Ours) 5)
FP8
1.79
80M
0.3322
0.3086
0,3858
0,3795
0,3491
0,3573
0.3130
0.3776
0.3708 0.3481
TorchA
160M
00M
ODM
OOM
0OM
0OM
0OM
FP6-LLM
E2M3
10.19
160M
0.2421
0.2141 0.2171 0.2398
0.2298
0.2640
0.2518
0.2424
0.2626
0.2562
138
FP6-LLM
E3M2
10.19
160M
0.2425
0.2211 0.2236 0,2417 0.2334 0.2425
02119 0.2190
0.2401
0.2300
Fishman ol.
14
FP8
160M
0OM
00M 00M 00M
00M
00M
0OM 0OM
00M 00M
FALQON (Ours)
FP8
3.24
80M
0,4408
0.3578
0.4774
0.4799
0,4644
0,4436
0,3571
0.4771
0.4754
0.4645
Table 11: Comparison on common sense reasoning benchmarks
Benchmark Results
Dataset
Model
Method
ARC-C
ARC-F
BoolQ
HellaSwag
OBQA
PIQA
Winogrande
Average
QLoRA
0.4735
0.7891
0.7856
0.5787
0.3660
0.7976
0.7159
0.6438
QA-LoRA
0.4735
0.7723
0.7511
0.5618
0.3620
0.7867
0.7230
0.6329
IR-QLoRA
0,4812
0.7786
0,7902
0.5819
0,3680
0,8009
0.7230
0.6463
Alpaca
7B
FP6-LLM (E2M3)
FP6-LLM (E3M2)
0.2125
0.2073
0.2647
0.2681
0.3783
0.3783
0,2600
0.2616
0.1020
0.1400
0,5229
0,5239
0.4957
0.5075
0.3188
0,3273
TorchAO (FP8)
0.4753
0.7870
0,7933
0,5824
0,3640
0,7960
0.7206
0,6455
Fishman (FP8)
0.4804
0.7896
0.7792
0.5826
0.7222
FAI FALQON (Ours)
0.4608
0.7786
0,7676
0,5711
0,3420
0.7905
0,7135
0,6320
OLoRA
0,5299
0.8161
0.8269
0.6087
0.3700
0.8085
0.7640
0.6749
QA-LoRA
0,5179
0,8051
0,8064
0,5930
0,3580
0,8074
0,7719
0.6657
IR-QLoRA
0,5247
0,8110
0.8407
0.6064
0,3640
0,8118
0.7537
0.6732
Alpaca
13B
FP6-LLM (E2M3)
FP6 L.M (E3A/2)
0.2048
0,2133
0.2731
0.2685
0,3783
0,3783
0.2609
0,2605
0,1340
0,1440
0,5310
0,5283
0.4949
0.5075
0,3290
0.3249
TorcAAO (FPS)
OOM
0OM
0OM
0OM
OOM
OOM
Fishman et al. (FP8)
OOM
0OM
0OM
OOM
OOM
OOM
0OM
OOM
FALQON (Ours)
0.5179
0.8064
0.7887
0.6010
0,3560
0.8030
0.7648
0.6625
QLoRA
0,4650
0.7849
0.7813
0.5921
0.3640
0.790n
0.7222
0.6428
QA-LoRA
0.4727
0.7765
0.7731
0.5816
0,3700
0.7954
0.7167
0.6409
IK-QLoRA
0.4761
0.7715
0.7991
0.5956
0,3780
0.7889
0.7072
0.6452
OASSTI
7B
FP6-LLM (E2M3)
0,2031
0.2614
0.6162
0.2581
0,1280
0,5234
0.4862
0,3538
L.M (E3M2)
0,1911
0.2736
0.3783
0.2599
0.1260
0.4949
0.3235
TorchAO (FPS)
0.4727
0.7854
0,7991
0,5932
0,3640
0.7911
0.7206
0.6466
(FP8)
0.4804
0.7875
0.7911
0.5966
0.7922
0.7222
FALQON (Ours)
0,4514
0.7769
0,7703
0,5716
0,3400
0,7889
0,7174
0.6309
OLoRA
0,5213
0.8110
0.8110
0.6170
0.3520
0,8074
0.7672
0.6696
QA-LoRA
0.5341
0.8114
0.8193
0.6178
0.3560
0.8107
0.7743
0.6748
IR-OLoRA
0,5307
0.8026
0,7963
0.6227
0,3660
0,7965
0.7616
0
OASSTI
13B
FP6-LLM (E2M3)
0.2585
0,5024
0.3266
0.6409
0,4008
F (E3A/2)
0,2065
0.2702
0,4550
0,2589
0,1180
0.5261
0.4791
0.3305
TorchAO (FP8)
OOM
00M
0OM
0OM
OOM
OOM
0OM
OOM
Fishman et al. (FP8)
0OM
0OM
00M
0OM
OOM
OOM
0OM
OOM
FALQON (Ours)
0.5154
0.8030
0.7902
0.6012
0,3460
0,8079
0.7743
0.6626
QLoRA
QA-LoRA
FALQON
Throughput (stepsis)
2.5
2.0
1.5
1.0
0.5
0.0
2
Batch Size
Figure 6: Throughput at varying batch sizes
18

---

Table 12: Performance across different learning rates (1) and top-k rows for Alpaca and OASSTI datasets
Number of Top-k Rows
Dataset
ILaMA (n)
5
10
20
30
50
2e-1
0.2465
0.2347
0.2413
0.2302
0.2295
2e-2
0.3209
0.3052
0.2971
0.2849
0.3015
2e-3
0,3410
0.3488
0.3310 0.3460
0.3363
0.3330
2e-4
0.3462 0.3491
0.3468
0.3426
2e-5
0.3436
0.3440
0,3469
Alpucu
2e-1
0.3047
0,2300
0.2322 0.2295
0.2295
0.2294
2c-2
0.4507
0.4591
0,4472 0.4394
0.3820 0,3774
13B
2e-3
0.4648
0,4694
0.4634
0.4629
0,3924 0.4160
2e-4 0.4638
0,4657 0,4644
0,4650 0,4301 0,4294
2e-5
0.4273
0,4270
0,4284
0,4298
0.4269 0,4290
2e-1
0.5509
0.3931
0,3570
0.2697
0.2571
0.2605
2e-2
0.3544
0,3627
0,3494
0.3475
0.5799 0,3400
7B
2e-3
0.3484
0,3483
0,3495 0.3434
0.5855 0,3504
2e-4
0.3452
0,3484
0,3481
0.3462
0.5732 0.3472
2e-5
0.3468
0.3493
0,3462
0.3458
0.5721 0.3454
OASST1
2e-1
0.3446
0,2632
0.2404
0.2352
0.2295
0.2423
2e-2
0,4580
0.4611
0,4611
0,4586
0.4519
0.4358
13B
26-3
0.4637
0.4616
0,4578
0.4574
0.4566 0,4514
2e-4
0.4645
0.4651
0.4645
0.4662
0.4624
0.4645
2e-5
0.4671
0.4654
0.4655
0 0.4656
0.4647
0.4659
Table 13: Sensitivity study batch size and rank
Rank (r)
Dataset
Model
Batch
16
32
64
128
0.3465
0.3457
0.3473
0.3484
0.3431
0.3494
0.3428
0.3462
7B
0.3418
0.3456
0.3463
0,3482
0.3458
0.3486
0.3491
0.3462
Alpaca
2
0.4657
0.4662
0.4641
0,4668
4
0.4650
0.4626
0.4640
0.4654
13B
8
0.4649
0.4634
0.4656
0.4643
0.4647 16
0.4647
0.4644
0.4648
0.3477
0.3467
0.3490
0.3502
0.3439
0.3442
0.3461
0.3463
7B
0.3469
0.3476
0.3464
0.3448
0.3460
0.3482
0.3481
0.3467
OASSTI
2
0.4658
0.4647
0.4635
0.4640
4
0.4662
0.,643
0.4646
0.4639
13B
8
0,4624
0.4646
0.4644
0.4652
16
0.4637
0.4673
0.4645
0.4638
19

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We clearly state our contribution and scope in the abstract and introduction sections. Our experimental results show significant speedup, and our breakdown analysis provides the reason for the computational benefit.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations the Appendix under the "Limitations" section due to space constraints in the main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
20

---

Answer:
[NA]
Justification:
We have no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed method section and corresponding algorithms. Also, provide detailed hyperparameter settings for the reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide experimental code and scripts to reproduce the main results in a separate zip file. Additionally, all datasets used are publicly available through the Huggingface Datasets library.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guides Codeeuumissionooolle
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips wwtp::ccpppppiiiicee/c/s////c/// (odeeubmissiinnnnn...o ierobbmmsssoooooolim.oo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide details in the experimental settings section in the main paper and the Appendix. Also, we provide reproducible code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We measure computation time in fixed and controlled local GPU environment, observing negligible variance; thus, we omit error bars. For accuracy, we use the default data splits from publicly available libraries without test-set training, conducting all experiments with fixed random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. neggaive error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We clearly specify GPU type, memory capacity, and computational resources in the experimental settings. Additionally, we include detailed computational costs and execution times central components of our experimental evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:: http://wwrrraa.opuuauaaaaaaaaaaa
Answer:
[Yes]
Justification:
We check and follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our method is one of the optimization methods mentioned in the example.
Thus, we provide potential societal impact solely from our proposed framework.
Guidelines:
23

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We propose an accelcration method of the existing low-rank fine-tuning Therefore, we may share the potential risk with LoRA-based fine-tuning, but no risk added beyond that.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all datasets, models, and libraries used in our experiments, and include appropriate licensing information within our provided code repository.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
24

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release code for FALQON and details on usage, training procedures, licensing, and known limitations.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We did not use L.M for any core development, besides of editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
26


Reviews:
[{"review_id": "LPravcTVR5", "reviewer": "Reviewer_kFb2", "summary": "The paper analyzes the benefits and overhead of FP8 utilization for low-rank quantization. The authors observed that the speed-up gained by using FP8 is offset by the quantization overhead due to the small dimension of low-rank adapters. To address this issue, the paper proposes the FALQON framework, which interleaves an FP8-quantized backbone with LoRA adapter models. This integration requires modifications to the forward path and backward pass for gradient computation, as proposed in the paper. The results show an improvement of 1.8x to 3x for LLM models compared to using LoRA and FP8 quantization separately. An ablation study over various training parameters such as learning rate, batch size, and number of ranks is also presented.", "questions": "In the Efficient Gradient Computation section, it is mentioned that gradients are computed only for matrix B, which is empirically efficient for effective fine-tuning. Could you elaborate on this point in more detail and explain why gradient computation for matrix A is not required?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n1- The proofs for the claims in the paper are presented clearly.\n\n2- The idea behind the efficient gradient calculation by combining the gradients of the weights and the low-rank matrix (A) is novel and significant.\n\nWeaknesses: \n\n1- The new approach seems to be valuable when multiple agents use multiple quantized LoRA adapters, as mentioned in the paper. It is suggested that multiple LoRA adapter fine-tunings are evaluated on a specific benchmark in the paper, demonstrating the trade-off between performance and the number of quantized LoRA adapters.\n\n2- FP4 fine-tuning has been suggested in previous papers [1,2,3], and it would be helpful to include it in the experiments section as a low-bit FP quantization baseline for comparison with the proposed approach.\n\n[1] Liu, Lian, et al. \"COMET: Towards Practical W4A4KV4 LLMs Serving.\" Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 2025.\n\n[2] Wang, Jie, et al. \"Fp4-quantization: Lossless 4bit quantization for large language models.\" 2024 IEEE International Conference on Joint Cloud Computing (JCC). IEEE, 2024.\n\n[3] Roy, Somnath. \"Understanding the impact of post-training quantization on large language models.\" arXiv preprint arXiv:2309.05210 (2023).", "was_revised": true, "final_justification": "In the rebuttal, the authors evaluate the trade-off between performance and the number of quantized LoRA adapters, demonstrating the consistent speed-up performance of FALQON compared to QLoRA. Moreover, they provide FP4 fine-tuning experiments, showing improvements over FP8 LoRA quantization. Furthermore, my question regarding the bypassing of the calculation of gradients for matrix A was thoughtfully addressed. Overall, the approaches presented in this paper are both valuable and novel.", "rebuttal": "We appreciate valuable feedback from the reviewer. We are encouraged that the reviewer finds our work novel and significant, with well-presented content. \n\nWe address the reviewer’s concerns and incorporate suggestions. Here is a brief summary of this rebuttal. \n\n* **W1-Suggesting evaluation of trade-off between performance and the number of LoRA adapters**: We provide experimental results that present robust performance gains.\n* **W2-Extension to FP4 fine-tuning**: We provide estimated speed up of FP4, where FALQON show 2.15$\\times$ faster fine-tuning than FP16. \n* **Q1-Request for the rationale of freezing A matrix**: Updating A matrix has small-to-no effects on the model quality [1], but incurs significant overhead. \n\nWe will incorporate these into the manuscript and thank the reviewer for the time. \nThe following is a detailed rebuttal. \n\n***\n\n### **W1: The new approach seems to be valuable when multiple agents use multiple quantized LoRA adapters. It is suggested that multiple LoRA adapter fine-tunings are evaluated on a specific benchmark in the paper, demonstrating the trade-off between performance and the number of quantized LoRA adapters.**\n\n$\\to$ Thank you for the comments. We believe the reviewer is asking for scalability on training multiple LoRA adapters at the same time.\nTo address this, we provide an analysis of the trade-off between performance and the number of LoRA adapters in Table R1.\n\n**Table R1:** Trade-off between performance and the number of LoRA adapters.\n|# of LoRA adapters:|1|2|4|8|16|32|64|\n|---|---|---|---|---|---|---|---|\n|QLoRA|3.99*s*|4.00*s*|4.00*s*|4.04*s*|4.19*s*|4.48*s*|6.82*s*|\n|FALQON|1.76*s*|1.77*s*|1.77*s*|1.77*s*|1.80*s*|2.34*s*|2.93*s*|\n|Speed Up|2.27$\\times$|2.26$\\times$|2.26$\\times$|2.28$\\times$|2.33$\\times$|1.91$\\times$|2.33$\\times$|\n||\n\nThis experiment directly evaluates the scenario mentioned by the reviewer, demonstrating that FALQON consistently outperforms the baseline in terms of time per step, even as the number of LoRA adapters increases.\n\nHowever, when we say “training numerous adapters” in our paper, we refer to running multiple independent fine-tuning jobs, each producing a single adapter. As far as we know, there is no use case to train multiple LoRA weights attached to the same backbone, simultaneously. Thus, we also provide experiments on training multiple LoRA weights independently. Table R2 reports the total training time required for such large-scale personalization, using the MovieLens-1M [C2] dataset scenario from [C1], which involves 6,040 users.\n\n**TABLE R2:** Comparison of temporal costs for fine-tuning LoRA\n|Time (days, 8 GPU)|QLoRA|QA-LoRA|FALQON|\\||vs QLoRA|vs QA-LoRA|\n|---|---:|---:|---:|---|---|---|\n|4090|89.3|153.7|35.7|\\||$\\Downarrow$ 53.6|$\\Downarrow$ 118.0|\n|L40S|98.3|164.0|37.7|\\||$\\Downarrow$ 60.6|$\\Downarrow$ 126.3|\n|H100|31.1|25.1|13.3|\\||$\\Downarrow$ 17.9|$\\Downarrow$ 11.8|\n||\n\nThese results confirm that FALQON provides significant temporal savings for both concurrent and large-scale fine-tuning scenarios.\n\nWe hope these results address the reviewer’s concerns. If further clarification or discussion is needed, we welcome additional questions in the discussion phase.\n\n[C1] Kong, Xiaoyu, et al. \"Customizing language models with instance-wise lora for sequential recommendation.\" NeurIPS (2024).\n\n[C2] Harper, F. Maxwell, and Joseph A. Konstan. \"The movielens datasets: History and context.\" ACM TiiS (2015).\n\n\n### **W2: It would be helpful to add FP4 fine-tuning baselines.**\n\n$\\to$ Thank you for this helpful suggestion. To address it, we provide a comparison in Table R3 below. The FP4 speedups are estimated by applying the FP4/FP8 matrix multiplication throughput ratio, as specified in NVIDIA GPU documentation and available hardware benchmarks:\n\n**Table R3:** Speed up Comparison for FP8 and FP4 compared to FP16\n|Model|Method|FP8|FP4|\n|---|---|---|---|\n|LLaMA-7B|TorchAO|1.31$\\times$|1.55$\\times$|\n||FALQON|1.61$\\times$|2.15$\\times$|\n||\n|LLaMA-13B|TorchAO|1.21$\\times$|1.35$\\times$|\n||FALQON|1.51$\\times$|1.91$\\times$|\n||\n\nThese results suggest FP4 could provide further speedup over FP8 for both TorchAO and FALQON, assuming similar implementation efficiency. However, current hardware and software support for full FP4 fine-tuning is still limited. We are actively seeking access to FP4-capable resources and will report empirical results as they become available.\n\nAdditionally, we would like to clarify that prior works [1,2,3] primarily focus on weight-only FP4 quantization for **inference** rather than end-to-end fine-tuning. Most of them do not speed up fine-tuning since computations still use high-precision arithmetic or do not support backward or update of weights. Recent papers such as [C3] and [C4] do address FP4 fine-tuning, but they target full-rank LLM training and focus mainly on large-dimensional matrix multiplications. In these scenarios, quantization overheads remain significant, and we believe FALQON’s approach would continue to offer benefits even in the FP4 regime.\n\n[C3] Castro, Roberto L., et al. “Quartet: Native FP4 Training Can Be Optimal for Large Language Models.” arXiv preprint arXiv:2505.14669 (2025).\n\n[C4] Chmiel, Brian, et al. “FP4 All the Way: Fully Quantized Training of LLMs.” arXiv preprint arXiv:2505.19115 (2025).\n\n### **Q1: Could you explain why gradient computation for matrix A is not required?**\n\n$\\to$ It is because updating matrix A does not significantly contribute to model quality [49], while slowing down the training. The earlier work, LoRA-FA [49], empirically showed that freezing matrix A has little effect on the final quality of fine-tuning. \n\nIn addition to LoRA-FA, we conduct further experiments showing the effect of freezing matrix A in Table R4. The results empirically verify that we may freeze matrix A during fine-tuning without sacrificing much of the model quality. Also, computing the gradient of matrix A affects the training speed, showing approximately a 16% slowdown (13B). This is because gradient of matrix $A$ ($\\frac{\\partial \\mathcal{L}}{\\partial A}$) is computed as $B^{\\top}\\frac{\\partial \\mathcal{L}}{\\partial O} x^{\\top}$, which cannot be pre-calculated unlike matrix $B$. We thank the reviewer for their insightful comments, and we will clarify this in the revised version of the manuscript. \n\n**Table R4:** Comparison of training speed and model quality regarding matrix A update\n|Model|Method|Time per Step|Alpaca (MMLU)|OASST1 (MMLU)|\n|---|---|---|---|---|\n|LLaMA-7B|FALQON|1.80s|0.3491|0.3481|\n|LLaMA-7B|+ Update matrix A|1.94s|0.3469|0.3470|\n||\n|LLaMA-13B|FALQON|3.26s|0.4644|0.4645|\n|LLaMA-13B|+ Update matrix A|3.89s|0.4634|0.4656|\n||\n\n[49] Zhang, Longteng, et al. \"LoRA-FA: Memory-efficient low-rank adaptation for large language models fine-tuning.\" arXiv preprint arXiv:2308.03303 (2023).\n\n***\nWe thank the reviewer for their insightful comments. We believe this rebuttal addresses the reviewer’s concerns and further clarifies our contribution. We welcome further questions or comments and would be happy to discuss them."}, {"review_id": "uattSdVNxI", "reviewer": "Reviewer_CKwV", "summary": "This paper identifies an interesting performance bottleneck in reduced-precision large language model (LLM) fine-tuning: while low-bit formats like FP8 can significantly speed up large matrix multiplications, their inherent quantization overheads lead to a net slowdown when applied to the small-dimensional matrices used in Low-Rank Adaptation (LoRA). To address this, the authors propose FALQON, a framework that eliminates the need for a separate computational path for LoRA adapters. The core idea is to \"meld\" the LoRA adapter directly into the FP8-quantized backbone weights from the start, using the initial quantization error as a form of adapter initialization. The framework reformulates the forward and backward passes to compute gradients efficiently within this merged structure and introduces a selective, row-wise update mechanism to apply only the most significant changes. The experimental results demonstrate that FALQON can achieve up to a 3x training speedup over existing quantized LoRA methods while maintaining comparable accuracy on various benchmarks.", "questions": "- The row-wise update mechanism introduces a top-k operation on the gradient matrix ΔB at every training step. This operation is not free. Could you provide an analysis of the computational overhead of this top-k selection and discuss how it impacts the overall wall-clock time speedup claimed by FALQON?\n\n- The dimensions (e.g., rank r) for the micro-benchmark in Figure 1a are not specified.\n\n- The trend in Figure 1b, where efficiency does not improve with increasing rank, is counter-intuitive and contradicts the paper's own analysis without further explanation.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- This paper pinpoints the less-explored inefficiency of FP8 acceleration for LoRA finetuning. It was interesting to note that LoRA would suffer from quantization overhead due to its small dimensional arithmetic. \n\n- The central idea of \"melded LoRA\" seems to be interesting as it would avoid the separate LoRA path entirely.\n\n\nWeaknesses\n- The paper's central premise—the need to quantize the LoRA path to FP8—feels somewhat constructed. LoRA adapters are intentionally small, and standard FP16 LoRA is a very strong and widely used baseline. The motivation for forcing the entire pipeline into an FP8 format, which creates the very problem this paper solves, is not sufficiently justified. The paper would be stronger if it presented compelling use cases (e.g., specific hardware constraints, or scenarios where training thousands of adapters simultaneously makes the FP16 overhead non-trivial) where this end-to-end FP8 acceleration is a critical, practical need rather than an academic exercise.\n\n- The most significant weakness is the omission of the standard FP16 LoRA as a performance baseline in the results tables (Tables 1, 2, 3). FALQON introduces several approximations (e.g., initializing from quantization error, fixing the adapter matrix A, and only updating B, avoid maintaining full-precision weight) that could potentially degrade fine-tuning quality. Without a direct comparison to the de-facto standard, it is impossible to assess whether the impressive speedup comes at a hidden cost to model performance. The claim of maintaining \"comparable accuracy\" is only validated against other quantized methods, which may already be degraded.\n\n- The paper suffers from a lack of clarity in its experimental reporting, hindering a full understanding of the results.  \n  - The experimental settings for baselines like QLoRA and TorchAO are ambiguous. The precision of the backbone, the LoRA adapters (presumably FP16 for QLoRA?), and optimizer states should be explicitly stated for all methods.\n  - The explanation for why TorchAO causes an OOM on the 13B model while FALQON does not is unclear. If FALQON's key contribution is to improve LoRA efficiency, how come it is more memory efficient than TorchAO (which does not have LoRA at all)? \n- The work's contribution is heavily weighted towards systems-level optimization and implementation efficiency rather than fundamental algorithmic advances in machine learning. While it involves algorithmic changes, the core novelty lies in overcoming a hardware-specific bottleneck. As such, the paper might be a more natural fit for a systems-focused conference like MLSys or DAC, rather than a general ML conference like NeurIPS.", "was_revised": true, "final_justification": "Thank you for the hard work in preparing the rebuttal. While I see potential in the proposed methods for reducing LoRA PEFT's memory requirements, the additional results (Table R3) reveal a concerning accuracy trade-off. Compared to FP16 and QA-LoRA, there's noticeable performance degradation, suggesting that FALCON's weight update approach needs refinement. I remain concerned whether the proposed methods can achieve fine-tuning performance comparable to the FP16 LoRA baseline.\n\nI agree that the inefficiency stems from the LoRA algorithm itself. However, given the rebuttal's emphasis on system-level benefits (e.g., memory savings), I believe a systems-focused venue like MLSys or DAC would be more appropriate. Therefore, I maintain my original rating.", "rebuttal": "We thank the reviewer for the constructive review. We are delighted that the reviewer acknowledged that the motivation and the central idea are interesting. \n\nHere, we address the reviewer’s concerns and questions. This is a brief summary of the rebuttal. \n\n* **W1-Use cases for FP8 LoRA**: We elaborate on real-world use cases (4 tasks from 6 papers) in the manuscript and further present cost comparison of FP16 and FP8 in rebuttal. \n* **W2-Accuracy comparison with FP16**: We show comparison with FP16 LoRA and FALQON also provides comparable accuracy. \n* **W3.1/Q2-Detailed experimental settings**: We provide more detailed experimental settings.\n* **W3.2-Why TorchAO causes OOM in 13B model?**: TorchAO store weights as FP16 which takes more memory than FALQON.\n* **Q1-Computational overhead regarding top-k selection**: We provide further analysis, and top-k selection incurs approximately 0.5% overhead of wall-clock time.\n* **Q3-Efficiency does not improve with increasing rank in the motivational study**: The efficiency starts to improve after 4k rank (which is infeasible).\n\nWe will include this in the manuscript and thank the reviewer for the effort. \nThe following is a detailed rebuttal. \n\n***\n### **W1: The motivation for FP8 format is not sufficiently justified. The paper would be stronger if it presented compelling use cases (e.g., scenarios where training thousands of adapters simultaneously makes the FP16 overhead non-trivial).**\n\n$\\to$ We respectfully highlight that large-scale training of LoRA adapters is crucial to real-world applications such as personalization [54, 50, 26], multitask learning [44], domain adaptation [11], and multilingual summarization (see lines 111–118). For example, PLoRA [54] targets millions of users, and iLoRA [C1] evaluates up to 6 million users. In these settings, FP16 overhead becomes significant.\n\nTo further support this, Tables R1 and R2 provide a direct cost comparison on the MovieLens-1M [C2] scenario with 6,040 users, showing that FALQON reduces training time by months and cost by thousands of dollars compared to FP16 LoRA. This demonstrates the real-world benefits of our approach.\n\n**TABLE R1:** Comparison of financial costs for fine-tuning LoRA\n|\\$ USD|QLoRA|QA-LoRA|FALQON|\\||vs QLoRA|vs QA-LoRA|\n|---|---:|---:|---:|---|---|---|\n|RTX4090|\\$6,001|\\$10,328|\\$1,971|\\|| $\\Downarrow$ \\$4,030  |$\\Downarrow$ \\$8,357 |\n|L40S|\\$35,126|\\$58,603|\\$10,070|\\||$\\Downarrow$ \\$25,057|$\\Downarrow$ \\$48,533|\n|H100|\\$41,122|\\$33,114|\\$13,419|\\||$\\Downarrow$ \\$27,703|$\\Downarrow$ \\$19,695|\n||\n\n**TABLE R2:** Comparison of temporal costs for fine-tuning LoRA\n|Time (days, 8 GPU)|QLoRA|QA-LoRA|FALQON|\\||vs QLoRA|vs QA-LoRA|\n|---|---:|---:|---:|---|---|---|\n|4090|89.3|153.7|35.7|\\||$\\Downarrow$ 53.6|$\\Downarrow$ 118.0|\n|L40S|98.3|164.0|37.7|\\||$\\Downarrow$ 60.6|$\\Downarrow$ 126.3|\n|H100|31.1|25.1|13.3|\\||$\\Downarrow$ 17.9|$\\Downarrow$ 11.8|\n||\n\n[C1] Kong, Xiaoyu, et al. \"Customizing language models with instance-wise lora for sequential recommendation.\" NeurIPS (2024).\n\n[C2] Harper, F. Maxwell et al. \"The movielens datasets: History and context.\" ACM TiiS (2015).\n\n### **W2: The most significant weakness is the omission of the standard FP16 LoRA as a performance baseline in the results tables (Tables 1, 2, 3). The claim of maintaining \"comparable accuracy\" is only validated against other quantized methods.**\n\n$\\to$ Thank you for this suggestion. We have included FP16 LoRA results below for direct comparison:\n\n**Table R3:** Comparison with FP16 LoRA\n|Model|Method|Time per Step|Alpaca (MMLU)|OASST1 (MMLU)|\n|---|---|---|---|---|\n|LLaMA-7B|FP16|2.87s|0.3527|0.3656|\n||QLoRA|5.45s|0.3272|0.3564|\n||QA-LoRA|9.44s|0.3548|0.3609|\n||IR-QLoRA|8.27s|0.3388|0.3605|\n||FALQON (Ours)|1.80s|0.3491|0.3481|\n||\n|LLaMA-13B|FP16|OOM|0.4641|0.4714|\n||QLoRA|9.37s|0.4443|0.4605|\n||QA-LoRA|18.02s|0.4729|0.4769|\n||IR-QLoRA|14.46s|0.4349|0.4620|\n||FALQON (Ours)|3.26s|0.4644|0.4645|\n||\n\nThe results show FALQON has competitive accuracy, even compared to FP16 LoRA. Please note that FP16 LoRA could not run on a single RTX4090 for LLaMA-13B due to its large memory consumption. Therefore, we run FP16 LoRA on an RTX A6000 GPU and only report accuracy without time per step (noted as OOM). Our focus on quantized methods is motivated by resource constraints common in practical deployments, but we agree that including FP16 LoRA enhances the comparison, and we will add these results to the manuscript. \n\n\n### **W3.1: The experimental settings for baselines like QLoRA and TorchAO are ambiguous.**\n\n$\\to$ We followed the original settings for each baseline: QLoRA and IR-QLoRA use NF4 for the backbone and FP16 for adapters; QA-LoRA uses INT4 for the backbone and FP16 for adapters. For FP8-based methods (TorchAO, Fishman et al., and ours), both the backbone and adapters use FP8. All experiments use a 32-bit paged AdamW optimizer. We will clarify these details in the revised manuscript. Thank you for the suggestion.\n\n### **W3.2: The explanation for why TorchAO causes an OOM on the 13B model while FALQON does not is unclear. How come FALQON is more memory efficient than TorchAO?**\n\n$\\to$ TorchAO stores master weights in FP16, so it does not reduce memory compared to FP16 LoRA. While TorchAO performs dynamic quantization for computation, FALQON maintains all weights in FP8, enabling significant memory savings. Consequently, FALQON can fine-tune the 13B model on a single RTX4090 GPU, which is not feasible with TorchAO or FP16 LoRA.\n\n### **W4: The work's contribution is heavily weighted towards systems-level optimization and implementation efficiency rather than fundamental algorithmic advances in machine learning. While it involves algorithmic changes, the core novelty lies in overcoming a hardware-specific bottleneck. As such, the paper might be a more natural fit for a systems-focused conference like MLSys or DAC, rather than a general ML conference like NeurIPS.**\n\n$\\to$ We respectfully disagree. The core idea of this paper is an algorithmic advance of the traditional quantization method by innovating the LoRA fine-tuning procedure. In addition, our target problem (the quantization bottleneck) is not a hardware-specific bottleneck, but an algorithm-induced slowdown. Because the quantization algorithm delivers the bottleneck, we improve the quantized training framework by modifying the LoRA algorithm. Lastly, quantization is one of the well-discussed topics in general-domain ML conferences such as NeurIPS. We believe our paper is beneficial to the general ML audience of the NeurIPS conference by providing practical values of efficient fine-tuning algorithms. \n\n### **Q1: Could you provide an analysis of the computational overhead of this top-k selection and discuss how it impacts the overall wall-clock time speedup claimed by FALQON?**\n\n$\\to$ The top-k selection step accounts for only about 0.5% of total training time (see Table R4), as it operates over $r$ rows (with $r \\ll d$).\n\n**Table R4:** Top-k selection overhead of LLaMA-7B and 13B models\n |Time (ms)|Train Step|Top-k selection|Overhead|\n|---|:---:|:---:|:---:|\n|LLaMA-7B|1769.09|9.97|0.56%|\n|LLaMA-13B|3210.36|13.85|0.43%|\n||\n\nMoreover, top-k selection reduces the update matrix dimension from $r$ to $k$, which lowers the weight update cost (See Figure 2 and Algorithm 3-line 6 in the paper.). As shown in Table R5, this leads to an overall speedup in step time:\n\n**Table R5:** Comparison of time per steps with and without top-k selection\n|Time per Step (ms)|w/ Top-k|w/o Top-k|Speed up|\n|---|:---:|:---:|:---:|\n|LLaMA-7B|1769.09|1814.98|1.026x|\n|LLaMA-13B|3210.36|3307.48|1.030x|\n||\n\nThus, top-k selection has negligible overhead and actually contributes to overall training speedup.\n\n### **Q2: The dimensions (e.g., rank r) for the micro-benchmark in Figure 1a are not specified.**\n\n$\\to$ Thank you for the comments. The rank is 64, which is a default setting of all our experiments, unless mentioned otherwise. \n\n### **Q3: The trend in Figure 1b, where efficiency does not improve with increasing rank, is counter-intuitive and contradicts the paper's own analysis without further explanation.**\n\n$\\to$ The efficiency does improve when we increase the rank over 1024 (Table R6). \n\n**Table R6:** Efficiency analysis across various LoRA ranks\n| |Rank:|16|32|64|128|256|512|1024|2048|4096|8192|16384|\n|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|LLaMA-7B|Speed Up|0.50$\\times$|0.55$\\times$|0.44$\\times$|0.48$\\times$|0.43$\\times$|0.62$\\times$|0.69$\\times$|0.60$\\times$|0.80$\\times$|**1.18$\\times$**|**1.22$\\times$**|\n|FT: 49.01GB|LoRA Mem.|0.67*GB*|0.78*GB*|0.98*GB*|1.39*GB*|2.20*GB*|3.83*GB*|7.09*GB*|13.60*GB*|26.63*GB*|**52.70*GB***|**104.82*GB***|\n|LLaMA-13B|Speed Up|0.55$\\times$|0.59$\\times$|0.50$\\times$|0.52$\\times$|0.46$\\times$|0.48$\\times$|0.64$\\times$|0.61$\\times$|0.88$\\times$|**1.21$\\times$**|**1.20$\\times$**|\n|FT: 95.48*GB*|LoRA Mem.|1.31*GB*|1.51*GB*|1.91*GB*|2.70*GB*|4.29*GB*|7.46*GB*|13.81*GB*|26.50*GB*|51.89*GB*|**102.66*GB***|**204.21*GB***|\n||\n\nThe results show that the efficiency starts to increase after 1024 rank and becomes beneficial in ranks 8192 and 16384. However, the memory consumption on those ranks exceeds that of full fine-tuning, making LoRA inefficient at all.\n\nWe believe that this is not counter-intuitive as the quantization overheads scale in $O(n^2)$ while the computation of matrix multiplication scales in $O(n^3)$. Therefore, in the small-dimensional region (widely used rank 16-512), the efficiency does not seem to increase due to its exponentially scaling characteristics. We will modify the motivational study to the extended range and add this explanation. \n\n***\nWe thank the reviewer for the valuable comments. We believe that this rebuttal effectively addresses the reviewer’s concerns and clarifies the paper's contributions. Also, we would be happy to address any additional comments the reviewer may provide."}, {"review_id": "ly8nbVE4jV", "reviewer": "Reviewer_EcqX", "summary": "The work suggests merging LoRA adapters into an FP8 quantized backbone to speed up quantized fine tuning with FP8. Instead of running a separate FP16 path for the two LoRA multiplications in $BAx$ the factor $\\hat{A}$ is folded into the FP8 matrix so a single operations produces both of the multiplications with $x$, $\\widetilde{W}x$ and the LoRA factor $\\hat{A}x$.\nThis means that gradients are only needed for $\\hat{B}$ and conversions between FP16 and FP8 are limited, providing a computational speedup. Experiments confirm wall time speedup and not large deviation from task evaluations.", "questions": "* What is special about FP8? Compared to say, INT4/NF4?\n* Section 4 / Figure 1: What is behind the dashed lines, theoretical maximum, and pre-training speed up? What is the empirical setup? Do you take random matrices of random dimensions? This is not clear.\n* The sign of the error term flips from equation 8 to 9, keep it as negation?\n* Line 184-185: I don’t see another interpretation as possible, what do you mean?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* The approach makes sense, removing these redundant operations is valuable (and could be implemented with a fused kernel for larger improvements?)\n* The low rank treatment has been used before LoftQ, LQLoRA, LoQT, initialising with SVD is empirically sound\n* Cutting memory and compute is important\n* Experimental comparisons and ablations are well rounded, good to see both performance and speed checked\n\nWeaknesses\n* Would have been good to have some sort of full picture overview or the key derivations together to get a concise overview, either with equations, a diagram or a combination of the two. I found it quite hard to follow. I think a more pedagogical approach could have been taken. The contribution is hidden away until page 5! \n* Writing and grammar can be improved. Some sentences are very long and logical flow is not always evident, e.g. line 92. Other sentences are vague, e.g. line 149.\n* You say in the introduction that FP8 quantization overhead outweighs the benefits from FP8 multiplication. I think you can make it clearer it is the speed and not the memory you refer to\n* Same in claims about cost of FP8<->FP16 in line 104-105. You say there will be a detailed analysis in section 4, but it is somewhat preliminary (as the section title says).\n* In the background on LoRA: you may want to mention that the adapters can be downcast.\n* In some places it would be good to make it clear you are talking about computational speed and not memory as is often the focus of the low rank papers.\n* I would say what your method does earlier, at least at a high level. You keep mentioning it is better (which gets repetitive), but only at page 5 do you say how it works!\n* Typo: Line 84, you say $\\widetilde{W}$ is high precision\n* Figure 2: For this to be easier to read it might be good to include the operations (addition and multiplication)\n* Equation 9 uses the same approach as LoftQ and LQLoRA to initialize the quantization error, if I'm not mistaken, maybe worth mentioning both\n* No error bars, you could have at least bootstrapped intervals for the evaluations\n* Note on code: it’s 244MB uncompressed! That seems very excessive. Also, not all .git folders were deleted.", "was_revised": true, "final_justification": "I leave my score at 4 since I am not sure about the broader impact of the work, the \"trick\" developed is nice though.", "rebuttal": "We genuinely appreciate the reviewer for the helpful comments and feedback. We are happy that the reviewer finds the proposed methods are valuable and sound. \n\nWe present this rebuttal to address the reviewer’s concerns and questions. Here is a brief summary of the contents. \n\n* **W1, W9-Would have been good to have a concise overview**: We clarified the algorithmic overview and improved the explanation of key derivations, including diagram updates and explicit operation descriptions.\n* **W4-Request for detailed analysis for matrix dimension**: We provide a detailed speed-up analysis across matrix sizes, showing that FP8 achieves empirical acceleration for dimensions above 4K.\n* **W11-Error bar for evaluations**: We provide two-sigma error bars for the experiments, confirming consistency. \n* **Q1-What is special about FP8 compared to INT4/NF4?**: We clarify that FP8 uniquely accelerates fine-tuning via fast GPU arithmetic, while INT4/NF4 primarily reduce inference memory.\n* **Typos and Writing suggestions**: We have revised and clarified key sentences as suggested.\n\nHere, we provide a detailed rebuttal.\n\n***\n### **W1, W9: Would have been good to have some sort of full picture overview or the key derivations together to get a concise overview. I think a more pedagogical approach could have been taken. Figure 2: For this to be easier to read it might be good to include the operations (addition and multiplication)**\n\n$\\to$ Thank you for the suggestion! Please note that we have included the complete algorithm overview in Appendix Section B. Due to the page limits, we provided partial algorithms for each of the proposed methods in the main body of the paper. \n\nRegarding Figure 2, we will add a modified version with the operations in the manuscript. Unfortunately, we could not post the revised figure in rebuttal, due to the system’s limitations. Therefore, we briefly mention how the change will happen. In Figure 2, activations ($\\tilde X$ and $O$) connected to the weights ($\\tilde W$, $\\tilde A$, and $\\tilde B$) with arrows are matrix multiplication. Weights and activations connected to (Quant) are quantization operations. Bold arrow in 3. Update figure is an addition for the weight update. \n\n\n### **W2: Some sentences are very long or vague.**\n\n$\\to$ Thank you for the constructive feedback. We will improve the sentences as follows:\n\n* Line 92 $\\Rightarrow$ Low-precision FP8 formats (e.g., E4M3 or E5M2) have a narrow representable range. Therefore, a higher-precision tensor $X$ (e.g., FP32 or FP16) must be scaled by a per-tensor scaling factor $s_X$. The scaled tensor $s_X \\cdot X$ is then rounded to FP8, as follows:\n\n* Line 149 $\\Rightarrow$ LoRA fine-tuning commonly involves small-dimensional matrices. In such cases, the computational overhead of quantization outweighs the speedup.\n\n### **W3 and W6: It would be good to make it clear you are talking about computational speed and not memory, as is often the focus of the low rank papers.**\n\n$\\to$ We thank the reviewer for the great suggestion! We will further clarify this throughout the manuscript. For example, we will add “Unlike most of the prior LoRA-based methods that aim to reduce the memory cost, our method aims to speedup the fine-tuning of LoRA by leveraging the FP8 arithmetics, suggesting an algorithmic approach to maintain the accuracy while maximizing the speed.”\n\n### **W4: Claims about cost of FP8<->FP16 in line 104-105. You say there will be a detailed analysis in section 4, but it is somewhat preliminary.**\n\n$\\to$ We further provide a detailed analysis regarding the FP8 efficiency across LoRA rank in Table R1, comparing computation time and memory usage with FP16. \n\n**Table R1:** Efficiency analysis across various LoRA ranks\n| |Rank:|16|32|64|128|256|512|1024|2048|4096|8192|16384|\n|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|LLaMA-7B|Speed Up|0.50x|0.55x|0.44x|0.48x|0.43x|0.62x|0.69x|0.60x|0.80x|**1.18x**|**1.22x**|\n|FT: 49.01GB|LoRA Mem.|0.67*GB*|0.78*GB*|0.98*GB*|1.39*GB*|2.20*GB*|3.83*GB*|7.09*GB*|13.60*GB*|26.63*GB*|**52.70*GB***|**104.82*GB***|\n|LLaMA-13B|Speed Up|0.55x|0.59x|0.50x|0.52x|0.46x|0.48x|0.64x|0.61x|0.88x|**1.21x**|**1.20x**|\n|FT: 95.48*GB*|LoRA Mem.|1.31*GB*|1.51*GB*|1.91*GB*|2.70*GB*|4.29*GB*|7.46*GB*|13.81*GB*|26.50*GB*|51.89*GB*|**102.66*GB***|**204.21*GB***|\n||\n\nThe results indicate that FP8 incurs a slowdown for typical LoRA ranks under 4K, and only becomes faster than FP16 at extremely high ranks (over 8192), where memory consumption exceeds that of full fine-tuning. These findings clarify the limitations and trade-offs of FP8 for LoRA fine-tuning, and we will incorporate this analysis into the manuscript.\n\n### **W5: In the background on LoRA: you may want to mention that the adapters can be downcast.**\n\n$\\to$ Thank you for the suggestion. We will clarify that by adding “While prior methods focus on integrating LoRA to a quantized model, it is also possible to quantize the low rank adapters themselves, such as [C1].” \n\n[C1] Guo, Hang, et al. \"Intlora: Integral low-rank adaptation of quantized diffusion models.\" arXiv 2410.21759 (2024).\n\n### **W7: I would say what your method does earlier, at least at a high level.**\n\n$\\to$ We will add high-level description after the introduction, such as “We leverage FP8 arithmetics to accelerate the LoRA finetuning task, where we maximize the speedup by not keeping a physical LoRA weight. Instead, we use the quantization error space and shape it to imitate the effect of a LoRA update. Then, the update is migrated and merged to the original weight using top-k row selection, saving both the memory and the expensive update cost.” \n\n### **W8: Typo: Line 84, you say $\\tilde W$ is high precision**\n\n$\\to$ We will correct that as follows: $DQ(\\cdot)$ reconstructs the high-precision approximation of quantized weight ($\\tilde W$). \n\n### **W10: Worth mentioning LoftQ and LQLoRA on initialization from quantization error.**\n\n$\\to$ Thank you for the suggestion. LoftQ is cited after Eq. 9 (line 195), and we will additionally reference LQLoRA in the revised manuscript.\n\n### **W11: Error bars for evaluation.**\n\n$\\to$ Here are the experimental results with two-sigma error bars. \n\n**Table R2:** LLaMA-7B Experimental results with error bars\n|Method|Time per Step|Alpaca|2-sigma error|OASST1|2-sigma error|\n|---|---|---|---|---|---|\n|FP16|2.87s|0.3456|$\\pm$0.0130|0.3656|$\\pm$0.0164|\n|QLoRA|5.45s|0.3278|$\\pm$0.0099|0.3524|$\\pm$0.0125|\n|QA-LoRA|9.44s|0.3472|$\\pm$0.0065|0.3627|$\\pm$0.0036|\n|IR-QLoRA|8.27s|0.3488|$\\pm$0.0031|0.3652|$\\pm$0.0035|\n|TorchAO|2.18s|0.3401|$\\pm$0.0132|0.3611|$\\pm$0.0218|\n|FP6-LLM (E2M3)|2.72s|0.2351|$\\pm$0.0073|0.2479|$\\pm$0.0321|\n|FP6-LLM (E3M2)|2.72s|0.2301|$\\pm$0.0018|0.2452|$\\pm$0.0157|\n|Fishman et al.|2.29s|0.3445|$\\pm$0.0095|0.3605|$\\pm$0.0259|\n|FALQON|1.79s|0.3471|$\\pm$0.0017|0.3469|$\\pm$0.0051|\n||\n\nFollowing the NeurIPS paper checklist, we provide two-sigma error bars of our experimental results. Table R2 presents the MMLU accuracy on LLaMA-7B architecture. The experimental results show that our method shows consistent performance. Due to the character constraints, we provide partial results with error bars. However, we will put more experimental results with error bars in the manuscript. \n\n### **W12: Excessive 244MB uncompressed code and not all .git folders were deleted.**\n\n$\\to$ The immense size is from the large contents in the 3rd-party libraries (Transformers, PEFT, lm-evaluation-harness), which we modified for our framework. Please consider the large size of the code as our efforts to provide runnable code to the reviewers. Also, we found that the .git folder of the lm-evaluation-harness library is not deleted. However, it is from the public and popular GitHub repository. Therefore, it does not harm the double-blind policy of the conference.  \n\n### **Q1: What is special about FP8? Compared to say, INT4/NF4?**\n\n$\\to$ FP8 quantization accelerates **fine-tuning** by enabling faster matrix multiplications, leveraging native GPU support for both weights and activations. In contrast, INT4 and NF4 quantization are primarily used for reducing **inference** memory and are typically weight-only. They do not speed up fine-tuning since computations still use high-precision arithmetic. Thus, FP8 uniquely enables significant training acceleration, which is the focus of our work.\n\n### **Q2: Figure 1: What is behind the dashed lines, theoretical maximum, and pre-training speed up? What is the empirical setup?**\n\n$\\to$ The theoretical maximum of FP8 is the Tensor Core throughput ratio of GPU (e.g., 330.3TFLOPS/165.2TFLOPS = 2.0 for RTX4090), pretraining speed up is reported from LLaMA-70B pretraining with 8xH100 GPUs, and the empirical speed up is measured speed of LoRA layers with TorchAO baseline on an RTX4090 GPU. \n\n### **Q3: The sign of the error term flips from equation 8 to 9, keep it as negation?**\n\n$\\to$ We respectfully argue that equations 8 and 9 are mathematically correct; therefore, the sign should be flipped. The sign is flipped because we did transposition of the error term $\\Delta_Q W$ at equation 9: $DQ(\\tilde W) + \\Delta_Q W = W $ (Eq. 8) $\\Rightarrow DQ(\\tilde W) = W - \\Delta_Q W$. However, if we misunderstood the reviewer’s question, please feel free to add a comment and elaborate on this. \n\n### **Q4: Line 184-185: I don’t see another interpretation as possible, what do you mean?**\n\n$\\to$ We mentioned that the key idea of melded LoRA interprets quantization error $\\Delta_Q W$ as an additive tensor to the original high-precision weight $W$. We agree that it is a well-known interpretation of quantization error, and we apologize for the possible confusion. \n\n***\nWe appreciate the reviewer for the constructive review. We believe our rebuttal addresses the reviewer’s concerns and better highlights the contributions of our paper. We would be happy to discuss if there are any additional concerns regarding our rebuttal."}, {"review_id": "BZVVWef8QL", "reviewer": "Reviewer_UHn5", "summary": "This paper presents FALQON, a novel framework designed to accelerate LoRA fine-tuning for large language models by leveraging FP8 arithmetic. The key insight is that while FP8 quantization accelerates large matrix multiplications, it incurs significant overhead for small matrices like those used in LoRA, often negating performance gains. To address this, the authors propose a method to merge LoRA adapters directly into the FP8-quantized backbone weights, thus avoiding the computational cost of separate adapter paths. They introduce a melded LoRA formulation, which repurposes quantization error as a low-rank approximation, and develop an efficient gradient computation strategy that does not require explicitly storing adapter parameters. Moreover, a row-wise proxy update mechanism is introduced to selectively update meaningful components of the quantized backbone, further enhancing efficiency. Experimental results show that FALQON achieves up to 3× speedup compared to existing quantized LoRA approaches while maintaining comparable accuracy across multiple benchmarks.", "questions": "Motivation for FP8:\n- The paper argues that quantized LoRA methods have substantial overhead but does not fully justify the use of FP8 specifically for LoRA. Can the authors elaborate on the real-world scenarios or hardware settings where such speed gains justify potential accuracy losses or increased complexity?\n\nPreliminary Analysis Scope:\n- The discussion on when FP8 becomes beneficial (e.g., matrix size ≥ 4K) is theoretical. Could the authors include experimental data showing the turning point where FP8 outperforms FP16 in practice for LoRA modules? This would better inform practitioners when FP8 should be preferred.\n\nClarification on Table 1 Results:\n\n - Is QLoRA implemented as in the original paper with 4-bit NF4 quantization?\n\n - Why are QA-LoRA and IR-QLoRA slower than QLoRA despite having fewer trainable parameters, especially when QA-LoRA claims to outperform QLoRA in speed in its original paper?\n\nLoss Curve Comparisons:\n- Could the authors provide training loss curves for all quantized LoRA baselines compared to FALQON? This would help verify convergence behaviors, training stability, and potential trade-offs between speed and optimization quality.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "This paper introduces a technically solid framework, FALQON, to accelerate LoRA fine-tuning using FP8 arithmetic. The design is clear and thoughtful, particularly the use of quantization error as a low-rank approximation and the integration of LoRA adapters into the quantized backbone. The melded LoRA formulation and row-wise proxy update mechanism are original contributions that show strong potential to reduce overhead while preserving accuracy. The paper is clearly written, with well-structured explanations and informative figures that effectively convey the technical design and empirical results. Its contributions are timely and relevant, addressing the under-explored challenge of improving computational efficiency in parameter-efficient fine-tuning, which enhances its significance. \n\nHowever, the motivation to adopt FP8 training specifically for LoRA is not fully convincing — it remains unclear whether the potential accuracy trade-offs are justified across practical use cases. Furthermore, some inconsistencies in the experimental results raise questions. For instance, QA-LoRA and IR-QLoRA, which have fewer trainable parameters than QLoRA, appear slower in Table 1, contradicting their original papers. Clarification on these implementation details is necessary to assess the fairness and validity of the comparisons.", "was_revised": true, "final_justification": "Addressed my concerns", "rebuttal": "We appreciate the valuable comments from the reviewer. We also thank the reviewer for acknowledging our paper to be technically solid, the design is clear and thoughtful, and the paper is clearly written and the contribution is timely. \n\nThis rebuttal addresses the reviewer’s concerns and suggestions. Below is a brief summary of this rebuttal. \n\n* **W1, Q1-Motivation for FP8 is not fully convincing**: We elaborate on the real-world scenarios (4 tasks from 6 papers) in the paper (line 111-118). \n* **W2, Q3.2-Why are QA-LoRA and IR-QLoRA slower than QLoRA despite having fewer trainable parameters, especially when QA-LoRA claims to outperform QLoRA in speed in its original paper?**: The original claim is about inference speed. Our paper focuses on fine-tuning speed, which is determined by the computational overhead rather than the number of trainable parameters.\n* **Q2-Could the authors include experimental data showing the turning point where FP8 outperforms FP16 in practice for LoRA modules?**: We present the experimental results and the turning point is around rank 8192. \n* **Q3.1-Is QLoRA implemented as in the original paper with 4-bit NF4 quantization?**: Yes. \n* **Q4-Loss curve comparisons**: We provide a training loss table of every hundred steps instead of a figure, due to the limitation of the conference policy. \n\nWe will improve our manuscript to address the reviewer. Please refer to the following for the detailed rebuttal.\n\n***\n### **W1, Q1: Can the authors elaborate on the real-world scenarios or hardware settings where such speed gains justify potential accuracy losses or increased complexity?**\n\n$\\to$ For this, we can refer to the real-world scenarios provided in the paper (line 111-118), which trains up to millions of LoRA adapters, and therefore the speed up is especially beneficial. In lines 111-118, we present four scenarios that train a large amount of LoRA adapters for *personalization [54, 50, 26], multitask learning [44], domain adaptation [11], and multilingual summarization*. For example, PLoRA [54] assumes millions of users, and iLoRA [C1] conducts evaluation on a few hundred to 6 million users. Therefore,  we believe our framework significantly provides benefits to these scenarios.\n\n[C1] Kong, Xiaoyu, et al. \"Customizing language models with instance-wise lora for sequential recommendation.\" NeurIPS (2024).\n\n### **W2, Q3.2: QA-LoRA and IR-QLoRA, which have fewer trainable parameters than QLoRA, appear slower in Table 1, contradicting their original papers. Why are QA-LoRA and IR-QLoRA slower than QLoRA despite having fewer trainable parameters, especially when QA-LoRA claims to outperform QLoRA in speed in its original paper?**\n\n$\\to$ We respectfully disagree that our results contradict the baselines’ original papers. The QA-LoRA claims that they are faster in the **inference** stage, while we report the **fine-tuning** speed. \n\nIn the original QA-LoRA paper, they claimed as *“The major advantage of QA-LoRA, compared to QLoRA, lies in the **inference** stage where it is faster.”* on page 5. This clearly shows that they mentioned the inference stage. However, in our target scenario, the fine-tuning speed is primarily determined by computational overhead, rather than the number of trainable parameters. As QA-LoRA and IR-QLoRA utilize auxiliary computations in addition to QLoRA, they are slower than QLoRA in **fine-tuning**, but they are faster in the **inference stage** due to optimization focused on inference efficiency. We believe this does not contradict the original paper, but aligns with their proposed algorithms. \n\n### **Q2: Could the authors include experimental data showing the turning point where FP8 outperforms FP16 in practice for LoRA modules?**\n\n$\\to$ The turning point of the LoRA adapter is rank 8192, which is an exceptionally large rank compared to common settings. We provide a detailed analysis across LoRA rank in Table R1. \n\n**Table R1:** Efficiency analysis across various LoRA ranks\n| |Rank:|16|32|64|128|256|512|1024|2048|4096|8192|16384|\n|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|LLaMA-7B|Speed Up|0.50$\\times$|0.55$\\times$|0.44$\\times$|0.48$\\times$|0.43$\\times$|0.62$\\times$|0.69$\\times$|0.60$\\times$|0.80$\\times$|**1.18$\\times$**|**1.22$\\times$**|\n|FT: 49.01GB|LoRA Mem.|0.67*GB*|0.78*GB*|0.98*GB*|1.39*GB*|2.20*GB*|3.83*GB*|7.09*GB*|13.60*GB*|26.63*GB*|**52.70*GB***|**104.82*GB***|\n|LLaMA-13B|Speed Up|0.55$\\times$|0.59$\\times$|0.50$\\times$|0.52$\\times$|0.46$\\times$|0.48$\\times$|0.64$\\times$|0.61$\\times$|0.88$\\times$|**1.21$\\times$**|**1.20$\\times$**|\n|FT: 95.48*GB*|LoRA Mem.|1.31*GB*|1.51*GB*|1.91*GB*|2.70*GB*|4.29*GB*|7.46*GB*|13.81*GB*|26.50*GB*|51.89*GB*|**102.66*GB***|**204.21*GB***|\n||\n\nThe experiments compare the computation time of LoRA adapters on FP16 and FP8 arithmetic. In addition, we provide the memory consumption of full fine-tuning (FT) and LoRA adapter to show the efficiency in relation to LoRA rank. Even if FP8 computation is beneficial in ranks 8192 and 16384, the memory consumption on those ranks exceeds that of full fine-tuning, making LoRA inefficient.\n\n### **Q3.1: Is QLoRA implemented as in the original paper with 4-bit NF4 quantization?**\n\n$\\to$ Yes, we used the NF4 quantization and also used the original code provided by the QLoRA’s authors. \n\n### **Q4: Could the authors provide training loss curves for all quantized LoRA baselines compared to FALQON?**\n\n$\\to$ Unfortunately, we could not present the figures due to the NeurIPS conference policy this year. Therefore, we provide a table showing the average training loss of every hundred steps (Table R2).\n\n**Table R2:** Loss comparison across training steps\n|Steps|100|200|300|400|500|600|700|800|900|1000|1100|1200|1300|1400|1500|1600|1700|1800|\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|FP16|4.379|4.290|4.337|4.361|4.235|4.326|4.244|4.160|4.194|4.341|4.227|3.971|4.156|4.194|4.203|4.156|4.047|4.003|\n|QLoRA|4.397|4.302|4.347|4.371|4.244|4.335|4.252|4.169|4.202|4.349|4.235|3.979|4.164|4.202|4.211|4.164|4.055|4.009|\n|TorchAO|4.382|4.293|4.339|4.364|4.237|4.328|4.246|4.162|4.196|4.343|4.230|3.975|4.161|4.198|4.207|4.160|4.054|4.012|\n|Fishmanetal.|4.381|4.291|4.338|4.363|4.236|4.327|4.245|4.161|4.195|4.342|4.229|3.974|4.159|4.197|4.205|4.158|4.051|4.008|\n|FALQON|5.009|4.653|4.544|4.393|4.361|4.368|4.405|4.378|4.406|4.476|4.387|4.346|4.367|4.418|4.384|4.467|4.329|4.378|\n||\n\nOverall, the results show the training curve of each method. The magnitude of the loss is similar to non-quantized FP16 LoRA, and the loss is reduced as the training continues. Also, please note that the training loss is not directly related to the performance of the downstream task (e.g., MMLU accuracy), due to the nature of the language models. Therefore, a larger training loss does not necessarily mean lower quality of the model. \n\n***\nWe thank the reviewer for the helpful comments. We believe that our rebuttal well addresses the reviewer’s concern and potentially clarifies our contribution. We welcome further comments and would be happy to discuss them."}]

Author rebuttal:

Meta-review: This paper proposes FALQON, a framework to accelerate LoRA fine-tuning by using FP8 arithmetic. The key idea is to merge LoRA adapters into FP8-quantized backbone weights, avoiding the overhead that makes FP8 inefficient for small matrices and introducing a tailored gradient method for the merged design. The paper accomplishes this by viewing quantization error as an additive component of quantized weights, which allows LoRA adapters to be included in the FP8-quantized weights. Because the adapters are merged into the backbone, standard autograd does not directly compute their gradients. The authors address this by introducing a customized gradient computation strategy tailored to the melded structure, which further reduces overhead. After extensive rebuttals and new experiments introduced by the authors, reviewers appreciated the problem focus, experiments, and improvements over prior art. 

Comments: 

1. Expand the caption for Fig 1 to cater to a more general AI/ML audience. For instance, this caption can mention finetuning explicitly and detail how these speed-up trendlines were evaluated. Also, "Up-Proj" and "Down-Proj" are not defined in the paper. 
2. Include all new experiments that were introduced in the rebuttals to improve the manuscript.
Decision: Accept (poster)