Title: FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic
Venue: NeurIPS 2025
Accepted: yes
Keywords: FP8

Quantization

LLM

LoRA

Quantized training
Abstract: Low-bit floating-point (FP) formats, such as FP8, provide significant acceleration and memory savings in model training thanks to native hardware support on modern GPUs and NPUs. However, we analyze that FP8 quantization offers speedup primarily for large-dimensional matrix multiplications, while inherent quantization overheads diminish speedup when applied to low-rank adaptation (LoRA), which uses small-dimensional matrices for efficient fine-tuning of large language models (LLMs). To address this limitation, we propose FALQON, a novel framework that eliminates the quantization overhead from separate LoRA computational paths by directly merging LoRA adapters into an FP8-quantized backbone during fine-tuning. Furthermore, we reformulate the forward and backward computations for merged adapters to significantly reduce quantization overhead, and introduce a row-wise proxy update mechanism that efficiently integrates substantial updates into the quantized backbone. Experimental evaluations demonstrate that FALQON achieves approximately a 3$\times$ training speedup over existing quantized LoRA methods with a similar level of accuracy, providing a practical solution for efficient large-scale model fine-tuning. Moreover, FALQON’s end-to-end FP8 workflow removes the need for post-training quantization, facilitating efficient deployment. Code is available at https://github.com/iamkanghyunchoi/falqon.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents FALQON, a novel framework designed to accelerate LoRA fine-tuning for large language models by leveraging FP8 arithmetic. The key insight is that while FP8 quantization accelerates large matrix multiplications, it incurs significant overhead for small matrices like those used in LoRA, often negating performance gains. To address this, the authors propose a method to merge LoRA adapters directly into the FP8-quantized backbone weights, thus avoiding the computational cost of separate adapter paths. They introduce a melded LoRA formulation, which repurposes quantization error as a low-rank approximation, and develop an efficient gradient computation strategy that does not require explicitly storing adapter parameters. Moreover, a row-wise proxy update mechanism is introduced to selectively update meaningful components of the quantized backbone, further enhancing efficiency. Experimental results show that FALQON achieves up to 3× speedup compared to existing quantized LoRA approaches while maintaining comparable accuracy across multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper introduces a technically solid framework, FALQON, to accelerate LoRA fine-tuning using FP8 arithmetic. The design is clear and thoughtful, particularly the use of quantization error as a low-rank approximation and the integration of LoRA adapters into the quantized backbone. The melded LoRA formulation and row-wise proxy update mechanism are original contributions that show strong potential to reduce overhead while preserving accuracy. The paper is clearly written, with well-structured explanations and informative figures that effectively convey the technical design and empirical results. Its contributions are timely and relevant, addressing the under-explored challenge of improving computational efficiency in parameter-efficient fine-tuning, which enhances its significance. \n\nHowever, the motivation to adopt FP8 training specifically for LoRA is not fully convincing — it remains unclear whether the potential accuracy trade-offs are justified across practical use cases. Furthermore, some inconsistencies in the experimental results raise questions. For instance, QA-LoRA and IR-QLoRA, which have fewer trainable parameters than QLoRA, appear slower in Table 1, contradicting their original papers. Clarification on these implementation details is necessary to assess the fairness and validity of the comparisons.\n\nQUESTIONS:\nMotivation for FP8:\n- The paper argues that quantized LoRA methods have substantial overhead but does not fully justify the use of FP8 specifically for LoRA. Can the authors elaborate on the real-world scenarios or hardware settings where such speed gains justify potential accuracy losses or increased complexity?\n\nPreliminary Analysis Scope:\n- The discussion on when FP8 becomes beneficial (e.g., matrix size ≥ 4K) is theoretical. Could the authors include experimental data showing the turning point where FP8 outperforms FP16 in practice for LoRA modules? This would better inform practitioners when FP8 should be preferred.\n\nClarification on Table 1 Results:\n\n - Is QLoRA implemented as in the original paper with 4-bit NF4 quantization?\n\n - Why are QA-LoRA and IR-QLoRA slower than QLoRA despite having fewer trainable parameters, especially when QA-LoRA claims to outperform QLoRA in speed in its original paper?\n\nLoss Curve Comparisons:\n- Could the authors provide training loss curves for all quantized LoRA baselines compared to FALQON? This would help verify convergence behaviors, training stability, and potential trade-offs between speed and optimization quality.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work suggests merging LoRA adapters into an FP8 quantized backbone to speed up quantized fine tuning with FP8. Instead of running a separate FP16 path for the two LoRA multiplications in $BAx$ the factor $\\hat{A}$ is folded into the FP8 matrix so a single operations produces both of the multiplications with $x$, $\\widetilde{W}x$ and the LoRA factor $\\hat{A}x$.\nThis means that gradients are only needed for $\\hat{B}$ and conversions between FP16 and FP8 are limited, providing a computational speedup. Experiments confirm wall time speedup and not large deviation from task evaluations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The approach makes sense, removing these redundant operations is valuable (and could be implemented with a fused kernel for larger improvements?)\n* The low rank treatment has been used before LoftQ, LQLoRA, LoQT, initialising with SVD is empirically sound\n* Cutting memory and compute is important\n* Experimental comparisons and ablations are well rounded, good to see both performance and speed checked\n\nWeaknesses\n* Would have been good to have some sort of full picture overview or the key derivations together to get a concise overview, either with equations, a diagram or a combination of the two. I found it quite hard to follow. I think a more pedagogical approach could have been taken. The contribution is hidden away until page 5! \n* Writing and grammar can be improved. Some sentences are very long and logical flow is not always evident, e.g. line 92. Other sentences are vague, e.g. line 149.\n* You say in the introduction that FP8 quantization overhead outweighs the benefits from FP8 multiplication. I think you can make it clearer it is the speed and not the memory you refer to\n* Same in claims about cost of FP8<->FP16 in line 104-105. You say there will be a detailed analysis in section 4, but it is somewhat preliminary (as the section title says).\n* In the background on LoRA: you may want to mention that the adapters can be downcast.\n* In some places it would be good to make it clear you are talking about computational speed and not memory as is often the focus of the low rank papers.\n* I would say what your method does earlier, at least at a high level. You keep mentioning it is better (which gets repetitive), but only at page 5 do you say how it works!\n* Typo: Line 84, you say $\\widetilde{W}$ is high precision\n* Figure 2: For this to be easier to read it might be good to include the operations (addition and multiplication)\n* Equation 9 uses the same approach as LoftQ and LQLoRA to initialize the quantization error, if I'm not mistaken, maybe worth mentioning both\n* No error bars, you could have at least bootstrapped intervals for the evaluations\n* Note on code: it’s 244MB uncompressed! That seems very excessive. Also, not all .git folders were deleted.\n\nQUESTIONS:\n* What is special about FP8? Compared to say, INT4/NF4?\n* Section 4 / Figure 1: What is behind the dashed lines, theoretical maximum, and pre-training speed up? What is the empirical setup? Do you take random matrices of random dimensions? This is not clear.\n* The sign of the error term flips from equation 8 to 9, keep it as negation?\n* Line 184-185: I don’t see another interpretation as possible, what do you mean?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies an interesting performance bottleneck in reduced-precision large language model (LLM) fine-tuning: while low-bit formats like FP8 can significantly speed up large matrix multiplications, their inherent quantization overheads lead to a net slowdown when applied to the small-dimensional matrices used in Low-Rank Adaptation (LoRA). To address this, the authors propose FALQON, a framework that eliminates the need for a separate computational path for LoRA adapters. The core idea is to \"meld\" the LoRA adapter directly into the FP8-quantized backbone weights from the start, using the initial quantization error as a form of adapter initialization. The framework reformulates the forward and backward passes to compute gradients efficiently within this merged structure and introduces a selective, row-wise update mechanism to apply only the most significant changes. The experimental results demonstrate that FALQON can achieve up to a 3x training speedup over existing quantized LoRA methods while maintaining comparable accuracy on various benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- This paper pinpoints the less-explored inefficiency of FP8 acceleration for LoRA finetuning. It was interesting to note that LoRA would suffer from quantization overhead due to its small dimensional arithmetic. \n\n- The central idea of \"melded LoRA\" seems to be interesting as it would avoid the separate LoRA path entirely.\n\n\nWeaknesses\n- The paper's central premise—the need to quantize the LoRA path to FP8—feels somewhat constructed. LoRA adapters are intentionally small, and standard FP16 LoRA is a very strong and widely used baseline. The motivation for forcing the entire pipeline into an FP8 format, which creates the very problem this paper solves, is not sufficiently justified. The paper would be stronger if it presented compelling use cases (e.g., specific hardware constraints, or scenarios where training thousands of adapters simultaneously makes the FP16 overhead non-trivial) where this end-to-end FP8 acceleration is a critical, practical need rather than an academic exercise.\n\n- The most significant weakness is the omission of the standard FP16 LoRA as a performance baseline in the results tables (Tables 1, 2, 3). FALQON introduces several approximations (e.g., initializing from quantization error, fixing the adapter matrix A, and only updating B, avoid maintaining full-precision weight) that could potentially degrade fine-tuning quality. Without a direct comparison to the de-facto standard, it is impossible to assess whether the impressive speedup comes at a hidden cost to model performance. The claim of maintaining \"comparable accuracy\" is only validated against other quantized methods, which may already be degraded.\n\n- The paper suffers from a lack of clarity in its experimental reporting, hindering a full understanding of the results.  \n  - The experimental settings for baselines like QLoRA and TorchAO are ambiguous. The precision of the backbone, the LoRA adapters (presumably FP16 for QLoRA?), and optimizer states should be explicitly stated for all methods.\n  - The explanation for why TorchAO causes an OOM on the 13B model while FALQON does not is unclear. If FALQON's key contribution is to improve LoRA efficiency, how come it is more memory efficient than TorchAO (which does not have LoRA at all)? \n- The work's contribution is heavily weighted towards systems-level optimization and implementation efficiency rather than fundamental algorithmic advances in machine learning. While it involves algorithmic changes, the core novelty lies in overcoming a hardware-specific bottleneck. As such, the paper might be a more natural fit for a systems-focused conference like MLSys or DAC, rather than a general ML conference like NeurIPS.\n\nQUESTIONS:\n- The row-wise update mechanism introduces a top-k operation on the gradient matrix ΔB at every training step. This operation is not free. Could you provide an analysis of the computational overhead of this top-k selection and discuss how it impacts the overall wall-clock time speedup claimed by FALQON?\n\n- The dimensions (e.g., rank r) for the micro-benchmark in Figure 1a are not specified.\n\n- The trend in Figure 1b, where efficiency does not improve with increasing rank, is counter-intuitive and contradicts the paper's own analysis without further explanation.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the benefits and overhead of FP8 utilization for low-rank quantization. The authors observed that the speed-up gained by using FP8 is offset by the quantization overhead due to the small dimension of low-rank adapters. To address this issue, the paper proposes the FALQON framework, which interleaves an FP8-quantized backbone with LoRA adapter models. This integration requires modifications to the forward path and backward pass for gradient computation, as proposed in the paper. The results show an improvement of 1.8x to 3x for LLM models compared to using LoRA and FP8 quantization separately. An ablation study over various training parameters such as learning rate, batch size, and number of ranks is also presented.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1- The proofs for the claims in the paper are presented clearly.\n\n2- The idea behind the efficient gradient calculation by combining the gradients of the weights and the low-rank matrix (A) is novel and significant.\n\nWeaknesses: \n\n1- The new approach seems to be valuable when multiple agents use multiple quantized LoRA adapters, as mentioned in the paper. It is suggested that multiple LoRA adapter fine-tunings are evaluated on a specific benchmark in the paper, demonstrating the trade-off between performance and the number of quantized LoRA adapters.\n\n2- FP4 fine-tuning has been suggested in previous papers [1,2,3], and it would be helpful to include it in the experiments section as a low-bit FP quantization baseline for comparison with the proposed approach.\n\n[1] Liu, Lian, et al. \"COMET: Towards Practical W4A4KV4 LLMs Serving.\" Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 2025.\n\n[2] Wang, Jie, et al. \"Fp4-quantization: Lossless 4bit quantization for large language models.\" 2024 IEEE International Conference on Joint Cloud Computing (JCC). IEEE, 2024.\n\n[3] Roy, Somnath. \"Understanding the impact of post-training quantization on large language models.\" arXiv preprint arXiv:2309.05210 (2023).\n\nQUESTIONS:\nIn the Efficient Gradient Computation section, it is mentioned that gradients are computed only for matrix B, which is empirically efficient for effective fine-tuning. Could you elaborate on this point in more detail and explain why gradient computation for matrix A is not required?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-06"}