LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuning
Authors: Nurbek Tastan
Stefanos Laskaridis
Martin Takáč
Karthik Nandakumar
Samuel Horváth
Venue: ICLR 2026 Poster (2026)
Keywords: parameter-efficient fine-tuning
low-rank adaptation
llms
large models
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Large pre-trained models are commonly adapted to downstream tasks using parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA), which injects small trainable low-rank matrices instead of updating all weights. While LoRA dramatically reduces trainable parameters with little overhead, it can still underperform full fine-tuning in accuracy and often converges more slowly. We introduce LoFT, a novel low-rank adaptation method that behaves like full fine-tuning by aligning the optimizer's internal dynamics with those of updating all model weights. LoFT not only learns weight updates in a low-rank subspace (like LoRA) but also properly projects the optimizer's first and second moments (Adam's momentum and variance) into the same subspace, mirroring full-model updates. By aligning the low-rank update itself with the full update, LoFT eliminates the need for tuning extra hyperparameters, e.g., the LoRA scaling factor $\alpha$. Empirically, this approach substantially narrows the performance gap between adapter-based tuning and full fine-tuning and consistently outperforms standard LoRA-style methods, all without increasing inference cost. The code is available at https://github.com/tnurbek/loft.

Published as a conference paper at ICLR 2026
LOFT: LOW-RANK ADAPTATION THAT BEHAVES LIKE FULL MUN-TNGNIG
Nurbek Tastan' Stefanos Laskaridis Martin Takáč'
Karthik Nandakumar Samuel Horváth'
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE
2 "Amazon Science, UK
*Michigan State University, USA
Bourbek.tastare martin. hartin.takac, horvath) heterrattthmmmuuumaeaea @mbzuai. acc.acc... ae ma mail@stefanos.co cc, nandakumêmsu. edu
ABSTRACT
Large pre-trained models are commonly adapted downstream tasks using parameter-effiiient fine-tuning methods such as Low-Rank Adaptation (LoRA), which injects small trainable low-rank matrices instead of updating all weights. While LoRA dramatically reduces trainable parameters with little overhead, it can still underperform full fine-tuning in accuracy and often converges more slowly, We introduce LoFT, novel low-rank adaptation method that behaves like full fine-tuning by aligning the optimizer's internal dynamics with those of updating all model weights. LoFT not only learns weight updates in a low-rank subspace (like LoRA) but also properly projects the optimizer's 's first and second moments (Adam's 's momentum and variance) into the same subspace, mirroring full-model updates. By aligning the low-rank update itself with the full update, LoFT elim- inates the need for tuning extra hyperparameters, e.g., the LoRA scaling factor a. Empirically, this approach substantially narrows the performance gap between adapter- based tuning and full fine-tuning and consistently outperforms standard LoRA-style methods, all without increasing inference cost. The code is available at https://githab..co//ubbb///////
INTRODUCTION
Fine-tuning large-scale pre-trained models for specific tasks has become standard paradigm in natural language processing and other domains. However, as model sizes grow into the billions of parameters, full fine-tuning (i.e., updating every weight) becomes computationally expensive and impractical, especially in multi-task (Chronopoulou et al. 2023) or multi-user (Yi et al. 2023) settings. crrameter-efffcieee fine-tuning (PEFT) techniques address this challenge by updating only a small subset of parameters while reusing the vast majority of pre-trained weights. Among these, Low-Rank Adaptation (LoRA) has emerged popular and effective solution. LoRA freezes the original weights and injects trainable low-rank matrices into selected layers, substantially reducing the number of learnable parameters. Remarkably, LoRA often matches and sometimes can exceed the performance of full fine-tuning on certain benchmarks, all while incurring minimal runtime overhead and no additional inference latency. This makes it an attractive alternative other methods like sequential adapters (Houlsby et al. PPeiifer et al., (202T), which typically introduce new layers and increased latency. Despite success, LoRA and similar low-rank approaches still fall short of full fine-tuning in some settings. Empirical studies have reported a persistent performance gap and slower convergence rates compared to full-model updates (Biderman al.) 2024 Wang et al. 2024). These gaps indicate that the optimization dynamics of LoRA differ in important ways from those of full fine-tuning. Recent work (Liu et al., 2024; Wang et al. 2025) has attempted to close this gap by focusing on more accurate gradient approximations within the low-rank subspace. This is motivated by the observation that LoRA's updates can omit or misestimate important directions in the full gradient, leading to suboptimal solutions. In this work, we demonstrate that this is only part of the story: optimizer state misalignment specifically in the first and second moments used by
Work done independently of Amazon.

---

Published as a conference paper at ICLR 2026
Full Fine-Tuning AdamW
Full
U-quiate
LoFT-AdamW
Sequentaly projected an vin I1 Py: Projection
Velk,
Mameraum:
Momeritam:
W
6s) It 10
(1 Ei
W
+
C'
F
Swezrrd monnrrd Ed. p5 m
Swardd momeent
El 2 of
Adam Upeine:
LoFT Upenne
ma/(1
mh//1
West W2
Un- We
Pis
Alsays Uppieed
v-w((1-89 - ++
/6/01 -
AbMeTnEtT Mictate
Figure 1: LoFT visualization. LoFT can be interpreted as the tightest approximation to full fine- tuning under the constraint that each update lies the subspace defined by V (when updating U). The LoFT-Adam W update consists of momentum and second- moment estimate constructed using projected gradients. The final update is then projected back onto the subspace of V to respect the low-rank constraint. When V is the updated component instead of U, the roles of U and V are simply exchanged, and the update is applied to W instead of W.
AdamW (Loshchilov & Hutter 2019), the de facto optimizer in large-scale training also plays critical role. When these internal statistics are not properly aligned with the low-rank constraint, undermines the effecttivnness of the adaptation.
Finally, practical complication in standard LoRA is the introduction of scaling hyperparameter, a, often normalized by the rank. This scaling factor modulates the contribution of the low-rank update and must be carefully tuned. Improper settings can lead to poor performance or even divergence by overpowering the backbone model (Lee et al., 2025] MMlinovsky al.) (024). Altogether, these challenges i.e., the gradient and optimizer state misalignment, as well as the additional hyperparameter sensitivity limit LoRA's ability to fully replicate the robustness and effectiveness of unconstrained full fine-tuning.
Our main contributions are summarized as follows:
We identify that not only gradients but also optimizer states (i.e., first and second moments) suffer from misalignment when approximating full fine-tuning with low-rank updates.
We propose Low rank adaptation that mimics Full fine-Tuning (LoFT), novel LoR A-based opti- mizer that addresses these issues by closely approximating full fine-tuning across all optimization dimensions. LoFT consists of five core components: gradient scaling, alternating updates, optimizer state calibration, construction of projected full fine-tuning update followed by low-rank projection, and projected full fine-tuning-awarc clipping.
To the best of our knowledge, LoFT is the first low-rank adaptation method that exactly reduces to
AdamW (Loshchilov & Hutter 2019) in the full-rank limit.
We conduct extensive experiments on both synthetic and real-world tasks across multiple modalities, demonstrating the effectiveness and generality of LoFT.
METHOD
We focus on the standard fine-tuning setup, where pre-trained model is adapted a downstream task. In full fine-tuning, each weight matrix W is updated by full-rank increment AW. To reduce computational cost, LoRA proposes low-rank reparameterization
W Wo + AW W. + +U
where W E Rmxn U R V Rexr, and < min min,m,,r) n.. Only U and V trainable. reducing the gradient and optimizer state footprint to O(mm + n)r) compared to O(mn) in full fine-tuning. LoRA typically introduces scaling factor > modulate the magnitude of the low-rank update. However, in our study, we set = and attribute the need for this hyperparameter misalignment between LoRA and full fine-tuning, which address in the subsequent sections.
2.1 GRADIENT DESCENT FOR FULL FINE-TUNING LORA
Let f(W): m + E denote a scalar loss function, with W representing the parameters of single linear layer. standard full fine- the-tunnng with gradient descent, the one-step update
W W V (WW)
(1)
2

---

Published as a conference paper at ICLR 2026
Table The six core building blocks of LoFT for aligning low-rank adaptation with full fine- lin-ttunigg.
Component
Purpose
Alternating Updates
Eliminate second order cross terms from LoRA dynamics.
Gradient Scaling
Ensure scale -nvariance of low-rank updates.
Optim. States Calibration
Align moments estimates across changing low-rank spaces.
Projected Full Update
Reconstruct the full-model update and project onto the low-rank subspace. Match full fine-tuning clipping behavior.
Gradient Clipping
where 7 > 0 is the learning rate, and V w (W) is the gradient of the loss with respect to W. With LoRA parametrization, the one-step update becomes
W+ = Wo + U++++)+++++ Wo + (U
(2)
Applying the chain rule yields
Vuf(W)= V
Substituting these into
gives
W W 7 (VwJ(W)VVV + V / Vw T(W).
(3)
Equation 33) highlights the first discrepancy between LoRA and full fine-tuning: the additional r/
term, which depends quadratically on the gradient. While seemingly small, this term can materially affect convergence,
show later in a controlled experiment. A straightforward way to eliminate this term through alternating updates.
Building Block 1: Alternating Updates
Do not update U and V simultaneouslyy but perform alternating updates.
Without loss of generality, assuming we update only U, the resulting update to W becomes
W+ = W VWW
However, this update suffers from scale ambiguity: for any c & O, UV (cU) (V/e) but the update scales differently with To resolve this, observe that the update direction always lies in the column space of V, allowing us scale the update using an x matrix (VTV)-
W+ = W - Vw / VVT = W - vV (WW)s
(5)
is the projection matrix onto the column space of V. This ensures the update is the closest low-rank approximation to Vw f(W under the given subspace. The associated computational cost is Onr? r)) This update defines our second building block.
where Pv
Building Block 2: Use Scaled Gradients
- vo.
-v/(W)= (w)
We are not the first suggest this: Zhang Pilanci 2024 derived similar result from the perspective of Riemannian optimization.
2.2 FIRST MOMENT MISALIGNMENT
In practice, gradients often estimated using momentum. Specifically, the first moment my is computed as my = 81m--1 + (1 - Br 31)gk: where 31 E 10 is the momentum coefficient and gl is
We assume V of full rank. If not, the pseudo-inverse.

---

Published as a conference paper at ICLR 2026
the stochastic gradient, and the subscript denotes iteration counter. For full fine-tuning, the resulting momentum update
mg = (1 (1) (WW.).
(6)
2=0
When updating U under the LoRA parameterization, the effect on W becomes
mV V = 31 8
C VVof(W,VV = (1 (1-3))
31)
Σ P V WwfffwWWy (V,
ViT 'V..
which does not represent proper projection due to the mismatch between Vi and Vg- To address this, we introduce a recalibration step
mi = Bim Bim + 81) (()),,(Wk),
where C/ def Vi)(VV Vx
is a calibration matrix. Substituting this back gives
k PI, (1 (1-3:) Σ
m = m Vil
81
Σ 8 V V ((W,
5 9
(8)
where 11 Pv, is the sequential projection. Let 9.¹ provides the tightest possible estimate (in l distance) of the momentum under the constraints of evolving low-rank subspaces defined by V/'s. Storing previous iterates {Vk-1, Uk-1) - incurs an additional memory cost of O((m + n)r).
Vw f(W) I1 Pv, This expression
Building Block 3: Recalibrate Momentum
U 3,m + 31 f(W.), m V
m -.mmk + - 81 vv
2.3 SECOND MOMENT MISALIGNMENT
Analogically, for Adam-style updates, the ideal update to W when U is being updated, given subspace constraints, would be
# 8/1--0)) Pv, s.t. m (1 (1- B1) //11--5)+ +
Σ
3 sk-ig
= 32)
3 Σ 1=0
9
9
(9)
1=0
where m is as defined in 8 and oh (1 (1- (1-3)) Σx gk- ogy is the second moment estimate.
The symbol & denotes element- the subspace defined by Vk, since this is a necessary constraint due to the update rule; see (4) To compute if efficiently, we use the following identities from Slyusar 1999)
multiplication. Note that this update is constructed lie in
(A (C & D) (AC) (BD), (AB) O (CD) (A C) D)
(10)
where @ is the Kronecker product, is the transposed Khatri-Ran Rao product, and is the standard Khatri-Rao product. We define the calibrated second-moment accumulator
U Pk 1ark-1
CK) (1 (o) V f(W)),
(11)
where o is matrix of size nr X r that stores the cross-terms necessary to reconstruct the second moment after transformation. The associated memory overhead is O( n)r²), which is the main limitation of our approach. For this reason, maintaining small rank r is crucial for memory efficiency. In practice, this constraint is acceptable as long as r S / /min (m, n}. which we find to be both reasonable and sufficient for capturing effective low-rank updates. In the experiment, LoFT leads to the memory increase of up to 25.65% compared to LoRA (Hu al. 2022). but improves or matches the memory of more performant DoRA (Liu et al. 2024). Furthermore, observe
4

---

Published as a conference paper at ICLR 2026
that omitting accond-moment calibration limits memory increase to less than 6% relative to LoRA and only incurs marginal performance degradation (~~ 0.1%). Details are provided in Appendix E.2
Finally, in Appendix D.I introduce version of LoFT based the Muon (Jordan et al. 2024 optimizer that does not exhibit this issue, as all optimizer states are linear functions of gradients.
Building Block 4: Second Moment Alignment
Use cross-terms for second moment accumulation to enable second moment recalibration
p i (C CC) + (1 (1- (1-8)) Vuf(W,)).
(12)
Pk = 3 & + (1- VvS(W)))
Using pK we compute oy = p ( (Vi * Vy.) and apply the following update.
Building Block 5: Reconstruct Full Update Followed by Projection
For the Adam version of LoFT, update U and V as
m Vv (1-
Uk+1 = Uk
VA (V V+
V (V2 VV++v +
(13)
m (1-
Vi+1 = V/ Tl (Uh(Uuua)//a-0p) +
Un Uz)
2.4 GRADIENT CLIPPING AND WEIGHT DECAY
We apply no special modifications weight decay. Since only one of or V updated at time, the effect of standard weight decay correctly reduces the low-rank update as UVT + (1- UV)UV. The full AdamW-LoFT algorithm is provided the appendix. With all six building blocks described above, LoFT-Adam W recovers full fine-tuning when r max(mn n and Un. Vv are full-rank. To our knowledge, LoFT is the first low-rank adaptation method that provably recovers full fine- fime-tuning.
Building Block 6: Gradient Clipping
To approximate full fine-tuning during gradient clipping, when updating U, we use Wus(WVVT = (W)(W)P as the effective gradient for the corresponding layer W.
2.5 SIMULATED EXPERIMENT
In the previous remark, we argued that for full-rank adaptation, LoFT recovers full fine-tuning, We now demonstrate that if the target solution is low-rank, LoFT matches the performance of full fine-tuning the correct rank is selected. We consider the op- timization problem /(W)- = ||W AlE where a randomly generated matrix with rank(A) r. We compare LoFT, LoRA, and full fine-tuning using the AdamW optimizer. To demonstrate how LoFT can efficiently approximate full fine-tuning, the step size is tuned for full fine-tuning and reused for all baselines. We initialize W 0, and for LoFT we follow the standard LoRA initialization (Hu et al.) 2022), which also yields UV
10a
10a
10
10-+
10
LoFT (No Alternate Update)
LoFT (No State Calibration)
10-10
LoFT
Fine-Tuning
50
200
250
Steps
Figure Comparison LoRA, LoFT, and Full Fine-tuning with Adam f(W) IW-1B
initially. We set
1024, 512, and 8. In addition to LoFT and LoRA. also include ablated variants of LoFT to highlight the importance of its design components: one without alternating updates, and one without optimizer state calibration. As shown in Figure LoFT closely matches the performance of full fine-tuning. In contrast, omitting any its core components leads to significantly slower convergence and worse final performance, confirming the necessity of the full LoFT design.
5

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison of parameter-efficiet fine-tuning methods, LoRA, DoRA, and our method LOFT, suite of commonsense reasoning benchmarks using LLaMA-7B, LLaMA2-7B, and LLaMA3-8B models. The table reports accuracy scores across multiple tasks with average performance shown in the final column. denotes the rank used in the respective adaptation method. Bold and underlined scores highlight the best and second-best performance per task, respectively.
Model
Method
BoolQ
PIQA
SIQA
HS
WG
ARC-C
ARC-E
OBQA
avg.
LoRA,-16
65.38
76.71
75.69
79.81
68.03
65.27
80.30
77.40
73.57
D
54.13
73.94
79.38
58.01
79.40
64.68
79.76
79.60
71.11
LoFT.
68.62
82.80
78.27
82.69
73.32
64.30
80.26
78.40
76.08
LLaMA-7B
LoFT
67.34
80.96
76.20
80.50
76.40
63.62
79.21
75.40
74.95
LoFT,
68.03
79.16
75.84
78.86
76.24
64.51
78.03
71.00
73.96
LoFT,
67.09
78.35
74.46
76.14
74.82
58.87
76.85
70.80
72.
LoRA, 16
50.09
59.03
76.41
65.45
77.51
64.68
79.12
77.20
68
DoRA,-16
71.93
82.92
79.22
88.90
83.03
66.98
82.70
82.00
79.71
10
71.80
83.51
79.02
90.59
82.72
70.65
84.43
$1.00
80.46
LLaMA2-7B
LoFT.=4
70.49
81.94
79.80
88.85 85
81.37
69.11
84.88
79.80
79.53
LoFT,
70.55
81.18
77.74
83.01
79.01
66.72
82.83
78.80
77.48
LoFT,-1
68.69
80.58
76.36
72.95
76,80
641.88
82.37
77.20
84.88
LoRA,
74.46
88.14
81.37
94.81
85.08
80.72
89.18
86.00
$4.97
DoRA,-16
74.56
88.52
80.09
95.17
86.74
79.78
90.19
84.60
LoFT,-16
75.63
88.85
80.35
95.64
86.11
80.89
91.16
86.40
85.63
LLaMA3-8B
LoFT.=4
74.53
88.52
80.04
95.45
85.32
78.92
89.73
84.20
84.59
LoFT.
73.76
87.11
79.84
94.72
84.29
79.61
80.98
84.60
84.24
LaFT
69.33
87.49
79.27
93.79
84.06
76.1
87.12
82.20
82.42
3
EXPERIMENTS
We conduct extensive experiments across both language and vision domains to evaluate the effec- tiveness of our method. Our primary baselines include LoRA (Hu et al.) (022), DoRA (Liu et al.) 2024). and full fine-tuning, and we apply these methods range of model backbones: LL
TTovron et al.) 2023a), LLaMA2-77 7B (Touvron et al. 2023b). LLaMA3 8B (Grattafiori et al. 2024]. 1LaMA3.1-70B (Grattafiori al., 2024). and ViT-Base (wu et al.) 2020). The evaluation spans two major fronts: (i) commonsense reasoning tasks in the language domain, and (ii) image classification tasks involving highly imbalanced and nomain-specific datasets, including several medical imaging datasets and DomainNet. We focus on LoRA and DoRA as our primary baselines since they are the most widely adopted and directly comparable PEFT methods, while results with additional baselines (namely full finetuning, rsLoRA (Kalajdzievski 2023), AdaLoRA (Zhang et al.. 2023). LoRA-Pro (Wang et al. 2025). LoRA-GA provided in Appendix E.4
Wang et al. 2024). LoRA+ + (Hayou et al. 2024 are
In addition to the typical low-rank configuration (c.g., rank 2 4), we explore extremely constrained settings by reducing the rank to low as 1, demonstrating the robustness of our method under stringent parameter budgets. This allows us to highlight not just absolute performance but also the parameter efficiency and scalability of our approach relative existing baselines. Further implementation and dataset details are provided in Appendix C For additional baselines, LoFT derivatives (LoFT (simple), which removes second-moment calibration to reduce memory and latency overhead, and quantized LoFT), as well as ablations, memory footprint, and latency analysis, please refer Appendix E Large-scale scaling results on LLaMA-3, 3.1-70B are provided in Appendix
3.1
COMMONSENSE REASONING
Setup. To evaluate the efficacy of LoFT in the language domain, we conduct experiments on a suite of commonsense reasoning benchmarks, including BoolQ, PIQA, SIQA, HellaSwag (HS), Winogrande (WG), ARC-Challenge (ARC-C), ARC-Easy (ARC-E), and OpenBookQA (OBQA). We fine-tune three prominent large language models, LLaMA-7B (Touvron et al.) 2023a), LLaMA2-7B (Touvron et al. 2023b), and LLaMA3-SB (Grattatiori et al. 2024), using parameter-efficient methods: LoRA, DoRA, and our proposed LOFT, each evaluated at multiple rank settings, notably including very low ranks (e.g.. 1,2,4) 4). Following the setting of Hu et al. 2023). we combine the training sets from all eight benchmarks into single unified training dataset, and then conduct evaluation separately on each task's 's official test set. This unified training strategy enables more stable fine-tuning and fairer comparisons across tasks and adaptation methods.

---

Published as a conference paper at ICLR 2026
LLaMA-7B: LoRA
LLaMA- 7B: vs LoFT
(+55%)
(22.060)
1-225%
1+22.781
Aug Accurecy
-50
40
LoRA
LoFT
LoFT
30
16
32
16
Rank (r)
Rank tr
Figure 3: Rank-wise comparison of LoFT against LoRA (left) and DoRA (right) on LLaMA-7B across commonsense reasoning tasks. LoFT maintains significantly higher accuracy, especially at low ranks. Percentage gains denote improvement of LoFT over the respective baseline at each rank.
LORA
DoRA
LoFT
SIQA
SIQA
SIQA
HS
80
80
2.20
WG
B WG
Boolq WG
BooiQ
ARC-C
OBOA
ARC-C
OBQA
ARCIC
O60A
ARC-E
Figure 4: Task-wise performance comparison across LoRA (green), DoRA (red), and LoFT (blue) at lower ranks (r = [4, 1}} on LLaMA-7B. LoFT maintains high performance across all tasks, even under extreme compression, unlike baselines that degrade sharply several benchmarks.
LoFT consistently achieves superior perfor- mance across all model scales and rank configurations. For LLaMA-7B, LoFT at rank 16 achieves the highest average accuracy 76. 76.08%, outprrformng both LoRA (73.57%) and DoRA (71.11%) by notable margins. Even at lower ranks, LoFT maintains strong performance, only 1.1% drop at rank and 3.9% at rank 1, demonstrating its robustness in extremely low-rank regimes. The trend continues for LLaMA2-7B, where LoFT at rank 16 reaches average accuracy of 80.46%, surpassing LoRA by 11.7% and slightly edging out DoRA. Remarkably, LoFT remains highly competitive down to rank 1, scoring 74.88% which still outperforms LoRA by significant margin. For the largest model, LLAMA3-8B, LoFT achieves the highest average accuracy of 85.63% at rank 16. The gains over LoRA and DoRA are less dramatic, but LoFT's performance remains consistently on top. Importantly, the drop-off in performance with decreasing rank is significantly more graceful for LoFT.
Overall Performance Results. As shown in Table 2
Rank- Wise Comparison. To better illustrate LoFT's robustness and performance scalability, we present rank-wise comparison in Figure 3 The left panel compares LoFT against LoRA, and the right panel compares against DoRA, both LLaMA-7B. We observe that LoFT consistently outperforms both baselines across all rank settings, but the gap becomes especially pronounced at low ranks. Notably, rank 4, LoFT surpasses DoRA by an impressive +40% and LoRA by +25%. highlighting LoFT's extreme efficiency in constrained settings.
Interestingly, while LoRA and DoRA both suffer steep accuracy drops at lower ranks, LoFT exhibits much flatter accuracy curve, showing that it retains high performance even with minimal trainable parameters. This makes LoFT particularly appealing for low-resource deployment scenarios.
These results validate two important properties of our method: (i) LoFT matches exceeds the performance of existing PEFT methods even at high capacity (r = 16), and (ū) it remains highly effective extremely low ranks, highlighting its efficiency and applicability in constrained settings. Overall, LoFT achieves the best balance between accuracy and parameter count across diverse commonsense reasoning tasks, while using the same number of parameters as LoRA.
Task-Specific Analysis at Low Ranks. To further analyze performance under parameter-come constrained settings, we examine how LoRA, DoRA, and LoFT behave across individual tasks at lower ranks r (4, 2, 1} using LLaMA-7B. Figure shows radar plots for all eight commonsense reasoning benchmarks each of these low ranks. These visualizations reveal that while LoRA and DoRA suffer
7

---

Published as a conference paper at ICLR 2026
Table Comparison of parameter-cffiieen fine-tuning methods on image classification benchmarks using the ViT-Base model. We evaluate full fine-tuning (Full FT), LoRA, DoRA, and proposed method, LoFT, across four datasets: ISIC2019, HAM10000, Diabetic Retinopathy, and DomainNet. Accuracy (mean + standard deviation) is reported for each setting.
Diabetic Retinopathy
Model
Method
ISIC2019
HAM10000
DomainNet
avg.
Full FT
80.69
0.18
93.27 1 0.64
56.07
0.23
73.46
1.20
75.86
L0
81.02
1.10
91.56 + 0.66
57.87
0.43
71.39
0.10
75.46
ViT-Base
DoRA
80.35 0.17
90.78 1 0.81
57.66
0.56
70.18
2.02
74.74
LoFT,-16
81.06
0.13
93.13 + 0.28
58.33
0.19
71.97
0.16
76.12
LoFT
80.36 0.21
91.78 1: 0.68
57.89
0.48
70.11
0.77
75.04
LoFT
79.31 0.36
91.45 + 0.73
56.98
0.27
69.32
0.55
74.27
inconsistent and often sharp performance drops across tasks, LoFT maintains stable and competitive accuracy across the board.
In particular, DoRA shows substantial instability at ranks 4 and 2, with near zero scores on certain tasks such as WinoGrande, whereas LoRA suffers large dips on more complex tasks like HellaSwag and SIQA. In contrast, LoFT retains high task-wise accuracy, especially on harder benchmarks (e.g.. HellaSwag, ARC-C), even at rank I, demonstrating its robust generalization when adaptation budgets are extremely constrained. For comprehensive view of the exact numerical breakdowns per task and rank, we refer readers to the appendix.
3.2 IMAGE CLASSIFICATION
To assess the generality of approach beyond the language domain, evaluate it image classification tasks using the ViT-Base model (Wu et al., 2020) pretrained on ImageNet-21K K (Deng al. 2009). Vision models are known be more sensitive to low-rank constraints, often requiring higher intrinsic ranks to preserve performance. Therefore, we restrict our analysis to ranks 2 4 and focus on whether LoFT can match or exceed strong baselines under these challenging constraints.
We conduct experiments on four diverse and challenging datasets: ISIC2019 (Codella et al. 2019) and HAM10000 (Tschandl al., 2018): medical skin lesion classification datasets with long-tail label distributions; Diabetic Retinopathy (Graham 2015): a medical imaging dataset with ordinal severity levels, and DomainNet (Peng et 2019): large-scale scale highly skewed benchmark.
We compare our method (LoFT) against full fine-tuning (Full FT), LoRA, and DoRA using consistent configuration (rank r 16 unless specified otherwise). For each dataset, we report the mean and standard deviation over three runs.
As shown on Table 3 LoFT at rank 16 achieves the highest average accuracy 76. 76,,2, outperform- ing both LoRA (75.46%) and DoRA (74.74%), and even slightly surpassing full fine-tuning (75.86%).
LoRA
LOFT
10a
Full
Loss (leg):
LoFT also achieves the top score on two of four in- dividual datasets, including ISIC2019 and Diabetic Retinopathy. Notably, LoRA performs competitively on ISIC2019 but exhibits degraded performance on HAM10000 and DomainNet, suggesting it may strug- gle with skewed datasets. DoRA generally underper- forms across datasets, indicating instability in visual Figure 5: Training log. loss on HAM10000.
220
0.055
100
200
300
Training lterations
domains with skewed/out-ol -otp/ouut----omminn datasets. In contrast, LoFT maintains strong performance, even when the rank is reduced to 8 and 4, with only a 2-point drop in average accuracy at rank 4, further reinforcing its resilience to low-rank degradation vision tasks.
In addition to the final accuracy gains reported in Table 3 we also present the training dynamics on HAM10000 in Figure 5 Remarkably, LoFT's training loss curve closely overlaps with that of full fine-tuning from the very first iterations, indicating that our updates follow the same optimization trajectory as Full FT right from the start. In contrast, LoRA starts with a noticeably higher loss and converges more slowly, never fully matching Full FT's initial descent. This early alignment between LoFT and Full FT demonstrates that, despite updating far fewer parameters, LoFT preserves the model's 's capacity to adapt rapidly.
8

---

Published as a conference paper at ICLR 2026
Throughout the remainder of training, LoFT maintains small gap behind Full FT, which attribute to the growing rank of the full fine-tuning solution, as explained by greedy low-rank learning theory (Li et al. 202T). Nevertheless, LoFT significantly outperforms LoRA across the full training trajectory. Interestingly, LoFT ultimately achieves better final performance than full fine-tuning, suggesting that Full FT' may overfit, whereas LoFT benefits from implicit regularization due the low-rank structure of its updates.
RELATED WORK
Effacieer-ffficient Fine- Tonn-T As aforementioned, the advent of Large Language Models has exploded the computational and memory requirements of running neural workloads, at training and inference time, thus limiting rumning such tasks to a few players. Towards this end, a significant amount of research has focused on efficient ways of fine-tuning LLMs for downstream tasks. Param- eter Efficient Fine-Tuning (PEFT) collectively refers to techniques that only tune a small number of parameters towards the optimization objective. Such methods take various shapes, ranging from token-level (i.e., prompt-tuning (Lester et al.) 2021] and intermediate state parameters (i.e., prefix- tuning) (Li & Liang 2021) to block-level parameters interspersed in the transformer block, either sequentially (Houlsby et al. 2019 Pfeiffer et al. 2021 or in parallel (He et al. 2022).
Low-Rank Adaptation. Closer to our method, LoRA (Hu et al. 2022) introduces low-rank adapters parallel to the attention and linear layers of the transformer block, which build upon the assump- tion that the changes in model weights during adaptation exhibit a low-rank structure and thus reparametrize updated weights such. While seminal, LoRA often falls short of the full fine-tuning potential of the model. Subsequent work has tried to tackle this in various ways. Specifically, DoRA (Liu et al., 2024) decomposes the model weights into their directional and magnitude compo- nents and fine-tunes both, but only the former remains low-rank. Similar in nature is DeLoRA (Bini al. 2025), decouples the direction and strength of low-rank weight updates via normalization and learnable scaling. On the contrary, Zhuu et al.] (2024) note the distinct function of A and B low-rank matrices and propose training only the latter for efficiency, while Hayou al. (2024) adopts different learning rates for each matrix. LoRA-Pro (Wang et al.) (025)) shows the equivalence of low-rank adaptation and low-rank gradient and enhances LoRA by minimizing the distance between the true gradient and the low-rank matrices A and B in closed form. Zhang & Pilanci (2024) introduce Riemannian preconditioner to enhance the stability and efficiency of LoRA with SGD and Adam optimizers across tasks. PiSSA (Meng al. 2024), on the other hand, pinpoints the issue with the initialization of LoRA matrices and proposes SVD decomposition and freezing only the residual components of the weights. All of the above methods attempt more faithfully approximate the gradients in the low-rank subspace and close the performance gap of LoRA with full fine-tuning. Contrary to prior work, our primary goal focuses on the optimization dynamics of low-rank models and aligning the optimizer state to full fine-tuning, By doing so, we are able to get state-of-the-ar results without sacrificing accuracy or efficiency. Concurrently with our work, AltLoRA (Yu et al., 2025) proposes an alternating optimization scheme for LoRA that mitigates second-order coupling effects via alternating projections P Both approaches share the central idea of alternating updates to alleviate coupling effects in low-rank parameterizations (i.e., alternating optimization constitutes the common core). However, they differ substantially in scope and objective. AltLoRA primarily focuses improving gradient approximation within the low-rank subspace and establishes recovery of SGD with momentum. In contrast, our method extends beyond gradient alignment to match the full optimizer dynamics, aligning not only gradients but also all internal optimizer moments. This enables exact recovery of full Adam W dynamics in the low-rank regime and provable reduction to full fine-tuning in the full-rank limit. Moreover, we introduce projected full-update reconstruction and moment calibration matrices, and eliminate the need for the LoRA scaling hyperparameter. To our knowledge, our method is the first to establish transformation invariance under full AdamW dynamics, encompassing first- and second-momen alignment, gradient clipping, and weight decay.
More efficient LoRA. While low-rank adaptation significantly drops the computational and memory requirements of training large-scale LLMs, it still can require significant amount of resources, espe- cially constrained edge or cross-deviss federated learning settings (Cho et al.) 2024). Towards this end, several approaches further optimize low-rank adaptation to minimize the overhead. Specifically,
AAltL and our work were developed independently and appeared on arxiv within days of cach other.

---

Published as a conference paper at ICLR 2026
VeRA (Kopiczko et al. 2024) proposes freezing shared random low-rank matrices and only training scaling vectors. LoRA-X -- (Batazy et al. 2024) freezes iniitiaalizee low-rank matrices and only trains small r x matrix for adaptation. Last, LoRA-SB (Ponkshe al., 2024) more carefully initializes the low-rank matrices to more faithfully approximate the full fine-tuning gradient directions during adaptation. Contrary to such approaches, LoFT can scale truly low ranks by careful tuning of the optimization process, rather than altering the adaptation modeling.
5 CONCLUSION
In this work, have presented LoFT, low-rank adaptation framework that aligns the optimizer's internal dynamics to full fine-tuning by means of alternating LoRA updates, gradient projection and scaling, first and second moment calibration, and gradient clipping approximations. These mechanisms enable significant performance and efficiency gains with minimal loss in accuracy across different tasks and model sizes and pave the way for training even more efficiently for downstream tasks. Towards this end, we plan to explore the interplay between our LoFT and quantization to further boost efficiency and sustainability in training, as well as how can be combined with noisy Differential Privacy updates, which can enable distributed private training at scale.
REFERENCES
Klaudia Balazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprini arXiv:2405. 17604, 2024.
Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Con- nor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. LoRA learns less and forgets less. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL httppss//ppeerreeieeeett////// al aloEru2qcG Featured Certification.
Massimo Bini, Leander Girrbach, and Zeynep Akata. Decoupling angles and strength in low-rank adaptation. In The Thirteenth International Conference on Learning Representations, 2025. URL
ptttp:://openreveee net forum? id X1074 IwuxG
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Paviov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.00337, 2021. URL http::tt:///tt
/ttp:::///arxiva///t/ http://wwxxvooooooooo/oo/o abs
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In The Eleventh International Conference on Learning Representations, 2023. /RLL/http:///peeeee wwweeenrvviewwe........ www.eeeeee/ooooeeee.eoeooeeeoeee www.foorruuoooooo worrumi.d=pluu22yycCM
Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous lora for federated fine-tuning of on-device foundation models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 12903-12913, 2024.
Alexandra Chronopoulou, Matthew Peters, Alexander Fraser, and Jesse Dodge. AdapterSoup: Weight averaging to improve generalization of pretrained language models. In Andreas Vla- chos and Isabelle Augenstein (eds.), Findings of the Association for Computational Linguistics: EACL 2023, pp. Linguistics. doi: 10. 10.186533/1/2023. findings-eacl.... 1533 URL https: //clt//////laattooooooo///////// 2023. findi eac1.153/
Dubrovnik, Croatia, May 2023, Association for Computational
10

---

Published as a conference paper at ICLR 2026
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo. Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXivv:902..0 03060, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Ieee, 2009.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances Neural Information Processing Systems, volume 36, pp. 10088-10115. Curran Associates, Inc., 2023.
Ben Graham. Kaggle diabetic retinopathy detection competition report. University of Warwick 22 (9), 2015.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint arXiv:2407 2tt.:::200.2118... 21733, 2024.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 Proceedings of Machine Leurning Research, pp. 17783-17806. PMLR, 21-27 Jul 2024. /tt.:ppee/bbccccccn..aaaaas/////
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrickk and Graham Neubig, Towards unified view of parameter-efficient transfer learning. International Conference on Learning Representations, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NI.P. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1790-2799 PMLR, 09-15 Jun 2019. URL http:::/ppreeeeci..........eeeee houl sallsby199 9b.y19..ttml
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL htthh:t:://// P
nžeVKeeFYf9!
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. LLM-adapters. An adapter family for efffmietr-efficientt fine-tuning of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5254-5276, Singapore, December 2023, Association for Computational Linguistics. doi: 18653/v 1/2023.cm2l2gppp m11n.319. /tt/:/ttes///elllatooooooooo////
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie. Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European conference on computer vision, pp. 709-727. Springer, 2022.
Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https 7/kel
Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03332, 2023.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024, URL

---

Published as a conference paper at ICLR 2026
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009.
Ariel Lee, Cole Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms. In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following. 2025.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp- 0044--059, 2021.
Xiang Lisa Li and Percy Liang. Prefixttuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582 0892-4597, 2021.
Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In International Conference Learning Representations, 2021. URL http::tttss//t
worrmoorummm/mm do d=AHCs 75557R
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https: 7 //openeeveew.eet www.eee///ooommeeeeeoooeee pipoormmiidll.
Shih- Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Beight-decomposed low-rank adaptation. In Forty-first International Conference Machine Learning, 2024.
Shih-Yang
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Con- ference Learning Representations, 2019. URL http:ttpps///peeneeeeeeeeeee/ee
Bkg6RiCqY7
Grigory Malinovsky, Umberto Michieli, Hasan Abed Al Kader Hammoud, Taha Ceritli, Hayder Elesedy, Mete Ozay, and Peter Richtárik. Randomized asymmetric chain of lora: The first meaningful theoretical framework for low-rank adaptation. arXiv preprint arXiv:2410.0832 2024.
Fanxu Meng. Zhaohui Wang, and Muhan Zhang. Pissa: adaptation of large language models. Advances in Neural Information Processing Systems, 37: 121038-121072, 2024.
Principal singular values and singular vectors
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint 1xXxx:2402.1483000 2024.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1406-141, 2019.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfu- sion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 487-503, 2021.
Kaustubh Ponkshe, Raghav Singhal, Eduard Gorbunov, Alexey Tumanov, Samuel Horvath, and Praneeth Vepakomma. Initialization using update approximation is a silver bullet for extremely efficient low-rank fine-tuning. arXiv preprint arXiv:24/ 1.557777, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):, 2019.
12

---

Published as a conference paper at ICLR 2026
VI Slyusar. A family of face products of matrices and its properties. Cybernetics and systems analysis, (5(3):379-384 1999.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09288 2023b.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, large collection of multi-source dermatoscopic images common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Kerem Turgudu, Jonathan Whitaker, and J. H. Fsdp, glora. and Ilama A recipe for efficient fine-tuning. http:s//www.aanser..a///tt/t/222
fedp qdora http:://www.aneee...aaaat//////. Accessed: 2025-05-21.
11tma 2024.
@dora- 1anraallaama 11maa 3. mm
Shaowen Wang. Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation.
Advances in Neural Information Processing Systems, 01154995--59333 2024.
Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. LoRA-pro: Are low-rank adapters
properly optimized? In The Thirteenth International Conference Learning Representations, 2025. URL https:
Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan. Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision, 2020.
Liping Yi, Han Yu, Gang Wang, Xiaoguang Liu, and Xiaoxiao Li. pfedlora: model- heteroencts personalized federated learning with lora tuming. arXiv preprint arXiv:2310. 13283, 2023.
Xin Yu, Yujia Wang, Jinghui Chen, and Lingzhou Xue. AltloRA: Towards better gradient approxi- mation in low-rank adaptation with alternating projections. In The Thirty-nimth Annual Confer- ence on Neural Information Processing Systems. 2025. URL
Fangzhao Zhang and Mert Pilanci. Riemannian preconditioned LoRA for fine-tuning foundation models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Confer- Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. v235/
59641-59669. PMIR, 21-27 Jul 2024. zzang24ax.html
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for crrameter-efffcien fine-tuming, In The Eleventh Inter- national Conference on Learning Representations, 2023. URL httts::: htattts:://oenreveee..t
forum? i (62.w...662211111
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. OpenCodelnteppreter: Integrating code generation with execution and refinement. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL. 2024, pp. 12834-12851 12859, Bangkok, Thailand, August 2024. Association for Computational Lingutattiooallniiisiiiitiii Linguisties. doi: 10. 10.18653/w1/224 http5/////222....mnnnnd......... and.ings-acl.762. httpttss
ac1.762/
Jaacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. In Forty-first International Conference on Machine Learning, 2024. URL http:tttss//pppeeeieeeeeee/////a
13

---

Published as a conference paper at ICLR 2026
Table of Contents
Limitations
B Theoretical Properties of LoFT for Matrix Factorization
15
C Implementation Details
17
Datasets
17
Hyperparameters
17
LoFT Algorithm
D.
Lo
19
E
Optimization and Efficiency Analysis
22
E.1
Ablation Study
22
E.2
Memory Footprint
................................
23
E.3
Training Latency
24
Comparison Additional Baselines
25
F
Language Model Experiments
27
F1
Commonsense Reasoning
27
F2
Mathematical Reasoning and Quantized LoFT
27
F3
Scaling to LLaMA-3-70B
28
Vision Experiments
28
Training Dynamics
................................
28
DomainNet: Domain-Specil Results
"e..............................
29
G.3
Comparison with Vision Cononic Baselines
30
Coding Experiments
30
14

---

Published as a conference paper at ICLR 2026
A LIMITATIONS
In this paper, we have proposed a technique for producing parameter-effiient fine-tuning via low-rank adaptation that behaves like full fine-tuning. While LoFT offers significant efficiency gains compared to full fine-tuning and DoRA, it has increased memory consumption compared to LoRA (due to storing previous iterates 1 U). to the benefit of increased accuracy (see Section E.2). Having said that, even for the same memory footprint of LoRA, LoFT is able to achieve better downstream accuracy.
Lastly, we applied our technique on top of the AdamW optimizer, as it is the most widely used in LLM optimization. We leave applications other optimizers, like Muon, for future work.
B THEORETICAL PROPERTIES OF LOFT FOR MATRIX FACTORIZATION
In Section 2.4
argue that when UVT is of full rank, then LoFT recovers full fine-tuning. Furthermore, for the matrix factorization problem, we showed that if the true solution is of low rank, then LoFT also empirically recovers full fine-tuning. this section, we further extend these results. In particular, we focus on the matrix factorization problem
{ {T(VVV del
min UEBmxr SERmX",VER=E VEREX
LIUV
AlE
(14)
Let A = ÜΣVT be the SVD decomposition of A. Then, by the Eckart- Young theorem, we have that every solution of (14) has the following form:
U = Ü,E,,
V V'=
where Ür, E,, V, contain the first singular vectors of A and E Rexr is full rank matrix. In the next lemma, we show that and start in the correct space, then LoFT applied to gradient descent with momentum recovers full fine-tuning with momentum.
Lemma Let Un U,Xo and Vo V,Yo, where Xo, Yo Taxx are full rank matrices. Then, LoFT-GD with momentum applied to the matrix factorization problem exactly recovers GD with momentum applied to f(W) = W|l - All initialized at Wo = Vo
Proof. The gradient of ((W) with respect to W has the following form:
W fWW) W. A Ü, (XoYg V.
The left and right spaces correspond to U, and V, respectively. Using ( and (8) we get
v 9o go Vw ((W) and my m. mm Vw (WW)..
Since momentum also the update, we have by induction that 4k: 2 0. U2 U, and Vs V,Yr. where Xs, Y2 E Rr Therefore,
gk = gh -Ww and
& iig m = m. = (1 31) Σ
gk- V
1=0
One interesting consequence of the above lemma is that if we apply LoFT with step size 1 with the initialization in the correct space, LoFT finds the optimal solution single step. Notice that without scaling, the smoothness constant of [4) with respect to both optimization variables can be unbounded, since
Wu.(U,,V). - V V - A)V (U.VT- - -A)VIl
V)le
=
15

---

Published as a conference paper at ICLR 2026
be unbounded as ||V| can be unbounded. In practice, would need restrict || and e to guarantee smoothness. On the other hand, LoFT scaled version of the gradient satisfies:
[Vuff(U,,V) - f(U, = |lU1 V (V) - (U2V" - A)V
= =I(U1 - VV)VTV(VVVV ||
1| - U22le.
Therefore, LoFT gradients smooth with the smoothness constant without any restrictions. The above highlights another desirable property LoFT introduced the following lemma.
Lemma 2. LoFT-GD with step size 17 = 1 applied to the matrix factorizatiioonnnnn Alternating Least Squares algorithm.
(4) corresponds to the
Proof. Without loss of generality, we assume U is updated. Let E = V
A, then:
E = Uk+1 A
(Ux E&Vx (V. V/s) -1 VW -
FE-- F&Ve (Vr Vz) -1 Vi
E& (I - -Vv).
Therefore,
J(UR+1, V&) 2 | 11
- 11 (1 - -vv, 11
min UERmx.
GlUVE -A1
min meRmxr
S(U, Va).
2
Analogically, we can derive
S(U,, V\+1) = min VEReX
[(U,, V):
which concludes the proof.
16

---

Published as a conference paper at ICLR 2026
C IMPLEMENTATION DETAILS
Compute Information. All experiments reported in this paper were conducted using a single NVIDIA 1100-SXM440GGB GPU. This setup was used consistently across all experimental runs. Time of execution and memory usage varied slightly depending on the model configuration, but all runs were completed on single-GPU setup. No additional or external compute (e.g., cloud services) was used during these experiments.
The implementation of LoFT used our experiments can be found at: http://gathbbbccom/muuukk/o/ffff
C.I DATASETS
Commonsense Reasoning. To evaluate language models' reasoning capabilities, we use curated commonsense reasoning benchmark COMMONSENSE: 17KK (Hu et al., 2023) consisting of 1 17KK diverse examples. These examples are drawn from multiple existing commonsense QA datasets and span a variety of tasks, including physical reasoning, social intuition, temporal understanding, and cause-effect inference.
Image Classification.
We conduct experiments on four diverse and challenging datasets to evaluate the generalization ability of our method in the image classification domain:
ISIC2019 (Codella et al. 2019) is medical dataset composed of 25300 training and 8238 test dermoscopic images spanning eight skin lesion categories. It presents a long-taile distribution, with the largest class heavily overrepresented relative such as dermatofibroma or vascular lesions. The dataset is particularly challenging due to inter-class visual similarity and intra-class variability.
malignancies
HAM10000 (Tschandl et al. al.||20111 20188) contains {8.2K + 1.2K) (training test) high-resolution dermoscopic images categorized into seven skin lesion types. In includes lesions from diverse populations and acquisition sources. Similar to ISIC2019, this dataset suffers from severe class imbalance.
Diabetic Retinopathy (Graham 2015) consists of (115K 14.2K} (training test) retinal fundus images annotated with ordinal labels representing five stages of diabetic retinopathy severity. The task involves predicting these severity levels from fundus scans.
DomainNet (Peng et al. 2019) is large-scale dataset designed for domain generalization and adaptation. It contains approximately 587000 images from 345 categories across six domains: real, clipart, infograph, painting, quickdraw, and sketch. Its substantial domain shift and high class diversity make it a valuable benchmark for testing superiority of the methods.
Math Reasoning. To assess mathematical reasoning in large language models, we use the ORCA- MATH dataset (Mitra et al. 2024). benchmark of 200K diverse math problems spanning arithmetic, algebra, geometry, calculus, and probability. Each problem requires multi-step reasoning and symbolic manipulation, making the dataset well-suited for evaluating fine-tuning strategies.
Language Modeling, To evaluate language modeling and text generation under low-resource con- ditions, we use the WIKITEXT2 dataset (Merity et al. (2017). a widely adopted benchmark consisting of over 100K tokens from cleaned Wikipedia articles. The dataset preserves natural long-range dependencies by retaining full articles and punctuation, making it suitable for assessing perplexity and generalization in autoregressive models. We follow the original data split and preprocessing protocol established by RRafordd et al. (2019).
C.2 HYPERPARAMETERS
We report training configurations for the main experiments: commonsense reasoning and image classification. For clarity and reproducibility. the full hyperparameter settings for each task presented tables below. Hyperparameters for the remaining tasks, including math reasoning, language modeling, and code generation, are detailed separately in Sections F2
E.4
respectively.
17

---

Published as a conference paper at ICLR 2026
Table 4: Hyperparameter configurations used for LoRA/DoRA (as in (Liu al. 2024 and our method, LoFT, across LLaMA model variants on commonsense reasoning tasks. Unlike prior method that tune the LoRA scaling factor a, LoFT sets consistently across all models without the need for tuning.
LoRA/DoRA
LoFT
Hyperparameter
LLaMA-7B
LLaMA2-7B
LLaMA3-8B
LLaMA-7B
LLaMA2-7B
LLaMA3-8B
Rank
Alpha scaler
2xr
r
Dropout
0.05
0.05
Optimizer
AdamW
Adam
Learning rate
2x10-1
3x10-4
1x10-1
2x10-4
3x 10
1x10-4
LR scheduler
Lincar
Lincar
Batch size
16
16
Micro-bate batch size
16
16
Warmup steps
100
100
Training epochs
3
Low-rank targets
Q V, Up, Down
Q. K, V. Up, Down
Table 5: Training hyperparameters for ViT-B/16 across four image classification datasets. All methods
(Full FT, LoRA, DoRA, and LoFT) are trained using the same configuration for fair comparison.
Dataset
Rank
Batch Size
LR
Epochs
Target Modules
LoRA/DoRA a LoFT a
ISIC2019
64
5x10-4
3
Q.K. K. Dense
2xr
HAM10000
r
5x10
3
Q, K, V, Dense
2xr
Retinopathy
r
64
5x10-4
3
Q, K, V, Dense
2xr
DomainNet
256
5x10-1
3
Q K, V, Dense
2xr
Common settings: Optimizer = AdamW, LR scheduler Linear, Warmup ratio =0.1, Dropout = 0.1, Micro-batch size
Batch size.
Commonsense Reasoning. We evaluate three generations of LLaMA family models, LLaMA-7B, LLaMA2-7B, and LLaMA3-8B, test whether our proposed LoFT approach scales consistently across architectural updates. For each backbone, we compare against two strong parameter-efficieee baselines, LoRA (Hu al., 2022) and DoRA (Liu et 2024). For these experiments, we adopt the optimal hyperparameter settings reported in (Liu et al. [024]. We adopt the same learning rate, learning rate scheduler, warmup steps, batch size, and the same Q, K, V, Up, Down matrices for applying LoRA. The full configuration is summarized in Table
Image Classification. We conduct image classification experiments using the ViT-B/16 model across four datasets: ISIC2019, HAM10000, Diabetic Retinopathy, and DomainNet. The input resolution fixed to 224x224 pixels, and the patch size is set to 16. All methods, including full fine-tuning, LoRA, DoRA, and our proposed LoFT, share the same training configuration to ensure fair comparison.
Specifically, fix the learning rate to 5x 5x across all datasets. The batch size medical datasets and increased to 256 for DomainNet due to its scale. All models are trained for 3 epochs using the AdamW optimizer, with linear learning rate scheduler and a warmup ratio of 0. 0.1. A dropout rate of 0.1 is applied, and low-rank methods target both the Q, K, V attention layers and the Dense layers. These hyperparameters are summarized in Table
64 for
Scaling Factor. We clarify the role of the scaling hyperparameter As noted in Section ? set a = 1 in LoFT. In practice, the HuggingFace PEFT library implements scaling as a/r, rr so setting yields effective scaling factor of thereby removing the need for hyperparameter tuning. For LoRA and DoRA baselines, we followed the recommended setting a 2r (see Table 10 in Liu al.] (2024)) to ensure fairness. One of LoFT's design goals is precisely to eliminate this hyperparameter, which we emphasize in the main text.
18

---

Published as a conference paper at ICLR 2026
D LOFT ALGORITHM
Algorithm LoFT-Adam W with Alternating Updates
Require: Pretrained weights Wo. low-rank factors Uo. Vo. learning rate 7/k, weight decay rate A,
AdamW parameters $1, 82,
1: Initialize first and second moments: my mi Po PD + 0
y
2: Set alternating update flag: update_ False
3: for k = 1, 2.... do
4: W f- Wo + +UVE
Reconstruct full weight matrix
5:
gw Vwf(WN)
Get full gradient (only for notational purposes) # Project gradients to low-rank factors
6:
9U - gw VK. gV t- Un
7:
C - (V),VVe(VV,
Vx)-1 V+.
C. + (U U0)
8:
DU + (V v1)= .9v + (V(
9:
m - k CV -1--
# First moment calibration
10:
R + 3ym , +(1 -112vv
11:
PE - 1 (Ck @ Ck + (1 B2) g0)
# Second moment calibration
PK /2 (C @ C + (1 (gv gv)
13:
if updatee U then
Alternating updates Reconstruct second moment projected space
Uy +- Pk (V& Vx))
m + m V /(1 8
Uy t / 35)
mik
AU + TiK
Vi(VV
Update U with projection
v oo +E
18:
UR+1 + (1 UU AUU
19:
V4+1 V/
20:
else
Uy < PX (UR (UR
UP)
Reconstruct second moment in projected space
22:
my my U. /(1 34)
23:
oo < ux /(1(11 BB)
mk
AV + Ti
UN(UTUL UK)
Update V with projection
V o +
25:
Ve+1 + (1 Xm Vr - AV
26:
UR411 + Uv
27:
end
update_Us not update_
# Alternate update direction
29:
end for
D.1 LOFT-MUON
In this section, provide extension of approach to Muon (Jordan et al.) 2024) algorithm. Firstly, we introduce the original Muon in Algorithm 2
Algorithm 2 Muon
Require: Learning rates 7,, momentum
1: Initialize mo +-
2: for k 1.2.... do
3: 9w + Vw (WA)
Compute full gradient
4: m4 pm4-1 + 9W
Compute momentum
5: Ok + NewtonSchhlls(mm))
# Algorithm Update Parameters
6: W +- Wt 7
7: end for
19

---

Published as a conference paper at ICLR 2026
Algorithm 3 NewtonSchulz5
Require: number of steps Nstepss E le l G Rmxx (a, (3.4445, -4.7700, 2.0315) 1: X + G/[[]]]| + +)
# Proper initialization
2: if m then
# For efficient computations
3:
X X
4: end if
5: for k
1.2.....
Rstpps do
6:
A f- X
7: 3+ cA2
8:
X+ + BX
9:
end for
10:
if m then
11:
X X
end if
13: return X
Examining the Muon algorithm, we observe that, like Adam, it employs first-order momentum; therefore, adapt the LoFT setting, we can directly apply the first three building blocks. Furthermore, we can reconstruct an estimate of the full-finetuning momentum using (8), i.e. m. = m Vi." We note that the m is at most rank but NewtonSchulz5 (Algortihm 3) can take advantage of that and directly plugging in mig to NewtonSchulzS would not benefit computations as we would be working with large m X n matrix. Therefore, we design efficient version of NewtonSchulz5 algortihm that accounts for low-rank inputs, see below.
Algorithm 4 NewtonSchulls__ookee
Require: number of steps Rateps: & 10-7, U mxr V E Re (G UVT), (a,
(3.4445, -4.7750, 2.0315)
1: if m n then
# For efficient computations (mirror of dense case) Flip U, V
2: U.V V,U
3: end if
4: UtU UTE U E RTX;; + INVVRVER'M
5: ||G|| + V (I((UTU)(TVV))
# Proper initialization (low-rank)
6: X₂
r I Dsttess
Core variable; X V
7: for k = 1,2...., Doteps do
8:
8 X-VIVX!
# r form of XXX
9:
A 4 SUU
10:
B+bA- + c
11:
X. X. + BXc
end for
13: Xu < UX
#X + UX.V, we use only UX2 as added implicitly via V
Uupd/V-update) 14: mm > n then
Xu + VXT
16: end if
return Xy
# Partial polar factors of G: cost per step O(m n)) r3)
We note that the cost per step of this algorithm is only ((m + n)) + +r3). Finally, ready to proceed with LoFT-Muon, which only requires extra memory O(m + n)r), thus matching the standard LoRA memory requirements.
20

---

Published as a conference paper at ICLR 2026
Algorithm 5 LoFT-Muon
Require: learning rates Te, momentum parameter
1: Initialize m6 my +- O, weight decay rate
2: Set alternating update flag: update_U False
3: for = 1.2...... do
4: Reconstruct full weight matrix
5: W* +- Wo + UgVV
6: Get full gradient (only for notational purposes)
7: 9w + Ww (WW)
8:
Project gradients to low-rank factors
9:
+ gw Va, gV +- Un-UU
10:
CK - VV VA)
Cf < +UL
11:
IU t (V Vi)= Iv < (V(
12:
First moment calibration
13:
THE U umg C + gv
14:
m 1 C + 5v
15:
Alternating updates
if update_0 then Av +- NewtonSchulz5 Looitllkm__L Vv.)
17:
Uk+1 + (1 Um AU Ve+1 f- V&
19:
else
21:
Av +- NewtonSchulz5 Lowaalmmm Uk)
22:
V441 4- (1 - \ - AV
23:
Uk+1 + Uk
24:
end
25:
Alternate update direction update_U + not update_U
26:
27:
end for
21

---

Published as a conference paper at ICLR 2026
E OPTIMIZATION AND EFFICIENCY ANALYSIS
E. ABLATION STUDY
In this ablation study, we investigate the contribution of key components in our proposed LoFT method by selectively disabling them and observing the impact performance. The goal is to isolate the effectiveness of (i) state calibration, and (ii) alternate updates. The experiments are conducted on the WikiText-2 dataset using GPT-2 model in causal language modeling setup.
We evaluate four variants:
LoFT (full method): includes both alternate updates and state calibration.
LoFT without alternate updates: removes the alternation mechanism while keeping calibration. LoFT without state calibration: disables calibration while retaining alternating updates.
LoFT without either: disables both the alternation and state calibration.
Training
Evaluation
70
47.5
45.0
60
Perplextty
50
37.5
35.0
32.5
Training Iterations
Training iterations
Callbration)
Calibration)
Update)
Figure 6: Ablation study of the proposed approach on language modeling task. We train GPT-2 model on the WikiText-2 dataset and evaluate the effect of key components of LoFT by incrementally removing (i) state calibration, (ii) alternate update, and (iii) both. Training perplexity (left) shows smoothed curves with shaded raw values, while evaluation perplexity (right) presents the unsmoothed results.
Training and evaluation perplexities are reported in Figure 6 perplexity (3-step centered moving average) with raw values shaded underneath; evaluation perplexity shown unsmoothed.
For training curves, we show smoothed
The pess-perforning variant in this specific setting is LoFT without alternate updates, which slightly outperforms the full LoFT setup. This is likely due to the fact that removing alternation effectively doubles the update frequency of LoFT parameters, which proves beneficial on WikiText-2 with GPT-2. We can see a significant decrease in performance when considering variants that do not have state calibration.
These results highlight the importance of state calibration, while they also suggest that LoFT can be slightly improved if we consider parallel updates. We attribute this to the small step size and gradient clipping, which limit the impact of the cross term that could be problematic in some cases.
Scaling up to LLaMA and VIT. To further assess generality, we conduct additional ablations on one large language model benchmark and one vision benchmark. Specifically, we evaluate LoFT on LLaMA-7B with commonsense reasoning tasks and on ViT-Base for CIFAR- classification. In each case, remove LoFT components one time. The results are reported in Tables
Across both language and vision benchmarks, the results align with our GPT-2 findings. LoFT without alternation sometimes matches slightly outperforms the full method, likely due increased update frequency. contrast, removing state calibration consistently causes large performance drops, particularly dramatic LLaMA-7B. Overall, these ablations confirm that both alternation and state
22

---

Published as a conference paper at ICLR 2026
Table Ablation results on CIFAR-100 with ViT-Base.
LoFT variant
Accuracy
Full method
91.18
No alternation
91.60
No state calibration
89.38
No alternation state calibration
89.94
Table Ablation results on LLaMA-7B with commonsense reasoning benchmarks.
LoFT Variant
BoolQ
PIQA
SIQA
HS
WG
ARC-C
ARC-E
OBQA
avg.
Full method
67.34
80.96
76.20
80.50
76.40
63.62
79.21
75.40
74.95
No alternation
68.56
79.26
77.33
77.16
78.37
62.97
79.42
74.40
74.68
No state calibration
03.18
48.80
66.43
21.17
68.90
55.03
75.63
66.00
50.64
No alt. state cal.
57.31
62.24
55.58
17.27
65.27
56.48
73.06
68.60
56.98
calibration are important contributors, with calibration being indispensable for LoFT's stability and effectiveness.
E.2 MEMORY FOOTPRINT
We evaluate the memory efficiency of LoFT in comparison to LoRA, DoRA, and DoRA (simple) under two configurations: rank r=16 and rank r=4. All experiments were conducted using the LLaMA-7B model on commonsense reasoning tasks (Tables ()
Theoretical analysis. For AdamW, LoRA requires
mn [Wo] + (m + n)r (U, + 4(m + n)r [optimizer states] mn + 5(m + n)r,
while LoFT requires
mn [Wo] + (m n)r U. 2(m n)r [previous iterates] 2(m + n)r [momentum] 2(m + n)r² [cross-terms] mm 5(m + n)r + 2(m + n)r
+
The additional 2(m + n)r² term arises from cross-terms for optimizer state recalibration. Crucially, this scales with (m++n) rather than mn, ensuring LoFT remains far more efficient than full fine-tuning when is small.
Empirical results. rank r-16, LoFT matches LoRA in terms of trainable parameter percentage (0.4145%) with no increase, while incurring only a 25%56%% increase in memory usage. This memory cost is nearly identical to DoRA (simple), which also maintains low overhead (+25.23%) and significantly lower than full DoRA, which increases memory by over 341%.
At the lower rank setting r=4, LoFT maintains parameter parity with LoRA (0.1040%) and achieves a very modest memory increase of just +6.71% %. compared to the large 342% increase with DoRA. While DoRA (simple) also limits memory to some extent, it still shows over 25% overhead and increases trainable parameters by 12.4%.
Table Comparison of trainable parameter percentage and memory usage for different methods rank 16 using L-LMA-BB on commonsense reasoning tasks.
Method
Trainable params (%)
+ Relative Iner.
Memory (GB)
Relative Iner.
LoRA
0.4145
+0.00%
28.50
+0.00%
DoRA
0.4274
+3.1 13%
125.95
+341.933
DoRA (simple)
0.4274
+3.11%
35.69
+22. 23%
LoFT
0.4145
+0.00%
35.81
+25.65%
23

---

Published as a conference paper at ICLR 2026
Table Comparison of trainable parameter percentage and memory usage for different methods at rank r=4 using LLaMA-7B on commonsense reasoning tasks.
Method
Trainable params (%)
+ Relative Incr.
Memory (GB)
Relative Iner.
LoRA
0. 1040
+00.0%%
28.15
+0.00%
DoRA
0.1169
+12.40%
124.47
+342.17%
DoRA (simple)
0.1169
+12.40%
35.27
+-25
LoFT
0.1040
+0.00%
30.04
+6.71%
Accuracy-efficiency trade-off. Although LoFT introduces extra memory overhead relative to LoRA, it achieves higher accuracy at substantially smaller ranks. For instance, LoFT with 4 surpasses LoRA with r = 16 on LLaMA-7B, and LoFT with r = surpasses LoRA with r = 16 on LLaMA-2-7B (see T Importantly, unlike DoRA, LoFT adds no backward- pass memory overhead (cf. Section 4.3 in DoRA (Liu et al.) 2024)). Thus, the modest increase is offset by substantial performance gains at lower ranks.
LoFT (simple). We further identify that the main bottleneck in LoFT stems from the second- moment calibration. Since its effect on accuracy is marginal (~ 0.1 1 drop at LLaMA--B, r= (Table TO), we propose LoF' (simple), which omits this step. As shown in Table [1] LoFT (simple) reduces overhead to under 6% compared to LoRA, while maintaining nearly identical accuracy.
Table 10: Performance comparison between LoFT and LoFT (simple) (LLaMA-7B, rank
16).
Method
BoolQ
PIQA
SIQA
HS
WG
ARC-C
ARC-E
OBQA
avg.
LoFT
68.62
82.80
78.27
82.69
73.32
64.30
80.26
78.40
76.08
LoFT (simple)
68.50
81.18
78.20
76.87
78.93
64.85
81.14
78.20
75.98
Table 11: LoFT (simple) memory overhead LLaMA-7B under ranks 16 and 4.
Method
Memory (GB)
+ Relative Iner.
LoRA
28.50
+0.00%
LoFT (simple) (r = 16)
30.02
+5.35%
LoFT (simple) (r
29.61
+5.18%
Overall, LoFT offers the same parameter efficiency LoRA while delivering competitive per- formance with substantially lower memory demands than DoRA variants. This makes LoFT memory-efficient alternative suitable for deployment in resource-constrained settings.
We refer the reader to (Liu et al. 2024) for detailed definitions DoRA and DoRA (simple). In our experiments, we exclusively used DoRA (simple), recommended by A'sA's authors. Also, the full DoRA implementation requires substantially more memory and impractical to run on one GPU.
E.3 TRAINING LATENCY
We also evaluate the training latency of LoFT relative to LoRA and DoRA (simple). LoFT introduces additional overhead due to optimizer state alignment and recalibration. Table 12 summarizes the latency (forward + backward + optimizer step) on LLaMA-7B across different ranks.
To better understand the main sources of overhead, conducted ablations of LoFT variants at rank 4-4. The results are shown in Table
These results indicate that the main bottleneck of LoFT arises from second-momen calibration. Omitting this step yields LoFT (simple), which reduces latency to within ~u 30% of LoRA while being 2x faster than the stronger baseline DoRA.
24

---

Published as a conference paper at ICLR 2026
Table 12:
Relative training latency of LoRA, DoRA (simple), LoFT, and LoFT (simple) on LLaMA- 7B. Latency is reported as multiplicative factor relative to LoRA.
Method
16
r= 4
r= 1
LoRA
1.00x
1.00x
1.00x
DoRA (simple)
2.38
2.54x
2.54x
LoFT
3.23x
2.26x
1.76x
LoFT (simple)
1.32x
1.27x
1.22x
Table 13: Latency breakdown (in seconds) for LoFT variants at rank (r 4) (LLaMA-7B).
LoFT Variant
Latency (s)
LoFT (full)
1.0903
No alternation
1.5703
No state calibration
0.5810
No alternation state calibration
0.6146
No second moment calibration [LoFT (simple))
0.6265
Implementation note. We expect substantial speed-ups with dedicated CUDA kernel implementations, which we plan as future work.
All latency measurements are based on plain PyTorch implementation.
E.4 COMPARISON WITH ADDITIONAL BASELINES
Experimental Setup. We first fine-tune the original GPT-2 (137M) on WikiText2 using the same data split and preprocessing as Radford et al. 2019). All methods share the same training hyperparameters: epoch, AdamW optimizer, batch size 64, learning rate 2x with linear decay, For adapter-based baselines (Hu et al. 2022 Kalajdzievski 2023 Zhang et al. 2023 Wang et al.) 2024 2025) we set the rank r=4; LoRA uses its default temperature and dropout as the official repository. After convergence, evaluate the WikiText-2 validation set and report perplexity (lower better).
Limitations of certain baselines. VeRA (Kopiezko et al. 2024 and DoRA Liu al. 2024 only handle Linear layers. Because GPT-2 implements attention weights as ConviD layers, reproducing these methods would require serious surgery and major rewrite; we, therefore, omit them. In practice, this means VeRA and DoRA cannot be applied unchanged to a large family of models that rely on ConvlD parameterizations.
Table 14:
Perplexity (PPL) results the Wiki- Text2 dataset for various fine-tuning methods ap- plied to GPT-2. Lower values indicate better per- formance. LoFT achieves the best result, outper- forming other parameter-efficient techniques.
Table 15:
Perplexity (PPL) WikiTexx2 for GPT-2 Large using various fine-tuning methods. LoFT achieves the best performance, outperform- ing full fine-tuning and other parameter-efficient techniques.
Model
Method
WikiText2 (PPL J)
Model
Method
WikiText2 (PPL J)
Zero-Shot
60.38
Zero-Shot Full FT
38.87
Full FT
29.51
19.42
LoRA,=1
34.80
LoRA, rsLoRA,
19.78
ISLORA,
32.96
GPT-2 Large
19.62
GPT-2
AdaLoRA,
55.67
AdaLoRA,=,
23.31
LoRA-PrO,
32.79
LoRA-Pro,
20.06
LoRA-GA, -ARA-
37.34
LoRA-GA,
21.44
LoRA+r=1
36.15
LoRAt,=1
19.73
LoFT,
31.75
LoFT,
19.26
25

---

Published as a conference paper at ICLR 2026
Results GPT-2. Table 14 reports validation perplexity. LoFT yields the lowest PPL (31.75), outperforming all other parameter-efficien baselines and coming within 2.2 points of full fine- tuning while updating only small fraction of parameters. AdaLoRA (Zhang et al. 2023) performs noticeably worse this low-resource regime. Training and evaluation curves are visualized in F LoFT converges smoothly and tracks Full FT closely throughout training, whereas other methods plateau higher.
Training
Evaluation
3.0
10
107
Perplcdyy
10
4 10
xx000
100
100
200
Training heeraiion
Training Rerations
FullFT
LoRA
IsLoRA
AdeLoRA
LoRA-Pro
LoRA-GA
LoRA+
LofT
Zero-shot
Figure 7: Training and evaluation perplexity curves for GPT-2 on WikiText-2 dataset. The left panel shows smoothed training perplexity (3-point moving average) for seven fine-tuning methods (Full-FT, LoRA, rsLoRA, AdaLoRA, LoRA-Pro, LoRA-GA, LoRA and LoFT), with the raw PPL shaded beneath each curve. The right panel reports evaluation PPL for the same methods, with dashed horizontal line at 60.38 marking the zero-shot baseline. Table for a reference: Table 14
Training
Evaluation
1x
4 10
20
Parplocs locky
1.00
15
8.0
1000
3 10a
19.25
102
2x
1x
Training Iterations
Training Reertions
LoRA-GA
Figure 8: Training and evaluation perplexity curves for GPT-2 LARGE on WikiText-2 dataset. The left panel shows smoothed training perplexity (3-point centered moving average) for seven fine-tuning methods (Full-FT, LoRA, rsLoRA, AdaLoRA, LoRA-Pro, LoRA-GA, LoRA+ and LoFT). The right panel presents evaluation perplexity curves, with dashed horizontal line at 38.87 marking the zero-shot baseline. Table for reference: Table
Scaling to GPT-2 Large. We repeat the experiment on GPT-2 Large (812M) with the same data and hyper-parameter (batch size reduced to to memory for full fine-tuning). Table 15 extends the comparison to this larger model. The zero-shot model perplexity is 38.87. Full fine-tuning brings this down to 19.42, but LoFT achieves an even lower 19.26 while updating only small fraction of weights. The other adapter-style baselines cluster few tenths higher (LoRA 19.78, rsLoRA 19.62, LoRA* 19.73). and AdaLoRA again lags behind at 23.31. 31. In relative terms, LoFT improves on the vanilla LoRA baseline by 2.6% and narrows (indeed, slightly surpasses) the gap full fine-tuning, confirming that the gains observed on the smaller GPT-2 model persist and even strengthen at a larger scale.
26

---

Published as a conference paper at ICLR 2026
Figure 8 highlights interesting trend: GPT-2 Large. Full FT achieves the lowest training perplexity, but its evaluation perplexity stalls above LoFT, evidence overfitting as model capacity grows. By contrast, the low-rank structure of LoFT provides built-in regularizer: it follows Full-FT during training yet generalizes better, maintaining leading evaluation PPL. On the smaller GPT-2 (137M), Full-FT still wins both train and evaluation there, capacity is not large enough overfit the WikiText2 dataset whereas at 812M parameters, the risk of memorization rises and LoFT's parameter-effiiien updates prove more robust.
LANGUAGE MODEL EXPERIMENTS
COMMONSENSE REASONING
For completeness. Table 16 provides the exact task-wise accuracy scores for all methods and rank settings shown in Figure of the main paper. These results quantify how LoRA, DoRA, and LoFT behave across eight commonsense reasoning benchmarks when applied to LLaMA-7B with rank r E 14.
As noted in the main text, LoFT maintains high and stable accuracy across all tasks, even under extreme compression (rank I), whereas both LoRA and DoRA degrade substantially - especcally on more complex tasks like HellaSwag (HS), Winogrande (WG), and SIQA. Notably, DoRA at r-4 and r=2 exhibits drastic task-level failures, with near-zero performance on WG and erratic behavior across others, reflecting instability under constrained adaptation. contrast, LoFT consistently performs well across ranks, confirming its robustness under limited parameter budgets.
See Table 16 for the exact per-task numbers.
Table 16: Task-wise performance of LoRA, DoRA, and LoFT on commonsense reasoning benchmarks at lower ranks (r {4, 2, 1}) using LLaMA-7B. While LoFT maintains stable accuracy across all tasks, both LoRA and DoRA show significant drops particularly on complex benchmarks such as HellaSwag and Winogrande indicating their limited reliability under extreme parameter constraints.
Model
Method
RoolQ
PIQA
SIQA
WG
ARC-C
ARC-E
OBQA
avg.
LoRA.=4
66.15
43.47
42.12
24.46
72.85
47.18
33.03
48.80
49.76
LoRA.
67.77
66.50
40.63
21.85
53,28
50.26
63.51
52.00
51.97
LoRA..1
66.15
74.05
73,58
35.24
77.19
59.56
76.43
70.80
66.62
DoRA,
32.35
7.13
47.03
27.54
0.00
52.65
66.37
46.60
34.96
LLaMA-7B
DoRA,
57.55
70.38
76.41
48.55
9.71
62.03
78.66
75.40
59.84
D
67.16
77.26
76.25
31.38
20.60
57.34
70.50
64.00
58.06
LoFT, 67.34
80.96
76.20
80.50
76.40
63.62
79.21
75.40
74.95
LoFT, 68.03
79.
75.84
78.86
76.24
64.51
78.03
71.00
73.96
LoFT,=1
67.09
78.35
74.46
76.14
74.82
58.87
76.85
70.80
72.17
F.2 MATHEMATICAL REASONING AND QUANTIZED LOFT
Setup. We evaluate exact-match accuracy on the Orca-Math dataset (Mitra et al. 2024) using LLaMA2 and LLaMA3 models. Our experimental setup is largely based on the QLoRA fine-tuning recipe outlined by Answer.ai (Turgutlu al., 2024), with a few key modifications. Specifically, we quantize the pre-trained model to 4-bit and fine-tune each model for 3 epochs on 200k training examples using bf16 precision, a global batch size of 32, the AdamW optimizer, and a shortened context window of 256 tokens. Evaluation is performed on 500 held-out examples using exact-match comparison, following the original methodology. We adopt the zero-shot and five-shot prompting results directly from the blog post: for LLaMA2, these are 0.07 and 0.08, and for LLaMA3, 0.23 and 0.27, respectively.
For parameter-effficien fine-tuning, we compare QLoRA (Dettmers et al. 2023) with proposed method, QLoFT - a quantized variant of LoFT designed for greater efficiency. We evaluate QLoRA at fixed rank of 16, yielding 0.15 accuracy LLaMA2 and 0.292 accuracy LLaMA3. Under the same rank (r=16), QLoFT achieves higher accuracy: 0.16 on LLaMA2 and 0.324 on LLaMA3. To assess robustness under constrained parameter budgets, further reduce QLFFT's rank to 8,
27

---

Published as a conference paper at ICLR 2026
LLaMA2
LLaMA3
0.175
0.150
0.150
0.154
0.148
0.164
0.30
0.292
0.324
0.312
0.318
0.276
0.25
0:230
Accuracy
0.100
0.20
0.080
0.075
0.070
0.15
0.050
0.025
0.05
0.000
Zero-shot
Five-shot
QLORA (r=16)
QLoFT (r=16)
QLoFT (r=8)
QLoFT (r=4)
QLaFT (r=1)
Zero-shot
Fivesshot
QLORA (r=16)
QLoFT (r=16)
QLOFT (r=
QLOFT (r=4)
QLoFT (r=1)
Figure 9: Accuracy comparison on the Orca-Math dataset using LLaMA2 and LLaMA3 models. We compare our method, QLoFT, quantized version of LoFT, with QLoRA. QLoFT is evaluated at various ranks (r = {16, 8, 4, 1J) and consistently outperforms QLoRA, demonstrating superior performance in parameter-efiii
fine-tuning for mathematical reasoning.
and 1. Even with 75% fewer trainable parameters (r=4), QLoFT maintains strong performance
0.148 on LLaMA2 and 0.318 on LLaMA3 matching or exceeding QLoRA's results. At 1= it still performs competitively, reaching 0.164 LLaMA2 and 0.276 on LLaMA3.
Overall, QLoFT consistently outperforms QLoRA at equivalent ranks across both model backbones, demonstrating better adaptation capacity with identical parameter budgets. More importantly, the performance drop as the rank decreases is surprisingly small, highlighting QLoFT's ability to retain strong accuracy even highly constrained regimes. On LLaMA3, the benefits are even more pronounced: QLoFT outperforms QLoRA by over 3 points at r=16, and continues to lead (8 4). This suggests that QLoFT better leverages the capacity of larger models, effectively leveraging their increased capacity for improved tuning.
F.3 SCALING TO LLAMA-3-708
To further evaluate the scalability of LOFT, we conduct experiments on the substantially larger LLaMA-3-70B model. These experiments follow the same unified training protocol used in the main commonsense reasoning evaluation. All hyperparameters follow the same configuration as explained in Section C.2 with varying rank and the learning rate to 5 10-3 Table 17 reports the results for LoFT at ranks r E {16, 4, 2, 13. These results confirm that the robustness trends observed on smaller backbones persist the 70B scale, even rank 1.
Table 17: Commonsense reasoning results on LLaMA-3-70B using LoFT at varying ranks.
Method
BoolQ
PIQA
SIQA
HS
WG
ARC-C
ARC-E
OBQA
avg.
LoFT (r = 16)
79.45
92.82
82.24
97.02
90.06
92.75
97.52
93.40
90.66
LoFT (r 4)
79.17
92.60
81.58
96.70
89.34
92.06
97.31
92.60
90.17
LoFT (r -2)
78.38
92.55
81.78
96.61
88.40
92.06
97.31
92.40
89.94
LoFT (r -1)
77.83
91.51
80.35
95.79
86.03
91.47
97.05
91.40
88,93
G VISION EXPERIMENTS
G.1 TRAINING DYNAMICS
In Figure of the main paper, we presented the training performance curves on the HAM10000 dataset. Here, in Appendix Figure 10 show analogous training-loss dynamics (log scale) for the
28

---

Published as a conference paper at ICLR 2026
three remaining image- coage-classification benchmarks: ISIC2019, Diabetic Retinopathy, and DomainNet. Each panel plots the raw per-step loss (a=0.225) beneath a 10-step centered moving average, with zoomed inset in the upper-right corner of the latter two datasets to highlight differences in the final epochs.
1SIC2019
Diabetic Retinopathy
DomainNet
2,00
Loss (log)
107
10"
475a
5000
$250
3 3000
Loos0, 18503 15030
2 10"
x 10-1
250 500 750
2000
4000
0
5000 10000 15000 20000 Training Iterations
Training Iterations
Training Iterations
LoRA
Full FT
LoFT
Figure 10: Additional training-loss dynamics for image classification. For the remaining bench- marks, ISIC2019 (left), Diabetic Retinopathy (center), and DomainNet (right), we plot training loss. LoFT (magenta) consistently outperforms LoRA (bluc) and closely tracks full fine-tuning (black), achieving the lowest loss Diabetic Retinopathy and substantially narrowing the gap ISIC2019 and DomainNet. See Figure 5 in the main paper for the HAM10000 curves.
Across all three tasks, LoFT (magenta) consistently outperforms LoRA (blue) and closes much the gap to full fine-tuning (black). In particular:
Diabetic Retinopathy: LoFT achieves the lowest training loss all three methods through- demonstrating its strongest advantage in this medical imaging dataset.
LoFT again reduces loss more quickly than LoRA and tracks very closely to full fine-tuning, especially in the later stages. While full FT still attains the absolute minimum loss, LoFT narrows the difference relative to LoRA.
ISIC2019 & DomainNet:
G.2 DOMAINNET: DOMAIN-SPECCIFC RESULTS
We would like to include the extended results of the experiment on the DomainNet dataset, including domain-specific performance results.
Table 18 complements the cross-dataset comparison in Table 3 (main paper) by breaking down the DomainNet dataset results by domain (clipart, infograph, painting, quickdraw; real, and sketch). All runs use the same ViT-Base backbone and optimization protocol described in Section
Table 18: Domain-specific accuracy results on the DomainNet dataset. While overall DomainNet results presented in the main paper, this table provides detailed per-domain accuracy for various parameter-efficient fine-tuning methods.
DomainNet Dataset
Model
Method
clipart
infograph
painting
quickdraw
real
sketch
avg
Full FT
78.92
44.09
73. 11
69.15
83.92
69.00
69.70
LoRA,
77.64
42.86
72.44
66.59
84.50
67.21
68.54
ViT-Base
DoRA,=16
73.15
40.14
69.46
60.83
82.60
64.38
65.09
LoFT,
78.11
42.95
72.80
68.10
84.55
68.37
69.15
LoFT,
76.77
42.04
71.56
65.99
84.30
67.09
67.96
LoFT, 73.38
40.15
69.58
60.98
82.83
64.10
65. 17
The overall DomainNet numbers reported Table BaeeBBalreadd already show that LoFT, narrows the gap to Full FT and outperforms both LoRA and DoRA. However, DomainNet's six domains differ markedly
29

---

Published as a conference paper at ICLR 2026
in style and label distribution; the per-domain breakdown reveals how each method copes with this heterogeneity.
Main observations:
Full fine-tuning remains strongest on average (69.7%) topping five of six domains.
LoFT with r=16 trails Full FT by only 0.55pp on average and surpasses the full FT on the real domain (84.55%)
LoRA lags LoFT on every domain except real, where both methods are statistically tied.
DoRA and low-rank LoFT variants (r= 4}) show the expected accuracy drop, but LoFT retains at least parity with the corresponding LoRA/DoRA settings.
In the main paper, we reported validation-set accuracy keep the test labels unseen. For the extended analysis here we evaluate on the official test split (176743 images) to give a complete picture of domain-lever generalization. No hyper-parameters were tuned on the test set; models are exactly those used in the main paper.
G.3 COMPARISON WITH VISION CENONCCENTRIC BASELINES
We further compare LoFT against two representative vision adaptation baselines: Visual Prompt Tuning (VPT) (Ti et al. 2022) and ViT-Adapter (Houlsby al. 2019 Chen et al. 2023). To cover
both standard and challenging scenarios, we evaluate on CIFAR 100 (general image classification)
(Krizhevsky al.] 2009) and Diabetic Retinopathy medical imaging dataset) (Graham 2015).
Table summarizes the results.
Table 19:
Comparison of LoFT with VPT and ViT-Adapter on CIFAR-100 and Diabetic Retinopathy datasets using ViT-Base.
Method
CIFAR-100
Diabetic Retinopathy
Adapter
85,93
51.82
VPT
90.03
50.28
LoFT
91.20
58.49
LoFT achieves the best performance across both datasets. While VPT and Adapter perform well on CIFAR- 100, they struggle on the more complex medical dataset, whereas LoFT maintains strong results in both settings.
H CODING EXPERIMENTS
To further evaluate LoFT representative large language model benchmarks for code genera- tion, we conduct additional experiments on HumanEval and HumanEval+ (Chen et al., (021). We use LLaMA3-8B as the backbone model and fine-tune it the CodeFeedback (Python-only) dataset (Zheng et al. 2024), which consists of curated instruction-respons pairs focused on Python code completion and correction. All experiments follow the same training recipe described in the main paper; the only variable changed across configurations is the rank r of the low-rank adaptation. Evaluation is performed using the EvalPlus framework (Liu et al. 2023). For each problem, we sample 10 completions at temperature 7 0.5 and compute @ k for k E {1,5, 10). HumanEval+ a stricter version of HumanFval with significantly expanded unit tests per task, providing a more rigorous assessment of functional conrectness.
We first study the effect of increasing the rank beyond the range r < 32 considered the main paper. Table 20 reports results for LoFT (simple) at r {8, 16, 32, 64, 128, 256} and compares against LoRA under identical training and evaluation settings. At low rank (r = 8), LoFT already achieves strong performance, with pass@) I of 0,460 HumanEval and 0.405 HumanEval+. As the rank increases, performance improves modestly, reaching peak pass at = 128 (0.485 on HvaanEval and 0.436 HumanEval+), before flattening 256. For broader sampling metrics,
30

---

Published as a conference paper at ICLR 2026
Table 20: Rank scaling results on HvmanEval and HvmanEval+.
HumanEval
HumanEval+
Method
rank
pass@1
pass
pass@ 10
pass@1
pass@5
pass
8
0.460
0.682
0.738
0.405
0.624
0.683
16
0,455
0.674
0.744
0.409
0.623
0.695
32
0,468
0.673
0.732
0.423
0.619
0.679
LoFT (simple)
64
0.465
0.684
0.768
0.415
0.626
0.707
128
0,485
0.678
0.750
0.436
0.627
0.697
256
0.482
0.665
0.736
0.432
0.623
0.681
8
0.421
0.631
0.720
0.368
0.572
0.659
LoRA
64
0.463
0.670
0.738
0.416
0.622
0.701
Table 21: Context length comparison at fixed rank r LoFT consistently outperforms LoRA across both context settings and both benchmarks.
HumanEval
HumanEval+
Method
Context
pass @
pass @5
pass @ 10
pass @
pass@5
pass @ 10
LoRA
256
0.421
0.631
0.720
0.368
0.572
0.659
LoFT
256
0.482
0.681
0.756
0.439
0.614
0.689
LoRA
512
0.433
0.640
0.695
0.380
0.583
0.646
LoFT
512
0.488
0.674
0.738
0.445
0.612
0.671
pass@ 10 reaches its highest value r 64 both benchmarks (0.768 on HumanEval and 0.707 on HvmanEval+)). These results indicate clear diminishing returns as rank increases, showing that small ranks already capture most of the achievable gains.
Importantly, LoFT demonstrates strong efficiency in the low-rank regime. At matched rank r LoFT outperforms LoRA (0.460 vs. 0.421 pass@ on HumanEval, and 0.405 vs. 0.368 on HumanEval+). Notably. LoFT at = 8 matches or exceeds the performance of LoRA at 64, indicating that LoFT achieves comparable aceuracy with substantially smaller rank. This highlights improved expressivity per parameter and better optimization alignment in the low-rank setting.
We further examine the effect of context length by evaluating both LoFT and LoRA at sequence lengths of 256 and 512 tokens with a fixed rank r 8. Results are shown in Table 21 LoFT consistently outperforms LoRA across both context lengths and both benchmarks. On HumanEval, LoFT achieves pass @ of 0.482 and 0.488 at context lengths 256 and 512. respectively, compared to 0.421 and 0.433 for LoRA. On HumanEval+, we observe a similar trend. The improvements remain stable across context windows, suggesting that the gains not sensitive to moderate changes in sequence length.
31


Reviews:
[{"review_id": "FOfnmZ2dI3", "reviewer": "Reviewer_Rnyc", "summary": "This paper introduces a new LoRA method which explicitly tries to mimic full-finetuning dynamics. Crucially, the paper identifies the importance of matching the optimizer state in addition to the updates. This is accomplished via \n\n1. alternating updates\n2. gradient rescaling\n3. momentum recalibration\n4. second moment recalibration\n5. projecting reconstructed AdamW update\n6. approximating gradient clipping\n\nThese \"building blocks\" ensure that in the limit of full-rank the full-finetuning dynamics are recovered. In the low-rank regime the experiments demonstrate improved performance over vanilla LoRA.", "questions": "Do the authors have any intuition about when mimicking the full finetuning update is optimal or not?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The authors provide a principled derivation for a LoRA method meant to explicitly mimic full-finetuning updates. The approach recovers the correct dynamics in the full-rank limit. The authors provide extensive experiments showing the promise of the method especially for low-ranks. The method is practically efficient, it requires modest memory and runtime overhead, and is simple to implement.", "weaknesses": "The experiments are only conducted with $r \\leq 32$ and models with $\\leq 8$B parameters.\n\nSecond-moment calibration appears to have low-impact at a high cost, however it is still valuable to derive and test this idea.\n\nIt is unclear if alternation is helpful or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v2YgGa0OLI", "reviewer": "Reviewer_mFX4", "summary": "This paper proposes LoFT (Low-rank adaptation that behaves like Full fine-Tuning), a novel parameter-efficient fine-tuning (PEFT) method designed to closely approximate the optimization dynamics of full fine-tuning within a low-rank subspace. Building on the LoRA framework, LoFT introduces several key components: alternating updates, gradient scaling, first- and second-moment state calibration, projected full-model updates, and gradient clipping. Together, these allow LoFT to mimic AdamW’s optimizer behavior while maintaining the computational and inference efficiency of low-rank tuning.\n\nEmpirical results across language (LLaMA-7B/2-7B/3-8B) and vision (ViT-Base) models demonstrate that LoFT consistently outperforms existing PEFT methods such as LoRA and DoRA, particularly under extreme low-rank constraints (e.g., rank ≤ 4). Ablation studies confirm that optimizer state calibration is critical to LoFT’s strong performance.", "questions": "See Weaknesses.\n\nIf the authors are willing to carefully clarify the relationship and differences between LoFT and AltLoRA during the rebuttal phase, I would be inclined to raise my score.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Strong conceptual motivation**: The paper identifies a previously underexplored source of suboptimality in LoRA — optimizer state misalignment — and provides a well-motivated correction grounded in optimization theory.\n\n**Methodological completeness**: The framework integrates multiple components (gradient projection, alternating updates, moment calibration) into a cohesive, well-defined optimizer (LoFT-AdamW), which provably reduces to full fine-tuning when rank = full.\n\n**Theoretical insight**: The analysis on matrix factorization clearly shows how LoFT recovers full fine-tuning dynamics, with formal smoothness guarantees and equivalence to alternating least squares in the special case.\n\n**Extensive empirical validation**: The experiments span multiple model families and domains, including large LLMs and ViTs, with clear, consistent performance improvements over LoRA and DoRA.\n\n**Careful ablation studies**: The paper convincingly demonstrates the necessity of each component, especially the importance of first-moment calibration for stable convergence.\n\n**Practical relevance**: LoFT eliminates the need to tune the LoRA scaling factor (α), reducing hyperparameter sensitivity and simplifying deployment.", "weaknesses": "**Missing citation and discussion of concurrent work**:\nThe Alternating Updates component (Building Block 1) reproduces an idea conceptually similar to AltLoRA [1], which independently proposed alternating optimization of low-rank factors to eliminate second-order coupling in LoRA updates.\n\nThe absence of a citation or discussion of AltLoRA is a notable omission, especially since the “alternating update” mechanism is presented as a key innovation. This should be acknowledged as concurrent or parallel work, with clarification of LoFT’s additional contributions beyond AltLoRA (notably optimizer-state alignment).\n\n**Complexity and memory overhead**: While the paper discusses the cost of storing previous iterates and cross-terms, the actual scalability to very large models (≥70B parameters) remains untested; empirical results are limited to ≤8B models.\n\n**Presentation clarity**: The main text can be dense, with many mathematical expressions introduced in rapid succession. It would be helpful to include more detailed mathematical explanations or derivations in the appendix to improve readability and reproducibility.\n\n[1] Yu, Xin, et al. \"AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections.\" arXiv preprint arXiv:2505.12455 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5MHgK7VW93", "reviewer": "Reviewer_5NnX", "summary": "The paper proposes a new method called LoFT for parameter-efficient fine-tuning of large pre-trained models. LoFT extends the Low-Rank Adaptation (LoRA) approach by aligning the internal states of the optimizer (including momentum and second moments) with full fine-tuning, thereby attempting to reduce the accuracy gap typically seen between low-rank and full fine-tuning methods. The authors test their method across multiple language and vision tasks, showing performance improvements compared to previous low-rank adaptation methods, especially at very low ranks. They also discuss trade-offs in terms of memory usage and computational overhead, presenting simpler variants with lower overhead.", "questions": "In Table 6, several DoRA results (e.g., r=4 with BoolQ=32.35, PIQA=7.13, Winogrande=0.00) are anomalously low and LoRA sometimes degrades as rank increases (e.g., r=4 worse than r=1 on PIQA/HellaSwag), suggesting hyperparameter/setup issues—could you explain these discrepancies?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- **Substantive technical contribution with theory.** The paper proposes a concrete improvement over standard LoRA-style adaptation and backs it up with clear derivations/analysis. The core ideas are technically motivated (e.g., aligning updates with full fine-tuning dynamics), and the method’s components are explained rather than presented as ad-hoc tricks.\n- **Broad empirical validation across domains.** Experiments cover multiple modalities/datasets (e.g., language and vision) and a range of ranks/settings, suggesting the approach is not narrowly tailored to a single task.", "weaknesses": "- **Gap between theory and the strongest claim.** While the derivations are compelling, there remains a gap between the formal analysis and the paper’s strongest claim(s) (e.g., exact equivalence to full fine-tuning/AdamW under certain limits). A precise theorem with assumptions, or a more cautious phrasing, would strengthen the work.\n- **LLM evaluation is too basic.** The large-language-model experiments rely on relatively easy, small benchmarks. For a model like Llama-3-8B, a more representative LLM evaluation suite (e.g., code, math/reasoning, or long-context benchmarks) would be more convincing. Multi-seed runs with statistical reporting would further solidify the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mr8EjZxaHg", "reviewer": "Reviewer_vRxF", "summary": "This paper introduces LoFT, a new parameter-efficient fine-tuning method designed to make low-rank adaptation behave like full fine-tuning. Existing LoRA-based approaches mainly focus on gradient approximation but ignore the optimizer state misalignment, particularly in the first and second moments of the AdamW optimizer. LoFT explicitly aligns both gradients and optimizer states with full fine-tuning dynamics through six carefully designed components: alternating updates, gradient scaling, optimizer state calibration, second-moment alignment, projected full update reconstruction, and gradient clipping. Theoretical analysis proves that LoFT reduces exactly to AdamW when the rank equals the full dimension. Experiments on multiple large language models and vision transformers demonstrate that LoFT achieves higher accuracy and faster convergence than LoRA and DoRA while maintaining the same inference cost and number of trainable parameters.", "questions": "1. What is the quantitative impact of the additional memory requirement on training efficiency and GPU utilization?\n\n2. Can LoFT be extended to other optimizers such as Muon that use different moment estimation mechanisms?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe method directly addresses the optimizer state misalignment problem that has been largely overlooked in prior low-rank adaptation research.\n2.\tThe theoretical analysis is rigorous and provides a clear guarantee that LoFT degenerates to AdamW in the full-rank case.\n3.\tThe six-component design is systematic, and is validated through detailed ablation studies.\n4.\tLoFT consistently outperforms LoRA and DoRA on both natural language and vision benchmarks, showing broad applicability.", "weaknesses": "1.\tThe additional memory cost, which can reach about twenty-five percent compared to LoRA, is not fully analyzed for its impact on large-scale training.\n2.\tExperiments are limited to models of eight billion parameters or smaller, leaving scalability to larger models unverified.\n3.\tThe effect of optimizer state projection on stability and convergence speed is discussed conceptually but lacks quantitative analysis.\n4.\tThe paper does not report concrete throughput or training speed measurements compared with LoRA or DoRA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)