LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuning

## Abstract

Abstract Large pre-trained models are commonly adapted to downstream tasks using parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA), which injects small trainable low-rank matrices instead of updating all weights. While LoRA dramatically reduces trainable parameters with little overhead, it can still underperform full fine-tuning in accuracy and often converges more slowly. We introduce LoFT, a novel low-rank adaptation method that behaves like full fine-tuning by aligning the optimizer’s internal dynamics with those of updating all model weights. LoFT not only learns weight updates in a low-rank subspace (like LoRA) but also properly projects the optimizer’s first and second moments (Adam’s momentum and variance) into the same subspace, mirroring full-model updates. By aligning the low-rank update itself with the full update, LoFT eliminates the need for tuning extra hyperparameters, e.g., LoRA scaling factor α \alpha . Empirically, this approach substantially narrows the performance gap between adapter-based tuning and full fine-tuning and consistently outperforms standard LoRA-style methods, all without increasing inference cost.

## 1 Introduction

Fine-tuning large-scale pre-trained models for specific tasks has become a standard paradigm in natural language processing and other domains. However, as model sizes grow into the billions of parameters, full fine-tuning (i.e., updating every weight) becomes computationally expensive and impractical, especially in multi-task (chronopoulou2023adaptersoup) or multi-user (yi2023pfedlora) settings. Parameter-efficient fine-tuning (PEFT) techniques address this challenge by updating only a small subset of parameters while reusing the vast majority of pre-trained weights. Among these, Low-Rank Adaptation (LoRA) has emerged as a popular and effective solution. LoRA freezes the original weights and injects trainable low-rank matrices into selected layers, substantially reducing the number of learnable parameters. Remarkably, LoRA often matches – and sometimes can exceed – the performance of full fine-tuning on certain benchmarks, all while incurring minimal runtime overhead and no additional inference latency. This makes it an attractive alternative to other methods like sequential adapters (houlsby2019parameter; pfeiffer2021adapterfusion), which typically introduce new layers and increased latency.

Figure: Figure 1: LoFT visualization. LoFT can be interpreted as the tightest approximation to full fine-tuning under the constraint that each update lies in the subspace defined by $V$ (when updating $U$). The LoFT-AdamW update consists of a momentum and second-moment estimate constructed using projected gradients. The final update is then projected back onto the subspace of $V$ to respect the low-rank constraint. When $V$ is the updated component instead of $U$, the roles of $U$ and $V$ are simply exchanged, and the update is applied to $W^{\top}$ instead of $W$.
Refer to caption: /html/2505.21289/assets/x1.png

Despite its success, LoRA and similar low-rank approaches still fall short of full fine-tuning in some settings. Empirical studies have reported a persistent performance gap and slower convergence rates compared to full-model updates (biderman2024lora; wang2024lora). These gaps indicate that the optimization dynamics of LoRA differ in important ways from those of full fine-tuning. Recent work (liu2024dora; wang2024lorapro) has attempted to close this gap by focusing on more accurate gradient approximations within the low-rank subspace. This is motivated by the observation that LoRA’s updates can omit or misestimate important directions in the full gradient, leading to suboptimal solutions. In this work, we demonstrate that this is only part of the story: optimizer state misalignment – specifically in the first and second moments used by AdamW (loshchilov2017decoupled), the de facto optimizer in large-scale training – also plays a critical role. When these internal statistics are not properly aligned with the low-rank constraint, it undermines the effectiveness of the adaptation.

Finally, a practical complication in standard LoRA is the introduction of a scaling hyperparameter, $\alpha$, often normalized by the rank. This scaling factor modulates the contribution of the low-rank update and must be carefully tuned. Improper settings can lead to poor performance or even divergence by overpowering the backbone model (lee2023platypus; malinovsky2024randomized). Altogether, these challenges – i.e., the gradient and optimizer state misalignment, as well as the additional hyperparameter sensitivity – limit LoRA’s ability to fully replicate the robustness and effectiveness of unconstrained full fine-tuning.

Our main contributions are summarized as follows:

- •
We identify that not only gradients but also optimizer states (i.e., first and second moments) suffer from misalignment when approximating full fine-tuning with low-rank updates.
- •
We propose Low rank adaptation that mimics Full fine-Tuning (LoFT), a novel LoRA-based optimizer that addresses these issues by closely approximating full fine-tuning across all optimization dimensions. LoFT consists of five core components: gradient scaling, alternating updates, optimizer state calibration, construction of a projected full fine-tuning update followed by low-rank projection, and projected full fine-tuning-aware clipping.
- •
To the best of our knowledge, LoFT is the first low-rank adaptation method that exactly reduces to AdamW (loshchilov2017decoupled) in the full-rank limit.
- •
We conduct extensive experiments on both synthetic and real-world tasks across multiple modalities, demonstrating the effectiveness and generality of LoFT.

## 2 Method

We focus on the standard fine-tuning setup, where a pre-trained model is adapted to a downstream task. In full fine-tuning, each weight matrix $W$ is updated by a full-rank increment $\Delta W$. To reduce computational cost, LoRA proposes a low-rank reparameterization

$$ $\displaystyle W=W_{0}+\Delta W=W_{0}+UV^{\top},$ $$

where $W\in\mathbb{R}^{m\times n}$, $U\in\mathbb{R}^{m\times r}$, $V\in\mathbb{R}^{n\times r}$, and $r\ll\min\{m,n\}$. Only $U$ and $V$ are trainable, reducing the gradient and optimizer state footprint to $\mathcal{O}((m+n)r)$ compared to $\mathcal{O}(mn)$ in full fine-tuning. LoRA typically introduces a scaling factor $\alpha>0$ to modulate the magnitude of the low-rank update. However, in our study, we set $\alpha=1$ and attribute the need for this hyperparameter to a misalignment between LoRA and full fine-tuning, which we address in the subsequent sections.

### 2.1 Gradient Descent for Full Fine-Tuning vs. LoRA

Let $f(W):\mathbb{R}^{m\times n}\to\mathbb{R}$ denote a scalar loss function with $W$ representing the parameters of a single linear layer. In standard full fine-tuning with gradient descent, the update is

$$ $\displaystyle W^{+}=W-\eta\nabla_{W}f(W),$ (1) $$

where $\eta>0$ is the learning rate, and $\nabla_{W}f(W)$ is the gradient of the loss with respect to $W$. With LoRA parametrization, the update becomes

$$ $\displaystyle W^{+}=W_{0}+U^{+}(V^{+})^{\top}=W_{0}+(U-\eta\nabla_{U}f(W))(V-\eta\nabla_{V}f(W))^{\top}.$ (2) $$

Applying the chain rule yields

$$ $\displaystyle\nabla_{U}f(W)=\nabla_{W}f(W)V,\qquad\nabla_{V}f(W)=\nabla_{W}f(W)^{\top}U.$ $$

Substituting these into ([2](#S2.E2)) gives

$$ $\displaystyle W^{+}=W-\eta\left(\nabla_{W}f(W)VV^{\top}+UU^{\top}\nabla_{W}f(W)\right)+\eta^{2}\nabla_{W}f(W)UV^{\top}\nabla_{W}f(W).$ (3) $$

**Table 1: The six core building blocks of LoFT for aligning low-rank adaptation with full fine-tuning.**
| Component | Purpose |
| --- | --- |
| Alternating Updates ([2.1](#S2.SS1)) | Eliminate second-order cross terms from LoRA dynamics. |
| Gradient Scaling ([2.1](#S2.SS1)) | Ensure scale-invariance of low-rank updates. |
| Optim. States Calibration ([2.2](#S2.SS2), [2.3](#S2.SS3)) | Align moments estimates across changing low-rank spaces. |
| Projected Full Update ([2.3](#S2.SS3)) | Reconstruct the full-model update and project it onto the low-rank subspace. |
| Gradient Clipping ([2.4](#S2.SS4)) | Match full fine-tuning clipping behavior. |

Equation ([3](#S2.E3)) highlights a key discrepancy between LoRA and full fine-tuning: the additional $\eta^{2}$ term, which depends quadratically on the gradient. While seemingly small, this term can materially affect convergence, as we show later in a controlled experiment. A straightforward way to eliminate this term is through alternating updates.

Without loss of generality, assuming we update only $U$, the resulting update to $W$ becomes

$$ $\displaystyle W^{+}=W-\eta\nabla_{W}f(W)VV^{\top}.$ (4) $$

However, this update suffers from a scale ambiguity: for any $c\neq 0$, $UV^{\top}=(cU)(V/c)^{\top}$, but the update scales differently with $c$. To resolve this, observe that the update direction lies in the column space of $V$, allowing us to scale the update using an $r\times r$ matrix(^1^11We assume $V$ is of full rank. If not, we can use the pseudo-inverse.) $\left(V^{\top}V\right)^{-1}$

$$ $\displaystyle W^{+}=W-\eta\nabla_{W}f(W)V{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\left(V^{\top}V\right)^{-1}}V^{\top}=W-\eta\nabla_{W}f(W){\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\cal P}_{V}},$ (5) $$

where $\mathcal{P}_{V}=V(V^{\top}V)^{-1}V^{\top}$ is the projection matrix onto the column space of $V$. This ensures the update is the closest low-rank approximation to $\nabla_{W}f(W)$ under the given subspace. The associated computational cost is $\mathcal{O}(r^{3})$. This update defines our second building block.

We are not the first to suggest this; zhang2024riemannian derived a similar result from the perspective of Riemannian optimization.

### 2.2 First Moment Misalignment

In practice, gradients are often estimated using momentum. Specifically, the first moment $m_{k}$ is computed as
$m_{k}=\beta_{1}m_{k-1}+(1-\beta_{1})g_{k},$
where $\beta_{1}\in[0,1)$ is the momentum coefficient and $g_{k}$ is the stochastic gradient at step $k$. For full fine-tuning, the resulting momentum update is

$$ $\displaystyle m^{W}_{k}=(1-\beta_{1})\sum_{i=0}^{k}\beta_{1}^{k-i}\nabla_{W}f(W_{i}).$ (6) $$

where $W_{i}$ denotes the parameter value at step $i$. When updating $U$ under the LoRA parameterization, the effect on $W$ becomes

$$ $\displaystyle m^{U}_{k}V^{\top}$ $\displaystyle=(1-\beta_{1})\sum_{i=0}^{k}\beta_{1}^{k-i}\tilde{\nabla}_{U}f(W_{i})V^{\top}=(1-\beta_{1})\sum_{i=0}^{k}\beta_{1}^{k-i}\nabla_{W}f(W_{i})V_{i}\left(V_{i}^{\top}V_{i}\right)^{-1}V_{k}^{\top},$ $$

which does not represent a proper projection due to the mismatch between $V_{i}$ and $V_{k}$. To address this, we introduce a recalibration step

$$ $\displaystyle m^{U}_{k}$ $\displaystyle=\beta_{1}m^{U}_{k-1}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}C_{k}}+(1-\beta_{1})\tilde{\nabla}_{U}f(W_{i}),$ (7) $$

where ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}C^{V}_{k}\overset{\text{def}}{=}(V_{k-1}^{\top}V_{k})(V_{k}^{\top}V_{k})^{-1}}$ is a calibration matrix. Substituting this back gives

$$ $\displaystyle\tilde{m}^{U}_{k}=m^{U}_{k}V^{\top}=(1-\beta_{1})\sum_{i=0}^{k}\beta_{1}^{k-i}\nabla_{W}f(W_{i}){\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\cal P}_{\cap_{j=i}^{k}V_{j}}}=(1-\beta_{1})\sum_{i=0}^{k}\beta_{1}^{k-i}g^{V}_{i},$ (8) $$

where ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\mathcal{P}_{\cap_{j=i}^{k}V_{j}}=\prod_{j=i}^{k}\mathcal{P}_{V_{j}}}$ is the projection onto the intersection of the column spaces of $V_{j}$ for $j=i$ to $k$. Let $g^{V}_{i}\overset{\text{def}}{=}\nabla_{W}f(W_{i})\mathcal{P}_{\cap_{j=i}^{k}V_{j}}$. This expression provides the tightest possible estimate of the momentum under the constraints of the evolving low-rank subspaces defined by $V_{i}$’s. In implementation, storing previous iterates $\{V_{k-1},U_{k-1}\}$ incurs an additional memory cost of $\mathcal{O}((m+n)r)$.

### 2.3 Second Moment Misalignment

For Adam-style updates, the ideal update to $W$ when $U$ is being updated given subspace constraints would be

$$ $\displaystyle\frac{\nicefrac{{\tilde{m}^{U}_{k}}}{{(1-\beta_{1}^{k})}}}{\sqrt{\nicefrac{{\tilde{v}^{U}_{k}}}{{(1-\beta_{2}^{k})}}+\varepsilon}}{\cal P}_{V_{k}},\text{ s.t. }\tilde{m}^{U}_{k}=(1-\beta_{1})\sum_{i=0}^{k}\beta_{1}^{k-i}g^{V}_{i},\quad\tilde{v}^{U}_{k}=(1-\beta_{1})\sum_{i=0}^{k}\beta_{1}^{k-i}g^{V}_{i}\odot g^{V}_{i},$ (9) $$

where $\tilde{m}^{U}_{k}$ is as defined in ([8](#S2.E8)), and $\tilde{v}^{U}_{k}=(1-\beta_{2})\sum_{i=0}^{k}\beta_{2}^{k-i}g^{V}_{i}\odot g^{V}_{i}$ is the second moment estimate. The symbol $\odot$ denotes element-wise multiplication. Note that this update is constructed to lie in the subspace defined by $V_{k}$, in accordance with ([4](#S2.E4)).

To compute $\tilde{v}^{U}_{k}$ efficiently, we use the following identities from slyusar1999family

$$ $\displaystyle(A\bullet B)(C\otimes D)=(AC)\bullet(BD),\qquad(AB)\odot(CD)=(A\bullet C)(B\ast D)$ (10) $$

where $\otimes$ is the Kronecker product, $\bullet$ is the transposed Khatri–Rao product, and $\ast$ is the standard Khatri–Rao product. We define the calibrated second-moment accumulator as

$$ $\displaystyle p^{U}_{k}=\beta_{2}p^{U}_{k-1}(C^{V}_{k}\otimes C^{V}_{k})+(1-\beta_{2})(\tilde{\nabla}_{U}f(W_{i})\bullet\tilde{\nabla}_{U}f(W_{i})),$ (11) $$

where $p^{U}_{k}$ is a matrix of size $nr\times r$ that stores the cross-terms necessary to reconstruct the second moment after transformation. The associated memory overhead is $\mathcal{O}((m+n)r^{2})$, which is the main limitation of our approach. For this reason, maintaining a small rank $r$ is crucial for memory efficiency. In practice, this constraint is acceptable as long as $r\leq\sqrt{\min\{m,n\}}$, which we find to be both reasonable and sufficient for capturing effective low-rank updates. To address this limitation in future work, we plan to investigate variants of LoFT using LLM-specific optimizers where all optimizer states are linear functions of stochastic gradients, such as Muon (jordan2024muon).

Using $p^{U}_{k}$, we compute $\tilde{v}^{U}_{k}=p^{U}_{k}(V_{k}\ast V_{k})$ and apply the following update.

### 2.4 Gradient Clipping and Weight Decay

We apply no special modifications to weight decay. Since only one of $U$ or $V$ is updated at a time, the effect of standard weight decay correctly reduces the low-rank update as $UV^{\top}\rightarrow(1-\lambda\eta_{k})UV^{\top}$. The full AdamW-LoFT algorithm is provided in the appendix.
*With all six building blocks described above, LoFT-AdamW exactly recovers full fine-tuning when $r=\min\{m,n\}$ and $U_{k}$, $V_{k}$ are full-rank. To our knowledge, LoFT is the first low-rank adaptation method that provably recovers full fine-tuning in this limit.*

### 2.5 Simulated Experiment

Figure: Figure 2: Comparison of LoRA, LoFT, and Full Fine-tuning with Adam on $f(W)=\|W-A\|_{F}^{2}$.
Refer to caption: /html/2505.21289/assets/x2.png

In the previous remark, we argued that for full-rank adaptation, LoFT recovers full fine-tuning. We now demonstrate that if the target solution is low-rank, LoFT matches the performance of full fine-tuning if the correct rank is selected.
We consider the optimization problem $f(W)=\|W-A\|_{F}^{2}$, where $A$ is a randomly generated matrix with $\mathrm{rank}(A)=r$. We compare LoFT, LoRA, and full fine-tuning using the AdamW optimizer. The step size is tuned for full fine-tuning and reused for all baselines. We initialize $W=0$, and for LoFT we follow the standard LoRA initialization (hu2022lora), which also yields $UV^{\top}=0$ initially.
We set $m=1024$, $n=512$, and $r=8$. In addition to LoFT and LoRA, we also include ablated variants of LoFT to highlight the importance of its design components: one without alternating updates, and one without optimizer state calibration.
As shown in Figure [2](#S2.F2), LoFT closely matches the performance of full fine-tuning. In contrast, omitting any of its core components leads to significantly slower convergence and worse final performance, confirming the necessity of the full LoFT design.

## 3 Experiments

We conduct extensive experiments across both language and vision domains to evaluate the effectiveness of our method. Our primary baselines include LoRA (hu2022lora), DoRA (liu2024dora), and full fine-tuning, and we apply these methods to a range of model backbones: LLaMA-7B (touvron2023llama1), LLaMA2-7B (touvron2023llama), LLaMA3-8B (grattafiori2024llama), and ViT-Base (wu2020visual). The evaluation spans two major fronts: (i) commonsense reasoning tasks in the language domain, and (ii) image classification tasks involving highly imbalanced and domain-specific datasets, including several medical imaging datasets and DomainNet.

In addition to the typical low-rank configuration (e.g., $\operatorname{rank}\geq 4$), we explore extremely constrained settings by reducing the rank to as low as $1$, demonstrating the robustness of our method under stringent parameter budgets. This allows us to highlight not just absolute performance but also the parameter efficiency and scalability of our approach relative to existing baselines. Further implementation and dataset details are provided in the appendix.

**Table 2: Performance comparison of parameter-efficient fine-tuning methods, LoRA, DoRA, and our method LoFT, on a suite of commonsense reasoning benchmarks using LLaMA-7B, LLaMA2-7B, and LLaMA3-8B models. The table reports accuracy scores across multiple tasks with average performance shown in the final column. $r$ denotes the rank used in the respective adaptation method. Bold and underlined scores highlight the best and second-best performance per task, respectively.**
| Model | Method | BoolQ | PIQA | SIQA | HS | WG | ARC-C | ARC-E | OBQA | avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA-7B | LoRA${}_{r=16}$ | $65.38$ | $76.71$ | $75.69$ | $79.81$ | $68.03$ | $65.27$ | $80.30$ | $77.40$ | $73.57$ |
| DoRA${}_{r=16}$ | $54.13$ | $73.94$ | $79.38$ | $58.01$ | $79.40$ | 64.68 | $79.76$ | $79.60$ | $71.11$ |  |
| LoFT${}_{r=16}$ | $68.62$ | $82.80$ | 78.27 | $82.69$ | $73.32$ | $64.30$ | 80.26 | 78.40 | $76.08$ |  |
| LoFT${}_{r=4}$ | $67.34$ | 80.96 | $76.20$ | 80.50 | 76.40 | $63.62$ | $79.21$ | $75.40$ | 74.95 |  |
| LoFT${}_{r=2}$ | 68.03 | $79.16$ | $75.84$ | $78.86$ | $76.24$ | $64.51$ | $78.03$ | $71.00$ | $73.96$ |  |
| LoFT${}_{r=1}$ | $67.09$ | $78.35$ | $74.46$ | $76.14$ | $74.82$ | $58.87$ | $76.85$ | $70.80$ | $72.17$ |  |
| LLaMA2-7B | LoRA${}_{r=16}$ | $50.09$ | $59.03$ | $76.41$ | $65.45$ | $77.51$ | $64.68$ | $79.12$ | $77.20$ | $68.69$ |
| DoRA${}_{r=16}$ | $71.93$ | 82.92 | 79.22 | 88.90 | $83.03$ | $66.98$ | $82.70$ | $82.00$ | 79.71 |  |
| LoFT${}_{r=16}$ | 71.80 | $83.51$ | $79.02$ | $90.59$ | 82.72 | $70.65$ | 84.43 | 81.00 | $80.46$ |  |
| LoFT${}_{r=4}$ | $70.49$ | $81.94$ | $79.80$ | $88.85$ | $81.37$ | 69.11 | $84.88$ | $79.80$ | $79.53$ |  |
| LoFT${}_{r=2}$ | $70.55$ | $81.18$ | $77.74$ | $83.01$ | $79.01$ | $66.72$ | $82.83$ | $78.80$ | $77.48$ |  |
| LoFT${}_{r=1}$ | $68.69$ | $80.58$ | $76.36$ | $72.95$ | $76.80$ | $64.08$ | $82.37$ | $77.20$ | $74.88$ |  |
| LLaMA3-8B | LoRA${}_{r=16}$ | $74.46$ | $88.14$ | $81.37$ | $94.81$ | $85.08$ | 80.72 | $89.18$ | 8 |  |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: Do the authors have any intuition about when mimicking the full finetuning update is optimal or not?
soundness: 4
strengths: The authors provide a principled derivation for a LoRA method meant to explicitly mimic full-finetuning updates. The approach recovers the correct dynamics in the full-rank limit. The authors provide extensive experiments showing the promise of the method especially for low-ranks. The method is practically efficient, it requires modest memory and runtime overhead, and is simple to implement.
summary: This paper introduces a new LoRA method which explicitly tries to mimic full-finetuning dynamics. Crucially, the paper identifies the importance of matching the optimizer state in addition to the updates. This is accomplished via 

1. alternating updates
2. gradient rescaling
3. momentum recalibration
4. second moment recalibration
5. projecting reconstructed AdamW update
6. approximating gradient clipping

These "building blocks" ensure that in the limit of full-rank the full-finetuning dynamics are recovered. In the low-rank regime the experiments demonstrate improved performance over vanilla LoRA.
weaknesses: The experiments are only conducted with $r \leq 32$ and models with $\leq 8$B parameters.

Second-moment calibration appears to have low-impact at a high cost, however it is still valuable to derive and test this idea.

It is unclear if alternation is helpful or not.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses.

If the authors are willing to carefully clarify the relationship and differences between LoFT and AltLoRA during the rebuttal phase, I would be inclined to raise my score.
soundness: 3
strengths: **Strong conceptual motivation**: The paper identifies a previously underexplored source of suboptimality in LoRA — optimizer state misalignment — and provides a well-motivated correction grounded in optimization theory.

**Methodological completeness**: The framework integrates multiple components (gradient projection, alternating updates, moment calibration) into a cohesive, well-defined optimizer (LoFT-AdamW), which provably reduces to full fine-tuning when rank = full.

**Theoretical insight**: The analysis on matrix factorization clearly shows how LoFT recovers full fine-tuning dynamics, with formal smoothness guarantees and equivalence to alternating least squares in the special case.

**Extensive empirical validation**: The experiments span multiple model families and domains, including large LLMs and ViTs, with clear, consistent performance improvements over LoRA and DoRA.

**Careful ablation studies**: The paper convincingly demonstrates the necessity of each component, especially the importance of first-moment calibration for stable convergence.

**Practical relevance**: LoFT eliminates the need to tune the LoRA scaling factor (α), reducing hyperparameter sensitivity and simplifying deployment.
summary: This paper proposes LoFT (Low-rank adaptation that behaves like Full fine-Tuning), a novel parameter-efficient fine-tuning (PEFT) method designed to closely approximate the optimization dynamics of full fine-tuning within a low-rank subspace. Building on the LoRA framework, LoFT introduces several key components: alternating updates, gradient scaling, first- and second-moment state calibration, projected full-model updates, and gradient clipping. Together, these allow LoFT to mimic AdamW’s optimizer behavior while maintaining the computational and inference efficiency of low-rank tuning.

Empirical results across language (LLaMA-7B/2-7B/3-8B) and vision (ViT-Base) models demonstrate that LoFT consistently outperforms existing PEFT methods such as LoRA and DoRA, particularly under extreme low-rank constraints (e.g., rank ≤ 4). Ablation studies confirm that optimizer state calibration is critical to LoFT’s strong performance.
weaknesses: **Missing citation and discussion of concurrent work**:
The Alternating Updates component (Building Block 1) reproduces an idea conceptually similar to AltLoRA [1], which independently proposed alternating optimization of low-rank factors to eliminate second-order coupling in LoRA updates.

The absence of a citation or discussion of AltLoRA is a notable omission, especially since the “alternating update” mechanism is presented as a key innovation. This should be acknowledged as concurrent or parallel work, with clarification of LoFT’s additional contributions beyond AltLoRA (notably optimizer-state alignment).

**Complexity and memory overhead**: While the paper discusses the cost of storing previous iterates and cross-terms, the actual scalability to very large models (≥70B parameters) remains untested; empirical results are limited to ≤8B models.

**Presentation clarity**: The main text can be dense, with many mathematical expressions introduced in rapid succession. It would be helpful to include more detailed mathematical explanations or derivations in the appendix to improve readability and reproducibility.

[1] Yu, Xin, et al. "AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections." arXiv preprint arXiv:2505.12455 (2025).

---

confidence: 2
contribution: 2
initial_rating: 4
presentation: 3
questions: In Table 6, several DoRA results (e.g., r=4 with BoolQ=32.35, PIQA=7.13, Winogrande=0.00) are anomalously low and LoRA sometimes degrades as rank increases (e.g., r=4 worse than r=1 on PIQA/HellaSwag), suggesting hyperparameter/setup issues—could you explain these discrepancies?
soundness: 2
strengths: - **Substantive technical contribution with theory.** The paper proposes a concrete improvement over standard LoRA-style adaptation and backs it up with clear derivations/analysis. The core ideas are technically motivated (e.g., aligning updates with full fine-tuning dynamics), and the method’s components are explained rather than presented as ad-hoc tricks.
- **Broad empirical validation across domains.** Experiments cover multiple modalities/datasets (e.g., language and vision) and a range of ranks/settings, suggesting the approach is not narrowly tailored to a single task.
summary: The paper proposes a new method called LoFT for parameter-efficient fine-tuning of large pre-trained models. LoFT extends the Low-Rank Adaptation (LoRA) approach by aligning the internal states of the optimizer (including momentum and second moments) with full fine-tuning, thereby attempting to reduce the accuracy gap typically seen between low-rank and full fine-tuning methods. The authors test their method across multiple language and vision tasks, showing performance improvements compared to previous low-rank adaptation methods, especially at very low ranks. They also discuss trade-offs in terms of memory usage and computational overhead, presenting simpler variants with lower overhead.
weaknesses: - **Gap between theory and the strongest claim.** While the derivations are compelling, there remains a gap between the formal analysis and the paper’s strongest claim(s) (e.g., exact equivalence to full fine-tuning/AdamW under certain limits). A precise theorem with assumptions, or a more cautious phrasing, would strengthen the work.
- **LLM evaluation is too basic.** The large-language-model experiments rely on relatively easy, small benchmarks. For a model like Llama-3-8B, a more representative LLM evaluation suite (e.g., code, math/reasoning, or long-context benchmarks) would be more convincing. Multi-seed runs with statistical reporting would further solidify the results.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What is the quantitative impact of the additional memory requirement on training efficiency and GPU utilization?

2. Can LoFT be extended to other optimizers such as Muon that use different moment estimation mechanisms?
soundness: 3
strengths: 1.	The method directly addresses the optimizer state misalignment problem that has been largely overlooked in prior low-rank adaptation research.
2.	The theoretical analysis is rigorous and provides a clear guarantee that LoFT degenerates to AdamW in the full-rank case.
3.	The six-component design is systematic, and is validated through detailed ablation studies.
4.	LoFT consistently outperforms LoRA and DoRA on both natural language and vision benchmarks, showing broad applicability.
summary: This paper introduces LoFT, a new parameter-efficient fine-tuning method designed to make low-rank adaptation behave like full fine-tuning. Existing LoRA-based approaches mainly focus on gradient approximation but ignore the optimizer state misalignment, particularly in the first and second moments of the AdamW optimizer. LoFT explicitly aligns both gradients and optimizer states with full fine-tuning dynamics through six carefully designed components: alternating updates, gradient scaling, optimizer state calibration, second-moment alignment, projected full update reconstruction, and gradient clipping. Theoretical analysis proves that LoFT reduces exactly to AdamW when the rank equals the full dimension. Experiments on multiple large language models and vision transformers demonstrate that LoFT achieves higher accuracy and faster convergence than LoRA and DoRA while maintaining the same inference cost and number of trainable parameters.
weaknesses: 1.	The additional memory cost, which can reach about twenty-five percent compared to LoRA, is not fully analyzed for its impact on large-scale training.
2.	Experiments are limited to models of eight billion parameters or smaller, leaving scalability to larger models unverified.
3.	The effect of optimizer state projection on stability and convergence speed is discussed conceptually but lacks quantitative analysis.
4.	The paper does not report concrete throughput or training speed measurements compared with LoRA or DoRA.


Decision: Accept (Poster)