Title: Balanced Low-Rank Adaptation: Removing Invariance for Fast and Stable Fine-Tuning
Primary area: optimization
Keywords: Fine-tuning, LoRA, Transformers, balanced manifold, invariance

Abstract: Low-Rank Adaptation (LoRA) is the most widely used method for fine-tuning large language models.
LoRA is overparameterized: multiple pairs of low-rank factors correspond to the same adapted weight matrix.
We observe both theoretically and numerically that these pairs can have significantly different condition numbers: converging to different minimizers of the loss affects the convergence rate of LoRA.
Building on this remark, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold where the conditioning of the loss is improved while keeping the same adapted matrix. 
This projection step is computationally inexpensive and integrates seamlessly with existing fine-tuning pipelines. 
Empirically, BaLoRA converges faster than standard LoRA and exhibits greater robustness to hyperparameter choices across a range of fine-tuning tasks.

TL;DR: BaLoRA accelerates convergence and improves robustness to hyperparameter tuning by projecting LoRA iterates onto a balanced manifold.