Title: LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Large Language Models, Low-Rank Representation, Efficient Fine-tuning

Abstract: Fine-tuning large language models (LLMs) is crucial for improving their performance on downstream tasks, but full-parameter fine-tuning (Full-FT) is computationally expensive and memory-intensive. Parameter-efficient fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), address this by optimizing only a small subset of parameters. However, LoRA's performance often lags behind Full-FT due to limitations of the low-rank gradient.
In this work, we reveal an asymmetric, collapsible structure in LoRA’s update: the low-rank modification to $W$ can be reformulated as a single-layer linear regression, implying one of the LoRA factors can be frozen without sacrificing expressivity. Leveraging this, we introduce LoRA-FA, which freezes the projection-down matrix $A$ and trains only the projection-up matrix $B$. We further close the gap to Full-FT by deriving closed-form gradient corrections that minimize the discrepancy between the induced low-rank gradient and the full gradient.
Through extensive experiments on diverse benchmarks, including GLUE, GSM8K, and HumanEval, we demonstrate that LoRA-FA consistently achieves comparable performance with existing PEFT methods and Full-FT. Experiments on system efficiency shows that LoRA-FA reduces significant activation memory consumption and computation workload in fine-tuning.