LoRA Learns Less and Forgets Less
Authors: name: Dan Biderman; name: Jose Gonzalez Ortiz; name: Jacob Portes; name: Mansheej Paul; name: Philip Greengard; name: Connor Jennings; name: Daniel King; name: Sam Havens; name: Vitaliy Chiley; name: Jonathan Frankle; name: Cody Blakeney; name: John P. Cunningham
Published: 2024-05-15 19:27:45

Abstract: Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning
method for large language models. LoRA saves memory by training only low rank
perturbations to selected weight matrices. In this work, we compare the
performance of LoRA and full finetuning on two target domains, programming and
mathematics. We consider both the instruction finetuning (approx100K
prompt-response pairs) and continued pretraining (approx10B unstructured
tokens) data regimes. Our results show that, in most settings, LoRA
substantially underperforms full finetuning. Nevertheless, LoRA exhibits a
desirable form of regularization: it better maintains the base model's
performance on tasks outside the target domain. We show that LoRA provides
stronger regularization compared to common techniques such as weight decay and
dropout; it also helps maintain more diverse generations. We show that full
finetuning learns perturbations with a rank that is 10-100X greater than
typical LoRA configurations, possibly explaining some of the reported gaps. We
conclude by proposing best practices for finetuning with LoRA.