Title: LoLoRA: Locally Fine-Tuned Low Rank Adapters
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: efficient fine-tuning, language models, localized learning

Abstract: Low-rank updates of weights (LoRA) have proven to be a memory-efficient approach to fine-tuning large models. If we keep matrix $A$ fixed during training, this further reduces memory consumption by avoiding storage of LoRA-related input activations, but this usually leads to a drop in final performance. In this work, we mitigate this trade-off and propose to perform local updates of adapter $A$ in a forward pass based on the input of the corresponding layer. These updates allow adapting to shifts in the input distribution directly in the forward pass and do not require saving input activations for the backward pass, thereby keeping performance comparable to standard LoRA while noticeably reducing the memory required for fine-tuning.