MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
Authors: name: Ting Jiang; name: Shaohan Huang; name: Shengyue Luo; name: Zihan Zhang; name: Haizhen Huang; name: Furu Wei; name: Weiwei Deng; name: Feng Sun; name: Qi Zhang; name: Deqing Wang; name: Fuzhen Zhuang
Published: 2024-05-20 15:48:32

Abstract: Low-rank adaptation is a popular parameter-efficient fine-tuning method for
large language models. In this paper, we analyze the impact of low-rank
updating, as implemented in LoRA. Our findings suggest that the low-rank
updating mechanism may limit the ability of LLMs to effectively learn and
memorize new knowledge. Inspired by this observation, we propose a new method
called MoRA, which employs a square matrix to achieve high-rank updating while
maintaining the same number of trainable parameters. To achieve it, we
introduce the corresponding non-parameter operators to reduce the input
dimension and increase the output dimension for the square matrix. Furthermore,
these operators ensure that the weight can be merged back into LLMs, which
makes our method can be deployed like LoRA. We perform a comprehensive
evaluation of our method across five tasks: instruction tuning, mathematical
reasoning, continual pretraining, memory and pretraining. Our method
outperforms LoRA on memory-intensive tasks and achieves comparable performance
on other tasks.