PaSEFT: Gradient-based Partial Parameter Selection in Efficient Fine-Tuning for Large Language Models

Authors: Shuoran Jiang, Wentao Zhu, Qingcai Chen, Xiangping Wu, Junjie Chen, Yang Xiang, Youcheng Pan, Kai Chen
Primary area: optimization
Keywords: partial parameter, parameter efficient fine-tuning, large language models, multilingual capability

Abstract:
As the severe unfairness of language distribution in the pre-training data, large language models (LLMs) require further fine-tuning for multilingual capability. However, existing parameter efficient fine-tuning (PEFT) methods are unable to enhance multilingual translation capabilities in the incremental learning for LLMs. This paper proposes a language specified gradient attribution (LanGA) method to select the trainable subset from the full-parameter LLMs, which is used for the multilingual translation PEFT. Specifically, LanGA first select a language feature specific subset from full parameters for training, and the selected subset is much smaller than full parameters. Next, LanGA reconstruct the AdamW as a sparse optimizer for parameter subset training. Our experiments demonstrate that LanGA outperforms different low-memory fine-tuning methods for multilingual fine-tuning. At the same time, LanGA costs less memory usage that existing PEFT methods. Importantly, LanGA keeps the translation performance on resource-rich languages in highest measure.

Decision: reject

Reviews and discussion:

Metareview: