GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs

Authors: Maxim Zhelnin, Viktor Moskvoretskii, Egor Shvetsov, Venediktov Egor, Maria Krylova, Zuev Aleksandr, Evgeny Burnaev
Primary area: optimization
Keywords: large language models, parameter efficient fine-tuning, Gaussian noise, quantization

Abstract:
Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and democratized the usage of Large Language Models (LLMs). Recent studies have shown that a small subset of weights significantly impacts performance. Based on this observation, we introduce a novel PEFT method, called Gaussian noise Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only salient columns, while injecting Gaussian noise into non-salient ones. To identify these columns, we developed a generalized sensitivity metric that extends and unifies metrics from previous studies. Experiments with LLaMA models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT methods under the same computational budget. Moreover, GIFT-SW offers practical advantages to recover performance of models subjected to mixed-precision quantization with keeping salient weights in full precision.

Decision: desk reject

Reviews and discussion:

Metareview: