GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs
Authors: name: Maxim Zhelnin; name: Viktor Moskvoretskii; name: Egor Shvetsov; name: Egor Venediktov; name: Mariya Krylova; name: Aleksandr Zuev; name: Evgeny Burnaev
Published: 2024-08-27 14:41:14

Abstract: Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and
democratized the usage of Large Language Models (LLMs). Recent studies have
shown that a small subset of weights significantly impacts performance. Based
on this observation, we introduce a novel PEFT method, called Gaussian noise
Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only
salient columns, while injecting Gaussian noise into non-salient ones. To
identify these columns, we developeda generalized sensitivity metric that
extends and unifies metrics from previous studies. Experiments with LLaMA
models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT
methods under the same computational budget. Moreover, GIFT-SW offers practical
advantages to recover performance of models subjected to mixed-precision
quantization with keeping salient weights in full precision.