GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs
Authors: name: Maxim Zhelnin, name: Viktor Moskvoretskii, name: Egor Shvetsov, name: Egor Venediktov, name: Mariya Krylova, name: Aleksandr Zuev, name: Evgeny Burnaev

Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and
democratized the usage of Large Language Models (LLMs). Recent studies have
shown that a small subset of weights significantly impacts performance. Based
on this observation, we introduce a novel PEFT method, called Gaussian noise
Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only
salient columns, while injecting Gaussian noise into non-salient ones. To
identify these columns, we developeda generalized sensitivity metric that
extends and unifies metrics from previous studies. Experiments with LLaMA
models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT
methods under the same computational budget. Moreover, GIFT-SW offers practical
advantages to recover performance of models subjected to mixed-precision
quantization with keeping salient weights in full precision.