Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Merging
Authors: name: Hua Farn; name: Hsuan Su; name: Shachi H Kumar; name: Saurav Sahay; name: Shang-Tse Chen; name: Hung-yi Lee
Published: 2024-12-27 08:03:22

Abstract: Fine-tuning large language models (LLMs) for downstream tasks is a widely
adopted approach, but it often leads to safety degradation in safety-aligned
LLMs. Currently, many solutions address this issue by incorporating additional
safety data, which can be impractical in many cases. In this paper, we address
the question: How can we improve downstream task performance while preserving
safety in LLMs without relying on additional safety data? We propose a simple
and effective method that maintains the inherent safety of LLMs while enhancing
their downstream task performance: merging the weights of pre- and
post-fine-tuned safety-aligned models. Experimental results across various
downstream tasks, models, and merging methods demonstrate that this approach
effectively mitigates safety degradation while improving downstream task
performance, offering a practical solution for adapting safety-aligned LLMs.