Course-Correction: Safety Alignment Using Synthetic Preferences
Authors: name: Rongwu Xu; name: Yishuo Cai; name: Zhenhong Zhou; name: Renjie Gu; name: Haiqin Weng; name: Yan Liu; name: Tianwei Zhang; name: Wei Xu; name: Han Qiu
Published: 2024-07-23 16:54:28

Abstract: The risk of harmful content generated by large language models (LLMs) becomes
a critical concern. This paper presents a systematic study on assessing and
improving LLMs' capability to perform the task of course-correction,
\ie, the model can steer away from generating harmful content autonomously. To
start with, we introduce the C^2-Eval benchmark for quantitative
assessment and analyze 10 popular LLMs, revealing varying proficiency of
current safety-tuned LLMs in course-correction. To improve, we propose
fine-tuning LLMs with preference learning, emphasizing the preference for
timely course-correction. Using an automated pipeline, we create
C^2-Syn, a synthetic dataset with 750K pairwise preferences, to
teach models the concept of timely course-correction through data-driven
preference learning. Experiments on 2 LLMs, Llama2-Chat 7B and
Qwen2 7B, show that our method effectively enhances course-correction
skills without affecting general performance. Additionally, it effectively
improves LLMs' safety, particularly in resisting jailbreak attacks.