Jailbreaking as a Reward Misspecification Problem
Authors: name: Zhihui Xie; name: Jiahui Gao; name: Lei Li; name: Zhenguo Li; name: Qi Liu; name: Lingpeng Kong
Published: 2024-06-20 15:12:27

Abstract: The widespread adoption of large language models (LLMs) has raised concerns
about their safety and reliability, particularly regarding their vulnerability
to adversarial attacks. In this paper, we propose a novel perspective that
attributes this vulnerability to reward misspecification during the alignment
process. We introduce a metric ReGap to quantify the extent of reward
misspecification and demonstrate its effectiveness and robustness in detecting
harmful backdoor prompts. Building upon these insights, we present ReMiss, a
system for automated red teaming that generates adversarial prompts against
various target aligned LLMs. ReMiss achieves state-of-the-art attack success
rates on the AdvBench benchmark while preserving the human readability of the
generated prompts. Detailed analysis highlights the unique advantages brought
by the proposed reward misspecification objective compared to previous methods.