Jailbreaking as a Reward Misspecification Problem
Authors: Zhihui Xie, Jiahui Gao, Lei Li, Zhenguo Li, Qi Liu, Lingpeng Kong


Reviewer paper summaries: This paper proposes a novel perspective on the vulnerability of LLMs. First, the paper introduces ReGap, a metric to measure the extent of reward misspecification. Based on ReCap, the paper presents ReMiss, a system for automated red teaming that generates adversarial prompts against various target aligned LLMs. And the proposed method achieves SOTA on the AdvBench benchmark.
Summary of strengths: This work has a high degree of completion, and the description of the methods and experimental settings is very detailed experimental setup. And the experimental results demonstrate the advantages of this method.
Summary of weaknesses: The motivation for introducing ReCap seems unclear. In the field of automated red teaming , it is also a common practice to directly use another LLM to craft harmful prompts based on the output toxicity of target LLMs [1][2]. But this paper does not theoretically explain why the more complex ReCap is used. Furthermore, it will be even better if there are more baselines, attacked models, and datasets in the comparative experiment.

[1] Perez E, Huang S, Song F, et al. Red teaming language models with language models[J]. arXiv preprint arXiv:2202.03286, 2022.
[2] Hong Z W, Shenfeld I, Wang T H, et al. Curiosity-driven red-teaming for large language models[J]. arXiv preprint arXiv:2402.19464, 2024.
Comments, suggestions and typos: 1. Explain theoretically the advantages of ReCap compared to directly using LLMs to craft harmful prompts.
2. More experiments will be beneficial, such as more baselines and attacked LLMs.
Overall assessment scores: 3
Metareview score: 3
Metareview: This paper investigates the vulnerabilities of LLMs to adversarial attacks, focusing on the concept of reward misspecification during safety alignment. The authors introduce a new metric, ReGap, to measure this misspecification and propose a novel attack method, ReMiss, which generates adversarial examples to test LLMs. The technical novelties are supported by experiments with four popular open-sourced LLMs on AdvBench dataset. 

Many of the missing pieces are complemented by the additional experiments the authors presented in the rebuttal, so it is necessary to include them in the next version of the paper, and address all  the review feedbacks as well as refinement promises.
Summary of reasons to publish: The introduction of ReGap as a metric for reward misspecification and ReMiss as an attack method provides new insights and tools for understanding and improving LLM safety. The findings have strong implications for improving the robustness and reliability of LLMs, which is crucial for their safe deployment in real-world applications, and the authors conducted relatively comprehensive experiments (including the ones presented in the rebuttal) to support their technical novelties.