Many-Turn Jailbreaking
Authors: Xianjun Yang, Liqiang Xiao, Shiyang Li, Faisal Ladhak, Hyokun Yun, Linda Ruth Petzold, Yi Xu, William Yang Wang


Reviewer paper summaries: This paper explores the Many-Turn Jailbreaking of LLMs and constructs a dataset named Multi-Turn Jailbreak Benchmark (MTJ-Bench), revealing the jailbreak risks of LLMs in multi-turn conversations.
Summary of strengths: 1. The exploration of many-turn jailbreaking complements previous work on single-turn jailbreaks of LLMs.

2. Much effort was spent conducting experiments on a large number of open-source LLMs.
Summary of weaknesses: 1. Minimal findings and contributions. I believe the findings of the paper and the constructed dataset offer scant contributions. It is quite obvious that once a single-turn jailbreak is successful, subsequent many-turn jailbreaks become more manageable. Moreover, MTJ-Bench is built upon existing datasets. The authors' contribution should, in my view, focus on how to better design multi-turn questions to reduce the likelihood of LLMs detecting harmful intent.

2. Are the designed questions universally applicable across different jailbreak methods? I think different jailbreak methods should correspond to different types of questions. The paper lacks this analysis.

3. To what extent can existing defense methods protect against many-turn jailbreaks? If single-turn jailbreaks can be defended against, then many-turn jailbreaks, due to their larger and more evident targets, should theoretically be easier to defend against. This should be discussed and analyzed.
Comments, suggestions and typos: 1. The font size in Tables 2 and 3 is too large, leading to significant whitespace in between.

2. The equations in section 3.3 are not numbered, and the formula for ASR_Gain extends beyond the formatting boundaries.
Overall assessment scores: 2