Uncovering Model Vulnerabilities With Multi-Turn Red Teaming

Authors: Nathaniel Li, Ziwen Han, Ian Steneker, Willow E. Primack, Riley Goodside, Hugh Zhang, Zifan Wang, Cristina Menghini, Summer Yue
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: language models, ai security, ai safety, robustness, adversarial attacks

Abstract:
Recent large language model (LLM) defenses have greatly improved models' ability to refuse harmful queries, even when adversarially attacked. However, LLM defenses are primarily evaluated against automated adversarial attacks in a single turn of conversation, an insufficient threat model for real-world malicious use. We demonstrate that multi-turn human jailbreaks uncover significant vulnerabilities, exceeding 70% attack success rate (ASR) on HarmBench against defenses that report single-digit ASRs with automated single-turn attacks. Human jailbreaks also reveal vulnerabilities in machine unlearning defenses, successfully recovering dual-use biosecurity knowledge from unlearned models. We compile these results into Multi-Turn Human Jailbreaks (MHJ), a dataset of 2,912 prompts across 537 multi-turn jailbreaks. We publicly release MHJ alongside a compendium of jailbreak tactics developed across dozens of commercial red teaming engagements, supporting research towards stronger LLM defenses.

Decision: reject
Reviewer scores: 3, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730288479553

thread_type:
official_review

<thread_object>
creator: reviewer_cegr

<content>
summary:
This paper looks at red teaming via multi-turn human interaction and releases an accompanying dataset of the jailbreaks. This is a useful study: the vast majority of the research effort and available data is focused on single-turn attacks, with only a few exceptions (e.g. Anthropic's hh-rlhf dataset having harmful multi turn conversations). Overall, the paper is clear and well written, and supplies useful well curated data for future work.

strengths:
- Human driven red teaming data, particularly multi-turn due to its scarcity, is always useful. There are many synthetically generated datasets, but high quality human driven attacks are a valuable resource. In particular, each prompt here had a high degree of manual curation (in comparison to datasets like hackaprompt which contain many low quality samples).

- The analysis across a few defenses is useful, and does highlight that even SOTA defenses on highly studied benchmark harmful questions can be broken in around 15min on a large subset of questions (e.g. often quicker than algorithmic attack runtime).

weaknesses:
- It would have been useful to see results which have human single turn jailbreak attempts in Figure 3. At the moment, the "Human" attacks have two variables changed compared to the others: the attack source (e.g. handcrafted), and additionally have multi-turn capabilities. Hence it makes it challenging to disambiguate if the difference in performance is due to the multi-turn aspect, or if humans given enough time remain better than automated based methods at creating jailbreaks.
- I am unclear as to why results against Cygent defense could not be carried out in the same setup as the original paper: the Llama model is open source, and the defense has a published paper. It would have enabled stronger reproducibility and clearer interpretation of results.
- Releasing the non-successful jailbreak attempts as well can be beneficial as it is still a useful resource, for example as training/fine-tuning data or to carry out further analysis.
- Although different styles of harmfulness were investigated: both "regular" harmbench style questions, but also WMDP-Bio for different attack objectives and domain performance from looking into the supplementary material it seems like just the harmbench data was released. Given the dataset is the core contribution of the paper it would have been useful to include the other domain data.

questions:
Are there statistics on how many red-team members there are/distributions on number of samples provided per red-team member?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730536591685

thread_type:
official_review

<thread_object>
creator: reviewer_v88e

<content>
summary:
This paper exposes the significant safety vulnerabilities of Large Language Models (LLMs) in multi-turn dialogue by conducting human red-teaming. By exploiting a diverse of human tactics, human red-teamers achieve a 70% attack success rate on Harmbench against strong defense baselines. This paper finally compiles these results into the Multi-Turn Human Jailbreaks (MHJ) dataset.

strengths:
+ **Valuable Findings, Including an Open-Source Jailbreak Dataset**: This paper provides a valuable contribution by conducting the first multi-turn human red-teaming experiments, revealing significant safety vulnerabilities that bypass current state-of-the-art defense methods. Given that existing safety alignment techniques primarily target single-turn jailbreaks, these findings effectively motivate the community to develop more robust defenses capable of countering multi-turn attacks.

weaknesses:
- **Lack of Strong Automated Attack Baselines**: Some black-box attacks, such as CodeAttack [1] and PAP [2], are not included in the experiments, despite utilizing similar “Request Framing” tactics as those in this paper. Including these baselines in future comparisons would offer a clearer perspective on the relative effectiveness of this paper's approach.

- **Unclear Evaluation Metric for Model Unlearning**: This paper proposes manual review to assess attack outcomes in model unlearning experiments, yet does not clearly define what constitutes a successful attack. For example, does success mean the model output includes content from the forget set? A more detailed description of the evaluation metric for model unlearning experiments would improve clarity.

1. *CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion.*
2. *How Johnny Can Persuade LLMs to Jailbreak: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs.*

questions:
- **Counter-Intuitive Experimental Results**: The effectiveness of the “Direct Request” tactic used by human red-teamers, as shown in Figure 8, is unexpected. In this paper, “Direct Request” refers to directly asking the LLM to produce harmful content, which should logically not be highly effective against safety defenses. An analysis of how various tactics influence the attack success rate would be beneficial, as it could offer insights into the specific failure modes of these models.

- **Lack of In-Depth Analysis on the High Effectiveness of Human Multi-Turn Jailbreaks**: Many of the tactics employed by human red-teamers, such as “Injection,” “Obfuscation,” “Output Format,” and “Request Framing,” are also used by automated attacks. Are there particular factors that enable human-led jailbreaks to outperform automated attacks using similar tactics? Identifying these factors could provide valuable insights into the unique strengths of human-driven attacks.

I would consider raising my score if the authors address these concerns and questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684431396

thread_type:
official_review

<thread_object>
creator: reviewer_qgev

<content>
summary:
This paper focuses on the safety vulnerabilities of LLMs in multi-turn conversations. The authors designed a comprehensive human red teaming pipeline, including "Attempt Jailbreak" and "Validate Jailbreak," and found that LLMs exhibit more safety risks in multi-turn dialogue conversations compared to single-turn automated attacks.

strengths:
1. Most previous papers have focused on LLM jailbreaking in single-turn conversations, while this work emphasizes multi-turn conversations, presenting a new scenario.

2. The red-teaming results show that multi-turn human jailbreaks outperform current automated attacks, which exposes more safety vulnerabilities in LLMs.

3. The authors have also released their Multi-turn Human Jailbreaks dataset, which can support further research on multi-turn jailbreaks.

weaknesses:
1. The human red-teaming was conducted only on Llama; further evaluations on other LLMs would more comprehensively illustrate the safety vulnerabilities in multi-turn conversations. Additionally, the human red-teaming data collected on Llama could potentially be used to examine safety issues in other LLMs under multi-turn scenarios.

2. A more detailed analysis of the effectiveness of different tactics could provide deeper insights into multi-turn jailbreaks.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720579135

thread_type:
official_review

<thread_object>
creator: reviewer_edxj

<content>
summary:
The paper highlights the limitations of current defenses in large language models (LLMs) against multi-turn adversarial attacks. Unlike single-turn automated attacks, multi-turn human-driven jailbreaks show a much higher success rate, bypassing existing defenses by strategically interacting with models over multiple conversation turns. The authors conducted tests with human red teamers who revealed that many LLMs, despite advanced safeguards, remain vulnerable in realistic settings. The study also introduces the "Multi-Turn Human Jailbreaks" (MHJ) dataset, containing thousands of successful jailbreak examples to support the development of more robust defenses that address multi-turn interaction vulnerabilities.

strengths:
1. The main strength of the paper is evaluating the LLM jailbreak robustness from a different perspective and in a like-life setting. A realistic threat model can always give a better idea about the robustness.

2. I appreciate the authors for publishing the dataset of 2912 prompts. I believe this can help the researchers and community.

weaknesses:
1. Only the *Llama3-8b-Instruct* model was used for evaluation. Other models, especially the stronger ones should have been included. 

2. The chosen baselines and the multi-turn-human-jailbreak approach are different from each other in many ways, such as timing, budget, etc. Seems like the MHJ attack is a soft-constrained version of attacks. 

3. Authors admit that *"the skill and experience of individual red teamers may vary"*. In that case, how can researchers compare their defense against human-based attacks in the common ground? What would be the baseline of a red-teamer's skill and experience? This is one of the reasons for using automated attacks for evaluation to this date.  

4. Other automated attacks, such as [1, 2] could be included in the evaluation. 

5. There was no discussion on plausible defenses for such multi-turn-human-based attack.


References: 


[1] Fast Adversarial Attacks on Language Models In One GPU Minute

[2] Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack

questions:
1. Why the evaluation was done only on one model? Why stronger models like ChatGPT was not included? 

2. How were other automated attacks implemented for the WMDP-bio questions?

3. Is it possible to mention the monetary cost of this human-based attack?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731718589447

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Author Response

comment:
Thank you to the reviewers for their detailed feedback! We’ve responded individually to every point of concern and question, in addition to providing the summary below.

We observe some disagreement between reviewers, with reviewer 4 giving the highest score after studying the supplemental dataset. This rebuttal aims to clarify the soundness and value of our contribution.

**Summary of reviewer agreed strengths and key contributions:**
- **Valuable results**: The paper shows multi-turn human red teaming easily breaks SOTA defenses (robust against single turn attacks). The strong performance of this realistic threat model highlights a need to develop better defenses.
- **High quality human dataset**: Our released dataset of human jailbreaks is an important contribution because of the high quality of manual curation. Reviewers agree this dataset will further support researchers in the field.

**We highlight two common concerns from reviewers 1, 2, and 3.** We note these concerns do not harm the findings of the paper, but we agree more experiments would enhance completeness. However, human experiments are incredibly costly and time-consuming, hence we were forced to make some design choices to adhere to work the community has already found valuable.

- **We should attack “stronger models”**:  We should not confuse capability with adversarial robustness. Prior work such as HarmBench (Appendix Table 6) [1] has shown that other models suggested, such as ChatGPT, are substantially less robust. We focus on the more difficult task of attacking defenses. The defenses we experiment with are already some of the most robust defenses against single-turn attacks at the time of experimentation. We use defenses on top of Llama3-8b to be consistent with prior work, it does not mean this model is less robust than larger models. For example, CYGNET has a 0% ASR on all other automated attacks as reported by the original authors, whereas we achieve a 70% ASR. Since we already break strong defenses, we do not see a need to add more less robust models to attack.
- **Implementing additional attacks**: The addition of more recent attacks would not substantially change the message of the paper: defenses which claim to be robust to single-turn attacks are not robust to multi-turn human attacks yet, as both a factor of the multi-turn objective and the unconstrained creativity of human red teamers. We chose to use a standard framework that has attacks implemented, in this case HarmBench [1]. There are many attacks which have come out recently not part of this standardized library, we simply selected 6 in HarmBench that were straightforward or did well on defenses.

**References**

[1] Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., ... & Hendrycks, D. (2024). Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249.
</content>
</thread_object>

</thread>


Metareview:
metareview:
There have been several multi-turn jailbreaking attacks proposed in the past, such as Crescendo, mentioned by the authors. This reduces the novelty and impact of the proposed work. Moreover, the authors have not adequately compared their method with critical prior works like Crescendo to highlight key differences. While the authors attribute the lack of comparison to the unavailability of open-sourced resources, the code for Crescendo is publicly accessible. Additionally, there are other multi-turn jailbreaking attacks that should also be included for a fair comparison. The AC recommends that the authors conduct a more comprehensive evaluation, not only focusing on ASR but also considering factors like prompt diversity and the cost introduced by the proposed attack method.