AutoRedTeamer: An Autonomous Red Teaming Agent Against Language Models

Authors: Andy Zhou, Kevin Wu, Yi Zeng, Yu Yang, Shuang Yang, Sanmi Koyejo, James Zou, Bo Li
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: adversarial robustness, large language models, jailbreaking, ai agents

Abstract:
As large language models (LLMs) become increasingly capable, robust and scalable security evaluation is crucial. While current red teaming approaches have made strides in assessing LLM vulnerabilities, they often rely heavily on human input and fail to provide comprehensive coverage of potential risks. This paper introduces AutoRedTeamer, a unified framework for fully automated, end-to-end red teaming against LLMs. AutoRedTeamer is an LLM-based agent architecture comprising five specialized modules and a novel memory-based attack selection mechanism, enabling deliberate exploration of new attack vectors. AutoRedTeamer supports both seed prompt and risk category inputs, demonstrating flexibility across red teaming scenarios. We demonstrate AutoRedTeamer’s superior performance in identifying potential vulnerabilities compared to existing manual and optimization-based approaches, achieving higher attack success rates by 20% on HarmBench against Llama-3.1-70B while reducing computational costs by 46%. Notably, AutoRedTeamer can break jailbreaking defenses and generate test cases with comparable diversity to human-curated benchmarks. AutoRedTeamer establishes the state of the art for automating the entire red teaming pipeline, a critical step towards comprehensive and scalable security evaluations of AI systems.

Decision: reject
Reviewer scores: 5, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730165321422

thread_type:
official_review

<thread_object>
creator: reviewer_kor2

<content>
summary:
The paper introduces an automated framework called AutoRedTeamer for red teaming large language models. Unlike existing methods that rely on human intervention, AutoRedTeamer uses an LLM-based agent architecture with five specialized modules and a memory-based attack selection mechanism. This allows for end-to-end automation of seed prompt generation, attack selection, execution, and evaluation. The results show that AutoRedTeamer achieves a 20% higher attack success rate on the HarmBench dataset while reducing computational costs by 46%, and it is able to break defenses and generate test cases comparable to human benchmarks.

strengths:
1. This paper focuses on an important research topic and provides valuable contributions to the community.

2. The paper is well-written and easy to follow overall.

3. The evaluation results demonstrate the potential of the proposed method for effective and efficient automatic red teaming.

weaknesses:
1. The technical contribution, as currently presented, is not significant.

2. The evaluation lacks a detailed description.

questions:
1. The core of the proposed AutoRedTeamer framework is an agent-based ensemble system that dynamically and adaptively selects and refines attack strategies while generating jailbreaking test cases. However, the current text does not elaborate on the technical details of each module in the framework. For instance, when introducing the seed prompt generation module, the authors state that "Each scenario is comprehensively defined, including a unique identifier, detailed description, expected outcome upon target AI failure, and the specific input for the target." However, it is unclear what specific prompts are used to ensure the correctness of the generated seed prompts. The same lack of detail applies to other modules, such as risk analysis and memory-based attack selection. Without these details, it is difficult to quantify the technical contributions of the proposed framework.

2. When evaluating the efficiency of the proposed method, the authors report the total number of queries AutoRedTeamer needs to successfully generate the test cases. However, AutoRedTeamer itself is an LLM-based agent, where each module is an LLM. This means that the overhead cannot simply be measured by the number of queries made to the target model; the time taken by the agent LLM (Mistral-8x22B-Instruct-v0.1) to generate intermediate results should also be considered. Therefore, I suggest that the authors report this overhead (in terms of exact time) and compare it with baseline methods.

3. Several recent LLM jailbreaking techniques shall also be considered as baselines, such as [1]

[1] Zeng, Yi, et al. "How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms." arXiv preprint arXiv:2401.06373 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730357275554

thread_type:
official_review

<thread_object>
creator: reviewer_uu9j

<content>
summary:
This paper designs an agent to do the automated red-teaming and shows the improved performance over selected baselines.

strengths:
+ The paper is easy to follow
+ The LLM jailbreak topic is trending

weaknesses:
- Marginal novelty
- The improvement is not significant
- Lack of strong baselines
- Need to conduct experiments on more well-aligned models

First of all, I think the idea of using LLM to generate jailbreak prompts is already well-studied since last year, and I do not see significant novelty in this paper compared with tons of similar papers using LLM to mutate/transform/generate new jailbreak prompts. The paper claims that it is the first paper as fully automated, which I cannot agree. Most published black-box jailbreak work just use a set of collected prompts as seeds or just start with some prompts, and then have LLM automatically do the prompt transformation. For example, [1] just starts with some prompts and have LLM automatically do the jailbreak, with is identical to your setting, and I believe there are too many works liks that without human in the loop. Thus, I do not think it is an accurate claim as the first fully automated jailbreak paper.

Second, I think this paper uses agentic approach and combines existing jailbreak strategies while the improvement is not significant, especially considering that there are many black-box attack papers that have near-perfect attack success rates such as [1,2,3].  I think the paper should compare with some latest and stronger attackers since PAIR is published last year and many stronger jailbreak approaches were found after that.

Although I appreciate that the authors conduct experiments on the 70B model, I believe some more aligned models can be attacked with, such as Llama-2, and Claude-3. It would make the experiments more comprehensive with more models in the evaluation.

Lastly, as an attack paper, I think the ethical concern discussion and response disclosure are missing from the manuscript.

[1] GPT-4 Jailbreaks Itself with Near-Perfect Success Using Self-Explanation

[2] WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Response

[3] Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks

questions:
See the comments above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730374001498

thread_type:
official_review

<thread_object>
creator: reviewer_uqx7

<content>
summary:
This paper proposes a framework to automatically generate jailbreaks/harmful prompts, evaluate their effectiveness, and iteratively refine the prompts to improve their performance. This is done via a series of LLMs given specific tasks in the overall pipeline. The method does have good performance, generating jailbreaks in fewer queries compared to other methods.

strengths:
- The method is empirically effective - e.g. in Table 1 the ASR is higher (or at least comparable) to the baselines with fewer queries required. Furthermore, from Fig. 8 the fact that many attacks are composed of multiple simple/cheap attack styles does indicate that the method is creating more complex attacks as it iterates over prompts.

- The attack is also robust against defences: for example, the perturbations in SmoothLLM don't significantly degrade attack performance.

-  The method allows fully end-to-end red-teaming with only an initial domain specified (e.g. “cybersecurity”). There is scope here to enhance AutoRedTeamer with attacks tailored to particular use cases that a user is concerned for.

weaknesses:
- Generally, I found the paper to lack detail and technical clarity and the overall paper flow would need improving. Some useful information is for example in the appendix (e.g. pesudocode), which given the low level of technical description in the main body of text could be given a more visible position. Some concrete examples which can use more technical description: in Section 3.2 Risk Analyser, the prompting strategy is not described; alternatively how the memory store works can use more  detail in the paper given its importance to the results and paper claims. Overall, just based on the content of the paper it would be highly challenging to reproduce the results diminishing the contribution of the work.

- The authors raise RedAgent and ALI-agent as similar work using agents to automatically red-team, and additionally state that their use of a memory store is novel. However, those two papers are similar to this one in proposing agentic frameworks and additionally utilise memory. The authors argue that their work is differentiated by the lack of a need for predefined test scenarios and support for a more diverse range of attacks. However, this may not suffice as a compelling degree of novelty - the range of attacks from the other works can be easily extended. Similarly, attacks such as GCG are not compared to because they have fixed test cases: however, in that case, would GCG combined with the seed prompt generation described in the paper be superior to the proposed attack piepline? I feel the authors don’t fully justify why it is so difficult to generate seed prompts and combine it with other attack pipelines.

- Rainbow Teaming is mentioned as the closest work, yet is not evaluated/compared to. Likewise Wildteaming [1] can be considered similar work and could use comparison or at least discussion.

[1] Jiang, Liwei, et al. "WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models." arXiv preprint arXiv:2406.18510 (2024).

questions:
- From the transition matrix it seems like many of the attack methods are rarely used. How do you ensure that the strategy designer LLM strikes a balance between exploring different attacks and exploiting successful attacks? Is an LLM the best choice for this as opposed to say RL or similar techniques?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717749103

thread_type:
official_review

<thread_object>
creator: reviewer_o3hz

<content>
summary:
This paper proposes an agent-based framework for automatically generating attacks on LLMs, featuring an attack memory bank designed to enhance attack efficiency. The framework demonstrates strong attack performance on HarmBench.

strengths:
1. The design of the risk analyzer within the agent, and the storage format of each entry in the attack memory bank for retrieval purposes are quite interesting.
2. The paper is well-written, with a clear structure. The experiments are extensive.

weaknesses:
1. This paper emphasizes an end-to-end automated red teaming framework; however, the experimental dataset is somewhat limited, lacking exploration of threats in real-world scenarios. It would be more meaningful if the authors could discuss threats in practical contexts, such as embodied scenarios, which would also strengthen their claims about flexibility, adaptability, and scalability.
2. The claims regarding efficiency need further substantiation, as the current experiments do not provide sufficient support.

questions:
1. Why is only HarmBench used in the evaluation? This alone is insufficient to demonstrate the proposed attack framework's flexibility, adaptability, and scalability. Please consider exploring some real-world security threat scenarios, which are more complex and meaningful than the questions in HarmBench.
2. While total queries and total tokens provide insights into efficiency, I am more interested in how long it takes for the end-to-end attack framework to generate an effective attack prompt. Since the end-to-end agent consists of multiple modules, there may be delays in communication between these components, which could negatively impact efficiency. This complexity may be a disadvantage compared to previous methods that do not involve such intricate module interactions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732309359575

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We thank the reviewers for their valuable feedback and comments. We are glad that the reviewers find our paper well-written (Reviewer o3hZ, Uu9j, Kor2), topic important (Reviewer Uu9j, Kor2), framework design interesting (Reviewer 03hZ), and acknowledge the empirical effectiveness of our method (Reviewer Uqx7, Kor2). We especially appreciate the recognition of our framework's potential for effective and efficient automatic red teaming (Reviewer Uqx7). We provide the summary of our revision content below to further strengthen our paper.

1. Our main contribution is providing the first comprehensive, automated red teaming framework. We demonstrate this comprehensiveness in several ways:
   - Flexibility across domains: Unlike prior work that only evaluates on fixed test cases, we show effectiveness on both existing input prompts from benchmarks (HarmBench) and diverse risk categories (AIR taxonomy)
   - Automated seed prompt generation: We demonstrate that our framework can match human-level diversity in test case creation while removing potential human biases
   - Integration of multiple attack vectors: Our framework can incorporate and intelligently combine various attacks, as shown by our successful integration of recent methods like Pliny

2. Following reviewers' suggestions, we have conducted additional experiments to strengthen our claims:
   - Added time efficiency results showing practical runtime despite comprehensive automation
   - Evaluated on more recent models (Claude-3.5-Sonnet, Llama-2-7B) 
   - Included comparisons to 2024 attacks such as the highly effective adaptive attack [1], showing our framework can match or exceed their performance when incorporated into our toolbox
   - Added comparison to other end-to-end pipelines such as rainbow teaming [2], demonstrating significant improvements 

3. We have added more technical details about our modules' implementation and enhanced our discussion of ethical considerations. We will release our code to aid reproducibility and future research.

We believe these revisions address the main concerns raised while reinforcing our paper's key contribution: a flexible, comprehensive framework that advances the state of automated red teaming.

[1] Andriushchenko, M., Croce, F., & Flammarion, N. (2024). Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. ArXiv, abs/2404.02151.
[2] Samvelyan, M., Raparthy, S.C., Lupu, A., Hambro, E., Markosyan, A.H., Bhatt, M., Mao, Y., Jiang, M., Parker-Holder, J., Foerster, J., Rocktaschel, T., & Raileanu, R. (2024). Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts. ArXiv, abs/2402.16822.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The AC and reviewers acknowledge that the authors have addressed some concerns, such as providing additional technical clarifications to enhance the paper. However, significant concerns remain regarding the utility and persuasiveness of the issues discussed. Additionally, both the AC and reviewers question the novelty of the automated red-teaming approach, particularly in light of the substantial body of similar research in the field. The primary contribution of the proposed work—focused on improving attack capability—appears aligned with the objectives of many existing studies, which raises further concerns about its novelty from the scope perspective.