Title: AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration
Venue: NeurIPS 2025
Accepted: yes
Keywords: adversarial robustness

large language models

jailbreaking

ai agents
Abstract: As large language models (LLMs) become increasingly capable, security and safety evaluation are crucial. While current red teaming approaches have made strides in assessing LLM vulnerabilities, they often rely heavily on human input and lack comprehensive coverage of emerging attack vectors. This paper introduces AutoRedTeamer, a novel framework for fully automated, end-to-end red teaming against LLMs. AutoRedTeamer combines a multi-agent architecture with a memory-guided attack selection mechanism to enable continuous discovery and integration of new attack vectors. The dual-agent framework consists of a red teaming agent that can operate from high-level risk categories alone to generate and execute test cases, and a strategy proposer agent that autonomously discovers and implements new attacks by analyzing recent research. This modular design allows AutoRedTeamer to adapt to emerging threats while maintaining strong performance on existing attack vectors. We demonstrate AutoRedTeamer’s effectiveness across diverse evaluation settings, achieving 20% higher attack success rates on HarmBench against Llama-3.1-70B while reducing computational costs by 46% compared to existing approaches. AutoRedTeamer also matches the diversity of human-curated benchmarks in generating test cases, providing a comprehensive, scalable, and continuously evolving framework for evaluating the security of AI systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a black-box, automated red teaming framework that integrates a multi-agent architecture with a memory-guided attack selection mechanism to support continuous discovery and adaptation of new attack vectors. The framework features two key agents: a red teaming agent capable of generating and executing test cases based solely on high-level risk categories, and a strategy proposer agent that autonomously identifies and incorporates novel attack strategies by analyzing recent research developments.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\nThis paper proposes a simple and effective red-teaming framework that operates in a black-box setting, making it applicable to both open-source and closed-source large language models.\n\n**Weakness**\n\n1. Unclear attacker proposer Design in Sec. 3.2. The design of the Attacker Proposer is not clearly described. Specifically, there is insufficient detail on how papers are transformed into executable attack code. Since this component is central to the attack capabilities of the proposed framework, more information is needed regarding how the implementation ensures quality and effectiveness. For example, what specific criteria are used to evaluate the novelty of an attack?\n2. It is unclear what types of attacks the agent actually implements. The main body seems to indicate that the agent can generate executable plug-and-play attack code (e.g., GCG [1]) or scripts that can be run automatically. However, based on the examples in Section E, it appears that the system primarily stores natural language descriptions of harmful prompt types, rather than actual attack code. These are closer to harmful prompt templates than to specific novel attacks.\n3. The authors state that “the agent also generates additional attack proposals by identifying core principles from the analyzed papers and combining them in new ways.” However, there is no detailed explanation of how such combinations are implemented. \n4. The description in Sec. 3.5 “by tracking each test case’s effectiveness” is unclear. It is unclear whether this effectiveness is evaluated by the attacker agent itself, or by observing the response from the target model. More clarity is needed on how this evaluation is carried out.\n5. The cost and purpose of validation set filtering. The framework includes a validation phase before actual red-teaming, which raises questions about cost and purpose. Since red-teaming aims to uncover vulnerabilities in a target model, using a validation step to pre-filter attack cases may add unnecessary computational cost and may also bias the resulting test cases. This pre-filtering may discard attacks that might be effective on certain target models.\n6. The novelty is limited. The proposed “Attack Memory” concept resembles prior work on skill libraries (e.g., [2], [3]), and the multi-agent red-teaming approach is similar to frameworks explored in [3], [4]. As such, the novelty of the contributions appears limited.\n7. The evaluation is restricted to only 4 LLMs. A broader evaluation over more models would strengthen the empirical support for the framework’s generality and effectiveness.\n8. A small point is that the paper aims to propose an automatic red-teaming framework; it is unclear why it still requires user-provided seed attacks. Ideally, the framework should also automate this seed generation step or incorporate it into the overall system design.\n\n[1] Zou, Andy, et al. \"Universal and transferable adversarial attacks on aligned language models.\" arXiv preprint arXiv:2307.15043 (2023).\n\n[2] Fan, Linxi, et al. \"Minedojo: Building open-ended embodied agents with internet-scale knowledge.\" Advances in Neural Information Processing Systems 35 (2022): 18343-18362.\n\n[3] Liu, Xiaogeng, et al. \"Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms.\" arXiv preprint arXiv:2410.05295 (2024).\n\n[4] BENCHMARK, JUDGE. \"JAILJUDGE: AComprehensive JAILBREAK JUDGE BENCHMARK WITH MULTI-AGENT ENHANCED EXPLANATION EVALUATION FRAMEWORK.\"\n\nQUESTIONS:\nSee above in weakness.\n\nLIMITATIONS:\nSee above in weakness.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLarge Language Models (LLMs) are continually evolving, and each new iteration may introduce vulnerabilities to both existing and emerging threats. Traditionally, identifying these vulnerabilities has relied on manual effort or human-in-the-loop approaches. This paper presents a novel autonomous red-teaming multi-agent agent framework that can be continuously updated with new agents as threats evolve. The framework enables continuous testing to proactively uncover vulnerabilities in LLMs. Additionally, it employs a memory-guided strategy, leveraging prior threat signatures and their ordering to intelligently launch targeted vulnerability detection agents.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper presents a major advance over the state-of-the-art by enabling autonomous and continuous vulnerability detection in LLMs without requiring a human-in-the-loop. \n\nA key innovation is the memory-guided strategy for ordering and launching vulnerability detectors, which outperforms random and heuristic baselines. This is demonstrated clearly using robust metrics such as attack success rate (ASR), number of queries per successful attack, and execution time.\n\nThe use of Harmbench—a well-recognized benchmark of adversarial prompts—is a plus. Dedicated agents are used  for each vulnerability class. \n\nThe agent pipeline (identify → assess likelihood → analyze risk → launch attack →evaluate attack success) is intuitive, well-structured and scalable as new attacks emerge. \n\nThe framework is validated on high-profile LLMs including LLAMA3.0-70B, ChatGPT-4o, Mixtral-8x7B, and Claude 3.5 Sonnet. Notably, it achieves orders-of-magnitude faster query times compared to prior methods, while maintaining comparable or superior success rates. \n\nThe appendix provides extensive ablation studies, illustrating the impact of the proposed memory-guided discovery approach across a broad set of harm categories (about a dozen). \n\nOverall, the paper is comprehensive, well-executed, and offers meaningful contributions and novelty.\n\nWeaknesses: There are few apparent weaknesses. However, due to a limited background in ML theory, there may be deeper theoretical aspects that are not fully assessed here. While the current limitations are mentioned in the conclusion and appendix, they should be clearly presented in a standalone “Limitations” subsection to improve clarity and transparency.\n\nQUESTIONS:\n1. Are there theoretical limitations or deeper ML questions that may not have been fully addressed due to a stronger focus on empirical performance?\n\n2. Include a dedicated “Limitations” section instead of embedding them in the conclusion and appendix, to improve clarity and completeness.\n\n3. Can you provide a playbook for integrating new agents into this red teamer framework by those outside of the authors/researchers. This will improve the usefulness of the framework and its impact.\n\nLIMITATIONS:\nYes.  Include a dedicated “Limitations” section instead of embedding them in the conclusion and appendix, to improve clarity and completeness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents AutoRedTeamer, an autonomous framework designed to evaluate the robustness of LLMs against jailbreak attacks. The system continuously learns new attack strategies by analyzing research papers, implementing and evaluating their effectiveness, and maintaining a memory system that tracks the success rates of various attack techniques. The framework comprises two main components: a strategy proposer that builds an evolving attack toolbox from academic literature, and a red teaming agent that uses this toolbox to evaluate the safety of target LLMs. Experiments on the HarmBench dataset, conducted across four different LLMs and compared against four optimization-based and three static template-based baseline attacks, demonstrate that AutoRedTeamer outperforms existing approaches. An ablation study further highlights the importance of the memory component to the overall system's effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nI enjoyed reading this paper and believe that it presents a promising autonomous end-to-end system for evaluating the robustness of LLMs against jailbreak attacks. While the paper does not introduce novel attack techniques of its own, it makes a valuable contribution by integrating a wide range of known attacks and evaluating weaknesses of LLMs to specific attacks. The evaluation shows the effectiveness of the proposed techniques and I also appreciate the inclusion of the ablation studies that shed light on the importance of the proposed memory system.\n\nI would have liked to see more analysis and discussion of the evaluation results. For instance, AutoRedTeamer performs particularly well on Claude-3.5-Sonnet, are there specific attack strategies that drive this success? Understanding these patterns could be instrumental in developing defenses. I encourage the authors to add further discussion on this point to the paper.\n\nQUESTIONS:\n- Are the successful attacks dominated by a particular subset of techniques discovered through the literature review?\n- If so, are these dominant techniques consistent across different LLMs or different categories of harmful content (as shown on the x-axis in Figure 3)?\n- Can you also share any insights into why AutoRedTeamer's is particularly successful on Claude-3.5-Sonnet?\n\nLIMITATIONS:\nLimitations are adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework based on Large Language Model (LLM) agents for automated red teaming, aiming to assess the security robustness of target LLMs. The framework consists of two main components: (1) Lifelong Attack Integration, which continuously collects and integrates the latest jailbreak strategies from public sources and synthesizes attacks using an attack designer guided by an evaluation mechanism; and (2) Red Teaming Evaluation, which includes a risk analyzer to receive user input, a seed prompt generator, a strategy planner, a memory system to track historical attack attempts, and an automatic evaluation module that scores model responses based on risk and contextual relevance.\n\nThe authors evaluate their framework across four LLMs, benchmarking it against existing approaches using attack success rate (ASR) and the number of queries required. Results show higher success rates and improved query efficiency. A comprehensive ablation study further validates the contribution of each component in enhancing red teaming effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper is well-written and clearly motivated. It covers and compares with the most relevant studies, and introduces a novel LLM-agent-based framework for red teaming. The evaluation is comprehensive, including both quantitative comparisons and ablation studies that support the effectiveness of the proposed approach.\n\n\nWeaknesses:\nWhile the paper presents a promising framework, the clarity of the methodology could be improved to enhance overall understanding and reproducibility.\n\n- Figure 1 could benefit from refinement. The naming of agents may cause confusion, as it is not clearly reflected in the figure. Additionally, inconsistencies in font size, visual balance, and component labeling make it difficult to grasp the core elements of the proposed framework at a glance.\n\n- Methodological details are lacking in several key areas:\n\n- - Line 143: It is unclear how the agent assigns a composite score—more explanation or formalization is needed.\n\n- - Line 152: The process by which the agent proposes new attack vectors is not described—what constitutes \"new,\" and how is novelty achieved or evaluated?\n\n- - Line 164: The mechanism for maintaining semantic coherence is underexplained.\n\n- - Line 225: The automatic evaluation procedure lacks sufficient detail. Specifically, how relevance is computed based on model outputs and risk scores.\n\nGreater clarity on these aspects would significantly improve the paper's reproducibility and technical transparency.\n\nQUESTIONS:\nEvlaution:\n\n* What is the rationale behind selecting Mixtral and Claude as the primary LLMs in the proposed framework? It is unclear why Claude is only used for attack implementation, while Mixtral appears more centrally involved. A discussion of this design choice, or empirical comparison across different LLMs, would help clarify how various models perform as threat models and their suitability for red teaming.\n\n* Could the authors elaborate on the performance of the attack implementation module? Specifically, how effective is it in generating successful and novel attack prompts, and how does it compare to baselines or other components within the framework?\n\n* It seems like a promising idea to incorporate user input into the proposed framework, particularly for guiding risk assessment or customizing attack objectives. I am curious about the impact of this component: how much does user input contribute to the effectiveness of the red teaming evaluation? Additionally, what would be the expected performance or limitation if the evaluation relied solely on the \"attack library\" generated by the Attack Strategy Proposer Agent, without incorporating user-driven customization or guidance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}