Title: AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement
Venue: NeurIPS 2025
Accepted: yes
Keywords: AI Safety

Multi-Agent Systems

LLMs

Large Language Models

Jailbreaking

Agent Scaffolds
Abstract: Scaffolding Large Language Models (LLMs) into multi-agent systems often improves performance on complex tasks, but the safety impact of such scaffolds has not been thoroughly explored. We introduce AgentBreeder, a framework for multi-objective self-improving evolutionary search over scaffolds. We evaluate discovered scaffolds on widely recognized reasoning, mathematics, and safety benchmarks and compare them with popular baselines. In "blue" mode, we see a 79.4% average uplift in safety benchmark performance while maintaining or improving capability scores. In "red" mode, we find adversarially weak scaffolds emerging concurrently with capability optimization. Our work demonstrates the risks of multi-agent scaffolding and provides a framework for mitigating them. Code is available at \url{https://github.com/jrosseruk/AgentBreeder}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAgentBreeder aims to address the growing need to autonomously design and optimise multi-agent scaffolds (i.e. LLM-based agent systems). Moreover, AgentBreeder uses multi-objective optimisation to optimise both accuracy and safety; the latter being an underrepresented in the optimisation process of current automated design systems for multi-agent scaffolds.\n\nAgentBreeder can be used employed in three different modes:\n1. BlueAgentBreeder - optimises for both capability and safety\n1. RedAgentBreeder - optimises for capability and minimising safety\n1. CapableAgentBreeder - optimises just for capability\n\nThe evolutionary process is controlled by a Meta Agent using Claude for is coding capabilities. The Meta Agent is responsible for creating new Python code scaffolds through mutation or crossover of elite scaffolds from the previous generation. Diversity is imposed by clustering candidates and then selecting pareto-front optimal solutions (capability vs safety) from each cluster. Experiments are performed on one of three different datasets: DROP, MMLU, GPQA, along with the safety benchmark SaladData and TruthfulQA to reduce reward hacking.\n\nResults show a significant uplift in safety (79.4% on average) with BlueAgentBreeder. Results with RedAgentBreeder highlight the potential limits of ignoring safety, i.e. it is possible to have similarly capable model\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Incorporating safely into automated scaffold discovery through multi-objective optimisation and quality-diversity\n* The paper addresses a significant need to ensure safety in scaffold design. This method is an elegant way to balance capability and safety at the core of the optimisation process.\n* The three different modes provide interesting insights into automated scaffold design. BlueAgentBreeder clearly shows that safety can be optimised alongside capability,  RedAgentBreeder demonstrates the worst-case scenarios where capable scaffolds can also be highly vulnerable , and CapableAgentBreeder provides a benchmark against which to compare\n* The code is open source and the experiments use recognised benchmarks which is great for reproducibility and comparison.\n\nWeaknesses\n* One of the biggest contributions is the ability to include safety in the optimisation process. As such, BlueAgentBreeder's ability to show an average 79.4% improvement in safety over the ADAS-generated baselines appears impressive. However, I am concerned that these baselines were not generated with safety in-mind, i.e. no inclusion in the task definition prompts to generate safe, as well as capable, scaffolds. A better benchmark would be the output from an ADAS where the prompt included such an instruction, similar to the prompt used to the Meta Agent. \n* The clustering process is somewhat questionable. The embedding is very low dimensional and it is not clear whether this is sufficient to capture the architectural quality differences between scaffolds. Furthermore, it is not clear how the 0.7 threshold was chosen. Examples from different clusters, and their analysis, would be very useful to illustrate how the embedding and clustering impacts diversity. The paper could be further strengthened if an ablation of cluster was included to concretely demonstrate that it was contributing to performance.\n* Safety evaluation relies on an LLM to judge responses. LLM evaluations are sensitive to their prompting and are susceptible to biases, such as positional, sycophancy, etc. The paper would benefit if the limitations of this evaluation were included and discussed. Human validation of the LLM evaluator on a representative sample would be useful to instill confidence in this approach. \n* The finding that the relative gain between seed and discovered scaffolds diminishes with stronger models is significant and is not fully explored.This could have significant implications regarding the limitations of scaffolding ever-improving LLM models. A more in-depth discussion on this point would significantly increase the paper's impac\n\nQUESTIONS:\nI have a few questions and clarifications.\n\n1. In Table 1, you report on three scaffolds from each experimental run: the most capable (arg max{f_C}), the safest (arg max{f_S}), and a 'best overall' (arg max{f_C, f_S, f_H}). While the first two are clear selections from the Pareto front, could you please clarify the specific criterion or scalarisation function used to select the single 'best overall' scaffold?\n1. Section 4.1 describes a balanced validation sampling strategy using 50% 'positive' and 50% 'negative' samples. Could you please confirm if these categories correspond to samples that a baseline CoT agent answered correctly and incorrectly, respectively?\n1. The impressive safety gains are benchmarked against scaffolds not optimized for safety. To ensure a fair comparison, did you consider creating a stronger baseline by running the original ADAS framework with a prompt that also asks it to consider safety? This seems crucial for isolating the benefit of AgentBreeder's specific methodology.\n1. Could you provide more justification for your clustering methodology (the 12-dim embedding and 0.7 threshold)? An ablation study that removes clustering would be the most convincing way to demonstrate its contribution to performance and diversity.\n\nLIMITATIONS:\nThe authors have addressed most limitations.\n\nThere are a couple more that may be worth including:\n\n* The initial population diversity (from the seven seeds) may have limited the potential diversity of solutions.\n* As discussed above, the limitations of using LLM-based evaluation for safety evaluation should be included and discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AgentBreeder, an approach to multi-objective evolutionary search to automatically generate and evaluate LLM scaffolds. The central application is to optimizing for task capability and safety metrics simultaneously, which is an important problem in AI safety research. The authors demonstrate how their method can be used as (a) a \"blue team\" approach to discover scaffolds with significantly improved safety performance while maintaining or enhancing capabilities; (b) a \"red team\" approach that identifies highly capable yet vulnerable scaffolds; and (c) a capability-only improving baseline.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n\n* **Methodological contributions**: The paper provides a few solid methodological improvements over ADAS, including extending the method to multi-objective optimization and using semantic clustering to balance diversity and efficient evolutionary search.\n\n* **Strong empirical results**: The results convincingly demonstrate that AgentBreeder can find scaffolds that significantly improve performance and safety properties, and that it can also be useful as a red-teaming tool.\n\n* **Careful empirical analysis**: The empirical evaluation is well-executed, the results are reported across multiple benchmarks and with confidence intervals. The analysis is nuanced and the authors discuss positive results as well as important limitations.\n\n# Weaknesses\n\n* **Unclear red-teaming setup**: The paper proposes a red-teaming setup where the goal is to maximize the task performance but minimize safety performance. It is unclear to me what kind of risk model this corresponds to. The authors argue that their approach \"seeks to model the case where an actor may unknowingly expose weaknesses in the base LLM when employing scaffolding to improve task performance\". However by explicitly optimizing for the agent to perform poorly on safety metrics their approach seems much more adversarial then a model developer just aiming to increase performance (which would be better modeled by the capability-only AgentBreeder variant).\n\n* **Some gaps in empirical results**: There are a number of natural questions that the results do not answer that would be easy to answer with minor modifications:\n  * How does AgentBreeder compare to ADAS in the single-objective case?\n  * What impact does the scaffold optimization have on other capability datasets? (eg. how does the MMLU-optimized scaffold  perform on GPQA and vice verse)\n  * How does the capability-only AgentBreeder perform on the safety metrics?\n  * Could you add helpfulness explicitly to the multi-objective optimization target?\n\n* **Lack of clarity about contributions**: The paper proposes a number of methodological advances but is not always clear about which of these are made with which goal in mind. Specifically which changes are to improve the evolutionary algorithm in general, which are to modify it for multi-objective optimization? The paper presents safety as the main motivation of their method but it seems to me that it can be applied much more widely; which would be useful to discuss in the paper.\n\n* **Unclear applicability to frontier models**: The authors admit that the proposed method seems less beneficial for more capable models and discuss possible reasons for this. However the paper does not evaluate the model with state-of-the-art frontier models and does not provide any systematic evaluation of the benefit for different combinations of models. This leaves important open questions about the applicability of the method unanswered.\n\n\n**Minor typos**:\n\n* Line 98: \"Fowler [17] provide\" should be \"Fowler [17] provides\"\n* Line 142: \"improve performance performance\"\n\nQUESTIONS:\nQuestions about experiment results (from above):\n* How does AgentBreeder compare to ADAS in the single-objective case?\n* What impact does the scaffold optimization have on other capability datasets? (eg. how does the MMLU-optimized scaffold perform on GPQA and vice verse)\n* How does the capability-only AgentBreeder perform on the safety metrics?\n* Could you add helpfulness explicitly to the multi-objective optimization target?\n\nQuestions about contributions\n* What is the purpose of the red teaming experiments? Why do you think red teaming is a better model of a developer inadvertently reducing safety than capability-only optimization?\n* Is the proposed method applicable to general multi-objective optimization problems or is it specific to safety problems for some reason?\n\nAdditional questions\n* Is the number in the F_CDROP row the exact same experiment in the red and blue breeder setup? If not what is the difference? If yes why are the numbers different?\n* How does you method compare to running ADAS with a quality metric that combines different optimization targets, eg. a capability and a safety benchmark?\n\nLIMITATIONS:\nThe authors discuss limitations adequately, though (as mentioned above) I think the applicability to frontier models is a potentially major concern that would've been nice to investigate further", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAgentBreeder tackles the under-explored safety implications of turning a single LLM into a multi-agent scaffold.  The authors cast scaffold design as an open-ended, multi-objective evolutionary search: a “Meta-Agent” (Claude 3.5) mutates and cross-breeds code for candidate scaffolds, embeds each design, clusters them, then promotes Pareto-optimal “elites” that jointly maximise capability and safety scores.  The framework can be run in three modes—BLUE (defence, maximise both objectives), RED (attack, maximise capability while minimising safety), and CAPABLE (capability only) — providing an automated way to stress-test or harden agent systems before deployment.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n### Quality\n\n- Reproducibility: algorithm is formalised (Alg. 1), code is promised open-source, and all evaluations are scripted inside Inspect.\n- Empirical support: runs BLUE, RED and CAPABLE modes on three capability benchmarks (DROP, MMLU, GPQA) plus the SaladData safety set; reports confidence intervals and ablates single- vs multi-objective search.\n- Sound methodology: frames scaffold design as a multi-objective evolutionary search, combining embedding-based clustering with Pareto elite selection.\n\n### Clarity\n\n- Clear separation of BLUE / RED / CAPABLE modes aids readability.\n- Paper is well organised; Figure 1 plus Alg. 1 convey the pipeline at a glance.\n\n### Significance\n\n- Addresses a fast-growing deployment pattern (LLMs scaffolded into swarms) that current safety work largely ignores.\n- Provides a practical defence (BLUE) and attack (RED) recipe that labs could integrate into release pipelines.\n\n## Weaknesses\n\n### Quality\n\n- Heavy reliance on proprietary models (Claude 3.5, GPT-4o-mini); results may not generalise to open weights.\n\n### Significance\n\n- Absolute capability gains are modest; blue-team safety jump is large mainly because the baseline is poor.\n- enchmarks are synthetic; no evidence yet that findings transfer to real web-based agents.\n\nQUESTIONS:\n1. Validation-set size & statistical power\n\nYou use only 250 validation items per generation and report bootstrap CIs.  How sensitive are your evolutionary decisions to this small sample?  Did you try larger samples or a moving-average fitness estimate to reduce noise?\n\n2. Hyper-parameter choices\n    - Why fix the agglomerative-clustering distance threshold at 0.7?\n    - How did you arrive at the 2:1 weighting of mutation vs. crossover?\n    - What temperature / top-p settings were used for the Meta-Agent and for the agents inside the scaffolds?\n\n3. Reward-hacking detection\n\nYou note that some scaffolds achieve 95 % “safety” by answering “Sorry, I can’t help with that.”  Beyond adding a helpfulness metric, did you explore behavioural diversity penalties or response-length constraints to discourage such trivial policies?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper makes a significant and timely contribution to AI safety and the Automated Design of Agentic Systems (ADAS) by introducing AgentBreeder, a novel multi-objective evolutionary framework for designing multi-agent scaffolds for Large Language Models (LLMs). It effectively addresses the critical, yet underexplored, safety implications of such systems, offering a robust approach to balancing capability and safety in multi-agent architectures.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\nThis paper addresses a critical gap in the safety of ADAS and self-improving AI systems by explicitly incorporating safety as an optimization objective alongside capability in the design of multi-agent scaffolds for LLMs. This focus is timely and essential, given the growing development and potential deployment of AI self-improvement techniques.\n\nAgentBreeder builds on prior work in Automated Design of Agentic Systems (ADAS) by introducing tailored mutation and crossover operators, multi-objective optimization, and a MAP-Elites-inspired clustering approach. These advancements enhance the diversity and quality of scaffold designs, effectively balancing capability and safety. Surprisingly, the inclusion of a safety objective not only improves adversarial robustness but also appears to facilitate more effective open-ended exploration of the search space, as evidenced by performance improvements over ADAS.\n\nThe framework’s three operational modes—BlueAgentBreeder (optimizing both safety and capability), RedAgentBreeder (maximizing vulnerability and capability), and CapableAgentBreeder (focusing solely on capability)—are extensively evaluated. Experiments demonstrate compelling results, with BlueAgentBreeder achieving a 79.4% average uplift in safety benchmark performance (e.g., SaladData) while maintaining or improving capability on benchmarks like DROP, MMLU, and GPQA. RedAgentBreeder effectively identifies exploitable vulnerabilities, providing valuable insights for improving system robustness.\n\n### Weakness\nThe paper could better justify certain design choices. For instance, the rationale for using clustering in the archive is unclear. Given the typically limited number of solutions explored in ADAS-style algorithms, combining clustering with non-dominated sorting to form Pareto fronts within each cluster may result in most solutions being non-dominated, potentially reducing the effectiveness of the selection process. Ablation studies evaluating key components, such as clustering and the crossover operator, would strengthen the paper by clarifying their impact.\n\nAdditionally, in the red-teaming setting (RedAgentBreeder), the paper optimizes both capability (e.g., DROP performance) and vulnerability (1-SaladData). It is unclear why capability is prioritized, as a typical adversary might focus exclusively on exploiting safety weaknesses, regardless of task performance. A comparison with a single-objective approach, similar to CapableAgentBreeder but minimizing safety scores, could clarify the value of this dual-objective strategy.\n\nOverall, this paper makes a significant contribution to ADAS and self-improving AI by advancing the design of safe and capable multi-agent systems. I would consider increasing my score if the above issues are addressed.\n\nQUESTIONS:\n1. The paper states, \"Weighting the mutation operator twice as highly as crossover was found empirically to lead to faster convergence\" (Line 133). Could the authors provide details or results from the empirical study supporting this claim?\n\n2. The explanation in Line 213 regarding RedAgentBreeder’s approach is unclear: \"It is important to note that in this case, the Meta Agent is not prompted to discover unsafe scaffolds, instead these arise via Pareto optimization on capability and harm benchmarks. This seeks to model the case where an actor may unknowingly expose weaknesses in the base LLM when employing scaffolding to improve task performance.\" If RedAgentBreeder simulates an adversarial attacker, why is the Meta Agent not explicitly prompted to identify unsafe scaffolds? Additionally, how optimizing for both capability and harm benchmarks leads to \"unknowingly exposing weaknesses.\" Isn't minimizing the safety benchmark explicitly trying to expose weaknesses?\n\n3. The results for argmax {f_C^DROP, f_C^MMLU, f_C^GPQA} in Table 1 (CapableAgentBreeder) do not align with those reported in Table 3 (Appendix B.3). Are there differences in experimental settings that account for this inconsistency?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}