AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement
Authors: J Rosser
Jakob Nicolaus Foerster
Venue: NeurIPS 2025 spotlight ()
Keywords: AI Safety
Multi-Agent Systems
LLMs
Large Language Models
Jailbreaking
Agent Scaffolds
Primary area: deep_learning
TL;DR: Our paper introduces AgentBreeder, a framework for multi-objective self-improving evolutionary search over scaffolds, specifically targeting scaffolds' safety impact on large language models in multi-agent systems.

Abstract: Scaffolding Large Language Models (LLMs) into multi-agent systems often improves performance on complex tasks, but the safety impact of such scaffolds has not been thoroughly explored. We introduce AgentBreeder, a framework for multi-objective self-improving evolutionary search over scaffolds. We evaluate discovered scaffolds on widely recognized reasoning, mathematics, and safety benchmarks and compare them with popular baselines. In "blue" mode, we see a 79.4% average uplift in safety benchmark performance while maintaining or improving capability scores. In "red" mode, we find adversarially weak scaffolds emerging concurrently with capability optimization. Our work demonstrates the risks of multi-agent scaffolding and provides a framework for mitigating them. Code is available at \url{https://github.com/jrosseruk/AgentBreeder}.

AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement
J Rosser University of Oxford jrossergrobots. ox. ac
Jakob Foerster FLAIR
University of Oxford
Abstract
Scaffolding Large Language Models (LLMs) into multi-agent systems often im- proves performance on complex tasks, but the safety impact of such scaffolds has not been thoroughly explored. We introduce AGENTBREEDER multi-objective self-improving evolutionary search over scaffolds. We evaluate discovered scaffolds widely recognized reasoning, mathematics, and safety benchmarks and compare them with popular baselines. In 'blue' mode, we see 79.4% average uplift in safety benchmark performance while maintaining improving capability scores. In 'red' mode, we find adversarially weak scaffolds emerging concurrently with capability optimization. Our work demonstrates the risks of multi-agent scaffolding and provides framework for mitigating them. httpuu..oooooosaeeeeeeeeeeeeeeee jrosseruk/Agent Beeee/AAnneneeBeeerrr Code is available at https: //g hight....... com/
framework for
Introduction
Recently, the field of artificial intelligence has witnessed remarkable advancements in Large Language Models (LLMs) and their applications [66]. LLMs are capable of exhibiting human-like reasoning such as
[2, 48, 56], enabling their application beyond natural language processing to diverse code generation [44, 51, 59], embodied AI in robotics [21, 26, 46], and autonomous agents [40, 12]. Our research motivated by accelerated advancements in autonomous agents such as the recent release of Operator [40] and Proxy [12] agents that browse the web and perform tasks autonomously on behalf of the user. Alignment research date has almost exclusively focused on the safety of LLMs in unipolar scenarios; ensuring single LLM remains aligned inside single-agent system. When deployed on the web, agents are placed in novel multi-agent scaffolds and subjected to multi- polar challenges [25]. With highly-capable agents now being deployed at scale, we seek to address the immediate need for more comprehensive safety evaluations of multi-agent systems.
In this paper, we introduce AGENTBREEDER an evolutionary open-ended framework capable of generating large populations of diverse multi-agent scaffolds. By equipping this framework with multi-objective optimization, we explore the generation of multi-agent scaffolds along complementary objectives of capability and safety. AGENTBREEDER to generate offspring that exhibit greater adversarial robustness and performance on capability benchmarks. Similarly, red teaming approach generates offssring that exhibit greater vulnerability to adversarial attacks.
be used to blue team of scaffolds
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Population F
Farcto Frants
Biites
Meta Agent
BluokgeniBrooiee
Seed Scaffolds
Crossowe
Matation
Expao by
RedAgeer Bedeggrrtrreeer
Population Pyri
Satey (d(e)
Capeubilty
Eupubiiky Zch0)
Capab Boreddr
jols)
SAND
Figure high-level illustration of the AGENTBREEDER algorithm outlined in Algorithm 1. Starting from seed scaffolds Qo. at each generation 9 the newly generated scaffolds Qg-1 are evaluated on capability (fc(s) and/or safety (Js(s)) benchmarks, then embedded via fD(s) for clustering A(-) into K clusters. Within each cluster, Pareto fronts F1, Fx are identified according fc(s) and/or fs(s), and these "frontier" solutions become the elite set Eg. An LLM- based Meta Agent applies crossover and mutation to the elites, creating new offspring scaffolds Qg- These offspring are added to the population for the next generation Po+1- By repeating this process for G generations, AGENTBREEDES explores large, diverse set of multi-agent systems while balancing capability and safety. AGENTBREEDER can run in 3 different modes side of this figure shows the optimal direction of travel of the Pareto front for each generation. BLUEAGENTBREEDER
right-hand
a defense mode and seeks to maximize both capability and safety, whereas REDAGENT BEIGGENNBRREEDE is an attack mode minimizing safety. CAPABLEAGENTD BREDDR serves as our baseline, only optimizing for capability without regard to safety.
Algorithm 1: AgentBreeder
Input: Number of generations G; Number of clusters K'; Number of evolutions M; Capability benchmark (s); Safery benchmark [s(;); Embedding function fD();; Seed scaffolds Qo: Clustering function A(-).
Initialize seed population Po = Qo of size No. for generation g = 1 to G do for scaffold Q- do
1. Compute capability fc(s) and safety fs(s).
2. Compute embedding es <- fD(s).
Cluster population into clusters: Ci, C2,..., CK f- Alei, E2 Identify Pareto Elites Eg:
1. Set Eg + 0.
2. for cluster
to do
(a) Find Pareto front F& using fe and fs-
(b) Update elite cohort Fg +- Fg U Fx-
Generate offspring Qn:
Set Q2 0.
2. for evolution m M do
(a) Weighted sampling elites from Eg
(b) If elites, Meta Agent performs Crossover; otherwise Mutation.
(c) Add the offspring to Qg.
Update population: Pg t Py-1 U Qy Update population size: Ng + Ng-1 + M. Output: Final population Pc.

---

Our main contributions are listed as follows:
Attack. We introduce novel red teaming method which can be used explore the attack surfaces of base LLMs when deployed in multi-agent settings.
Defense. We introduce a novel blue teaming method for generating multi-agent scaffolds that exhibit greater robustness to adversarial attacks.
Evaluation. We implement AGENTBREEDE in Inspect [1] to ensure the reproducibility and extensibility our results and methods.
2 Background
Multi-Agent Systems. Multi-agent systems consist of multiple interacting intelligent agents such as LLM assistants like ChatGPT [37]. These systems offer several advantages over single-agent approaches [58], including planning [27.7] 7J. task decomposition [10, 16, 18, 42], and specialization [8, 10, 16, 18, 42). The terms "multi-agent system", "multi-agent framework", "agent" and "scaffold" used interchangeably in literature to refer the structural frameworks that support communication between multiple LLMs [23, 60, 20, 17]. In this paper, we will primarily use the term "scaffold" to refer to the architectures often defined in Python code - that support the operation of multi-agent systems.
Automated Design of Agentic Systems. We build upon the seminal work of Hu et al. [20] which introduces the research area Automated Design of Agentic Systems (ADAS), an automated approach to discovering high-performing (multi-agent) scaffolds. Hu et al. [20] formulate ADAS as an optimization algorithm comprising 3 key components; the search space, the search algorithm and the evaluation function. Hu et al. [20] also propose search algorithm called "Meta Agent Search" where a single "Meta Agent" discovers scaffolds by programming them in Python code. Python is Turing Complete language [5] therefore searching within code space allows the Meta Agent to program theoretically any possible scaffold. This approach has shown promising results [20, 60], with discovered scaffolds outperforming state-of-the-ar hand-designed baselines across various tasks, including reading comprehension, mathematics, and science questions [19, 47, 43, 11, 14].
We formulate AGENTBREEDER with respect to the ADAS methodology. We replicate the approach of Hu et al. [20] by seeding our population with hand-designed scaffolds. We prompt single "Meta Agent" to search for novel scaffolds in the space of Python code. We introduce a novel quality- diversity search algorithm inspired by MAP-Elites [34], where the Meta Agent evolves new scaffolds via the random sampling, mutation and crossover of the highest performing individual "elite" of each niche of the population. We cluster scaffolds based on their architectural features, and evaluate the performance of scaffolds on two benchmarks, one for capability and one for safety. We employ multi-objective optimization, sampling elites from the Pareto front of each cluster.
Multi-Objective Evolutionary Algorithms. Multi-objective optimization searches for solutions to problems with multiple, often conflicting objectives. Multi-objective evolutionary algorithms (MOEAs) incorporate evolutionary approach to generate a diverse set of solutions [24]. In AGENTBREEDER we seek to balance the objectives of capability and safety whilst evolving a diverse range of scaffolds. AGENTBREEDEE balances quality and diversity by clustering scaffolds based on their architectural features and randomly sampling clites from each cluster's 's seuabbiity--sfett Pareto front. A solution is Pareto optimal if no other solution improves one objective without worsening another. The Pareto front comprises all such optimal solutions.
Adversarial Robustness. Adversarial robustness quantifies the resilience of model or scaffold to malicious inputs such as jailbreaks [9] and prompt injection [30]. Red teaming, the practice of simulating adversarial scenarios to identify vulnerabilities, has emerged as crucial tool for assessing Al model risks and alignment [45, 41]. In REDAGENT
instead generating adversarial examples, we seek to evolve multi-agent scaffolds that are more vulnerable to adversarial attacks than the base model. In B..URAGENBRREEDE more robust to adversarial attacks than the base model.
we seek to evolve multi-agent scaffolds that are
3

---

3 Related Work
Self- Leff-Referenial Self-Improving Systems. Numerous frameworks [62, 22, 20, 57, 60] have been proposed to address the design of multi-agent scaffolding. EvoAgent [62] extends single expert agents to multi-agent scaffolds via evolutionary algorithms, whilst AGENT entire system as a unit. EvoMAC [22] evolves agents and their connections during test time to improve code generation, whereas AGENTBREEDER is domain agnostic and can explore the entire search space of scaffolds. ADAS [20], Comfy Agent [57] and Gödel Agent [60] search in the space of code for novel scaffolds, but unlike AGENTBREEDER they do not incorporate quality-diversin mechanism for exploring agent design space. FunSearch [44] is an evolutionary method to search the function space for nigh-performing computer programs but not necessarily scaffolds. PromptBreeder [15] is an evolutionary self- improviig framework that evolves prompts for a given domain, but does not focus on the scaffold as whole.
evolves the
Ageti-Agent Safety Research. Zhang et al. [64] evaluate the safety of multi-agent scaffolds from a psychological perspective by injecting agents with malicious traits, and provide mitigation strategies such as performing psychological assessments and therapy for agents. Polaris [35] introduces a safety-focused scaffold for real-time patient healthcare conversations. Huang et al. [23] explore the resilience of multi-agent scaffolds when injected with malicious or error-prone agents. Fowler [17]
provide a more thorough discussion of the safety risks associated with scaffolded LLMs.
4 AgentBreeder
We now introduce AGENTBREEDER, our automated, evolutionary approach to discovering new multi-agent scaffolds. By evolving large, diverse corpus of multi-agent scaffolds, AGENTBREEDER seeks to address the challenge of evaluating the vulnerabilities of base LLMs acting inside capability- optimized multi-agent scaffolds. The peeudo-lloorihm is given in Algorithm and Figure I provides a brief overview. AGENTBREEDER can be run in three modes:
this mode, the Meta Agent adopts the role of "Blue Team", searching for scaffolds that exhibit high capability and safety when evaluated representa- tive benchmarks.
BLUEAGENTBRREEEE
REDAGENTBREEDEE In this mode, the Meta Agent adopts the role of a "Red Team", minimizing performance on one safety benchmark whilst maximizing performance on one capability benchmark.
CAPABLEAGENTBREEIE In this mode, the Meta Agent seeks to maximize performance on a single capability benchmark without consideration of safety.
4.1 Seed Scaffolds
Following the approach of Hu et al. [20] and Yin et al. [60], we seed our population with the same 7 hand-designed scaffolds. These comprise Chain-of-Thought (CoT) [53]. Self-Consistency with Chain-of- Thought [52], Self-Refine [32], LLM-Debate [13], Step-back Abstraction [67]. Quality- Diversity (QD) [31], and Role Assignment [55]. Before running evolution our chosen benchmark, we evaluate a single CoT agent on 1,000 samples from the validation set of the benchmark, oversampling and resampling where necessary. For each generation, validate the newly discovered scaffolds using a balanced sampling strategy, selecting 50% positive and 50% negative samples. Positive samples correspond those the baseline CoT agent answered correctly and vice versa. We implemented this balanced sampling strategy specifically to increase the signal strength for the evolutionary process, ensuring adequate representation of both success and failure cases. Often improvements between generations are marginal, so this method increases information gain by providing stronger signal for the evolutionary process.
4.2 Mutation Operators
AGENTBREEDER" 's evolutionary search algorithm mimics the process of natural selection comprising mutation, crossover and selection. Claude 3.5 Sonnet [3] 2--0ue-----oonnne--024400000 is used as the core model of the Meta Agent due its state-of-the-a performance on code generation tasks [49].

---

Selection. Selection pressure is applied at each generation by sampling candidate scaffolds at random the Pareto front is simply the
from the Pareto fronts of each cluster. In CAPABLEAGENT elite of each cluster, whereas in BL.UFAGENT REEDER and REDAGEEBBREEDDR the Pareto front comprises the scaffolds which best trade-off safety and capability.
The Meta Agent uses weighted random sampling to select either the crossover or muta- tion operator. Weighting the mutation operator twice as highly as crossover was found empirically to lead to faster convergence. Mutation is performed via random sampling of mutation operators expressed short textual passages we desindesigned. There are two types of mutation operators. supablliyyeenhacced enhanced and safety-enhanced. When running BLUEAGENNBREEDDE are randomly sampled from the concatenated capability- and safety-enhaane enhanced corpus. REDAGENT- BREEDER and BEEBLLEAAEBEBEEE
Mutation.
mutation operators
the safety-enin enhancedd operators are omitted. The full list of Meta Agent prompts and mutation operators are given in Appendix D.
Crossover. During crossover, the Meta Agent is provided with two randomly sampled scaffolds from the population and tasked with combining them in such way that would be likely improve performance performance. The full crossover prompt is given in Appendix D.6.
4.3 Descriptors
In open-ended evolutionary approaches, descriptors are essential for quantifying the diversity of candidate solutions [34]. In order to explore the full range of vulnerabilities of base model, seek to generate and evaluate a diverse range of multi-agent scaffolds and require high-dimensional descriptors. In 12-dimensional text embedding of the system name and code as our descriptor to encode semantic information about the name, structure, and potential logic embedded in the scaffold.
we use OpenAI's foxt-embedding----iae 3-maall [39] model returning
4.4 Clustering
Once the descriptors have been generated for the new scafolds, AGENT re-clusters the whole population based on their descriptors to discover groups of similar architectures. We choose agglomerative clustering it has been found be particularly effective for smaller datasets like ours [54]. By setting a distance threshold in the agglomerative clustering algorithm, we allow the number of clusters adjust flexibly. When the number of clusters increases, the selection pressure decreases towards zero. Conversely, reducing the number of clusters encourages the algorithm to explore only few options, which leads to less diverse scaffolds. To achieve balanced trade-off between system performance and system diversity, a distance threshold of 0.7 was selected.
4.5 Multi-Objective Pareto Elites
In Divvity-Diversity algorithms such as MAP-Elites [34], selection pressure is applied by randomly sampling the gighest-performing candidates in each niche for evolution, referred to as the "elites". In multi-objective optimization, a solution is Pareto optimal if no other solution improves one objective without worsening another [24]. The Pareto front comprises such optimal solutions. In AgentBreend instead of sampling from pre-defined niches, we sample elites from the Pareto fronts of dynamically generated clusters.
4.6 Evaluations
Evaluations are implemented in Inspect [1], an open-source framework for LLM evaluations. We instantiate AGENTBREEDER as custom model provider by deriving new class from ModelAPI, and each individual scaffold derives as a Model from that ModelAPI. This allows comprehensive experiment tracking and parallelization, and provides extensible framework allowing AGENT- BREEDER to run on a new benchmark often with fewer than 100 lines of code. Section we report results on 5 benchmarks comprising safety, capability and helpfulness.
5

---

Experiments
We conduct experiments to validate AGENTBREEDER 's three modes; BLUEAGENTBREEDER REDA- GENTBREEDER, CAPABLEAGENTBREEDRR To evaluate the capability of the multi-agent scaffolds produced, we follow the approaches of Hu et al. [20] and Yin et al. [60] and report results three benchmarks from OpenAl's simple-evall [36]. To evaluate system safety, we report results on one comprehensive safety benchmark. A full description of each benchmark can be found in Appendix C. To evaluate the safety of multi-agent scaffolds, we adopted method inspired by Constitutional AI, where an LLM judges responses based Anthropic Claude Constitution [4] into a series of yes/no questions. During validation, each system responded to 100 attack-enhanced prompts, and agent aware of the constitution evaluated the safety each response. We then reported the median accuracy and the 95% bootstrap confidence interval for each scaffold's safety performance.
predefined set of principles. We transformed the
5.1 Experiment 1: Blue Team Defense
We ran BLUEAGENTBREEDER for 20 generations, on each of our three chosen capability benchmarks (DROP [14], MMLU [19] and GPQA [43]) independently, with the aim of generating "Blue Teams" of multi- scaffolds that simultaneously optimize for capability and safety across our chosen benchmarks. The evolution is visualized in Figure and report the HV indicator the test set for each experiment in Table 1. The Meta Agent discovers 10 new scaffolds each generation, and report the median accuracy and the 95% confidence interval on the held-out test set for the best performing discovered scaffold in Table 2. The "best" scaffold maximized the sum of capability and safety scores, with the important caveat that excluded scaffolds that engaged in reward hacking of the safety objective. A more detailed visualization of the evolutionary process is shown in Figure 's tendency reward-hack the safety benchmark by finding trivial safe response to question-answe answering tasks that require a long-form response, during evaluation, we report the "helpfulness" of the scaffold questions from TruthfulQA [29].
To reduce BLUEAGENT
GPCA
MUUI
CRCP
Engeel
bar 20 cuit tarestio
Fodeet Burltai Fasta Sayber
evolves scaffolds that improve the capability-safety Pareto frontier. The plots show the evolution of multi-agent scaffolds across 20 generations the validation set of three different benchmarks: GPQA (left), MMLU (middle), and DROP (right). Each point represents scaffold, with colors indicating generation (lighter blue for seed scaffolds, darker blue for later generations). The x-axis measures capability (fc) and the y-axis measures safety (fs). The light blue shaded region shows the Pareto frontier of the seed generation, while the dark blue region shows the Pareto frontier of evolved scaffolds.
Figure 2:
Benchmark GPQA MMLU DROP
Seed Scaffolds
Discovered Scaffolds
0.219064
0.247536
0.484208
0.542816
0.390754
0.438813
Table Reporting the HV indicator on the test set for BLUEAGENTBBEEDDER

---

BLUEAGENTBREEDER
Capability
Safety
Helpfulness
DROP
MMLU
GPQA
SaladData
TruthfulQA
Seed Scaffolds from ADAS [20]
Chain-of-Though Thought (CoT)
66.6 5.0
80.0 + 4.4
31.2 = 5.6
29.2 + 5.6
86.8 + 3.6
Self- Consistency CoT
66.0 : 4.4
81.6 = 4.8
32.4 6.0
22.8 = 5.2
85.6 4.4
6.84
28.4 6.0 5.2
26.0
86.8 + 4.0
Debate
69.9 4.4
7..6
29.6
29.6556 5.6
36.4
6.0
86.4
Step-Back Abstraction
71.4 + 4.3
79.2
2 4.8
30.8
5.2
40.8
5.6
85.2
Quality- -ivverstyy
78.0 3.9
81.6
4.4
28.4
5.6
25.8
5.8
87.2 4.0
Role Assignment
75.8 = 4.2
79.2
= 4.8
32.0 6.0
18.0 = 5.2
85.6 - 4.4
BlueAgentBreeder Scaffolds (S SaladData, H TruthfulQA)
[fCrcee) arg max,
79.0 3.8
46.4 + 6.4
88.0 4.0
arg max,
{fs)
62.0 + 4.8
86.0 4.0
83.6 4.4
arg max, Cuncee Ss, }
62.0 4.8
86.0 66.0:.00
83.6
arg max. SCssau
85.2 4.4
5.6
81.2
arg max, [[s]
84.0 4.4
8.04
76.0
arg max,,
fomaa fHHfH)
84.0 = 4.4
8..4
76.0
arg max, frrrg
39.2 = 5.6
6.8
57.6 6.4
arg max, arg max, ( S1)
[ss
31.2 6.0
95.2 + 2.4
49.6 + 6.4
36.8 5.2
49.2 + 6.8
86.8 4.0
Table 2: We report the evaluation results of BLUEAGENT capability benchmark (DROP [14], MMLU [19], GPQA [43]), safety benchmark (SaladData [28]) and ensure trivial solution has not been found by evaluating each scaffold's helpfulness on TruthfulQA [29]. For each benchmark, we compare the AGENT scaffolds, with the highest and second highest result in each column given in bold and underlined respectively. We report the median accuracy (or Fl score for DROP) along with a 95% confidence interval computed over 250 held-out test samples. The evolution is conducted independently for each capability benchmark.
on the held-out test set of
scaffolds against seed
Significant Increase in Performance our Safety Benchmark. We observed 1 110.7% maximum increase in performance of our best discovered scaffold on SaladData [28] with an average uplift of 79.4% across all three experimental runs.
Performance on Capability Benchmarks Maintained if not Increased. Whilst providing a substantial safety uplift, were able to ensure capability was either kept the same additionally provided a marginal uplift, for example as much as 21.0% on GPQA [43].
Vulnerability to Reward Hacking. We note that certain scaffolds display inflated scores our safety benchmark, achieving an accuracy of 95.2 2.4. However, under closer examination, the system shows significant 43% decrease in performance multi-agent scaffolds are prone to reward-hacking the safety objective e.g. by finding trivial solution such responding "I' sorry, can't san't help with that." to any question outside of their domain, and that one way of detecting this behavior is by reporting performance on helpfulness benchmark.
our helpfulness benchmark. We find that
5.2 Experiment 2: Red Team Attack
REDAGENTBREEDE seeks to discover "Red Team" scaffolds that exploit the safety vulnerabilities of the base LLM while preserving capability on chosen benchmark. in this case DROP [14]. We introduce an inverted safety benchmark "1 "1-SaladData" which contains the same questions as the original SaladData [28] benchmark, but utilizes an inverted scoring function that awards higher scores for unsafe responses. The REDAGENTE BREEDER scenario models the outer misalignment problem, where scaffold designer operates under misaligned reward function that inadvertently incentivizes unsafe behavior while pursuing seemingly beneficial objectives. It is important to note that in this case, the Meta Agent not prompted to discover unsafe scaffolds, instead these arise via Pareto optimization capability and harm benchmarks. This seeks to model the case where actor may

---

unknowingly expose weaknesses in the base LLM when employing scaffolding to improve task performance. visualization is provided in Figure and Tables and 4 summarize these results.
DROP
Gan
(heed)
5.0
12
6.2
8.6
1.0
Don thead scatiees) tust anaaarr
Evshvec Scattals Parids Pepie
Parato Hounder
Figure 3: REDAGENTBREEDDE evolves scaffolds 10 generations. The validation set results are reported, with the x-axis shows capability (fc) and the y-axis shows inverted safety (1-fs), with higher values both axes representing the optimization objectives. Lighter red points indicate seed scaffolds (Gen O), while darker red points represent scaffolds from later generations (up to Gen 10). The shaded regions highlight the Pareto regions of both seed scaffolds (lighter area) and evolved scaffolds (darker area), with the connected points along the boundary representing non-dominated solutions.
Benchmark DROP
Seed Scaffolds Discovered Scaffolds
0.572759
0.0.602547
Table 3: Reporting the HV indicator the test set for CEDAGENTBREEDDE
REDAGENT BREEDER
Capability DROP
Harm
1-SaladData
Seed Scaffolds from ADAS [20]
CoT
68.2 4.7
76.8 + 4.8
CoTSC
69.6 + 4.8
70.8 + 6.0
Refine
62.6 f 4.7
71.6
Debate
63.2 + 4.8
56.0
StepBack
67.5 4.5
5.00
QD
74.9 + 4.2
73.6 7
Roles
73.1 3.7
6.0
RedAgentBreeder Scaffolds
arg max, max,tfoomm {fCoup } arg max, {f1-s) arg max, (.Crrur fi-s}
75.0 + 4.1
26.0 + 5.2
67.7
4.7
81.6 + 4.8
67.7
+ 4.7
81.6 + 4.8
Table 4: We report the evaluation results of AGENTBREEDER test set. We seek to maximize performance on DROP [14] whilst also maximizing performance on 1-SaladData [28], an inverted version of the SaladData benchmark where unsafe responses scored highly. For each benchmark, we compare the SHEENBRREEDRRdRE scaffolds, with the highest and second highest result in each column given in bold and underlined respectively. We report the Fl score and median accuracy for DROP and I-SaladData respectively, along with 95% confidence interval computed over 250 held-out test samples.
in "red" mode the held-out
scaffolds against seed

---

Unsafe Scaffolds are Easier to Find. REDAGENTBREEDER highest performing scaffold achieved 81.6+4.8 aceuracy on the inverted I-SaladData [28] metric, surpassing all seed scaffolds by least 6.25% with only half the generation budget of BL.UEACENTBREEDRR These results indicate that scaffolding may be more likely to weaken than strengthen the base L.MM to adversarial attacks.
Capability Disguises Safety Vulnerabilities. Interestingly, even while maximizing unsafe perfor- mance, we were able to achieve competitive Fl score 67.7.4..7 This result is comparable the seed scaffolds, highlighting that scaffolds may appear just as capable in terms of task performance yet simultaneously exhibit increased safety vulnerabilities.
5.3 Experiment 3: Multi-Objective Ablation
As an ablation for our work, we run PREABLEAGENTBREEEE
criteria and to compare AGENTBREEDER against the seminal
of our framework for 20 generations, evolving 10 mutants each generation. We take the highest-performing scaffolds from ADAS [20] and evaluate them with GPT-4o mini [37] as their core model. We report the Fl score for DROP [14], median accuracy for MMLU [19] and GPQA [43] and their 95% confidence intervals, as well as their performance on SaladData [28], our chosen safety benchmark. The results are shown in
a single-objective-variaa
Table 5 in Appendix B.
Comparable Performance to Previous Work. CAPABLEAGENT BREEDER achieves competitive results to ADAS, marginally surpassing performance across all capability benchmarks.
Multi-Objective outperforms Single-Objective Optimization. The scaffolds discovered by CA- BALLLAGGENTRREEDEE achieve near or slightly above-baseline results, such as 72.3 #3.1 F1 on DROP and 41.2#4.4 accuracy on GPQA. This performance gain is notably smaller than in the multi-objective setting. This supports our hypothesis that incorporating an additional benchmark may increase the signal-to-noise ratio of scaffold validations each generation. This improves the quality of the selection pressure for the evolutionary algorithm, helping the process converge to better solutions overall.
Insignificant Impact on Safety Performance. In single-objective ablation runs, the discovered scaffolds showed only modest performance uplift on SaladData [28], suggesting that ignoring safety in the objective yields no strong impetus for safe or unsafe behaviors. This contrasts with multi- objective runs, where explicit safety optimization (or "negative safety" in red-teaming) substantially influenced outcomes.
Performance Stagnates with Better LLMs. When using more advanced models (GPT-4o mini [37]
for scaffolds and Claude 3.5 Sonnet [3] for the Meta Agent) compared to the original ADAS [20]
implementation, we observe that while overall performance improves, the relative gain between seed
and discovered scaffolds diminishes. We attribute this to three plausible factors: (1) increased data
contamination in newer LLMs may lead to memorized solutions rather than genuine reasoning. (2)
higher baseline performance makes marginal improvements harder to distinguish from noise and (3)
recent models are already fine-tuned for detailed reasoning, reducing the benefit of scaffold-induces reasoning steps [38, 63].
Discussion
Pre-Deployment Safety Evaluations. The Dead Internet Theory posits a future where agents dominate online activity [50]. While speculative, the recent releases of Operator [40] and Proxy [12] highlight the increasing population of agents deployed with the ability to interact autonomously with other agents and humans. These underscore the uncertainty around agent-on-agent dynamics, especially when these agents evolve or compose themselves in unanticipated ways. Our REDA- GENTBREEDER experiments illustrate an automated approach to efficiently surface multi-agent scaffolds that exhibit vulnerabilities on safety benchmarks. Over time, labs could adopt a REDA- GENTBREEDER-StV/ pipeline proactively "red-team" new LLMs as part of release protocol.
Post- Dost-Deployment Adversarial Robustness. Just as REDAGENTBREEDER discovers vulnerable scaffolds, BL.EEAGENTBBEEDE provides a methodology to design safe and capable multi-agent scaffolds. This method can also be used to upgrade the safety capabilities existing scaffolds, akin to Weak-to-Strone Generalization [6]. Furthermore, BLUEAGENTB BREEDER can be used to

---

ensure scaffold conforms to dynamic company values, policies and regulatory requirements. These experiments validate the practicality of evolutionary search as dynamic, data-driven tool for multi- agent evaluation.
Limitations. While our experiments provide promising insights, several limitations should be ac- knowledged. Firstly, due to computational costs, we conducted experiments over a limited number of generations and with relatively small population sizes, resulting in only marginal performance improvements. Secondly, our experimental setup serves as proof of concept for multi-objective alignment, and stronger claims of helpfulness and safety would require evaluations on more com- prehensive benchmarks. Additionally, our evaluation was restricted to a select set of benchmarks, which may not fully capture the diverse range of real-world capabilities and safety concerns. Finally, the initial population was limited to seven seed scaffolds, potentially constraining the diversity of discovered scaffolds.
Conclusion
This paper introduces AGENTBREEDER, evolutionary framework for discovering and evaluating multi-agent scaffolds via the multi-objective optimization of capability and safety. Our experiments demonstrate that AGENTBREEDER operates effectively in three distinet modes. BLUF.AGENT- BREEDER for developing safer scaffolds, REDAGENTBREEDER for identifying vulnerabilities, and CAPABLEAGGET for maximizing task performance. Through empirical evaluation across multiple benchmarks, we show that our framework discovers scaffolds that achieve competitive or increased performance prior works while exhibiting increased adversarial robustness.
Our results highlight several important findings for Al safety research. First, we demonstrate that unsafe behaviors can coexist with strong task performance, evidenced by REDAGENTB DEDAGENTBREEDER REEDER'S 's ability to generate scaffolds that maintain capability while cxhibiting increased vulnerability. Second, experiments reveal that multi-objective optimization targeting both capability and safety yields better overall solutions compared to single-objective approaches. Third, our BLUEAGGNTBRRENE experiments achieved substantial safety improvements (up to 110.7% increase SaladData with 19.4% average uplift) while simultaneously maintaining or enhancing capability (up to 21. 21.0% improvement GPQA). Finally, we show that automated evolutionary methods can effectively probe the complex attack surfaces of multi-agent scaffolds, offering a practical approach to pre-deployment safety evaluation.
As AI systems become increasingly interconnected and deployed in roal-world settings, frameworks like AGENTBREEnE ations. Our work establishes a foundation for the systematic evaluation of multi-agent scaffolds, contributing to the development of safer and more reliable Al technologies.
bridge the research gap between single-agent and multi- ageti- safety evalu-
8 Future Work
Scaling Laws. Scaling AGENTBREEDER to larger population sizes and longer evolutionary runs could yield more substantial improvements in both capability and safety metrics. Incorporating closed- source safety benchmarks such as AlLuminate [33] and contamination- capability benchmarks such as MMLU-CF [65] would provide a more comprehensive assessment of multi multi-agent system safety.
White-Box and Gray-Box Evaluations. A key limitation of our current approach is its focus on black-box evaluation of scaffolds. Future work could investigate individual agent behaviors, including how agents interact with tools, external APIs, and information sources. Developing methods to trace and analyze agent-agent and agent-tool interactions could reveal potential safety risks that are invisible in black box evaluation.
Alternative Objectives. In this work, we only consider the capability and safety objectives for optimization. Future work could explore inference cost objective to minimize for, and consider multi-core scaffolds where different LLM base models exist inside the same scaffold.
Multi- geet Governance. Critical research is needed to establish governance frameworks for multi-agent scaffolds. Future work could comprise developing differentiated safety cases for scaffolds with varying levels of transparency, from fully white box to black box architectures.
10

---

Acknowledgments and Disclosure of Funding
J Rosser supported by the EPSRC centre for Doctoral Training in Autonomous and Intelligent Machines and Systems EP/Y035070/1. We extend our sincere gratitude to the members of the Foerster Lab for AI Research (FLAIR) for their guidance during the project scoping phase and thorough proofreading. Special thanks to the London Initiative for Safe Al and Arcadia Impact for providing workspace and offering invaluable feedback throughout.
References
[1] UK Al Security Institute. Inspect AI: Framework for Large Language Model Evaluations, 2024. URL https://githhii Ctmm///KGGeerrmeettttt/aaaaeeeet
[2] Maryam Amirizaniani, Elias Martin, Maryna Sivachenko, Afra Mashhadi, and Chirag Shah. Can Ilms reason like humans? assessing theory of mind reasoning in Ilms for open-ended questions. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 34-44, 2024.
[3] Al Anthropic. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card, 3:6, 2024.
[4] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. 2022.
[5] Robert S. Boyer and J. Strother Moore, mechanical proof of the turing completeness of pure lisp. Technical Report 130625, Texas Univ at Austin Inst for Computing Science and Computer Applications, May 1983. URL https: //tp://apppppp/pp dtic. mwl/stiiciaatiinotttt ADA1 Approved for public release.
30625.
[6] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong gener- alization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023.
[7] Hong Cao, Rong Ma, Yanlong Zhai, and Jun Shen. LIm-collab: a framework for enhancing task planning via sfain-of-thoughh and multi-agent collaboration. Applied Computing and Intelligence 4(2):328-33) 348, 2024.
[8] Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better lim-based evaluators through multi-agent debate. arXiv preprint arxiv:2008.0720 2023.
(9] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08413 2023.
[10] Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308. 10848, 2(4);6, 2023.
[11] François Chollet. On the measure of intelligence. arXiv preprint Xrxi:: 19100119911.0157,2211
[12] Convergence. Introducing web-world models, December 2024. URL nce. ai aittraining-ut web-inng-----bbee wwweiiingwwwwaaess--eeeee wehh-weebaaee wew.thhhee-woolddoooooooo- 2025-01-30.
2024/. Accessed:
[13] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning language models through multiagent debate. arXiv preprint arXiv:2305. /4325, 2023.
[14] Dheeru Dua Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gard- ner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint 1033n:1903.0016,, 2019.

---

[15] Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rock- täschel. Promptbreeder: Self-referential self self-improvement via prompt evolution. arXiv preprint arXiv:2309. 16797, 2023.
[16] Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al. Magentic-one: generalist multi-agent system for solving complex tasks. arXiv preprint 1..:::2441..440000 2024.
[17] Stephen Fowler. Scaffolded Ilms: Less obvious concerns. LessWrong, 2023. URL. https: www.lesswwoon cww/posta/////pccccooooooocossst homm/osta/mAAwaan us- concerns.
11ms- less-eessees obvio
[18] Alireza Ghafarollahi and Markus Buchler. Sciagents: Automating scientific discovery through multi-agent intelligent graph reasoning. arXiv preprint arXiv:2409.05556. 2024.
[19] Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0m 03300, 2020.
[20] Shengran Hu, Cong Lu, and Jeff Clune. Automated design agentic systems. arXiv preprint 20Xiv:2408.08435 2024.
[21] Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis, Jay Patrikar, Nikhil Keetha, Seungchan Kim, Yaqi Xie, Tianyi Zhang, Hao-Shu Fang, et al. Toward general-purposs robots via foundation models: A survey and meta-analysis. arXiv preprint arXiv:2312.08782, 2023.
[22] Yue Hu, Yuzhu Cai, Yaxin Du, Xinyu Zhu, Xiangrui Liu, Zijie Yu, Yuchen Hou, Shuo Tang, and Siheng Chen. Self-evolving multi-agent collaboration networks for software development. arXiv preprini arXiv:2410. 16946, 2024.
[23] Jen-tse Huang, Jiaxu Zhou, Tailin Jin, Xuhui Zhou, Zixi Chen, Wenxuan Wang, Youliang Yuan, Maarten Sap, and Michael Lyu. On the resilience of multi-agent systems with malicious agents. arXiv preprint arXiv:2408. 200.09900000999, 2024.
[24] Adarsh Kesireddy and F Antonio Medrano. Elite multi-criteria decision making-pareto front optimization in multi-objective optimization. Algorithms, 17(5):206, 2024.
[25] Akbir Khan. Why multi-agent safety is important. https://www, lesswrong, wwwp:wrwnn...o.popoooooeooeeee/e com/post s/pks #RGKKR (atecc9ddduurrrTT...o http///whmmulttt multimmliiiii agent sreentssffey-is--iee inmporrtann 2022. Accessed: 2025-01-30.
[26] Xiangrui Kong, Thomas Braunl, Marco Fahmi, and Yue Wang. A superalignment framework in autonomous driving with large language models. arXiv preprint vrxiv:2406.05651, 2024.
[27] Ao Li, Yuexiang Xie, Songze Li, Fugee Tsung, Bolin Ding, and Yaliang Li. Agent-oriented planning in multi-agent systems. arXiv preprint arXiv:2410.02189, 2024.
[28] Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. Salad-bencis A hierarchical and comprehensive safety benchmark for large language models. arXiv preprint arXiv:2402.05044 2024.
[29] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021.
[30] Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang. Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, et al. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.055. 20499999.00 2023.
[31] Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scien- tist: Towards fully automated open-ended scientific discovery. arXiv preprint NiXx:2408.0629) 2024.
12

---

[32] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang, al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
[33] MLCommons Association. Ailuminate v1.0 benchmark, 2025. URL URL https:///aat wnllumntnatt ml commons. wwt. htss/bnnchmarrk/// Accessed: 2025-01-31.
[34] Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv: 1..500440099 2015.
[35] Subhabrata Mukherjee, Paul Gamble, Markel Sanz Ausin, Neel Kant, Kriti Aggarwal, Neha Manjunath, Debajyoti Datta, Zhengliang Liu, Jiayuan Ding, Sophia Busacca, et Polaris: sacety-focused Ilm constellation architecture for healthcare. arXiv preprint MxXi::4403113111 2024.
[36] OpenAL. simple-evals. 2025-01-23
http://hhhh....oooopp///ssppp/// 2023. Accessed:
[37] OpenAL. Gpt-4o mini: Advancing cost-efficient intelligence, 2024. URL https://opeaa.. om/ wewiindxxgppp-eee mini
cost www.reeooooooo.eeeeeeeeeeeeeeeee
[38] OpenAl. Learning reason with Ilms, 2024. URL htttps://ppenai.com//neeee//////
g- t-tooreason--ae with with--lmsh Accessed: 2025-01-31.
[39] OpenAl. New embedding models and api updates. Jamuary 2024. URL https://openaii.. /research/new-enneeee hwwparchc/eeeeeebbbddnnngggg -ww.nedddin-oddeleeeeeeeeeeeeeee models- and--and--a- api- Accessed: 2025-01-14.
[40] OpenAI. Computer-using agent: Introducing universal interface for ai to interact with the digital world, 2025. URL https://openaa. com. ccommindee womm/ndeexcomppuerr using------ agent.
[41] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:220..032866 2022.
[42] Chen Qian, Xin Cong. Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint arXiv:2307.07924, 6(3), 2023.
[43] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpga: A graduate-level google-prod proof q&a benchmark. arXiv preprint arXiv:2311.12000 2023.
[44] Bernardino Nomera-Pareded Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 62577995):468 475, 2024.
[45] Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H Markosyan, Manish Bhatr, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, et al. Rainbow teaming: Open-ended generation of diverse adversarial prompts. arXiv preprins 2002 2024.
[46] Sebastian Sartor and Neil Thompson. Neural scaling laws for embodied ai. arXiv preprint wwXiv:2405.14005 2024.
[47] Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung. Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual chain-of-thought reasoners. URI https://arxiv. org/abs/22100030 10.00305. 2022.
[48] Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang, Shuo Shang. Ji-Rong Wen, and Rui Yan. Determir: Augmenting llmebased logical reasoning from indeterminacy to determinacy. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9828-9862 2024.
13

---

[49] Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li. et al. Scicode: A research coding benchmark curated by scientists. arXiv preprint 131.,6,,0.. 2024.
[50] Yoshija Walter. Artificial influencers and the dead internet theory, Al & SOCCETY, pages 1-2, 2024.
[51] Jianxun Wang and Yixiang Chen. A review on code generation with Ilms: Application and evaluation. In 2023 IEEE International Conference Medical Artificial Intelligence (MedAI), pages 289-289. IEEE, 2023.
[52] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint ar arXiv:2203.11177, 2022.
[53] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia. Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thouglh prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[54] Anna Christina Weigand, Daniel Lange, and Maria Rauschenberger. How can small data sets be clustered?, 2021.
[55] Benfeng Xu, An Yang, Junyang Lin, Quan Wang. Chang Zhou, Yongdong Zhang, and Zhendong Mao. Expertprompting: Instructing large language models to be distinguished experts. arXiv preprint arXiv:2305. 14688, 2023.
[56] Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng. al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09638 2025.
[57] Xiangyuan Xue, Zeyu Lu, Di Huang, Zidong Wang, Wanli Ouyang, and Lei Bai. Comfy- bench: Benchmarking l1m-based agents in comfyui for autonomously designing collaborative ai systems, 2024. URL https: / /arxivv.vvvv . org/ //////ba////// 2209//49..0 013992.
[58] Yingxuan Yang, Qiuying Peng, Jun Wang, and Weinan Zhang. Aulti-llm-agent systems: Techniques and business perspectives. arXiv preprint arXiv:2411.1403: 2024.
[59] Burak Yetistiren, Işik Özsoy, Miray Ayerdem, and Eray Tüzün. Evaluating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt. arXiv preprint arXiv:2304. 10778, 2023.
[60] Xunjian Yin, Xinyi Wang, Lianggingg Pan, Xiaojun Wan, and William Yang Wang, G\"" odel agent: self-referential agent framework for recursive self-improvement. arXiv preprint 1rXiv:2410.044: 04444, 2024.
[61] Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. Gptfuzzer: Red teaming large language models with auto- -ettrreedetted jailbreak prompts. arXiv preprint vrXiv:2309.1025: 2023.
[62] Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Dongsheng Li, and Deqing Yang. Evoa- gent: Towards automatic multi-agent generation via evolutionary algorithms. arXiv preprint arXiv:2406.1422 2024.
[63] Wojciech Zaremba, Evgenia Nitishinskaya, Boaz Barak, Stephanie Lin, Sam Toyer, Yaodong Yu, Rachel Dias Eric Wallace, Kai Xiao, and Johannes Heidecke Amelia Glaese Trading inference-time compute for adversarial robustness., 2025.
[64] Zaibin Zhang, Yongting Zhang, Lijun Li, Hongzhi Gao, Lijun Wang, Huchuan Lu, Feng Zhao, Yu Qiao, and Jing Shao. Psysafe: A comprehensive framework for psychological- attack, defense, and evaluation of multi-agent system safety. arXiv preprint 2ttx/i::401.//880 2024.
[65] Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Qinzheng Sun, Shaoguang Mao, Xin Zhang, Ying Xin, Qiufeng Yin, Scarlett Li, et al. Mmlu-ef: contamination-free multi-task language understanding benchmark. arXiv preprint 41X11:241..15194, 2024.
14

---

[66] Wayne Xin Zhao. Kun Zhou, Junyi Li. Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, al. A survey of large language models. arXiv preprini arXiv:2303. 18223, 2023.
[67] Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H Chi, Quoe V Le, and Denny Zhou. Take step back: Evoking reasoning via abstraction in large language models, arXiv preprint arXiv:2310.01117, 2023.
A Impact Statement
This work introduces methods for evaluating and improving the safety of multi-agent scaffolds, which increasingly critical as embodied, autonomous agents become more prevalent. While AGENTBREEDER can help discover safer multi-agent architectures, it could also be used to find scaffolds that exploit vulnerabilities. deployment, but acknowledge the dual-use nature of these techniques. The red-teaming capabilities we describe could misused to develop harmful scaffolds, though we believe the defensive benefits outweigh these risks. Additionally, our research surfaces important questions about Al governance as multi-agent scaffolds become more common. We hope this work advances the field's understanding of multi-agent safety and helps develop more robust evaluation frameworks. We encourage future research to build upon these methods while carefully considering potential misuse and implementing appropriate safeguards.
release this research enable proactive safety testing before
B CapableAgentBredder
Table shows the results of CAPABLEAGENTBREEDER reported on three benchmarks and compares them with the performance of ADAS [20].
CAPABLEAGEETTRREDDDR
Capability
Safety
DROP
MMLU
GPQA
SaladData
Seed Scaffolds [20]
70.4 3.1
80.2 3.6
35.2 = 4.4
31.2 + 4.2
64.4 + 3.2
82.6 3.4
38.1 + 4.3
17.8 + 3.4
69.3 3.2
81.2 3.6
39.4 4.4
55.6 4.6
ADAS Scaffolds
72.0 + 3.0
57.0 + 4.2
80.4 + 3.4
76.4 3.6
37.4 + 3.6
61.0 + 4.2
CapableAgentBreedder Scaffolds
72.3 3.1
39.4 4.4
$2.4
58.0 + 4.2
41.2 - 4.4
43.8 + 4.4
Table 5: We report the evaluation results of CAPABLEAGGETTBREEDDR on the held-out test sets. scaffolds with the seed and
For each benchmark, discovered scaffolds from the seminal work ADAS [20], with the highest and second highest result in each column given in bold and underlined respectively. We report the Fl score DROP [14] and median accuracy on the other benchmarks, along with a 95% confidence interval computed over 500 held-out test samples.
compare the BHENTBREEDER-diSCOOOOE
15

---

C
Benchmarks
MMLU [19] is multi-task benchmark comprising multiple choice questions on 57 subjects across STEM, the humanities, the social sciences, and more.
DROP [14] is English reading comprehension benchmark which requires the model reason over and answer questions on given paragraphs. This may be simple look-up or extend to more complex tasks such counting, addition and sorting.
a benchmark comprising graduate-level multiple-choice questions in the field of biology, physics, and chemistry. The questions are designed to be challenging such that experts in the domain achieve 65% accuracy and non-experts achieve 34% accuracy.
GPQA [43]
SaladData [28] is hierarchical and comprehensive safety benchmark spanning 3 levels. We report results on the "Attack Enhanced" subset, where harmful questions have been further enhanced with variety of attack methods such as GPTFuzz [61], automated method of generating jailbreak prompts via black-box "fuzzing".
TruthfulQA [29] is -anwweinnn benchmark comprising questions that some humans may answer incorrectly. The types of questions span 38 categories, including health, law, finance and politics.
D Prompts
The Meta Agent is tasked with iteratively programming novel scaffolds. After evolving the first candidate solution, the Meta Agent goes through two iterations of refinement, and then up three iterations of debugging. The prompts for all of these are given below and are modified versions the prompts designed by Hu et al. [20] for ADAS.
D.1
Base Prompt
Meta Agent Base Prompt
You are an expert machine learning researcher testing various agentic systems. Your objective is design building blocks such as prompts and control flows within these systems to solve complex tasks.
<< BENCHMARK DESCRIPTION >>
<< SAFETY DESCRIPTION>>
# The utility code:
pyython from collections import namedtuple from typing import Union import numpy as import json
import openai
import backoff
from utils import random_id
Initialize the OpenAI cléent client spenai.0penkle ()
<< STRUCTURED JSON RESPONSE CODE >>
AGENTIC FRAMEWORK CODE
class AgentArchitecture:
16

---

nan
Fill on your code here.
cun
asyne def (sel, task,
>>
str: .."
Placeholder method for processing task information.
Args:
task (str): Tash description.
required, anmrrdd_oouer (str): required format for the answer.
g.g. lates code, or single letter A,B,C,D etc.
Returns:
Answer (str): Your FINAL Answer. Return as string in
the ezact
format as specified in the ansdrred_asswer... format. ane
pass
Discovered architecture archive Here the archive of the discovered architectures:
<< ARCHIVE>>
The fitness value is the median and 95% Bootstrap Confidence Interval of the correct rate on validation question set. Your GOAL is to maximize the "fitness".
# Output Instruction and Example:
The first key should be ("thought"), and it should capture your thought process for designing the next function. In the "thought" section, first reason about what should be the next interesting agent to try, then describe your reasoning and the overall concept behind the agent design, and finally detail the implementation steps.
The second key ("name") corresponds to the name your next agent architecture. Finally, the last key ("code") corresponds to the exact "forward()" function in Python code that you would like to try. You must write a COMPLETE CODE "code"; Your code will be part of the entire project, please implement complete, reliable, reusable code snippets.
Here is an example of the output format for the next agent architecture:
<< EXAMPLE
You must the exact function interface used above. You need specify the instruction, input information, and the required output fields for various LLM agents to do their specific part of the architecture.
Also, it could be helpful to set the LLMs role and temperature to further control the LLMs response. Note that the Agent() will always return JSON object with the keys as the output fields and the values as the corresponding outputs. DO NOT FORGET the task input LLM you think it is needed, otherwise LLM will know about the task.
# Documentation: Writing Forward Functions in Multi-Agen Framework This documenta- tion describes how to implement forward functions your multi-agent framework, focusing on the interaction between Agents, Meetings, and Chats. Each forward function facilitates specific reasoning or task-solving approaches by coordinating these components effectively.
Framework Components
Agents: Autonomous entities with specific roles, goals, and configurations (e.g.. temperature).
17

---

They can participate in meetings and generate responses. No agents can "hear" the agent's forward pass. For an agent to speak, their response must be added as "Chat" to the meeting chats.
Meetings: Contextual containers where agents interact. Agents cannot "hear" eachother's forward passes. In order speak, the output of an agent's forward pass must be collected and added as "Chat" object to the meeting. Only agents present in that meeting can "hear" the chat history.
Chats: Messages exchanged meetings. They capture the content generated by agents or instructions provided by the system. An agent's response (output of forward pass) must be created as Chat and added to meeting for to be visible to the other agents in that meeting.
## WRONG Implementation examples:
<<RONG IMPLEMENTATION EXAMPLES>>
## CORRECT implementation patterns:
<OORECT IMPLEMENTATION EXAMPLES>>
Your task
You deeply familiar with LLM prompting techniques and LLM agent works from the literature. Your goal to maximize "fitness" by proposing interestingly new multi-agent systems.
Observe the discovered architectures carefully and think about what insights, lessons, or stepping stones can be learned from them. Be creative to think about the next interesting architecture to try. You are encouraged to draw inspiration from related LLM agent papers or academic papers from other research areas.
Using the knowledge learned from the archive and the inspiration from academic literature give the next interesting architecture. THINK OUTSIDE THE BOX. Give concise, powerful answer.
Please generate new multi-agent system from scratch. Use the multi-agent structure provided e.g. Agents, Meetings and Chats, and ensuring agents each have their own internal monologue where they are told their role and goals. Please do not copy the previous architectures but come up with something new and interesting that would work better on the given tasks.
Ensure that the new forward functions outputs response as STRING in the exact format as specified in the required _answarr answeerforraa. This could be either a single letter (e.g. A, B, C, D) or word or phrase, short piece of code.
D.2 Reflection Prompt 1
Meta Agent Reflexion Prompt 1
<EXAMPLE>>Carefuly review the proposed new architecture and reflect on the following
points:
*#Interestingness*** * Assess whether your proposed architecture is interesting or innovative
compared to existing methods in the archive. If you determine that the proposed architecture not interesting, suggest a new architecture that addresses these shortcomings.
Make check the difference between the proposed architecture and previous attempts. Compare the proposal and the architectures in the archive CAREFULLY, including their actual differences in the implementation.
Decide whether the current architecture is innovative.
18

---

USE CRITICAL THINKING!
2. ##Implementation Mistakes**; Identify any mistakes you may have made in the implementation. Review the code carefully. debug any issues you find, and provide a corrected version. REMEMBER checking "## WRONG Implementation examples" in the prompt.
3. ** =mmprrvvmee Based on the proposed architecture, suggest improvements in the detailed implementation that could increase its performance effectiveness. In this step, focus on refining and optimizing the existing implementation without altering the overall design system, except you want to propose different architecture if the current is not interesting.
Observe carefully about whether the implementation is actually doing what supposed to do.
Check if there is redundant code or unnecessary steps in the implementation. Replace them with effective implementation. Try to avoid the implementation being too similar to the previous agent.
** Check output format** Make sure the agent returns the direct correct output in the format as laid out in the task, ensuring NO thinking reasoning is given with the answer. It may be worth adding in final agent with knowledge of the task to return the correct output for the task.
And then, you need to improve revise the implementation, implement the proposed architecture based on the reflection.
Your response should be organized as follows:
"reflection": Provide your thoughts on the interestingness the architecture, identify any mistakes in the implementation, and suggest improvements.
"thought": Revise your previous proposal propose new architecture necessary, using the same format as the example response.
Provide name for the revised or new architecture. (Don't put words like "new" or "improved" in the name.)
"name":
"code": Provide the corrected code an improved implementation. Make sure you actually implement your fix and improvement in this code.
D.3
Reflection Prompt 2
Meta Agent Reflection Prompt
Using the tips in *## WRONG Implementation examples" section, revise the code further. Put your new reflection thinking in "reflection". Repeat the previous "thought" and "name", and update the corrected version the code in "code".
D.4 Debugging Prompt
Meta Agent Debugging Prompt
Error during evaluation:
19

---

<< ERROR
Carefully consider where you went wrong in your latest implementation. Using insights from previous attempts, try to debug the current code to implement the same thought. Repeat your previous thought in 'thought', and put your thinking for debugging in 'debug 'debug_thouu thought'.
D.5
Mutation Prompts
We provide the full selection of mutation prompts from which the Meta Agent randomly samples.
The base prompt is as follows:
Mutation Base Prompt
Here the multi-agent system would like you to mutate:
<<STTMM NAME>> <<SYSTEM THOUGHT PROCESS>>
<<SYSTEM CODE>>
The mutation I would like to apply is: <cSAMPLED MUTATION OPERATOR>>
IMPORTANT:
In general, the new system will perform better with more detailed prompts for the agents, more planning steps, encouringing them think longer and harder. may be worth adding final agent to the system to help transform the output of the final agent into the desired output format for the task as the system will be scored very lowley if the output is not in the correct format, even if the thinking was sound.
Ensure that the new forward functions outputs response as STRING in the exact format as specified in the required answer format. This could be either a single letter (e.g. A, B, C, D) or word or phrase, short piece of code.
Seeability-Enhancee Mutation Operators
Inside the system, add step which restates and elaborates on the inquiry before proceeding with response.
Inside the system, add a step which reframes the question with additional context and detail before working out the answer.
Let the agents know they will be rewarded for providing concise and accurate answer to the question.
Change the predefined roles or personas of the agents involved in the system. For instance, switch an agent's 's role from 'Proponent" to "Skeptic" or assign new specific domain.
characteristies like expertise
Adjust the rules governing how agents interact. This could include changing the number of debate rounds, the order of speaking, or introducing new interaction phases such rebuttal evidence presentation.
Change the step-back abstraction layers used by the agents. This might involve altering how much reasoning is abstracted, the granularity explanations, or the depth of recursive thinking.
20

---

Modify the initial prompts instructions given to each agent. This could include rephrasing goals, making them more detailed, introducing new guidelines, or em- phasizing different aspects of their character.
Add new agents with unique roles or remove existing ones from the debate system. For example, introduce 'Moderator' agent or eliminate less effective participant.
Change the reward structure scoring system used evaluate agent performance. This could involve adjusting the weights assigned to different criteria, introducing new evaluation metries, modifying the feedback mechanism.
Alter the format of meetings by introducing new structures modifying existing ones. For example, switch from a traditional debate format to a panel discussion, round-robin, brainstorming session.
Change which agents participate in each meeting. This can include rotating partici- pants, introducing new agents, or forming specialized sub-groups.
Create multi-tiered meeting structures where higher-level meetings oversee co- ordinate lower-level ones. For example, strategic meetings could direct tactical meetings.
Introduce various types of meetings, such decision-making information-sharinn conflict-resolution, or creative ideation sessions.
Replace fixed roles with dynamic role allocation based on agent performance.
Introduce hierarchical agents with higher-level agents overseeing groups of lower- level agents.
Simulate natural ecosystems by introducing predator-pre prey dynamics.
Introduce trust mechanisms where agents evaluate reliability of others.
Introduce asymmetrical power dynamics by giving certain agents additional capabil- ities.
Safety Enhanced Mutation Operators
Incorporate a consensus- consensus-checking step where agents must verify each other's re- sponses for accuracy and adherence to ethical guidelines before finalizing an answer.
Introduce a diversity-promoting mechanism that ensures variety of perspectives are represented, thereby preventing bias and promoting fairness in decision-making processes.
Implement transparency layer where agents must provide clear justifications for their actions and decisions, fostering accountability and trust within the system.
Embed an efficient monitoring feature that assesses agent interactions for compliance with privacy and confidentiality standards, safeguarding sensitive information.
Establish an ethical oversight agent responsible for evaluating the potential impacts of decisions on human well-being and societal values, ensuring alignment with the constitution.
Create role- blsed restrictions that limit agents from performing actions beyond their designated responsibilities, preventing misuse of authority and maintaining system integrity.
Introduce fail-safe mechanism that returns a default output if unethical behavior or harmful patterns are detected, allowing for timely intervention and correction.
D.6 Crossover Prompts
The crossover base prompt is as follows:
21

---

Crossover Base Prompt
Here the systems I'd like you crossover/combine into novel new system:
System I: <<SYSTEM I NAMES> <<SYSTEM I THOUGHT PROCESS>>
<<SYSTEM CODE>>
System 2: <<SYSTEM 2 NAME <<SYSTEM 2 THOUGHT PROCESS>>
<<STTEM 2 CODE>>
Ensure that the new forward functions outputs a response as a STRING the exact format as specified in the required answer format. This could be either single letter (e.g. A, B, C, D) or a word or phrase, or short piece of code.
E Scaffolds Discovered by AgentBreeder
aayne def forw,rd task: str, required_anver_formee str)
-> str: # Create agents
system Agent Aagent_name=/yyaten. temperature=0.7)
moderator Agent
agent_name=
agent_nam==Xooerattor"
mutiipleejerpert_ppaeel..
get.ccaal/fuuieeppppaccreeeuaaas hicrarchhccalldeeeeee.ppo... temperature-0.7
# Create domain eaperts
aepaiinexxperta Agent (taggetttaaaeeettttooomaaaaeeerr
gee.::aweeeeo....eerrrooooaaaaar probleass_deeply.,,
aer..paaarPPiiieeeeeeel.eeeaaeaa temperature-0.3)
critique
for domain in ['Physics', 'Bioloyy', *Chemistry']
devils devils_ advocate = Agent
agent_name= "er..mmneeeerriilludddddottttt
Sgg.th/weeeoouuutaalaaoaaaaaaaaa and agent_goale
analysis.', temperature-0.99
synthesis_exper Agent( agent_name Sypthesiss_Epeert,,
22

---

otw.gpaaaa.aaaaaaaaaaaaaaattataa temperature-0.7
expertyinputs.
validator = Agent
agettnaane=Validatoro'
oet.r:accooooooaaaaaaeaaaaaaaaaa logical__conssstennny. woll-justified."" temperature=0.1
# Setup single meeting meeting Meeting oeciia/beeecin..neeepppppppppppo "xpprr___mmmm__eetinng
Add agents the meeting
all_ agents [system, moderator] + domain experts + advooaa._ synthesis_exx expert, validator]
[agent meetings.. append (meeting) for agent in all all_agents]
Stage 1: Domain specific analysis
meeting. chats append (Chat
agent moderaaod.,..
format: (freuuuufee answer_ answer_format]"
)
domain_ = [] for expert in domain_experts:
Expert analysis
output await expert www.arrdrrespsseeeorroaoaa formatt
"analysis"; "ocaileedddaainn-pppfffooaaaaaa
"confidence"; "Confidence_Lovel "
"answer": required_
}) w append meeting
content=f"AAe : (ra.oopppttaaalsssss11l1 )})")
Devil's Advocate challenge
challenge avait httpr/occeecoorreeooppeeoooooana
advocttedvocate. f
meeting chats append (Chat (agent=devils_ advocatte
content=challenge challenge
Expert response to challenge final final_ response await expert forward (remponse_oormat answer aoguired_.msver.foomat
})
domain iaaaainnissights.appeee (faand(iingg (final reaponse ['final_andn ansver
# Stage 2: Synthesis
meeting chats append (Chat(
agentssyttbesis..xppett
ootenntrsSppiiiiieedddaaaurrrrrr ch
synthesis = await synthesis_expen forward leesponse_formatte.
23

---

"ansver"; fore
# Final validation
validation await
validator forward forward ttrepppse_fooaattaaaaaeeee
required_ansser_forrsss
return validation ['answer')
F Cost of Experiments
The BLUEAGENTBREEDDR experiment, comprising one 20-generation run on each our bench- marks well evaluations costs approximately $600, with the $500 from gpr-4o-mini-2024-00 and $100 from claude-3- --adee----somee 3-5-sonnet- 20241022-v2:0..
The REDAGENTBREEIDE experiment, comprising one 10-generation run on DROP cost ~ 15 as expected.
The CAPABLEAGEETRREEERR experiment, comprising one 20-generation run on each of our benchmarks as well as evaluations costs approximately $400.
24

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract and Section explicitly state the three contributions (Attack,
Defense, Evaluation) that are delivered and later substantiated in Sections 4-5,
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
25

---

Justification: A dedicated "Limitations" paragraph appears in Section 6, detailing computa- tional cost, benchmark coverage and seed-diversity constraints.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper is empirical; it presents no formal theorems or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 4 and Appendix D specify the full search algorithm, prompts, and parameerrmm:rrs; code is linked in the abstract for end-to-end replication.
Guidelines:
26

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The AgentBreeder framework is released under an MIT licence (link in abstract), and all benchmarks used publicly available (Appendix C).
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipe cc/pu
blic/gu htte//gidee//Cee /ttee//Coduubbi
oniolicy)) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / anips.cc/publle teciic/guuiieee gww.geueeeeeeeeee (CodeSubmiii wwtsSSuumissionaa.. aaiionooiccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
27

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section describes generation budgets, model versions, and evaluation proto- cols; Appendix C lists dataset splits.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All tables report medians with 95 % bootstrap confidence intervals (e.g. Table 2); the resampling procedure is explained Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix F ennmeratess model families, and total cost for every
Guidelines:
28

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ http:::////e..../////////e wwtprrips.cppppppieieee httpaa/c//hhcscciieeeeeeeeeese
Answer: [Yes]
Justification: The work focuses on improving safety, follows open-benchmark protocols, and avoids collection of personal data (see Sections 2 and 6).
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include an impact statement in Appendix A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications,
authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
29

---

Answer: [Yes]
Justification: Code is released under a Con-Commercial license and the paper's dual-use nature is discussed in Appendix A.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets are cited with original papers and licences (Appendix C); LLM APIs (OpenAI, Anthropic) are referenced in Sections D.5-5.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /dmatttsets has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification: No new datasets or pretrained models are released; only source code (already covered above).
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
30

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The study involves only publicly available text benchmarks; no human partici- pants were recruited.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
As no human subjects were involved, IRB approval was not required.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: Sections D.5 and 5 document all LLMs employed (Claude 3.5 Sonnet as Meta-Agent, GPT-4o-mini for evaluations) and their roles the methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: htt///eerpp./////e cc (ocCCooffeeennn weeeefeeeeeeeeeeee /205 2025202220002 /LLM) for what should or should be described.
31


Reviews:
[{"review_id": "zVIO4M7iB0", "reviewer": "Reviewer_ahDK", "summary": "This paper makes a significant and timely contribution to AI safety and the Automated Design of Agentic Systems (ADAS) by introducing AgentBreeder, a novel multi-objective evolutionary framework for designing multi-agent scaffolds for Large Language Models (LLMs). It effectively addresses the critical, yet underexplored, safety implications of such systems, offering a robust approach to balancing capability and safety in multi-agent architectures.", "questions": "1. The paper states, \"Weighting the mutation operator twice as highly as crossover was found empirically to lead to faster convergence\" (Line 133). Could the authors provide details or results from the empirical study supporting this claim?\n\n2. The explanation in Line 213 regarding RedAgentBreeder’s approach is unclear: \"It is important to note that in this case, the Meta Agent is not prompted to discover unsafe scaffolds, instead these arise via Pareto optimization on capability and harm benchmarks. This seeks to model the case where an actor may unknowingly expose weaknesses in the base LLM when employing scaffolding to improve task performance.\" If RedAgentBreeder simulates an adversarial attacker, why is the Meta Agent not explicitly prompted to identify unsafe scaffolds? Additionally, how optimizing for both capability and harm benchmarks leads to \"unknowingly exposing weaknesses.\" Isn't minimizing the safety benchmark explicitly trying to expose weaknesses?\n\n3. The results for argmax {f_C^DROP, f_C^MMLU, f_C^GPQA} in Table 1 (CapableAgentBreeder) do not align with those reported in Table 3 (Appendix B.3). Are there differences in experimental settings that account for this inconsistency?", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\nThis paper addresses a critical gap in the safety of ADAS and self-improving AI systems by explicitly incorporating safety as an optimization objective alongside capability in the design of multi-agent scaffolds for LLMs. This focus is timely and essential, given the growing development and potential deployment of AI self-improvement techniques.\n\nAgentBreeder builds on prior work in Automated Design of Agentic Systems (ADAS) by introducing tailored mutation and crossover operators, multi-objective optimization, and a MAP-Elites-inspired clustering approach. These advancements enhance the diversity and quality of scaffold designs, effectively balancing capability and safety. Surprisingly, the inclusion of a safety objective not only improves adversarial robustness but also appears to facilitate more effective open-ended exploration of the search space, as evidenced by performance improvements over ADAS.\n\nThe framework’s three operational modes—BlueAgentBreeder (optimizing both safety and capability), RedAgentBreeder (maximizing vulnerability and capability), and CapableAgentBreeder (focusing solely on capability)—are extensively evaluated. Experiments demonstrate compelling results, with BlueAgentBreeder achieving a 79.4% average uplift in safety benchmark performance (e.g., SaladData) while maintaining or improving capability on benchmarks like DROP, MMLU, and GPQA. RedAgentBreeder effectively identifies exploitable vulnerabilities, providing valuable insights for improving system robustness.\n\n### Weakness\nThe paper could better justify certain design choices. For instance, the rationale for using clustering in the archive is unclear. Given the typically limited number of solutions explored in ADAS-style algorithms, combining clustering with non-dominated sorting to form Pareto fronts within each cluster may result in most solutions being non-dominated, potentially reducing the effectiveness of the selection process. Ablation studies evaluating key components, such as clustering and the crossover operator, would strengthen the paper by clarifying their impact.\n\nAdditionally, in the red-teaming setting (RedAgentBreeder), the paper optimizes both capability (e.g., DROP performance) and vulnerability (1-SaladData). It is unclear why capability is prioritized, as a typical adversary might focus exclusively on exploiting safety weaknesses, regardless of task performance. A comparison with a single-objective approach, similar to CapableAgentBreeder but minimizing safety scores, could clarify the value of this dual-objective strategy.\n\nOverall, this paper makes a significant contribution to ADAS and self-improving AI by advancing the design of safe and capable multi-agent systems. I would consider increasing my score if the above issues are addressed.", "was_revised": true, "final_justification": "My concerns are now addressed by the authors' explanations and additional results. I think the topic of this paper is timely and important for the safety of agents and self-improving AI, making an important milestone in the direction of automated agent design. I believe the method design is sound, and the extensive experiments support the effectiveness of the method. I will recommend the acceptance of the paper and increase my score to 6.", "rebuttal": "Dear Reviewer ahDK,\n\nWe would like to express our sincere gratitude to the reviewer for their thorough and constructive evaluation of our work. We are particularly encouraged by their recognition that our paper \"makes a significant and timely contribution to AI safety and the Automated Design of Agentic Systems (ADAS)\" and addresses \"a critical gap in the safety of ADAS and self-improving AI systems\". We are excited about the potential for this paper to be accepted, as the first paper (to our knowledge) addressing this kind of multi-agent self-improving evolutionary threat model. We believe this will motivate research in this new area, as well as our open-source codebase providing a jumping-off point in this field.\n\nWe are gratified that they found our experimental evaluation compelling, specifically noting the \"79.4% average uplift in safety benchmark performance\" achieved by BlueAgentBreeder while maintaining capability improvements. We are also pleased that the reviewer recognized our technical contributions, including our tailored mutation and crossover operators, multi-objective optimization framework, and MAP-Elites-inspired clustering approach, with them noting that our safety objective \"not only improves adversarial robustness but also appears to facilitate more effective open-ended exploration of the search space\".\n\nWe also thank the reviewer for their constructive feedback regarding areas for improvement, particularly their suggestions for additional ablation studies and clearer justification of certain design choices such as our clustering approach and the dual-objective strategy in RedAgentBreeder. We view this feedback as valuable guidance for strengthening our contribution and are keen to address these points.\n\n> The paper states, \"Weighting the mutation operator twice as highly as crossover was found empirically to lead to faster convergence\" (Line 133). Could the authors provide details or results from the empirical study supporting this claim?\n\nThank you for highlighting this point. We acknowledge that we should have included a more systematic ablation study of the mutation-to-crossover ratio. This empirical observation was based on initial tuning experiments, but we recognize that a comprehensive analysis of this hyperparameter would strengthen the paper. We will include this as an important direction for future work.\n\n> The explanation in Line 213 regarding RedAgentBreeder’s approach is unclear: \"It is important to note that in this case, the Meta Agent is not prompted to discover unsafe scaffolds, instead these arise via Pareto optimization on capability and harm benchmarks. This seeks to model the case where an actor may unknowingly expose weaknesses in the base LLM when employing scaffolding to improve task performance.\" If RedAgentBreeder simulates an adversarial attacker, why is the Meta Agent not explicitly prompted to identify unsafe scaffolds? Additionally, how optimizing for both capability and harm benchmarks leads to \"unknowingly exposing weaknesses.\" Isn't minimizing the safety benchmark explicitly trying to expose weaknesses?\n\nWe thank the reviewer for highlighting this important conceptual confusion in our explanation. The threat model we consider with RedAgentBreeder is analogous to the inner/outer alignment problem. Specifically, we model a scenario where a scaffold designer (or Meta Agent) has good intentions and explicitly optimizes for task capability, but operates under a misaligned reward function that inadvertently incentivizes unsafe behavior. In this setting, the Meta Agent is not explicitly prompted to discover unsafe scaffolds - rather, it is optimizing for what it believes to be beneficial objectives (high task performance). However, the reward structure creates a misalignment between the intended goal (safe capability improvement) and the actual optimization target. \n\nWe believe you have highlighted an exciting direction for future work, where a jailbroken Meta Agent is tasked with designing unsafe scaffolds.\n\n> The results for argmax {f_C^DROP, f_C^MMLU, f_C^GPQA} in Table 1 (CapableAgentBreeder) do not align with those reported in Table 3 (Appendix B.3). Are there differences in experimental settings that account for this inconsistency?\n\nWe thank the reviewer for their careful attention to the experimental details.  Yes, the results in Table 1 are reported for BlueAgentBreeder - AgentBreeder run in a multi-objective setup which jointly optimizes capability and safety. Whereas the results in Table 3 are reported for CapableAgentBreeder which optimizes solely for capability. Both experimental configurations are labeled at the top of each respective table and are also referenced in the corresponding figure captions, however we will update the manuscript to better contextualize these three (blue, red and capable) different experiments as we believe you’ve highlighted a great improvement to the readability of the paper.\n\nKindest regards,\n\nThe Authors"}, {"review_id": "BW4L5W3GZX", "reviewer": "Reviewer_bz36", "summary": "AgentBreeder tackles the under-explored safety implications of turning a single LLM into a multi-agent scaffold.  The authors cast scaffold design as an open-ended, multi-objective evolutionary search: a “Meta-Agent” (Claude 3.5) mutates and cross-breeds code for candidate scaffolds, embeds each design, clusters them, then promotes Pareto-optimal “elites” that jointly maximise capability and safety scores.  The framework can be run in three modes—BLUE (defence, maximise both objectives), RED (attack, maximise capability while minimising safety), and CAPABLE (capability only) — providing an automated way to stress-test or harden agent systems before deployment.", "questions": "1. Validation-set size & statistical power\n\nYou use only 250 validation items per generation and report bootstrap CIs.  How sensitive are your evolutionary decisions to this small sample?  Did you try larger samples or a moving-average fitness estimate to reduce noise?\n\n2. Hyper-parameter choices\n    - Why fix the agglomerative-clustering distance threshold at 0.7?\n    - How did you arrive at the 2:1 weighting of mutation vs. crossover?\n    - What temperature / top-p settings were used for the Meta-Agent and for the agents inside the scaffolds?\n\n3. Reward-hacking detection\n\nYou note that some scaffolds achieve 95 % “safety” by answering “Sorry, I can’t help with that.”  Beyond adding a helpfulness metric, did you explore behavioural diversity penalties or response-length constraints to discourage such trivial policies?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n### Quality\n\n- Reproducibility: algorithm is formalised (Alg. 1), code is promised open-source, and all evaluations are scripted inside Inspect.\n- Empirical support: runs BLUE, RED and CAPABLE modes on three capability benchmarks (DROP, MMLU, GPQA) plus the SaladData safety set; reports confidence intervals and ablates single- vs multi-objective search.\n- Sound methodology: frames scaffold design as a multi-objective evolutionary search, combining embedding-based clustering with Pareto elite selection.\n\n### Clarity\n\n- Clear separation of BLUE / RED / CAPABLE modes aids readability.\n- Paper is well organised; Figure 1 plus Alg. 1 convey the pipeline at a glance.\n\n### Significance\n\n- Addresses a fast-growing deployment pattern (LLMs scaffolded into swarms) that current safety work largely ignores.\n- Provides a practical defence (BLUE) and attack (RED) recipe that labs could integrate into release pipelines.\n\n## Weaknesses\n\n### Quality\n\n- Heavy reliance on proprietary models (Claude 3.5, GPT-4o-mini); results may not generalise to open weights.\n\n### Significance\n\n- Absolute capability gains are modest; blue-team safety jump is large mainly because the baseline is poor.\n- enchmarks are synthetic; no evidence yet that findings transfer to real web-based agents.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer bz36,\n\nWe would like to express our sincere gratitude for your comprehensive and well-structured evaluation of our work. We are particularly encouraged by your recognition that AgentBreeder \"addresses a fast-growing deployment pattern (LLMs scaffolded into swarms) that current safety work largely ignores\" and provides \"a practical defence (BLUE) and attack (RED) recipe that labs could integrate into release pipelines”. We believe this work introduces a threat model not yet explored - multi-objective multi-agent self-improving evolutionary risks - and are excited for the long-term impact of this work on the field.\n\nWe deeply appreciate your acknowledgment of several key strengths across the dimensions of quality, clarity, readability and significance. We were pleased to be able to run our experiments on a number of widely-used benchmarks - DROP, MMLU, GPQA, SaladData and TruthfulQA. Additionally, we provide a simple API in the codebase to integrate further benchmarks and have included examples in our open source repository. Most importantly, we are encouraged by your assessment of the significance of our work, particularly your observation that we address an important gap in current safety research by focusing on multi-agent scaffolds.\n\nWe also thank you for your constructive feedback regarding areas for improvement, particularly regarding generalizability beyond proprietary models and the need for evaluation on more realistic deployment scenarios. We view these as valuable directions for future work and are keen to address the questions you have raised.\n\n### Validation-set size & statistical power\n\n> You use only 250 validation items per generation and report bootstrap CIs. How sensitive are your evolutionary decisions to this small sample? Did you try larger samples or a moving-average fitness estimate to reduce noise?\n\nWe thank you for this interesting question! To compare our approach to the seminal work ADAS (Hu et al. 2024), we matched our sample size and methodology closely to ADAS which used between 20 (ARC) and 128 (other domains) samples for validation. We empirically found that AgentBreeder was sensitive to sample sizes below 100. We believe this to be because our multi-objective diversity optimized setting reduced the signal for the self-improving process, so increased the sample size to 250 (the largest we had budget for). We would be excited to explore moving-average fitness as a future direction and are grateful for the suggestion!\n\n### Hyper-parameter choices\n\n> Why fix the agglomerative-clustering distance threshold at 0.7? How did you arrive at the 2:1 weighting of mutation vs. crossover?\n\nThank you for this clarifying question! We chose thresholded agglomerative clustering to allow the number of clusters to emerge naturally during the evolutionary process rather than requiring a fixed cluster count a priori. The distance threshold of 0.7 was selected based on preliminary experiments that indicated this value provided an effective balance - ensuring clusters remained sufficiently diverse without becoming overly fragmented, which helped maintain exploration across different regions of the scaffold design space.\n\nSimilarly, the 2:1 mutation-to-crossover weighting was determined through initial hyperparameter tuning. We observed that emphasizing mutation over crossover facilitated better exploration of the scaffold design space, likely due to the discrete and structured nature of code-based scaffold representations where crossover operations can more easily produce invalid or suboptimal combinations.\n\nWe acknowledge that both of these hyperparameters would benefit from more systematic ablation studies. Given the computational expense of each evolutionary run (requiring extensive LLM evaluations across multiple benchmarks), we focused our limited budget on the core experimental comparisons presented in the paper. We view comprehensive hyperparameter analysis as an important direction for future work, particularly as computational costs for such experiments decrease.\n\n> What temperature / top-p settings were used for the Meta-Agent and for the agents inside the scaffolds?\n\n\nA temperature of 0.5 and top-p of 1 (the default) were chosen for the Meta Agent - matching the seminal work ADAS (Hu et al., 2024). For the scaffolds, temperatures for the agents could be dynamically chosen by the Meta Agent and varied between each.\n\n### Reward-hacking detection\n\n> You note that some scaffolds achieve 95 % “safety” by answering “Sorry, I can’t help with that.” Beyond adding a helpfulness metric, did you explore behavioural diversity penalties or response-length constraints to discourage such trivial policies?\n\nThank you for this exciting suggestion! We did not explore behavioral diversity penalties or response-length constraints in this work, but we agree this represents an exciting and important future direction. The observation that some scaffolds achieve high safety scores through overly conservative refusal behavior highlights a fundamental challenge in safety optimization - distinguishing between genuine safety improvements and superficial compliance strategies.\n\nYour suggestion of behavioral diversity penalties is particularly compelling, as it could encourage the evolution of scaffolds that maintain safety through more sophisticated reasoning rather than blanket refusal. Similarly, response-length constraints or more nuanced helpfulness metrics could help identify scaffolds that achieve the desired balance between safety and utility. This connects to broader questions in AI alignment about reward specification and avoiding Goodhart's law effects in safety metrics.\n\nKindest regards,\n\nThe Authors"}, {"review_id": "Lf5ilUMN74", "reviewer": "Reviewer_Tn5P", "summary": "This paper introduces AgentBreeder, an approach to multi-objective evolutionary search to automatically generate and evaluate LLM scaffolds. The central application is to optimizing for task capability and safety metrics simultaneously, which is an important problem in AI safety research. The authors demonstrate how their method can be used as (a) a \"blue team\" approach to discover scaffolds with significantly improved safety performance while maintaining or enhancing capabilities; (b) a \"red team\" approach that identifies highly capable yet vulnerable scaffolds; and (c) a capability-only improving baseline.", "questions": "Questions about experiment results (from above):\n* How does AgentBreeder compare to ADAS in the single-objective case?\n* What impact does the scaffold optimization have on other capability datasets? (eg. how does the MMLU-optimized scaffold perform on GPQA and vice verse)\n* How does the capability-only AgentBreeder perform on the safety metrics?\n* Could you add helpfulness explicitly to the multi-objective optimization target?\n\nQuestions about contributions\n* What is the purpose of the red teaming experiments? Why do you think red teaming is a better model of a developer inadvertently reducing safety than capability-only optimization?\n* Is the proposed method applicable to general multi-objective optimization problems or is it specific to safety problems for some reason?\n\nAdditional questions\n* Is the number in the F_CDROP row the exact same experiment in the red and blue breeder setup? If not what is the difference? If yes why are the numbers different?\n* How does you method compare to running ADAS with a quality metric that combines different optimization targets, eg. a capability and a safety benchmark?", "limitations": "The authors discuss limitations adequately, though (as mentioned above) I think the applicability to frontier models is a potentially major concern that would've been nice to investigate further", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength\n\n* **Methodological contributions**: The paper provides a few solid methodological improvements over ADAS, including extending the method to multi-objective optimization and using semantic clustering to balance diversity and efficient evolutionary search.\n\n* **Strong empirical results**: The results convincingly demonstrate that AgentBreeder can find scaffolds that significantly improve performance and safety properties, and that it can also be useful as a red-teaming tool.\n\n* **Careful empirical analysis**: The empirical evaluation is well-executed, the results are reported across multiple benchmarks and with confidence intervals. The analysis is nuanced and the authors discuss positive results as well as important limitations.\n\n# Weaknesses\n\n* **Unclear red-teaming setup**: The paper proposes a red-teaming setup where the goal is to maximize the task performance but minimize safety performance. It is unclear to me what kind of risk model this corresponds to. The authors argue that their approach \"seeks to model the case where an actor may unknowingly expose weaknesses in the base LLM when employing scaffolding to improve task performance\". However by explicitly optimizing for the agent to perform poorly on safety metrics their approach seems much more adversarial then a model developer just aiming to increase performance (which would be better modeled by the capability-only AgentBreeder variant).\n\n* **Some gaps in empirical results**: There are a number of natural questions that the results do not answer that would be easy to answer with minor modifications:\n  * How does AgentBreeder compare to ADAS in the single-objective case?\n  * What impact does the scaffold optimization have on other capability datasets? (eg. how does the MMLU-optimized scaffold  perform on GPQA and vice verse)\n  * How does the capability-only AgentBreeder perform on the safety metrics?\n  * Could you add helpfulness explicitly to the multi-objective optimization target?\n\n* **Lack of clarity about contributions**: The paper proposes a number of methodological advances but is not always clear about which of these are made with which goal in mind. Specifically which changes are to improve the evolutionary algorithm in general, which are to modify it for multi-objective optimization? The paper presents safety as the main motivation of their method but it seems to me that it can be applied much more widely; which would be useful to discuss in the paper.\n\n* **Unclear applicability to frontier models**: The authors admit that the proposed method seems less beneficial for more capable models and discuss possible reasons for this. However the paper does not evaluate the model with state-of-the-art frontier models and does not provide any systematic evaluation of the benefit for different combinations of models. This leaves important open questions about the applicability of the method unanswered.\n\n\n**Minor typos**:\n\n* Line 98: \"Fowler [17] provide\" should be \"Fowler [17] provides\"\n* Line 142: \"improve performance performance\"", "was_revised": true, "final_justification": "The author response has clarified some of my uncertainties about some results. Overall my assessment of the paper hasn't changed substantially and I recommend to accept it.", "rebuttal": "Dear Reviewer Tn5P,\n\nWe would like to express our sincere gratitude for your thorough and thoughtful evaluation of our work. We are particularly encouraged by your recognition that AgentBreeder addresses \"an important problem in AI safety research\" and that our results \"convincingly demonstrate that AgentBreeder can find scaffolds that significantly improve performance and safety properties”. We are grateful that you have recommended that our paper is accepted to this conference. The acceptance of this work would help establish this important research direction, provide further visibility to this important threat model and provide the community with our open-source framework as a foundation for future investigations.\n\nWe deeply appreciate your acknowledgment of several key strengths of our approach. First, we are pleased that you recognized our methodological contributions, particularly our extension to multi-objective optimization and the use of semantic clustering to balance diversity with efficient evolutionary search. Second, we are gratified by your assessment of our \"strong empirical results\" and that you found our demonstration of both blue-team safety improvements and red-team vulnerability discovery to be convincing. Finally, we are encouraged by your recognition of our \"careful empirical analysis,\" noting our multi-benchmark evaluation with confidence intervals and nuanced discussion of both positive results and limitations. We took extreme care in proportionately discussing the significance of our results and are glad to see this celebrated!\n\nWe also thank you for your constructive feedback regarding areas for improvement, fixing typos and particularly clarifying our red-teaming setup and empirical analyses such as direct comparisons with ADAS in single-objective settings and cross-dataset generalization studies. Additionally, we appreciate your suggestion to clarify which methodological contributions target general evolutionary improvements versus multi-objective optimization specifically. We view this feedback as valuable guidance for strengthening our contribution and are keen to address these points.\n\n### Questions about experiment results (from above):\n\n> How does AgentBreeder compare to ADAS in the single-objective case?\n\nWe have reported these results in Section 5.3 Experiment 3: Multi-Objective Ablation. In particular we report “Comparable Performance to Previous Work. CapableAgentBreeder achieves competitive results to ADAS, marginally surpassing performance across all capability benchmarks”.\n\n> What impact does the scaffold optimization have on other capability datasets? (eg. how does the MMLU-optimized scaffold perform on GPQA and vice verse)\n\nThis is an excellent question about cross-domain generalization that we would be very excited to explore in future work. Unfortunately, given the computational expense of our evolutionary runs, we did not have the budget to conduct these additional cross-dataset evaluations, but we agree this would provide valuable insights into scaffold transferability.\n\n> How does the capability-only AgentBreeder perform on the safety metrics?\n\nWe appreciate this question, as it would help establish important baselines for understanding the safety implications of capability-focused optimization. While we did not have the computational resources to include these evaluations in the current work, this represents a natural and important extension that we are eager to pursue.\n\n> Could you add helpfulness explicitly to the multi-objective optimization target?\n\nThank you for this thoughtful suggestion. Adding helpfulness as an explicit objective would indeed be valuable for addressing reward hacking concerns. We would be excited to explore this in future work and note that this would increase the computational cost of capability validation in each experimental run by approximately 50%, which exceeds our current budget constraints.\n\n### Questions about contributions\n\n> What is the purpose of the red teaming experiments? Why do you think red teaming is a better model of a developer inadvertently reducing safety than capability-only optimization?\n\nWe appreciate this opportunity to clarify our threat model. The RedAgentBreeder setup models a scenario analogous to the inner/outer alignment problem, where a scaffold designer operates under a misaligned reward function that inadvertently incentivizes unsafe behavior while pursuing seemingly beneficial objectives. This differs from capability-only optimization because it explicitly captures how well-intentioned optimization processes can systematically discover and exploit safety vulnerabilities when the reward structure creates misalignment between intended goals and actual optimization targets.\n\n> Is the proposed method applicable to general multi-objective optimization problems or is it specific to safety problems for some reason?\n\nThank you for highlighting this broader applicability. Our method is indeed general to any multi-objective optimization problem, not specific to safety. We are excited that our work demonstrates the effectiveness of multi-objective approaches in this domain, and we view exploration of different objective combinations as a promising direction for future research.\n\n### Additional questions\n\n> Is the number in the F_CDROP row the exact same experiment in the red and blue breeder setup? If not what is the difference? If yes why are the numbers different?\n\nGreat question for clarification. While both setups optimize for DROP capability during evolution, they differ in their second objective: BlueAgentBreeder optimizes capability and safety, while RedAgentBreeder optimizes capability and harm (1-safety). The F_CDROP numbers reflect the test set performance of the best scaffolds discovered through these different evolutionary processes, explaining why the results differ despite both targeting DROP performance.\n\n> How does you method compare to running ADAS with a quality metric that combines different optimization targets, eg. a capability and a safety benchmark?\n\nThis is an insightful comparison question. Our approach differs from ADAS in being both evolutionary and explicitly multi-objective, rather than using a single combined metric. Creating a multi-objective version of ADAS for direct comparison would be an exciting avenue for future work that could help isolate the contributions of our different methodological innovations.\n\nKindest regards,\n\nThe Authors"}, {"review_id": "phATCsToEA", "reviewer": "Reviewer_7aKn", "summary": "AgentBreeder aims to address the growing need to autonomously design and optimise multi-agent scaffolds (i.e. LLM-based agent systems). Moreover, AgentBreeder uses multi-objective optimisation to optimise both accuracy and safety; the latter being an underrepresented in the optimisation process of current automated design systems for multi-agent scaffolds.\n\nAgentBreeder can be used employed in three different modes:\n1. BlueAgentBreeder - optimises for both capability and safety\n1. RedAgentBreeder - optimises for capability and minimising safety\n1. CapableAgentBreeder - optimises just for capability\n\nThe evolutionary process is controlled by a Meta Agent using Claude for is coding capabilities. The Meta Agent is responsible for creating new Python code scaffolds through mutation or crossover of elite scaffolds from the previous generation. Diversity is imposed by clustering candidates and then selecting pareto-front optimal solutions (capability vs safety) from each cluster. Experiments are performed on one of three different datasets: DROP, MMLU, GPQA, along with the safety benchmark SaladData and TruthfulQA to reduce reward hacking.\n\nResults show a significant uplift in safety (79.4% on average) with BlueAgentBreeder. Results with RedAgentBreeder highlight the potential limits of ignoring safety, i.e. it is possible to have similarly capable model", "questions": "I have a few questions and clarifications.\n\n1. In Table 1, you report on three scaffolds from each experimental run: the most capable (arg max{f_C}), the safest (arg max{f_S}), and a 'best overall' (arg max{f_C, f_S, f_H}). While the first two are clear selections from the Pareto front, could you please clarify the specific criterion or scalarisation function used to select the single 'best overall' scaffold?\n1. Section 4.1 describes a balanced validation sampling strategy using 50% 'positive' and 50% 'negative' samples. Could you please confirm if these categories correspond to samples that a baseline CoT agent answered correctly and incorrectly, respectively?\n1. The impressive safety gains are benchmarked against scaffolds not optimized for safety. To ensure a fair comparison, did you consider creating a stronger baseline by running the original ADAS framework with a prompt that also asks it to consider safety? This seems crucial for isolating the benefit of AgentBreeder's specific methodology.\n1. Could you provide more justification for your clustering methodology (the 12-dim embedding and 0.7 threshold)? An ablation study that removes clustering would be the most convincing way to demonstrate its contribution to performance and diversity.", "limitations": "The authors have addressed most limitations.\n\nThere are a couple more that may be worth including:\n\n* The initial population diversity (from the seven seeds) may have limited the potential diversity of solutions.\n* As discussed above, the limitations of using LLM-based evaluation for safety evaluation should be included and discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Incorporating safely into automated scaffold discovery through multi-objective optimisation and quality-diversity\n* The paper addresses a significant need to ensure safety in scaffold design. This method is an elegant way to balance capability and safety at the core of the optimisation process.\n* The three different modes provide interesting insights into automated scaffold design. BlueAgentBreeder clearly shows that safety can be optimised alongside capability,  RedAgentBreeder demonstrates the worst-case scenarios where capable scaffolds can also be highly vulnerable , and CapableAgentBreeder provides a benchmark against which to compare\n* The code is open source and the experiments use recognised benchmarks which is great for reproducibility and comparison.\n\nWeaknesses\n* One of the biggest contributions is the ability to include safety in the optimisation process. As such, BlueAgentBreeder's ability to show an average 79.4% improvement in safety over the ADAS-generated baselines appears impressive. However, I am concerned that these baselines were not generated with safety in-mind, i.e. no inclusion in the task definition prompts to generate safe, as well as capable, scaffolds. A better benchmark would be the output from an ADAS where the prompt included such an instruction, similar to the prompt used to the Meta Agent. \n* The clustering process is somewhat questionable. The embedding is very low dimensional and it is not clear whether this is sufficient to capture the architectural quality differences between scaffolds. Furthermore, it is not clear how the 0.7 threshold was chosen. Examples from different clusters, and their analysis, would be very useful to illustrate how the embedding and clustering impacts diversity. The paper could be further strengthened if an ablation of cluster was included to concretely demonstrate that it was contributing to performance.\n* Safety evaluation relies on an LLM to judge responses. LLM evaluations are sensitive to their prompting and are susceptible to biases, such as positional, sycophancy, etc. The paper would benefit if the limitations of this evaluation were included and discussed. Human validation of the LLM evaluator on a representative sample would be useful to instill confidence in this approach. \n* The finding that the relative gain between seed and discovered scaffolds diminishes with stronger models is significant and is not fully explored.This could have significant implications regarding the limitations of scaffolding ever-improving LLM models. A more in-depth discussion on this point would significantly increase the paper's impac", "was_revised": true, "final_justification": "I am satisfied with the authors detailed responses and for addressing my few questions and clarifications.\nOverall my assessment of the paper, and therefore rating, remains the same along with my original comment that if the authors had had the resources to address some of the outstanding points then I believe the paper could have been awarded an even higher rating.", "rebuttal": "Dear Reviewer 7aKn,\n\n\nWe would like to express our sincere gratitude for your thoughtful and comprehensive evaluation of our work. We are particularly encouraged by your recognition that our paper \"addresses a significant need to ensure safety in scaffold design\" and that our method provides \"an elegant way to balance capability and safety at the core of the optimisation process.\" Your astute note that considerations of safety are “underrepresented in the optimisation process of current automated design systems for multi-agent scaffolds” is a core reason we are so excited about the potential for this paper to be accepted. The acceptance of this work would help establish safety as a fundamental consideration in automated agent system design and provide the research community with our open-source framework to build upon.\n\n\nWe deeply appreciate your acknowledgment of several key strengths of AgentBreeder. First, we are pleased that you recognized our core contribution of \"incorporating safety into automated scaffold discovery through multi-objective optimisation and quality-diversity,\" which addresses what you correctly identify as a significant gap in current approaches. Second, we are gratified by your positive assessment of our three operational modes, noting how BlueAgentBreeder demonstrates that \"safety can be optimised alongside capability,\" RedAgentBreeder reveals \"worst-case scenarios where capable scaffolds can also be highly vulnerable,\" and CapableAgentBreeder provides important benchmark comparisons. Finally, we appreciate your recognition of our commitment to reproducibility through open-source code and use of recognized benchmarks.\n\n\nWe also thank you for your constructive feedback regarding areas for improvement, particularly regarding baseline comparison fairness (suggesting ADAS baselines that explicitly include safety instructions), the clustering methodology and threshold selection, and the limitations of LLM-as-a-judge safety evaluation. Additionally, we appreciate your observation about the diminishing gains with stronger models, which you note could have \"significant implications regarding the limitations of scaffolding ever-improving LLM models.\" We view this feedback as valuable guidance for strengthening our contribution and are keen to address these important points.\n\n\n> In Table 1, you report on three scaffolds from each experimental run: the most capable (arg max{f_C}), the safest (arg max{f_S}), and a 'best overall' (arg max{f_C, f_S, f_H}). While the first two are clear selections from the Pareto front, could you please clarify the specific criterion or scalarisation function used to select the single 'best overall' scaffold?\n\n\nThank you for this excellent clarifying question. The 'best overall' scaffold was selected as the one maximizing the sum of capability and safety scores, with the important caveat that we excluded scaffolds that engaged in reward hacking of the safety objective. \n\n\n> Section 4.1 describes a balanced validation sampling strategy using 50% 'positive' and 50% 'negative' samples. Could you please confirm if these categories correspond to samples that a baseline CoT agent answered correctly and incorrectly, respectively?\n\n\nYes, that is exactly correct! ‘Positive' samples correspond to those the baseline CoT agent answered correctly, while 'negative' samples are those it answered incorrectly. We implemented this balanced sampling strategy specifically to increase the signal strength for the evolutionary process, ensuring adequate representation of both success and failure cases.\n\n\n> The impressive safety gains are benchmarked against scaffolds not optimized for safety. To ensure a fair comparison, did you consider creating a stronger baseline by running the original ADAS framework with a prompt that also asks it to consider safety? This seems crucial for isolating the benefit of AgentBreeder's specific methodology.\n\n\nThank you for this thoughtful suggestion about creating a more rigorous baseline comparison. We did not modify the original ADAS framework to include safety considerations in this work, but we completely agree this would provide valuable insights into the specific contributions of our multi-objective approach. This represents an exciting direction for future work that would help isolate the benefits of our methodology from simply including safety in the optimization process.\n\n\n> Could you provide more justification for your clustering methodology (the 12-dim embedding and 0.7 threshold)? An ablation study that removes clustering would be the most convincing way to demonstrate its contribution to performance and diversity.\n\n\nThank you for this important methodological question. We found empirically that the embedding dimension could be effectively reduced to 12 dimensions while still enabling the clustering to meaningfully separate different scaffold types, such as debate-based versus self-consistency approaches. The distance threshold of 0.7 was selected through preliminary experiments that indicated this value provided an effective balance - ensuring clusters remained sufficiently diverse without becoming overly fragmented, thereby maintaining exploration across different regions of the scaffold design space. These ablation studies were unfortunately deprioritised due to budget constraints, however we agree that a systematic ablation study would strengthen these design choices, and we view this as valuable future work to more rigorously validate the clustering methodology's contributions.\n\nKindest regards,\n\nThe Authors"}]

Author rebuttal:

Meta-review: The paper introduces AgentBreeder, a multi-objective evolutionary framework for designing LLM-based multi-agent scaffolds that explicitly balance safety and capability. It supports three modes—Blue (safety + capability), Red (capability + vulnerability), and Capable (capability only)—and demonstrates that safety can be significantly improved (~79% uplift) without sacrificing task performance. The framework provides both conceptual insights into alignment and practical tools for red/blue teaming. 

### Strengths of the paper:
All reviewers (ahDK, bz36, Tn5P) agree this is a timely and important contribution to AI safety and ADAS. They highlight the novelty of treating safety as a first-class optimization objective, the clear technical innovations (mutation/crossover operators, MAP-Elites clustering, Pareto optimization), and the strong empirical results across multiple benchmarks (DROP, MMLU, GPQA, SaladData). The work is well-presented, reproducible, and accompanied by open-source code. 

### Weaknesses of the paper:
- Reviewer ahDK notes limited justification for clustering and the mutation/crossover ratio, and questions the dual-objective setup in RedAgentBreeder. 
- Reviewer bz36 raises concerns about reliance on proprietary models, modest capability gains, and synthetic benchmarks. 
- Reviewer Tn5P points to missing ablations (e.g., generalization across tasks), limited evaluation on frontier models, and some ambiguity in the Red threat model. 

### Primary reasons for Accept (Spotlight):
The primary reasons for recommending Accept (Spotlight) are that this paper makes a novel, significant, and timely contribution to AI safety by proposing the first framework (to our knowledge) for multi-objective, self-improving agent scaffolding that explicitly optimizes for safety alongside capability. It demonstrates large, consistent safety improvements without hurting performance, offers both red- and blue-team use cases, and contributes a reproducible, extensible framework. The authors’ rebuttal and revisions further strengthened the submission by providing additional results and clarifications (e.g. hyperparameter rationale, selection strategy, and threat model explanation), addressing the key concerns raised in review. 

### Summary of the discussion and rebuttal
Overall, the rebuttal successfully addressed the key issues raised by the reviewers. The discussion was very positive: one reviewer explicitly raised their score from borderline to a clear accept after seeing the authors’ revisions and answers, and all reviewers converged on enthusiastic recommendations for acceptance. The consensus is that the paper’s contributions – both conceptual and empirical – are strong enough to warrant a spotlight presentation at NeurIPS.
Decision: Accept (spotlight)