Automated Design of Agentic Systems
Authors: Shengran Hu
Cong Lu
Jeff Clune
Venue: ICLR 2025 Poster (2025)
Keywords: LLMs
Language Model Agents
Agents
Agentic Systems
Reasoning
Meta Learning
Open-endedness
Primary area: foundation or frontier models, including LLMs
TL;DR: We describe a newly forming research area which aims to automatically create powerful agentic system designs and present an algorithm to demonstrate that agents can automatically designing themselves by programming in code.

Abstract: Researchers are investing substantial effort in developing powerful general-purpose agents, wherein Foundation Models are used as modules within agentic systems (e.g. Chain-of-Thought, Self-Reflection, Toolformer). However, the history of machine learning teaches us that hand-designed solutions are eventually replaced by learned solutions. We describe a newly forming research area, Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways. We further demonstrate that there is an unexplored yet promising approach within ADAS where agents can be defined in code and new agents can be automatically discovered by a meta agent programming ever better ones in code. Given that programming languages are Turing Complete, this approach theoretically enables the learning of any possible agentic system: including novel prompts, tool use, workflows, and combinations thereof. We present a simple yet effective algorithm named Meta Agent Search to demonstrate this idea, where a meta agent iteratively programs interesting new agents based on an ever-growing archive of previous discoveries. Through extensive experiments across multiple domains including coding, science, and math, we show that our algorithm can progressively invent agents with novel designs that greatly outperform state-of-the-art hand-designed agents. Importantly, we consistently observe the surprising result that agents invented by Meta Agent Search maintain superior performance even when transferred across domains and models, demonstrating their robustness and generality. Provided we develop it safely, our work illustrates the potential of an exciting new research direction toward automatically designing ever-more powerful agentic systems to benefit humanity.

Published as a conference paper at ICLR 2025
AUTOMATED DESIGN OF AGENTIC SYSTEMS
Shengran Hul Cong , Jeff Clune 'University of British Columbia, VVector Institute, 'Canada CIFAR AI Chair (srhu, congl ubc. ca, jel une gmail 1. com
ABSTRACT
Researchers are investing substantial effort in developing powerful general- purpose agents, wherein Foundation Models are used modules within agen- tic systems (e.g. Chain-of-Thought, Self-Reflection, Toolformer). However, the history of machine learning teaches us that hand-designed solutions are eventu- ally replaced by learned solutions. We describe a newly forming research area, Automated Design of Agentic Systems (ADAS), which aims to automatically cre- ale powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways. We further demonstrate that there is an un- explored yet promising approach within ADAS where agents can be defined in code and new agents can be automatically discovered by a meta agent program- ming ever better ones in code. Given that most programming languages Complete, this approach theoretically enables the learning of any possible agentic system: including novel prompts, tool use, workflows, and combinations thereof. We present a simple yet effective algorithm named Meta Agent Search to demon- strate this idea, where a meta agent iteratively programs interesting new agents based on an ever wver-growing archive of previous discoveries. Through extensive ex- periments across multiple domains including coding, science, and math, we show that our algorithm can progressively invent agents with novel designs that greatly outperform thrte-r hand land-designed agents. Importantly, we consistently observe the surprising result that agents invented by Meta Agent Search maintain superior performance even when transferred across domains and models, demon- strating their robustness and generality. Provided we develop it safely, our work illustrates the potential of an exciting new research direction toward automatically designing ever-more powerful agentic systems benefit humanity. All code is open-sourced at https://tthabccom/Seeggaanalaal
Turing
INTRODUCTION
Foundation Models (FMs) such GPT (OpenAl 2024
2022) and Claude (Anthropic 2024b)
quickly being adopted as powerful general- agents for agentic tasks that need flexible rea- soning and planning (Wang al. 2024 Despite recent advancements in FMs, solving problems liably often requires an agent to be a compound agentic system with multiple components instead of monolithic model query (Zaharia al. 2024 Rocktäschel] (024). Additionally, to enable agents to solve complex real-world tasks, they often need access to external tools such as search engines, code execution, and database queries. As a result, many effective building blocks of agentic systems have
been proposed, such as chain-of- -f-thuuhttt planning and reasoning (Wei et al. 2022 Yao et al. 2023
Hu & Clune 2024), memory structures (Zhang et al., 2024c| Lewis et al. 2020)
tool use (Schick et al. 2023 Qu et al.) 2024), and self-refiection (Madaan et al. |224 Shiin et al. 2023). Although
these agents have already seen significant success across various applications (Wang et al., 2024). developing these building blocks and combining them into complex agentic systems often requires domain-specific manual tuning and substantial effort from both researchers and engineers.
However, the history of machine learning reveals recurring theme: manually created artifacts be- come replaced by learned, more efficient solutions (Clune 2019) over time as we get more compute and data (Sutton 2019). An early example is from computer vision, where hand-designed features like HOG (Dalal & Triggs 2005) were eventually replaced by learned features from Convolutional Neural Networks (CNNs, Krizhevsky al. 22112). More recently, AutoMI. methods (Hutter al.,

---

Published as a conference paper at ICLR 2025
Next interesting agent
Refine until novel and error-free
Summary and motivation: "Based on the insights from previous agents ....
Name: "Divide and Conquer Agent", Code: forward[Task]:
Meta Agent
New Agent
return Answer"
Input
Test performance on tasks
Agent Archive
and add to archive
Dypars
Took
Task
Anelyne
Vicuil
Texk
Sub piobion
Visual Poined gi
Answer
Arsweis
Bisemble
Veetter
CaT
VarTfod gm
Answers
Dapere
Mutti step Poer Rexiew Agent
Ver Sed Multimadil Agent
Divile and Coneguer Agant
Examples of Discovered Agents
Overview of the proposed algorithm Meta Agent Search and examples of discovered agents. In our algorithm, we instruct the "meta" agent to iteratively program new agents, test their performance on tasks, add them to an archive of discovered agents, and use this archive to inform the meta agent in subsequent iterations. We show three example agents across runs, with all names generated by the meta agent. The detailed code of example agents can be found in Appendix H
Figure 1:
2019) and Al-Generating Algorithms (AI-GAs, Clune (2019) have also demonstrated the superior- ity of learned AI systems compared to hand-designed AI systems. For example, the current best- performing CNN models come from Neural Architecture Search (Elsken et al. 2019 Shen al.,
2023) instead of manual design; in LLM alignment, learned loss functions (Lu et al. 2024a) out-
perform most hand-designed ones such DPO (Rafailov et al. 2024); The AI Scientist (Lu al.)
2024b) demonstrates an automated research pipeline, including the development of novel ML algo- rithms: and an endless number of robotics learning environments can be automatically generated in works like OMNI-EPIC (Faldor et al. 2024), which demonstrate surprising creativity in generated environments and allow more efficient environment creation than the manual approach (see more examples in Section 55) Therefore, in this paper, we propose new research question: Can we automate the design of agentic systems?
To explore the above research question, we describe newly forming research area we call Automated Design of Agentic Systems (ADAS), which aims to automatically invent novel build- ing blocks and design powerful agentic systems (Section 2). We argue that ADAS may prove to be the fastest path to developing powerful agents, and show initial evidence that learned agents can greatly outperform hand-designed agents. Considering the tremendous number of building blocks yet to be discovered in agentic systems (Section 5). would take a long time for our research com- munity to discover them all. Even if we successfully discover most the useful building blocks, combining them into effective agentic systems for massive real-world applications would still be challenging and time-consuming, given the many different ways the building blocks can combine and interact with each other. In contrast, with ADAS, the building blocks and agents can be learned in an automated fashion. ADAS may not only potentially save human effort in developing powerful agents but also could be a faster path to more effective solutions than manual design.
Although few existing works can be considered as ADAS methods, most of them focus only on designing prompts (Yang et al. 2024 Fernando et al. 2024), greatly limiting their ability to invent flexible design patterns in agents (Section 5). In this paper, we show that there is an unexplored yet promising approach to ADAS where we can define the entire agentic system in code and new agents can be automatically discovered by a "meta" agent programming ever better ones in code. Given that most programming languages, such as Python, which Complete (Boyer& & Moore 1983] Ladha 2024), searching within a code space theoretically enables an ADAS algorithm to discover any possible agentic systems, including all components such as
in this paper, are Turing
2

---

Published as a conference paper at ICLR 2025
prompts, tool use, workflows, and more. Furthermore, with recent FMs being increasingly proficient in coding, we can use FMs as meta agents to create new agents in code for ADAS, enabling novel agents to be programmed in automated manner.
Following the aforementioned ideas, we present Meta Agent Search in this paper as one the first algorithms in ADAS that enables complete design in code space (Figure D. The core concept of Meta Agent Search is to instruct a meta agent to iteratively create interestingly new agents, evaluate them, add them to an archive that stores discovered agents, and use this archive to help the meta agent in subsequent iterations create yet more interestingly new agents. Similar to existing open-endedness algorithms that leverage human notions of interestingness (Zhang al.) 2024a, Lu al., 2024c), we encourage the meta agent to explore interesting (c.g.. novel or worthwhile) agents. To validate the proposed approach, we evaluate the proposed Meta Agent Search on: (1) the challenging ARC logic puzzle task (Chollet, (019)) that aims to test the general intelligence of an Al system, (2) four popular benchmarks on reading comprehension, math, science questions, and multi-task problem solving, and (3) the transferability of discovered agents to held-out domains and models (Section 4
Our experiments show that the discovered agents substantially outperform state-of-the-art hand- designed baselines. For instance, our agents improve Fl scores reading comprehension tasks in DROP (Dua et al., 2019) by 13.6/100 and accuracy rates on math tasks in MGSM (Shi et al.) 2023) by 14.4%. Additionally, they improve accuracy over baselines by 25.9% and 13.2% on GSM8K (Cobbe et al. (021) and GSM-Hard (Gao et al. 2023) math tasks, respectively, after trans- hand-designed solutions
ferring across domains. The promising performance of algorithm illustrates the potential of ADAS in automating the design of agentic systems. Furthermore, the experiments demonstrate that the discovered agents not only perform well when transferring across similar domains but also exhibit strong performance when transferring across dissimilar domains, such as from mathematics to reading comprehension. This highlights the robustness and transfer- ability of the agentic systems discovered by Meta Agent Search. In conclusion. our work opens up many exciting research directions and encourages further studies (Section 6
2
AUTOMATED DESIGN OF AGENTIC S SYSTEMS (ADAS)
Search Space
Search Algorithm
Evaluation Function
Sample New Agent
Whenl Cr
1 1+1=7
Agent
LLM
E.g. Agonts defined by code
E.g. defines ogants usingooda code
Evaluate the Objectives
E.g. Accuracy the task
The three key components of Automated Design Agentic Systems (ADAS). The search space determines which agentic systems can be represented in ADAS. The search algorithm specifies how the ADAS method explores the search space. The evaluation function defines how to evaluate candidate agent target objectives such performance.
Figure 2:
At the time of writing, the community has not reached consensus on the definitions or terminolo- gies of agents. Here, by agents we refer to agentic systems that involve Foundation Models (FMs) as modules in the workflow solve tasks by planning, using tools, and carrying out multiple, iterative steps of processing (Chase 2024 Ng 2024). In this paper. we describe a newly forming research area Automated Design of Agentic Systems (ADAS). Similar research areas AI-GAs (Clune) 2019) and AutoMI. (Hutter et al. 2019). such as Neural Architecture Search (Elsken et al. (019),, we formulate ADAS as an optimization process and identify three key components of ADAS algorithms (Figure 2).
Formulation
Automated Design of Agentic Systems (ADAS) involves using search algorithm to dis- cover agentic systems across a search space that optimize an evaluation function.
3

---

Published as a conference paper at ICLR 2025
Search Space: The search space defines which agentic systems can be represented and thus discovered in ADAS. For example, works like PromptBreeder (Fernando et al., 2024) mutate only the text prompts of an agent, but their other components, such as workflow, remain the same. Thus, in these search spaces, agents that have a different workflow than the predefined one can not be represented. Existing works also explore search spaces such graph structures (Zhuge al., 2024) and feed-forward networks (Liu et al. )
The search algorithm defines how ADAS algorithms explore the search space. Since the search space is often very large or even unbounded, the coaloration-expliitttoon trade- off (Sutton & Barto 2018) should be considered. Ideally, the algorithm can both quickly discover high-performance agentic systems and avoid remaining stuck in a local optimum. Existing ap- proaches include using Reinforcement Learning (Zhuge et al., (024)) or an FM iteratively gener- ating new solutions (Fernando et al. 2024) as search algorithms.
Search Algorithm:
Evaluation Function: Depending on the application of the ADAS algorithm, we may consider different objectives to optimize, such as performance, cost, latency, or safety of agents. An eval- uation function defines how to evaluate a candidate agent on those objectives. For example, to assess the agent's 's performance on unseen future data, a simple method is to calculate the accuracy rate on the validation data for a task, which is commonly adopted existing works (Zhuge al.) 2024
Fernando et al. 2024).
Although many search space designs possible and some have already been explored (Section 5 there is an unexplored yet promising approach where we can define the entire agentic system in code and new agents can be automatically discovered by a meta agent programming ever better ones in code. Searching within a code space theoretically enables the ADAS algorithm to discover any possible building blocks (e.g., prompts, tool use. workflow) and agentic systems that combine any of these building blocks in any way. This approach also offers better interpretability for agent design patterns since the program code is often readable, making debugging easier and enhancing AI safety. Additionally, compared to search spaces using networks (Liu et al.) 2023) or graphs (Zhuge et al.) 2024), searching in a code space allows us to more easily build on existing human efforts. For ex- ample, it is possible to search within open-source agent frameworks like LangChain (LangChainA]] 2022) and build upon all existing building blocks (c.g.. RAG, search engine tools). Finally, since FMs are proficient in coding, utilizing code search space allows to leverage existing expertise from FMs during the search process. In contrast, search algorithms in custom search spaces, such as graphs, may be much less efficient due to the absence of these priors. Therefore, we argue that the approach of using programming languages as the search space should be studied more in ADAS.
3 OUR ALGORITHM: META AGENT SEARCH
In this section, we present Meta Agent Search, a simple yet effective algorithm to demonstrate the approach of defining and searching for agents in code. The core idea of Meta Agent Search is to adopt FMs meta agents to iteratively program interestingly new agents based on an ever- wer-growing archive of previous discoveries. Although any possible building blocks and agentic systems can theoretically be programmed by the meta agent from seratch, it is inefficient in practice to avoid providing the meta agent any hasic functions such as FM query APIs or existing tools. Therefore, in this paper, we define simple framework (within 100 lines of code) for the meta agent, providing it with a basic set of essential functions like querying FMs or formatting prompts. As a result, the meta agent only needs to program "forward" function to define a new agentic system, similar to Cooeera-Paredee et al. 2024). This function takes in the information of
the practice in FunSearch the task and outputs the agent's 's response to the task. Details of the framework codes and examples of the agents defined with this framework can be found in Appendix
As shown in Figure I the core idea of Meta Agent Search to have a meta agent iteratively program new agents in code. The algorithm proceeds as follows: (1) The archive is (optionally) initialized with baseline agents such as Chain-of-Thhughh (Wei et al., (022) and Self-Refine (Madaan et al., 2024 Shinn et al. 2023). (2) Conditioned on the archive, the meta agent designs new agent by generating high-leve description of the new idea for an agentic system and then implementing it in code. The design then undergoes two self-reflection (Madaan et al. 2024 Shinn et al. 2023) steps by the meta agent to ensure is novel.
(3) The generated agent is evaluated using validation data from the target domain. errors occur during evaluation, the meta agent performs a self-reflection step to
4

---

Published as a conference paper at ICLR 2025
refine the design, repeating this process up five times if necessary. (4) Finally, the agent is added to the archive along with its evaluation metrics, and the process continues with the updated archive until the maximum number of iterations is reached. A pseudocode of the algorithm is provided in A
Similar to existing open-endedness algorithms that leverage human notions of interesting- ness (Zhang et al.] 2024a Lu et al. 2024e), we encourage the meta agent to explore interestingly new (e.g., novel or worthwhile) agents based on an ever-growing archive of previous discoveries. Here, we calculate the performance (e.g., success rate or Fl score) as the metrics for the meta agent maximize. The prompt and more details are presented in Appendix
4 EXPERIMENTS
(2) four popular benchmarks assessing the agent's abilities on reading comprehension, math, science ques- tions, and multi-task problem solving (Section 4.2p. and (3) the transferability of discovered agents on math to held-out math tasks and non-math tasks (Section 4.3). We use an identical implemen- tation of the algorithm across different tasks, with the only variation being task-specific descriptive text included the prompt (details are available in Appendix [.. Across experiments, we find that the discovered agents substantially outperform baseline state-of-the-arr hand-designed agents and maintain superior performance even when transferred across domains and models.
We conduct extensive experiments on: (1) the ARC challenge (Chollet 2019) (Section 4.1
4.1
CASE STUDY: ARC CHALLENGE
Meta -egent Search ARC
The Best Discovered Agent
14
COT
Held-out Test, Accuracy (96)
Best agent inbroduedd mulipi
Task
Answers
12
Efficiency Expert
Intpauol spant manry dong more refnemett
Experts
Readability Exert
Human-like
Critic
Simplicity Expert
dtaonsy emenged Hollle cors
to panerate sosatee ar crowers rutt them and essemoiig
Final
Answer
Feedback
Foodback
(befoen Inzlementing lew level dera
Ensemble
All Answers
Refinement
3 times
Iteration
Top-3 Answers
Evaluate
Chain thought
COT-CC
Self-Refine
Qualtty-Diversity
LLM Debate
Aeta-Agent Search
Structured F Feedback and Ensemble Agent
(a)
(b)
Figure 3: The results of Meta Agent Search on the ARC challenge. (a) Meta Agent Search progressively discovers high-performance agents based on ever- ever-growing archive of previous dis- coveries. We report the median accuracy and the 95% bootstrap confidence interval on a held-out test set by evaluating agents five times. (b) The visualization of the best agent discovered by Meta Agent Search on the ARC challenge. Detailed implementation of this agent is available in Appendix
We first demonstrate how Meta Agent Search discovers novel agentic systems and outperforms ex- isting state-of-the-an hand-designed agents in the Abstraction and Reasoning Corpus (ARC) chal- lenge (Chollet, 2019). This challenge aims to evaluate the general intelligence of AI systems through their ability to acquire new skills. Questions in ARC include (1) showing multiple examples of vi- sual input-output grid patterns, (2) the AI system learning the transformation rule of grid patterns from examples, and (3) predicting the output grid pattern given test input grid pattern. Since each question in ARC has a unique transformation rule, it requires the Al system to learn efficiently with few-shot examples, leveraging capabilities in number counting, geometry, and topology.
5

---

Published as a conference paper at ICLR 2025
require the agent to write code for the transformation rule instead of answering directly. We provide tool functions in the framework (de- scribed in Section that evaluate the generated transformation code. Given the significant challenge that ARC poses to current Al systems, we sample our data from questions with grid dimensions < 5 x in the "Public Training Set (Fasy)". We sample validation set and test set with 20 and 60 questions, respectively, for searching and testing. We calculate the validation and test accuracy of an agent by assessing it over the validation and test sets five times to reduce the variance from the stochastic sampling of FMs. We evaluate all discovered agents on the held-out test set and report the test accuracy in Figure Meta Agent Search runs for 25 iterations and the meta agent uses GPT-4 (OpenAl 2024). while discovered agents and baselines are evaluated using GPT-3.5 (Ope-) nAI 2022 to reduce compute cost. More algorithmic details and examples of ARC questions can be found in Appendix
Setup. Following common practice (Greenblatt 2024
Baselines. We compared against five state-of-the-art hand-designed agents: (1) Chain-of-
(COT. Wei et al. (2022), ) which instructs the agent to output the reasoning before answering to improve complex problem-solving through intermediate steps; (2) Ceff-ssnist with Chain-of- Thought (COT-SC, Wang et al. (023b)), which ensembles multiple parallel answers from COT to produce a more accurate answer; (3) Self-Refine (Madaan et al.] 2024] Shinn et al., 2023), which allows iterative self-reflection to correct mistakes made in previous attempts; (4) LLM- (Du iet al. (023). which enables different L.MS to debate with each other, leveraging diverse perspec- tives find better answers; (5) Quality-Diversity, Diversitty a simplified version of Intelligent Go-Explore (Lu iet al. (202c)), which produces and ensembles diverse answers to better explore potential solutions. The selected baselines represent widely adopted agent designs in the agent literature, embodying key design patterns and approaches frequently utilized across various applications. By "state-of- the-art," we refer to these baseline designs as exemplifying important advancements and practices within the field. We also use all baselines as initial seeds in the archive for Meta Agent Search, with additional results for empty initialization provided in Appendix [] To ensure fair comparisons, all baseline implementations were developed using the same framework as the Meta Agent, providing a consistent and equitable evaluation environment. More details about baselines can be found in Appendix G
Results and Analysis. As shown in Figure 3a Meta Agent Search effectively and progressively discovers agents that perform better than ffate-of-the-aee sand-designed baselines. Important break- throughs are highlighted in the text boxes. As is critical in prior works on open-endedness and
AI-GAs (Zhang et al. 2024a Faldor et al.. 2024 Wang et al., 2019 2020; Lehman & Stanley 2011]. Meta Agent Search innovates based on a growing archive of previous stepping stones. For example, an important design pattern emerged in iteration 3 where uses multiple COTs to gener- ate possible answers, refines them, and finally ensembles the best answers. This became a crucial stepping stone that subsequent designs tended to utilize. Additionally, the best-discovered agent is shown in Figure where a complex feedback mechanism is adopted to refine answers more effec- tively. Careful observation of the search progress reveals that this sophisticated feedback mechanism did not appear suddenly. Instead, the ideas of incorporating diverse feedback, evaluating for various specific traits (via experts) such as efficiency and simplicity, and simulating human-like feedback emerged in iterations 5, 11, and 12, respectively. The final mechanism is innovation based on these three stepping stones. This illustrates that even though these stepping stones did not achieve high performance immediately upon emergence, later discoveries benefited from these innovations by combining different stepping stones, resembling crossover in evolution via LLMS (Meyerson cet al.) (023). Overall, the results showcase the potential of ADAS and the effectiveness of Meta Agent Search to progressively discover agents that outperform state-of-the-art hand-designed base- lines and invent novel design patterns through the innovation and combination of stepping stones.
4.2 REASONING AND GROLLIGG DOMAINS
Setup. Next, we investigate the potential of our algorithm to improve the capabilities of agents across math, reading, and reasoning domains. We test Meta Agent Search on four popular bench- marks: (1) DROP (Dua et al. 2019) for evaluating Reading Comprehension; (2) MGSM (Shi et al.) 2023 for evaluating Math capability under a multi-lingual setting; (3) MMLU (Hendrycks et al, 2 for evaluating Multi-task Problem Solving: and (4) GPQA (Rein et al. 2023) for evaluating the capability of solving hard (graduate-level) questions in Science The search is conducted inde- pendently within each domain. Meta Agent Search runs for 30 iterations. The meta agent uses GPT-

---

Published as a conference paper at ICLR 2025
4 (OpenAl| 2024), while the discovered agents and baselines are evaluated using 5.5-3.5 (OenAA
2022). More details about datasets and experiment settings can be found in Appendix
Baselines. We adopt all baselines introduced in Section 4.1 Additionally, since the above do- mains require strong reasoning skills, we include two additional baselines that specifically focus on enhancing the reasoning capabilities of agents for more thorough comparison: (1) Step-back Abstraction (Zheng et al. 2023), which instructs agents to first consider the principles involved in solving the task for better reasoning; (2) Role Assignment (Xu et al., 2023). which assigns different roles to FMs to obtain better answers. Furthermore, we compare our approach with the state-of-the- art prompt optimization baseline OPRO (Yang et al.) 2024) highlight the advantages of learning all possible components of agents rather than focusing solely on prompts. More details about the baselines can be found in Appendix G
Table I: Performance comparison between Meta Agent Search and state-of-the-art hand- designed agents across multiple domains. Meta Agent Search discovers superior agents compared to the baselines in every domain. We report the test accuracy and the 95% bootstrap confidence in- terval on held-out test sets. The search is conducted independently for each domain. Here, and in all tables below, we bold the entry with the highest performance for each domain, as well as all entries whose median falls within the 95% confidence interval of the highest-nerforming treatment.
F1 Score
Accuracy (%)
Agent Name
Reading Comprehension
Math
Multi-task
Science
State-of-the-art Hand-designed Agents
Chain-of- Thought Wei al. 2022
64.2 +
28.0 3.1
65.4 +
29.2 + 3.1
COT-SC (Wang et al. 2023b
64.4 = 0.8
28.2 = 3.1
65.9 3.2
30.5 + 3.2
Self-Refine (Madaan al.)
2024
59.2 = 0.9
27.5= 3.1
63.5 33
31.6 - 3.2
LLM Debate (Du et al.
2023
60.6 + 0.9
39,0 3,0=30 34
65.6 3.3
31.4 3.2
Step-back Abstraction (heegg al.
2023
60.4 + 1.0
31.1 3.2
65.1
26.9 3.0
(Lu al. 2024c
61.8 1 0.9
23.8 3.0
65.1 3.3
30.2 1 3.1
Role Assignment (uu al.
2023
65.8 = 0.9
30.1 3.2
61.5
31.1 + 3.1
Automated Design of Agentic Systems on Different Domains
Prompt Optimization Yang et al. 2024
69.1 = 0.9
30.6 3.2
67.6 = 3.2
32.9 + 3.2
Meta Agent Search (Ours)
79.4 - 0.8
53.4 3.5
69.6 - 3.2
34.6 = 3.2
Results and Analysis. The results across multiple domains demonstrate that Meta Agent Search can discover agents that outperform state-of-the-art hand-designed agents (Table []. We want to highlight the substantial gap between the learned agents and hand-designed agents the Reading Comprehension and Math domains, with improvements in F1 scores by 13.6/100 and accuracy rates by 14.4%, respectively. While Meta Agent Search also outperforms baselines in the Multi-task and Science domains, the gap is smaller. We hypothesize that for challenging questions in the Science and Multi-task domains, the knowledge in FMs is not sufficient to solve the questions, limiting the improvement through optimizing agentic systems, which is problem that will diminish as FMs improve. In contrast, in the Reading Comprehension and Math domains. FMs possess adequate knowledge to solve the questions, and errors could mainly be hallucinations or calculation mistakes, which can be mitigated through well-designed agentic systems, like the ones discovered by Meta Agent Search. Additionally, when compared to prompt optimization methods, the results demon- strate that our proposed Meta Agent Search consistently outperforms them across all domains. This comparison further strengthens of all components offer significant advantages. Overall, the results across various domains showcase the effectiveness of Meta Agent Search in searching for agents tailored to specific domains. This could be increasingly useful for saving human efforts and developing better task-specific agents as continue to create agents for a diverse set of applications (Wang et al. 2024).
argument that defining agents in code and enabling the learning
4.3 GENERALIZATION AND TRANSFERABILITY
In the previous sections, illustrated that Meta Agent Search find effective agents for in- dividual tasks. In this section, we further demonstrate the transferability and generalizability of the discovered agents. To demonstrate the generalizability of the invented building blocks and de-
7

---

Published as a conference paper at ICLR 2025
sign patterns, transfer discovered agents from the MGSM (Math) domain both math and non-math domains to test their ability to generalize across different tasks. We evaluate the top agents from MGSM by transferring them to (1) popular math domains: GSM8K (Cobbe et al. 2021), GSM-Hard (Gao et al., 2023), and (2) non-math domains: MMLU (Multi-task) and DROP (Reading Comprehension).
detailed in Section As shown in Table Meta Agent Search con- sistently outperforms the baselines. Notably, our agents improve accuracy by 25.9% on GSM8K and 13.2% on GSM-Hard compared the baselines when transferring within math domains. More surprisingly. we find that agents discovered in the math domain can also be transferred to non-math domains. While their performance does not fully match agents specifically designed for the target domains, they still outperform state-of-the-arr hand-designed baselines. More results of transfers across domains are shown in Appendix
We also observe similar superiority when transferring agents across different FMs on ARC. We test the top agents with the best test accuracy evaluated with GPT-3.5 on ARC and then transfer them to Claude-Haiku (Anthropic 2024a), GPT-4 (OpenAl 2024), and Claude-Sonnet (Anthropic) 2024b). As shown in 3] we observe that the searched agents consistently outperform the hand- designed agents, with substantial gap. Notably, we found that Claude-Sonnet the most powerful model from Anthropic, performs the best among all tested models, enabling our best agent to achieve nearly 50% % accuracy on ARC. These results transferring across domains and models highlight Meta Agent Search 's ability to discover generalizable design patterns and agentic systems.
Table 2: Performance on held-out math and non-math domains when transferring top agents from MGSM (Math). GSM8K and GSM-Hard are the held-out math domains, while MMLU is for Multi-task, and DROP is for Reading Comprehension. Agents discovered by Meta Agent Search consistently outperform the baselines across all domains. We report the test accuracy and the 95% bootstrap confidence interval. The names of the top agents are generated by Meta Agent Search.
Accuracy (%)
F1 Score
Agent Name
MGSM
GSM8K
GSM-Hard
MMLU
DROP
Manually Designed Agents
Chain-of- WWi al.
2022
28.0 1 3.1
34.9 1 3.2
15.0 2.5
65 45.4-1 3.3
64.2 0.9
COT-SC (Wagg et al 2023b
28.2 1 3.1
37.8 1 3.4
1
65.9 1 3.2
64.4 0.8
Self- Self-Refine (Madaan al. 2024
27.5 3.1
38.9 3.4
1..1
63.5 3.4
59.2
LLM Debate Du et al. 2023
39.0 - 3.4
43.6 3.4
2.6
65.6 = 3.3
60.6 60.6=099
Step-back Abstraction [Zheng al.
2023
31.1 3.2
3.3
1.3
65.1 3.3
60.4
Quality- (Lu et al.
2024c
23.8 - 3.0
28.0
14.1
65.1 = 3.1
61.8 0.9
Role Assignment (uu
2023
30.1 : 3.2
37.0
18.0
64.5 3.3
65.8 0.9
Top Agents Searched on MGSM (Math)
Transferred within Math Domains
Transferred beyond Math Domains
Dynamic Role-Playing Architecture
53.4 1.4 3.5
69.5 3.2
31.2 + 3.2
6.44
70.4= 0.9
Structured Multimodal Feedback Loop
50.2 3.5 3.5
3.4
30.1 3.2
67.0 3.2
70.4 0..4
Interactive Multimodal Feedback Loop
47.4 3.5
64.9 + 3.3
27.6 3.2
64.8 = 3.3
71.9 0.8
5 RELATED WORK
Agentic Systems. Researchers develop various building blocks and design patterns for different
applications. Important building blocks for agentic systems include: prompting techniques (Chen
ett al. 2023a
Schulhoff et al. 2024), Shain-of--toughte
plamning and reasoning methods ( (Wei
let al.
2022, Yao et al. 2023 Hu & Clune 2024). reflection (Madaan et al. 2024 [Shinn et al.
2023 developing new skills for embodied agents in code (Wang et al. 2023a Vemprala et al.
2023), , external memory and RAG (Zhang et al. 2024c Lewis et al.) (220), tool use (Qu et al. 2024
Schiik et al. 2023 Nakano et al. 2021). assigning FM modules in the agentic system with different
roles and enabling them to collaborate (Hong et al., 2023 Wu et al. 2023 Qian et al. 2023 Xu
et al. 2023 Qian al. 2024). and enabling the agent instruct itself for the next action (Richards) 2023 etc. While the community has invested substantial effort in developing all the above important
Techniques, this is only partial list of the discovered building blocks, and many more remain to be
8

---

Published as a conference paper at ICLR 2025
Table 3: Performance on ARC when transferring top agents from GPT-3.5 to other FMs. Agents discovered by Meta Agent Search consistently outperform the baselines across different models. We report the test accuracy and the 95% bootstrap confidence interval. The names of top agents are generated by Meta Agent Search. We manually changed this name because the original generated name was confusing.
Accuracy ARC (%)
Agent Name
GPT-3.5
Claude-Haiku
GPT-4
Claude-Sonnet
Manually Designed
Agents
Ofain-of-Thoughee Wei et al. 2022
6.0 i 2.7
4,3 2.2
17.7 4,4
25.3 +: 5,0
COT-SC Wang 2023b
8.0 + 3.2
5.3
19.7 4.5
26.3 + 4.9
L.M Dehate (Du al. 202.
4.0 + 2.2
1.7
19.0 4.5
24.7 4.8
Self-Refine Madaan et al. 2024
6.7 2.7
6.3 2.8
23.0 5.2
39.3 4: 5.5
Quality-Diverity (Lu et al.
2024c
+ 2.9
3.3 + 2.2
23.0 4.7
31.7 + 5.3
Top Agents Searched with GPT-3.5
Transferred to Other FMs
Structured Feedback and Ensemble Agent
13.7 1: 3.9
5.0 2.5
30.0 = 5.2
38.7 2 5,5
Hierarchical Committee Reinforcement Agent
13.3 3.8
8.3 3.2
32.3 + 8.9
39.7 + 5.5
Dynamic Memory and Refinement Agent'
12.7 1:
= 3.3
37.0 5.3
48.3 5.7
uncovered. Therefore, in this paper, we describe a newly forming research area, ADAS, which aims to invent novel building blocks and design powerful agentic systems in an automated manner.
Existing Attempts to ADAS. There are two categories of works that attempt ADAS: those focused on learning better prompts and those that learn more components beyond prompts. Most works fall into the first category, where FMss are used to automate prompt engineering, primarily enhancing the phrasing of instructions to improve reasoning (Yang et al. 2024 Fernando et al. [2024] Zhou jet al. 2024a Yuksekgonul et al. (024). However, these prompts are often domain-specifhe and dif- ficult to generalize. Some works optimize role definitions within prompts (Yuan al.) [2024] Chen iet al., [2023c bj Wu et al. 2023). as assigning personas or roles to agents has been shown to be beneficial (Xu et al. 2023). Although tuning prompts can improve performance, other components remain fixed, limiting the space of agents that can be discovered. The second category, which is less explored, involves learning additional components such as workflows, often representing agents as networks or graphs. In these formulations, the FM with a certain prompt is considered trans- formation function for text on nodes, and the information flow of the text considered as edges. For example, DyLAN (Liu et al.) (023) uses FMs to optimize connections between nodes in net- work, DSPy (Khattab et al. 2024) and Trace (Cheng et al., 2024) optimizes across the Cartesian product of a set of possible nodes, and GPT-Swarm (Zhuge et (0024) uses reinforcement learning to optimize node connections. Although these approaches optimize workflows, many components like tool usage remain fixed. AgentOptimizer (Zhang et al., (024b) learns the tools used in agents, AutoFlow (Li et al. 2024) proposes new language to optimize workflow, Agent Symbolic Learn- ing (Zhou et al. 2024b) attempts to learn prompts, tools, and workflows together. While these works share similar motivations to learn more components in agents, they either fail to cover all possible designs in agentic systems have harder search spaces for search algorithms. In contrast, our work represents all components in code, allowing all possible designs in agentic systems and resulting in a promising search space for FM- FM-guided search, as coding tasks are one of the most important tasks in FMs' training. We also include additional related work in Appendix A.1
6 DISCUSSION AND CONCLUSION
Safety Considerations. While is highly unlikely that generagge code will perform overtly malicious actions in our current settings with the Foundation Models (FMs) we employ, such code could still act destructively due to limitations in model capability or alignment (Rokon et al. 2020 Chen et al. 2021). To address these risks, we have implemented safety measures including con- tainerized execution of all generated code in secure, isolated environments, thorough manual in- spections to verify the absence of harmful behaviors, and clear warnings in our codebase to alert users to potential risks. These practices align with established safety standards in the literature, such

---

Published as a conference paper at ICLR 2025
as those in SWE-Bench (Jimenez et al. 2024 and Voyager WWang et al. 2023a). which similarly prioritize controlled execution environments.
The proposed Automated Design of Agentic Systems (ADAS) introduces novel area in AI-GA research, potentially accelerating the development of Artificial General Intelligence (AGI) beyond current manual approaches (Clune 2019). This raises broader questions about advancing AI ca- pabilities, a topic extensively debated in prior works (Clune 2019 Ecoffet et al.) 2020; Bostrom 2002 Yudkowsky et al.) 2008; Bengio et al. 2024), though beyond this paper's scope. We argue that publishing this work is net beneficial. It reveals that powerful ADAS algorithms can be easily programmed using API access FMs, without requiring expensive hardware like GPUs, inform- ing the community of their accessibility and implications. Moreover, ADAS can enhance safety in agentic systems by automating the design of explicit, interpretable workflows, reducing the risk of malicious behavior through greater controllability and auditability.
We believe the discussion on ADAS and its safety impact is timely given the growing adoption of agentic systems in real-world applications Turow 2024), where ADAS can streamline the creation of safe, reliable agents, amplifying Al's potential to benefit humanity domains like health and economics (Amodei (024). Furthermore, self-improving Al systems become prominent (Clune) 2019 Fernando et al. 2024; Lu et al.) 2024a; Zelikman et al., 2022). their continued development appears inevitable. By sharing this work, we aim to inspire further research into safe-ADAS algo- incorporating mechanisms like Constitutional AT (Bai et al., 2022 to ensure that advancements in AI-GA and self- self-improving AI yield systems that are both powerful and aligned with human values, ultimately fostering safer Al development.
Future Work. Our work also opens up many future research directions. Below, we discuss few, with additional directions provided in Appendix A.2
Higher-order ADAS. Since the meta agent used in ADAS to program new agents in code is also an agent, ADAS can become self-referential where the meta agent can be improved through ADAS as well. It would be an exciting direction to have higher order of meta-learning to allow the learning of the meta agent and even the meta-meta agent, etc. (Lu et al. 2023 Schmidhuber) 1987 2003 Zelikman et al., 2024)
Online Continual Learning. As agents are deployed, they will receive vast amounts of feedback from both task environments and users. Continuously improving agents based on this extensive feedback is challenging for human developers. However, with ADAS automating the design and enhancement of agents, online continual learning becomes feasible post-deployment.
We only consider one objective (i.e., performance) to optimize in this paper, but in practice, multiple objectives are often considered, such as cost, latency, and robust- ness of agentic systems (Hu et al. [2021] Huang et al., 2023). Thus, integrating multi-objective search algorithms (Deb et al. (0022) in ADAS could be promising.
Multi-objective ADAS.
Towards a Better Understanding of FMs. Works from Neural Architecture Search (Huang ctt al. 2023) show that by observing the emerged architecture, could gain more insights into Neural Networks. In this paper, we also gained insights about FMs from the results. For exam- plc, the best agent with GPT-3.5 involves complex feedback mechanism, but when we transfer to other advanced models, the agent with a simpler feedback mechanism but more refinement becomes better agent (Section 4.3). This shows that GPT-3.5 may have worse capability in evaluating and refining the answers, so it needs a complex feedback mechanism for better refine- ment, while other advanced models benefit more from simpler feedback mechanism.
Conclusion. In this paper, we propose a new research problem, Automated Design of Agentic Systems (ADAS), which aims to automatically invent novel building blocks and design powerful agentic systems. We demonstrated that promising approach to ADAS is to define agents in code, allowing new agents to be automatically discovered by a "meta" agent programming them in code. Following this idea, we propose Meta Agent Search, where the meta agent iteratively builds on previous discoveries to program interesting new agents. The experiments show that Meta Agent Search consistently outperforms ofate-of-the-an hand-designed agents across an extensive number of domains, and the discovered agents transfer well across models and domains. Overall, our work illustrates the potential of an exciting new research direction toward full automation in developing powerful agentic systems from the bottom up.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the Vector Institute, the Canada CIFAR AI Chairs program, grants from Schmidt Futures and Open Philanthropy, an NSERC Discovery Grant, and generous donation from Rafael Cosman. We thank Jenny Zhang, Rach Pradhan, Ruiyu Gou, Nicholas Ioannidis, and Eunjeong Hwang for insightful discussions and feedback.
REFERENCES
Dario Amodei. Machines of loving grace, October 2024. URL http:://bar.oammeeeeett com/ Bachines--f--ooing---e
Anthropic. Introducing the next generation of claude.
http:://ww.anchrppicc..o
famill March 2024a. Blog post.
Anthoopic. Introducing claude 3.5 sonnet.
hecres//www.attarrpic.ccom/
claude 3- 5--onnet] June 2024b. Blog post.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yu- val Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, et al. Managing extreme ai risks amid rapid progress. Science, 384(6699): 2024.
N Bostrom, Existential Risks: analyzing human extinction scenarios and related hazards. Journal of Evolution and Technology, 9, 2002.
Robert S Boyer and Strother Moore, A mechanical proof of the Turing completeness pure LISP. Citeseer, 1983.
Harrison Chase.
What is agent?
http:://booq.aagchhala dev/
what -ist-is-- an annatttt/ June 2024. Blog post.
Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. Unleashing the poten- tial of prompt engineering in large language models: a comprehensive review. arXiv preprint MrXiv:2310.14735 14735, 2023a.
Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Sesay Jaward, Karlsson Börje, Jie Fu, and Yemin Shi. Autoagents: The automatic agents generation framework. arXiv preprint, 2023b.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03377 2021.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representa- tions, 2023c.
Ching-An Cheng, Allen Nie, and Adith Swaminathan Trace is the next autodiff: Generative op- timization with rich feedback, execution traces, and 1lms. Advances in Neural Information Pro- cessing Systems, 37:71596-7164 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating Iims by human preference, 2024.
François Chollet. On the measure of intelligence. arXiv preprint arXiv: 191100191 2019.
Jeff Clune. Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv preprint arXiv: 1eee: 2019.

---

Published as a conference paper at ICLR 2025
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. 2021.
Antoine Cully and Yiannis Demiris. Quality and diversity optimization: A unifying modular frame- work. IEEE Transactions on Evolutionary Computation, 22(2):245-259, 2017.
N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE Com- puter Society Conference on Computer Vision and Pattern Recognition (CVPR 05), volume 1, pp. 886-893 vol. 1, 2005. doi: 10.1 0..110//VVR..200..77.
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist mul- tiobjective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary computation, 6(2): 182-197, 2002.
Aaron Dharna, Amy K Hoover, Julian Togelius, and Lisa Soros. Transfer dynamics in emergent evolutionary curricula. IEEE Transactions Games, 15(2):157-170, 2022.
Yilun Du, Shuang Li, Antonio Torralba, Joshua Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint
arXiv:2305. 140.5 2023.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 2368-2378, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N199 1246.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL'2: Fast reinforcement learning via slow reinforcement learning. International Conference on Learning Representattonss, 2017.
Adrien Ecoffet, Jeff Clune, and Joel Lehman. Open questions in creating safe open-ended Al: Tensions between control and creativity. In Conference on Artificial Life, pp. 27-35. MIT Press, 2020.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune. Omni-epic: Open-endedness via mod- cls of human notions of interestingness with environments programmed in code. arXiv preprint arXiv:2405.1556, 2024.
Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution, 2024.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1 1126-1 1135. PMLR, 2017.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference Machine Learning, pp. 10764-10799, PMLR, 2023.
Ryan Greenblatt. Getting 50% sota on arc-agi with gpt-4. fatpss:/eedwodreeeerss July 2024. Techni-
substack. htt.://c.........o..ooooo.aoaao.
cal Report.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2025
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi- agent collaborative framework. arXiv preprint arXiv:2308.00
2023.
Shengran Hu and Jeff Clune. Thought Cloning: Learning to think while acting by imitating human thinking. Advances in Neural Information Processing Systems, 36, 2024.
Shengran Hu, Ran Cheng, Cheng He, Zhichao Lu, Jing Wang, and Miao Zhang. Accelerating multi- objective neural architecture search by random-weight evaluation. Complex & Intelligent Systems, pp. 1-10, 2021.
Shihua Huang, Zhichao Lu, Kalyanmoy Deb, and Vishnu Naresh Boddeti. Revisiting residual net- works for adversarial robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8202-8211, 2023.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated machine learning; methods, sys- tems, challenges. Springer Nature, 2019.
Carlos E Jimenez, John Yang. Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL htthstss:// //pp///vie/......
Det/Eorum?id-VIE 6yNQM66
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas Joshi, Hanna Moazam, Heather Miller, et al. Dspy: Compil- ing declarative language model calls into state-of-the-art pipelines. In The Twelfth International Conference on Learning Representations, 2024.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Abrahim Ladha. Lecture 11: furing-completenes. https: http://accltt..o/oo /tttps://facutyy.comm cc cc. ww..y.cchh..at.....cccc edu/
- Fladha/ Blacha/244/00 022/4551/II fff. 2024. CS 4510 Automata and Complexity, February 21st,
2024, Scribed by Rishabh Singhal.
LangChainAL. Langchain: Build context-aware reasoning applications. https://github. 2022.
com/
Joel Lehman and Kenneth Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computution, (992):1892222 2011.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval augmented genera- tion for knowledge-intente -eteede--nniniveeeeeeeeeeeee nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, and Yongfeng Zhang. Autoflow: Automated workflow generation for large language model agents. arXiv preprint arXiv:2407. /2821 2024.
Fei Liu, Tong Xialiang, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. Evolution of heuristics: Towards efficient automatic algorithm design using large language model. In Forty-first International Conference on Machine Learning, 2024.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic Ilm-agent network: An Ilm- agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02110, 2023.
Chris Lu, Sebastian Towers, and Jakob Foerster. Arbitrary order meta-learning with simple population-bail -assdd evolution. In ALIFE 2023: Ghost in the Machine: Proceedings of the 2023 Artificial Life Conference. MIT Press, 2023.
13

---

Published as a conference paper at ICLR 2025
Chris Lu, Samuel Holt, Claudio Fanconi, Alex J Chan, Jakob Foerster, Mihaela der Schaar, and Robert Tjarko Lange. Discovering preference optimization algorithms with and for large language models. arXiv preprint 240.v::200.084414, 2024a.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The Al Scien- tist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.0622 2024b.
Cong Lu, Shengran Hu, and Jeff Clune. Intelligent go-explore: Standing the shoulders of giant foundation models. arXiv preprint arXiv:2405. 1iti:24555.1143 15143, 2024c.
Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. Nsga-net: neural architecture search using multi-objective genetic algorithm. In Proceedings of the genetic and evolutionary computation conference, pp. 419-427 427, 2019.
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayara- man, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In The Twelfth International Conference on Learning Representa- tions, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Elliot Meyerson, Mark Nelson, Herbie Bradley, Adam Gaier, Arash Moradi, Amy K Hoover, and Joel Lehman. Language model crossover: Variation through few-shot prompting. arXiv preprint 1.Xi::2302.12170. 2023.
Shen-yun Miao, Chao-Chun Liang, and Keh- Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. Proceedings of the 58th Annual Meeting of the Association .orr Computational Linguisties, pp. 975-984, 2020.
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909, 2015.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprimi 2rXiv:2112.093322 2021. 2021.
Andrew Ng. Issue 253, June 2024. Newsletter issue.
253/
Ben Norman and Jeff Clune. First-explore, then explit:: Meta-leaming intelligent exploration. arXiv preprint arXiv:2307.02226. 2023.
OpenAl. Introducing chatgpt. http:://ppeaa..ccomm/meee///aaap Blog post.
Novenee November 2022.
OpenAl. Simple evals, 2023. URL hta.lt/pp///hhhhb..ooo/////////p evals Ac- cessed: 2024-08-10.
OpenAl. Gpt-4 technical report, 2024.
Joon Sung Park, Joseph O' OrBrin,, Carric Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pp. 1-22, 2023.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 2080- 2094, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. naaci-main.163
14

---

Published as a conference paper at ICLR 2025
Chen Qian, Xin Cong, Cheng Yang. Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint arXiv:2307.07924, 2023.
Chen Qian, Zihao Xie, Yifei Wang. Wei Liu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling large-language Language-model-aee moodel-based multi-agent collaboration. arXiv preprint vrXiv24466.07/55 2024.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with large language models: A survey. arXiv preprint arXiv:2405. 17935, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D) Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023.
Toran Bruce Richards. Autogpt. AutoGPT 2023. GitHub repository.
G
Tim Rocktäschel. Artificial Intelligence: 10 Things You Should Know. Seven Dials, September 2024. ISBN 978-1399626521,
Md Omar Faruk Rokon, Risul Islam, Ahmad Darki, Evangelos E Papalexakis, and Michalis Falout- sos. SourceFinder: Finding malware Source-Code from publicly available repositories in GitHub. In 23rd International Symposium on Research Attacks, Intrusions and Defenses (RAID 2020),
pp. 149-163, 2020.
Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468 475, 2024.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In seieth- Conference on Neural Information Processing Systems, 2023. URL httpp:://opeeeee////////////// wtwpeerreweweeeee//eoe..ee
Jurgen Schmidhuber: Evolutionary principles in self-referential learning. on learning now to learn: The meta-meta-meeta..hhon Diploma thesis, Technische Universitat Munchen, Germany, 14 May 1987. URL http::/ http://wwww... idsia..................iii...iii ch/ http///eeeen//ieeeeeeeeee http:///ppppppmaaaaaaeaaa html
Jürgen Schmidhuber. Gödel machines: self-referential universal problem solvers making provably optimal self-improvements. arXiv preprint cx/0309488, 2003.
Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, et al. The prompt report: A system- atic survey of prompting techniques. arXiv preprint arXiv:2406.0666 2024.
Xuan Shen, Yaohua Wang, Ming Lin, Yilun Huang, Hao Tang, Xiuyu Sun, and Yanzhi Wang. Deep- mad: Mathematical architecture design for deep convolutional neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6163-6173, 2023,
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Lan- guage models are multilingual ofain-of-thought reasoners. In The Eleventh International Confer- ence on Learning Representations, 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023.
15

---

Published as a conference paper at ICLR 2025
Kenneth O Stanley and Joel Lehman. Why greatness cannot be planned: The myth of the objective. Springer, 2015.
Kenneth Stanley, Jeff Clune, Joel Lehman, and Risto Mikkkulainen. Designing neural networks through neuroevolution. Nature Machine Intelligence, 1(1):4 35, 2019.
Richard S. Sutton. The bitter lesson, 2019. URL http:///ww..nncmmmleeeeeeeeeeeee net/
[ncldees//iittrreesoo. htmll
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Jon Turow. The rise of ai agent infrastructure. June 2024. URL https: www www.maadron.aommmmmm www..tts////w...aaaaoooo..
com/
the-rise- rise-of-aa -atss-----a--aeee------a---a----
Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: De-
sign principles and model abilities. Technical Report MSR-TR-2023-8, Microsoft, Febru- ary 2023. URL httpa///w//////o..oooooooo///tta for ottpa//w....ooooooo......ooooooa
Guanzhi Wang. Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: Arxiv-2305.1622 2023a.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arviv: 1et::111111001 L00763, 2016.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 1866):186345, 2024.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Poet: open-ended coevolution of environments and their optimized solutions. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '19, pp. 142-151, New York. NY, USA, 2019. Association for Computing Machinery. ISBN 0781450361110 111. 18. doi: 10.1 10.1145/3321707:33211999
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International conference on machine learning, pp. 9940-9951. PMLR, 2020.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of thothut prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen m1m applications via multi- agent conversation framework. arXiv preprint arXiv:2308.08873 08115500.0115,... 2023.
Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. Expertprompting: Instructing large language models to be distinguished experts. arXiv preprint arXiv:2305.44600 2023.
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le, Denny Zhou, and Xinyun Chen. Large language models optimizers. In The Twelfth International Conference Learning Representations, 2024. URL
net forum? in Bb4BBB44LLL.. B
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023, URLhttps: http//peeervveeeeee / net forum?
id=wE vluYUI--X
16

---

Published as a conference paper at ICLR 2025
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kinmani, Kuang-Huei Lee, Montserrat Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al. Language to rewards for robotic skill synthesis. In Conference Robot Learning, pp. 374 404. PMLR, 2023.
Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Dongsheng Li, and Deqing Yang. Evoa- gent: Towards automatic multi-agent generation via evolutionary algorithms. arXiv preprint arXiv:2406. 14228, 2024.
Eliezer Yudkowsky et al. Artificial Intelligence as a positive and negative factor in global risk. Global catastrophic risks, 1(303):184, 2008
Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic" differentiation" via text. arXiv preprint 27X499440..77496,
2024.
Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. The shift from models to compound ai systems. compound ai sasisssssems//// 2024.
www.paa...ee.ee....p..........ee hedb//loog/224//2/111
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STar: Bootstrapping reasoning with reasoning. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022. URL httpps://ppenreview
Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer (stop): Recursively self-inm code generation. In First Conference on Language Modeling, 2024.
Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. OMNI: Open-endedness via mod-
els of human notions of interestingness. In The Twelfth International Conference on Learning Representations, 2024a. URL httt:tttt///eeeeeeeee
Shaokun Zhang, Jieyu Zhang, Jiale Liu, Linxin Song, Chi Wang, Ranjay Krishna, and Qingyun Wu. Offline training of language model agents with functions as learnable weights. Forty-first International Conference on Machine Learning, 2024b.
Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents. arXiv preprint arXiv:2404. /3501, 2024c.
Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H Chi, Quoe V Le, and Denny Zhou. Take a step back: Evoking reasoning via abstraction large language models. arXiv preprint arXiv:2310.061 17, 2023.
Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Heng-Tze Cheng, Quoc V Le, Ed H Chi, Denny Zhou, Swaroop Mishra, and Huaixiu Steven Zheng. Self-discover: Large language models self- compose reasoning structures. arXiv preprint 2xX.::2402.03620, 2024a.
Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, et al. Symbolic learning enables self-evolving agents. arXiv preprint 240. 2024b.
Mingchen Zhuge, Wenyi Wang. Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference Machine Learning, 2024.
Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: Variational adaptivee deep rl via meta-learning. Journal of Machine Learning Research, 22(289): 2021a.
Luisa M Zintgraf, Leo Feng, Cong Lu, Maximilian Igl. Kristian Hartikainen, Katja Hofmann, and Shimon Whiteson. Exploration in approximate hyper-state space for meta reinforcement learning. In International Conference on Machine Learning, pp. 12991-13001 PMLR, 2021b.
17

---

Published as a conference paper at ICLR 2025
SUPPLEMENTARY MATERIAL
TABLE OF CONTENTS
A More Related Work and Future Work
19
A.1 More Related Work
19
A.2 More Future Work
19
Generalization and Transferability
20
Prompts
21
D Framework Code
23
Experiment Details for ARC Challenge
26
Experiment Details for Reasoning and Problem-Solving Domains
29
Baselines
30
Example Agents
Pseudocode of the Meta Agent Search
33
J Impact of Initialization
33
Cost of Experiments
34
18

---

Published as a conference paper at ICLR 2025
A MORE RELATED WORK AND FUTURE WORK
A.1 MORE RELATED WORK
AI-Generating Algorithms and AutoML. Research in Al-Generating Algorithms (AI-
GAs. (2019) and AutoML (Hutter et al. (2019) aims to replace handcrafted components in Al
systems by learning them. This field has three key pillars: (1) meta-learning architectures, (2) meta-
learning learning algorithms, and (3) generating learning environments and training data (Clune)
2019). Neural Architecture Search (Elsken et al.
2019; Lu et al.) 2019 Hu et al. (021) exem-
plifies the first pillar by automating neural network design, while works like MAML (Finn et al.
2017) and Meta-RI. (Wang et al. [016]] Duan et al.
2017: Norman & Clune 2023 Zintgraf et al.,
2021abb) b) exemplify the second pillar, focusing on "learning to learn" for improved sample efficiency and generalizability The third pillar includes works like POET (Wang et al.) 2019 Dharna et al., 2022 Wang et al. 2020) and OMNI-EPIC (Faldor et al.] 2024). which generate learning environ- ments in open-ended manner. We position Automated Design of Agentic Systems in both the first and second pillars: meta-learning agentic architectures and leveraging in-context learning to "learn to learn," " as shown in the ARC challenge (Section 4.1 Furthermore, recent AI-GA and AutoMI. advances have also integrated Foundation Models (FMs) to write code, as seen in Fun- Search (Romera-Paredes
al. 2024) and EoH (Liu et al. 2024), where FMs discover optimization algorithms. In DiscoPOP (Lu et al. (0224)), FMs program loss functions for preference learning, and Eureka (Ma al. 2023) and aanuage-to-rewaan
(Yu al., (223) enable FMs write reward func- tions for reinforcement learning. OMNI-EPIC (Faldor et al. (024] allows FMs to create robotics learning environments. Similarly, we enable FMs to program new agents in code.
A.2 MORE FUTURE WORK
More complex domains. Currently, we only evaluate Meta Agent Search on single-step QA tasks in this paper. It would be interesting to extend the method to more complex domains, such as real-world applications involving multi-step interaction with complex environments.
Seeding ADAS with more existing building blocks. Although we can theoretically allow any components in agentic systems to programmed from scratch in the code space, it is not efficient in practice. Therefore, it would be interesting to explore ADAS by standing on the shoulders of existing human efforts, such as search engine tools, RAG (Lewis et al., (020)), or functions from existing agent frameworks like LangChain [LangChainA]] 2022). Additionally, it is interesting to support multi-modal capabilities (e.g. vision) in FMs or allow different FMs to be available in agentic systems. This will enable the meta agent to choose from different FMs flexibly according to the difficulty of the instruction and whether data privacy is priority.
Novelty search algorithms. In Meta Agent Search, the design of the search algorithm is rela- tively simple. focusing solely on exploring interesting new designs. more careful design of the search algorithm can be promising future direction. For example, one could incorporate more sophisticated ideas from Quality-Diversity (Mouret & Clune 2013 Cully & Demiris 2017). AI- generating (Clune 2019). and Open-endec Algorithms (Faldor et al.) |024] Zhang et al. 2024a
Stanley& & Lehman 2015; Stanley et al. 2011)). One could also include more classic approaches to balance exploration and exploitation (Sutton & Barto, 2018 Liu et al. 2024).
More Intelligent Evaluation Functions. In this work, we simply evaluate discovered agents on the evaluation set and use the numerical performance results. However, this approach is both expensive and misses lot of information. A promising future direction is to enable the meta agent to analyze detailed running logs during the evaluation, which contain rich information on the failure and success modes for better debugging and improving agentic systems (Zhou al, 2024b). Also, many tasks involve subjective answer evaluations (Chiang et al.) 2024 Lu et al,, 2024b) that do not have ground-truth answers. It is also important to design novel evaluation functions in ADAS to address these tasks. Finally, in this work, we targeted only one domain during the search. It would be interesting to explore whether ADAS algorithms can design even better generalist agents when specifically searching for agents capable of performing well across multiple domains.
Understanding the emergence of complexity from human organizations. Beyond potentially saving researchers' efforts and improving upon the manual design of agentic systems, the research
19

---

Published as a conference paper at ICLR 2025
in ADAS is also scientifically intriguing as it sheds light on the origins of complexity emerging from human organization and society. The agentic system is a machine learning system that oper- ates primarily over natural language- representation representtinnnnnnnnnnnnnnnnn that is interpretable humans and used by humans constructing our organization and society. Thus, there a close connection between agentic systems and human organizations, structure for human companies in agents (Hong et al. (022)) or simulating human town with agents (Park al., 2023). Therefore, the study in ADAS may enable to observe how create a simple set of conditions and have an algorithm to bootstrap itself from simplicity to produce complexity in system akin human society.
shown in works incorporating the organizational
B GENERALIZATION AND TRANSFERABILITY
In this section, present more details of the experiments in Section 4.3 and the complete results of transferring agents across different domains.
For the results shown in Table 3 20240307" for Haiku, and "claude-33-ssomee-22220222 for Claude-Sonnet.
20pt-44-2024-05-- for GPT-4, "claude-3-haiie"
Table 4: Performance different math domains when transferring top agents from MGSM to other math domains. Agents discovered by Meta Agent Search consistently outperform the base- lines across different math domains. We report the test accuracy and the 95% bootstrap confidence interval. The names of top agents are generated by Meta Agent Search.
Accuracy (%)
Agent Name
MGSM
GSM8K
GSM-Hard
SVAMP
ASDiv
Manually Designed Agents
Chain-of-Thought et
2022
2.10
1.
2.8
88.9
COT-SC (Wang et al. 2025b
28.2 3.1
37.8=3.
15.
78.2 2.8
89.0 09.0:22 2.1
Self Refine (Madaan et al. 2024
27.5 + 3.1
38.9 = 3.4
15.1
78.5 1:5=224 2.8
89.2 2.2
L.MM Dehate (Du
2023
39.0 + 3.4
43.6 3.4
17.4+
76.0 3.0
88,9 2.2
Step-back Abstruction Zheng
2023
31.1
313
22
76.1
87.8
Quality-Dvvesity
Lu et 2024c
23.8 3.0
28.0=3.1
11.1224
09.8 3.2
80.1
Role Assignment
Xu et
2023
30.1 + 3.2
37.0 - 3.4
18.0 2.7
73.0 + 3.0
83.1 2.6
Top Agents Searched on MGSM (Math)
Transferred within Math Domains
Dynamic Role-Playing Architecture
:: 3.5
69.5
31.2 3.2
81.5 2.6
91.8 1.8
Structured Multimodal Feedback Loop
50.2 :: 3.5
64.5 3.4
30.1 3.2
82.6 2.6
89.9
Interactive Multimodal Feedback Loop
47.4 1: 3.5
64.9
27.6:::3.2
80.6 1: 2.8
89.8 1: 2.1
Table 5: Performance across multiple domains when transferring top agents from the Math (MGSM) domain to non-math domains. Agents discovered by Meta Agent Search in the math domain can outperform match the performance of baselines after being transferred to domains beyond math. We report the test accuracy and the 95% bootstrap confidence interval.
Agent Name
Accuracy (%)
F1 Score
Accuracy (%)
Math
Reading Comprehension
Multi-task
Science
Manually Designed Agents
Chain-of-Thought Wei ml 2022
28.0 3.1
64.2 + 0.9
65.4 3.3
29.2 = 3.1
COT-SC [Wang et ail 202cb
28. 2 3.1
61.1 0.8
65.9 3.2
30.5 3.2
Self-Refine Refine Madaan et 2024
27.5 3.1
59.2 + 0.9
63.5 3.4
31.6++3.2
LLM Debate (Du et
6..
31.4
Step Absiraction Zheng
2023
60.4
65.1
Quality Diversity LL
2024
23.5
61.8 +
65.1 + 3.1
Role Assignment Xu et
2023
30.1 3.2
65.8 1 0.9
64.3 1 3.3
31.1 =: 3.1
Top Agents Searched on Math (MGSM)
Transferred beyond Math Domains
Dynamic Playing Architecture
62.4
Structurd Multimodal Feedback Loop
50.2 3.5
70.1
67.0 3.2
28.7
Interactive Multimodal Feedback Loop
47.4 3.5
71.9 - 0.8
64.8 - 3.3
29.9 3.2
We transfer the discovered agent from the MGSM (Math) domain to other math domains to test whether the invented agents can generalize across different domains. Similarly, test the top
20

---

Published as a conference paper at ICLR 2025
3 agents from MGSM and transfer them to (1) four popular math domains: GSM8K (Cobbe al.
2021), GSM-Hard (Gao et al. 2023), SVAMP (Patel et al. 2021), J. and ASDiv (Miao et al. (020) and
(2) three domains beyond math adopted in Section 4.2 As shown in Table 4 observe similar superiority the performance of Meta Agent Search compared to baselines. More surprisingly, we observe that agents discovered in the math domain can be transferred non-math domains (Table(5).
While the performance of agents originally searched in the math domain does not fully match that of agents specifically designed for the target domains, they still outperform (in Reading Comprehension and Multi-tash or match (in Science) the state-of-the results illustrate that Meta Agent Search can discover generalizable design patterns and agentic systems.
hand-designed agent baselines. These
C PROMPTS
We use the following prompts for the meta agent in Meta Agent Search, Variables in the prompts that vary depending on domains and iterations are highlighted.
We use the following system prompt for every query in the meta agent.
System prompt for the meta agent.
You are helpful assistant. Make sure return in WELL-FORMED JSON object.
We use the following prompt for the meta agent to design the new agent based on the archive of previously discovered agents.
Main prompt for the meta agent.
You are an expert machine learning researcher testing various agentic systems. Your objective is to design building blocks such prompts and workflows within these systems solve complex tasks. Your aim is to design an optimal agent performing well on [Brief Description of the Domain].
[Framework Code
[Output Instructions and Examples]
[Discovered Agent Archive] (initialized with baselines, updated at every iteration)
Your task
You are deeply familiar with prompting techniques and agent works the literature. Your goal to maximize the specified performance metrics by proposing interestingly new agents.
Observe the discovered agents carefully and think about what insights, lessons, stepping stones can be learned from them.
Be creative when thinking from related agent papers or academic papers from other research areas. Use the knowledge from the archive and inspiration from academic literature to propose the next inter- esting agentic system design.
the next interesting agent to try. are encouraged to draw inspiration
THINK OUTSIDE THE BOX.
The domain deseriptions are available in Appendices E and F and the framework code is available in Appendix D We use the following prompt to instruct and format the output of the meta agent. Here, we collect and present some common mistakes that the meta agent may make in the prompt. We found it effective in improving the quality of the generated code. These formatting prompts inspired by Lu et al. (2024a).
Output Instruction and Example.
Output Instruction and Example:
should capture your thought process for designing next function. In the "thought" section, first reason about what the next interesting agent to try should be. then describe reasoning and the overall concept behind the agent design, and finally detail
The first key should be ("thought"),
21

---

Published as a conference paper at ICLR 2025
the implementation steps. The second key ("name") corresponds to the name of your next agent architecture. Finally, the last key ("code") corresponds to the exact "forward()" function in Python code that you would like to try. You must write COMPLETE CODE in "code": Your code will be part of the entire project, please implement complete, reliable, reusable code snippets.
Here is an example of the output format for the next agent: ("thought": "#*Insights: Your insights what should be the next interesting agent. **Overall Idea: your reasoning and the overall concept behind the agent design. ##Implementation:** describe
implementation step by step.".
"name": "Name of your proposed agent".
"code": "def forward(self, taskInfoj: Your code here"}
## WRONG Implementation examples:
[xammles of potential mistakes the meta agent may make in implementation)
After the first response from the meta agent, perform rounds of self-reflection make the generated agent novel and error-free (Shinn et al. 2023 Madaan et al. 2024).
Prompt for self-reflection round
[Generated Agent from Previous Iteration]
Carefully review proposed new architecture and reflect following points:
1. "IInerrettingness* Assess whether your proposed architecture is interesting innovative compared to existing methods the archive. If you determine that the proposed architecture not interesting, suggest new architecture that addresses these shortcomings.
Make sure to check difference between proposed architecture previous attempts.
Compare the proposal and the architectures in the archive CAREFULLY, including their actual differences in the implementation.
Decide whether the current architecture is innovative.
USE CRITICAL THINKING!
Identify any mistakes you may have made in the implementation. Review the code carefully, debug any issues you find, and provide corrected version. REMEMBER checking "## WRONG Implementation examples" in the prompt.
2. **Implementation M
3. ##Improvement*:: Based on the proposed architecture, suggest improvements in the detailed implementation that could increase its performance effectiveness. In this step, focus on refining and optimizing the existing implementation without altering the overall design framework, except if you want to propose different architecture the current not interesting.
Observe carefully about whether the implementation is actually doing what it supposed to do. Check if there is redundant code or unnecessary steps in the implementation. Replace them with effective implementation.
-rr avoid the implementation heing similar the previous agent.
And then, you need to improve or revise the implementation, or implement the new proposed architecture based on the reflection.
Your response should organized as follows:
Provide your thoughts on the interestingness of the architecture, identify any mistakes in the implementation, and suggest improvements.
"reflection":
"thought": Revise your previous proposal propose architecture if necessary, using the format as the example response.
"name": Provide name for the revised or new architecture. (Don't 't put words like "new" or "improved" in the name.)
"code": Provide the corrected code improved implementation. Make you actually implement your fix and improvement this code.
22

---

Published as a conference paper at ICLR 2025
Prompt for self-reflection round 2.
Using the tips WRONG Implementation examples" section, further revise code. Your response should be organized as follows:
Include your updated reflections in the "reflection". Repeat the previous "thought" and "name". Update the corrected version of the code in the "code" section.
When an error encountered during the execution of the generated code, we conduct reflection and re-run the code. This process is repeated up to five times if errors persist. Here is the prompt we use to self-reflect any runtime error:
Prompt for self-reflection when a runtime error occurs.
Error during evaluation:
[Runtime errors]
Carefully consider where you went wrong in your latest implementation. Using insights from previous attempts, try to debug the current code to implement the same thought. Repeat your previous thought in "thought", and put your thinking for debugging in thouug.thought".
D FRAMEWORK CODE
In this paper, provide the meta agent with simple framework to implement basic functions, such as querying Foundation Models (FMs) and formatting prompts. The framework consists of fewer than 100 lines of code (excluding comments). In this framework, information into a namedtuple Info object, making it easy to combine different types of information (e.g., G.. FM responses, results from tool function calls, task descriptions) and facilitate communication between different modules. Additionally, in the FM module, we automatically construct the prompt by concatenating all input Info objects into a structured format, with each Info titled by its metadata (e.g., name, author). Throughout the appendix, we renamed some variables in the code to match the terminologies used in the main text.
encapsulate every piece of
Code 1: The simple framework used in Meta-Agent Search.
Namea tuple for holding task information
Info namedtuple (" Info', I' n'me'' 'author', 'content', 'iteration
Format instructions for FM response
FORMAT_INST lambda request_keys: f"Reply EXACTLY with the following JSON format. /n/str (request keys) AnDD NOT MISS ANY FIELDS AND MAKE SURE THE JSCN FORMAT Is CORRECT! \n"
Description of the role of the EM Module ROLE_DESC lambda role: f"You are (role),"
benkkoff.en_excepiee (backoff. openai Rateldmit&rror) (msg, model, system_message temperature)
11 def get_iscn_respone__free *.**
Function to get JSON response from GPT model.
15
Args:
16
msg (str): The user message.
model (str): The model to use.
systen_message (str): The system message.
19
temperature (float): Sampling temperature.
20
Returns:
dict: The JSON response.
23
**
24
25
return dict
class PM_Mudule:
23

---

Published as a conference paper at ICLR 2025
\"""
29
Base class for an module.
30
31
Attributes:
output_fields (list): Fields expected in the output.
name (str): Name of the EM medule.
role (str): Role description for the FM module. 35
model (str): Model to be used.
36
temperature (Eloat): Sampling temperature.
ia (str): Unique identifier for the FM module instance.
-"*
39
40
def init__ssel (self, fiells_ list, name: str, role='helpfui te -> None:
assistant', model= --tt..-
42
43
def generate_ (sclf, input_infos, instruction) -> str: \""
44
Goncrates prompt for the FK.
47
Args:
48
input_infos (list): List of input information on.
instruction (str): Instruction for the task.
51
Returns:
52
tuple: System prompt and user prompt.
53
An example of generated prompt:
""
56
You are a helpful assistant.
57
58
# Output Format:
Reply EXACTLY with the follewing JSON format.
61
62
Your Task: You will given some number of paired example inputs and outputs.
The outputs
66
**# thinking #1 by Chain-of-Thoughe hkFo (yourself):
Instruction:
69
Please think step by step and then solve the task by writing the code.
"""
72
73
return system_promppt prompt
34
def (sel,, input_infos: 1ist, instruction, iteration_idx--")
list (00
76
77
Queries the FM with provided input information and instruction.
Args:
input_infos (list): List of input information.
81
instruction (str): Instruction for the task. iteration_idk (int): Itcration index for the task.
82
Returns:
85
cutput_infos [Iist(Inf]]) Output information.
36
return output infos
24

---

Published as a conference paper at ICLR 2025
def re____rr____1___
91
return FM(self.agent_namg (self.id)"
92
def _aa__lllll__ (self, input_infos: list, instruction, iteration_idx--1) self.query (input_infos, instruction, iteration_idks _dx= iteration_idx)
95
96
class AgentSystem:
def (srward(self, taskInic) Inion|Info, strl:
\"""
99
Placeholder method for processing task information.
100
Args:
taskInfo (Info) Task information.
103
101
Returns:
105
Answer (Union [Info, strl): Your FINAL Answer. Return either a namedtuple Info or string for the answer.
"*"
107
pass
With the provided framework, an agent can be easily defined with a "forward" function. Here we show example of implementing self-reflection using the framework.
Code Self-Reflection implementation example
def (oear taskInfo)
Instruction for initial reascning
vot_initial_instrustte "Please think step by step and then solve the task."
Instruction for reflecting previous attempts and feedback improve
cot oot_reelect_inntruutee = "Given previous attempts and feedback, carefully consider where you could wrong in your latest attempt. Using insignts from previous attempts, try to solve the task better."
7
cot_module PM_Module ([' t'innknng'', 'answer'1, 'Chain-of-Thought))
Instruction for providing feedback and correcting the answer critic insiruciiott "Please review the answer above and criticize where might be wrong. If you are absolutely sure correct, output "True' in 'correct'..
critic_medule FMLModule ([' feedback', 'correct'), 'Critic')
Mama Maximum number of attempts
15
Initial attempt
cot jat_inputs_ [taskInfo]
thinking, answer cod_module (cet inputs, cot_ initial instruction,
19
for i in range (N max):
20
$ Get feedback and correct status from the critic feedback, correct critic_module [[taskinfo, thinking, answerl, oritic_instruction, i)
22
if correct.content break
"True';
23
Add feedback to the inputs for the next iteration cot_inputs.co extend (Ithinking, answer, feedback))
26
27
28
# Reflect previous attemps and refine the answer thinking, answer cot_medule (cot_inputs, coc_reelect_insteoe i 1)
25

---

Published as a conference paper at ICLR 2025
return
E EXPERIMENT DETAILS FOR ARC CHALLENGE
Test grid
Example Input- outttooupuu grid #1
Example Input- outttouttutuuu grid #2
A Anwee
Figure 4: An example task from the ARC challenge (Chollet 2019). Given the input-output grid examples, the system asked learn the transformation rules and then apply these learned rules to the test grid to predict the final answer.
An example task from the ARC challenge shown in Figure 4 In the ARC challenge experiments (Section 4.1. we represent the grids strings of 2-D arrays, where each color is represented by an integer. We instruct the meta agent to design agents that generate code as solutions rather than directly outputting answers. Additionally, (1) to test whether the generated code can solve the example grids and (2) to obtain the task's answer by applying the generated code the test grid. The accuracy rate calculated by the Exact Match between the reference solution and the predicted answer. The meta agent uses "gpt-4o-2024- 05-13" (OpenAl 2024). while discovered agents and baselines are evaluated using "gpt-3.5-turbo 0125" O 2022) to reduce compute cost.
provide two tool functions within the framework:
The domain description of ARC for the meta agent is shown below:
Description of ARC for the meta agent.
Your aim is to find an optimal agent performing well on the ARC (Abstraction and Reasoning Corpus) challenge.
In this challenge, each task consists of three demonstration examples, and test example. Each Example consists of "input grid" and "output grid". Test-takers need to use transformation rule Iearned from the examples to predict the output grid for the test example.
# An example task from ARC challenge:
## Task Overview:
You will be given some number of paired example inputs and outputs grids. The outputs were produced by applying transformation rule to the input grids. In addition to the paired example inputs and
26

---

Published as a conference paper at ICLR 2025
outputs, there is also one test input without a known output.
The inputs and outputs are each "grids". A grid is rectangular matrix of integers between 0 and color. 0 is black.
(inclusive). Each number corresponds Your task is to determine the transformation rule from examples and find out answer, involving determining the size the output grid for test and correctly filling each cell the grid with appropriate color or number.
The transformation only needs to be unambiguous and applicable to the example inputs and the test imput. doesn't need work for possible inputs. Observe examples carefully, imagine the grid visually, and try to find the pattern.
## Examples:
### Exampic 0:
imput = [10.0000000550000000000.0 10.000000000.00000000000 [0..000.4440.0....0000000 (0.0.0044..4444400000 10....3.00000 [0.0.0.3.5.0.000.00, [0.000.33.533333300,, [0.00003355000000000, [0.000000.5500000000,, 100,0000000000000000000000000000 [0.000.005500000000] output = [[0.0,0,0], [0,0,0.0], [0,0.0.4), [0.0.4.4), [0.0.3.3], [0.0.0.3], [0.3,3.3], [0.0,0.3], [0.0.0.]]. [0.0,0.0]]
### Example 1:
input= = [[000000000000050000000000,, 0tt.0..............0002000000000 [0.00002.55226600000,, 10.0.022.55000000000, 10.00.2.2.000.0002022222..0..010 10.0.666.5566000000,,, 10.00022.5500.00.00,,, [0.22200055200000000,, 10.00.22.550000000000 [0.00000.550000..000 output [[0,0.0.0], [0,0,0,2], [0.0,6,21, [0,0.0.2,, [0,2,2,1]. [0.0,6.6], 10.0,0.21, 10,2,2,1, [0.0.0.]]. [0.0.0.0]]
## Example 2:
input = [mpp===[f..0000055.00000000 00000050000000000000000000000000 10.0.0.0.5588800000,, 10.000.8.5.0000000,,, 10.0008.5.00000000011,, 10.00000.5.877.000011.. 10.0000000000......0000 output= [[0,0,0,0], [0,0,0,7], [0,0,0,8], 10,0,0,8), 10,7,888, 10.0,8,8). (0.0,0.8, 10,0,0,8), 10.0,7,8), [0,0,0.0]]
1.0.0000000.....0.00000000000000 00000000000000000000000000000000 10..
### Test Problem:
[popt=[[000.000.5.0000 10.000.1.500..00000,, 10.000.1.5.1000.00. 101.I.155...1..611 [0.000.6.5.66660000), [0.0I0.0.5.II111.00... (0.000.1.5.0.0.0000,, (0.0.0.1.5.16660000), [0.000.0.5.6.0.0000,, 10.00000.000..0
Analyze the transformation rules hased on the provided Examples and determine what the output should be for the Test Problem.
Here we present the best agent on ARC discovered by Meta Agent Search.
Code 3: The best agent on ARC discovered by Meta Agent Search
Structured Feedback and Ensemble Agent
def forward (self, taskInfo):
# step 1: Genorate initial candidate solutions using multiple FM Modules
mnitial_instruetice
'Please think step by step and then solve the task by writing the code.'
num_candidates 5 # Number of initial candidated Sittial_modula [FM__oddu]]]]
'code' 1, 'Initial Solution', Lemperature=0.8) for in range (num cauuidaanddates)
solitiaosolutions
for i in range inmmccandidates)
thoughts thinking, code thoughts thoughts [1]
[i] ([taskInfo), initial_instructin
11
12
feedback, correct_cxamples wrong_examples = self.
run_ can__exmples_aae and__ge__feedbac (code)
if mantcorrettexxmppl. passed least one example
> 0: Only consider solutions that
thinking, 'code's code, 'feedback': feedback, ccorrect_count' len(
correct_exampled
Step 2: Simulate human-1 feedback for each candi dadi solution
27

---

Published as a conference paper at ICLR 2025
human like Feedback module FM Module (oduul(("thinkingn l'thinking',, Human-like Feedback', temperature-0.5) htt.__ffeccacc_ssssstttticce 'Please provide human-like feedback for the code, focusing on common mistakes, heuristic corrections, and best practices.'
'feedback') ,
18
20
for sol in initial initial_solution solallsssotiin::
21
thoughts = auman_like htt__/eeecddddolleeeeee ([aasIIafo, soll tolltthinking", ), code' J., seeaanneeeeeeeddeeeeeeee_ instruction) human thinkigh human feedback thoughts (0), thoughts [cl["huan__eeedacc
Juman_feedback
24
25
Step 3: Assign expert adviscrs to evaluate and provide targeted feedback
expert_roles l'Efficiency Expert', 'Readability Expert', Simplicity Expert'l
27
expert_advisors = [oduMe ([" thinkiing',, 'feedback'] ), role, tcmperature=0. 6) for role in expert_rolcs] expert_instruction 'please evaluate the given code and provide targeted feedback for improvement.'
29
30
for sol in
31
sol_ieedback = ()
for advisor in expert adviscrs:
33
thoughts advisor (ItaskIIn,, sol I' ttinkinngg ), sol l' code" ]],
expert_instriction)
34
thinking, feedback theughts[0] [0], thoughts [1]
sol__eedback [advisor...lei..ee....... sol (eeeepertt_ffeebax"
feedback
sol_ feedback
37
38
Step 4: Parse and structure the feedback to avoid redundancy and refine the solutions iteratively
max_rofiiemen__tteraiioo =
40
refinement_module MMLModule (I' thinking'' 'code'), 'Refinement Module', temperature-0.5)
41
re refined_sclution []
for sol in initial_soluta
44
for range (max refinement iterations):
45
combined feedback soll' feedback' 1. c.content soll'
human feedback' .conentt + " "'.join contecontentt for fb in vvalues ()])
structuree_foccaack '. (set (comblned__eecdaack splitt
# Avoid redundancy
47
UUsing the structured feedback, refine the solution to improve its performance. thoughts aeriinement_mcdulee [taskInfo, sol s'hhnnkikingg ], sol
refinement_instrue
'code' J, Info feedback', 'Structured Feedback', i)
feruuttured_feedbacc.. }) sefinement_instructiin
thougats thoughts feedback, correcttexxmples,, wrong_examples self.
refinement_tie ttiieement__thiiiingg refined_codd
run_evaappee__on______efffe (reedhaackreiiieed_coom (refined_code)
51
if len (entreeect__xxamlles)) 0:
52
sol.
refinement teffinemntttthnkiing, 'code'
refined_code 'feedback': feedback, 'correct_count"
len (ecoorrect_exmmppeeeee
refince_.oolltt.ooo..apaddd
54
55
step 5: Select the best-perferming solutions and make final decision using ensemble approach corted_sclutae sorted (refteedreiieed key=lambda x: x[' correct_counti reverse-True)
57
top_solutions
Select the top 3 solutions
28

---

Published as a conference paper at ICLR 2025
final _eseisddeiiiionncce instruction 'Given all the above solutions, reason over them carefully and provide final answer by writing the code."
60
refinement_module (I"thnkking 'code'), Final Decision Module', temperature-C.1)
final_inputa [taskInfo] [item for solution in top_solutions for
item in [solution]" (thiikiinnti solution c'doe' ], solution[' teedback' ]1]
decniidecssinnniiiiee modille (final (final_inpts,
final_ dww.sieniinnnooeeeee instruction) insteueeeiioo
final_thinking, final_code final final_thoughts [0], final_ tinal_thoughts answer
64
putttttoouuutttte foo___rrm_oooooo__ (final_code) return answer
65
F EXPERIMENT DETAILS FOR REASONING AND PROBLEM-SOLVING DOMAINS
To reduce costs during search and evaluation, we sample subsets of data from each domain. For GPQA (Science), we use GPQA diamond and the validation set consists of 32 questions, while the remaining 166 questions form the test set. For the other domains, the validation and test sets are sampled with 128 and 800 questions, respectively. We evaluate agents five times for GPQA and once for the other domains to maintain a consistent total number of evaluations. Each domain uses zero- shot style questions, except DROP (Reading Comprehension), which uses one-shot style questions following the practice in (OpenAl 2023). The meta agent uses "gpt-4o-2024-05-13"" (OpeAA 2024). while discovered agents and baselines are evaluated using "gpt-3.5-turbo 2022 to reduce compute cost.
OOenAA
We present the description of each domain we provide the meta agent.
Description of DROP (Reading Comprehension).
Your aim is find an optimal agent performing well on Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (DROP), which assesses the ability to perform discrete reasoning and comprehend detailed information across multiple paragraphs.
## An example question from DROP:
You will be asked to read a passage and answer a question.
Passage:
Non-nationals make up more than half the population Bahrain, with immigrants making up about 55% of the overall population. Of those, the vast majority come from South and Southeast Asia: according to various media reports and government statistics dated between 2005-2009 roughly 290,000 Indians, 125,000 Bangladeshis, 45,000 Pakistanis, 45,000 Filipinos, and 8,000 Indonesians.
Question: What two nationalitics had the same number of people living in Bahrain between
2005-2009?
Answer [Not Given]: Pakistanis and Filipinos
Description of GPQA (Science) for the meta agent.
Your aim an optimal agent performing well on GPQA (Graduate-I.eve Google-Proof Q& Benchmark). This henchmark consists of challenging multiple-choice questions across the domains of biology, physics, and chemistry, designed by domain experts ensure high quality and difficulty.
## An example question from GPQA:
Two quantum states with energies El and E2 have lifetime of 10 9 see and 10 8 sec, respectively. We want to elearly distinguish these energy levels. Which one of the following options could be their energy difference so that they be elearly resolved?
29

---

Published as a conference paper at ICLR 2025
Answer choices:
10TH eV
10 8eV
10 7eV
10 6eV
Correct answer [Not provided]:
10 7eV
Explanation [Not provided]:
According to the uncertainty principle, Delta E* Delta t=hbar/2. Delta is the lifetime and Delta E is the width of the energy level. With Delta t=10 9 >=> Delta E1= 10 7 ev. And Delta t=10 11 gives Delta E2-3.310 8 eV. Therefore, the energy difference between the two states be significantly greater than 7 ev. So the answer
ev.
Description of MGSM (Math) for the meta agent.
Your aim find optimal agent performing well the Multilingual Grade School Math Benchmark (MGSM) which evaluates mathematical problem-solee abilitics across various languages to ensure broad and effective multilingual performance.
## An example question from MGSM:
*uestionn**:
C _ - M E
MI マットカナナ t= 9 tn AVD 11 غ . 4 CANTIIA
htt.://.wo...........e.......... AIF
Answer (Not Given)**; 348
Description of MMLU (Mult-task) for the meta agent.
Your to an optimal agent performing well on the MMLU (Massive Multitask Language Understanding) benchmark, a challenging evaluation that assesses a model's ability to amswer questions across wide range of subjects and difficulty levels. It includes subjects from STEM, social sciences, humanities, and more.
## An example question from MMLU:
Answer the following multiple-choice question.
The constellation.. bright W-shaped constellation the northern sky.
(A) Centaurus
Cygnus
Cassiopeia
(D) Copheus
G BASELINES
In this paper, implement five state-of-the-art hand-designed agent baselines for experiments
on ARC (Section 4 : (1) Chain-of-Thouue Thought (COT) (Wei al. 2022), (2) Self-Consistency with
Chain-of- -hainhof-Thought (COT-SC((Wang et al. 2023b), (3) Relf-Refine (Madaan al. 2024 Shinn al. 2023), (4) LLLM-Dbate (Du et al. 2023), and (5) Quality-Diversiin a simplified version Intelli- gent Go-Explore (Lu al. 2024c).
In addition to these baselines, we implement two more for experiments on Reasoning and Problem- Solving domains (Section 4.2) (6) Step-back Abstraction (Zheng al. 2023) and (7) Role Assign- ment (Xu et al. 2023). An example implementation of Relf-Retine with our simple framework is
shown in Appendix
30

---

Published as a conference paper at ICLR 2025
In COT, we prompt the FM to think step by step before answering the question. In COT-SC, sample N answers and then perform an ensemble using either majority voting or an FM query. In Self-Refine, allow up to five refinement iterations, with early stop if the critic deems the answer correct. In LLM--ebatee, each debate module is assigned a unique role, such as Physics Expert Chemistry Expert, and the dehate lasts for rounds. In Muality-Diversity, three iterations to collect diverse answers based on previously proposed ones. In Role Assignment, use FM query to first choose role from a predefined set, and then use another FM query to answer the question by acting within the chosen role.
conduct
H EXAMPLE AGENTS
In this section, we present the detailed implementation of three example discovered agents by Meta Agent Search shown in Figure [ The "Multi-Step Peer Review Agent" and "Divide and Conquer Agent" were discovered during the search in the Reading Comprehension domain (GPQA) (Rein itt al. 2023), while the "Verified Multimodal Agent" was discovered during the search in the Math domain (MGSM) (Shi et al. 2023).
Code 4: Example discovered agent: Multi-Step Peer Review Agent
def forward taskInfo):
mnitial_instructinn = "Please think step by step and then solve the task."
critique_instruatio
"Please review the answer above and provi de feedback on where it might be wrong. If you are absolutely sure it is correct, output "True' in 'correct'."
refine instruction "Given previous attempts and feedback, carefully consider where you could go wrong your latest attempt. Using insights from previous attempts, try to solve the task better." dimal_ddcisionnnnnsttttt
= "Given all the above thinking and answers, reason over them carefully and provide final answer."
FM_modules [FM
'answer',, 'EM Module', role= role) for role in ('Physics Expert', 'Chemistry Expert', 'Biology Expert', 'Science Generalist' critic critic_medules modules [FM (1 Hledbbbac'', 'correct'), 'Critic', role=
zole) for role in L'Paysics Critie', 'Chemistry Critie', 'Biology Critic', Generall Critic']] final_ lel_e_diiio_modllee Decision', temperature-0.1)
#M_moduled (M_mcdule[['thinkkii l'tinink 'answer'), "Final
11
all ali_thinning = [[] for range (len (FM modules) ]
12
all answer [[] for in range (len (FM [FM_modules)) ]
all fcedback = [[] for in range (len (FM_modulcs))
for range (len (FM_modulcc))
thinking, answer FM medules [i] ([taskInfo], initial_instruationn all all_thinking aapendd (thinking) all all_answer[ [i] append (answer)
20
for range (Len (FM modules)
21
for j in range (len (FM modules)
22
if i !- j:
feedback, correct critic_modules [j] ([taskInfo,
all_thinning(ii) [0], all_answer [i] [0]],
critique_instructe
24
all all_feedback(1] (faeddd
25
for i in range (FM (FM_modules))
27
refine_inputs [taskInfc, all_thinking [1] (0), all_answer [4] [0]]
all_
28
thinking, answer = [M_modules(i] (refine_inpute,
iesinneiisttuctt all all_thinking [i] (..pan (thinking)
31

---

Published as a conference paper at ICLR 2025
all answer [i] append (answer)
31
32
final_inputs = [taskInfo] [all_thinking [1] for i in range (Len
[M_modules) }] + [nswer]][]]]]] for 1 in range (len (FMLmoculcss)
thinking, answer fiaal_decissinnmmodiile (f
fina__eccisioo_instrrtttt
34
35
return answer
Code 5: Example discovered agent: Divide and Conquer Agent
def forward (self, taskinfo):
Step 1: Decompose the problem into sub-problems
"please decompose the problem into smaller, manageable sub-problems. List each sub-problem clearly." decomposition_modiee module = FM Module l'thinking', "sub_problems") Decomposition Module')
Step 2: Assign each sub-preblen to specialized expert = "Please think step by step and then solve the sub-problem."
7
sub_ wwwplleeninsrrueeeeeeeeeeeeeeeee
special zed_experts - ([__Modlle((thinknnng
x 'sub_salution's Specializer Expert', role=role) for role in ('Physics Expert', Caemistry Expert', 'Biology Expert', GGeneaal Expert'
# Step 3: Integrate the sub-problem solutions into the final answer integration_instrusttoo = "Given the solutions to the sub-problens, integrate them provide a final answer to the original problem.
12
integration_modiee = FM MM_ulel ('thinking', 'answer'), "Integration Module', temperature=0.1)
Decompose the proplem
15
thinking, sub__rrul pub_pprolems = decomposition_modue (oodulee[(taan ([taskInfo),
knsure sub_probleng problems
string and split into individual sub-
18
sub problems list sub pwoblems..ee.oepe content..... content.split (' \n() if isinstance sub problems. content,, str) else []
20
Solve each sub-problem
21
sub_solutions = []
22
for i, sub_problem in enumcrate peroblems_listee
sub_prablem_info = I'
decomposition_modui module....
-repr_() sub_problem,
24
sub_thinking, sub sub_ sub_solution specialized _ [i & len
[ssu__problen_inff
sub wtt.ppobbemiissstrrrrep sub soo_sclutions..o append (sub (sub_schuue sclution)
26
27
Integrate the sub-problen sclutions integration_inputs [taskInfe] + sub_solutions
28
thinking, answer integration_moduu (integratiio_innutss. integration instruction)
30
31
return answer
Code 6: Example discovered agent: Verified Multimodal Agent
def forwardseeff taskInfo):
Instruction for generating visual representation of the problem *please create visual representation (e.g., diagram, graph) the given problem.
32

---

Published as a conference paper at ICLR 2025
Instruction verifying the visual representation corffiiaaion_instuuutee "Please verify the accuracy and relevance of the visual representation. Provide feedback and suggestions for improvement if necessary."
Instruction for solving the problem using the verified visual aid cot_instruction = "Jsing the provided visual representation, think step by step and solve the problem."
Instantiate visual representation module, verification module, e, and Chain-of-thought module vi sual_mcdulee FM_Module ([' (['visual'), /Visual Representation Module')
verification = FM Module Verification Module')
feedback', 'verified_visin ('eedba''
cot_ vot_module_ Module ([' thinking', 'answer') ) Mhain-of-Thoughe Module')
Generate the visual representation of the problem visual_cutput visual sesssualuulleeeeeeeeeeeeeeeeaeae ( ([takklffo) visual_instruction) visual_represnntaiin visual _isual_cutput[00 [0] Using Info object directly
18
Verify the visual representation
21
feecback, verified visual verification module (ItaskInfo, visual_reprrsennattion]] verification_instrettiie
22
Use the verified visual representation solve the problem thinking, answer cot cot_module ([taskInfo, verified_visuu visual), cot cot_instruction)
25
return answer
I PSEUDOCODE OF THE META AGENT SEARCH
In this section, we provide the pseudocode for the Meta Agent Search algorithm to clarify its im- plementation and workflow. The pseudocode outlines the iterative process of designing, evaluating, and refining agents using meta agent, as described in the main text.
Algorithm Meta Agent Search Algorithm
1: Input: Target domain validation data, maximum iterations N
2: Output: Archive discovered agents
3: Initialize archive AA with baseline agents (e.g., Chain-of-Thougeee Self-Refine'
4: for = 1 to N do
5: Design Step: Meta agent generates new agent:
6:
(a) Outputs design reasoning
7:
(b) Implements the design in code
8:
(c) Performs two self-reflection steps to ensure novelty and correctness
9:
Evaluation Step: Evaluate the new agent on target domain validation data:
10:
(a) If the agent produces eтrоrs during evaluation, refine the design up to 5 iterations
(b) Re-run the evaluation after each refinement
12: Update Step: Add the refined agent and its evaluation metrics to the archive A
end for
14:
Return: Final archive A
J IMPACT OF INITIALIZATION
One of the key claims of our work is that the code space representation allows for better utilization of existing human efforts (Section 2 enabling more efficient search process than starting entirely
33

---

Published as a conference paper at ICLR 2025
from scratch. To further investigate the effects of initialization, conducted experiments where the Meta Agent Search algorithm was run without any initial agent designs, contrasting with our standard approach that incorporates human- demannde solutions into the search process.
The results, presented in Table 6 demonstrate that even without initial agent designs, Meta Agent Search discovers agents that outperform all hand-crafted baselines across all evaluated domains. This finding underscores the robustmess of our method, as it effectively leverages the inherent struc- ture of the code space explore and optimize agent designs.
Interestingly, while the inclusion of good initial solutions generally leads to improved performance, the math domain exhibited unique outcome: starting from scratch resulted in superior performance. We hypothesize that the absence of predefined design patterns in this case encouraged broader and more diverse exploration of reasoning strategies within the limited number of iterations. Such diversity appears particularly beneficial for math tasks, which demand flexible and varied approaches reasoning.
This observation opens up an intriguing avenue for future research: exploring how the choice and quality of initialization impact search effectiveness across different domains. For instance, would be valuable to identify conditions under which starting without initial solutions may yield perfor- mance gains, or to design strategies that combine the advantages of both initialization and broad exploration.
Table 6: Performance comparison of Meta Agent Search with and without initial agent designs across multiple domains. The results show that even without initialization, Meta Agent Search out- performs hand hand-designed baselines in all domains. However, incorporating initial solutions generally leads to better performance, except in the math domain, where starting without initialization yields superior results.
F1 Score
Accuracy (%)
Agent Name
Reading Comprehension
Math
Multi-task
Science
State-of-the-art Hand-designed Agents
Chain-of- Thoughh Wei 2022
64.2 +
65.4 +
29.2 + 3.1
COT-SC (Wang et al. 2023b
64.4 = 0.8
28.2 3 3.1
65.9 3.2
30.5 - 3.2
Self-Refine (Madaan
2024
59.2 =
27.5
63.5
31.6 -
LLM Debate (Du et al.
2023
60.6 + 0.9
39.0 00,0=30 34
65.6
31.4 3.2
Step-back Abstraction (heng al
2023
60.4 +
31.1 31.1-32 322
65.1
26.9 3.0
Quality-Diversioy (Lu al.
2024c
61.8 1 0.9
23.8 3.0
65.1 3.3
30.2 1 3.1
Role Assignment Xu 2023
65.8 - 0.9
30.1 3.2
64.5 - 3.3
31.1 + 3.1
Automated Design of Agentic Systems on Different Domains
Meta Agent Search (Empty Initialization)
73.9 = 0.9
67.5 + 3.3
68.5 - 3.3
32.7 + 3.2
Meta Agent Search
79.4 + 0.8
53.4 3.5
69.6 - 3.2
34.6 - 3.2
K CoST OF EXPERIMENTS
single run of search and evaluation on ARC (Section 4. costs approximately $500 USD in OpenAl API costs, while run within the reasoning and problem-solving domains (Section costs about $300 USD.
The primary expense comes from querying the "gutb-.5-turoo-0125" model during the evaluation of discovered agents. Notably, the latest GPT-4 model, "gpt-40-mini," is less than one-third the price of 1.pt-3.5-ttubbbl 5-tuubbo-0125""" and offers better performance, suggesting that could achieve improved results with Meta Agent Search at just one-third of the cost. Additionally, as discussed Section the current naive evaluation function both expensive and overlooks valuable information. We ticipate that future work adopting more sophisticated evaluation functions could significantly reduce the cost of ADAS algorithms.
34


Reviews:
[{"review_id": "akSWRvp4UB", "reviewer": "Reviewer_wS7u", "summary": "In the style of neural architecture search, the authors argue that the design of agentic systems can be automated and provide a system that is a step toward solving that problem. They show that their system can achieve strong performance on several benchmarks compared to hand-designed systems.", "questions": "Please respond to the above when possible as well.\n\n- What exactly is the flowchart for the system here? It would be good to see something like that covering all the details in Section 3 (something more detailed than Figure 1).\n- Do LLMs act as Turing Complete systems?\n- What is and isn't hand-engineered here? Maybe use Figure 3(b) to explain.\n- The system seems to build itself up bit by bit. This isn't exactly a nice optimization space, though, so how do you counteract local minima? Neural architecture search has quite a bit of focus on this (even the original NEAT is mainly focused on this).\n- Can you clarify why you are using grids to sample in line 267?\n- Can you compare this with Voyager? I'd like to see something differentiating it from generic code-generation tasks here.\n- It's maybe a bit of an unfair comparison but did you compare against a vanilla GPT-o1 (I don't view this is required here)?\n\nIf the other reviewers bring up no serious issues, I can see myself changing my score if you can address most of the above.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The work is decently well presented and does read like a paper one would expect to see at ICLR.\n- The topic is relevant, the paper is timely, and the results are somewhat strong.\n- I can see this naturally integrating with work trying to design better evaluators (LLM-as-a-Judge, Agent-as-a-Judge, etc.).", "weaknesses": "- My most serious concern lies with the novelty of this work. There are papers like Voyager out there which incrementally build up sets of tools for solving a specific task. Is this actually fundamentally different from such code-generating systems? Or just a new paint job?\n- The authors keep citing the Turing Complete nature of their proposal. Most things in AI are, in one way or another, Turing Complete, so I wonder if that is really a big deal here. Like everything else here, the Turing Completeness of their system is wholly dependent on the components of the actual system (here, whether the LLMs actually act as Turing Complete systems in practice).\n- From the main text, it is quite unclear how their proposed system works exactly. Is it just a trivial loop writing a basic program?\n- I think you want to cite \"The Bitter Lesson\" alongside Clune in Line 50. It's more disconnected from the other works this one cites, so it is a bit less biased. While a blog post, it predates the cited work from Clune here and was quite influential. I'm sure there's earlier stuff that's good to cite here, too (this is one of the core motivations of this work, so it's worth fleshing out).\n- I like the presence of the research question on lines 88-89. But given that it's so important to the text, maybe polish it a bit by refining it (maybe just drop everything from \"rather\" onwards).\n- When bolding results, you need to bold everything that has overlapping error bars. Statistically, the results probably wouldn't pass a t-test and that's normal in the field, but we still can't get away with totally ignore statistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FyK8D5A529", "reviewer": "Reviewer_MJM9", "summary": "The paper introduces a new research area called Automated Design of Agentic Systems (ADAS), aiming to automate the creation of powerful agentic systems by inventing novel building blocks and combining them in innovative ways. The authors propose an approach where agents are defined in code, allowing a meta-agent to program new agents iteratively. They present an algorithm called Meta Agent Search, where a meta-agent utilizes Foundation Models (FMs) to generate, evaluate, and improve agent designs based on an ever-growing archive of previous discoveries. Through extensive experiments across multiple domains—including logic puzzles, reading comprehension, mathematics, and science—they demonstrate that agents discovered by Meta Agent Search outperform state-of-the-art hand-designed agents. Moreover, these agents show strong generalization capabilities when transferred across different domains and models.", "questions": "1. How does Meta Agent Search address potential biases or unsafe behaviors that might emerge from automatically generated agents? Are there mechanisms in place to detect and mitigate these issues?\n2. How does the approach scale with more complex tasks or larger domains? Are there computational limitations that need to be considered?\n3. Could the authors elaborate on how higher-order ADAS (e.g., improving the meta-agent itself) might be achieved and what challenges are anticipated in that direction?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The introduction of ADAS as a research area is novel, and framing the agent design problem as code generation by a meta-agent is a creative approach.\n2. The Meta Agent Search algorithm is well-conceived, and the experimental results are robust, showing consistent improvements over existing methods.\n3. The paper clearly articulates the motivation, methodology, and implications of the work, making it accessible to readers.\n4. Automating agent design has the potential to accelerate AI development significantly, and the demonstrated generalization capabilities suggest wide applicability.", "weaknesses": "1. While the paper mentions safety concerns, a more in-depth analysis of potential risks and mitigation strategies associated with ADAS would be beneficial.\n2. A deeper discussion on why certain baselines were chosen and how they were implemented would strengthen the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5tnq5awemr", "reviewer": "Reviewer_955X", "summary": "This paper proposes an approach that automatically searches building blocks and combinations of them as code generation for the Automated Design of Agentic Systems. Meta Agent Search, an iterative optimization method with foundation models generating new agents, is presented. The experiment shows that the Meta Agent Search is capable of inventing novel designs of agents. It is claimed that Meta Agent Search consistently outperforms state-of-the-art hand-designed agents in various domains. The optimized agents are shown to be transferable to different domains and foundation models.", "questions": "1. How should I interpret the legend \"E.g. LLM defines agents using code\" in the search algorithm box of Figure 2?\n2. What is \"the framework\" on line 265 referring to?\n3. What is the motivation for reporting the median accuracy instead of the mean? Is this a common practice? Could you also so the mean?\n4. The paper states that the method is compared with five state-of-the-art baselines, namely COT, COT-SC, Self-Refine, LLM-Debate, and Quality-Diversity. Could you elaborate in what sense they are state-of-the-art? Are they the best-performed known method for at least one of the tasks being tested? To my knowledge, the best handcrafted solution of ARC achieves at least 20% accuracy (see the reference below). Please specify the scope of your baselines.\n\nFerré, S. (2021). First steps of an approach to the arc challenge based on descriptive grid models and the minimum description length principle. arXiv preprint arXiv:2112.00848.\n\n5. How exactly is the meta agent implemented? Is it a single query of an LLM with a handcrafted template presented in Appendix C?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The work is well-motivated.\n2. The transferability results are interesting.", "weaknesses": "1. Some of the sentences are not written with care. For example, \"Given that programming languages are Turing Complete...\". The fact is that some programming languages are not Turning complete. This is awarded by the authors on line 106.\n2. It should be stated in the main text that the implementation of the meta optimizers is different for different tasks.\n3. As an optimization method, it is only compared with handcrafted baselines. More comparison with other optimization methods is expected.\n4. I think there are missing citations for self-referential higher-order meta-learning. See Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: The meta-meta-... hook. Diploma thesis, Institut für Informatik, Technische Universität München, 1987 and its follow-up works.\n5. I think there is a lack of explanation of how the work illustrates the potential of this direction to benefit humanity, which is claimed in the abstract. \n6. The optimization starts with the baseline method as the initial solution. For a fair comparison, I think the optimization result starting without any manually optimized initial solution should be presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mGk8j5lLsW", "reviewer": "Reviewer_gtAT", "summary": "Through this paper the authors propose to \"form\" a new research area they call \"Automated Design of Agentic Systems\", in which the high level general idea is to automate the generation of \"new agents\" (in this case the word agent is loosely used for a piece of code, apparently), that are continuously evolved.", "questions": "no specific question.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- Thought-inducing subject, as the authors propose to work on \"agent-generating agents\".\n- Contemporary work, following the trend of developing approaches to improve inference once the LLM is already trained, which makes it of interest to a good portion of the community.", "weaknesses": "- The biggest and greatest critique to this paper is that on the high level the authors are proposing to create a sub-community dedicated to exploring what could be seen as a primitive form of self-replication. Given that in practice the proposed method is pretty much just a code generator (easy to isolate and secure by making the code run in a Domain Specific Language), I won't reject or reduce the score of the paper due to this point, but I urge the authors to reflect on whether if push for the development of such general \"Agent generation procedures\" is really a good idea. Conceptually speaking, it's not a big jump to go from an agent that generates code that generates other agents, to a robot that builds other robots.\n\n- Related to the first matter, such a research should have safety as a cornerstone of the design, with clear descriptions of how to make the agent generation safe. However, the only mention of \"safety\" throughout the paper is pretty much the discussion on section 6, that only talks about the \"danger of creating AGI\" (which in my opinion is totally eclipsed by the danger of self-replication in the case of this paper), and to which the author's only response is to push to future works to think of safety, which in my opinion is completely inappropriate. Every single building block of the algorithm should have been thought of in terms of safety and clear recommendations to avoid dangers should have been given.\n\n- The practical description of the method is very unclear and far too high-level in the main text. I cannot understand from the description exactly what type of code is generated and what are the building blocks that are given as prior to the agent. The expanded analysis of the related works should have been moved to the appendix to make room for a better explanation of the method. Overall, it seems like the authors' approach would require a lot of compute AFTER a lot of compute has already been dedicated to train the LLM, yet, the computational cost addition to the inference was not compared to the other, clearly less-computationally expensive, approaches such as chain of thought in table 1. The method has very limited applicability if it clearly cannot be applied to hundreds of billions of parameters even if you have the supercomputers for that.\n\n- The author statement \"community has not reached a consensus on the definitions or terminologies of agents\" is at the very least a weird one. Certain communities have been working on agents since the 80s and while there is always space to argue about certain gray areas of the definition, textbooks used to teach undergrad students already have a pretty decent definition for \"agents\", for example AI a modern approach. The authors seem to be too focused in the LLM subcommunity and disregarding past work and definitions that are relevant to the work they are trying to develop.\n\n- Unless the authors are part of openAI I suggest against using their logo in figure 2.\n\n- Despite being at some extent intriguing to think about some aspects of the authors' work, in my opinion the work is not very technically mature, with relatively simple and overly high level descriptions, and very limited experimentation. In my judgement this paper is more suited to an ICLR workshop.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)