AgentSquare: Automatic LLM Agent Search in Modular Design Space
Authors: Yu Shang
Yu Li
Keyu Zhao
Likai Ma
Jiahe Liu
Fengli Xu
Yong Li
Venue: ICLR 2025 Poster (2025)
Keywords: LLM agent
Modular design space
Agent search
AutoML
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a modular design space of LLM agents and an automatic agent search framework named AgentSquare.

Abstract: Recent advancements in Large Language Models (LLMs) have led to a rapid growth of agentic systems capable of handling a wide range of complex tasks. However, current research largely relies on manual, task-specific design, limiting their adaptability to novel tasks. In this paper, we introduce a new research problem: Modularized LLM Agent Search (MoLAS). We propose a modular design space that abstracts existing LLM agent designs into four fundamental modules with uniform IO interface: Planning, Reasoning, Tool Use, and Memory. Building on this design space, we present a novel LLM agent search framework called AgentSquare, which introduces two core mechanisms, i.e., module evolution and recombination, to efficiently search for optimized LLM agents. To further accelerate the process, we design a performance predictor that uses in-context surrogate models to skip unpromising agent designs. Extensive experiments across six benchmarks, covering the diverse scenarios of web, embodied, tool use and game applications, show that AgentSquare substantially outperforms hand-crafted agents, achieving an average performance gain of 17.2% against best-known human designs. Moreover, AgentSquare can generate interpretable design insights, enabling a deeper understanding of agentic architecture and its impact on task performance. We believe that the modular design space and AgentSquare search framework offer a platform for fully exploiting the potential of prior successful designs and consolidate the collective efforts of research community. Code repo is available at https://github.com/tsinghua-fib-lab/AgentSquare.

Published as a conference paper at ICLR 2025
AGENTSQUARE: AUTOMATIC LLM AGENT SEARCH IN MODULAR DESIGN SPACE
Yu Shang'; Yu ; Zhao', Likai Ma', Jiahe Liu', Fengli Xu Yong Li 'Department of Electronic Engineering, Tsinghua University "Shenzhen International Graduate School, Tsinghua University
{ denglixu, 1iyong07 siiiigguaa.......... edu. en
ABSTRACT
Recent advancements in Large Language Models (LLMs) have led to a rapid growth of agentic systems capable of handling a wide range of complex tasks. However, current research largely relies on manual, task-specific design, limiting their adaptability to novel tasks. In this paper, we introduce new research prob- lem: Modularized LLM Agent Search (MoLAS). We propose a modular design space that abstracts existing LLM agent designs into four fundamental modules with uniform I0 interface: Planning, Reasoning, Tool Use, and Memory. Build- ing on this design space, we present novel LLM agent search framework called AgentSquare, which introduces two core mechanisms, i.., module evolution and recombination, to efficiently search for optimized LLM agents. To further ac- celerate the process, we design performance predictor that uses in-context sur- rogate models to skip unpromising agent designs. Extensive experiments across six benchmarks, covering the diverse scenarios of web, embodied, tool use and game applications, show that AgentSquare substantially outperforms hand-crafted agents, achieving an average performance gain of 17.2% against best-known hu- man designs. Moreover, AgentSquare can generate interpretable design insights, enabling a deeper understanding of agentic architecture and its impact on task performance. We believe that the modular design space and AgentSquare search framework offer a platform for fully exploiting the potential of prior successful designs and consolidate the collective efforts of research community. Code repo is available
INTRODUCTION
The past few years have witnessed remarkable progress in the development of Large Language Mod- els (LLMs) (Achiam et al. 2023
Touvron al. 2023). giving rise to the proliferation of numer- ous agentic systems (Weng 2023: Shen et al.] 2024). For example, "chain-of sociin-of-hhouree thought" prompting
has unlocked the general-purpose reasoning capabilities of LLMS (Wei al. 2022), and memory mechanisms have been proven effective in simulating human behavioiur (Park et al. 2023). These emerging LLM agents have demonstrated astonishing abilities transform wide range of tasks,
cluding solving mathematical problems Gkomera-Paredes et al. 2024), navigating the web (Nakano) al. 2021). providing financial advice (Ding et al., (2024)) and informing medical decisions (Li et al. 2024a). Therefore, the design of agentic systems plays crucial role in harnessing the power of LLMs for various downstream applications.
However, current research predominantly relies manually designed agentic systems tailored for specific tasks, which often depend heavily on expert insight and intensive human labor. Furthermore, these task-specific agent designs frequently struggle to adapt to novel tasks. A few recent studies have explored using LLMs to rewrite and optimize the prompts of existing agents (Fernando et al.
"Equal contribution. Corresponning author.

---

Published as a conference paper at ICLR 2025
Diverse agents
Standardized agents
Module poo!
Agent Square
Performance
Evaluation
im letion
Tool Use
Sef-dffiing
perpeealpurpose R casoning
Planning
Reasonnng
Tooluse
Memoory
Figure I: AgentSquare is a modular framework for designing and optimizing LLM agents.
2024 Yang al. 2024). A more recent work introduces the idea to leverage LLMs to search the entire agentic systems defined in code space (Hu et al. 2024), enabling the discovery of agents with more flexible prompts, control flows, etc. However, these previous approaches are limited in their ability to explicitly recombine the strengths of agentic modules discovered by different researches and located in separate codebases. Another line of research focuses on optimizing the configuration of muuti--geent systems (Chen et al., 2023] Yuan et al., 2024, Li et al., 2023 Zhuge et al.) [022]. Wang et al. 2023b). These efforts are orthogonal to the optimization of single-agent systems, as they focus more on the role-playing and interaction patterns among multiple agents, rather than the design of agentic modules.
This paper addresses a novel research problem Modularized LLM Agent Search (MoLAS). The goal is to automatically optimize LLM agent designs by leveraging the experience of published or evaluated modules. Therefore, the core of our work is a modular design space for L.M agents, comprising 4 categories of modules: Planning, Reasoning, Tool Use, and Memory. This design space is abstracted from a thorough literature review of existing agentic systems (details provided in Section 2). It is important to note that our goal is not to propose the most comprehensive, one-size- fits-all LLM agent design space, but rather to demonstrate that our modular design space enables researchers and intelligent search algorithms to fully exploit the potential of prior successful designs. MoLAS is a guided and constrained searching problem in the modular design space, which subset of the entire code search proposed in ADAS (Hu et al. 2024). However, MoLAS has nice feature of providing standardized I0 interfaces for agent modules, facilitating easy recombination of modules from different agentic systems and hence enabling efficient search for novel agents. Our design space is also highly extensible, allowing new agentic systems to be integrated as plug- in modules. Therefore, it provides platform to consolidate the collective efforts of the research community on LLM agents. The overview of this work is illustrated in Figure []
Building this modular design space, propose novel LLM agent search framework called AgentSquare. Specifically, AgentSquare optimizes LLM agents through the mechanisms of module evolution and recombination. The module evolution mechanism leverages an evolutionary meta- prompt to explore new modules through prompt-level optimization, which jointly models task de- scriptions, existing modules. and the performance of evaluated modules. Besides, the module re- combination mechanism performs modulc-level optimization by leveraging the reasoning power of LLMs to strategically search for promising module combinations. To reduce the expensive eval- uation costs of LLM agents, we further introduce a performance predictor that implements an in- context surrogate model for newly proposed LLM agents, enabling us to skip unpromising candi- dates and significantly accelerate the search process.
We conduct comprehensive evaluations on six widely adopted benchmarks, covering diverse use cases in web, embodied, tool use and game scenarios. Our experiments show AgentSqaure can discover novel LLM agents that outperform hand-crafted agents across all six benchmarks, scor- ing an average performance gain of 17.2% compared the best known human designs. Besides, AgentSqaure also surpasses other search algorithms in terms of having a steeper optimization tra- jectory. More importantly, case studies reveal that AgentSquare can provide human interpretable design insights for newly discovered, good-perforning agents.
The key contributions of this work follows:
2

---

Published as a conference paper at ICLR 2025
prom Naysne specens USI
Task Description
feedback
Agentic Workflow
Planning
Environment
bulput
Subers
Subart
Subartc
Memmayy Deralese
Rcorove
Memory
feedback
Abstract
Reasoning
Add
Tool
act
dass Planigg
i lax deverptees 0.5g1
liac iben detiptin
Module
select
relum
Dupal: reuchigg toult
Marayy
Task Descriptin
Tool Pool
dua Foolae3:
Resources
scl cuattr)
Dupi: bo repoets
Ougurr enlove)
Modular Agent Design Space
S ttandard 1O Interface
Figure 2: Illustration of the modular agent design space and agentic workflow (left) and the stan- dardized 10 interface of four types of modules (right).
We propose a novel modular design space for LLM agents, enabling researchers easily build on previous successful designs and accumulate new discoveries as a community.
We design the AgentSquare framework that efficiently searches for novel and good- performing I.IM agents via the novel mechanism of module evolution, module recom- bination, and performance predictor.
Experiments across six diverse tasks show that our method discovers novel LLM agents that outperform all known human designs. Besides, AgentSqaure can generate human interpretable design insights for these novel agents.
A MODULAR DESIGN SPACE OF LLM AGENTS
2.1 BACKGROUND
Using LLMs for automatic optimization has been a widely explored topic, such as applications in code generation (Lehman et al.] 2023; Romera-Paredes et al. 2024) and neural architecture search (Nasir et al.
2024 Chen et al.) 2024a). There are several recent studies that explore the problem of prompting L.MSs to design LLM agentic systems. OPRO Yang et al., (024) and Prompt- breeder [Fernando et al. 2024) can be viewed as leveraging the reasoning power of LLMs to improve the prompt of LLM agents. More importantly, ADAS introduces the idea of searching the entire agentic system defined code space, and propose a Meta Agent Search algorithm that discovers LLM agents outperforming of-the-rtt human designs (Hu et al. 2024). Our main difference and contribution lie in introducing modular design space for LLM agents, which can provide a standard framework support the convenient reuse of existing successful agent components and fruitful innovative agent module discovery.
modular design space for LLM agents facilitates the reuse of prior successful designs and sup- ports the exploration of new architectures. At the core of such modularization is the standardization of input-output interfaces, which ensures both extensibility and seamless integration with existing designs. Many experts in the field have proposed building LLM agentic systems with key modular components from engineering (Weng) 2023) and cognitive perspectives (Sumers al.) 2023). How- ever, these proposals remain largely conceptual, lacking implementable solutions to unify existing LLM agents. Besides, current I.I.M workflow program frameworks (e.g., LangChain and Auto- GPT) only provide operation-leve level components, which cannot support module-level search that best exploits the potential prior successful designs.
To address these problems, we perform comprehensive literature review of publications from NeurIPS, ICML, and ICLR over the past three years. The review focuses on papers with the key- words "LLM". "Agent", or "Large Language Model" in their titles while excluding works related to multi-agent systems or agents that require additional training. Note that our aim not to propose the most comprehensive, one-for-all L.LM agent design space, but offer standardized framework
3

---

Published as a conference paper at ICLR 2025
that enables the recombination of existing agents and facilitates the discovery of new ones. As result, we sort out 16 popular LLM agents and abstract a modular design space with 1050 possible combinations, which can be easily extended when new modules are discovered. Below, we describe the agentic workflow and the function of four modules in our design space.
2.2 WORKFLOW OVERVIEW
The proposed agent workflow operates through an iterative process with the intercomnection of the above four modules, shown in F Upon receiving a task d, the agent starts with the planning module, decomposing it into n sub-tasks($1; . .. Next, these sub-tasks are passed to the reasoning module sequentially. Taking the sub-task 8; description explores to prompt LLMs to give the result. When reasoning encounters limitations in internal knowledge of LLMs, the tool use module is activated to select an appropriate tool from the pre- defined tool pool т, supporting problem-solving memory module which reads and writes necessary observations and experiences from a memory database mem to help reasoning. The reasoning result of each sub-task will be transformed into actions, guiding the agent to interact with the external environment. After all sub-tasks are finished or the reasoning process gets stacked, the agent will activate the planning module to adjust the plan with the received feedback. The agent conducts such a trial-and-en -nd-rrr- loop until the task is completed or the set maximum trial number is reached.
input, the reasoning module
Besides, the reasoning process also accesses the
Planning. The planning module is responsible for decomposing the targeted task into smaller sub- tasks. Given task description d and optional feedback information f. the planning module P strategically decomposes the targeted task into a sub-task sequence {5) S2...., s,} = P(d, f)- Such decomposition is critical for handling very complex tasks with long-term characteristics, espe- cially for agents in open-world environments such as MineCraft (Wang et al., c))
Reasoning. LLMs have exhibited remarkable reasoning abilities under advanced prompting ap- proaches such as CoT (Wei et al. 2022). ToT (Yao et al., [0024). and SoT (Shang et al. 2024). shaping the foundation of the intelligence of LLM agents. The reasoning module invoked to solve the sub-tasks sequentially after planning, which takes each sub-task and optional feedback information fi as input and outputs a solution r; R(s;, fi).
Tool use. The ability of using external tools (Shen et al. 22224] Schick et al., 2024) overcomes the limitations of the L.M''s internal knowledge during the reasoning process. Formally, given certain problem Pıy derived from the reasoning process of sub-task and pre-defined tool pool r, the tooluse module T selects the best-matched tool f.j to address the problem, denoted as fij T(P;, T), where tij T.
Memory. Memory plays a critical role by storing past thoughts, actions, and observations of agents (Park et al. 2023) Shinn et al. 2024). During the reasoning process, these internal logs are dynamically written to and retrieved from the memory database mem, controlled by the mem- ory module M. The writing process can be expressed mem Murute (o, mem), where denotes the current observations. The retrieval process is m = Mretrieve (o, mem), where m denotes the retrieved knowledge relevant the current situation.
AGENTSQUARE FRAMEWORK
3.1 PROBLEM FORMULATION OF MoLAS
In the proposed modular design space, an LLM agent A can be instantiated with the combination of a planning module P, reasoning module R. tooluse module T and memory module M, denoted as A = (P, R, T,, M). Given the task description d and the set of all possible modules with standardized 10 interface {F, I. T, MJ. We formulate an optimization problem for searching LLM agent architectures within the modular design space. The objective is to identify the optimal module combination in solution space defined by Cartesian product of four design dimensions to maximize agent performance. Let the performance evaluation function the task be Eval.(.). where the specific metric varies in different tasks discussed in Appendix A.I The optimization
4

---

Published as a conference paper at ICLR 2025
Task E Evaluation
Webshop Travciplanner
Attiorid
Scoworld
M3Tbol
65
Initial Agent
LLM
Episode
New Module
New
New Agents
Evolution
Recombination
Planning Reasoning
Memory
05
Spach carch
05
reeback
Module Foo
Experience Fool
Figure 3: Overview of AgentSquare search framework. AgentSquare optimizes LLM agents through the mechanisms of module evolution and recombination. We further introduce a performance pre- dictor that implements in- context surrogate model for efficient evaluation of novel agents.
problem of MoLAS is defined follows:
arg max PEP.NERTTET,ME
(1)
3.2 AGENTSQUARE SEARCH ALGORITHM
Solving the optimization problem of MoLAS features three key challenges: (1) The search space, defined the Cartesian product of four orthogonal modules, is vast and hard explore; (2) the module sets encompass any code with standard I0 interfaces, making the module selection an open- ended problem; (3) the high costs agent evaluation during the search process constrain the overall search scale. To tackle these issues, we introduce AgentSquare, an automatic search framework to optimize LLM agents within the modular design space. Facing the vast search space of MoLAS, propose module recombination operation utilizing LLMs to strategically reason to identify more promising module combinations. Such operation broadens the coverage of child samples, overcom- ing the limitations of prompt rewrite methods that explore only restricted space. However, only searching in the existing module combinations also narrows the search space, thus we propose mod- ule evolution operation which employs evolutionary meta-prompt to search new modules through code-level optimization. This operation, combined with module recombination, enables the search of any module combination in the open-ended solution space. Finally, mitigate the high costs of frequent evaluations of searched agents, we design performance predictor as an in-context surro- gate model for evaluating searched agents, significantly accelerating the search process and reducing real-valued costs.
The overall framework of AgentSquare is illustrated in Figure and the algorithm presented in Algorithm Next, we detail the key components of the AgentSquare search process.
3.3 INIIIALIZATION
Insights from existing AutoML studies indicate that a well-chosen initialization enhances warm-up and improves search efficiency by avoiding unpromising populations (So et al. 2019 Yuan et al. 2024]. AgentSquare starts by initializing a global experience pool P. Ro E T. E T, My E M) to seed agents that are well-designed (as mentioned in Section 2) along with their realevalued performance v. The module pools {IP, R. T, M+ are set to the standard- ized modules extracted from these seed agents.
[P, R, T. M, v Pa E
5

---

Published as a conference paper at ICLR 2025
3.4 MODULE RECOMBINATION
Given the vast solution space of MoLAS, relying solely on prompt rewriting leads to a lim- ited exploration confined to the neighbor of the initial state. To expand the exploration space, we propose leveraging LLMs as a self-adaptive proposer, which iteratively reason to identify promising module combinations with accumulated experience beyond the original agent config- uration. Denote the initial agent of the recombination phase as A = (Po. Ro. To, Mo), where Po E P. Ro E RR Tu T, Mp E M. The module combination proposer LLM wo incorporates targeted task description d, existing module pools {P, R, T, M and the performance experience of searched module combinations E to propose promising new agents Ap:
A, = xo ((P( Ro. To, Mo), d, N. R. T,, E.
Based on the initial agent configuration AG, the LLM proposes N offspring {A,, 42, replacing certain modules of A with alternatives from the module pool. For instance, a possible solution could be (Po, R To. Mo). where R' E R. is different reasoning module selected from the module pool. Then, the created N new agents are evaluated with a performance predictor xp (detail in Seciton 3.6) and the best one goes to the next episode as initialization.
42, ..., AN) by
3.5 MODULE EVOLUTION
As mentioned above, the solution space for each module type is open-ended, allowing any code with standardized I/O interface. Consequently, searching only with module recombination narrows the solution space and limits the upper bound of agent performance. To address this problem. we design module evolution operation with evolutionary meta-promppe to search for new modules through program-level optimization. This design is inspired by the iterative pipeline of FunSearch (Romera- Paredes et al. 2024), which prompts LLMs to propose new solutions based on the target problem and performance feedback from existing solutions. Building on this concept, we introduce a module- programming LLM TT to conduct agent search in our modular design space by jointly modeling task descriptions, existing modules, and the performance of previously evaluated modules. Please note we reuse parts of the open-source code from ADAS (Hu al. 2024) to implement the optimization procedure. Leveraging LLMs to search in the modular agent design space has several appealing advantages. Compared with the unconstrained design space of L.LM agents, searching functional modules can produce a more focused and fruitful search space. Additionally, integrating existing successful module designs with standard I0 as in-context examples can better elicit the reflective reasoning abilities of LLMs to identify previous key designs to help propose innovative ones. Denote the initial agent in the module evolution stage A (Pá: Rg To, Mý), the module programmer LLM produces a population of child agents by evolving current modules of A Formally the module evolution operation is denoted follows:
A = (e((Pp Rg. Tú. Mo), N,P,,,,,,,
(3)
The created new modules are appended to the standardized module pools (P,R, T, M) and each module is used individually mutate the initial agent, resulting in N child agents (AL A ...... A }- For example, ( (P*, Ro. To. Mo) represents a solution where the planning module is mutated into a new variant pa. These child agents are then real-tested and updated to the historical experience pool E. The best-performing one is selected as the initial agent for the subsequent recombination phase.
3.6 PERFORMANCE PREDICTOR
The last challenge in automatic agent search is the high API cost incurred during the evaluation of each candidate agent. Many agent tasks require multiple steps and involve substantial input and output tokens, leading to prohibitive evaluation costs. For instance, evaluating a simple CoT agent based on GPT-4o in ALFWorld (Shridhar et al. (021) requires around $60, making the agent search economically unsustainable at scale. To tackle this issue, we propose incorporating an additional LLM Tp performance predictor to serve as an in-context surrogate model for novel agent eval- uation, enabling the exclusion of unpromising candidates and significantly accelerating the search process. Compared to real environment evaluation, such in-context surrogate model requires significantly fewer tokens, making it more cost-efficient and supporting larger-scale searches. Sim- ilar approaches have been effectively applied in neural architecture search (NAS), where LLMSs

---

Published as a conference paper at ICLR 2025
Web Webshop
Embodied
Tool
Game
Baseline Type
Method
ALFWorld SeiWorld M3Tool
Travel
PDDL
CoT
0,485
0,405
0.697
0,448
0,487
0.542
Cot-SC
0.512
0.426
0.656
0,461
0.413
0.495
Self-refine
0.461
0.567
0.654
0,442
0.000
0.514
ToT
0.501
0.437
0.741
0.453
0.380
0.476
Step Back
0.468
0.279
0.220
0.434
0.000
0.486
TP
0.398
0,404
0.576
0.387
0.430
0.518
Hand-crafted Agents
HuggingGPT
0,519
0.481
0.680
0.354
0.510
0.584
Voyager
0.366
0.425
0.776
0.247
0.523
0.412
Generative Agents
0.499
0.477
0.663
0,402
0.480
0.553
DEPS
0.481
0.459
0.740
0.278
0.540
0.591
OPENAGI
0.506
0.510
0.718
0.322
0.533
0.616
Dilu
0.451
0.433
0.682
0.475
0.360
0.463
Random
0.533
0.620
0.704
0,438
0.563
0.660
Module Search
Bayesian
0.549
0.634
0.749
0.502
0.537
0.650
Prompt Search
OPRO 0.505
0.380
0.569
0.309
0.523
0.589
Agent Search
ADAS 0.521
0.543
0.754
0.475
0.373
0.568
AgentSquare 0.607
0.695
0.781
0.524
0.583
0.669
Table 1: Performance comparison of searched agents from AgentSquare and (1) existing human- designed agents (2) module search baselines (3) prompt search baselines (4) agent search baselines based on GPT-4o on six tasks across different domains.
leveraged to evaluate the performance of generated network architectures (Jawahar et al., 2023 Chen et al. 2024a).
During the search process, newly created agents from module evolution are still tested in the real task environment because these new modules never appear in the experience pool, and it is unsuitable to use the performance predictor to provide predictions. During the module recombination operation, the newly proposed agents are evaluated by the performance predictor, which leverages in-context reasoning based past agent combination performance to provide efficient performance prediction. Here, given a newly searched agent A', the performance predictor Fp thoroughly considers task descriptions d, module profiles and in-context performance examples of previously tested agents E to score novel agents:
= ",P,
(4)
where u' is the predicted performance of the evaluated agent. Empirical results demonstrate that the predicted performance of agents closely matches their actual performance, verifying the effective- ness of the proposed performance predictor, which is detailed in Section 4.3
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Task setup. We conduct experiments on six representative tasks covering four domains: embodied, game, web and tool applications, which are widely adopted by existing LLM agent benchmarks (Ma
ct al. 2024 Xi et al. 2024), more details are presented in Appendix(A..
Baselines. We compare AgentSquare with four types of baselines including hand-crafted agents, module-level search, prompt-level search and agent-search methods. More details are presented in Appendix A.I
AgentSquare setup. We implement AgentSquare and conduct experiments using both GPT-3.5- turbo-0125 and GPT-4o (Achiam al. (023). To ensure fair comparison. of few-shot examples across methods. The initial agent is set as a random module combination, and the search process terminates after 5 consecutive iterations without performance improvement.
the same number

---

Published as a conference paper at ICLR 2025
AgentSquare Alfworld
ApentSquare Webshop
RATPPE 16
Genewzze Agents
0.50
090
Pbu
DEPS
Herrary Dil
Success rate
0.20
0.50
0.24
0.45
0.52
0.48
Iteration
Iteration
Figure 4: AgentSquare search trajectory on Alfworld and Webshop.
4.2 EXPERIMENTAL RESULTS
Main results. We conduct extensive experiments to compare our method against three types of baselines on six tasks and present results based on GPT-4o in Table [and results on GPT-3.5 in Ta- ble Additionally, we evaluate the agents' API costs and provide a performance-cost comparison in Figure A.7 to Figure A.12 From these results, we have the following observations:
AgentSquare can effectively discover better agents compared with human-designed agents. On the six representative agent tasks, the best agent searched by AgentSquare consistently out- performs designd agents in terms of performance. Specifically, as shown in Table T and Table A.3 compared with the best human-designed agent, AgentSquare achieves an average 14.1% performance improvement on Webshop, 26. 1 1% improvement on ALFWorld, 20.5% im- provement SciWorld, 30.6% improvement on M3Tool, 6.0% improvement on Travelplanner, 6.0% improvement on PDDL. Simultancously. the best agent from AgentSquare is commonly cost-efficient, which strikes the best performance-cost trade-off among all compared agents as seen in Figure A.7 FFiuure A.12 Since the search cost is a one-time expense and the searched modules can be reused, it is not included in the above analysis, but separately listed in Table 6
AgentSquare provides more efficient searching approach for LLM agent optimization. To further demonstrate the effectiveness of the search of AgentSquare, we compare three types of searching methods including module search, prompt search and agent search. Compared with the best agent crafted from these searching methods, AgentSquare achieves average 8.4% per- formance improvement on Webshop, 8.1% improvement on ALFWorld, 11.0% improvement on SciWorld, 12.8% improvement M3Tool, 2.5% improvement Travelplanner, 1.4% improve- ment on PDDL. The comparison of search-based methods is conducted with a fixed LLM token budget to ensure fairness by maintaining the same number of search iterations. While in princi- ple ADAS has the potential to discover more sophisticated agents by searching in the entire code space, it may require more iterations (and thus higher LLM token usage) to achieve this.
Search trajectory in AgentSquare. We present the search trajectory under 15 iterations using AgentSquare based on GPT-4o and other searching methods on ALFWorld and Webhop tasks in Figure Results other tasks are presented in Figure A.13 and A.14 AgentSquare demonstrates steady convergence trajectory, where more ad advanced agents are continually emerging during search. In contrast, module-level searching methods including random and Bayesian search lack clear and insightful search direction. Prompt-level search methods such as OPRO are constrained by a limited modification space, leading to minimal performance improvements. As a result, they all encounter performance bottlenecks during the search process, resulting in sub-optimal agent architectures. Be- sides, we find that simple module-level search methods such as random recombination greatly out- performs prompt-level search, indicating the importance of searching in the modular design space.
4.3 ABLATION STUDY OF AGENTSQUARE
Effectiveness of module evolution and recombination. There are two key operations in the search- ing framework of AgentSquare: module evolution which creates new modules and module recom-
8

---

Published as a conference paper at ICLR 2025
Method
Webshop
ALFWorld
SciWorld
M3Tool
TravelPlanner
PDDL
AgentSquare (full)
0.607
0.695
0.781
0.524
0.583
0.669
w/o module evolution
0.564
0.649
0.736
0.502
0.577
0,614
w/o module recombination 0.560
0.616
0.710
0.481
0.280
0.669
Table 2: Ablation study of AgentSquare on GPT-4o on six tasks across different domains.
Actual tested perfermance
Wensmos bead on paid. Alkand based 3
Actual Lested per formance
Sewere beard gaodss
Actual tocterd cerffomance
Klooto and gakals
Thareptanna: baied goi-35
Fod beasd en ap-25
Predicted performance
d
Predicted performance
Predicted performance
Figure 5: Validation of the effectiveness of the performance predictor (correlation between the actual and predicted performance) each task.
bination which strategically recombines existing ones. To verify the effectiveness of each design, tested three variants: the full model, version without module evolution, and version without module recombination. The results based on GPT-4o and GPT-3.5 are presented in Table [ Ta- ble A.5] respectively. It can be seen that dropping each design results in noticeable performance decline and the module recombination has larger impact. Module recombination significantly ex- pands the search space, reducing the risk of falling into local optima. Mcanwhile, module evolution facilitates the discovery of more advanced modules tailored to specific tasks. These two operations collaborate well ensuring the effectiveness of the search process in AgentSquare.
Effectiveness of performance predictor. In this part, we verify the effectiveness of this design empirically. Figure iilustrates the predicted performance of given agents versus their actual tested performance based on both GPT-3.5 and GPT-4o on all six tasks. The tested agents were generated through random sampling by randomly combining existing modules. It can found that the pre- dicted performance closely aligns with the actual performance, demonstrating the effectivenesss of the performance predictor. For instance, the evaluation cost of the predictor is only about 0.025% of the cost of full evaluation based GPT-4o in ALFWorld, demonstrating its remarkable cost- efficiency. We provide more experiment results of predicting the performance of the dynamically searched agents in Figure Af the Appendix.
4.4 DISCOVERED BEST AGENTS FROM AGENTSQUARE
In this section, provide some illustrations of the searched best agents, especially some discovered promising modules. Table A.4 summarizes the searched best agent from AgentSquare and the best hand-crafted agents all tasks. We can observe that AgentSquare can adaptively identify promising agents with both previously existing and newly programmed modules tailored to the given task. For instance, the discovered best agent for AL. combines existing well-designed memory module from Generative Agents with newly created planning (named 71)) and reasoning modules (named SF-ToT). By comparison, the best hand-crafted agent Self-refine focuses only on reasoning module design while overlooking other functional modules, leading to suboptimal performance. Moreover, we illustrate two new modules and the human interpretable design insights discovered on ALFWorld in Figure More illustrations are listed in the Figure A.16 Figure A.2I
5 RELATED WORK
5.1 LLM-BASED AUTONOMOUS AGENTS
LLM-based autonomous agents are an advanced AI system using a core LLM to manage external functional modules and interact with the world (Ding et al. 2024b). Recent studies have equipped

---

Published as a conference paper at ICLR 2025
SE-TOT
TD
Insights: Develop module that only generates multiple paths and evaluates them but also incorporates self-reflection and self improvement strategies.
Insights: Use timeline- based approach. where tasks are decomposed into sub- tasks with explicit temporal dependen- cics.
def o0th loelff pronge presisporact wich
def 2001.,
prospe aetetor are planner who disides (aaee tack into tengaral dependenclire Cansider the order of actiooe and theie dependencied to essure logtcal requereina.
housabeld salvo a tack,
eamplees, exaplees. respensee Iin_rerparant..))
rempense seit.get vetenturel censooras,
rubupians Hiuraapponte.... cotuze eut_plant
retuen response
Figure 6: New module discovered through AgentSquare search on ALFWorld.
LLM agents with several LLM-centric functional modules including planning (Hao al., 2023
Zeng et al.
2024: Shao et al.
2025). reasoning (Wei et al., 2022 Yao et al. 20224, Shang et al. 2024
Xu al. 2025), using tools Shen al., 2024 Schickk et 2024). and monitoring memory (Wang et al. 2024a] Park et al. 2023), greatly enhancing the capabilities of LLM agents. Along with the improvement of the single agent, there's ' another line of work trying to build more advanced multi- agent systems by strategically organizing individual agents for both simulation (Li et al., 2023 Chen et al. 2023) and targeted task solving (Qian et al. 2024 Chen et al. 2024b Li et al. 2024b). The emergence of more and more sophisticated agent produces remarkable performance improvement, however, their architectures and codebases differ greatly with each other. The lack of unified design space and consistent terminologies across individual works makes it hard to compare different agents, understand their evolution routes, and guide new agent design directions.
5.2 AUTOMATIC DESIGN OF LLM-BASED AGENTS
LLM-based agent system, as the most advanced AI system, has not yet formed a unified design space and an automatic design approach. Engineering-orrenter open resources like LangChair and BabyAG[ have provided convenient ways to build an LLM-centric agentic system, however, they still need human participation to organize different modules and can't support the optimization of the designed agent. Besides, there have been some conceptual frameworks trying to provide a unified design principle of LLM agents, such as CoALA (Sumers et al.) 2023). However, it's still vision of how LLM agents should be in the future, without providing a practical design framework. More importantly, there are several recent works that explore the problem of automating (at least part of) the design of LLM agent systems defined on different search spaces. OPRO (Yang et al.) 2024) and Promptbreeder (Fernando et al. 2024) can be considered as using LLMs optimize LLM agent defined on prompt space. More relevantly, ADAS (Hu et al., (024) proposes to search the entire agentic system defined on code space, enabling the search for LLM agents with more flexible prompts, tool uses, control flows and more.
6 CONCLUSION
In this work, we introduce a novel modular design space for LLM agents, allowing researchers to build upon successful prior designs and collectively accumulate new insights. Based on this, we pro- pose novel research problem, Modularized LLM Agent Search (MoLAS), which aims to automat- ically optimize LLM agent designs by leveraging the knowledge gained from previously published or evaluated modules. To address the challenge of vast search spaces, present AgentSquare, an automatic search framework to optimize LLM agents through module evolution and recombination. We further introduce performance predictor as an in-context surrogate model for evaluating novel L.M agents to accelerate the search process. Overall, our work offers a transition from studying individual LLM agent designs studying LLM agents within a modular design space, further con- solidating the collective efforts of the research community.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China under 231AA02114 and 62472241.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Angelica Chen, David Dohan, and David So. Evoprompting: language models for code-level neural architecture search. Advances in Neural Information Processing Systems, 36, 2024a
Lin Chen, Fengli Xu, Nian Li, Zhenyu Han, Meng Wang, Yong Li, and Pan Hui. Large language model-driven meta-structure discovery in heterogeneous information network. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 307-318, 2024b.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representa- tions, 2023.
Han Ding, Yinheng Li, Junhao Wang, and Hang Chen. Large language model agent in financial trading: A survey, arXiv preprint arXiv:2408.066: 2024a.
Jingtao Ding, Yunke Zhang, Yu Shang, Yuheng Zhang, Zefang Zong, Jie Feng, Yuan Yuan, Hongyuan Su, Nian Li. Nicholas Sukienník, et al. Understanding world predicting future? a comprehensive survey of world models. arXiv preprint arXiv:2411.14499 2024b.
Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. In Forty- first International Conference Machine Learning, 2024.
Yingqiang Ge. Wenyue Hua, Kai Mei, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang, et al. Openagi: When Ilm meets domain experts. Advances Neural Information Processing Systems, 36, 2024.
Shibo Hao, Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference Empirical Methods Natural Language Processing, pp. 8154-8173, 2023.
Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint w0XVv22408.08435. 2024.
Ganesh Jawahar, Muhammad Abdul-Mageed Laks VS Lakshmanan, and Dujian Ding. Llm perfor- mance predictors are good initializers for architecture search. arXiv preprint arXiv:2310.16712, 2023.
Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O Stanley. Evolution through large models. In Handbook of Evolutionary Machine Learning, pp. 331-366. Springer, 2023.
Junkai Li, Siyu Wang, Meng Zhang, Weitao Li, Yunghwei Lai, Xinhui Kang, Weizhi Ma, and Yang Liu. Agent hospital: A simulacrum of hospital with evolvable medical agents. arXiv preprint arXiv:2405.02957 2024a.
songwei Li, Jie Feng, Jiawei Chi, Xinyuan Hu, Xiaomeng Zhao, and Fengli Xu. Limp: Large language model enhanced intent-aware mobility prediction. arXiv preprint arXiv:2408, 12832, 2024b.
11

---

Published as a conference paper at ICLR 2025
Yuan Li, Yixuan Zhang, and Lichao Sun. Metaagents: Simulating interactions of human behav- iors for I1m- llm-based task-oriented coordination via collaborative generative agents. arXiv preprint arXiv:2310.06500, 2023.
Chang Ma. Junlei Zhang. Zhihao Zhu, Cheng Yang, Yujuu Yang. Yaohui Jin, Zhenzhong Lan, Ling- peng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn Ilm agents. arXiv preprint arXiv:2401. 1..40/1.117.. /3778, 2024.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uni Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted cueestion-answeriggg with human feedback. arXiv preprint a1Xiv:2112.09332, 2021. 2021.
Muhammad Umair Nasir, Sam Earle, Julian Togelius, Steven James, and Christopher Cleghorn. Llmatic: Neural architecture search via large language models and quality diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1110-1118 1118, 2024.
Joon Sung Park, Joseph O' O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pp. 1-22, 2023.
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang. Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. In Pro- ceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15174-151888 2024.
Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 62597595):468 475, 2024.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves use tools. Advances in Neural Information Processing Systems, 36, 2024.
Yu Shang, Yu Li, Fengli Xu, and Yong Li. Defint: A reasoning with hybrid large language models. arXiv preprint arXiv:2402.02563, 2024.
framework for efficient
Chenyang Shao, Xinyuan Hu, Yutang Lin, and Fengli Xu. Division-of-thougee Harnessing hybrid language model synergy for efficient on-device agents. arXiv preprint uxXiv:2502.04392, 2025.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugging- gpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, Advances in Neural Information Processing Systems, 36, 2024.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandrr Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021.
David So, Quoc Le, and Chen Liang. The evolved transformer. International conference on machine learning, pp. 5877-5886. PMLR, 2019.
Theodore R Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. Cognitive archi- tectures for language agents. arXiv preprint 22X 2023.
12

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint a1Xt1:2302.13977, 2023.
Guanzhi Wang, Yuqi Xie. Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024a. ISSN 2835-8856.
Ruoyao Wang, Peter Jansen, Maro-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11279-1122 1918, 2022.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Exe- cutable code actions elicit better Ilm agents. In Forty-first International Conference Machine Learning, 2024b.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi. Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023a.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint arXiv:2307.05300 2023b.
Zihao Wang. Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Shawn Ma, and Yitao Liang. De- scribe, explain, plan and select: interactive planning with iims enables open-world multi-task agents. Advances in Neural Information Processing Systems, 36, 2024c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of- thouht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 25:24724-44877, 2022.
Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao MA, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. Dilu: A knowledge-drie approach to autonomous driving with large language models. In The Twelfth International Conference Learning Representations, 2024.
Lilian Weng. Lim-powered autonomous agents. lilianweng. Jun 2023. URL https:
//lilianweng, www.gggthhh.....
Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural architectures for neural architecture search. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 10293-10301, 2021.
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, et al. Agentgym: Evolving large language model-based agents across diverse environments. arXiv preprint arXiv:2406.0413 04..24466,,1,,, 2024.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: benchmark for roal-world planning with language agents. In Forty-firs International Conference on Machine Learning, 2024.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprini urXiv:2501.00668 2025.
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe Le, Denny Zhou, and Xinyun Chen. Large language models optimizers. In The Twelfth International Conference Learning Representations, 2024.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35:20744-20755 2022.
13

---

Published as a conference paper at ICLR 2025
Shunyu Yao. Dian Yu, Jeftrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimban. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
Junchi Yu, Ran He, and Zhitao Ying. THOUGHT PROPAGATION: AN ANALOGICAL AP- PROACH TO COMPLEX REASONING WITH LARGE LANGUAGE MODELS. In The Twelfth International Conference on Learning Representations, 2024.
Siyu Yuan, Kaitao Song. Jiangjie Chen, Xu Tan, Dongsheng Li, and Deqing Yang. Evoa- gent: Towards automatic multi-agent generation via evolutionary algorithms. arXiv preprint arXiv:2406. 14228, 2024.
Qingbin Zeng, Qinglong Yang, Shuman Dong, Heming Du, Liang Zheng, Fengli Xu, and Yong Li. Perceive, reflect, and plan: Designing Ilm agent for goal-directed city navigation without instructions, arXiv preprint arXiv:2408. 04168, 2024.
Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, and Denny Zhou. Take a step back: Evoking reasoning via abstraction large language models. In The Twelfth International Conference on Learning Representations, 2024.
Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. Bayesnas: A bayesian approach for neural architecture search. In International conference on machine learning, pp. 7603-7613, 2019.
Mingehen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents optimizable graphs. In Forty-first International Conference on Machine Learning, 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 EXPERIMENTAL SETUP
Task setup. We evaluate AgentSquare and compared methods on six representative tasks covering four key domains which are widely adopted by existing LLM agent benchmarks (Ma et al. 2024 Xi al. 2024):
Embodied:
ALFWorld (Shridhar al. 2021 with text-based household tasks where agents
igate and interact with objects using text commands, ScienceWorls (Wang et al. (2022) with inter- active science tasks requiring agents navigate rooms and perform experiments, testing scientific commonsense;
Game: PDDL (a et al.
2024) including many strategic games where agents use PDDL expres-
sions to complete tasks;
2022 focusing on online shopping tasks where agents browse and purchase products based on user instructions;
Web: WebShop Yao et al.
Tool: TravelPlanner (Xie et al. 2024) with many travel planning tasks where agents use tools and data to create detailed plans, (6)M3 EoolEval (Wang et al. 2024b) including complex tasks requiring multi-turn interactions with multiple tools.
The specific performance evaluation metric varies in different tasks, following the evaluation set- tings in their original work. Specifically, the evaluation metric is "success rate" for ALFWorld and M3ToolEval, "task score (defined as the average reward obtained across episodes)" for Webshop, "progress rate" for SciWorld and PDDL, and "micro pass rate" for TravelPlanner.
Baselines. We compare AgentSquare with four types of baselines:
Hand-crafted agents. We compare with 12 hand-crafted agents including CoT
Wei al. 2022
CoT-SC (Wang et al.) 2023a), Self-refine (Madaan et al. 2024). ToT Yao et al. 2024). Step
back (Zeng et al., (024). Thought propagation Yu et al. (024). HuggingGPT (Shen al. 2024
Voyager (Wang et al. 2024a), Generative Agents (Park et al. 2023
DEPS (Wang et al. 2024c
OPENAGI (Ge al. 2024 Dilu (Wen et al. 2024).
Module search methods. We compare with two module-level agent optimization methods in- cluding the random combination existing modules and Bayesian (Zhou et al. 2019) module combination optimization inspired by Bayesian optimization in NAS WWite al.
2021
Prompt search methods. We select OPRO Yaang et al. 2024) as representative prompt-level optimization approach, which leverages LLMs as optimizers by generating and refining instruc- tions through iterative prompts.
Agent search methods. We select ADAS (Hu et al. 2024 which optimizes the entire agentic system in code space as the agent search baseline. We use the official code of ADAS and make slight modifications to adapt it to our tasks.
AgentSquare setup. We implement AgentSquare and conduct experiments using both GPT-3.5- turbo-0125 and GPT-4o (Achiam al. 2023). To ensure fair comparison, we use the number of few-shot examples across all methods. The initial agent is set as a random module combination, and the search process terminates after 5 consecutive iterations without performance improvement.
15

---

Published as a conference paper at ICLR 2025
Algorithm 1: Algorithm of AgentSquare
Input: Initial agent Ag, targeted task descriptions maximum evolution episode K, population size N per evolution phase, standardized module pools (P, IR, T,, M). experience pool E
Output: The evolved agent A
t t- /// Current search episode
4 < Ag // Initialization of the module evolution phase
while < do
{A) 42.
A ".. N, // Module evolution
4 < arg
Evala(A)) (.ral)))). // Select the
best-performing general rated agent
(4
p1 A A + (A( d, MR Module recombination
4 - arg max (ax(aaalx(a))) (eala(A)) Evala(A)) [) Select the
best pestrerronning generated agent
t ++1
end
A +- 4
return A"
Web Webshop
Embodied
Tool
Game
Method Type
Method
ALFWorld SciWorld M3Tool
Travel
PDDL
CoT
0.504
0.369
0.142
0.172
0.080
0.151
CoT-SC
0.527
0.381
0.105
0.181
0.167
0.178
Self-refine
0.439
0.388
0.222
0.098
0.000
0.109
ToT
0.510
0.381
0.143
0.189
0.163
0.147
Step Back
0.478
0.375
0.027
0.128
0.120
0.137
TP
0.429
0.299
0.168
0.139
0.063
0.122
Hand-crafted Agents
HuggingGPT
0.518
0.502
0.270
0.012
0.470
0.212
Voyager
0.427
0.369
0.301
0.008
0,480
0.149
Generative Agents
0.539
0.388
0.153
0.144
0.060
0.123
DEPS
0.555
0.474
0.308
0.017
0.500
0.186
OPENAGI
0.507
0.448
0.257
0.008
0.430
0.178
Dilu
0.418
0.291
0.000
0.131
0.137
0.054
Random
0.562
0.569
0.367
0.235
0.473
0.216
Module Search
Bayesian
0.581
0.611
0.269
0.217
0.497
0.210
Prompt Search
OPRO
0.507
0.376
0,032
0,193
0.513
0.179
Agent Search
ADAS 0.519
0.274
0.217
0.193
0.410
0.186
AgentSquare 0.617
0.651
0.432
0.285
0.520
0.219
Table A.3: Performance comparison of searched agents from AgentSquare and (1) existing human- designed agents (2) module search baselines (3) prompt search baselines based on GPT-3.5 on six tasks across different domains.
Task
Planning Reasoning Tooluse
Memory
Best Hand-crafted Agents
Webshop
10
HTSS
Dilu
HuggingGPT
ALFWorld
TD
SF-ToT
Generative Agents
Self-refine
SciWorld
Voyager
COT
Hier
Voyager
M3Tool
COT-SC
ToolBF
Toolbench
TravelPlanner
DEPS
CoT
TH
DEPS
PDDL
IR
CASRC
Generative Agents
OPENAGI
Table A.4: Comparison between the searched best agent from AgentSquare and the best human-
designed agent on all tasks.
16

---

Published as a conference paper at ICLR 2025
Method (gentSquareefull) 0.617
Webshop
ALFWorld
SciWorld
M3Tool
TravelPlanner
PDDL
0.651
0.432
0.285
0.520
0.219
w/o module evolution 0.595
0.623
0.288
0.236
0.483
0.202
w/o module recombination 0.578
0.546
0.310
0.258
0.267
0.173
Table A.5: Ablation study of AgentSquare on GPT-3.5 on six tasks across different domains.
Wehshop
ALFWorld SeiWorld M3Tool
TravelPlanner
PDDL
Avg cost (GPT-3.5)
$3.16
$4.25
$1.92
$2.43
$1.84
$2.70
Iterations (GPT-3.5)
23
21
8
14
9
17
Avg cost (GPT-40)
$10.51
$13.96
$42.14
$26.03
$29.75
$26.94
Iterations (GPT-40)
18
15
9
18
8
12
Table A.6: Average API cost per search iteration and the total number of iterations until termination for AgentSquare using GPT-3.5 and GPT-4o across six tasks.
Alfworld based GPT-3.5
Altworld based GPT-4o
0.65
0.7
AgentSquare
AgentSquare
0.6
Accuracy
HuggingGPT
Accuracy
Self-refine
DEPS
0.5
OPENAGI
HuggingGPT
Generative Agents
0.45
OPENAGI
DEPS
CoT-SC
Dilu
Self-refine Generative Agents
Veyager
0.40
0.4
COT
TP
0.35
Voyager
Step Back
CoT-SC
0.30
Step Back
80
100
API cost(S)
API cost(s)
Figure A.7: Performance versus API costs visualization on ALFWorld task.
Webshop based on GPT-3.5
Webshop based on GPT-4o
0.625
AgentSquare
0.60
AgentSquare
0.575
0.55
Accuracy
0.550
CoT-SC
DEPS
Accuracy
Co HCggglgggT
OPENAGI
0.525
HuggingGPT
Generative Agents
0.50
DEPS
ToT
Generative Agents
0.500
OPNNAI
Co
Step Back
Col
0.45
Self-refine
0.475
Step Back
Dilu
0.450
Selt-refine
0.40
Voyager
Dilu
TP
Voyager
60
80
100
120
API
API cost(S)
Figure A.8: Performance versus API costs visualization on Webshop.
Sciworld based on GPT-3.5
Sciworld based on GPT-4o
AgentSquare
0.4 AgentSquare
CoT 0.7
Voyager
OPENAGI
Generative Agents
To
Dilu
HuggingGPT
0.3
Voyager HuggingGPT
CoT-SC
Accuracy
OPENAGI
Self-refine
0.2
Generative Agents
COT
0.4
ToT
0.3
Back
Step Back
1.5
15
30
cost(S)
cost(S)
Figure A.9: Performance versus API costs visualization on Sciworld.
17

---

Published as a conference paper at ICLR 2025
M3Tool based on GPT-3.5
M3Tool based on GPT-4o
0.25
AgentSquare
0.50
AgentSquare
Dilu TD
CoT-SC
Selfrefine
Accuracy
T-SC TDT
Accuracy
Step Back
0.15
0.40
Generative Agents
0.10
Step Back Dilu Se feffree
Agents TP
0.35
HuggingGPT
TP
OPENAGI
0.05
HuggingGPT
DEPS
0.00
OPNNAGI Voyager
0.25
Vayager
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
10
15
20
25
30
API cost(s)
API cost(S)
Figure A.10: Performance versus API costs visualization on M3tool.
Travelplanner based on GPT-3.5
Travelplanner based GPT-4o
0.5
AgentSquare
OPENAGI
AgentSquare
HuggingGPT
DEPS
Voyager
Cot
Veyager
0.4
OPENAGI
Generative Agents
TP
Accuracy
Accuracy
CoT-SC
0.3
To
COT-SC
Dilu
0.1
CoT
Step Back
Generative Agents
0.0
Self-refine
0.0
Self-refine
Step Back
API cost(S)
API cost(S)
Figure A.11: Performance versus API costs visualization on Travelplanner.
Pddl based on GPT-3.5
Pddi based on GPT-4o
0.225
HuggingGPT AgentSquare
0.65
AgentSquare
0,200
DEPS
OPENAGI
Arcuracy
0.150
0.175
OPENAGI
COT-SC
Accuracy
0.60
HuggingGPT
DEPS
Step
Voyager CoT
TOT
Generative Agents
0.125
Generative Agents
Cal
TP
TP
0.50
CoT-SC
0.100
Step
ToT
0.075
0.45
0.050
Dilu
0.40
Veyager
3.5
4.0
cost(s)
cost(S)
Figure A.12: Performance versus API costs visualization on PDDL.
18

---

Published as a conference paper at ICLR 2025
AgentSquare Search M3tool
AgentSquare Search PDDL
Ferala"
0.25
Soiaes Fiddle koll
0.66
human suge
0.64
Progress rate
0.24
0.62
Success
0.60
0.58
0.20
0.56
0.18
0.54
10
14
6
10
11
Iteration
Iteration
Figure A.13: AgentSquare search trajectory on M3tool and PDDL (more hand-crafted agents, spe- cific module combinations when surpassing best hand-crafted and the final evolved agent, other search baselines).
AgentSquare Search on Sciworld
AgentSquare Search on Travelplanner
0.58
Auyito mobil
0.56
0.54
Progress
0.1-
0.50
0.0
0.48
8
Iteration
Iteration
Figure A.14: AgentSquare search trajectory Sciworld and Travelplanner (more hand-crafted agents, specific module combinations when surpassing best hand-crafted and the final evolved agent, other search baselines).
Actual performance
Alare bosd opi-3.
performance
Sowett g7-4e
Deads base on 20-3.3
Redure bestoee perfoon eenc
not besed 97-335
Predicted performance
Predicted performance
Predicted performance
Figure A.15: Validation of the effectiveness of the performance predictor dynamically searched agents for each task.
19

---

Published as a conference paper at ICLR 2025
HTSS
Observing the current performance of the reasoning modules, it seems that tech- niques like Chain-of-Thought (CoT) and Tree-of-Thoughts (ToT) have offered improve- ments by breaking down the task into smaller steps and evaluating multiple reasoning paths. The Self-Consistency (SC) approach also shows promise by generating multiple answers and voting them, while the Self-Refine module uses iterative improvement based on feedback. Overall Idea: To further enhance the performance, we can integrate combination of these techniques into single module. Specifically, we can combine the Tree-of- Thoughts (ToT) approach with Self-Consie Consistency (SC) and Self-Refine. This combined approach would volve generating multiple reasoning paths, evaluating them to choose the best path, and then iteratively refining the chosen path based feedback.
Insights:
Implementation: 1. Generate Multiple Reasoning Paths: Use the Tree-of- -ho-ughs (ToT) approach to generate multiple reasoning paths.2. 2. Evaluate and Choose the Best Path: Use Self-Consistency (SC) to evaluate these paths by voting and selecting the most common or accurate path.3. Refine the Chosen Path: Use Self-Refine iteratively improve the cho- sen path based on feedback.4. Prompt Structure: Craft prompts that encourage step-by-step problem-solving, refer to similar solved examples, and refine output based feedback.
class REAGONING_IYDRE TOI_SC_ BOAOIID_TTTSS__LLTREEEEIEI cO_SELTREFINE()
def
iini:_(se1,, profile_cype__rrmpp menory, tocluse, 1ims_sype):
self self. feedback se1 pttpproft"__type_p_" self.memory memory
- prarile_type_pree
se1 self.11n_type 11ms_type(0)
sollue Locluse
def _call_(se1, (self, task_ descripttri; sti, instsuuuctt :stz=!',
feedback iatse '1]
task_neme re.findsll (z' # (et
task_descrsptiinn
if self self.memory is not None:
self. Laik.aas__aame__e_ cache Lask neme helf.memory_cacte iE Lask_descrigt.on. count ('Reasuning") 2:
self. self.memory cccck_ddserrpption
seiffaemory_cachh alif baaskdescription.co count keasening' == 4:
melf.mencry_cecaee split ('Observation")
self.
"Ubservztion' joonn
menory -joinntal..mmmarrcccci
''Observation' [0:3])
alse:
metoryemoryccaccc self.memory_cacie
melo.me
eemory__cchee
self. tovluse
tcoluse self.tooluse htl..ttoolueettsiiddersipiiiipie tcol_instruction)
alse:
tcoluse
split_text task wtw..daarrrption.rpplttttt examples aplit_text [G]
deskciiscttttio 'KebStop' split_texi[]]
prompt ri1 [tooluse:Solve the taak by step. Your output must
fellow the examples process. Don't refine your search. You have choose from list of itens.
[mamory] (examples)
prompt prompt.
format format (task saankk_dssrrpttionttaa.. www.eertte..kkkeoo..ooooos.oeeee
examples-examples, memory=sel memory_pache, toolise-tool reasoning reasoning_resull resuuls rem_responaee (pwunpllpzummee model=self. lin tempezature-O..1 stop_strs" \z")
Evaluate and choose the best path collections import countt Counter (reassninggg results)
bast_path string_onunts, moott_ooommnn (11 [0] [0;
Refine the chosen path based on feedback nefined_reaut self, selffrefino (hest_psth)
freasoning self.cefine (eweeeiieeeeeennnnaaa (reasoning_resull) retura reesutd_result
Figure A.16: New module discovered through AgentSquare search on Webshop.
20

---

Published as a conference paper at ICLR 2025
Hier
Insights: The proposed memory module's hierarchical structure offers significant advan- tages task management for intelligent agents. By breaking down each task into smaller sub-tasks stored separately, the system enables focused information retrieval, allowing the agent to access only the relevant data instead sifting through an entire task trajectory. Overall Idea: My proposed memory module will focus creating hierarchical memory structure, where each task is broken down into smaller sub-tasks and each sub-task is stored separately. This approach allows the agent retrieve focused information on specific sub- tasks rather than an entire task trajectory. Additionally, this memory module will include a feedback mechanism to improve memory relevance and accuracy over time.
Implementation: The implementation involves modifying the memory module to store and retrieve sub-task trajectories and introduce a feedback loop for continuous improvement.
class MEMORY SOOOYYJHIRRLL
def
init (self, lime_rype)
sel 1im.imme Lypmm__pee_____ [0]
embedding Cpensl[mbeeddaags
da_path po.. //db', 'memory/")
e5 es.path.exista. (dh_pcth):
shutil. umtre (db_peth)
self..cerraroomancc Chroma
enbedding__uucttieeee eebbediingg parstot_dhracttry--tt
def callllll (self, curreat_situation str="";;
11 ' 'The correct trajectory le' in current_situation:
dedEfedddemmoryyeee (eusrent__ituuttee
else:
raturn self, retrieveManory (currant_aituatinn)
def retriaveMemorysslll (self, pcery_acenario):
querr_scenaal
if selt.scensria sccosrriamemooyy.. www.y//ooooocoooooooooo. count = print ("The memory search.")
datsbase empty. Canrot perform
return "
8 similarity_ resultss secll.scenaioomemmaa wtt.:ruw...ueyy.....l.....eeeeed sub nam_, k=3)
Swwahot_rrsulle []
senge Jen(emmilarityyreeniaz)
[C] .matadata
['aub trajectory'
return "insere are similar sub-tasks and the corract handling
trajectories these casea: + (fowshotooor_eeeeee def addKemury (self, current_siie sitzenl_iitualonnl current cursent_situation
sub wtt.tttssstseeettteeets sub Lasi_name re.search Sub Lask (.42) AnThe corsect LeajecLozy
is', cusrent_situation muh_task_name
sub_task_descrip sub_task__nmeegroup
doc Document
sage_contert=sube task desertp,
metadata= "esdatta=""ubb_takk prub_task_name aub_task_descrip.
bub_taak__raaecton aub_tase trajstttrjjeorr
id def (eebbaciiechhanisnneeell surcen-_situalior. vas_succesalul)
douusttsssseeeee.ddueeueeeessss ([dec]
if sac_ssccsssful:
addd,addMemory teurrent. ceurrnnt_pituation)
else:
sub_task_namd re.search (r' sub-task
(.+2)
trajactory 's', purrent_situatt
ir tub_lask_nane is not None:
sub_task _as__akk_desorppp aub_task ttt.kaaamm.grrupppp. doc_id
self. wwt.sscceerooeeeeee decrueeddecuentts bersbbttte__eeeeeeeeeeeeeeeeeeee [0] aecoenarro_memeiieee commory.aai deweleeeeeeeeeeeee document (doc_id)
Figure A.17: New module discovered through AgentSquare search on Seiworld,
21

---

Published as a conference paper at ICLR 2025
ToolBF
Insights: The previously discovered architectures indicate that leveraging multiple interac- tions or multiple attempts to identify the most suitable tool can enhance performance (as in Toolformer). Additionally, using vector similarity approach to retrieve the most relevant tools (as Toolbench) seems promising.
Overall Idea: propose combining the vector similarity approach with multiple attempts to maximize the chances selecting the optimal tool. Specifically, I will augment the Tool- bench approach by making multiple calls to the L.M to generate several potential solutions and then selecting the best through voting mechanism.
Implementation: The implementation will involve converting instructions and API docu- mentation into vector representations, retrieving the most relevant APIs, generating multiple responses using the LLM, and finally selecting the best response using voting mechanism.
class :COLUSE TOCLEENCHEURMERRII
def _init (self, line_Lype): seLe.lin_typ -ims_type.. self.svenar.o__eeee for tools toc.us__t__pco..teee...
db_path os.psth. join(' //db', posp___olllnmmee//
sel geif.enbedding OpenATPmbeddingg
sewnaaie [name] Chroma
embed6ing_ fon r.enbeddd..dnddggg pero.at_di2atter--dd.ooo
_ol_atttern_.. re.comp' 1e("" ((d+)) ([":)+);
(.+7) (7-\ [( rO.DCTALL]
apl_mate _ol_matchess..
_at.tere...taaaa11.. (tools)
documenta -
for match in api_matches:
api_id, apt_nams, jpi_descripttin match
Eiret_santeece api_descriptionn.s..tt (0) strrip))
Euli_descriptien ["(iapiidd)) (spi_name]:
[api_descriptionn
Document
pep__contenttiiitt__eeeee
netsdats";
"name"; api_name.
"descript_en": fuli_descriptienn)
appuren (doc)
self. calnsatatttommeeerr (nmee_e add_cccmntaaas (document.))
def c0lllll (self, dask_deser:.tion, tool_instruction,
feedback_oo ptt.oooff_rrroo_roooosll
sinilar: sinilarity_reeute
welf.acenario_ memerrrrionmoorooooomooomoo destekdddesrrpptoon stmilarity_.eareh__u itt.
k=4)
tool_pool []
r0nga10, lan (otmilarity reau ts)l
iix [0] .metadata( description")
prompt g'>.
You have access to the following tools:
(tocl_peol)
You need select the appropriate Lout from the 1ist of available Louis sccording to the task description complete the task:
[Lool_instructiinn
You must the tools cutputing the tool followed its arguments, delimited by commas.
You can optiona" express your thoughta vaing natural language before acLion. For example, 'Thought: weat use Loul_neme tu something. Aotions syour act ion to call
Rnd Action".
You only inveka
tima.
You muat begin your tool invocation with 'Actions" and and it with <End Action'.
Your tool invocation fornat must follow the invocation on format in the too1 description.
Steccback_of_ppeeviouu.t
strings lin (prompt sodel=self.1lm.ee Lemperature a=3)
string self. (tocs tcol inoliinstuuttinn, deediack_oi_peevioos strings)
return string
Figure A.18: New module discovered through AgentSquare search M3tool.
22

---

Published as a conference paper at ICLR 2025
TH
Insights: From the currently explored architectures, the 'Toolformer' approach seems have the highest performance at 0.56, which suggests that generating multiple candidate responses and then voting the best is effective. Another observation is that hierar- chical search approach like Anyttoo'' might help better categorizing and selecting tools based on the task.
Overall Idea: I'll combine the hierarchical search strategy with the candidate response gen- cration and voting method. This will involve first categorizing the tool based on the task description and then generating multiple candidate responses select the best This should leverage the strengths of both methods.
Implementation: will implement class where the tool is first selected using hierarchi- cal search strategy, and then multiple responses are generated for the selected tool, followed by a voting mechanism to identify the best response.
ala TOOMUSF__R
der _init_ (seli,, 11ms_type]:
self. 11ms_type[0]
celf.tool_desarpptioo functions_info
self.tocl_pool traveiplamee-ttooppeel
cazegory-prompt category-prompt-)
string =in_sesponse ptg.rpppposspppppppaaaapaaapappp
podel=self.AAmmeyee
temperature=Q.11
dict_strings ze. (e..fladal1:=MMMMM
string)
self.dicls [eal, leet. ids) for da in dici_strings)
del call__ll_ (seli, Lack Soll_instrution, prompl ['elfsse.
You need to select the appropriate teel categery from the list available Loels accordiny Lo the Lask desccipLion complete Lhe teak: (task daac__deecrrpooon
Moollinnttruhh
You can only invoke one eategory at time.
Sreebbcck_u_jpp se___rrvvss__ooll
Outpet category name rectly
Your oulpul ehould be of the following furmaL: Canegory name:
category_name 11m_ response (prorgt=pronpt, model=sel fn tarperature-G.11 (") macching Docchingdllet None for in seli.dicta:
d.gez d'ateggyyy name") category_nane
matching_dict
break
if matching_eics and 'tool list' in matching_dict and betching_dict("sca) list"]:
masched_tools - [tool: selfitool, dol
for tool in
matching_dict
list'] tool self.tool_dessriitiinn
slae:
masched_tools random. (list (slf leeeeeaad....o..dderrriee....... (1)) prompt prrr
(matched_tools)
The user's query is : (task__escrigtien) tool-use instruction You can only invoke one tool at time. (feesback_od wtt.e_efppeee____eeeeoeee You answer should follow the fornat: toul_ tyoi__yyeetocc__tg] [tocl_arg], such FlightBearch York,
surrent
intoci_
2022-13-01]
remporee (prompt-pronpt, madel=sel1.11nn2yyp.. Comperature=0.1 n=3) string vocev Inatched_tooe tosi_instruction, of ptwvooffo__evvvtttooooeeeo strings)
string
Figure A.19: New module discovered through AgentSquare search on Travelplanner.
23

---

Published as a conference paper at ICLR 2025
CASRC
Insights: The current approaches have explored direct reasoning, step-by-step (Chain-of- Thought), and self-refinemmnt aechniques.Notable the "Chain-of-Thougi"" and 'Self-Refine' methods have shown polential by decomposing the task and iteratively improving the solu- tion. However, despite these efforts, the performance still hovers around 50-55%, indicating room for improvement.
Overall Idea: To further enhance the performance, propose combining elements from the high-performing methods (Chain-of- Thought and Self- Self-Refine) with new focus on.
REASONING, CONTEXT AKARF_ BE__EEEEEECCCTTCC ():
der -init _init___ts (seli, protsle_typo_proopp. memozy, tooluse, lims_typel,
welf.proffle__te cate___ye__ppprreeee proffil__tye_prompg
self.memory menory
self.1ln_typt [ype_type[0]
self.tooluse tooluse
se1f.con:ext.
_csll (self, tesi__esccriptinn stt, tool_instruction: stc-!!,
feedback: str="/1 if memoryy menory selt.memory www.aaoooyyyttyaeeeeee (task_osscrspptionn
else:
memory
if self cooluse la not None: Louluse wew....oluse deseerpii, tool_instruction)
e1se:
tooluse
Set the initial prompt
prompt porvsolve the task step by step, Interact with a household to solve task. Your instructions ahould follow the examples.
[memory]
[tooluse]
"pddl", "goal"; goal satiafy the following conditiona: shott contains cocktai16. shot2 contains ingredientl. saot3 contains ingredient2. "subgoaia"; [*shot1 containa cocktaile.", *shot2 contains ingredferti.", "shot3 contaiins ingredient2.") "), "citficulty"; "hard", "additional_inee" "subtask": "barnaa"
exemple,
rid";
In this tack, the geal is to have shots containing specific ingrediencs or cocktaile. This mixing and serving drinks.
under "barnan" category, involves
Assuning a bartender zobot with a gripper hand, let's break dowa the steps to achleve this:
1. The cirst subgeal is to have shetl contain cocktails. The robet will need Locate cocktail6, grasp using gripper, and pour it into
The second subgeal is the same as in step i, but this sime locating ingredienti and pouring it into shot?.
shot2 contain ingredientl. the rebot will
3. The taird subgeal is to have shot contain ingredient2. The robst will again repeat the process, locating ingredient2 pouring
After stepa, robot wi1l ach! eved subgoals, thus completing the main task.
ATTENTION:YOn should ansver a valid action directly! Sow, here is the zask need to sclve: deask__escription)
('prompt:', prompt]
input (I
prompt prompt. format (task_descr descript dordtask_eesrrpttionn menory-memory, toolusa=cool cosee) reasoning_rasut iim_respense (pronpt-prompt, model-seir. 1im__ype, tesperature-0.1. stop_strssi [" 'aa "talll.rrppaa repplace (1) stssitpppi reasoning_cesult self, refine deveription, ressoning_resuls) return reasoning_reaull
Figure A.20: New module discovered through AgentSquare search on Pddl.
24

---

Published as a conference paper at ICLR 2025
IR
Insights: To maximize the performance of the agent on ALFworld tasks, we should con- sider incorporating feedback loops and iterative refinement in the planning process. From the discovered architectures, it seems that the most effective modules (DEPS and openagi) provide detailed sub-goals and make of iterative improvements based on feedback. Overall Idea: Our next planning module will focus iterative planning with feedhack in- corporation. After generating an initial set sub-tasks, the module will prompt the LLM refine plan by explicilly checking dependencies and completeness the sub-tasks. Implementation: We will create a planning module that generates an initial set of sub-tasks and then refines it based on feedback. This refinement will ensure that the sub-tasks are co- herent, minimal, and complete, ensuring better performance in sequential decision decision-making tasks.
clacs PLANNING FLAWWING_TEERATIV CEEERATIYS_REEEEEE REP-NEMENT()) init sima_typel:
self.plan
self. [ima_type[0]
_ossl (self, type, dask_descriptionn [eedback):
fev_shot rrrGoll The goal to saciaty folloving conditiona b" is on b2., b2 is on b3, Anibaervat onn BI is on the table. R2 on the table. B3 is on the table. Robot arn empty. The bl cleas, The sub-task 1: pedeerription
need seack h2 63 first', 'reasoning instruction's "b2 is b3', "cool use instruction' None)
sub-tack 2: 11' descriptionn TTen need stack bl b2', 'reasoning b2',
instruction':
instruction', None)
prompt You aze plannez who divides [task_type) task into
First, generate initial set aubcaska schieva final goal.
After generst refine the sustasks by ensuring in they cover al necessery atepa, are in the correct order, and have no redundancies. example The fallowing are some exarg' es:
outpuz
Tack: (example)*<
feedback
prompt prompt
Itask_ descrritiion!"
prompt prompt. format wexemplefev_sshe Las__deccrittoonttttl..eee.
task_typeeraak_eypa
pronpt
pronpt
end
(feedback) Taak: (teek descriptionn it
prompt, format texemple=fev..o shot, task_descripcconttakk des.cippcoottssk..ccrrtttee
task__yperrsakk.yyp
Seedback feecnack)
Initiel
reatiaae_
raa_responaa (prong:=prompt, model=selr,1lo :omperatureeD.j)
findsll (PMI""AA initia' _rtittrrrspoose
initisl_dicte Last.litorsl_evalla
initial_eict_ttringg
Refinenent phase
refinement
You excert planner
with zefining folloxing subtasks. Sneure all necessary stops are covered, they are in the correct order, and there are redundancies, Youz outgut format should follow the example beldw.
The following aone examples:
Task: (exampla)
end
Subtasks: (subreeks)
forratt wwwssasko--iitiiiaaaa...a dicto, sxample-fey_hhee
srinnad_responae rim_respones toromprrett
wtl.--eett..nntyp...... tomperature- C.11
socined bottned_eiet_string re. findall ("") (1*),, refined
sotinod_ literal_eeeallll
seflnod_diet_strinng
self.plan rafined_dicta
recurn self.plan
Figure A.21: New module discovered through AgentSquare search on Pddl.
25


Reviews:
[{"review_id": "nRERgG9rqA", "reviewer": "Reviewer_funC", "summary": "The paper introduces AgentSquare, a framework designed to automatically optimize LLM agent architectures within a modular design space. It proposes a novel approach, termed Modularized LLM Agent Search (MoLAS), by defining a modular architecture divided into Planning, Reasoning, Tool Use, and Memory modules. AgentSquare employs module evolution and recombination mechanisms, along with a performance predictor, to explore and identify optimal combinations within the design space efficiently. The framework is evaluated on six benchmarks, showing a 17.2% performance improvement over existing hand-crafted agent designs.", "questions": "The authors write that: “In contrast, module-level searching methods including random and Bayesian search lack a clear and insightful search direction.\" Where do we see this? When looking at the figures, it seems that random search is a pretty competitive baseline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Modularity and Reusability: The modular allows the reuse and recombination of components, which aligns well with LLM advancements in modularization and scalability. \n\n- Effective Search Mechanism: The combination of module evolution, recombination, and performance prediction seems to be a robust optimization strategy. The proposed performance predictor effectively reduces evaluation costs, addressing practical limitations in real-world deployments of LLM agents.\n\n- Comprehensive Evaluation: Benchmarks across domains such as web applications, embodied AI, and gaming provide evidence of AgentSquare’s efficacy and generalizability.\n\n- Interpretable Insights: The framework’s ability to provide human-interpretable design insights is a useful addition, potentially helping in  aiding the design and tuning of future LLM-based agents.", "weaknesses": "My main issue with the paper is that while the modular approach is beneficial in the short-term, it may limit flexibility by enforcing predefined components. Extending the modular design to allow more dynamic, task-specific modules could enhance its applicability.\n\nAdditionally, the framework’s reliance on LLM-driven suggestions for module evolution and recombination could inherit biases or inefficiencies from the LLM models themselves, potentially limiting the quality of novel configurations.\n\nMinor comment: \n- Many citations miss parenthesis around them. Try using \\citep instead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HnBI6FWU74", "reviewer": "Reviewer_T4be", "summary": "This paper introduces a modular design space for LLM agents named “Modularized LLM Agent Search (MoLAS)” and proposes an evolutionary framework named “AgentSquare” for optimizing and recombining the various modules (covering planning, reasoning, tool use and memory) within this design space. It further introduces a performance prediction model for ruling out unpromising candidate solutions, hence rendering the search process more efficient. Through comprehensive evaluations it is demonstrated that their autonomous, modular design of LLM agents significantly outperforms manual designs and other search algorithms, and offers interpretable insights that complement human knowledge. This paper thus contributes to more standardized and scalable development of agentic systems that exploit prior successful experience, minimizing the reliance on human intervention.", "questions": "1.Are there any specific criteria for selecting the 16 LLM agents when constructing module pools? Besides, human labor involved in standardizing these modules might become prohibitive if one hopes to leverage a larger number of existing agents. Are there any measures to tackle this problem? \n\n2.Could you elaborate a bit more on the evaluation of candidate solutions? Specifically, it is currently unclear when solutions are evaluated by the performance predictor and when they are tested in the real task environment. \n\nThank you!", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is well-organized and uses language that is easy to understand. \n\n2.The paper is well motivated, addressing an interesting research question. It innovatively consolidates existing (and potentially upcoming) LLM agent designs into a unified framework, and effectively leverages their successful experience for better design. \n\n3.The experiments are conducted with both quantitative evaluations (including task performances, API costs and search trajectories), and qualitative analyses such as the specific module combinations of optimized agents and insights drawn from the newly discovered modules. \n \n4.The authors have made their code repository available on GitHub, which ensures reproducibility and promotes future work.", "weaknesses": "1.There are some inconsistencies within the paper that might cause confusion. For example, Equation (2) and (3) indicate that both module recombination and evolution take past experience as input, which is not the case in Figure 3. Besides, the random initialization (as mentioned in experimental setup) seems to contradict the arguments made at the beginning of Section 3.3. \n\n2. It seems more fair to also take into account the API cost incurred by search when you compare the performance-cost trade-offs of AgentSquare and manual design. However, the paper did not mention this information. \n\n3.The experimental results appear to be obtained from a single run. The authors are suggested to carry out repeated experiments, and report averaged results and standard deviations to demonstrate the robustness of their approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qlnkKaz007", "reviewer": "Reviewer_fhp6", "summary": "This paper proposed a framework for automatically selecting and optimizing for off the shelf LLM-based methods from existing literature. It proposes a new concept:MoLAS to enhance agent performance through modular evolution and recombination. The design includes four fundamental modules: Planning, Reasoning, Tool Use, and Memory and uses an in-context performance predictor to efficiently evaluate designs, outperforming manually crafted agents across six benchmarks by an average of 17.2%.  In sum up, AgentSquare provides a platform for researchers to reuse and extend prior successful LLM agent designs.", "questions": "1. What is the evaluation and metric function to optimize the searching process ?\n2. How does it scales if extending to multi-agent problem?\n3. Is it able to solve manipulation tasks ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This method shows it's originality in using evolutionary and recombined mechanisms to automatically explore optimal combinations, effectively consolidating prior research efforts. It also provides empirical testing across diverse benchmarks, showing improvement over handcrafted and prior methods. The research extends to its broad applications and the potential to unify efforts within the LLM agent community, reducing reliance on task-specific human design part and enabling a more systematic exploration of agent architectures.", "weaknesses": "The paper lacks clarity regarding the definition of certain components of the method, particularly the performance evaluation function mentioned in Section 3.1. It is unclear whether this refers to the API cost introduced later or another evaluation metric. Additionally, the method shows limited novelty, as it primarily focuses on leveraging LLMs to recombine and select existing components rather than introducing a fundamentally new application or capability for LLMs. Although the proposed approach does achieve better performance than existing handcrafted methods, it does not address or eliminate the core limitations of prior works, given that it relies heavily on reusing previous designs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5SVpQNoI3C", "reviewer": "Reviewer_Dm8t", "summary": "I am disheartened to report I believe this paper involves a form of plagiarism and likely involves intentional academic misconduct (or at least suspect behavior). For full transparency, and because I am an author of the plagiarized work and thus have a stake beyond a normal reviewer, I am making my review public and non-anonymous. \n\nWe posted a paper called “Automatic Design of Agentic Systems” (ADAS) on August 15th to arXiv and shared it on X and LinkedIn. It was submitted for review at ICLR 2025. \n\nThis review is of AgentSquare: Automatic LLM agent search in modular design space. It was posted to arXiv on Oct 8 2024. \n\nThis AgentSquare paper clearly takes many components from the ADAS paper, including the main ideas (slightly modified), repackages them (after adding a bit of new work), and presents the paper as a completely new set of ideas, all without acknowledging the significant degree to which it has lifted many key elements from our paper. It does cite ADAS, but only as an afterthought in the last paragraph before the conclusion, rather than writing throughout the paper (including in the motivation) that the ADAS paper is an extremely relevant piece of prior work and from which they copied (and slightly adapted) core pieces: the main ideas, unique writing choices, data visualizations, code, and prompts (which have significant chunks directly copied without modification). \n\nIt is hard to believe this is an honest mistake. Instead, the evidence causes me to conclude it is an attempt to get an ICLR publication by hoping the reviewers are not aware of how similar this paper is to a previous arXiv paper. Given how similar this new piece of work is, I would expect (and feel scholarship demands) this new paper to mention ADAS early and often, including discussing how it inspired their work and how the works differ, and to acknowledge that this new paper uses many pieces of the ADAS paper. I would also expect AgentSquare to compare to ADAS, since a central claim is that it is an improvement over the approach ADAS took. Instead, the paper is written in such a way that a busy reviewer/reader might not know such a similar work exists, and thus mistakenly attribute credit for the big ideas, results, and novelty to this new work.\n\nIf the authors are somehow so unaware of what is appropriate and required in academic writing that this is all an honest mistake, then in my view they should let the community know that, apologize, and entirely rewrite the paper to properly inform readers. \n\nNote: this is almost certainly not a case of the work being done concurrently, because the paper copied so many pieces from ADAS (including prompts used in their experiments). \n\nTo be clear, there are some nice new innovations in the paper (if the data are to be believed, see below). Properly written (and with more careful comparisons to ADAS and ablations of what makes this paper different and if they help), this paper could have been accepted as a nice innovation on top of ADAS. Instead, I believe it should be rejected as plagiarism. There is not enough time and rounds of review in this conference's review process for reviewers to consider a full rewrite (including back and forth) at this stage. \n\nHere is evidence of untoward behavior:\n\n1. The prompts from ADAS are largely copied verbatim and then modified in a few places. The amount of overlap makes it certain they copied from our work, yet they did not acknowledge doing so. They should report using a prompt from another paper. But more importantly, it shows how similar the work is to, and was inspired by, ADAS, and thus that ADAS should have been mentioned throughout (as prior work, something to be compared to, something they built on and were inspired by, etc.). You can see the similarities here: https://drive.google.com/file/d/1vHPW2EXvx7LjFv-kDhQHyb_VGHTnndD8/view?usp=sharing\n\n2. ADAS did something rare: it said it was (A) recognizing and naming “a new research area” named Automatic Design of Agentic Systems, and (B) introducing a new search algorithm within that area named Meta Agent Search. AgentSquare clones that template, saying “we introduce a new research problem: Modularized LLM Agent Search (MoLAS)” and “building on this design space, we present a novel LLM agent search framework called AgentSquare.” It is uncommon for papers to name a new research area explicitly, let alone also introduce an instance within it. \n\n3. Moreover, MoLAS is not a new research area. It is a subset of ADAS. It is a more constrained subset of ADAS, at least the way it is initially presented, which is that MoLAS is constrained to combine pre-existing, human-designed agent modules. However, in a contradiction that is confusing and harms novelty, later the paper also says the modules can be changed, removing this constraint, meaning that MoLAS is thus just ADAS with a new name?\n\n4. Even if ADAS is not identical, it clearly is a very related prior work, yet it is not mentioned in the introduction, nor anywhere in the paper until the second-to-last paragraph. Yet other, much less related work is mentioned (line 053), suggesting an intentional choice to deceive the reader/reviewer by hiding work that hurts the novelty claims of this paper. Also, some of the claims in the introduction say effectively that things like ADAS do not exist or are rare (line 047): clearly ADAS should be mentioned in that context, but isn’t. \n\n5. The plot style of ADAS Fig. 3a is very similar to Fig. 4, providing further evidence of the central role the ADAS paper played in inspiring and catalyzing the AgentSquare paper. \n\n6. One (somewhat, see * below) new thing in AgentSquare vs. ADAS is an explicit surrogate model that predicts an agent’s performance, saving compute vs. running a real evaluation. However, I am very skeptical of this data. Even with much training and years of research, getting surrogate models to be highly predictive (e.g. in Neural Architecture Search) is very difficult. I have a hard time believing an LLM is even good zero shot, yet these data say it is nearly PERFECT zero shot. That raises an important question: why are there so few (5) points shown per run (e.g. versus the many more shown in many more per run in Fig. 4)? How did you select these 5? Can you provide code that replicates this experiment, so the community can run it independently and verify it? I do not mean the overall search, which is expensive, but just the zero-shot predictions and the evaluations for all agents produced per experiment. I have never asked for such a thing in review before, but extraordinary claims require extraordinary verification, in my mind, especially given the cloud of suspicion raised by all the other issues in this review.  If the surrogate model is indeed this good, that is a major discovery and worthy of publication in some form, and I recommend you share it in a properly written paper. \n\nThe net effect of all of this evidence forced me to conclude the paper is at best deeply flawed due to many innocent mistakes, and at worst intentionally dishonest, with the latter seeming far more likely. \n\nIf there were no plagiarism issues, I would also point out:\n\n- There is no ablation of memory, one of the key new things they claim they added, and thus no evidence it helps\n\n-  The paper should cite https://arxiv.org/abs/2206.08896\n\n- *Surrogate models arguably implicitly exist within ADAS since it asks for proposals the model thinks will be high-performing, meaning it could use its predictions to guide proposals. However, we know from things like Chain of Thought and Reflexion that one can coax better performance by asking an LLM in the right way and/or asking it to reflect post-hoc on a creation. \n\n- The paper claims that ADAS’ search in the space of code is an over-simplification. While it is true that describing the search space is simple, searching in it is MUCH more complex. I think it is much more appropriate to say that MoLAS is simplifying search by constraining it to a simple, small search space vs. ADAS, the latter of which allows any possible agent. \n\n- The paper claims that ADAS is proposed without consideration of existing agent architectures. I disagree, since one can easily inject them as seeds in the search archive (and we did consider that). But we prefer that to be a choice: it likely speeds up search, but also biases it. \n\nIt saddens me to make these accusations. However, I deeply believe in the integrity of the scientific process, including peer review. It relies on trust, and I think we need to stay vigilant and hold people accountable so we can maintain that trust as much as possible. \n\nJeff Clune\n\nProfessor, Computer Science\nUniversity of British Columbia\nJeffClune.com\n\nCanada CIFAR AI Chair & Faculty Member\nVector Institute", "questions": "See main review.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "See main review.", "weaknesses": "See main review.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)