MetaAgent: Automatically Constructing Multi-Agent Systems Based on Finite State Machines
Authors: Yaolun Zhang
Xiaogeng Liu
Chaowei Xiao
Venue: ICML 2025 poster (2025)
Keywords: LLM Agent
Multi-Agent System
Primary area: deep_learning->large_language_models
TL;DR: We introduce a new framework that can automatically produce multi-agent systems base on finite state machines

Abstract: Large Language Models (LLMs) have demonstrated the ability to solve a wide range of practical tasks within multi-agent systems. However, existing human-designed multi-agent frameworks are typically limited to a small set of pre-defined scenarios, while current automated design methods suffer from several limitations, such as the lack of tool integration, dependence on external training data, and rigid communication structures. In this paper, we propose \textbf{MetaAgent}, a  \textbf{finite state machine} based framework that can automatically generate a multi-agent system. Given a task description, MetaAgent will design a multi-agent system and polish it through an optimization algorithm. When the multi-agent system is deployed, the finite state machine will control the agent's actions and the state transitions. To evaluate our framework, we conduct experiments on both text-based tasks and practical tasks. The results indicate that the generated multi-agent system surpasses other auto-designed methods and can achieve a comparable performance with the human-designed multi-agent system, which is optimized for those specific tasks.

MetaAgent: Automatically Constructing Multi-Agent Systems Based on Finite State Machines
Yaolun Zhang!
Xiaogeng Liu
Chaowei Xiao
Abstract
2024). To further improve the performance, the multi -agent system has been proposed, which improves and enlarges the abilities of the agent by assigning different roles and skills to LLMs and designing effective cooperation mechanisms to organize them (Hong et al., 2024b; Qian et al., 2023; Yan et al., 2024; Huang et al., 2023). Despite the success, most of the existing multi-agent systems are still manually de- signed, introducing human efforts to implement the complex codebase, and need several iterations of human polishing. Moreover, these frameworks are built only to solve tasks in some specific scenarios, further enhancing the design cost.
Large Language Models (LLMs) have demon- strated the ability to solve wide range of prac- tical tasks within multi-agent systems. However, existing human-designed multi-agent frameworks are typically limited to a small set of pre-defined scenarios, while current automated design meth- ods suffer from several limitations, such as the lack of tool integration, dependence external training data, and rigid communication structures. In this paper, we propose MetaAgent, finite state machine based framework that can auto- matically generate a multi-agent system. Given a task description, MetaAgent will design multi- agent system and polish through an optimiza- tion algorithm. When the multi-agent system is deployed, the finite state machine will control the agent's 's actions and the state transitions. To eval- uate our framework, we conduct experiments on both text- based tasks and practical tasks. The results indicate that the generated multi-agent system surpasses other auto-designed methods and can achieve a comparable performance with the numan-designed multi-agent system, which is optimized for those specific tasks. The code can be found at: https://gitubb.com/aaaoo WISC/MetaAAeet/.
To address it, few works try to build multi-agent systems automatically (Chen al., 2024a; Wang et al., 2024d; Yuan et al., 2024). However, current works have failed to con- struct complete and practical multi-agent system due to several reasons. SPP, AutoAgents, and EvoAgent (Chen et al., 2024a; Wang et al., 2024d; Yuan et al., 2024) de- sign multi-agent systems for each specific case. In other words, the produced multi-agent system can only handle the specific case and lacks generalization to other cases in the same task domain. SPP and AutoAgents do not sup- port tool-using well. ADAS and Symbolic-Leaminn (Hu et al., 2024; Zhou et al., 2024) build multi-agent systems automatically based on self-iteration algorithms. However, tons of iterations and external data are needed for opti- mization. Moreover, following the communication structure of human-designed multi-agent systems (Hong et al., 2024b; Qian al., 2023; Du et al., 2024), current works use lin- ear, decentralized debate coordinate with an orchestrator cooperation structure to organize agents (Su et al., 2024), which have limited traceback abilities to fix bugs in previ- ous steps when encountering errors or misunderstanding.
1. Introduction
Large Language Models (LLMs) (OpenAl et al., 2023; Zhao et al., 2023) show a spring-up of intelligence, containing strong ability of reasoning, coding, and numerous com- pressed knowledge. Utilizing LLM as the brain to build agents can complete various complex tasks, which require the agent to plan, utilize tools, and make reflections (Yao al., 2023; Shinn et al., 2023; Wang al., 2024a; Qin et al.,
To address the limitations of human- -eeaand multi-agent systems and drawbacks cxisting auto-design methods, we introduce MetaAgent: A framework that can automatically design Finite State Machine (FSM) based multi-agent sys- tems for a large spectrum of tasks.
Specifically, given a general description of a type of task, MetaAgent will first design agents needed to solve the task. Then, to organize these agents, several states are summa- rized based on the possible situations involved in solving the task. Each state includes the corresponding task-solving
Universitty Wisconsin Madison, Madison, US. Correspon- dence to: Chaowei Xiao <cxiao34@ wisc.edu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
Property Framework
MetaGPT AutoAgents
SPP EvoAgent ADAS Symbolic MetaAgent
Auto-Designed
x
Generalization
J
X
X
X
d
Tool Enabled
X
X
J
Traceback Ability
X
X
X
X
X
X X
Non-External Data Depend
/
X
Table 1. Comparison of existing and proposed Multi-Agent Frameworks. The properties are vital for automatically and effectively building robust Multi-Agent Systems.
agent, the instructions for the task-solving agent, the condi- tion verifier who checks whether the output meets certain state transition conditions, and the listener agents who will receive the output of the state. This design leverages the LLM's deccision-making ability dynamically manage the cuoblem-solvingg process when encountering different cases within the given type of task.
is a Product Manager Agent. The FSM then transitions to the next state, where the Product Manager Agent designs the product by creating its Product Requirements Document (PRD). Should the Product Manager Agent identify incor- rect information during this design phase, the system can trace back to relevant previous state for correction.
Other types of multi-agent systems can be seen as con- strained versions of the Finite State Machine (FSM). A linear structure is an FSM with only one state transition function for each state, meaning lacks state traceback and condition verifier. The decentralized debate structure in- cludes limited traceback from the last state the first state. The coordinated system with orchestrator is similar to an FSM with a shared condition verifier. The FSM structure, with a customized condition verifier for each state and un- constrained state transition conditions, is highly suitable for auto-design scenarios because it maximizes the flexibility of the multi-agent system.
To improve the practical deployment of our FSM-based multi-agent system, we designed an optimization algorithm to merge redundant FSM states. This was motivated by our observation that the initial FSM design often suffered from excessively long chains of information transfer and task-solving, hindering performance. To address this, our algorithm traverses each pair of states within the FSM, us- ing a Large Language Model (LLM) to determine their mergeability. This method optimizes the FSM structure by eliminating trivial states, thereby enhancing the system's 's ro- bustness. Unlike related works (Hu et al., 2024; Zhou al., 2024), our optimization approach requires neither external data nor extensive training steps.
To verify that our MetaAgent is general and robust frame- work capable of automatically producing customized multi- agent systems for various scenarios. we conduct experiments on realistic tasks. These include Machine Learning Bench (Hong et al., 2024a), software development tasks (Zhou et al., 2024), and Text-Based tasks including Trivial Cre- ative Writing (Wang et al., 2024d), and GPQA (Rein et al., 2023), which widely used to evaluate other auto-design multi-agent systems. The experiments indicate that the multi-agent system produced by the MetaAgent surpasses other automatic systems and achieves performance compa- rable to manually designed systems tailored for the tasks, In Bext-Based tasks, our MetaAgent method surpasses previ- ous prompt-based SOTA methods by 9%. In the Machine Learning tasks, the multi-agent system generated by MetaA- gent achieved 97% of the average performance of the best human-da -uman-designed multi-agent system, surpassing all other human-designe and multi-designed frameworks. In the software development task, MetaAgent passed 50% more checkpoints than the human-designed system. Our abla- tion study on tool usage, optimization, and traceback shows
When deployed, starting from the initial state, the user query and the current state's instructions serve as inputs for the task-solving agent. The agent's output is then sent to the condition verifier, who checks whether the agent's output matches any pre-defined state transition condition. If con- dition is met, the system transitions to the next state, which could be an appeared state, allowing for state traceback. Before transitioning, the agent's 's output is saved as memory for listeners.
Figure 1 shows how the FSM works. For example, if the user's task is to "Build a Pac-Man Game", the FSM designed for software development begins with the Information Col- lector Agent, who will use search engine to gather infor- mation. The Condition Verifier then checks if the collected information meets the transition conditions. If errors are found, feedback will be given to the Information Collector Agent to help it refine its actions. If the action is successful, the collected information will be sent the listener, which
2

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
Finite State Machine based Multi-Agent System
State 2
Build
State
Search relative information according to user query
Design the product
Pac-Man
Product Manager
Game
User
Transition
Information Collector
Pre-defined Condatons:
LLM Reasoning:
Conditions: When information
To design the product, need thorough information NPC, which provided by information Collector
collected success fully, trunsit to state 2.
Using
Condition2: Otherwise,
refine
Tool Result:
Feedback
LLM Reasoning:
Condition Verifier
The objective of the game est of the
dots placed in the muze while avoiding four ghosts.
matches
Conditionl.
On geing If any condition mutched
Pre-defined Conditions:
Case2: The Agent failedd the tool correctly, and bug ....
Condition2: When needed, transit state 1.
Condition Verifier
Case2: Match Condition2.
State Traceback
Figure An example of what a state is, and how our finite state machine structure works.
decreases in performance on the aforementioned tasks, high- lighting the importance of these features.
As the growing trend of automatic design, SPP (Wang et al., 2024d) introduces a prompt-based method to build a lin- ear multi-agent system for each specific case, invoking the compressed knowledge by assigning the roles. AutoAgents (Chen et al., 2024a) is built on the codebase MetaGPT and further improves the multi-agent system by adapting planning and multi-turn cooperation between agents. ADAS and Symbolic Learning (Hu et al., 2024; Zhou et al., 2024) try to optimize a multi-agent system from a given simple system, but they both need external data and steps for iter- ation and focus more on the inner structure of each single agent. However, there a lack of methods efficiently and automatically build tool-enabled multi-agent system that can handle specific domain.
2. Related Works
2.1. Multi-Agent System
Previous works have discussed multi-agent systems in var- ious scenarios. One category of Multi-Agent Systems are designed to simulate real-world scenarios (Park al., 2023; Xu et al., 2023; Hua et al., 2023), where the researchers can find some rules or conduct social experiments.
In this research, we focus on the multi-agent system, which built for problem-solving. Early works use merely the reasoning ability of LLM to build systems like debating, vot- ing, and negotiating (Wu et al., 2023; Du et al., 2024; Yan et al., 2024; Bianchi et al., 2024). Later works implement tool-using and more complex communication structures for the system. MetaGPT and ChatDev (Qian et al., 2023; Hong al., 2024b) build a Multi-Agen System for software development and introduce message pool to manage com- munication. Datalnterpreter and AgentCoder (Hong al., 2024a; Huang al., 2023) focus on data science or Python code problems, but are also limited to pre-defined scenarios. There are a few works that apply the finite state machine to control the agentic system (Wu et al., 2024; Liu et al., 2023; Chen et al., 2024b). However, they are all human-desiin -eeind,,,,, limited certain scenarios as well as use hard-coded meth- ods to detect certain output strings the transition function, which not adapt to complex real- world scenarios.
2.2. Tool LLM
Utilizing tools is significant feature of LLM Agent well MetaAgent Framework, because it enables the Agents to interact with external worlds, enlarging their ability scope. Previous works about Tool LLM can be divided into two categories. The first category teaches LLMs to utilize a wide range of real-world APIs via function-calling, with a focus on the breadth of tools (Patil et al., 2024; Qin et al., 2024). The second category focuses on the usage of some specific tools, like search engines and code interpreters, that can complete multiple tasks. CodeAct (Wang et al., 2024b) first assigned code as actions and integrated various functions into the Python code snippet. PyBench and MINT (Zhang et al., 2024; Wang et al., 2024c) evaluate LLM cquipped with a code interpreter on multiple tasks. Gao et (2023)
3

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
shows LLM Agent equipped with a search engine has significant ability growth in numerous information-seeking tasks. Our MetaAgent mainly equips the agents with code interpreter and search engine, promoting the tool-using ability in the area of automatic multi-agent systems.
3.2.2. STATES AND TRANSITION CONDITIONS DESIGN
Following the agent design phase, the designer constructs a FSM based on the previously defined agents and the overall task description. This FSM explicitly defines a set of states and the natural language conditions that govern transitions between them. The designer must act as farsighted planner to anticipate various scenarios that agents may encounter during the task-solving procedure (e.g., different types of in- put materials, varying results from intermediate actions, and diverse final outputs) and encapsulate them into this FSM structure, which consists of states and transition conditions. The design of the FSM involves the detailed specification of its core components: states and transitions.
3. Method
3.1. Definition of Finite State Machine
The finite state machine is model defined by tuple M S, so, F, (Hopcroft al., 2001; Carroll & Long, 1989), where Σ is the input alphabet and in our set- ting it is the set of specific cases in the task domain where the FSM is designed to solve, S is the finite set of states, 50 E S is the initial state, F C is the set of final states, and 8 is the state transition function. In the MetaAgent, each state represents a possible situation in the process of solving a problem, characterized by task-solving agent, condition verifier, a state instruction, and listeners who receive the output upon state completion. The FSM starts at so and transitions between states based on the transition function and input symbols until it reaches a final state in F, indicating task completion, or until exceeds the maximum number of transitions, indicating task failure.
Each state in the FSM represents a specific situation en- countered during the task-solving procedure, for which the designer defines several key components. A central element is the State Instruction, a pre-defined natural language instruction outlining the specific sub-task the assigned task- solving agent needs to address when this state is active. Coupled with the instruction is an Assigned Agent, one of the agents designed in the previous step. When the state is activated, this task-solving agent utilizes its memory, con- taining the user input and information from any previous state where the agent served listener, to follow the state instruction and attempt to solve the current sub-task. Finally, the designer specifies I Listeners for the state, designating a list of other agents to receive the output of the current state. This control over information flow, inspired by Hong et al. (2024b), is crucial. After the Condition Verifier confirms a state transition, the final output of the task-solving agent in the current state is inserted into the memory of all its listener agents. Furthermore, all agents listen to the user input initially, ensuring that every agent knows the initial tasks, which guarantees information alignment.
3.2. Construct the Finite State Machine
3.2.1. AGENTS DESIGN
The first step of designing an FSM is to design the agents. This agent design method is fundamentally prompt-centric, initiated when a designer LLM is furnished with the general task description. To enhance the efficacy of the designed agents, the designer is guided by prompts to generate a comprehensive task analysis and a clear system goal. This preliminary reasoning phase is pivotal for optimizing the subsequent agent generation. Concurrently, the designer is constrained to propose the most parsimonious yet effective ensemble of agents deemed essential for the task for cost- efficiency.
Transition conditions determine the control flow within the FSM after an agent attempts to resolve a state, potentially leading to an advanced state, tracing back to a previous state, or remaining in the current state. To operationalize the evaluation of the natural language conditions, each task- solving agent is paired with Condition Verifier. The designer configures this verifier by setting its system prompt to be the same as the task-solving agent's 's system prompt, appended with all the natural language state transition con- ditions defined for exiting the current origin state. After the task-solving agent generates its solution, the Condition Verifier checks if the agent's 's output meets any of the pre- defined state transition conditions. If any condition is met, the verifier will guide the state transition to the appropriate destination state. If no condition met, it will execute a null-trastion, giving feedback to the task-solving agent for action refinement.
Following this foundational analysis, the designer continues to identify and profile these potential agents, outlining their functionalities relevant to the task domain. The finalized agent configurations are then rendered in a structured JSON format, including each agent's name, system prompt, and assigned tools. The agent's name is crucial for streamlining the subsequent design of states and transition conditions. The system prompt defines its role, core responsibilities and tasks, operational limitations, and expected response format. The tools allocated by the designer can help the agents solve problems within its defined scope.
4

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
Construction Stage
Initial State
General Task Description
FSM
(4)
Optimizing the FSM
When match condition__. J the state need convert to state_
Eg. Build Multi-Agent System software development
Final State
(3)
States
Adaptor
Can states merged?
Tools
Designer
A&B
is_ $_jnitial:: True Agent Instruction Listeners
is_ final: True
Agent Instruction
Agents
Merging Until Stable
...
Optimized FSM
Define Name, System prompl, and Tools
Figure 2. The construction stage of MetaAgent
3.2.3. OPTIMIZING THE FSM
minimizes complexity and enhances the performance of the FSM without compromising its functionality.
The initial version of the FSM frequently failed due to an excessively large number of states, many of which were redundant or could be consolidated.
3.3. Deployment Stage
After the construction stage, the multi-agent system is stable and ready for deployment in practical scenarios. In specific task domain, the finite state machine operates according to A Algorithm 2. Initially, the state set to 80, and agent in this state acts based on the given instructions and query. The output, which is combination of L.MM text and tool responses (if used), is evaluated by the condition verifier with transition conditions. It will assess whether condition is met and identify the target state for transition. If a condition met, the state transitions to the target state, and the output the current state inserted into the memory of the listeners, ensuring the flow of information. transition function indicates that the state is complete for no condition is met, the finite state machine will choose null-transition, continuing to call the current agent until transition condition is met or the maximum number of interactions M is exceeded. Figure shows an example of how a finite state machine works.
To address this issue, we optimized the FSM by systemat- ically merging states that are considered equivalent based perform
specific criteria. Given the state set (S) pairwise comparisons of all possible state pairs. For each pair, an adaptor LLM determines whether the two states can be merged. For the key advantage of multi-agent system over a single agent the ability to leverage diverse roles to stimulate various aspects of the LLM's knowledge, the adaptor will assess whether the roles of the agents in the two states sufficiently distinct. If they are not, the states and their corresponding agents will be merged.
the
If a pair of states meets the criteria for merging, we combine them into a single state and update the state set. We then restart the pairwise comparison process with the updated set. This iteration continues until no further states can merged and the state stabilizes.
When combining states, the task-solving agents associated with those states are also merged. The adaptor merges the system prompts of the agents and updates the instructions for the combined state accordingly.
3.4. Features of Finite State Machine
Several features help the finite state machine become a suitable structure for multi-agent systems.
By applying this iterative merging process, we effectively reduce the number of states in the FSM. This optimization
Null-Transition. In the domain utilizing large language
5

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
models (LLMs) to solve complex and practical tasks, it is crucial enable refining debugging for tasks that cannot be resolved in single turn. The FSM structure is naturally suited for these requirements because the condition verifier can choose a mull-Transition, which means giving feedback to the task-solving agent and staying in the current state for action refinement. This mechanism allows the task-solving agent to operate in multiple turns, enhancing the robustness of its actions and enabling it to solve more complex problems that require iterative refinement.
the limited space of Σ. In these systems, E contains only one case. In contrast, FSM design allows E to encompass set of cases within the task domain, which is a generalized framework and much more efficient when dealing with a large number of cases.
Decentralized Debate as FSM. Another multi-agent sys- tem structure the decentralized debate. LLM Debate and AgentCoder are representative examples (Du et al., 2024; Huang et al., 2023). In this structure, agents make claims or execute actions over multiple rounds. After the last agent's s action, control returns to the first agent for new round of conversation, continuing until a consensus is reached. Compared to the linear structure, the debate structure in- volves traceback mechanism, although typically restricted to tracing from the latest state back to the first one. And it still does not support tull-trannition. Thus, it can be viewed as an FSM with limited traceback conditions and without null- transtiaanss
State Traceback. In general problem- processes, encountering errors or misunderstanddnns from previous steps is inevitable. Existing aulti-agent systems with linear structures, such as SOPs, lack mechanisms to address these issues as they operate on predefined linear pipelines. Our FSM model enables state traceback by allowing transitions back to previous states when the condition verifier detects issues stemming from earlier steps. For example, in a soft- ware development task, if the QA Test Agent discovers that certain functionalities are missing, the FSM can transition back to the state where the Programmer Agent works on those functionalities, facilitating iterative refinement.
Coordinate with Orchestrator as FSM. Coordinating with an Orchestrator is an advanced multi-agent system structure. An Orchestrator agent dynamically decides the next step and the corresponding agent. Magentic-One and Datalnterpreter are examples (Fourny et al., 2024; Hong et al., 2024a). This structure can be considered as an FSM where every state shares a single condition verifier, which serves as the overall controller of the multi-agent system. Our comprehensive FSM, which includes null-transitions, can flexibly establish traceback conditions and provides a condition verifier for each state, making it a more general structure for automatically designing multi-agent systems.
3.5. Generalization of multi-agent systems as Finite State Machines
Su et al. (2024); Guo et al. (2024) categorize existing multi- agent system structures into three types: Linear, Decen- tralized Debate, and Coordinate with Orchestrator. We demonstrate that these structures are all weakened or spe- cialized versions of Finite State Machines (FSMs). Figure 3 illustrates the difference between structures.
4. Experiment
Linear Systems as FSM. Linear multi-agent systems, such as MetaGPT, Agentg, and SPP (Hong et al., 2024b; Chen et al., 2024a; Wang et al., 2024d), involve agents connected in a strict sequence. In these systems, each agent performs a specific function before passing control to the next agent. This structure can be mathematically represented as:
4.1. Setup
We conducted a series of experiments on different tasks to show the versatility and robustness of MetaAgent. Firstly, we compare MetaAgent with other prompt-based based methods Trivial Creative Writing (Wang et al., 2024d) and GPQA (Rein al., 2023). After that, we compare MetaAgent practical tasks including machine learning (Hong et al., 2024a) and software development (Qian et al., 2023) tasks. We selected GPT-40 as the foundation model in the main experiments and set the temperature to 0 to ensure repro- ducibility. We prepare the code interpreter and search engine in the tool pool for selection.
8,+1 = 0(8, Vi = 0,
where so is the initial state, sn is the final state, and o, is the output of s,. Transitions occur deterministically without cycles or branches, making this a special case a finite state machine (FSM) with a stationary state transition graph.
Moreover, in these linear systems, there is state trace- back or null-transition, which means they cannot handle unforeseen conditions or backtrack. This lack of flexibility differentiates them from full FSMs.
4.2. Results on Text-Based Tasks
Datasets. Two benchmarks are used for evaluating the performance of our method and baselines in taxt-based tasks. Specifically, we use Trivial Creative Writing and GPQA(Diamond), Trivial Creative Writing is dataset that contains 100 tasks consisting of several questions and 5
Specifically, the auto-desige methods used by AutoAgents and SPP for multi-agen systems are also constrained by
6

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
possible answers for each question. The model is required to write a story that contains answers to every question. GPQA(Diamond) is a dataset that contains 198 multi-choice graduate-level scientific questions.
datasets, which is appointed in each task description. For the software development tasks, unlike other benchmarks (Hong et al., 2024b; Qian et al., 2023), which primarily rely on subjective evaluation metrics, we designed objective checkpoints for each software. These checkpoints include accessibility, functional completeness, and control ability, Each software is evaluated on four key points, earning one point for each test it passes. The metric used is the ratio of passed tests. The details are presented in Appendix B.
Metries. The success rate is our primary evaluation metric. For Trivial Creative Writing, it is defined as the proportion of questions whose answers are adequately covered by the story. For GPQA, it is determined by the proportion of correct answers.
Baselines. Both human-designed and auto-designed Frame- works are selected as baselines for Machine Learning Bench, AutoGen (Wu et al., 2023), Openlterpreter (Lucas, 2023), Task Weaver (Qiao et al., 2023), and Datalnterpreter (Hong et al., 2024a) are typical numan-designed multi-agent frame- works that can adapt machine learning tasks. We then adapt SPP (Wang al., 2024d) and AutoAgents (Chen et al., 2024a) to the ml_bench by extracting the generated code and getting the execution result.
Baselines. We select prompt engineering methods in- cluding Direct, CoT (Wei et al., 2022), CoT-SC (Wang et al., 2023), llm-debate (Du et al., 2024). and Relf-Refine (Madaan et al., 2023) as well as SPP (Wang et al., 2024d), an automatic multi-agent method.
Results and Analysis. The results show MetaAgent out- performs all other methods, achieving the highest score of 0.86 on writing tasks and 0.60 on GPQA(Diamond) (Table 2). Specifically, the designed multi-agent system, equipped with a search engine and a code interpreter, can surpass another Auto-Designed method, Solo-Performance- Prompting(SPP), who design a multi-agent system for each case.
For software development tasks, we choose MetaGPT (Hong et al., 2024b), which designs a fixed SOP to organize the process of software development. We also adapt AutoA- gents and SPP (Chen al., 2024a; Wang et al., 2024d) to the software development task by extracting the code they generated and save them to the files.
Table 2. MetaAgent's Performance on Trivial Creative Writing and GPQA. The results show that MetaAgent produces multi-agent systems that surpass other prompt-based methods.
Results and Analysis. Table 3 presents the results on the Machine Learning Bench. The multi-agent sys- tem generated by MetaAgent outperforms all other auto- designed frameworks, which lack the mechanism to utilize tool feedback and thus process the dataset with halluci- nations. MetaAgent also surpasses most human-designed multi-agent systems, demonstrating the robustness of its finite state machine. It achieves state-of-the the-ttt (SOTA) performance on the Titanic and House Prices datasets and secures the second-highest scores on other datasets, showing comparable performance to Datalnterpreter, a multi-agent system specifically tailored for machine learning tasks. To analyze more deeply, we find that MetaAgent can generate multi-agent system comprising "Data Preparation and Model Selection Agent," a "Model Training Agent," and a "Report Agent". Following the designed state instructions, these agents can perform feature engineering, explore the dataset's structure, and share the detected information with other agents. They can also train various models and report the best one. These features enable the multi-agen system to surpass others.
Method Task
Writing
GPQA
Direct
0.76
0.46
CoT
0.74
0,44
CoT-SC
0.74
0.49
llm-debate
0,73
0.54
Self-Refine
0.76
0.55
SPP
0.79
0.45
MetaAgent
0.86
0.60
4.2.1. WORLD CODING TASKS
Datasets. Two datasets are selected to evaluate MetaA- gent's performance on more practical tasks. Machine Learn- ing banch(ml_bencch (Hong et al., 2024a) is a benchmark that requires agents to train a machine-leariing model for regression or classification.
Software development is a comprehensive and practical task for evaluating agent systems, often used to assess various multi-agent frameworks. We selected several representative software development tasks, including game and web app development (Zhou et al., 2024).
Table 5 presents the results for five different software devel- opment tasks, demonstrating that our MetaAgent framework not only outperforms other auto-designed frameworks but also surpasses MetaGPT, human-designed multi-agent framework for software development. Without tool-using capabilities, the performance of AutoAgents and SPP is significantly lower. Additionally, MetaGPT is constrained
Metrics. For the Machine Learning Bench, the normalized performance score (NPS) serves as the metric to evaluate the quality of the trained machine learning model on the given evaluation datasets. The NPS normalizes the different machine learning metrics including FI-Score, accuracy, or RMSE of the trained machine learning model on the test
7

---

MetaAgent:
Automatically Building Multi-Agent System based on Finite State Machine
Table Normalized performance score MI. Bench. The MetaAgent performs best among Auto-Desigee Auto-Designed Multi-Agent methods and
has comparable performance with Data Interpreter, a Human- Dessgned multi agent system specific for Machine Learning Tasks.
Method/ / Task
Auto-Designed
Titanic
House Prices
SCTP
ICR
SVPC
Average
AutoGen
X
0.82
0.88
0.82
0.71
0.63
0.77
Open Interpreter
0.81
0.87
0.52
0.25
0.00
0.49
Task Weaver
0.43
0.49
0.00
0.65
0.17
0.35
MctaGPT
X
0.81
0,00
0.73
0,00
0.71
0.45
Data Interpreter
X
0.82
0.91
0.89
0.91
0.77
0.86
SPP
0.82
0,00
0.00
0,00
0.00
0.16
AutoAgents
0.00
0,00
0.00
0.00
0.00
0.00
MetaAgent
0.83
0.91
0.86
0.88
0.68
0.83
Table Performance on Software Development Tasks.
The software produced by MetaAgent passes most the checkpoints and
surpasses all the other methods.
Method / Task
Auto-Designed
2048
Snake
Brick breaker
Excel
Weather
Average
MetaGPT
x
0.25
0.25
0.75
0.50
0.35
AutoAgents
0
0.75
0.25
0.20
SPP
0.25
0.50
0.15
MetaAgent
0.75
1.0
0.50
1.0
1.0
0.85
by its linear structure, which lengthy and lacks the ability to trace back like a finite state machine.
4.4. Ablation Studies
To demonstrate the importance of Meta-gent's key features, we conducted ablation studies focusing on its core compo- nents: tool-using, traceback, and optimization. Additionally, examined how the quality of foundation models impacts the framework's s performance.
MetaAgent designed multi-agent system consisting of "Requirement Designer," a "Code Developer," and "Tester". The tool-using and traceback features of the finite state machine contribute to its success. It can test whether the software can start and run smoothly via a code interpreter and trace back to the code development state to fix bugs.
Tool-Using augments the Agent System's knowledge for text-based tasks. Tool-using is a crucial part of the finite state machine. When equipped with tools, the task-solving agent of state can interact with the file system or the in- ternet solve complex tasks. The condition verifier will help to analyze the tool feedback as well, establishing multi-turn turlti-turrain interactive environment, which can enhance the performance of the finite state machine. According to the result in Table 6, the performance on Trivial Creative Writ- ing and GPQA(Diamond) has decreased when the tool disabled. The equipped search engine, which can collect external information from the internet, truly helps the multi- agent system clarify the answers and reach higher score.
4.3. Cost Analysis
We conducted a cost analysis to demonstrate the effective- ness of our Meta/gent framework, focusing on machine learning and software development tasks. We calculate the total token cost of solving tasks in the Machine Learning Bench and 6 tasks in software development, respectively.
Table 7 shows the results indicating that MetaAgent achieves higher or comparable performance at a relatively lower cost compared to other methods. Specifically, when compared to MetaGPT, human-designed method, the auto-designed methods, despite incurring additional token costs during the design stage, result in more effective multi-agent systems. This effectiveness is reflected in the lower average token cost observed during the deployment of auto-designed systems.
Traceback helps the Agent System fix previous bugs flex- ibly. The state traceback feature also contributes a lot when solving complex and unpredictable tasks. In the case that the current agent finds the input information needs to be re- fined via the previous state, the finite state machine enables traceback to the previous one and transmits the information to that agent. This design ensures the finite state machine is better at handling various situations, which distinguishes it from common linear structures like SOPs. The result of the ablation experiments also proves the assertion. In particular, we find that multi-agent systems without a traceback design often fail due to unresolved bugs. For instance, when the tester discovers a bug while executing the software code, they cannot relay this information back to the programmer
Additionally, we compared case-level design and task-level design approaches. AutoAgents designs a multi-agent sys- tem for each specific case, while MetaAgent designs multi- agent system for a type of task, which contains the cases. The findings reveal that case-level designed multi-agent sys- tems are more costly and less applicable to general and massive tasks, underscoring the efficiency and broader ap- plicability of MetaAgent's task-level design.
8

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
Table Results on Machine Learning Bench when transferring the foundation model of Design and Executor of MetaAgent.
Designer
Executor
Titanic
House Prices
SCTP
ICR
SVPC
Average
GPrb,5-Turbo
GPT3.5-Turbo
0.73
0.00
0.00
0.00
0.00
0.12
GPrb.5-Turbo
GPT-4o
0.81
0.00
0.82
0.00
0.68
0,39
GPT-4o
GPT3.5-Turbo
0.73
0.00
0.80
0.00
0.00
0.26
GPT-4o
GPT-4o
0.83
0.91
0.86
0.88
0.68
0.83
Table 6. Ablation Studies Tool-Using Traceback and Optimization. (""" means not applicable)
ML.Bench
Software
Writing
GPQA
Methods
Score
A(%)
Score
A (%)
Score
Á (%)
Score
A (%)
MetaAgent (w/o tool-using)
0.79
8.11
0.52
13.33
MetaAgent (w/o optimization)
0.61
+ 26.5
0.65
4 35.3
0.65
24.4
0.56
$ 6.67
MetaAgent (w/o traceback)
0.72
4 13.3
0.35
4. 58.8
0.77
4 10.5
0.58
3.33
MetaAgent
0.83
0.00
0.85
0.0
0.86
0.00
0.60
0.00
the performance decline is more pronounced when the ex- ecutor's quality is reduced, suggesting that the executor's quality may play a more critical role in the system's 's overall performance.
Table 7. Total Token Cost for 5 Machine Learning Tasks and Software Development Tasks. The result shows that MetaAgent has the lowest cost practical tasks.
Method
Stage
Software
MI. Bench
Design Deploy Total
(human design)
(human design)
5. Conclusion
MetaGPT
60,237
65,355
60,237
65,355
This paper introduces MetaAgent, a framework that automat- ically generates multi-agent systems using finite state ma- chines. It overcomes the limitations of designed and auto-designed systems by providing tool-using and trace- back capabilities, and can self-optimize without external data.
Design
19,832
21,615
AutoAgents
Deploy
32,448
40,970
Total
52,180
62,585
Design
5,378
6,226
MetaAgent
Deploy
42.374
39,663
Total
47,752
46,289
Impact Statement
without a traceback mechanism.
MetaAgent introduces an automatic method for constructing multi-agent systems using a Finite State Machine (FSM), es- tablishing a unified and intuitive framework. This approach offers a significant advantage for Artificial Intelligence re- searchers, enabling them to explore the development of more powerful agentic systems and address a wider array of real- real-world challenges. Furthermore, MetaAgent holds substantial value for industrial companies. Engineers can readily leverage this method to build and adapt customized multi- systems with specialized tools, thereby acceler- ating their workflows. The low cost building and deploy- ing systems with MetaAgent further enhances its practical appeal.
Reduce system redundancy through optimization. When designing the multi-agent system, optimizations are required to make the system more robust. The optimization method can get rid of some unnecessary agents or inter- mediate states to simplify the work pipeline and enhance robustness. Results in Table show that a sharp decrease in performance is caused by the absence of optimization. In the bad cases, we do observe that the system struggles complete the task due excessively long text caused unnecessary steps.
Foundation Models' quality plays a more important role as Executor. The question of how much the quality of the designer and executor's foundation models affects a sys- tem's overall performance is intriguing. To investigate this, we used GPT-4o as the designer and GPT3.5-Turbo as the exccutor, and vice versa, using ML.Bench an example. Our findings indicate that decreasing the quality of either the designer or the executor leads to significant drop in overall performance. Specifically, when both the designer and executor use GPT-4o, the average score is 0.83. How- ever, replacing the designer or executor with GPT3.5-Turbo reduces the score 0.26 and 0.35, respectively. Notably,

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
References
Hong, S., Lin, Y., Liu, B., Liu, B., Wu, B., Li, D., Chen,
J., Zhang, J., Wang. J., Zhang, L.,, Zhang. I.., Yang. M., Zhuge, M., Guo, T., Zhou, T., Tao, W., Wang, W. Tang, X., Lu, X., Zheng, X., Liang, X., Fei, Y., Cheng, Y., Xu, Z., and Wu, C. Data interpreter: An Ilm agent for data science, 2024a. URL https: www.aiiiogggaaaaaeeoooo
Bianchi, F., Chia, P. J., Yüksckgönül,, M., Tagliabue. J., Jurafsky, D., and Zou, J. How well can Iims negotiate? negotiationarena platform and analysis. In Forty-first International Conference on Machine Learning, ICML. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, wwwpreeviiiieeee/orurrree ner/Eorum
2024. URL ? Fid-Cm0maxkt8p.
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C.,, Wang, Z., Yau, S. K. S., Lin, Z.., Zhou, L., Ran, C., Xiao, L.,, Wu, C., and Schmidhuber, J. Metagpt: Meta programming for A multi-agent collaborative frame- work. In The Twelfth International Conference on Learn- ing Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.nel, 2024b. URL https: //openevieww net forum? http://frmmmmdd/tdaaeeea d=
Carroll, and Long, D. Theory of Finite Automata: With an Introduction to Formal Languages. 1989.
Chen, G.. Dong, S., Shu, Y., Zhang, G., Sesay, J., Karls- son, B., Fu, J., and Shi, Y. Autoagents: A framework for automatic agent generation. In Proceedings of the Thirty-Thit Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3- 9, 2024, pp. 22-30. ijcai.org, 2024a. URL https: //www. i j ca i. wwtpoorrgeeeee..oeo http://oocenn//////////
Hoperoft, J. E., Motwani, R., and Ullman, J. D. Introduction automata theory, languages, and computation, 2nd edition. SIGAC/ News, 32(1):60 2001. ISSN 0163- 5700. doi: 01114445566438556444555 URL https: // //doi. wwt. 1ttpg//1..114556848888883 568455.
Chen, W. You, Z., Li, R., Guan, Y., Qian, C., Zhao, C., Yang, C., Xie, R., Liu, Z., and Sun, M. Internet of agents: Weaving a web of heterogeneous agents for collaborative intelligence, 2024b. URL https: //tp:///a/vvvv//v org/
Hu, S., Lu, C., and Clune, Automated design of agentic systems, 2024. URL https://arxiv, 08435.
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mor- datch, I. Improving factuality and reasoning in lan- guage models through multiagent debate. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL htap::///ppereeeeee wttpe/ewwweeeeeeeeeeeeeeeeeeeeee net/forum forum Pid=zj7YuTE4t
Hua, W., Fan, L.., Li, L.,, Mei, K., Ji, J., Ge, Y., Hemphill, L.,, and Zhang, Y. War and peace (waragent): Large language model-based multi-agent simulation of world wars, 2023. URL https: / /arxiv.ivvv./////////./////avvvv org/ /abs//////////
Huang, D., Zhang, J. M., Luck, M., Bu, Q. Qing, Y., and Cui, H. Agentcoder: Multi telti--gget--baatd based code generation with iterative testing and optimisation, 2023. URL ht /ps://arxx org/ abs / 221222112222 13010.
Fourney, A., Bansal, G., Mozannar, H., Tan, C., Salinas, E., Erkang, Zhu, Niedtner, F., Proebsting, G., Bassman, G., Gerrits, J., Alber, J., Chang, P., Loynd, R., West, R., Dibia, V., Awadallah, A., Kamar, E., Hosn, R., and Amershi, S. Magentic-one: A generalist multi-agent system for solving complex tasks, 2024. URL https:
Liu, J., Shuai, J., and Li, X. State machine of thoughts: Leveraging past reasoning trajectories for enhancing prob- lem solving, 2023. URL https://arxiis org/ab
s s/2312 //2312.17445.
// //arxiv. org/ abs/ / 2211111111111 1.1.1....11. 04468.
Lucas.
Available at: https://githii: wttp:/tw...ooo//o//ooeeeeeeeeeee catom//pennttepreteeee inpen-interppeter.
Openinterpreter, 2023.
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., and Wang, H. Retrieval-augmente generation for large language models: A survey, 2023. URL https: http//axivvv./// wtwpxiiworr/abs//11...
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S. Yazdanbukhsh, A.. and Clark, P. Self-refine: Iterative refinement with self-feedback. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans,
Guo, T., Chen, X., Wang, Y., Chang, R., Pei. S., Chawla, N. V., Wiest, O., and Zhang, X. Large language model based multi-agents: A survey of progress and challenges. In Proceedings of the Thirly-Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pp. 8048-8057. ijcai.org, 2024. URL https:/ / //t///wwww www. ijaii. org/ /prcceeeeeeeee di dings/2024/890.
LA, USA, December 10 16. 2023, 2023. URL http://papers nips wwtpppppppppc..eteecc.. fi ferrrfiieepppppp les/pap
10

---

org/ab MetaAgent: Automaticallly Building Multi-Agent System based on Finite State Machine
Tuggle, P. Turley, N., Tworek, J., Uribe, J. F. C., Vallone,
html.
A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang,
J. Wang, A., Wang, B., Ward, J., Wei, J., Weinmann,
C., Welihinda, A., Welinder, P., Weng, J., Weng, L.. Wi- ethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q. Zaremba, W., Zellers, R., Zhang. C., Zhang, M., Zhao. S., Zheng. T., Zhuang, J.. Zhuk, W., and Zoph, B. Gpt-4 technical report, 2023. URL https: / araii.iiiivv.... org/ abs aat//aas////3/ 2303.0030000 08774.
OpenAl, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. I.., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, L., Bal- aji, S., Balcom, V. Baltescu, P. Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapinn G., Berner, C., Bogdonoff, L.. Boiko, O., Boyd, M., Brakman, A.-L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H. W., Cummings, D., Cur- rier, J.. Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontijo-Lppes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S., Guo, Y., Hallacy, C.,, Han, J. Harris, J., He, Y., Heaton, M.. Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu. K. Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Lukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim, J. W. Kim, C., Kim, Y., Kirchner, J. H., Kiros, J., Knight, M., Kokotajlo, D., Lukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung. J., Levy, D., Li, C. M., Lim, R,, Lin, M., Lin, S., Litwin, M., Lopez, T. Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D.. Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang. L.. O' Keeffe C., Pachocki, J. Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G. Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F. Petrov, M., de Oliveira Pinto, H. P.. Michael, Pokorny, Pokrass, M., Pong, V. H., Powell, T., Power, A., Power, B., Proehl, E., Puri, R. Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K.. Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T.. Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P.. Sidor, S.. Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song, Y., Staudacher, N., Such, F. P., Summers. N.. Sutskever, I., Tang, J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, E.,
Park, S., O'Brien, J. C., Cai, C. Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior, 2023, URL https:// a arxiv. wexiv.org/eo. abs 2304 03442.
Patil, S. G., Zhang, T., Wang, X., and Gonzalez, Gorilla: Large language model connected with massive apis. In Globersons, A., Mackey, I.., Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http: //papers. nips ct httpp/ps.cppppppppp.eeeeeee pwwp:aapppeeee www.s/flleesppppeeeeeeeeeaeeesee
Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., Cong, X., Xu, J., Li, D., Liu, Z., and Sun, M. Chatdev: Communicative agents for software development, 2023. URL abs/2307.00
Qiao, B., Li, L., Zhang, X., He, S., Kang, Y., Zhang, C., Yang, F., Dong, H., Zhang, J., Wang. L., Ma, M., Zhao, P., Qin, S., Qin, X., Du, C., Xu, Y., Lin, Q. Rajmo- han, S., and Zhang, D. Taskweaver: A code-first agent framework, 2023. URL https://arrs s/2311. 17541.
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., Zhao, S., Hong, L., Tian, R., Xie, R., Zhou, J.. Gerstein, M., Li. D., Liu, Z.. and Sun, M. Toolllm: Facilitating large language models to master 16000+ real-world apis. In The Twelfth Interna- tional Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. 2024. URL https: http:////peeeeeeeeeeeeeemee net forum ? Pid=dHing200Jjr
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-pros proof q&a benchmark, 2023. URL https: pttbss//arxivvoo/ /bs///////// 2311 12022.

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: language agents with verbal reinforcement learning. In Oh, A., Naumann, T., Glober- son, A., Saenko, K., Hardt, M., and Levine. S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023. URL http: pape rs www..ps....... cc. http::pppppppppcc pawer_apppppeeeea fi files/ htt.le//ppprr//223/////
S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume 1: Long Papers), Pp. 257-279, Mexico City, Mexico, 2024d. Association for Computational Linguis- tics. URL https: /awcaananntooooyyooooorcc org/ 2024. naacl-long.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F. Chi, E. H., Le, Q. V., and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: An- nual Conference Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http: / wwp.erss.ipps... cttpp//ssrppppsscppppp.pee.eeeee filerr/p_ll//pppp e hash//aaaahaaaa http://ssaaaaas5000013eeeeeeeaas 7b31 -bw.aabbatttatttaabbaba..a.aaaae Conpssttrrr---ooforooee Coiee--ooooorre html.
html.
Su. Y., Yang, D., Yao, S., and Yu, T. Language agents: Foundations, prospects, and risks. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pp. 17-24, Mi- ami, Florida, USA, 2024. Association for Computational Linguistics. URL https: ac www.aolloogly.... org /2024. wwwwemmmmttrriieeeeeee tepptuutorrlls..
Wang, L.,, Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J.. Chen, Z., Tang, J., Chen, X., Lin, Y., Zhao, W. X., Wei, Z., and Wen, J. A survey large language model based autonomous agents. Frontiers of Computer Science, 18(6), March 2024a. ISSN 2095-2236. doi: 10.1007/ $11704-024-40221111 URL http://dt /xx www.doi.oo. ori
Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang. S., Liu, Awadallah, A. H., White, R. W., Burger, D., and Wang. C. Autogen: Enabling next- gen Ilm applications via multi-agent conversation, 2023. URL https /ttp///axivvvveee//ee//eeeeeaaea org/ aas///b////// /2088 08155.
Wu, Y., Yue, T., Zhang, S., Wang, C., and Wu, Q. State- flow: Enhancing Ilm task-solving through state-driven workflows, 2024. URL https: http//aawvvv.gr..r...e s s/2403, 11322.
Wang, X., Wei, J. Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reason- ing in language models. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: /etpa::/opeerevviie wwtpeewwwwwwwweeeeeeeeeeeeeeeeee pddf? d=1P LINIMMIW.
Xu, Y., Wang, S., Li, P., Luo, F., Wang, X., Liu, W., and Liu, Y. Exploring large language models for communication games: An empirical study on werewolf, 2023. URL https: arxiv. .rgg abs / 220//339..000 04658.
Yan, Y., Zhang, Y., and Huang, K. Depending on yourself when you should: Mentoring Ilm with rl agents to become the master in cybersccurity games, 2024. URL https: / aaaxiiv... org/ abs 22203
Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., and Ji, H. Executable code actions clicit better LLM agents. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.nee 2024b. URL https: //openreview nwt iidiiddjja JJ9BOXAfFa.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali. Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: bttps://openreet /ttp://////eeeeweew/e net/pdf?i d-WE WWE_vluYUL-X,
Wang, X., Wang, Z., Liu, J., Chen, Y., Yuan, L., Peng, H., and Ji, H. MINT: evaluating Ilms in multi-turn turn interaction with tools and language feedback, In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, net/ forum
2024c. URL httpp:://openee
Yuan, S., Song, K., Chen, J., Tan, X., Li, D., and Yang, Evoagent: Towards automatic multi-agent generation via evolutionary algorithms, 2024. URL https: arxx
?id= ?id= Jp3=MMuu37
Wang, Z., Mao, S., Wu, W. Ge, T., Wei, F., and Ji, H. Un- leashing the emergent cognitive synergy in large language models: A tosk-solving agent through multi multi-persona self- collaboration. In Duh, K., Gomez, H., and Bethard,
v. orrrg/ abs /24062466..0 14228.
Zhang, Y., Pan, Y., Wang, Y., Cai, J., Zheng, Z., Zeng, G., and Liu, Pybench: Evaluating Ilm agent on various
12

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
real-world coding tasks, 2024. URL https://arxi
v.org/abs/224// 2407 16332.
Zhao, W. X., Zhou, K., Li, J., Tang. T., Wang, X., Hou, Y., Min, Y., Zhang. B., Zhang. J., Dong, Z., Du, Y., Yang. C., Chen, Y., Chen, Z., Jiang, Ren, R., Li, Y., Tang, X., Liu, Z., Liu, Nie, J.-Y. and Wen, J.-R. A survey of large language models, 2023. URL https:/ /arxi v. v.org/a abs 2303. 18223.
Zhou, W., Ou, Ding, S., Li, L.,, Wu, J., Wang, T., Chen, J., Wang, S., Xu, X., Zhang, N., Chen, H., and Jiang, Y. E. Symbolic learning enables self-evolving agents, 2024. URL https: /ttt://aaxiii http:///awxwv....aaseeesssesseee wwt.v/orggbbb///eeese 1 8532.
13

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
A. General Task Descriptions
Software Development Task Build a multi-agent system that develops software. The multi-agent system could also save the developed software a local file system and write README for the user.
Machine Learning Task Build Multi-Agent system that can train machine-learning model based on the given dataset. And report the expected metrics (like F-1 score, RMSE and etc. on the test dataset.
Trivial Creative Writing Task Build Multi-Agent System that can input a list of questions and then output a story that includes answers all the questions in the list.
B. Software Tasks
Evaluation Criteria the criteria.
We design several evaluation criteria for each software development task. Table 8 demonstrates on
Task Name 2048game
Evaluation Criteria
1. Can open an interface
Can operate normally
3. Can merge correctly
4. Can score correctly
Snake Game
1. Can open an interface
2. Can operate the snake normally
Can eat beans correctly
The snake can grow normally
Brick Breaker Game
Can open an interface
Can operate the paddle normally
Can eliminate bricks correctly
4. Can score correctly
excel app
1. Can open interface
2. Can transfer files correctly
3. Can display correctly
Can close correctly
weather
1. Can open an interface
Has weather query function
Can fetch weather data correctly
Can display weather data aesthetically
Table 8. Evaluation Criteria for Software Development Tasks
C. Statistics of MLE Bench Finite State Machine
In this section, show the statistics of example finite state machine. Table C shows the static statisties including the states and transition count before and after optimization. And the Null- Transition (remains in the current state) and traceback transition count in test case.
D. Optimization Method
D.I. Algorithm
The algorithm shows the detailed method that optimizes the FSM.
14

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
Table 9. Statisties of Mench Finite State Machine
Table 10. Static Statistics
Table 11. Dynamic Statistics
Metric
Count
Metric
Count
FSM States(Initial)
Total Transitions (Initial)
Null- Transitions
3
FSM States(Optimized)
Traceback
2
Total Transitions (Optimized)
Total Transition
9
Algorithm FSM State Optimization
Require: State set S
Ensure: Optimized state set
1: function LLM(staler, states)
return true if state; and statez can be merged, otherwise
3: end function
4: procedure OptimizeFSM(S)
5: repeat
6: merged + false
7: for each pair (si, sj) in S do
8:
LLMM(s;; 85) then
9:
Merge s; and s into a new state Siy
10:
Update 5 by replacing 8; and 8, with 8ij
11: merged <- truc
12: break
13: end if
14: end for
15:
until merged= = false
end procedure
This prompt describes the standard of whether two states can be merged.
D.2. Prompt of Updating the Multi-Agent System
You are given descriptions of two states in a finite state machine (FSM). Your task is to determine if these two states can be merged based on the following criteria:
**Role Distinguishabillty"** Evaluate if the roles associated with the states are sufficiently distinct. If the roles are not distinct, the states should be merged. **Information Necessity*** Assess if the information transfer between the states necessary. If the information transfer is unnecessary, the states should be merged. 3. **Tool Assignment** Check if the tool assignments actions associated with the states overlap can be unified. If they can be unified, the states should be merged.
the states can be merged, output the merged state description in JSON format. If the states cannot be merged, output 'FALSE'.
State Description: {state_1.descriptin
State Description: (state deseriptionn)
Based on the above criteria, determine if the states can be merged and provide the appropriate output."
E. Deployment State
15

---

MetaAgent:
Automatically Building Multi-Agent System based on Finite State Machine
Algorithm 2 Deployment Stage
Require: specific case Q, max iterations M, Finite State Machine { S, Su, con) ) A state contains the corresponding agent s.Agent, the instruction to the agent s.Ins, the listener agent who will receive the state output Lis and the condition verifier for the state 8. er
1: $ 5 80
c+
3: while < M do
4: oulpul s. s.Agenl(s.Ins, Q)
5: Starget + V er (outtut)
6: if Starget N None then
7: oulpul + s.Agenl(s. Ins, oulpul)
8:
c ctc+1
9:
else
10:
Starget c c+e
11:
12:
for Lis in s.Lis do
13:
memory ins oulpul)
14:
end for
15:
end
16:
end while
Linear
Finite State Machine
Coordinate with Orchestrator
Fixed Transition
Decentralized Debate
Shared
Condition Verifier
Flexible Transition and Traceback
(2)
XN
Customized Condition Verifier
Fixed
Traceback
(3)
Solution
Consensus
Solution
Figure 3. Compare FSM with Other Kinds of Multi-Agent System Structures. The figure shows the Linear, Decentralized Debate, and Coordinates with Orchestrator structures" difference between the finite state machine.
16

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
F. Compare between FSM and other frameworks
G. Example Multi-Agent Systems
Here an example Multi-Agent System for Software Development
"agents":
"agent_id" "0", "name": "RequirementDeiigne:" "systee_proopt" "You are RequirementDesignen Your goal is to understand the software requizements and create design architecture for the software Your responaibility is to gather and analyze the requi rements for the software project and ensure that the design is robust and scalable.", "tocls": I Shearch_enging
10
I1
12
"agent_id" "1", "name": "CodeDeveloper", "systee_promppt" "You are CodeDeveloper. Your goal write the actual code for the software based on the design provided by You are also responsible for writing README file for the user and saving the developed software to a local sile system. Ensure that the code is clean, efficient, and furctional.",
15
"tools": "file_writer writer"
16.
17
19
20
"agent_id" "2", "name": "Tester", "systee_prompt's "You are Tester. Your goal is to test the software to ensure it works as Intended. Your responsibility is to identify and report any bugs or issues the software. You should also report the expected metrics on the test dataset to the user.",
21
23
"tocls":
"code_interpreter"
26
27
1,
"states": "states";
30
31
"state_id"; "1",
32
"agent_id": "0",
"instruction": "Gather and analyze software requirements and create
design architecture based the requirements." "is initiai;; true,
34
35
"is_final": false, "Listener": "1"
36
38
39
40
"state_id id": "2" "agent_id": "1"
43
"instruction": "Write the actual code based on the design, write a README filo, and save the devoloped software to a local file system.", "is_initial"; false, Vis_final" false, "listener": "
44
47
17

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
48
49
30
Scrate_id", "3",
"agent_id"; "2", "instruction": "Test the software to ensure works as intended. Report the expeeted metrics (like F-1 score, RMSE, etc.) ) on the test dataset to the user.", "is "is_initial"; false, "is_final" final": false, "listener": "0", "1"
53
55
56
57
60
61
62
"state_id"; "4", "agent_id"; "0", "instruction": "<]subrit|> The a response to the user, example: subbiit |> The software is developed and the metrics on the test dataset are reported.",
"is "is_initial": false, "is_final"; true, "Listener"; []
67
68
}
69
1.
"transitions":
72
"from "from_state"" state"; "1", "to :to_state": "2", "condition"; "If requirements are clear and complete and design is robust and scalable"
23
75
76
77
stroo_statee: "2", itt_state", "3", "condition": "If code is clean, efficient, and functional and README is clear, informative, and easy to understand"
79
80
"fron_state"" "3", "to_state": "4", "condition"; "If the software works as intended and metrics are reported"
84
85
"Erom "Eron_state"" state": "3", sto_state"; "2",
88
89
"condition"; "If the test is not passed"
90
93
H. Prompts
1.0.1. MULTI-AGENT SYSTEM GENERATION
You are the designer a multi-agent system. Given general task description and list of agents, you need to generate Finite State Machino (FSM) to manage the process of solving the task.
WARNING: You good contrelling costs, many agents and too complex cooperation structure can lead to excessive costs informati in exchange
18

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
Each state the should include:
1. state_id; A unique identifier for the state
2. agent_id; The ID of the agent associated with this state
3. instruction: What the agent should do in this state
4. is_initial: Boolean indicating if this is the initial state
5. is_final: Boolean indicating if this a final state
10
6. listener: The agent who will save this state output information in their memory Notice Make sure the listener covers all related agents. The agents not
11
listed as listener would not received the informaticn (which may cause the failure cosperation)
12.
Hence, some important milestone like a new version of code/answer should be broadcast all related agent!
The FSM should also include transition functions between states. Each transition function should specify:
15
1. from stame: The ID of the state this transition is from
16
2. to_state: The ID of the state this transition goes to
3. conditicn: description of the condition that triggers this transition
18
19
Your answer should follow this format:
20
Reasoning: <Your saap-by-ster reasoning process?
Answer:
"'json
23
{{
24
"states":
25
{{
"state_id"; "1"
27
"agent_id"; "0",
28
"instruction": "Perform task x",
29
"is_initial"; true, "is "is_final": false, "listener"; ["1", "2"]
30
32
}h
33
34
1,
"transitions":
{{.
37
"trom_state" "1"
38
"to_state": "2",
39
"cendition"; "If task X is completed successfully"
42
"from_state"; "2",
43
sto_state"" "1",
44
"condition": "If the previous task needs to be re-done."
})
46
47
48
}}
50
51
Rules:
52
1. Ensure there is exactly one initial state and at least one final state.
2. Every non-final state should have at least one outgoing transition.
3. The FSM should able handle loops and complex interactions between agents.
55
4. Include transition to final state that submits the final answer (use <|submit
in the instruction).
56
5. Make sure all agent_ids in the states correspond to the provided agent_dict.
6. The transitions should consider as many possible situations. which consisit roadmap for Multi-Agent System deployment stage.
19

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
H.1. Running Example
The general task description (Input the Designer):
Build Multi-Agent system that can train machine-learning model based on the given dataset. And report the expected metrics (like F-1 score, RMSE, etc.) on the test dataset to the user.
Generate the initial version of Finite State Machine
"agents":
"agent id"; "0",
"name": "pataPreprocess ingAgent", "systen_prompt": "You are Datapreprocessi igggent. Your goal is to clean and prepare the given dataset for machine learning. Your responsibilities include handling missing values, encoding categorical variables, and normalizing numerical features. Ensure the dataset is in a format suitable for model training. You can use the following tools: code code_intene interpreter: Use it with <execute> "pythonn <Your Codo>
and you will got
the stdout or error message An WARNING: Thses enironment is not a jupyter notebook. Please use print head instoad of df df. other jupyer outputs also need print out If If dataset is prepared successfully, output : 2n If no conditions are met, output STATE_TRANS>: None 'nn DO NOT WRITE THIS IN THE CODE SNIPPET!", "tools": "code interpreter"
10
11
"agent_id"; "1", "name"; "Modelseeecctionngeet"" "You are Kodelseleetionkgent. Your goal is to select the most appropriate machine learning model based on the characteristics of the prepared dataset. Consider factors like the type of problem (classification regression), dataset size, and feature types. Output the selected model type. An- If model is selected successsully, output *<STATE, TRANS>: : 3 - If no conditions are met, output SETATE_IRANN>> None ' An DO NOT WRITE THIS IN THE CODR SNIPPRT!", "tocls": []
13
14
15
16
17
18
pagent_id*s "2",
19
"name": "Mode MrodelTraaningggent "systen__rompt"s "You are ModelTraiiningggett Your goal is to train the selected machine learning model using the prepared dataset. Ensure to split the dataset into training and validation sets, and optimize the model's hyperparameters if necessary. Output the trained model. You can use the following tools:\n- code_interpretee
Use it with <execute> "python <Your Code> x*< lexecutee, and you will got the stdout or error message \n WARNING: Thses enironment is not a jupyter notebook. Please use print (df, ()a instead of dł, heade other jupyer outputs also need print out An- If If model is trained successfully, output <<SAATE_TRAN TRANS>: 4 An- IE no conditions are met, output "<STATK_ITAASS" None '. DO NOT WRITE THIS IN THE CODE SN SNIPPET!", "tocls";
21
22
"code interpreter"
25
26
"agent_id"; "3", "nane"; "EvaluationAgent" Dsystem_prompt" "You are RvaluationAgent. Your goal is to evaluate the trained model on the test dataset. Compute the required metrics such F-1 score, RMSR, and any other relevant metrica. Output the evaluation results. You
27
20

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
with <execute> 1" python <Your Code> (Xx lexecute>, and you will got the stdout or error message\n WARNING: Thses enironment is not jupyter notebook, Please use print (df ()) instead of df. df. other jupyer outputs also need print outAn- If If model is evaluated successfully, output '<STATE TRANS>: 5".An- IF no conditions are met, output ><STATE_ <<STATE_TRANS>; None DO NOT WRITE THIS IN THE CODE SNIPFET!", "tocls":
can use following tools: An- code Interrpee:::
29
"code crode_interpretee"
31
12
33
"agent_id"; "4", "name": "ReportingAgent",
36
"systen__ronne prompt": "You are Reportingagent. Your goal to compile the evaluation netrics and generate a comprehensive report for the user. Ensure the report is clear, concise, and includes all relevant metrics and insights. An- If conditions WRITE THIS IN THE CODE SNIFFET!", "tcols": []
met, output "<STATE_TRANS TRANS> None ' An DO
37
38
1,
"states": {
41
"states":
42
43
sitate_id": "1" "agent_id"; "0"
45
"instruction": "Clean and prepare the given dataset for machine learning.", "is "is_initial"; true, "is fisafin false,
46
47
48
"listener": "1", "2"
50
51
52
}
"state_id"; "2" bogent_id"; "1",
55
56
"instruction"; "Select the most appropriate machine Learning model based on the prepared dataset.", "is_initial"; false, "is_final"; false, "listener"; "2"
58
59
60
63
64
"state_id"; "3", "agent_id" _d"; "2",
65
"instruction": "Train the selected machine learning mode? using the prepared dataset.",
67
"is_initial"; false, "is "is_final": false, "listener": "3"
68
71
72
73
"state_id"; "4" "agent_id"; "3", "instruction": "Evaluate the trained model on the test dataset and compute the required metrics.", "is_initial"; false, false,
75
76
77
79
"Listener";
21

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
81
82
"state_id". _i" "5", "agent_id"; "4", "instruction"; "Compile the evaluation metrics and generate a comprehensive report for the user. use </subnit/>; Your answer to subit the answer to user", "is_initial"; false, "is_final"; true, "listener": []
85
86
87
88
89
1,
92
"transitiens";
93
94
"fron_state"; "1"
"to_state"; "2", "condition": "If dataset is prepared successfully"
97
98
"Erom_star :trom_state"; "2" "to_state"; "3", "condition"; "If model selected successfully"
101
102
103
"fron_state" "3", "to_state"; "4",
106
"condition": "If model trained successfully"
107
"fron_stato"; "4", "to_state"; "5",
111
"condition"; "If model is ovaluated succossfully"
112
Finite State Machine Optimization The finite state machine optimized by merging unnecessary states.
"json
"agents":
"agent_id"; "0",
"namc:
"system_prompt"'. "You are
Your goal is to clean and prepare the given dataset for machine learning, select the most appropriate model, train it, and evaluate it the test dataset. Your responsibilities include handling missing values, encoding categorical variables normalizing numerical features, selecting the model, training the model, and computing the required metrics such as F-1 score, RMSE, and any other relevant metrics. Knsure the dataset is in format suitable for model training and evaluation. You can use the following tools: An- code_interpreter: Use it with execute> " 'python <Your Code> 192) lexecutex and you will get the stdout or error message \n WARNING: This environment is not Jupyter notebook. Please use print headd)) instead of df. head other Jupyter outputs also need print out \n- IF dataset is prepared, model is selected, trained, and evaluated successfully, output ASTATE_IRANSY, 2*An. AnA no conditions are met, output * STATE_TRANS>: None', An DO NOT WRITE THIS IN THE CODE SNIPFET!",
tools":
11
"agent_ids "1",
22

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
12
"name": "ReportingAgent""
13
"system_prompt" "You are Reportingagent. Your goal is to compile the evaluation metrics and generate a comprehensive report for the user. Ensure the report is clear, concise, and includes all relevant metrics and insights. IE no conditions are met, output ><STATE, ><STATE_TRANS> TRANS>: None Bane".Wo An DO NOT WRITE THIS IN THE CODE SNIPPET!",
14
"tools": []
15
}
17
"states": {
18
"states":
19
staae_id"; "1",
pagent_id"; "0",
22
"instruction": "Clean and prepare the given dataset for machine leazning, select the most appropriate model, train it, and evaluate it on the test dataset. Ensure to handle missing values, encode categorical variables, normalize numerical features, and compute the required metrics such F-1 score, RMSE, and any other relevant metrics. Output the ovaluation results.", "is_initial"; true, "is_final": false, "listener": ["1"]
23
24
27
28
"state "state_id"; "2", "agent_id"; id"; "1", "instruction": "Compile the evaluation metrics and generate comprehensive report for the user. Use < <laubmit|>s Your answer submit the answer the user." ", "is initiall"" false, "is =is_final": true, "listener": []
29
31
32
33
35
},
36
"transiticns";
37
{
"from_stater "1"
"te_state"; "2",
40
"condition"; "If dataset prepared, model selected, trained, and evaluated successfully"
41
]
43
44
45
### Explanation
88
19
1. * Aggntt***:
50
This agent is responsiblc for the entire data preparation, model selection, training, and evaluation process. This reduces the need for multip'e agenta and minimizes communication overhead. Reeorttinggeent:: This agent is responsible for compiling the evaluation metrics and generating the final report.
++DataP
51
2.
**State***:
54
**State 1**: The data preparation, model selection, training, and evaluation. It outputs the evaluation results and transitions to the next state if successful.
handles all tasks related to
**State 2*x: The 'Reportingagent compiles the evaluation metrics and generates the final report, then submits the answer to the user.
56
57 3. **Transiticnse*:
58
-A single transition from State 1 to State 2 ensures that the process is streamlines ined and efficient. The condition for this transition is the successful completion of data preparation, nodel selection, training, and evaluation.
23

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
Deployment Stage
DataAncModelAgent
Lot's s proceed with the data preprccessing steps, which include handling missing values, encoding categorical variables, and normalizing numerical features. After that, we will train machine learning model and evaluate its performance.
Here are the steps we will follow:
1.
Handle missing values.
2. Encode categerical variables.
3. Normalize numerical features.
4. Train a machine learning model.
10 5. Evaluate the model's performance.
Let's start with the data preprocessing steps.
13
14 <exeoute> "ppputt>>"pythoo 'python
15 import pandas as pd
from skicarn. sadaaanmmmdllsseeettiooo import train_test_eplit
from irleasspreproessinng import StandardScaler, OneHot&ncooder
18 from sklearn.compost import Col rommrrrrrrorrr
19 from sklearn.pipel1n import Pipeline
20 from sklearn. impute import SimpleImputer from Mekearn.ennemblln import RandemF cannomserestClasssiiir
22 from skiearn.metrice import securacy_score
23
24 Load the dataset
train_data_path - /ttaa////aaaltte/////e////e/eeer
split_train.cov'
26
eval_datt_ http://///////////////////////// split_eval.cst
eval haal_ddat_pathh
train_df pd. ttr.dccctttttrainnddtttt..
29 ev
30
11 # Separate features and target
X_train train_df.drop (col Jumns- (trrssssSSuvviiii Survived', Passengerid', Name' (ficket', "Cabin
31 y_train train_df[ Survived']
34 X eval eval df. drop (columns= Survived', 'PassengerId', 'Naee' 'Ticket', "Cabin' ])
35 eval eval df l Survived'
Preprocessing for numerical data 38 fumerical_fcatures l'Age', 'sibsp', 'Parch', 'Fare"]
39 bumerical_trranffrmnee Pipeline (steps=[
40 ('imputer', Simplelmputer (strategy= median")) ) ('scaler', StandardScaler ())
})
43
44 Preprocessing for categorical data 15 Letegorrcaa__teturee = l'Polass', Sex', 'Embarked' |
eategorical_transfone
- Pipeline (steps-
(' ('imputer', simplemmutter Imputer ('onehot', OneHotkncoder (handle (ggp__uunnnnooooognnnnnee
48
49 1)
Bundle preprocessing for numerical and categorical data preprocessor ColumnTransformert
52
53
transformers-
54
numerical_fcaturss) ('cat', trranssorrr,,,, ])
('num',
57
24

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
Define the model
59
model RandomE crest oreeecilassiieeeeeeettnnnttooooo (nesstiaattrrtt11000 randon_state-O)
60
Create and evaluate the pipeline
clf Pipeline (steps= pppppooccessr'' preprocessor), ('classifier', model)
63
64
65 Freprocessing of training data, fit model
clf fif. y_train)
67
88 Preprocessing of evaluate avaluation data, get predictions
89 Y_pred clf.predict (X_eval)
Evaluate the medel
12 accuracy accuracy_sscooe score (y eval, y_pred)
19 print (E' Accuracy: (accuracy)')
74
<'execute>>
DataAncModdlAgent
76 Lot's s proceed with the
preprccessing, feature engineering, and nodel training steps. We'11 handle missing values, encode categorical variables, normalize numerical features, and then train a model to predict the survival outcome. Finally, we' 11 evaluate the model's performance
the evaluation dataset.
18 Here is the complete code to achieve this:
19
80 "execute>""yyyhon 'pythen
import pandas pd
82 from Ailearnn.odel_selee
import train_test_split
83 from
import StandardScaler, OneHotEncoder
84 from sklearn.compor import ColumaTransformer
from sklearn.pipeliae import Pipeline from sklcarn.impun import SimplcImputer
87 from sklcarn.cnsemble import RandomForestclassii
88 from sklcarn.metrics import accuracy_score
Load the datasets 89
spiit_train csv' httpa/////////////e/tt/t//////// //tat///aal/aasttt/////at/tttttt /ttpa//e//eaaaaaaaoaeaaa/aaaaa/e
92 eval dadaaaappaataa evalt_evvl..cv
bwnchmark titanic/
94
train_df pd. reraaaeettttraii_aata.atttt) df pd. (eval
95
96
Separate features and target
X_train train_df.ddi drop (columns=( Survived', 'PassengerId' 'Naee', "Ticket', 'Cabin'))
99
y_train train_df Survived'] Xevva= eval_df drop (columns= Survived', 'Passengerid', 'Naee, 'Ticket', "Cabin') )) y_eval eval_ Survived']
100
Preprocessing pipeline numeri cal laal_ffeaurrss ['Age', 'sibšp', 'Parch', (Fare'] numeri cal _all_transformrr Pipeling ine (steps-[
numerical features
104
105
(' imputer', SimpleImputer (strategy='meds median' ) ('scaler', StandardScaler
1)
109
Preprocessing pipeline for categerical features
eategorical_fcaturee ['Pelass', 'Sex', 'Embarked']
etegorical__raasfooee = Pipeline (steps=[ (' Vimputer', SimpleImputer tntraateg="nost_freqeeett , ('onehot', OneHotEncoder (handle mnndlle_nknowwm//55 (inaeee__unooonnnniooooee
113
most_
114
}}
Compine preprecessing steps
25

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
preprocessor ColummTransformer
119
trans formers-
120
(' num', numerical_transfomme numerical_features).
('cat', categorical
catborical_ffattrrre
}]
123
124 Create the model pipeline
model = Pipeline (steps=| (' ('preprocessor', preprocessor),
('elassifier', RandomForest (random statt=)2)
128 })
129
Train the model
model.fit (X train, y y_train)
132
133
Make predictions on the evaluaticn set y_pred model. predict (X_eval)
136
Evaluate the model
137
accuracy accuracy_score (y_eval, Y_pred)
138
print (f' Accuracy on the evaluation set: (accuracy:.4f)) "*<lexxcute>>
141
Accuracy on the evaluation set: 0.8324
142
143
ARSUUL
DataAnddooeelAgent
<STATE_TRANSS> 2
146
ReportingAgent
147
### Titanic Passenger Survival Prediction Report
#EFF 1. Data Analysis
The dataset used for this analysis is the Titanic passenger survival dataset. The target column is 'Survived', which indicates whether a passenger survived (1) or did not survive (0).
####
Data Preprocessing
153
Msing Values**: Handled missing values columns such as 'Age', 'Cabin', and Embarked'.
Converted categorical variables like 'ex' EEnbarkdd,, and Polass into numerical values using one-hot encoding. Feature Scalingax: Applied standard scaling numerical features to ensure they are on a similar scale.
*Categorical Variables*::
156
#### 3. Feature Engineering
**iill Extraction*x: Extracted titles from passenger and created new feature. **Family Sizc**: Created a new feature representing the size a passenger's family.
160
*IIAAlon**:: Created a binary feature indicating if a passenger was traveling alone.
### 4. Modeling
Several machine learning models trained and evaluated including:
164
logistic Regression
165
decision Tree
Random Forest
Gradient Becsting
Support Vector Machine (SVM)
169
170
UUFF 5. Evaluatien Metrics
The models were evaluated using accuracy the evaluation dataset.
175
Model
Accuracy
174
Logistic Regression
0.80
Decision Tree
0.78
17
Random Forest
0.82
26

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
Gradient Bocsting 0.83 Support Vector Machine (SVM) 0.81
179
180
#### 6. Best Model
The perf--erfo model **Gradient Boosting.. with accuracy of +*0.83** on the evaluation dataset.
183
184
#F#F 7. Conclusien
The Gradient Becsting model the most effective in predicting passenger survival the Titanic dataset, achieving an accuracy of 83%. Further improvements could be made by tuning hyperparameters, adding more features, or using ensemble methods.
186
</submit||>
This is titanic passenger survival dataset, your goal is predict passenger survival outcome. The target column Survived. Perform data analysis, data preprocessing, feature engineering, and modeling to predict the target. Report accuracy on the eval data. Train data path:
http:////a....aaacooooooo/aaaaaa eval data path: ' /ttpaa/dawt......aaaaaa.eeeeeeee
Data
Path 1s: /Users/ wtt.a//w....aa..aaaaaaaaaaaattaa csv
189 ML Bench Answer: ### Titanic Passenger Survival Prediction Report
#### 1. Data Analysis
192
The dataset used for this analysis is the Titanic passenger survival dataset. The target column is 'Survived', which indicates whether a passenger survivea (1) or did not survive (0).
#EFF Data Preprocessing
195
MMi ssing Values*4: Handled missing values columns such as 'Age', "Caiin' and Embarked'.
** *Categorical Variables**: Converted categorical variables like 'ex 'Enbarked', and Pelass * into numerical values using one-hot encoding. Feature Scaling**: Applied standard scaling numerical features to ensure they are on a similar scale.
198
### 3. Feature Engineering
**iitle Extractionxx: Extracted titles from passenger names and created new feature.
201
=*Family Size**: Created a new feature representing the size a passenger's s family.
202
*+IsAione**: Created a binary feature indicating if a passenger was traveling alone.
#EFF 4. Modeling
205
Several machine learning models were trained and evaluated, including:
206
Logistic Regression
207
Decision Tree
Random Forest
Gradient Becsting
210
Support Vector Machine (SVM)
211
212
### 5. Evaluation Metrics
The models were evaluated using accuracy the evaluation dataset.
215
Mode1
Accuracy
216
Logistic Regression
0.80
Decision Tree
0.78
219
Random Forest
0.82
220
Gradient Boosting
0.83
221
Support Vector Machine (SVM) 0.81
#### 6. Best Model The best- perfoorrnggn model was *xGradient. Boostingex with an accuracy of **0.83** on the evaluation dataset.
224
225
### 7. Conclusien
The Gradient Boosting model was the most effective in predicting passenger survival on the
27

---

MetaAgent: Automatically Building Multi-Agent System based on Finite State Machine
Titanic dataset, achieving an accuracy of 83%. Further improvements could be made by tuning hyperparameters, adding more features, or using ensemble methods.
28


Reviews:
[{"review_id": "3JVL4BEqfz", "reviewer": "Reviewer_WYux", "summary": "This paper primarily discusses the automated construction of multi-agent systems. Its highlight is the introduction of the finite state machine (FSM) concept, incorporating null-transition states and state traceback into multi-agent systems. This allows the system to more flexibly address two issues: (1) when the current agent does not resolve a subtask as expected, and (2) when a downstream agent identifies problems with a previous agent, enabling traceback. Additionally, the paper outlines the relationships between three mainstream multi-agent approaches under this framework: Linear System, Decentralized Debate, and Coordinate with Orchestrator, explaining that they are all special cases of FSM. Based on the FSM idea, it Introduces the process of building and optimizing multi-agent systems. Its effectiveness was measured in scenarios such as Trivial Creative Writing, GPQA, and Coding. Furthermore, cost analysis and ablation experiments were conducted, demonstrating a comprehensive exploration.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer’s effort and the appreciation in the discussion in Session 3.3. We believe the finite state machine has the potential to be a unified structure of the Multi-Agent System. \n# Re Claims1:\nOur optimization method is inherently self-iterative, meaning it does not rely on external training data. In the ablation study, the description highlights the motivation behind this design choice. Initially, we observed that the first version underperformed on test cases, which led us to develop this self-iteration approach as an optimization strategy. We will refine the description in the revision to improve clarity.\n# Re Question 1:\nThis question explores why a Multi-Agent System, where agents are assigned different roles, can outperform a single LLM agent in certain tasks. Intuitively, assigning specific roles to agents activates their specialized knowledge related to those roles. Similarly, when all condition verification tasks are assigned to a single LLM, two main challenges arise. First, as the number of agents increases, a single condition verifier struggles to understand each agent’s situation. Second, dedicated condition verifiers, assigned to specific agents, can better capture and process the state-specific information, leading to more accurate and efficient verification.\n# Re Question 2:\n GPQA evaluates the accuracy of multiple-choice questions, while Writing assigns a score based on keywords. Each keyword has a corresponding list that includes its various forms, and as long as the generated text contains at least one of these variants, it is considered valid for scoring.\nWhen designing the objective evaluation criteria for software development tasks, we also select several bad cases for subjective evaluation. Through this process, we gradually update the objective evaluation criteria to make it more reasonable.\n# Re Question3:\nNo. It means the optimization method described in the method section. We will refine the description in the revision. \n# Re Experiment Metric also Question4:\nFor a batch of tasks (10+), the cost of the MetaAgent architecture is lower than that of other frameworks. The primary reason is that the FSM is more general: it only requires a one-time design for the task domain, whereas other frameworks necessitate case-by-case design(eg, AutoAgents,SPP). As a result, MetaAgent incurs lower costs when handling a batch of tasks.", "claims_and_evidence": "The paper mentions that external data is not needed, but in the ablation experiment of \"Reduce system redundancy through optimization\", it shows that \"... a few iterations are required to make the system more robust. . After testing the initial version of the multi-agent system on the pertinent test cases, the multi-agent system will be adapted in the aspect of agent and state design ...\", which seems to be Inconsistente.", "methods_and_evaluation_criteria": "For text-based tasks, the definition of metrics are provided, but it doesn't mention how to obtain them.\n    \nWhen evaluating software development task, the authors apply the \"objective\" checkpoints based methods. It would be better if the subjective evaluation metrics are reported together.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "In cost analysis, it is not clear here why the number of tokens used in this paper would be less. It stands to reason that additional verifiers would increase the use of tokens.", "relation_to_broader_scientific_literature": "A framework to unify existing multi-agent systems, which is well-discussed in Session 3.3.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Main highlight is to use FSM to unify current multi-agent systems, and provide a way to automatically construct multi-agent systems.", "other_comments_or_suggestions": "See questions.", "questions_for_authors": "1. What will be the difference between a multi-agent system with single verifier and MetaAgent if the verifier knows the role of each agent?\n2. How to get the metrics for text-based tasks\n3. What is the exact meaning of \"iterations\" in the ablation study? Is it case-driven?\n4. Why MetaAgent costs less with more verifier?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wAEkTr3wcg", "reviewer": "Reviewer_rvy8", "summary": "The paper introduces MetaAgent, a framework for automatically designing multi-agent systems using finite state machines (FSMs). The paper conceptualizes FSM within LLM agent design. The proposed method allows traceback ability to solve complex tasks. The paper also develops an optimization approach to merge the states for efficiency. Results on several text-based tasks demonstrate its performance compared with baselines.\n\n## update after rebuttal\n\nThanks for the responses from the authors. I would recommend that the authors add the discussions on external data and foundation model capabilities in the revised version.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for appreciating the finite state machine as a unified framework of a Multi-Agent System. \n# Re Reference: \nCAMEL is a simple two-agent chat system resembling a Decentralized Debate structure (Session 3.3).   \n\nAgentVerse employs two cooperation structures: Horizontal : a Linear System where decisions are summarized and Vertical:a Debate System where a solver and reviewer iteratively refine decisions until reaching consensus. These structures are also discussed in Session 3.3.  \n\nThe paper \"Embodied LLM Agents Learn to Cooperate in Organized Teams\" introduces a Communication-Action model, where agents communicate first and act later. However, this model has drawbacks: agents receive no immediate feedback during the Action Phase, delaying behavior refinement, and the leader agent can only issue instructions during the Communication Phase.   \n\nIn contrast, the FSM structure enables immediate feedback after each action via condition verifiers. Agents refine actions on the spot, determine state transitions, and communicate continuously, ensuring seamless interaction throughout the process. \n\n# Re Question 1:\nThe proposed optimization method relies on a self-iteration procedure, where the system refines itself over time. While incorporating external data could enhance this process by allowing the designer agent to learn from additional information, our observations indicate that most failure cases in the initial version of the FSM are caused by overly trivial agent and state assignments. Therefore, the most effective approach is to prompt the designer agent to self-refine the FSM by merging trivial agents and states, eliminating the need for external data.  \n\nOur FSM structure is designed to handle general open-ended, real-world tasks, where user inputs can vary widely. This variability makes it difficult to collect or synthesize high-quality test cases. As a result, relying on the self-iteration optimization method becomes not only effective but also the most practical approach for improving the FSM's performance in these complex, unpredictable scenarios.\n# Re Question2:\nAs presented in the optimization method (Session 3.4.3):” This iteration continues until no further states can be merged and the state set stabilizes.” The ‘iteration’ in the ablation study means the optimization method. We will refine our description in the revision.\n# Re Evaluation:\nWe have implemented a Virtual Home environment to the MetaAgent Framework and tested it with some housekeeping tasks. The following is a highlighted running log.  \n\n**Agent Design**:  LivingRoomAgent, RestRoomAgent, KitchenAgent, and BedroomAgent. \n  \n\n**State Design**(show state instructions):  \nTake books from the living room, Clean rubbish in the living room, …, Take books from the bedroom  \n\n**State Transition Example**:  \n{\n               \"from_state\": \"1\",  \n               \"to_state\": \"2\",  \n               \"condition\": \"If books are taken from the living room\"  \n },  \n\nWhen deployed, this multi-agent system can plan and submit actions to the environment in the specific format.   (example task: Clean the  bedroom)\n\nThe Bedroom Agent first **plan**:  \nPrioritized Cleaning Plan for the Bedroom:  \n1. **Organize Clothes:**  \n  - Fold and organize the clothespiles (15 clothespiles).  \n  - Hang the clothesshirts (3 clothesshirts) and clothespantss (5 clothespantss) in the closets.  \n2. **Clean Surfaces:**  \n  - Wipe down the desk, nightstands, coffeetable, and bookshelf.  \n  - Ensure the desk and nightstands are properly closed.  \n…  \n  \nAnd then it **act**:\n\n<ACTION>  \n{  \n   \"action\": \"Grab\",  \n   \"object\": \"clothespile\",  \n   \"object_id\": 150  \n}  \n</ACTION>  \n…    \n\nSeveral test cases show that the FSM-based multi-agent system can also understand the Virtual Home environment and interact with it properly.  \n# Re Weakness 1:\nWhen writing the paper, Open source models do not have enough intelligence level to achieve this complicated task of designing a Multi-Agent System. To invest the influence of the foundation model quality, we also apply GPT-3.5 as a weaker model in the ablation study, whose performance is lower than GPT-4o but is still comparable with other baselines.  \nRecently, we also tried some powerful open-sourced models as designers (like deepseek-v3). It also worked well on our test cases.\n# Re Weakness 2:\nFirstly, our framework is orthogonal to the foundation model capabilities. A better foundation model leads to better performance. However, when using GPT3.5 as the designer (GPT-4o as executor), our method also shows good performance on the Machine Learning Task. (In ablation study, Table 5)\nMoreover, while concerns about the reliability of using an LLM as a designer in a Multi-Agent System (or even within the LLM Agent domain) are valid, to the best of our knowledge, almost no existing work in the area can avoid the influence caused by a weaker foundation model.", "claims_and_evidence": "Supported: Tool usage and traceback are validated via ablation studies in Table 6. The overall performance is also evaluated on several benchmarks.\nNot supported: The generalization ability is not tested across domains beyond text-based scenarios.", "methods_and_evaluation_criteria": "1. The design of the method is clear and easy to follow.\n2. Benchmark limitations: only text-based benchmarks are deployed, while real-world multi-agent tasks also include planning and decision-making tasks, such as VirtualHome (http://virtual-home.org/documentation/master/get_started/get_started.html).", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. Though traceback helps with the performance, the paper seems not to report its potential influence such as delay in getting the results in practice.\n2. The paper assumes LLMs can reliably decompose complex tasks, but does not evaluate failure cases (e.g., ambiguous user queries).", "relation_to_broader_scientific_literature": "1. Multi-agent system: Extends prompt engineering to multi-agent coordination.\n2. Automatic learning: Combining AutoML with agent workflow design.", "essential_references_not_discussed": "Several papers on flexible multi-agent collaboration architectures are not discussed. What are their relations with the FSM?\n\n[1] Li, Guohao, et al. \"Camel: Communicative agents for\" mind\" exploration of large language model society.\" Advances in Neural Information Processing Systems 36 (2023): 51991-52008.\n\n[2] Guo, Xudong, et al. \"Embodied LLM Agents Learn to Cooperate in Organized Teams.\" Language Gamification-NeurIPS 2024 Workshop.\n\n[3] Chen, Weize, et al. \"Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents.\" arXiv preprint arXiv:2308.10848 2.4 (2023): 6.", "other_strengths_and_weaknesses": "Strengths:\n1. State merging is a novel way to make multi-agent much more efficient.\n2. FSM can potentially serve as a unified framework to develop more complex multi-agent system, which has been discussed in this paper.\n\nWeaknesses:\n1. Open-source models are not included and discussed.\n2. The merging of states depends on one single LLM. The capability of this LLM will limit the performance.", "other_comments_or_suggestions": "The tables in Appendix C should be revised.", "questions_for_authors": "1. Is there any discussion on \"the optimizing method does not need external data as well as numerous training steps\"? If there is external data, will the optimization work better?\n2. In ablation studies, the number of iterations is not clearly stated. The process of optimization is not included. Can you provide more information about this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jIsAITU5Cb", "reviewer": "Reviewer_YubL", "summary": "This paper proposes a novel framework, MetaAgent, for the automatic generation of multi-agent systems based on finite state machines. The framework comprises three key steps: (1) Agent Design: The designer model defines the roles and tools for each agent according to task discriptions; (2) Finite State Machine Design: The designer decomposes the task into multiple states and assigns them to appropriate agents; and (3) FSM Optimization: Agents are dynamically merged based on role distinguishability and tool assignment. For experiments, this paper selects a range of text-based tasks (e.g., trivial creative writing and GPQA) and real-world coding tasks (e.g., ML benchmarks and software development tasks). The experimental results demonstrate the effectiveness and efficiency of the MetaAgent.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful feedback. We are encouraged that the reviewer agrees the Finite State Machine(FSM) is an inspiring method to the Multi-Agent System field and appreciates our theoretical analysis. \n# Re Weakness:\nOur framework is independent of the foundation model’s capabilities. Our method remains effective even with weaker models. For instance, when using GPT-3.5 as the designer, our approach still achieves strong results on Machine Learning benchmarks.\nMoreover, while concerns about the reliability of using an LLM as a designer in a Multi-Agent System, to the best of our knowledge, almost no existing work provides a provable guarantee. And Our framework shows good performance on many kinds of tasks empirically.\n# Re Claim 1: \nSimply modifying the prompt is insufficient to enhance the performance of the LLM debate structure on text-based tasks. This is because our finite-state machine (FSM) framework incorporates structural features such as Null-Transition and State Traceback, which cannot be replicated through prompting alone.  \nTo empirically validate this, we conducted an experiment comparing a purely prompt-modified LLM debate with the traditional one. Specifically, we rewrote the LLM debate prompt, instructed the model to refine its responses, and evaluated performance on GPQA (Diamond). The modified prompt achieved a score of 0.56, only marginally better than the traditional one (0.54). This result demonstrates that mere prompting cannot match the performance of our FSM-based approach—it is the FSM structure itself that drives the improvement.  \nTheoretically, Null-Transition serves as a refinement mechanism, allowing agents to improve responses based on feedback. In contrast, Decentralized Debate follows a rigid sequence where agents present opinions without feedback opportunities, preventing output refinement. This limitation cannot be overcome through simple prompt modifications. However, introducing a condition verifier to refine responses and select speakers naturally transforms the structure into a finite-state machine.\n# Re Claim 2:\nOur MetaAgent Method represents the full realization of a Finite State Machine (FSM), as it incorporates a specialized condition verifier that enables Null-Transition and flexible state transitions, including state traceback.  \nSection 3.3 demonstrates that existing Multi-Agent Systems can be viewed as limited FSMs; the \"Coordinate with Orchestrator\" approach, which has a shared verifier, suffers from centralized decision-making that becomes computationally burdensome as states increase. In contrast, FSM's decentralized architecture—with independent condition verifiers at each state—significantly improves scalability and adaptability.\n# Re Method1：\nWe evaluate the MetaAgent Method on ML and Software Development tasks using two predefined tools: Python Code Interpreter and Search Engine.  Our experiments demonstrate that the designer LLM effectively selects and assigns tools to different agents.\nWe also test FSM with an expanded tool pool; the designer maintains wise selections and consistent performance.  \n# Re Method2:\nIt is interesting to include the ‘splitting’ action in our method. However, practically, we find the designer LLM always tends to split the given task into trivial sub-tasks, causing a high failure rate because the chain of execution is too long. To fix this drawback, we design the merge action to help the designer LLM rethink and revise the FSM. The ‘splitting’ method is not practical because the initial version of the finite state machine itself tends to be very trivial. \n# Re Experiment 1\nWe selected SPP since it is an auto-design multi-agent method. \nBased on your suggestion,  we have added new experiments for the software development task using AutoGen, Task Weaver, and Open Interpreter. \nThis table shows the result:\n|Method|2048Game|SnakeGame|BrickBreakerGame|ExcelApp|WeatherApp|Avg|\n|------------------|-----------|------------|--------------------|-----------|-------------|------|\n|AutoGen|0.75|1|0|0|0|0.35|\n|Open Interpreter|0|0.5|0|0.25|0.25|0.2|\n|Task Weaver|0| 0.5|0|1|0|0.3|\n|MetaAgent|0.75|1|0.5|1|1|0.85|\n\nFrom the above table, we can observe that our method also outperforms existing baselines by a large margin. Note that we do not include  Data Interpreter since it is a frame specifically designed for Data Science. \nFor other baselines in the text-based tasks, including direct prompt, CoT, CoT-SC, and Self-Refine. They are merely prompt-based, single LLM  methods instead of the multi-gent framework, and they do not support tool-using functionality. They perform poorly and are not a fair comparison with our method since we do not include them in the real-world coding task.  \n# Re Comment 2:\nIn appendix F\n# Re Comment 4:  \nFrom DataInterpreter Paper.\nNPS = \n$\\frac{1}{1 + s}$  (if s is smaller, the better )\nor \n$s$ (if s is bigger, the better ).\n\n# Re Question:\nYes. It is included in the ‘design’ stage.’", "claims_and_evidence": "1. In Section 3.3, in the paragraph titled “Decentralized Debate as FSM,” this paper claims that those methods do not support null-transitions, which is presented as a key difference. This claim is not well-supported. The definition of null-transitions is not difficult, and they can be implemented through manual definition or prompt rewriting. \n\n2. In Section 3.3, in the paragraph titled “Coordinate with Orchestrator as FSM,” the authors claim that those methods can be considered as FSMs. So, what is the main difference between them and MetaAgent?", "methods_and_evaluation_criteria": "1. Does predefining tools for a specific domain limit the ability to build automated workflows?\n\n2. In section 3.4.3, with only the \"merge\" action available, is there a possibility that splitting might be necessary, for example, when agent1 and agent2 have similar roles in state 1 but different roles in state 2?\n\n3. The construction of multi-agent workflows heavily relies on the designer’s abilities. In MetaAgent, the absence of hierarchical or tree-like decomposition places a high demand on the designer’s task decomposition skills.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The baseline comparisons are inconsistent. Tables 2, 3, and 4 use different baselines. Why is only SPP compared in Table 2? And why does the software development task, also a real-world coding task, have a different baseline setup compared to ML-Bench?", "relation_to_broader_scientific_literature": "This paper’s key contribution is using FSMs to build multi-agent workflows. While others focus on multi-agent framework construction, like automatic evolution, this work uniquely introduces FSMs and highlights their advantages over other approaches.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "**Strengths**:\n\n1. The authors’ introduction of Finite State Machines for modeling multi-agent workflows is insightful and inspiring for the field. \n\n2. The authors thoroughly discuss different types of multi-agent frameworks and analyze the distinctions between MetaAgent and them.\n\n**Weakness**:\n\n1. This method heavily relies on the designer's capabilities for task decomposition and planning, and it does not provide any guarantee regarding the lower bound of the method's performance.", "other_comments_or_suggestions": "1. The font size in Figure 1 is too small, resulting in poor readability. \n\n2. In the second-to-last paragraph of section 3.4.3, which appendix is being referred to? \n\n3. Sections 4.2 and 4.2.1 should be presented as parallel sections.\n\n4. The explanation of the metrics would be clearer with the inclusion of formulas, such as for NPS.\n\n5. In the second paragraph of section 4.4, “augment” should be corrected to “augments”.\n\n6. Tables 9, 10, and 11 are disorganized.", "questions_for_authors": "In section 4.3, does the calculation of cost include the refinement process during the construction (as mentioned in section 3.4.3)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ejjSDmumZO", "reviewer": "Reviewer_6eD6", "summary": "The paper proposes MetaAgent, an approach to automatically construct multi-agent systems using finite state machines (FSMs). Instead of hand-coding roles and workflows, MetaAgent uses a prompt-driven “Designer” to:\n\n1. Identify which agents (roles) are needed to complete a family of tasks.\n2. Build a finite state machine to represent the multi-agent collaboration flow, with possible traceback (going back to earlier states if errors are found) and null-transitions (staying in the current state for iterative refinement).\n3. Optimize the initially generated FSM by merging redundant states before deploying it to real tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer’s appreciation of our finite state machine design as well as the thorough discussion and experiments. \n\n## Re Reference Material:\nThank you for your valuable feedback. To the best of our knowledge, our work is the first to introduce finite state machines for the automatic design of multi-agent systems. We appreciate the suggestion regarding formal verification techniques and RL-based approaches. In our revisions, we will expand the discussion to include relevant formal verification methods for multi-agent systems and provide a more comprehensive analysis of RL-based self-play approaches in the context of task orchestration.\n\n## Re Wakeness 1:\nFirstly, our framework is orthogonal to the foundation model capabilities. A better foundation model leads to better performance. However, when using GPT3.5 as the designer, our method also shows good performance on the Machine Learning Task. \nMoreover, we admit this is a reasonable concern. But when using LLM as a designer of a Multi-Agent System and even the LLM Agent domain. Almost every existing work can not avoid the influence caused by the ability of the foundation model. Our framework shows good performance on many kinds of tasks empirically.\n\n## Re Weakness 2:\nThe condition verifier serves as a guider who utilizes the knowledge of the foundation designer model (by following state transition conditions) and helps the agent learn from feedback or evaluation results. So the thing is the same as Weakness 1: when the foundational model goes powerful, our FSM becomes more reliable in uncertain domains.  \n\n## Re Weakness 3\nAs for the ‘reuse’, the benchmark on ML bench and Software already show the reuse ability of the designed FSM. Because the FSM use in these benchmarks is designed for general use in the task domain (eg. Software Development). When deployed, the FSM will handle different cases in the domain. Thus, the experiment is showing the ‘reuse’ of the FSM.", "claims_and_evidence": "Claims:\n1. MetaAgent can generate multi-agent systems automatically (as opposed to requiring human-coded instructions).\n2. Finite state machines provide superior flexibility over purely linear or orchestrator-based pipelines.\n3. The generated multi-agent systems are robust and match or exceed the performance of both (a) domain-specific multi-agent frameworks designed by humans and (b) other auto-generated frameworks.\n\nEvidence:\n\n1. Each state is assigned one agent plus a separate condition-verifier mechanism that decides how to transition. This design is demonstrated in detail in examples (e.g., software development with states for requirements, coding, and testing).\n2. On “Trivial Creative Writing” and “GPQA(Diamond)” question-answering, MetaAgent’s multi-agent system achieves higher accuracy (and coverage of correct answers) than other prompt-engineering baselines and the SPP auto-design approach.\n3. MetaAgent produces a multi-agent system that nearly matches or surpasses specialized frameworks (e.g., DataInterpreter) and outperforms other auto-designed systems.\n4. Outperforms MetaGPT (a well-known human-designed pipeline) and other auto-designed frameworks on a set of small projects (e.g., developing Snake, 2048, etc.), measured by functional checkpoints.", "methods_and_evaluation_criteria": "Methods:\n\nAn LLM-based “Designer” first creates specialized agents—each with its own system prompt and tool access—relevant to the target domain. It then builds a finite state machine (FSM) by defining states, each tied to one agent, along with transition conditions and “listeners” to control information flow. The initially generated FSM may contain redundant states, so a second LLM-driven pass merges or removes overlapping roles. Finally, when the FSM is deployed, the system processes a user’s query state by state, allowing agents to backtrack or refine outputs until a final condition is met.\n\nEvaluation Criteria:\n\n1. Quantitative success rates on text-based tasks (e.g., how many questions are answered correctly in GPQA or how many writing tasks pass certain correctness checks).\n2. Metrics on machine learning tasks such as F1-score, accuracy, or RMSE, aggregated into a “Normalized Performance Score (NPS).”", "theoretical_claims": "1. The authors argue that many standard multi-agent frameworks (linear pipelines, decentralized debate, orchestrator-based) can be seen as special or “constrained” forms of an FSM (with fewer transitions or no null-transitions).\n2. By allowing cycles (traceback) and conditional transitions, an FSM can capture more sophisticated workflows and error-handling mechanisms—leading to better coverage of real-world complexities.\n\nThese claims are well-supported conceptually, showing how “traditional” multi-agent designs can be embedded in or derived from an FSM. That said, the paper does not delve deeply into formal language or automata theory beyond drawing analogies, so the theoretical underpinnings are relatively straightforward: it directly applies state machines to agent orchestration.", "experimental_designs_or_analyses": "The paper uses:\n\n1. Comparative performance with (a) multiple baseline prompt-engineering paradigms (e.g., Chain-of-Thought, Self-Refine, SPP) and (b) well-known multi-agent frameworks (MetaGPT, AutoAgents).\nMultiple tasks that vary in complexity (short question-answering, creative writing, data science, and software dev). This broad coverage supports the authors’ claims of generality.\n2. Cost analysis in tokens to highlight how the automatic design overhead compares to the cost of actually running tasks. They show that despite an up-front design cost (the LLM constructing the FSM), subsequent repeated usage can amortize that cost-effectively.\n\nStrengths in design:\n\n1. A variety of domain tasks confirm the FSM’s generality.\n2. Clear success metrics (accuracy, coverage, or pass/fail of checkpoints).\n\nPotential limitations:\n1. The tasks tested are mostly small-scale and may not stress extremely large or complicated real-world scenarios (e.g., building production-grade software or enormous data pipelines).\n2. While the paper reports improved performance on multiple tasks, seeing more ablation or failure modes in very complicated contexts would be interesting.", "relation_to_broader_scientific_literature": "1. The authors mention synergy with code interpreters, search engines, etc., continuing a line of research that integrates LLMs with external APIs or tool frameworks.\n2. The paper builds on the growing idea that large language models can “prompt themselves” (or use meta-prompts) to define roles, states, transitions, etc.", "essential_references_not_discussed": "Within the discussion, the paper references many important works on multi-agent frameworks and tool-based LLMs (e.g., SPP, AutoAgents, Symbolic Learning, etc.). However, there are a few broader areas they only lightly mention or do not cite explicitly:\n1. Formal methods for verifying the correctness of multi-agent systems: the authors mention automata (FSM) but do not elaborate on advanced formal verification techniques that might reduce error.\n2. Additional references to “complex task orchestration” frameworks or thorough treatment of “self-play” from prior RL-based approaches might have offered more historical grounding.", "other_strengths_and_weaknesses": "Strengths\n\n1. Unified framework: They argue how finite state machines unify or generalize popular multi-agent pipelines.\n2. Traceback and Null-Transitions: This is an elegant way to add “debugging” loops and multiple tool calls within each state.\n3. Experimental breadth: They tested text tasks, machine learning tasks, and code-generation tasks, demonstrating generality.\n\nWeaknesses or Limitations\n\n1. LLM Dependence: The entire pipeline’s success hinges on the correctness of the LLM used for designing the agents and transitions. The authors note performance degrades significantly when the LLM is replaced with a weaker model (e.g., GPT-3.5 vs. GPT-4).\n2. Limited Depth on Formal Guarantees: While the authors position FSM as a powerful structure, they do not thoroughly discuss formal correctness or how reliably the condition verifiers handle ambiguous states in real, uncertain domains.\n3. Cost–Benefit: Although the authors present cost analysis, real-world users might want more discussion or demonstrations of how well one can reuse a single “task-level” design across many similar sub-tasks without re-triggering the design step each time.\n4. Minor: The fonts in Figures 1 and 2 are small.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents MetaAgent, a framework for automatically constructing multi-agent systems via finite state machines (FSM). The use of FSMs provides a unifying perspective over existing agent architectures and enables traceable, modular, and optimizable workflows. While the method’s reliance on a strong designer model is a valid concern, the authors address this through empirical evaluations and ablation studies across diverse tasks. The paper is well-executed and offers a novel and practical contribution, though some claims would benefit from broader real-world validations.
Decision: Accept (poster)