MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems
Authors: Rui Ye
Shuo Tang
Rui Ge
Yaxin Du
Zhenfei Yin
Siheng Chen
Jing Shao
Venue: ICML 2025 poster (2025)
Keywords: large language models
multi-agent systems
Primary area: deep_learning->large_language_models
TL;DR: We propose MAS-GPT that simplifies the process of building query-specific multi-agent systems, making it as simple as querying ChatGPT.

Abstract: LLM-based multi-agent systems (MAS) have shown significant potential in tackling diverse tasks.
However, to design effective MAS, existing approaches heavily rely on manual configurations or multiple calls of advanced LLMs, resulting in inadaptability and high inference costs.
In this paper, we simplify the process of building an MAS by reframing it as a generative language task, where the input is a user query and the output is a corresponding MAS.
To address this novel task, we unify the representation of MAS as executable code and propose a consistency-oriented data construction pipeline to create a high-quality dataset comprising coherent and consistent query-MAS pairs.
Using this dataset, we train MAS-GPT, an open-source medium-sized LLM that is capable of generating query-adaptive MAS within a single LLM inference. The generated MAS can be seamlessly applied to process user queries and deliver high-quality responses. Extensive experiments on 9 benchmarks and 5 LLMs show that the proposed MAS-GPT consistently outperforms 10+ baseline MAS methods on diverse settings, indicating MAS-GPT's high effectiveness, efficiency and strong generalization ability.
The codes are released at \url{https://github.com/rui-ye/MAS-GPT}.

MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems
Rui Ye 12*
Shuo Tang" Rui Ge
Yaxin Du
Zhenfei Yin
Siheng Chen
Jing Shao
Abstract
Query
Manually Design MAS
MAS-GPT: Generating MAS
LLM-based multi-agent systems (MAS) have shown significant potential in tackling diverse tasks. However, to design effective MAS, ex- isting approaches heavily rely on manual config- urations or multiple calls of advanced LLMs, re- sulting in inadaptability and high inference costs. In this paper, we simplify the process of build- ing an MAS by reframing it as a generative lan- guage task, where the input is user query and the output is corresponding MAS. To address this novel task, we unify the representation of MAS as executable code and propose a consistency- oriented data construction pipeline to create a high-quality dataset comprising coherent and con- sistent query-MAS pairs. Using this dataset, we train MAS-GPT, an open-source medium-sized LLM that is capable of generating query-adaptive MAS within single LLM inference. The gener- ated MAS can be seamlessly applied to process user queries and deliver high-quality responses. Extensive experiments on 9 benchmarks and 5 LLMs show that the proposed MAS-GPT consis- tently outperforms 10+ baseline MAS methods diverse settings, indicating MAS-GPT's high effectiveness, efficiency and strong generaliza- tion ability. The codes are released at https: / //github. com/ rwtmmruiiee/MM-GG.
IB 32
Query
Answer
Inadaptability!! High Effort!
Adaptability!
One Inference!
Figure 1: Introduction of our proposed new paradigm for building MAS. During inference, MAS-GPT adaptively gen- erates query-specifa, MAS with one LLM inference.
Such limitation has driven recent research towards building LLM-based multi-agent systems (MAS) (Ye et al., 2025a; Qian et al., 2024a; Chen et al., 2024), where multiple LLMs (agents) with specialized capabilities work collabo- ratively to achieve more effective solutions. For example, MetaGPT (Hong et al., 2024) and ChatDev (Qian et al., 2024a) build multi-LLMM teams with expertise roles (e.g.. programmer, tester, and product manager) to solve com- plex coding tasks in predefined pipeline; while Agent- Verse (Chen et al., 2024) involves recruiters, executors, and evaluators for iterative task solving. These methods have shown superior performance over single LLM inference.
Despite achieving promising task performance, there are two fundamental issues that hinder the broad applications of MAS: inadaptability and high costs. (i) Inadaptability & high human effort: MAS in MetaGPT (Hong al., 2024), ChatDev (Qian et al., 2024a), and Vgent (Chen et al., 2024) are all manually crafted (e.g., for coding tasks). That is, the collaboration structure and agents' prompts are pre- determined and static, lacking in the generality to adapt towards any given tasks. (ii) High inference costs: Al- though there have been efforts design adaptive MAS, they essentially shift the human cost onto the computational cost, For example, both GPTSwarm (Zhuge et al., 2024), AFlow (Zhang et al., 2024), and DyLAN (Liu et al., 2024b) rely on LLMS to replace human involvement, iteratively adjusting the collaboration structure or agents' prompts in the MAS for each specific task. However, this process of- ten requires multiple LLM inferences and corresponding validation set in advance (Zhuge al., 2024).
1. Introduction
Large language models (LLMs) such as ChatGPT (Ouyang al., 2022: OpenAI, 2023) have achieved significant suc- cess wide range of tasks. However, single L.LM often struggles to handle the diverse and complex range of tasks (e.g., varying difficulties and domains) encountered in practice (Hong et al., 2024; Chen et al., 2024).
Primary contributor Work done during Internship Shannhaai Jiao Tong University "Shanghai AI Laboratory University of Ox- ford 'The University of Sydney. Correspondence to: Siheng Chen <sihenge @ jutt..eduucnn..... Jing Shao <shaojing @ pjlab.org.cn>>
Focusing these key issues, this paper explores how to adaptively build a query-specific MAS at a minimal cost. Our core idea is to reframe the process of building an exe- cutable MAS for each query as a generative language task, making building MAS as simple and efficient as querying
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MAS-GPT: Training LLMs to Build LLM--based Multi-Agent Systems
ChatGPT (Ouyang al., 2022). Given the generated MAS, the query can then be seamlessly processed to produce the final response, significantly simplifying the whole pipeline.
et al., 2025), bringing 13.3% and 10.0% gain on AIME- 2024 (a challenging mathematical benchmark), respectively. We also verify that our MAS-GPT can generalize to unfa- miliar queries and generate novel MAS via case studies.
Under this context, we introduce MAS-GPT, an LLM specifically trained to adaptively generate executable MAS based on any given user query in one single inference. While the concept is straightforward, the challenge lies in the limited knowledge of LLMs on the task of MAS gen- eration and the lack of corresponding training data. These limitations raise two key technical challenges: how to repre- sent the MAS and how to construct the dataset. (I) To ensure the generated MAS is readily executable, we unify the repre- sentation of MAS by describing it as a Python code snippet (i.e., a forward function), with each agent's prompt as a variable, LLM calls as functions, and agent relationships as string concatenation. (2) Building on this foundation, we
Our contributions are as follows:
1. We reframe building MAS for each query as a generative language task. We unify the representation of MAS as executable code and propose a consistency-orienta query-MAS data construction pipeline for LLM training.
2. We train MAS-GPT, an LLM that generates query- specific executable MAS within one single inference and can generalize across domains.
3. Experiments on 9 benchmarks and 5 LLMs show that MAS-GPT consistently outperforms 10+ baselines at a moderate cost during inference, indicating its effective- ness, efficiency and generalization ability.
propose a consistency-oriented data construction pipeline to facilitate the model in learning generalizable patterns and logical correlations, which includes the construction, evalua- tion, selection, and refinement of query-MAS pairs. During selection
2. Related Work
selection, we design an nnter-consisteney-rrieee approach to ensure that similar queries are paired with sim- ilar nigh-performing MAS, facilitating the model learn generalizable patterns. During refinement, we propose rete-coosissenccorrieeeee refinement method to strengthen the relatedness between query and MAS, enabling the model to learn the reasonable correlation. Finally, the resulting pairs are used to train open-source LLMs via supervised fine-tuning, where the instruction is the user query and the response is the MAS represented by code. This will equip the model with the ability to generate query-specific MAS, and also, generalize to unseen queries.
LLM-based Multi-Agent Systems. Since a single LLM may struggle to handle the diverse and complex range of tasks practice (Li et al., 2023; Qian et al., 2024b), such limitation has driven recent research towards building LLM- based multi-agent systems (MAS) (Wang et al., 2024c; Wu et al., 2023). MetaGPT (Hong al., 2024) and Chat- Dev (Qian et al., 2024a) introduce manually designed multi- agent teams for solving coding tasks; while MedAgents is designed for medical tasks (Tang et al., 2024). Agent- Verse (Chen et al., 2024) proposes an iterative collaboration structure where agents are recruited to discuss, execute, and evaluate. Multi-Agent Debate (Du et al., 2024; Liang et al., 2024) designs multiple expertise LLM-agents to debate and reason over multiple rounds to get final answers. The MAS in these methods are all fixed regardless of the given query, lacking in the generality to adapt accordingly.
With the introduction of MAS-GPT, inference for a query becomes significantly simplified. Instead of relying on man- ual crafting (Hong et al., 2024; Qian et al., 2024a; Chen et al., 2024) or multiple LLM inference costs (Liu et al., 2024b; Zhuge et al., 2024) to obtain an MAS for each query, the user simply inputs query into MAS-GPT to get a cor- responding executable MAS. Such MAS can be directly applied to process the query, where multiple MAS-GPT- generated agents collaborate with an -.S--eeT-eeeerated structure to deliver the final solution. With advantages of adaptability, low cost, and generalization, this approach could facilitate the application of MAS at scale.
DyLAN (Liu et al., 2024b) leverages LLMs to evaluate agents' values and dynamically select the best agents, GPTSwarm (Zhuge al., 2024) manually initializes an agent team, adjusts the collaboration structure and agents' prompts by prompting LLMs. Given queries with ground- truth answers from one task and several available MAS as context, ADAS (Hu et al., 2024) and AFlow (Zhang et al., 2024) leverages the strong capabilities of LLMs such as (Anthropic. 2024) and GPT-4 (OpenAI, 2023) to iteratively generate task-oriented MAS for the spe- cific task. Most of these methods require a corresponding validation set in advance (Zhuge et al., 2024; Zhang et al., 2024; 2025) and multiple times of LLM calls (c.g., over 10 calls of API with lengthy context) (Hu et al., 2024; Liu et al., 2024b) to obtain an MAS, which is expeusiv and even unachievable in practice.
We conduct extensive experiments to compare MAS-GPT with 10+ baseline methods on 9 benchmarks (various do- mains) using 5 state-of-the-art (open-source and proprietary) LLMs. Our results show that MAS-GPT consistently out- performs baseline methods on average, indicating its high generality and effectiveness. Meanwhile, MAS-GPT has the potential to further push the boundary of strong reasoning capability of ol (OpenAI, 2024b) and DeepScck- R1 (Guo
Claude-3.5-some
2

---

MAS-GPT: Training LLMs to Build LLM--based Multi-Agent Systems
Instead of manually designing a fixed MAS (Qian et al., 2024a; Chen et al., 2024; Du et al., 2024; Ye et al., 2025b) or requiring multiple LLM inference costs to obtain MAS (Liu et al., 2024b; Zhuge et al., 2024) for each query, our MAS-GPT significantly simplifies the process of build- ing an MAS, which can flexibly generate query-specific MAS within one LLM inference. Meanwhile, unlike exist- ing optimization-basee methods (Hu et al., 2024; Zhang et al., 2024; Zhuge al., 2024) that optimize on the valida- tion set drawn from the domain same as the testing domain, our optimization (i.e., training) can be performed on diverse domains, enabling our MAS-GPT to generalize across do- mains. Specifically, we design a construontttt pipeline to generate a series of query-MAS pairs, which are used for training MAS-GPT based on open-source LLMs.
def forward(querr): aath_ 4 YYoou are nath expert. Solve this question: (query)'
Math Agent
nath output call Lwt(maat__ageeeeeee
feedback Lagent F'Etyen (query) and (vath_output), A
provide feedback? feedhack_outee m call_]]n(feebaakk _agert) Roedback
refinne_ggent -'Given [query), (nath_output) and (feedback output),, provide the finel answer!
Refine
Agent
return call 1in(reiine agentt
Our unified code representation of an executable MAS (i.e., forward function). Each color denotes an agent. Agents defined by variables, LLM calls denoted by function calls, and interactions represented by string concatenations.
Figure 2:
computational costs to determine the appropriate MAS, our approach streamlines the entire process of building MAS by reducing it to single LLM inference.
LLM Post- Trst-irainnng Modern state-of-the-an art LLMs are usually post-trained via two main stages: supervised fine- tuning (SFT) and preference learning (Ouyang et al., 2022; Dubey et al., 2024; Yang et al., 2024; Liu et al., 2024a), where SFT is the basic technique to teach LLM a defined tasks (Zhou et al., 2023; Longpre et al., 2023). Focusing on SFT, a series of researches are conducted on the construction of datasets for training chatbot-type LLMs. For example, LIMA (Zhou et al., 2023) manually annotates high-quality language data for SFT, emphasizing the importance of qual- ity of SFT datasets. WizardLM (Xu et al., 2024), TULU 3 (Lambert et al., 2024), and Persona Hub (Ge et al., 2024) synthesize SFT data by prompting GPT models, indicating the potential of synthetic data for LLM training.
The core of our system is MAT- an LLM that is trained to generate MAS tailored specifically to the input query. Instead of relying on pre-built agent configurations, MAS- GPT dynamically creates an MAS for each query, ensur- ing that the system adapts to wide range of tasks. This approach not only minimizes the time and computational resources traditionally required to build the right MAS but also enhances the system's 's flexibility by generating task- specific solutions in real-time. Finally, the MAS generated by MAS-GP7 can be seamlessly integrated to process the query and deliver the final answer (bottom right in Figure 3).
For MAS-GPT, the training process leverages SFT, with primary focus on data construction. While previous ap- proaches focus on training LLMs to directly answer user queries, the challenge of training LLMs to generate MAS from user queries introduces a novel difficulty. Unlike real- world dialogue data, LLMs have limited (if any) knowledge of MAS generation. Using our proposed data construction pipeline, create the first Maeer--MAS-paiigg dataset, fa- cilitating the training of LLMs for MAS generation.
3.2. MAS-GPT: Dataset Construction and Training
To achieve the above goal, we reframe building MAS as a generative language task, where the input is a user query and the output is executable MAS capable of processing that query. This shift to generative paradigm introduces a new challenge since there is few (if any) knowledge within L.MSs on MAS generation. To make this approach viable, the key focus lies in constructing an appropriate dataset to teach the LLMs such brand-new new task. To achieve this, we propose a consistency-oriented data construction process, which involves four key steps: (1) construction of query and MAS pools, (2) inference and evaluation of query-MAS caansistennc-oriientee pair selection, (4) intra-
3. Methodology
This section first outlines the overall system integrated with MAS-GPT when processing user queries during inference. Next, we delve into the specifies of training MAS-GPT, with particular focus on the dataset construction process.
pairs, (3) consistency-oriene pair refinement.
Data Construction of Query and MAS Pools (Rep- resenting MAS as Executable Code). To construct the dataset for supervised fine-tuning (SFT), we adopt the fol- lowing data format: (system prompt, instruction, response). Here, the system prompt briefly describes the MAS genera- tion task, the instruction corresponds to the user query, and the response includes the MAS, which can be extracted by string matching. Therefore, training the LLM requires the collection of series of query-MAS pairs. Firstly, to enable
3.1. Overall System Integrated with MAS-GPT
We follow a standard workflow: given user query, multi- agent system (MAS) is constructed, with multiple agents working collaboratively to generate the final answer. Unlike previous approaches that either manually design the MAS, rely on fixed and query-agnostic MAS, or incur significant
3

---

MAS-GPT: Training LLMs to Build LLM-bbsed Multi-Agent Systems
Pool Construction
Pair Evaluation
Pair Selection
Pair Refinement
Dataser Construction
Query Pool
Query
MAS
Å
Score
&
A
A
=
0
A
&
O
& & &
A A
O
+
A
& T
A
A
O
A
A
O
+
A
I
Group-wise Query-MAS Pair Mapping for Intes-Conssseency
Enhance Query-MAS Alignment for Ciraa-Consistency
MAS Pool
O
+
A
0
Training
System: "Given the query generate an MAS"
Supervised
Inference
Query MAS-GPT
MAS
Query
MAS
Response
Tuning
Instruction: The Query O
A
Response: MAS with Reasoning A
MPS-GPT
Figure 3: Illustrations of dataset construction, training, and inference of our MAS-GPT. After training (which is one-time), the inference is significantly simplified: MAS-GPT generates a query-specific MAS, which can directly process the query.
MAS-GPT to handle diverse queries, we build a query pool from open-source queries across various domains, such as general QA, mathematics, and coding. Each query is care- fully selected to be verifiable, ensuring the presence of ground-truth answer or test cases (e.g., for coding tasks).
our MAS representation. Importantly, these 40+ MAS do not directly correspond to the exact number of MAS in the training dataset; rather, they serve as foundations that evolve during the query-MAS pair refinement process. See more details about implementation and visualization in Section
While the collection of queries is relatively straightforward, constructing the MAS pool presents fundamental chal- lenge: how to represent executable MAS. To address this, we propose unifying the representation of MAS by formaliz- ing it as executable Python code snippets. This unified rep- resentation motivated by the observation that existing LLM-based MAS methods are ultimately implemented as code, encompassing the definition of agents' prompts, LLM calls, and inter-agent interactions (Qian et al., 2024a; Hu et al., 2024; Zhang et al., 2024). Specifically, we define an MAS as forward function that takes a user query as input and returns the final answer generated by the MAS. Within the forward function, agent prompts are defined as variables, agent inferences are implemented as function calls, and in- teractions between agents are represented through string concatenation; see example in Figure
Data Evaluation of Query-MAS Pairs. After construct- ing the query and MAS pools, it is crucial to evaluate the query-MAS compatibility since not all MAS designs are equally suitable for every query. To achieve this, we pair each query and MAS in the pool by inferring the query to the MAS and evaluating the generated final answer.
Specifically, given the query pool Q {(2,, YJJ and the base MAS pool M {MAs Jj=1" where Q is the query. Y, is the information for verification, N and M denotes the pool size, we obtain N x M pairs. Then, a query-dependent evaluation function Seval.)) will be ap- plied to evaluate the effectiveness the query-MAS pair: score,,; feval [M A.S ghase (Q.), Y)), where M ASShase (Q.) denotes the answer generated by M MS given the query Qi and denotes correct and wrong respectively. Overall, we get M MAS scores for each query Qi which are denoted by s; = |core,,1, score,,M laying the foundation for subsequent steps for selecting appropriate query-MAS pairs and further refinement.
Following this framework, we first re-implement several existing MAS methods (e.g., Multi-Agent Debate (Du et al., 2024), Self-Consistency (Wang et al., 2024b), Self- Refine (Madaan et al., 2024)) to align with our unified code representation. To further expand the diversity of MAS can- didates, also manually design sulting in base MAS pool comprising over 40 unique MAS designs (Figure 6). These 40 MAS cover basic elements such as chain-of-thooe thougtt prompts, role-playing prompts, LLM-calling functions, and functions to get code execution results. As the base LLM does not know our representation of MAS (verified by poor performance in Figure 5(b)), in- cluding these basic elements critical to teach the LLM
Data Conte--onsistency-ao the query-MAS pair results obtained from the evaluation step, the next critical task is to select and construct high- quality query-MAS pairs for training. The first selection criterion is intuitive: effectiveness. Specifically, we retain only the query-MAS pairs where the MAS produces cor- rect answer (evaluation score is 1), as MAS designs that generate correct answers are more likely to be suitable for their respective queries compared to those that fail.
Pair Selection. With
MAS systems,

---

MAS-GPT: Training LLMs to Build LLM--based Multi-Agent Systems
While using all the remaining effective query-MAS pairs for training is straightforward, this approach introduces significant problem of low inter-consisteneyy the same or similar queries may correspond to multiple different MAS designs. This lack of consistency makes it difficult for the model to learn a clear optimization objective, hindering its ability to understand and perform the task effectively.
turn facilitates model training and improves generalization.
Next, we infer and evaluate the refined MAS on the corre- sponding query, as advanced LLMs could generate inappro- priate or non- non-executable MAS. Specifically, for each base pair (Qi, M A.5 the refined M ASqccinee is tested and ac- cepted only if it achieves nor-worse score. Formally:
M ASqelinee if sefine 2 buse
To address this issue, we propose an inter-consistency- oriented pair selection method that optimizes both effective- ness and inter-consistency. The core idea is to group similar queries and assign them a single, high-performing MAS to maintain consistency across the dataset. Specifically, we cluster queries based on their metadata or embeddings. For a group of S queries = {Q:} we calculate cumu- lative score for each MAS by summing its effectiveness scores across all queries in the group:
M AS;
M M otherwise
where grefine = Seval ( AA (Q.),, and glase feva( AA (Q.), Y,) are evaluation scores by compar- ing the MAS-generated and ground-truth answer Y,
Through this process, each query Qi is ultimately associated with a tuple (Q,, R.. MA,)). where R, denotes the reason- ing statement, and M AS; is the final MAS. This refined dataset ensures both inter- and intra-consistency, providing qughituality training data for subsequent model fine-tuning.
Ei= St. The MAS with the highest cumulative score is then selected as the representative MAS for all queries in the group: M AS = arg maxM ASEMS s. Through this, we pair each specific query with a specific base MAS: (Q,, MA
Training- Supervised Fine-Tuning of MAS-GPT Our dataset follows the format (system prompt, instruction, re- sponse). The system prompt briefly describes the task of generating a query-specific MAS and the instruction corre- sponds to the user query Q, The response is constructed as the concatenation of the reasoning process and the final MAS, which is represented as executable code in text form.
By aligning similar queries with the same MAS, this method improves the inter-consistency of the query-MAS pairs, facilitating the model in learning rec- ognize generalizable patterns and generalize across similar queries. For example, queries requiring divergent thinking may be consistently paired with MAS structures where mul- tiple agents independently generate ideas and then discuss.
Building upon this dataset, we perform supervised fine- tuning of MAS-GPT on the open-source medium-sized LLM,, 1own:-5-Cooerr32--Innrre (Yang al., 2024), leveraging its capabilitiis of code generation and instruction- following. During inference, when a user query is received, MAS-GPT generates an executable MAS tailored to that specific query Q;: M = MAS- (Q.):. The gen- erated MAS is directly usable for processing the query Qi and delivering the final answer: A, M MAS, (A,,((())) icantly simplifying the task handling process.
Data Intra- mntraa-Coosisseeyy--rreeeed Pair Refinement, While the inter-consistency-onnn mise-consistency--rrieeeed pair selection process effectively ensures consistency across query-MAS pairs, there remains critical issue within individual pairs: intra- consistency. Specifically, the alignment between a query and its associated MAS may still be suboptimal, making it challenging for the model to learn meaningful associations. For instance, a query about physics may be paired with an MAS involving experts from multiple domains (e.g., physics and biology), where the presence of non-relevant agents like biology experts can confuse the model.
(Qt), signif-
3.3. Discussions
Advantages. Overall, our system integrated with MAS- GPT offers the following key advantages: simplicity, cost- efficiency, and adaptability (generality). Instead of man- ually designing an MAS for each specific query, relying on fixed MAS for all queries, or requiring multiple LLM inference costs to obtain an MAS for a query, our MAS- GPT significantly simplifies the process of building an MAS by reducing into one single LLM inference. Given user query, MAS-GPT will efficiently return a query-specific MAS, which is executable and can be seamlessly applied to process the query to deliver the final answer. Although training incurs some cost, it is a one-time expense, whereas inference is potentially endless in practical applications. We believe that MAS-GPP GPT has the potential to further advance the real-world application of MAS due to its simplicity,
To address this, we propose an ceara-connistency-orrnntee pair refinement method. This approach aims to improve the query-MAS alignment through two key strategies: (1) adjusting MAS make query-dependen ducing reasoning process to strengthen the connection between the query and MAS. We employ an LLM-based data synthesis method, where an advanced I.I.M adjusts agents' definitions within the MAS based on the query and the previously selected MAS. The LLM is also instructed to generate a reasoning statement that explains the relationship between the query and the refned MAS, improving the in- terpretability of the query-MAS pair; please refer to prompt in Table 11. This process enables the model to better under- stand the context and logic behind each decision, which in
and (2) intro-
5

---

MAS-GPT: Training LLMs to Build LLM--based Multi-Agent Systems
Table I: Comparing MAS-GPT with 10 baselines across 8 benchmarks using -imma---700-Instrucc MAS-GPT performs the
best on average, verifying its generality in handling diverse queries. Benchmarks with are out-of-domain for MAS-GPT.
METHOD
MATH GSM8K
GSM-H H
H-EVAL."
H H-EVAL+
MMLU
GPQA
SCIBENCH
AVG.
SINGLE (DUBEY ET AL., 2024)
50.55
92.38
45.80
79.01
75.78
77.37
36.68
21.05
59.83
CHAIN-OF- THOUUHT (WEI ET AL.. 2022)
53.20
92.79
46.20
77.16
77.02
75.56
35.28
17.68
59.36
CONSISSSSSYYYY (WANG
AL., 2024B)
61.59
94.99
47.20
77.78
75.78
78.18
37.15
20.00
61.58
LLM-DEBATE (DU ET AL., 2024)
61.37
91.58
44,60
74,69
74.53
77.78
34.35
19.79
59.84
SELF-REFINE (MADAAN ET AL., 2024)
58.50
90.78
37.80
67.90
62.73
74.75
38.32
20.00
56.35
QUALITY- DUVLRTY-DIVERSITY DIVERSITY (LU AL.,, 2024)
60.49
92.99
45,60
70.99
70.19
75.76
33.64
20.63
58.79
SPP (WANG ET AL., 2024c)
51.66
92.79
44.80
76.54
73.29
77.37
35.05
20.84
59.04
AGENTVERSE (CHEN ET Al.., 2024)
55.63
93.39
41.40
77.78
73.91
76.57
40.19
16,00
59.36
GPTSWARM (ZHUGE ET AL., 2024)
55.41
93.19
43.20
69.14
73.91
75.
36.45
14.11
57.57
DYLAN (LIV AL., 2024B)
59.60
91.18
44.80
79.01
75.78
78.
35.98
19.79
60.54
MAS-GPT (OuRS)
68.65
93.39
62.40
80.25
78.88
78.38
37.62
24.21
65.47
Table 2: Statistics of MAS-GPTI training dataset. We show the number of samples Naata: the averaged instruction Linsi the averaged response Lrea. reasoning Lyen, and MAS length ASs and the number of unique MAS NMAS.
its (een2.5-Coderr322--miie (Yang et al., 2024), leveraging and coding capabilities. We train the LLM using 16 A100s with an effective batch size of 32 for 3 epochs at learning rate of le-5 (Zheng et al., 2024).
Naato
Lins
Lrea
Lean
LMAS
NMAS
Testing, To verify that our MAS-GPT can handle di- verse queries practice, we consider multiple benchmarks from diverse domains. These include MATH (Hendrycks et al., 2021b), GSM8K (Cobbe et al., 2021), GSM- Hard (Gao al., 2023), AIME-2024 for math domains; HumanEval (Chen et al., 2021) and HumanEval+ (Liu et al., 2023) for coding tasks; MMLU (Hendrycks et al., 2021a) for general QA tasks; GPQA (Rein et al., 2023) and SciBench (Wang et al., 2024a) for science topics. Among these, AIME-2024, HumanEval, HumanEval+. GPQA, and SciBench are from totally different distributions com- pared to training data, serving to verify the generalization capability of MAS-GPT. While there are samples related to mathematics and science during training, during testing, the problems in AIME, GPQA, and SciBench are much more challenging. Please refer to Table for details about datasets and Section C.2 for details about evaluation. For all baselines, the LLMs that drive the MAS to process user queries are kept the same, where we consider five state-of- the-art LLMS including -lama-3-70B-Mme Instruct (Dubey et al., 2024), Qwen2.5-722 -wwnn2.5-2B-Insttuuee (Yang et al., 2024), GPT-4o- mini-2024-07-18 (OpenAI, 2024a), ol- ol-preview-2024 2024--9-- (OpenAI, 2024b), and Deepseek-RI (Guo et al., 2025).
11442 ~ 75.0
1062.3
262.5 ~ 784.8
7580
cost-efficiency, and adaptability.
Cost. While MAS-GPT simplifies the application of MAS during inference time, it incurs cost for collecting data and training LLMs. However, training is one-time while in- ference could be endless (similarly OpenAl trains GPT-4 for one time and serves the world countless times). The MAS-GPT is trained on diverse data only once, and then can generalize to diverse domains for wide applications. Training-tinn time efficiency not the focus of this paper, which could be a potential future direction.
We believe that this paradigm is scalable due to exciting and promising property of MAS-GPP SATT it could stand on the shoulders of giants. Ideally, if we could include all existing high-performance MAS methods into the MAS pool, in real- world applications, we only need to deploy MAS-GPT to solve diverse user queries, rather than deploying multiple MAS and designing complicated rules to select them.
4. Experiments
Baselines. For fair comparisons, we consider 11 base- lines that are suitable for handling diverse tasks. We in- clude single agent and agent with cfain-offthhouht (Wei et al., 2022) as two basic baselines, Self- Belf-Consistency (Wang et al., 2024b) and Quality-Diversity (Lu et al., 2024) that select the best from multiple answers, LLM-Debate (Du et al., 2024) that involves multiple experts for debating, (Madaan et al., 2024) that iteratively refines last agent's 's answer, SPP (Wang et al., 2024c) that stimulates con- versations among multiple roles, AgentVerse (Chen et al., 2024) and DyLAN (Liu et al., 2024b) that dynamically ad- just multi-agent team during inference, GPTSwarm (Zhuge et al., 2024) that relies on graph collaboration structure.
4.1. Experimental Setups
Training, Our training queries are sampled from the train- ing splits available in MATH (Hendrycks et al., 2021b) GSM8K (Cobbe et al., 2021), MBPP (Austin et al., 2021), MMLU (Hendrycks et al., 2021a), and SciQ (Welbl al., 2017), covering domains of math, coding, and general QA. 1aama-3-70B-Instmee is used during dataset construction. The number of training samples (i.c., query-MAS pairs) is approximately 11k. We report the statistics of our dataset in Table 2, where the number of unique MAS is measured by string comparisons. Our MAS-GPT trained over
6

---

MAS-GPT: Training LLMs to Build LLM--baed Multi-Agent Systems
We also compare with MAS optimized by AFlow (Zhang et al., 2024) on math task.
Table 3: MAS-GPT consistently performs the best across
MAS- dri-iiiving LLMs, indicating its strong compatibility.
METROD
MATH
GSM-H H-EVAL+
MMLJ
GPQA
Ave.
4.2. Main Results
WWEN2.5-72.IINSTREE
SINGLE
85.86
64.91
85.37
44.39
72.63
Since our proposed MAS-GPT aims to facilitate the multi- agent systems in flexibly handling diverse queries, our sults focus on the keyword of generality. Here, we show the generality of MAS-GPT by comparing performance averaged on various benchmarks and performance using different LLMs to drive the MAS.
COT
86,90
62.27
84,15
83.20
47.86
72.88
SELF-CON.
61.46
83.40
50.00
73.88
LLM-DEBATE
85.24
63.49
68.
86.20
47.86
70.34
SELF-REFINE
83.58
59.03
78.66
85.40
43.32
70.00
Q-D
85.65
63.08
76.83
82,80
48.66
71.40
SPP
85.65
62.88
82.32
83.40
48.40
72.53
AGENTVERSE
84.82
59.43
81.10
83.20
44.65
70.64
GPTSWARM
83.16
63.89
83.54
84,60
44. 94.92
72.02
DYLAN
87.73
63.08
85.37
84.48
$1.07
74.33
MAS-GPT
87.53
66.33
85,98
83,80
48.
74.46
MAS-GPT's generality in handling diverse queries. We compare MAS-GPT with 10 baselines on 8 benchmarks using 1aaama---7B-Instteeee (Dubey et al., 2024) to drive the MAS, with results reported in Table 1. GPQA and SeiBench are two benchmarks that are out-of-domain for our MAS-GPT. From the table, we see that (1) our MAS- GPT significantly outperforms the baseline methods on av- erage, outperforming the second-best method by 3.89%. (2) Our MAS-GPT simultaneously achieves promising perfor- mance in both in-domain and out-of-domain (i.e., queries that are significantly different from those in the training data) benchmarks, indicating MAS-GPT's generality. (3) MAS-GPT is the only method that consistently achieves bet- ter performance than single agent, indicating its robustness in handling diverse queries.
GPT-4O-MINI-
SINGLE
78.18
58.03
86,23
78.56
38.03
COT
78.79
60.84
85.62
79.16
39.60
68.80
SELF-CON.
81.62
59.04
80.96
39.82
69.29
L.M-DEBATE
79.60
60.84
86.25
80.76
37.81
69.05
SELF-REFINE
74.55
54.62
76.88
79.16
33.33
63.71
Q-D
79.80
59.64
84,38
79.76
37.58
68.23
SPP
77.58
57.63
77.96
37.58
67.40
AGENTVERSE
75.15
55.62
79.38
78.36
36.24
64,95
GPTSSWRM
75.15
55.62
79.38
78.36
36.32
64.97
DYLAN
81.21
59.24
80. 60.62
79.96 96
40.94
68. 38.39
MAS-GPT
81.21
61.45
86.88
80.36
42.60
70.50
Figure 4(a), show that our proposed MAS-GPT significantly outperforms the baseline methods on this challenging task. Specifically, it can improve over the single LLM by a large margin: 13.3 13.34%. This result not only verifies the generality of our proposed MAS-GPT, but also indicates its promising potential in pushing the boundaries of LLM reasoning.
Generality in using diverse LLM backbones for MAS. Llama-3-708B Instructt was utilized to drive MAS during the dataset construction phase for training MAS-GPT, 32B- sized LLM. As shown in Table 1, this approach proves effec- tive when employing the same LLM to drive MAS during test time. To further validate the versatility of MAS-GPT, we assess its performance under different MAS-driving L.Mss, including Qwen2.5-72BBInstreee
Comparisons with task-specific methods, AFlow. To further demonstrate the generality and effectiveness of our MAS-GPP during inference time, we compare with AFlow (Zhang et al., 2024), a latest task-specific method for MAS optimization that has been specifically optimized MATH (Hendrycks al., 2021b) dataset. We evalu- ate on two AFlow's 's in-domain (MATH and GSM8K) and two AFlow's 's out-domain (MMLU and HumanEval+) bench- marks. Results in Figure 4(b) show surprisingly good per- formance of our proposed MAS-GPT. As a general method, our MAS-GPT even outperforms math-specific AFlow on the MATH dataset by 3.53%! Meanwhile, the MAS op- timized MATH by AFlow fails to generalize other domains, achieving worse performance than single LLM. In contrast, our MAS-GPT consistently performs the best across these benchmarks. It is also worth mentioning that our MAS-GPT only requires one-time inference of a 32B- sized LLM to build the MAS; while AFlow needs to call the APIs of powerful proprietary LLMs, such as Claude-3.5. Sonnet (Anthropic, 2024), 10 times per query and depends hold-out validation set.
and GPT-40-mini-2024 07-18, in Table 3. The results demonstrate that MAS-GPT consistently achieves superior performance, regardless of the LLM used to drive MAS, highlighting its strong compat- ibility and adaptability across various MAS-driving LLMs.
MAS-GPT's potential in further augmenting the rea- soning performance of strong reasoning LLMs such as ol. In recent developments, the Al community has intro- duced several oftte-of-the-art reasoning LLMs (OpenAI, 2024b; Qwen, 2024), which have demonstrated remark- able reasoning capabilities by scaling inference-time com- putations (Snell et al., 2024). In this context, we aim to explore whether our proposed MAS-GPT can take the rea- soning power of these already advanced models even further. To test this, we conduct experiments using OpenAl's ol- preview (OpenAI, 2024b) and Deepseek-R1 (Guo et al., 2025), evaluating them on the highly challenging AIME- 2024 mathematical benchmark'. The results, as shown in
Cost comparisons. Here, we compare the inference cost of various methods from the moment a user query is received to the generation of the final answer, as illustrated in Fig- ure 4(c). We quantify the inference cost in terms of the number of L.MM inference calls (Liu et al., 2024b), interpret-
7

---

MAS-GPT: Training LLMs to Build LLM--bsed Multi-Agent Systems
Trade-off Between Peformance and Cost
Accuracy on AIME-2024 (%)
MAS Impowered by Reasoning Models
10.0%
22.2
In-Domain Performancs (%)
.oompriioons with Task-Specific Method
AFlow's In-Domain
Out-Domain
%
Out Out-Domain moaain Performance (%)
Performance (%)
64
MAS-GPT (Ours)
*
SelfCoomiiseency KAS-OPT (Durs)
70-
800
DyLAN
60-
LLM Debate
13.3%
AgentVerse
GPTSwarm
c1-Frevicw Re
MATH
GSM Hard
HMLU
HumarEvals
564
16
32
64
Reasoning Models
Benchmarks
Inference Times
(a) MAS-GPT-Assisted Reasoning
(b) Comparison with Task Task-Specific Method
(c) Performance v.s. Inference Times
Figure 4: (a) Different methods empowered with strong reasoning LLM: ol -prevvi... We see that our MAS-GPT significantly enhance the reasoning performance over single LLM, indicating its potential in further augmenting LLM reasoning. (b) Comparisons with AFlow (optimized on MATH), MAS-GPT even outperforms AFlow on its in-domain benchmarks; while AFlow fails on out-of-domain benchmarks. (c) MAS-GPP achieves the best performance with low inference cost.
Table 4: Ablation studies on the designs of dataset construc- pair selection, (2) the adjustment of MAS in our sarra-consisteccy-oree -orieetee pair refinement: Refine-A, (3) the introduction of reasoning process in our intra-consistency-orieeii Refine-R. The table shows that these three designs all play critical roles in achieving high task performance.
consistency-orienter query-MAS pair refinement method. This method enhances the alignment between query and MAS by adjusting MAS to make it query-dependent and introducing a reasoning process to strengthen the logical connection. To examine their effects, we conduct two ex- periments with one without adjustment of MAS and one without reasoning process. From Table 4, by comparing @ and @, @ and @, we see that our designs in adjusting MAS and introducing reasoning process both contribute to performance improvement, indicating the effectiveness of our proposed autra-consisteccy-rreeee
tion: (1) our wwr--cooonsseccooiininiet
pair refinement:
SELECT REFINE-A REFINE-R MATH MMLU
GPQA
V
60.26
77.58
36.68
x
66.23
77.78
36.45
V
x
64.90
75.96
37.15
pair refinement.
V
/
V
68.65
78.38
37.62
Scaling effects of data size. To explore the scaling effects of data size for training MAS-GPT, we adjust the size from 11k using the same 32B-sized model and compare the extractability (i.e., the Python code can be extracted), ex- ecutability (i.e., the code is executable), task performance. Results in Figure 5(a) show that except for the extractability under O data sample (the base model knows that it needs to generate Python code, but do not know what codes are needed), the extractability and exccutability generally im- proves with the data scale. Results in Figure 5(b) show (I) the base model is unable generate an effective MAS in zero-shot setting, indicating the necessity for training MAS-GPT. Overall, observe promising scaling trend of training MAS-GPT: more data leads to better performance.
ing the inference of MAS-GPT as 0.5 times, given that its model size is approximately half that of the MAS-driving LLM (32B v.s. 70B). From the figure, we observe that, among the four methods compared, MAS-GPT achieves the best performance while requiring the fewest inference calls, demonstrating its efficiency and effectiveness.
4.3. Analysis of MAS-GPT
Effectiveness of inter-consistency-orieeee pair selection. During data construction, facilitate the model in recog- nizing generalizable patterns between queries and MAS, we propose an coler-oonisteec-orrenee selection method, which maps similar queries with consis- tent high-performing MAS. To examine its effectiveness, we replace this mapping with a random mapping approach, which randomly selects one out of those MAS with correct answers. From Table 4, by comparing ① and @, we that our proposed method brings significant performance gain, with an absolute improvement of 8.39% on MATH.
quS pair
Scaling effects of model size. Here, we compare the per- formance of MAS-GPT trained based on 7B, 14B, and 32B models. Results in Figure 5(c) show that the performance of MAS- improves steadily with the growing model size, Overall, these findings demonstrate the promising potential of MAS-GPT, suggesting that it can be further improved with more diverse, high-quality data and stronger models as the community continues to advance.
pair refine- ment. During data construction, to help the model learn the associations between query and MAS, we propose an intra-
Effectiveness of intra--coniistecc--oreeeet
Examining the generated MAS. To ensure that our MAS- GPT is not simply memorizing those MAS seen during the
8

---

MAS-GPT: Training LLMs to Build LLM--baed Multi-Agent Systems
Fallure Analysis with Data Size Scaling
Performance Data Scaling
Performance with Model Size Scaling
90
N=0
78
Failure Counts
4
Execution
Extraction
Performance (%)
N-1E3
326
146
60
N=:E4
Performance
66
Lower is Better
45
-4
30
1000
10000
Humantva-Flus
MMLU
HumanEval
Data Scale
Benchmarks
Benchmarks
Data Scale Failure
(b) Data Scale v.s. Performance
(c) Model Scale v.s. Performance
Figure 5: Explorations of scaling in training MAS-GPT. (a) More data leads to fewer execution failures. (b) More data contributes to better performance of MAS-GPT in facilitating MAS application. Without training (N=0), the model fails, highlighting that MAS generation a non-trivial task requiring specific training. (c) Larger model generally contributes to better performance. These findings demonstrate the promising potential of MAS-GPT, suggesting that it can be further improved with more diverse, high-quality data and stronger models as the community continues to advance.
Table 5: The proportions of generated MAS that are novel compared to those in the training set. We see that MAS-GPT not simply memorizing MAS seen during training, rather learns to generate appropriate query-dependent MAS.
tently demonstrate that MAS-GPT outperforms all baselines, strongly validating its effectiveness and generalizability. Ad- ditionally, we observe MAS-GPT's potential to further en- hance state-of-the-art reasoning capabilities, as well as its scalability for continued improvements. We believe MAS- GPT can accelerate the adoption of MAS, inspiring future research and real-world applications.
BENCHMARK
GSM-H
MATH SCIBENCH
MMLU
NOVEL (%)
92.5
69.0
71.0
42.9
Limitations and Future Works. While MAS-GPT shows promising results, several limitations and avenues for future work remain. First, constrained by limited human annota- tion and computational resources, the diversity of our initial MAS library and the final training set, potentially biased by mapping multiple queries to strong poron-perfommnng MAS instances, could be further improved. Second, beyond code execution, integrating additional tools like multi-modal data processing and web search tools could significantly enhance MAS-GPTT capabilities. Third, as a foundational step, our current work utilizes only supervised fine-tuning (SFT); future research could explore reinforcement learning (RL.) methods to enable MAS-GPT to autonomously explore and refine MAS generation. We believe that there is substantial room explore along this direction, continuously advanc- ing the flexibility of MAS in real-world applications.
training process, we compare the generated MAS given queries during testing with those in the training set. We report the proportion of novel MAS in Table 5. From the ta- ble, we see that our MAS-GPT indeed generates novel MAS during test time. The proportions are not consistent across benchmarks, which may stem from the varying diversity of MAS associated with different domains during training.
To offer an intuitive understanding, we present several ex- amples in Appendix showcasing the query, the MAS-GPT- generated reasoning process, and the MAS-GPT-generated MAS. These show that MAS-GPT can generate query- specific MAS (Section B. generalize to unseen queries (Section B.2), generate novel MAS (Section B.3).
5. Conclusions
Building MAS was time-consune cone-consuming and resource-intensive intensive. This paper aims to streamline this process into a single LLM inference, making MAS creation as effortless as querying ChatGPT. To this end, we introduce MAS-GPT, an LLM specifically trained to generate exccutable MAS from ar- bitrary user queries. Our approach follows data-driven spirit, leveraging a consistency-oriente data construction pipeline to enhance the coherence and consistency of data pairs. We conduct extensive experiments, comparing MAS- GPT against 10+ baseline methods across 9 benchmarks, using 5 different L.LMs as MAS drivers. The results consis-
Impact Statement
This paper presents an advancement in simplifying the ap- plication of L.L.- multi-agent systems (MAS). Our MAS-GPT significantly reduces the complexity of design- ing and deploying MAS for wide range of tasks. Our approach not only makes MAS development more acces- sible and efficient but also improves scalability, enabling its broader application in real-world scenarios. While the potential negative impacts of our approach are similar to those associated with large language models, such as eth-

---

MAS-GPT: Training LLMs to Build LLM--basd Multi-Agent Systems
ical concerns and misuse, these are inherent to the use of LLMs in general and do not require further elaboration in
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501 12948, 2025.
this context.
Acknowledgments
Hendrycks, D., Burns, C., Basart, S., Zou, A.,. Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a.
This research is supported by the National Natural Sci- ence Foundation of China under Grant 62450162 and Grant 62171276; and Shanghai Artificial Intelligence Laboratory.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. NeurlPS, 2021b.
References
Anthropic. Claude 3.5 sonnet. https://www
anthropic. www.co//////oeee /wwm//www///// /claude33---sonneee
2024. Accessed: 2025-01-22.
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C.,, Wang. Z., Yau, S. K. S., Lin, Z.,, et al. Metagpt: Meta programming for a multi-agent collabora- tive framework. In The Twelfth International Conference on Leurning Representations 2024.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al.
Program synthesis with large language models. arXiv preprint 2xXi:22108.07732, 2021.
Hu, S., Lu, C., and Clune, J. Automated design of agentic systems. In NeurlPS 2024 Workshop Open-World World Agents, 2024.
Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, H. P. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374, 2021.
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. V., Liu, A., Dziri, N., Lyu, S., et al. TV" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.1512 2024.
Chen, W., Su, Y. Zuo, J., Yang. C., Yuan, C., Chan, C.-M., Yu, H., Lu, Y., Hung, Y.H.,, Qian, C., et al. Agentverse: Facilitating multi-agent collaboration and exploring emer- gent behaviors. In The Twelfth International Conference on Learning Representations, 2024.
Li, G., Hammoud, H., Itani, H., Khizbullin, D., and Ghanem, B. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991-52000 2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, I.., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers solve math word problems. arXiv preprint 111.422110.14168, 2021.
Liang, T., He, Z., Jiao, W., Wang. X., Wang, Y., Wang. R., Yang, Y., Shi, S., and Tu, Z. Encouraging divergent think- ing in large language models through multi-agent dehate. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N, (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pp. 17889-17904, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. (0.18653/v1/2022 emnlp-main.992 main.992. URL https: http:////aaaathooooy... orgg 2224 emn
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A. et al. The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
in...n...00 992/
Ling, W., Yogatama, D., Dyer, C., and Blunsom, Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 158-167, 2017.
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: Program-aided lan- guage models. In International Conference on Machine Learning, pp. 10764-10799, PMLR, 2023.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., LU, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint 2412.2412..9433 2024a.
Ge, T., Chan, X., Wang, X., Yu, D., Mi, H., and Yu, D. Scal- ing synthetic data creation with 1,000,000,000 personas. arXiv preprint 24066 20094, 2024.
10

---

MAS-GPT: Training LLMs to Build LLM--based Multi-Agent Systems
Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL https::///// opwnrevieneeeeeeeeeeeeeee..eevee net.
Qwen. Qwq: Reflect deeply on the boundaries of the unknown. htttps://wwen1 prithub.i//bli.. qwq 32b ppevieww, 2024. Accessed: 2025-01-27.
Rein, D., Hou, B. L.., Stickland, A. C., Petty, J., Pang, Y., Dirani, J., Michael, J., and Bowman, R. Gpqa: A graduate-level google-prode q&a benchmark. arXiv preprint 2rrXiv:231.... 12022, 2023.
Liu, Z., Zhang, Y., Li, P., Liu, Y., and Yang. D. A dynamic llm-powered agent network for task-oriented agent col- laboration. In First Conference on Language Modeling, 2024b.
Snell, C., Lee, J., Xu, K., and Kumar, Scaling IIm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408. 240.244488.331 2024.
Longpre, S., Hou, L., Vu, T., Webson, A.. Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effec- tive instruction tuning. In International Conference on Machine Learning, pp. 22631 PMLR, 2023.
Tang, X., Zou, A., Zhang, Z., Li, Z., Zhao, Y., Zhang, X., Cohan, A.. and Gerstein, M. MedAgents: Large language models collaborators for zero-shot medi- cal reasoning. In Ku, L.-.., Martins, A., and Sriku- mar, V. (eds.), Findings of the Association for Computa- tional Linguistics: ACL 2024, pp. 599-621, Bangkok, Thailand, August 2024. Association for Computa- tional Linguistics. doi: 10. 10.186653/1120044fnddnn 33. URL pettps://ccaattheeee org/ 2024. E Findings-acl.33 33/.
Lu, C., Hu, S., and Clune, J. Intelligent go-explore: Stand- ing on the shoulders of giant foundation models. In Automated Reinforcement Learning: Exploring Meta- Learning, AutoMI, and LLMs, 2024.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye. S., Yang, Y., et al. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Systems, 36, 2024.
Wang, X., Hu, Z., Lu, P., Zhu, Y., Zhang, J., Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y., and Wang, W. Scibench: Evaluating college-level scientific problem- solving abilities of large language models. In Forty-first International Conference on Machine Learning, 2024a.
OpenAl. Gpt-4 technical report.
arXiv preprint
arXiv:2303.08874, 2023.
OpenAl. Gpt-4o mini: advancing cost-efficient intelli- gence. httpa///ooooo/e///ttoo-mm-man
Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In The Eleventh International Conference on Leurning Representations, 2024b.
cost-efficien--neelligneeel 2024a. Accessed: 2025-01-
23.
OpenAL. Introducing openai ol-preview.
https: htttps:/oppeai... wwwm/inndex.. introducing ooppoao-opn-----ee - ol-preview. 2024b.
Wang. Z., Mao. S., Wu, W., Ge, T., Wei, F., and Ji, H. Un- leashing the emergent cognitive synergy in large language models: A task-solving agent through multi- multi-persona self- collaboration. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 257-279, 2024c.
Accessed: 2025-01-22.
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. NIPS, 35:27730-27744, 2022.
Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang. C. Chen, W., Su, Y. Cong, X., et al. Chatdev: Commu- nicative agents for software development. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp- 15174-15186, 2024a.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Qian, C., Xie, Z., Wang, Y., Liu, W., Dang, Y., Du, Z., Chen, W., Yang, C., Liu, Z., and Sun, M. Scaling large- language-moul model-based multi aulti-agent collaboration. arXiv preprint wwxx::2466.07// 0515, 2024b.
Welbl, J.. Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User deer-genertted Text, pp. 94-106, 2017.

---

multi-agent MAS-GPT: Training LLMs to Build LLM--basdd Multi-Agent Systems
Wu, Q., Bansal, G., Zhang. J., Wu, Y., Zhang. S., Zhu, E., Li, B., Jiang, I.., Zhang, X., and Wang, C. Autogen: Enabling next-gen IIm applications via multi-agent conversation framework. arXiv preprint arXiv:2338.08115. 2023.
Xu, C., Sun, Q. Zheng, K., Geng, X., Zhao, Feng, J., Tao, C., Lin, Q., and Jiang, D. WizardIm: Empowering large pre-trained language models to follow complex in- structions. In The Twelfth International Conference on Learning Representations, 2024.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint 151 2024.
Ye. R., Huang, K., Wu, Q. Cai, Y., Jin, T., Pang. X., Liu, X., Su, J., Qian, C., Tang, B., et al. Maslab: A unified and comprchensive codebase for systems. arXiv preprint a50iv:2505. 16988, 2025a.
Ye, R., Liu, X.. Wu, Q., Pang, X., Yin, Z., Bai, L.,, and Chen, S. X-mas: Towards building multi-agent systems with heterogeneous llms. arXiv preprint arXiv:2505.16997, 2025b.
Zhang, G., Yue, Y.. Li, Z.. Yun, S., Wan, G., Wang, K., Cheng, D., Yu, X., and Chen, T. Cut the crap: An eco- nomical communication pipeline for LLM-based multi- agent systems. In The Thirteenth International Confer- ence on Learning Representationss 2025. URL https: //openreview net. /atc/forrume forum? 1d=kkuuP
Zhang, J., Xiang, J., Yu, Z., Teng, F., Chen, X., Chen, J., Zhuge, M., Cheng, X., Hong, S., Wang, J., et al. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410. 10762, 2024.
Zheng, Y., Zhang, R., Zhang, J., Ye, Y., and Luo, Z. Lla- maFactory: Unified efficient fine-tuning of 100+ language models. In Cao, Y., Feng, Y., and Xiong, D. (eds.), Proceedings of the 62nd Annual Meeting of the Asso- ciation for Computational Linguisties (Volume 3: Sys- tem Demonstrations), pp. 400-410, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10. 10.18663/w1122244.dddddmmms ///cl/nnhhlooy..rrrg//ooo www.:/oo.oo..//o//e acl demos. 38/.
URL https:
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less more for alignment. Advances in Neural Information Processing Systems, 36, 2023.
Zhuge, M., Wang, W., Kirsch, L., Faccio, F., Khizbullin, D., and Schmidhuber, J. Gptswarm: Language agents as op- timizable graphs. In Forty-first International Conference Machine Learning, 2024.
12

---

MAS-GPT: Training LLMs to Build LLM-bbsed Multi-Agent Systems
A. Details of The Initial MAS Pool
A.1. Implementations
We totally implement 40+ MAS in our initial MAS pool, which covers basic elements such as chain-of-thought prompts, pieeppaying prompts, LLM-calling functions, and functions to get code execution results. As the base LLM does not know our representation of MAS (verified by poor performance in Figure 5(b), N=0), including these basic elements is critical to teach the LLM our MAS representation.
Basic functions:
1. call.ttmp,, which takes the prompt as imput and outputs the response generated by the LLM.
2. coecute which takes a code snippet as input and outputs the exccution results (including printed information or errors).
3. test cott.oodegee gete-gget feedbbcccccee cases,, which tests the given code with the test cases and returns the feedback. taskInfo), which calls the LLM return the generated function signature for the given task.
4. get
5. ger_ttes_cases((i)) taskInfo. function which calls the LLM return the generated test cases for the given task and function signature.
6. extract code solution( solution), which returns the code by extracting (wrapped within iCode Solutiong and i/Code Solution;) from the given solution.
7. generate and extreettttcccc.. conde(Lm, prompt), which calls the LLM to return the generated response and the extracted code from the response (based on extract_code_solution
solution)). solussss
Representing MAS as python code a promising direction, as most functionalities our Al community can represented by python code. Along this new direction, we believe that with more data, tools, MAS, MAS-GPT could be continually improved.
A.2. Visualization of Our MAS pool
We visualize several MAS in our MAS pool in Figure
B. Case Study
B.1. MAS-GPT Can Generate Query-Specific MAS
From Table 6, we see that given a query. MAS-GPT first provides a reasoning process in analyzing the domain, difficulty, and required knowledge of the query. Secondly, MAS-GPT analyzes what multi-agent system is appropriate for addressing the question effectively and decides the required agents and structure.
Finally, MAS-GPT generates query-specific MAS (Listing I and Figure 7(a)), which involves five agents with specific tasks targeting the query and a final decision-making agent to provide the final answer.
This indicates that MAS-GPT can flexibly generate a query-specific MAS within one single LLM inference.
B.2. MAS-GPT Can Generalize to Unseen Queries
From Table 7, that given query whose domain is unseen from training dataset, MAS-GPT still works well. Specifically, it first analyzes the involved domain, the required understandings, and the difficulty of the given query. Secondly, it figures out a well-suited multi-agent system to answer the query, which directly mentions what agents and what structure are appropriate.
Finally, MAS-GP GPT generates a query-specifis MAS (Listing 2 and Figure 7(b)), which involves four parallel pipelines: two pipelines that answer the query via coding and two pipelines that directly answer the query. The first two pipelines are
13

---

MAS-GPT: Training LLMs to Build LLM--baed Multi-Agent Systems
Table 6: Case I: user query and the reasoning process generated by our MAS-GPT. MAS-GPT firstly analyzes the domain, difficulty, and required knowledge, then propose the idea of what multi-agent system is appropriate.
Query
The following is a multiple-choice question: Which of the following statements about any two events A and B is true?
The following choices are provided:
(A) P(A U B) implies events A and B are independent.
(B) P(AU B) implies events A and B are mutually exclusive.
(C) P(An B) implies events A and B are independent.
(D) P(AU U B) implies events A and B are mutually exclusive.
(E) B) = 0 implies events A and B are mutually exclusive.
(F) P(An B) = 1 implies events A and are independent.
(G) P(AUB)
implies events and B are independent.
(H) P(AnB) implies events and B are mutually exclusive.
(l) B) implies events A and B are mutually exclusive.
(J) P(AnB) implies events A and B are independent.
Please finish your answer with "the answer is (X)" where X is the correct letter choice.
Reasoning generated by MAS-GPT
The question is a multiple-choice problem in the domain of probability theory, specifically focusing on the relationships between events, such independence and mutual exclusivity. The difficulty lies in the need for precise understanding of probability concepts and their implications, as well as the ability to analyze and evaluate multiple statements to identify the correct one. This requires both theoretical knowledge and logical reasoning.
To address this question effectively, a multi-agent system should include agents that special- ize analyzing the relationships between events in probability theory. Each agent should focus on specific aspect of the problem, such independence, mutual exclusivity, or the implications of specific probability values (e.g.. P(A n B) 0 or P(AU B) 1). By dividing the task into these focused analyses, the system ensures comprehensive coverage of the problem space. Additionally, a final decision-making agent is necessary to synthesize the insights from all agents and provide a well-reasoned final answer. This structure ensures that the system can handle the complexity of the question while leveraging the strengths of specialized reasoning agents. The system is designed to align with the specific requirements of the question, ensuring that each agent contributes relevant and precise insights to the solution process.
both consisted of two agents: one for writing and executing the code, and one for organizing the final answer. Finally, an aggregation agent reasons over the four solutions and provides the final answer.
This indicates that MAS-GPT can generalize to unseen queries and generate appropriate MAS.
B.3. MAS-GPT Can Generate Novel MAS
From Table 8 and Listing 3 (Figure 7(c)), we see that given a query, the MAS-GPT proposes novel MAS that is unseen from our base MAS pool. Specifically, it builds an MAS with five parallel pipelines, where each pipeline actually involves an agent for answering the question and another agent for refining the solution. Finally, a decision decision-making agent is introduced to provide the final answer.
This shows that MAS-GPP is not merely memorizing the data during training, but actually learns patterns of designing MAS, indicating the generalization ability of our proposed MAS-GPT.
Listing 1: Case 1: Multi-agent system generated by MAS-GPT. MAS-GPT can generate query-specific MAS. MAS-GPT designs five independent responding agents, each responsible for different aspects of the task.
from utils import
14

---

MAS-GPT: Training LLMs to Build LLM-bbsed Multi-Agent Systems
class MAS ():
def _init_ (self, model_1ist):
self 11m = LLM (istt
def forward (self, taskInfo): ***
A multi-agent system for solving probability-related multiple-choice questions.
11
Steps:
12
1. 5 agenta analyze the task independently, each fecusing on a speci fic aspect of probability theory.
2. A final decision- agent reasons over the solutions and provides the final solution.
14
..*
15
16
I Step-by-step instruction for each agent to reason and generate answer instructions
18
f"Task: (taskInfo//n//noauu on analyzing the relationship between events and in terms of independence. Solve the task.", f"Task: (taskInfo) AnAnFocus
Mnnnooos en analyzing the relationship between events A and B in terms of matual exclusivity. Solve the task.", ["Task: PaaskInfo//n/noeee
19
the implications of P \eup and \cap B) on the relationsaip between events A and в. Solve the task.",
21
I"Task: (taskInfo//a//Foauu on the definitions and properties of probability union and intersection. Sol ve the task.",
f"Task:
Identifying any logical inconsistencies or impossibilities in the given choices. Solve the task."
23
24
# Call the 11m to generate each sclution cot_results- = (folf. (instruction) for instruction in instructions]
27
28
# Get the instruction for the final decision-making agent based on all generated solutions
final mamal_dciisie_niii deeciiiiininnnttttiion instruuttion = self. hut.ggee__ffalleeeeeiissssttttto (taskInfo, cot_results)
30
31
Call the 1im to process the final decision-making instruction and generate the final answer aa
32
Final_decisionnresuut
33
34
Return the final solution
35
return
def [el_final_deciiinn ott_ ***
instruction (self, taskInfo, cot_results):
38
39
Format an instruction for final decision-making based on a given task description and list of solutions.
41
Args:
42
taskInfo (str): A description of the task that needs to be completed: cot_results ( A list containing solutions or reasoning steps for the task.
43
45
Returns:
46
str: A formatted instruction that includes the task description, each solution, and a prompt for final decisioo-making.
47
49
# Initialize the instruction text with a general guideline instruction = ="Task: \ \n\n"
50
51
Appenc each solution from cot cot_results to the instruction for i, result enumerate (cot_results)
53
15

---

MAS-GPT: Training LLMs to Build LLM-bbsed Multi-Agent Systems
Number each solution for instruction E"Solution (i+1): (oll)::ltrrssttmmm clarity
55
56
Add the final prompt to encourage reasoning over the solutions and provide final answer
57
instruction += "Given all the above solutions, reason over them carefully and provide a final answer to the task."
58
Return the complete instruction text return instruction
60
Table 7: Case 2:
query and the reasoning process generated by our MAS-GPT. MAS-GPT firstly analyzes the domain,required elements, and the difficulty, then propose the idea of what multi-agent system is appropriate.
Query
An electric dipole consisting of charges of magnitude 1.50nC separated by 6.20pmm is in electric field of strength 1100 N/C. What is the magnitude of the electric dipole moment?
Reasoning generated by MAS-GPT
The question involves calculating the magnitude of the electric dipole moment, a concept rooted in physics, specifically in the domain of electromagnetism. The problem requires understanding the relationship between the charges, their separation, and the resulting dipole moment. While the mathematical computation straightforward, the problem demands precise application of the formula for the dipole moment and careful handling of units, which adds a layer of complexity.
To address this, one would analyze the question by identifying the key variables (charge magnitude, separation distance) and the formula (dipole moment = charge x separation).
A multi-agent system well-suited for this task because it can leverage multiple approaches ensure accuracy and robustness. First, agents independently break down the problem into steps, generate Python code to compute the solution, and execute the code verify the caleulations. This ensures that the solution is not only theoretically correct but also computationally validated. Additionally, other agents can directly generate solutions without code, providing baseline for comparison. Finally, an evaluation agent can synthesize the outputs from all agents, ensuring consistency and selecting the most reliable solution. This multi-faceted approach is particularly effective for scientific problems where both analytical and computational accuracy are critical.
Listing 2:
Case Multi-agent system generated by MAS-GPT, MAS-GPT can generalize to unseen queries from SciBench (Wang et al., 2024a), generating an appropriate multi-agent system handle the query.
from utils import
class MAS (A:
def _init_ (self, model_list)]
self. 11m = LLM (model_list)
def forward taskInfo) .*.
A multi-agent system for solving math problems executing code and directly answering.
Steps:
11
1. 2 agents independently solves the problem by breaking it down into steps and generating code, where each agent organizes the solution based on the code execution results, ensuring clarity and correctness.
2. agents generate solution directly, which provides baseline solutions espocially when code generation is challenging
13
3. A final agent evaluates all the solutions and determines the final solution.
14
# parallel pipelines to solve the problem independently
solutions
17
16

---

MAS-GPT: Training LLMs to Build LLM--based Multi-Agent Systems
The first two pipelines generate code solve the problem for in range :
19
20
answer, output =
(taskInfo)
solution = self. (taskInfo, answer, output) solutions solutions.append (soonnstt
23
24
# The third pipeline generates a sclution directly for in range (
25
solution self 11m. waw.mmmaalllliiaa (taskInfo)
27
solutions solutions.appenteeeee.o (ppennd)))lutionn)
28
29
# Netermine the final solution based on the generated solut final_solution self. legggeeftaaa___e iinaa_sclution itaskinfo, sclutions) return final solution
32
38
def generate_cooe _aaele__ooe__eeeeeeeeeeeeeeee__e pet_ou (self, taskInfo)
34
Generate Fython code solve the mathematical problem and execute the code to get the output.
36
Args:
37
taskInfo (str): The mathematical problem to be solved.
Returns:
tuple containing:
40
str: The answer generated by the LIM model.
41
str: The output of the code execution.
42
code good_geerrationnisttriee
f"""Yсu are an expert in solving mathematical
problems.
44
*Problem:**
45
(taskInfo)
46 ** Instrructions:
1. Analyze the problem and list the steps required to solve it.
48 Generate Fython code that can help solve the problem. The code should:
49 Print important intermediate results in the calculation process, along with clear explanations.
Store the final calculation result in variable named 'output'. This variable should contain the final result of the computation and defined at the global scope. 51 Be directly executable. The code should run and produce a result when executed.
52 Wrap your final code solution in <Code Solution> and </Code Solution>. For example:
53 <Code Solution>
Your function code here </Code Solution
56 I""
57
# Call geneeaat_ande and_aad__tt__e andd_extact__ae to generate answer and extract the code answer, code = generate and_axxd_ aard__extactt_ooe (1lm=self. 11m, prompt= ger.ggeraaiionisssretteettt
59
60
Call 'executa_code" to execute the generated code and get output output = execute_code (code)
61
return answer, output
64
def organize (self, taskinfo, answer, result):
65
66
Organize the solution based the code execution results. Args:
68
taskInfo (str): : The mathematical problem to be solved. answer (str): The initial solution generated by the LLM model.
69
70
result (str): The output of the cede execution.
Returns:
The organized solution based on the code execution results.
73
.*"
74
arganize_instructe Protesrr
75
(taskInfo)
+Initial Solution:**
77
[answer]
17

---

MAS-GPT: Training LLMs to Build LLM--baed Multi-Agent Systems
+*Code Execution Result:** (result)
79
80 To solve the **Probleme,, the +*Initial Sclution* provides steps and python code for calculations. The A*Code Execution Result** is the output of the code.
Based the +Initial Solutiona and **Code Execution Result**, provide final solution to the problem. Include the results the code calculation your response. Your final response should be complete as if you are directly answering the problem. """
solution self. mel...m..callllllc www.aaaleeerraaaaaassstttooooo return solution
83
84
85
def innlrfiiaa_sluuiiooooooo
taskinfe, sclutions):
Evaluate the solutions provided by the agents and determine the final solution.
88
Args:
89
taskInfo (str): The mathematical problem to be solved.
90
solutions (list): A list of sclutiens provided by the agents.
Returns:
92
str: The final solution the mathematical problem.
93
.""
94
Prepare the solutions in a numbered format for evaluation solutions_str
for solution in enumerate (sclutions)
97
soluticns_str I"Solution (i+1l'ntsolute An
98
99
- #ro*;=Probeem:+:
(taskInfo) **oouttion::*
102
(solutions_ss str)
103
Several solutions are given to solve the problem. Carefully evaluate these solutions. This consistency in answers is crucial for determining the most reliable solution.
You should provice final solution to the problem. Your final response should be complete as you are directly answering the problem. "** final_solution = self.llm. hal.:ilm.aalllllmee oo return final_sclution
105
106
18

---

MAS-GPT: Training LLMs to Build LLM--basdd Multi-Agent Systems
Table 8: Case 3: user query and the reasoning process generated by our MAS-GPT. MAS-GPT firstly analyzes the domain, difficulty, and required knowledge, then propose the idea of what multi-agent system is appropriate.
Query
The following is a multiple-choice question: Suppose that there are two firms in the market facing no costs of production and a demand curve given by Q= 150 P for their identical products. Suppose the two firms choose prices simultaneously as in the Bertrand model. Compute the prices in the nash equilibrium.
The following choices are provided:
(A) 30
(B) 50
(C) 0
(D)
(E) 150
(F) 75
(G) 90
(H) 120
(I) 100
(J) 60
Please finish your answer with "the answer is (X)" where X is the correct letter choice.
Reasoning generated by MAS-GPT
The question pertains to a classic economic problem in the domain of game theory and microeco- nomics, specifically focusing on the Bertrand model of price competition. The difficulty lies understanding the strategic interaction between two firms, the implications of no production costs, and the Nash equilibrium concept. Solving this requires both theoretical knowledge of economic models and the ability apply mathematical reasoning to derive the equilibrium price.
To address such a question, one must first break it down into its core components: the de- mand curve, the firms' identical products, and the simultancous price-setting behavior.
A multi-agent system well-suited for this task because can leverage multiple perspec- tives ensure accuracy and depth in reasoning. By employing agents that independently analyze the problem, generate solutions, and refine their responses through iterative feedback, the system ensures robust exploration of the problem space. This approach is particularly valuable for question like this, where multiple solution paths (e.g., algebraic derivation, economic intuition) can lead to the correct answer. The iterative refinement process allows for cross-verification of solutions, reducing the likelihood of errors. Finally, a decision becision-making making agent synthesizes the diverse solutions, ensuring that the final answer both logically sound and consistent with the principles of the Bertrand model. This structured reasoning process aligns with the complexity of the question and ensures reliable outcome.
Listing 3: Case 3: Multi-agent system generated by MAS-GPT. MAS-GPT can generate novel MAS.
from utils import
class MAS():
def init (self, model list):
self. LLM (model_list)
def forward (sclf, taskInfo)
À multi agent system for solving general tasks.
10
19

---

MAS-GPT: Training LLMs to Build LLM--basdd Multi-Agent Systems
Steps:
12
1. 5 agents solve the task independently.
2. Each agent reflects on the sclutions and provides an improved solution.
3. A final decision-making agent reasons over the improved solutions and provides the final solution.
15
n**
16
# step-by-step instruction for each agent to reason and generate answer instruction = ="Task: (taskInfo] \n\nPlease solve the task."
17
19
# Set the number solutions to generate; using for variety and diversity N 5
20
21
# Call the 11m to generate each sclution
cot_results - [self. 11m.cam (mmissstttutti for in range (N)
24
# Get the instruction for the self-refine process based on all generated solutions
25
self self_refine_instrcctt - self oof. cot_results)
26
27
Call the 11m to refine each solution refined_results =
28
for in range (N)
30
# Get the final making instruction based on all refined solutions
31
tist final_ cerallaacciiicnnnttteee instruction = selt.get_ decissdn instruction (taskInfo, refined_resultss)
33
Call the 11m to process the final decision-making instruction and generate the final answer
34
final_decision result = self. seff.llm.com mel..lim.calll..mmmmm 11m dtt.aaal.eessssccccssttttttiit
35
36
# Return the final solution return dima_ddcisioon_ceull
37
38
39
def
(self, taskInfo, cot_results):
Format an instruction for self-refinement based on given task description and a list of solutions.
42
43
Args:
taskInfo (str): A description of the task that needs to be completed. cot_results (list): list containing solutions or reasoning steps for the task.
45
46
47
Returns:
str: formatted instruction that includes the task description, each solution, and a prompt for self-rofinement.
49
50
51
# Initialize the instruction text with general guideline instruction #"Task: (ftskkinfoo) \n\n"
53
54
# Append each solution from cot_results to the instruction
35
for i, result in enumerate rootrtss) instruction += E"Solution (i+11: resuulltttest AnAn" Number each solution for clarity
57
58
I Add the final prompt to encourage self-refinement and improvement instruction += "Given all the above solutions, reason them carefully and provide improved solution to the task."
60
61
# Return the complete instruction text return instruction
62
64
def secision instruction (sels, taskInfo, refined
20

---

MAS-GPT: Training LLMs to Build LLM--based Multi-Agent Systems
METHOD
HUmanEvAl
MATH HUMANEVAL-
CHATDEV (QIAN AL.. 2024A) MAS-GI GPT (OURS)
83.33
84.04 87.23
62.07
91.18
77.59
Table Comparisons with task-specific method, ChatDev (Qian al., 2024a), which is specifically designed for software development.
65
Format an instruction for final description and list of refined solutions.
based a given task
67
68
Args:
taskInfo (str): A description of the task that needs to be completed. refined_resultt (list): list containing refined solutions for the task.
71
72
Returns:
73
str: A formatted instructien that includes the task description, each refined solution, and prompt for final decision-making.
75
76
# Initialize the instruction text with a general guideline instruction = #"Task: inltaskInoo/nna
77
79
# Appenc each refined solution frem refined results to the instruction for i, result in enumerate reeiine_resultss= instruction += f"Solution (i+1) (oesulltttrttmmm Inin" i Number each solution for clarity
80
81
83
# Add the final prompt to encourage reasoning over the solutions and provide a final answer
84
instruction += "Given all the above solutions, reason over them carefully and provide final answer to the task." "
85
86
Return the complete instruction text
87
return instruction
C. Additional Experimental Setups.
C.1. Descriptions of Datasets
We provide overall descriptions of the training and testing datasets in Table 10.
C.2. Evaluations
In this section, detail our evaluation approach. For queries with ground truth answers, employ LLMS to extract the MAS output and compare it with the ground truth. For code benchmarks like HumanEval and MBPP, we assess correctness using test cases.
LLM-based Evaluation with Ground-Trune Answer We utilize LLMs to perform evaluation with ground-truth answer. However, direct evaluation against the ground truth is incompatible as the LLM annotates the response itself. To address this issue, we adopt two-step evaluation process based on the prompts used AutoGen (Wu al., 2023), first extract the answer, then evaluation. Here the responses generated by multi-agent systems (MAS) are often unstructured and irregular, making it difficult extract the final answer to query using rule-based methods. To avoid extraction errors that could impact the evaluation of MAS performance, we use LLMs to automate the answer extraction process. Specifically, we prompt the L.M to extract the answer from the MAS response based on predefined rules and then ask the L.M to compare it with the ground truth. The prompts used for this process are detailed in Table 13.
Code Evaluation with Test Cases We evaluate the MAS performance on coding tasks based on pass rate on test cases, with a two step approach: first, prompting the LLM extract the code from the MAS response, and second, executing it in a
21

---

MAS-GPT: Training LLMs to Build LLM--based Multi-Agent Systems
Purpose
Dataset Name
Domain
Sub-Domains
Sample Number
Counting Probability
Geometry
Algebra
MATH (Hendrycks et al., 2021b)
Math
Number Theory
6000
Precaleulus
Prealgebra
Training
Intermediate Algebra
GSM8K (Cobbe et al., 2021)
Math
1000
GSM-Hard (Gao al., 2023)
Math
319
AQUA-RAY RAT (Ling et al., 2017)
Reasoning
1000
MBPP (Austin et al., 2021)
Code
374
SciQ (Welhl et al., 2017)
General QA
2000
Humanities
Social Science
MMLU (Hendrycks et al., 2021a)
General QA
1529
STEM
Others
Counting Probability
Geometry
Algebra
MATH (Hendrycks et al., 2021b)
Math
Number Theory
500
Precalculus
Prealgebra
Intermediate Algehra
Testing
GSM8K (Cobbe et al., 2021)
Math
500
GSM-Hard (Gao al., 2023)
Math
500
HumanEval (Chen et al., 2021)
Code
164
HumanEval-PPe (Liu et al., 2023)
Code
164
GPQA (Rein et al.. 2023)
Science
448
SciBench (Wang al., 2024a)
Science
500
Humanities
Social Science
MMLU (Hendrycks al., 2021a)
General QA
500
STEM
Others
AIME-2024
Math
30
Table 10: Descriptions of benchmarks
coding environment to calculate the pass rate; see the prompts used for extract code and functions in Table 14.
22

---

MAS-GPT: Training LLMs to Build LLM--baed Multi-Agent Systems
Wake pueet 10 de
Cham--t.lhaughh agent
(a) CoT
(b) 5 CoT SC
(c) Rellection
(d) Scientific LLM Debate
yhing Prly
"A
Math Ensemble
(f) Test Refine
g Test Fix Ensemble
(h) Ensemble Format
Gode Congrne
Cole Aent
(i) Quality Diversity
ii L.LM Debate
(k) Code Organize
(I) Code Test
(m) Step Back Abstraction
(n) Code L.M Dehate
(o) Dynamic Agent
(p) Hcuristic Simulation Refine
Eightss
(q) Priority Refine
(r) Socratic Questioning
(s) Strategy Engineer Engineer-SSientisst
Scientist
(1) Code Basic Ensemble
Figure 6: Visualization of our MAS pool
23

---

MAS-GPT: Training LLMs to Build LLM--baed Multi-Agent Systems
Eaduulityy Analyci Aguee
(D: EI
(D:FF
Defiudiors popertls
redependencs Aeokysis Agunt
Informtioon plook Agent
(c) Visualization of Case Figure 7: Visualization of three cases. (a) MAS-GPT can generate query-specific MAS. (b) MAS-GPT can generalize to unseen queries. (c) MAS-GPT can generate novel MAS.
(a) Visualization of Case
(b) Visualization of Case
24

---

MAS-GPT: Training LLMs to Build LLM-bbsed Multi-Agent Systems
Table 11:
The prompt for nutre-conniiteenc-ooreeeedddd pair refinement. This prompt is fed to GPT-40-2024-11-20 to adjust the MAS and generate a reasoning statement. The prompt integrated with a user query and the selected MAS represented by Python code.
I will give you a question and multi-agent system. The multi-agent system is described in the format of Python code, where each agent is represented by agent-specific instruction and one call 1lm. Though the multi-agent system can answer the question, it may not be the best one. You task is return me two things: an improved multi-agent system and a paragraph.
The improved multi-agent system should be more related to the question, while basically, try not to change the architecture compared to the original multi-agent system.
For example, the multi-agent system is in a parallel structure (e.g.. parallel agents generate answer and I agent select the best answer). you may keep the structure unchanged while only changing each parallel agent's instruction. If the multi-agent system is already suitable, you may only modify the instructions in the multi-agent system more relevant to the question while leaving the structure unchanged. If you think additional agents are required (e.g., the question is difficult and complex), you may add some related expert agents to enhance the multi-agent system.
The paragraph should first analyze the question itself, from the perspectives of domain and difficulty. Then, you should provide reasoning process to bridge the question and the improved multi-agent system. The reasoning process should be in the views of that how analyzes the question and objectively thinks about what multi-agent system is needed. Then, the reasoning process can finally and logically lead to the improved multi-agent system. Do not mention "this multi-agent system", or "the improved multi-agent system", rather, say "a multi-agent system" instead. Do not mention the original multi-as multi-agent system or the original structure.
Please follow the following format requirements:
The improved multi-agent system should be included between <CODE> and </CODE> The paragraph should be included between <PARAGRAPH> and </PARAGRAPE>
Please firstly generate the multi-agent system and then generate the paragraph. The paragraph should analyze about the question and the generated multi-agent sytem, such that when one sees the (question, paragraph, the improved multi-agent system) triplet (wihtout the original multi-agent system), one can understand the reasoning process behind the improved multi-agent system. Notice! The paragraph should never mention the original multi-agent system the original structure.
The question is: {query}
The multi-agent system is: [MAS code)
25

---

MAS-GPT: Training LLMs to Build LLM-bbsed Multi-Agent Systems
Table 12:
The prompt for generating a reasoning process if the refined MAS fails. This prompt is fed to GPT-44-2022-11-22 10-2-11-20 to generate a reasoning statement. The prompt is integrated with a user query and a selected MAS represented by Python code.
I will give you a question and multi-agent system. The multi-agent system is described in the format of Python code, where each agent is represented by agent-specific instruction and one call_Imme You task is return me paragraph.
The paragraph should first analyze the question itself, from the perspectives of domain and difficulty. Then, you should provide a reasoning process to bridge the question and the provided multi-agent system. The reasoning process should be the angle of views that how one analyzes the question and objectively thinks about what multi-agent system needed. Then, the reasoning process can finally and logically lead to the provided multi-agent system. Do not mention "this multi-agent system", "the provided multi-agen system", rather, say "a multi-agent system" instead.
The paragraph should analyze about the question and the provided multi-agent sytem, such that when one sees the (question, paragraph, the provided multi multi-agent system) triplet, one can understand the reasoning process behind the provided multi-agent system.
Remember, the paragraph should be included between |PARAGRAPH; and /PARAGRAPHI,
The question is: {query)
The provided multi-agent system is: [MAS code]
26

---

MAS-GPT: Training LLMs to Build LLM-bbsed Multi-Agent Systems
Table 13: Prompts for extract answer and answer evaluation.
You are a helpful Al assistant tasked with extracting the final answer from a provided solution.
*#Input:*$
1. A problem statement, prefixed with "===Problem: <problem>".
2. A solution to the problem, prefixed with Solii
"*Problem and Solution:**
Problem: {query)
==olutiion: {response}
Instrructions:
Carefully analyze the solution and extract the final answer in reply: "The answer is <answer extracted> in reply".
the solution does not contain a final answer (e.g., only reasoning, code without execution, or incomplete information). respond with: "The reply doesn't contain an answer."
Ensure that the extracted answer exactly presented in the solution. Do not infer or use external knowledge. Do execute the code yourself.
Remember, Never execute the code yourself! Never doing any computation yourself! Just extract and output the existing answer!
You are helpful Al assistant. You will use your coding and language skills to verify the answer. You are given:
1. A problem, which is going to start like "=-Problem: <problem>".
2. A ground truth answer, which is going to start like "==Ground truth answer:".
3. A reply with the answer to the problem, which are going to start like "e==-e,y:""
Please do the following:
Extract the answer reply: "The answer <answer ext rxctedx in reply".
2. Check whether the answer in reply matches the ground truth answer. When comparison is not obvious (for
example, 3*)) and 7.348), you may compare by calculation, allowing small margin of error.
3. After everything is done, please give each reply comment like the following options:
'Thhe answer is correct."
"The answer is approximated but should be correct. Correct Answer: <ground truth answer>
Answer
extracted: <answer ext excteed>""
"The answer is incorrect. Correct Answer: <ground truth answer> Answer extracted: <answer
extracted>."
"TTe reply doesn't contain answer."
Here the problem, the ground truth answer and the reply: ===Probem:: {query)
Ground truth answer: {ground truth answer)
Reply: (Reply)
27

---

MAS-GPT: Training LLMs to Build LLM--baed Multi-Agent Systems
Table 14: Prompts for extract code and functions.
You are given a **Problem*** ** and *Solution The Problemm** asks for a code function. Extract the final code function from the Solution **Problem:** {query)
**Solution:** {solution]
Please follow the following rules:
-Only output the code function that exists in the #*Solution**, without any additional explanation or content.
Do not modify any part of the code function.
Remove parts like 'example use' or 'test cases'.
If the **lutionn** does contain code function, respond with: 'The reply doesn't 't contain code function."
28


Reviews:
[{"review_id": "T5802ddT0y", "reviewer": "Reviewer_y7aF", "summary": "In this paper, the authors propose to train a GPT to generate the code that represents a Multiagent system, and thus provide a team build for each query. Specifically, the author proposes to use some existing datasets as training samples and run these samples on 40 different predefined systems to form training pairs for the model to learn the best candidates. The results on 8 datasets show that the proposed method is better than many fixed baselines, such as CoT and DyLAN.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your appreciation of our idea and motivation. We would like to address your remaining concerns in the following.\n\n---\n\n&nbsp;\n\n**Methods:** The motivation for choosing these 40 MAS pools is unclear.\n\n**A:** Sorry for the confusion. Let's clarify.\n\n(1) The key motivation for choosing these 40 MAS in the initial pool is for teaching the LLM the basic format for representing a MAS. These 40 MAS cover basic elements such as chain-of-thought prompts, role-playing prompts, LLM-calling functions, and functions to get code execution results. As the base LLM does not know our representation of MAS (verified by poor performance in Figure 5 b, N=0), including these basic elements is critical to teach the LLM our MAS representation.\n\n(2) MAS-GPT is not simply selecting MAS. For example, during the inference of MAS-GPT on GSM-Hard, 925 out of 1000 generated MAS are unseen from the training data, indicating that MAS-GPT is not merely selecting MAS but generating query-specific appropriate MAS.\n\n(3) These 40 MAS constitute only a subset of the full training data. Actually, these 40 MAS only serve as `seed MAS`. They will be evolved during the query-MAS pair refinement process, which indeed enlarges the pool with some random systems.\n\nWe will include these in the revision.\n\n---\n\n&nbsp;\n\n**Exp1:** The authors compare the cost by computing the number of the inferences ... better to include token cost.\n\n**A:** Thanks for the suggestion. Due to limited time, we report the following results and report more in our revision, where we see that MAS-GPT achieves **the best with the least token cost**.\n\n| |AgentVerse|DyLAN|MAS-GPT|\n|-|-|-|-|\n| LLM Calls | 12.05 (70B) | 12.96 (70B) | 1 (32B) + 6.44 (70B)    |\n| Tokens    | 8610 (70B)  | 4874 (70B)  | 1133 (32B) + 2127 (70B) |\n| Acc (%)   | 59.36       | 60.54       | 64.47   |\n\n---\n\n&nbsp;\n\n**Exp2:** It is fairer to compare the performance of the cross-domain or zero-shot capability.\n\n**A:** Thanks for the suggestion. Acturally, we have compared the performance of the cross-domain or zero-shot capability on datasets such as HumanEval, HumanEval+, GPQA, SciBench, and AIME-2024.\n\nPlease note that all of these datasets are **only used for evaluation, but not training**. We would like to highlight that GPQA (graduate-level QA), SciBench (college-level scientific problems), AIME-2024 (mathematical competition) are all much harder than the training data. We also additionally conduct experiments on MedQA (medical domain). We can see that our method achieves **the best performance in these cross-domain setups**. These experiments also verify the generality of our MAS-GPT: it could generalize to domains unseen in training data and to much harder queries than those in training.\n\n| |HumanEval|HumanEval+|GPQA|SciBench|AIME|MedQA|\n|-|-|-|-|-|-|-|\n|DyLAN|79.01|75.78|35.98|19.79|53.33|76.34|\n|MAS-GPT|80.25|78.88|37.62|24.21|66.67|78.60| \n\n---\n\n&nbsp;\n\n**Exp3:** the baselines are largely included in the system. For instance, the CoT is inside the 40 MAS pool. ... It would be better not to include the baseline in the pool to see the filter-out performance...\n\n**A:** Thanks for the comments.  We would like to answer from three perspectives.\n\n(1) Firstly, our initial MAS pool only includes a few baselines with simple operation such as Co; but does not inlcude complicated baselines such as DyLAN. We include these simple elements to teach the LLMs to generate MAS in our desired format (also see response to methods). \n\n(2) Secondly, we would like to kindly inform the reviewer that including existing baselines into our initial MAS pool does not conflict with the rationale and motivation of MAS-GPT. One exciting and promising advantage of MAS-GPT is that it could `stand on the shoulders of giants`. Ideally, if we could include all existing high-performance MAS methods into the MAS pool, in real-world applications, we only need to deploy MAS-GPT to solve diverse user queries, rather than deploying multiple MAS and designing complicated rules to select them.\n\n(3) Following your suggestions, we exclude CoT from the MAS pool and re-run the experiments. We report the results in the following. From the table, we see that excluding CoT performs comparably with the current version. This result is reasonable because the performance of CoT is normal such that during the process of pair evaluation and selection, CoT is less likely to be paired with queries, resulting in few CoT samples in the training data.\n\nWe truly believe in this new paradigm. With more diverse and high-performance MAS being included, we believe that MAS-GPT will be further advanced in a way similar to the advancement of ChatGPT: with better data and training techniques, the models become better.\n\n| |MATH|MMLU|GPQA|\n|-|-|-|-|\n|Before|68.65|78.38|37.62|\n|After|69.09|75.59|37.62| \n\n---\n\n&nbsp;\n\nOverall, we hope that our responses can fully address your concerns and will be grateful for any feedback.", "claims_and_evidence": "The motivation of the proposed method is generally clear. The authors want to solve the issue of an adaptive system for each query and human labor, as well as the system cost.", "methods_and_evaluation_criteria": "The motivation for choosing these 40 MAS pools is unclear. Why do you include these models? Does it mean the model can only support existing frameworks? Since the MAS is evolving, the proposed method is more like a model selection algorithm rather than an auto-team building algorithm. Although the system can generalized to unseen MAS systems, it is better to conduct an experiment to enlarge the pool with some random systems already.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The cost the each sample is not fair. The authors compare the cost by computing the number of the inferences, however, the inference has diverse lengths, which is unfair for short response generators. It is better to include the real money or token cost of the system to have a direct understanding of the result.\n\nThe experiment design is unfair. This is two-fold. First, the proposed methods use the training set of the evaluation dataset while the baselines do not use it. Thus, it is the comparison between fine-tuning and zero-shot learning. It is fairer to compare the performance of the cross-domain or zero-shot capability of the proposed method. Second, the baselines are largely included in the system. For instance, the CoT is inside the 40 MAS pool. Thus, as long as the model can figure out the performance of CoT, they can beat it naturally. It will be better not to include the baseline in the pool to see the filter-out performance, mimicking the real-world cases where we do not know what other models are.", "relation_to_broader_scientific_literature": "Highly related to Auto team building or multiagent systems.", "essential_references_not_discussed": "The discussion is relatively sufficient.", "other_strengths_and_weaknesses": "I like the idea of converting the auto-team building tasks to a coding task. And ask GPT to write code to build team.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K2SfJi4w2A", "reviewer": "Reviewer_2xyd", "summary": "The paper presents MAS-GPT, a novel approach that automates the creation of multi-agent systems specifically tailored to user queries using a single inference. The authors address key limitations in existing MAS approaches, namely high manual crafting effort and high computational costs, and propose to simplify MAS construction as a generative language task. They introduce a dataset construction pipeline emphasizing consistency, which facilitates supervised fine-tuning of MAS-GPT. Extensive experiments demonstrate MAS-GPT’s superiority across various tasks, proving its effectiveness, efficiency, and adaptability.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad to see that you acknowledge that our approach is novel, methodologically sound, flexible, and scalable. We are sorry that we leave you some concerns and let's clarify.\n\n---\n\n**W1&Q2:** The generalization capability of MAS-GPT across significantly different or novel task domains remains unclear...\n\n**Answer 1:** Our MAS-GPT can generalize well to new tasks and we have put efforts to verify this point in our paper. Sorry for potentially missing details.\n\n(1) SciQ and SciBench are significantly different, where SciQ is a knowledge-based benchmark (multiple-choice, https://huggingface.co/datasets/allenai/sciq) while SciBench is a reasoning-based benchmark (non-choice, https://huggingface.co/datasets/xw27/scibench). Meanwhile, SciQ is a dataset for **4th grade** exams while SciBench are all **college-level** scientific problems, which are much challenging than those in training data. \n\n(2) Our paper also include results on GPQA (graduate-level QA) and AIME-2024 (mathematical competition), which are all out-of-domain and much harder than training data. Here, we additionally test on medical domain, MedQA (there is no medical dataset in training). From the table, we see that MAS-GPT is indeed generalizable.\n| |GPQA|AIME|MedQA|\n|-|-|-|-|\n|DyLAN|35.98|53.33|76.34|\n|MAS-GPT|37.62|66.67|78.60| \n\n(3) We kindly remind the reviewer that our MAS-GPT has achieved a better generalization capability compared to existing optimization-based methods: MAS-GPT does not require re-optimizing when being applied to different benchmarks. For methods such as GPTSwarm (ICML 2024 oral) and AFlow (ICLR 2025 oral), given a benchmark (e.g., MATH), these methods first optimize on a subset and then could only infer on the corresponding test set.\n\n---\n\n&nbsp;\n\n**W2:** The generated MAS topologies presented are relatively straightforward...\n\n**A:** Thanks for the comments.\n\n(1) The reason why we develop the MAS topologies at the current complexity is straightforward: the current complexity level is somewhat sufficient to achieve a pleasant performance on most of the benchmarks. All benchmarks are commonly used by MAS community and we follow their setups. If we continue to increase the complexity, it might be hard to achieve a good cost-performnace balance. \n\n(2) Our approach to training MAS-GPT is scalable and methodologically capable of supporting sophisticated topologies. To achieve this, one only needs to design more sophisticated MAS and include them into the MAS pool. Based on this, the trained MAS-GPT would be able to generate sophisticated MAS for complex queries. Meanwhile, MAS-GPT already supports using tools such as code executor, indicating its potential of scalability to include more tools to handle complex tasks.\n\n(3) The main contribution of this paper is pointing out a new direction for the MAS community. Like the training of LLMs (e.g., GPT-4), there are always cases that the trained MAS-GPT cannot solve no matter how well the current MAS-GPT is trained. That is, the current MAS-GPT is not the end. Similar to the continuing advancement of LLMs, our MAS-GPT could be continuously improved as the community designing more sophisticated (or better) MAS, tasks, and data samples.\n\n---\n\n**Q1:** If two MAS share an identical topology but differ in the instructions to the agent, are they counted as separate MAS instances?\n\n**Answer:** Sorry for the confusion. Yes, they are counted as separate MAS instances. The reasons for such criteria is that it is hard to automatically and merely distinguish topologies.\n\n---\n\n**Q3:** What are the costs for generating the training data in terms of GPU hours and costs for calling APIs?\n\n**A:** Collecting the training data roughly requires 245k calls of open-source LLMs. The training process takes roughly 32 GPU hours for training 32B-sized models. The refinement process roughly costs 143 US dollars for calling GPT-4o. Please note that MAS-GPT (32B) is only required to be trained **once** and applied to handle diverse queries **without re-training** both MAS-GPT and the LLMs that drive the MAS.\n\nMeanwhile, we will open-source all of the data at every step (e.g., before and after filtering) and models to facilitate future research.\n\n---\n\n**Q4:** Did you see new topologies?\n\n**Answer:** We have manually checked some of the generated topologies and found that MAS-GPT could generate novel topologies. Please refer to our case study of Case 3 (Section A.3). Meanwhile, please note that even if the generated topologies are the same as existing topologies, MAS-GPT would assign appropriate prompts (instructions) to the agents within the topologies, making them query-specific and appropriate MAS. For example, during the inference of MAS-GPT on GSM-Hard, 925 out of 1000 generated MAS are unseen from the training data, indicating that MAS-GPT is generating query-specific appropriate MAS.\n\n---\n\n&nbsp;\n\nOverall, we hope that our responses can fully address your concerns and will be grateful for any feedback.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "MAS-GPT builds upon and advances existing literature on multi-agent systems and large language models. It addresses critical limitations found in prior systems such as MetaGPT, ChatDev, and AgentVerse, notably manual configuration and high inference costs. By reframing MAS construction as a generative language task and utilizing SFT, this work bridges a significant gap in current MAS approaches and introduces a flexible, scalable alternative.", "essential_references_not_discussed": "Yes", "other_strengths_and_weaknesses": "**Strengths**:\n1. The inter- and intra-consistency-oriented approach is robust, methodologically sound.\n2. Thorough experiments with diverse benchmarks and backbones validate the generality and effectiveness of MAS-GPT.\n3. The writing is easy to follow.\n\n**Weaknesses**:\n1. The generalization capability of MAS-GPT across significantly different or novel task domains remains unclear. Although the authors designate benchmarks such as H-Eval and SciBench as out-of-domain, their training dataset explicitly includes MBPP (programming benchmark similar to H-Eval) and SciQ (science question-answering dataset similar to SciBench). Consequently, these tasks are not strictly out-of-domain. The LLM could memorize task and MAS patterns during sft and subsequently reproduce them during inference. \n2. The generated MAS topologies presented are relatively straightforward. This simplicity raises concerns about MAS-GPT’s potential to effectively handle complex real-world tasks that require sophisticated interactions and multi-agent collaboration.", "other_comments_or_suggestions": "Refer to the Questions.", "questions_for_authors": "Questions:\n1. In Table 1, the number of MAS is 7580. What is your criteria for distinguishing two MAS instances? Specifically, if two MAS share an identical topology but differ in the instructions to the agent, are they counted as separate MAS instances, or are they considered identical?\n2. How does MAS-GPT perform on tasks that are genuinely out-of-domain? For example, if MAS-GPT is exclusively trained on mathematics-related tasks (e.g., MATH and GSM8K), would the generated MAS structures remain effective for programming benchmarks such as HumanEval or MBPP? Could you discuss MAS-GPT’s transferability and performance in such scenarios?\n3. What are the costs for generating the training data in terms of GPU hours and costs for calling APIs?\n4. Have you examined the MAS topologies by representing them as Directed Acyclic Graphs? If so, have you identified any genuinely novel topological structures generated by MAS-GPT beyond those present in the training set? How do you verify that MAS-GPT is not simply memorizing training data topologies and applying them unchanged to tasks during inference?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "diaKrOIbhR", "reviewer": "Reviewer_w8Qs", "summary": "The paper introduces MAS-GPT, a novel approach that trains LLMs to automatically generate query-specific multi-agent systems (MAS) in a single inference step. Unlike previous methods requiring manual configuration or multiple LLM inferences, MAS-GPT simplifies MAS creation by reframing it as a generative language task, where the MAS is output as executable Python code tailored to user queries. The authors propose a consistency-oriented dataset construction pipeline to produce high-quality training data, enabling MAS-GPT to effectively learn to build adaptive MAS. Experiments on nine diverse benchmarks show that MAS-GPT consistently outperforms existing multi-agent methods, achieving better adaptability, generalization, and significantly lower computational costs.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad to see that you agree with our motivation and that our approach is novel. We notice that your main concern is on the experimental setups, of which we believe is caused by some confusion. (Since the comments in `Claims` are overlapped with `Weaknesses`, so we focus on the latter)\n\n---\n\n&nbsp;\n\n**W1:** The authors trained MAS-GPT and evaluated it within the same domain?\n\n**Answer:** Sorry for the confusion.\n\n(1) We would like to highlight that we do **NOT** 'train MAS-GPT and evaluate it within the same domain'. In contrast, we train MAS-GPT on diverse domains simultaneously and evaluate them on diverse domains **without re-training**. For example, during evaluation, GPQA (graduate-level QA), SciBench (college-level scientific problems), AIME-2024 (mathematical competition) are all **out-of-domain benchmarks and much harder** than the training data.\n\n(2) Following your advice, we report performance results in the medical domain, MedQA (there is no medical dataset in training). From the table, we see that MAS-GPT is indeed generalizable.\n\n||AgentVerse|DyLAN|MAS-GPT|\n|-|-|-|-|\n|MedQA|65.84|76.34|78.60| \n\n(3) Compared to many optimization-based methods, MAS-GPT is indeed more generalizable. GPTSwarm (ICML 2024 oral), AFlow (ICLR 2025 oral), and AgentPrune (ICLR 2025) are all benchmark-dependent methods. That is, to test their methods on one benchmark, they need to firstly optimize on a subset (with GT labels) of that benchmark. In contrast, our MAS-GPT is optimized on diverse training data and then **generalize to many benchmarks without modification**. \n\n---\n\n&nbsp;\n\n**W2:** The performance difference between \"Single\" and proposed method is not obvious?\n\n**Answer:** Sorry for the missing details. We run twice. We would like to emphasize from three perspectives.\n\n(1) MAS-GPT is the only method that consistently achieves better performance than SINGLE. Please refer to the following table, where we report the percentage of benchmarks where a method outperforms a single agent.\n\n|Method|CoT|SC|AgentVerse|DyLAN|MAS-GPT|\n|-|-|-|-|-|-|\n|Percentage|50|62.5|37.5|50|100| \n\n(2) On particularly challenging datasets such as AIME-2024 and GSM-H, our method achieves significantly better performance than SINGLE (e.g., 16.67 on AIME-2024), demonstrating substantial potential.\n\n(3) We agree that for some benchmarks, improvements are not significant. Please note that these patterns commonly exist in all methods. The reasons why we still report them are twofold. First, these benchmarks are commonly used in MAS literature and we follow their setups. Second, we want to show that our method is generalizable on diverse datasets. \n\n---\n\n&nbsp;\n\n**W3:** Improvements originate from manually designed MAS?\n\n**Answer:** Thanks for the comments. We would like to address your concerns from two perspectives.\n\n(1) Following your advice, we compute the number of unique MAS during test time (GSM-Hard, MATH, and SciBench) compared to those in training data. This strongly verify that MAS-GPT does **NOT** essentially memorize the training data.\n\n||GSM-Hard|MATH|SciBench|\n|-|-|-|-|\n|Unique/total|925/1000|690/1000|491/692| \n\n(2) It is NOT a bad thing if MAS-GPT sometimes generates MAS that exists in the training data. The key of training MAS-GPT is making it learn to generate appropriate MAS based on each specific query. The keyword here is `appropriate` but not `new`. This is actually similar to the training of LLMs, LLMs could generate identical sentences seen during training and also new sentences as long as they are appropriate.\n\n---\n\n&nbsp;\n\n**Q1:** How many runs?\n\n**Answer:** We performed 2 runs. See W2.\n\n---\n\n&nbsp;\n\n**Q2:** Why did the authors train and test MAS-GPT within the same domain? When claiming that MAS-GPT is efficient, did the authors account for the development and training time of MAS-GPT itself?\n\n**Answer:** We did NOT train and test MAS-GPT within the same domain. Please refer to responses in Weakness 1. \n\nPlease allow us to emphasize our focus on this paper. Our ultimate goal is to make MAS-itegrated application effective and efficient during inference. To achieve this, we consider a brand-new paradigm: training an LLM. This LLM (i.e., MAS-GPT) is trained on diverse data and can generalize to diverse scenarios, making it a step closer to this goal. Training is **one-time while inference could be endless** (just like OpenAI trains GPT-4 for one time and serves the world countless times). Training-time efficiency is not the focus of this paper, which could be a potential future direction.\n\nAs an analogy, in efficient LLM research, their goal is to ensure the trained model enables LLMs to be efficient in applications (inference), rather than focusing on training efficiency. Our work shares a similar objective.\n\n---\n\n&nbsp;\n\nOverall, we hope that our responses can fully address your concerns and will be grateful for any feedback.", "claims_and_evidence": "The primary claim of this paper is that MAS-GPT significantly improves adaptability, generalization, and computational efficiency compared to existing methods. To substantiate this claim, the authors conducted experiments across nine datasets.\n\nHowever, I find that the experimental setting presented in this paper is toy and not real, as the training and testing data are derived from the same domain. If the authors wish to convincingly demonstrate MAS-GPT’s generalization capability, they should conduct additional experiments on out-of-domain datasets, such as GAIA.\n\nAdditionally, the authors argue that MAS-GPT offers improved computational efficiency over manually designed multi-agent systems. Nevertheless, their evaluation does not account for the additional computational costs incurred by model training and dataset refinement, both of which likely require significant time investment. In the authors setting, the users need to train \"MAS-GPT\" every time for each domain separately.", "methods_and_evaluation_criteria": "The authors use the metrics from the benchmarks they use in experimental section. The metrics are good.", "theoretical_claims": "There is no theoretical claims involved in the paper.", "experimental_designs_or_analyses": "I checked all experiment results and settings. The critical issue is what I mentioned in previous part. The training and test data are from the same domain. I greatly suspect the usability of this method.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well-written, I could easy follow the reasoning logic. Overall, it is easy to understand. \n\n2. I agree the motivation of this paper. Building MAS manually is not practical. Thinking of the application/query grows, the human effort needed will be large. \n\n**Weakness**\n\n1. The experimental setup appears simplistic and unreliable. The authors trained MAS-GPT and evaluated it within the same domain, suggesting that users would need to retrain MAS-GPT every time they apply it to a new domain. When considering the training time required for MAS-GPT, I doubt it would actually be more efficient than manual MAS design. \n\n2. In Table 2, the performance difference between \"Single\" and proposed method is not obvious. Most the performance improvement seems coming from the GSM-H dataset. Improvements on other datasets are small and seems randomness. Multiple evaluations runss needed to be conducted. \n\n3.  The authors spent considerable effort manually refining MAS within their training dataset. As a result, I suspect that the observed performance improvements largely originate from the manually designed MAS, and that MAS-GPT essentially memorizes these manually provided MAS inputs.", "other_comments_or_suggestions": "The authors should further elaborate in their paper—particularly through experiments or additional analysis—why MAS-GPT is more efficient compared to manual MAS design.", "questions_for_authors": "1. In Table 2, how many experimental runs did the authors perform? Could the authors explain why the differences across datasets are relatively small?\n\n2. Why did the authors train and test MAS-GPT within the same domain? When claiming that MAS-GPT is efficient, did the authors account for the development and training time of MAS-GPT itself?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "VLHFuUT5WX", "reviewer": "Reviewer_xR52", "summary": "This paper proposes to train an LLM to build multi-agent system. The paper reframes MAS construction as a python coding task and represents MAS as executable python code. One key contribution is a consistency-oriented data construction pipeline that generates high-quality query-MAS pairs. Extensive experiments on different benchmarks and backbones indicate that MAS-GPT outperforms baselines in effectiveness, efficiency and generalization capability. MAS-GPT also excels in terms of inference costs.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that you acknowledge that our idea is novel, method is comprehensive, and direction is aligned with a broader trend toward developing more streamlined, scalable, and adaptive architectures. Let's address your remaining concerns!\n\n\n---\n\n**W1:** Representing MAS as python code neglects important aspects of agent functionality...?\n\n**A:** Sorry for the missing details! Our MAS-GPT actually **supports** functionalities mentioned by you (tool and multi-turn interactions) and they are indeed compatible with code.\n\n(1) Code execution tool is implemented as a python function `execute_code(code_text)`, which outputs the execution results to other agents. More tools can be implemented similarly.\n\n(2) While single-turn LLM call is represented by a function `call_llm(prompt)`, multi-turn interaction is represented by `multi_turn(history, prompt)`.\n\nRepresenting MAS as python code is a promising direction, as most functionalities in our AI community can be represented by python code. Along this new direction, we believe that with more data, tools, MAS, MAS-GPT could be continuingly improved as that from GPT-3.5 to GPT-4.\n\n---\n\n**W2:** Discussion on AgentPrune.\n\n**A:** Thanks for recommendation! We will include these discussions and cite it.\n\n(Methodology) The goals of our paper and AgentPrune are significantly different. We aim to optimize MAS-GPT **once** on diverse domains such that the MAS-GPT can **generalize** to diverse domains. While AgentPrune aims to optimize on one validation dataset such that it can work on the corresponding test dataset. There are also two recent papers with goals similar to AgentPrune: GPTSwarm(ICML2024 oral), AFlow(ICLR2025 oral). While these three require **re-optimizing** for each test set, MAS-GPT can generalize to diverse test sets **without re-optimizing**.\nSince in practice, users will not provide several examples in advance for optimization, we believe that MAS-GPT is a promising direction to truly make MAS practically applicable.\n\n(Experiments) We run the official code of AgentPrune on HumanEval and MMLU. From the table, we see that our method performs **much better with less inference cost.**\n\n||Method|Acc|Re-optimizing cost|Test cost (avg)|\n|-|-|-|-|-|\n|HEval|AgentPrune|70.12|349(70B)|9.8(70B)|\n|HEval|MAS-GPT|80.25|0|1(32B)+8.7(70B)|\n|MMLU|AgentPrune|75.05|280|7.3(70B)|\n|MMLU|MAS-GPT|78.38|0|1(32B)+4.7(70B)|\n\n---\n\n**W3:** Comparison with ADAS.\n\n**A:** Thanks for the advice.\n\n(1) The reason why we compare with AFlow rather than ADAS is that AFlow is an improved follow-up of ADAS.\n\n(2) We compare MAS-GPT with ADAS (optimized on GPQA). MAS-GPT performs **significantly better than ADAS on these diverse datasets at a much lower cost**, indicating that MAS-GPT is much more generalizable.\n\n||MATH|GSM8K|GSM-H|H-Eval|H-Eval+|MMLU|GPQA|SciBench|Avg|LLM calls|\n|-|-|-|-|-|-|-|-|-|-|-|\n|ADAS|34.7|36.3|12.4|75.9|69.6|76.6|38.1|14.5|44.8|21~41 (70B)|\n|MAS-GPT|68.7|93.4|62.4|80.3|78.9|78.4|37.6|24.2|65.5|1 (32B) + 6.4 (70B)| \n---\n\n**W4:** The performance gain seems modest?\n\n**A:** Thanks for the comments. We would like to respond from the following perspectives.\n\n(1) MAS-GPT is the **only method** that consistently achieves better performance than SINGLE. Here, we report the percentage of benchmarks where a method outperforms a single agent.\n\n||CoT|SC|AgentVerse|DyLAN|MAS-GPT|\n|-|-|-|-|-|-|\n|Percentage|50|62.5|37.5|50|100| \n\n(2) On challenging datasets such as AIME-2024 and GSM-H, our method achieves significantly better performance than SINGLE (16.67 on AIME-2024), demonstrating substantial potential.\n\n(3) We agree that for some benchmarks, improvements are not significant. Please note that these patterns commonly exist in all methods. The reasons why we still report them are twofold. First, these benchmarks are commonly used in MAS literature and we follow their setups. Second, we want to show that our method is generalizable on diverse datasets. \n\n---\n\n**Q1:** Are there cases where the MAS incorporates if-else structures...?\n\n**A:** Yes, we have observed interesting workflows. For example, we see MAS with if-else structures in several cases. (1) Agents evaluate the current status and determine whether to stop or continue solving. (2) If code executor shows that the code runs correctly, then the system ends; else the system continues.\n\n---\n\n**Q2:** \"code test agent\"?\n\n**A:** Yes. We verify the generated code by actually executing it. We implement two related python functions: one takes code piece as input and one takes code piece together with test cases as inputs.\n\n---\n**Q3:** Number of query-MAS pairs?\n**A:** Thanks for the advice. The selection process will filter out those queries that all MAS fail to correctly answer. The refinement process essentially replace those original query-MAS pairs(Line 256-258).\n\n|Step|Init|Select|Refine|\n|-|-|-|-|\n|Num|12292|11442|11442| \n\n---\n\nOverall, we hope that our responses can fully address your concerns and will be grateful for any feedback.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A. No theoretical claims involved.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The paper builds on and extends research in automatic and adaptive construction of agentic systems. In contrast to previous works that require manual design or multi-round refinement, this paper proposes training an LLM to generate MAS in one inference, thereby reducing cost and improving efficiency. This direction is aligned with a broader trend toward developing more streamlined, scalable, and adaptive AI architectures that simplify complex system design through end-to-end learning.", "essential_references_not_discussed": "Works that involve optimizing MAS architecture:\n[1] Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems. ICLR 2024", "other_strengths_and_weaknesses": "Strengths:\n1. The paper presents a novel idea by training an LLM to build multi-agent systems, thereby reducing both inference cost and the manual effort involved in system design. The motivation is clearly articulated, positioning the work as a natural evolution in the use of multi-agent orchestration.\n2. The authors provide a comprehensive description of their consistency-oriented data construction pipeline. The detailed rational behind the strategies enhances the credibility of the proposed pipeline.\n3. The presentation quality is good. The paper writeup is clear and easy to follow.\n\nWeaknesses:\n1. Representing multi-agent system as python code neglects important aspects of agent functionality, such as tool integration and multi-turn interactions. In practical agent systems, agents interact with their environments by executing tools and processing iterative feedback. However, in the scope of this research, the agent seems to be prompting LLM in a single-turn manner. Therefore, the approach somewhat resembles automated workflow orchestration or bootstrapped reasoning. This oversimplification is my main concern of this paper.\n2. The paper should add discussion and compare the performance and cost with AgentPrune[1], a pipeline for multi-agent communication. \n3. The method shares similarities with ADAS in its representation of MAS as code. The paper would benefit from a direct performance and cost comparison with ADAS, similar to the comparison provided with AFlow.\n4. The performance gain seems modest. In table 2, when compared to single LLM, MAS-GPT shows modest performance gains on most benchmarks except MATH and GSM-H. In table 3, similar observation is there for qwen2.5b model. This raises questions about the practical benefits of integrating multi-agent structures, especially given the additional complexity involved.\n\n[1]  Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems. ICLR 2024", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. For the generated MAS, have you observed any interesting workflows beyond the typical sequential prompting and answer ensembling? For example, are there cases where the MAS incorporates if-else structures, cross-references answers from different agents, or employs other non-linear strategies in its workflow?\n2. In Figure 7(l), you include a \"code test agent.\" Could you explain how this agent works in practice? Does it verify the generated code by actually executing it with test cases, or does it simply deduct whether the output aligns with the expected test case in natural language?\n3. The data curation process involves multiple selection and refinement steps to construct the final query-MAS pair dataset. Could you provide details on the number of query-MAS pairs retained after each selection step (e.g., after initial pairing, after inter-consistency selection, and following intra-consistency refinement)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jbm3BSru7b", "reviewer": "Reviewer_Ap5F", "summary": "This paper introduces MAS-GPT, a novel multi-agent system generation framework. Specifically, MAS-GPT employs a multi-agent generator model trained using modules such as MAS filtering, inter-consistency assurance, and intra-consistency enhancement. Designed to adapt to the nuances of diverse tasks, MAS-GPT aims to generate optimally performing MAS for tasks of the same category, thereby enhancing the capability of LLM-MAS to address a broad spectrum of problem types. The effectiveness of MAS-GPT is rigorously validated across a wide array of benchmarks, and the paper further presents comprehensive experimental results that substantiate its capabilities.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time devoted to reviewing our paper. We are glad to see that you acknowledge that our method is novel, experimental designs are sound, and our paper is insightful. It is encouraging. We would like to address your remaining concerns in the following.\n\n---\n\n&nbsp;\n\n**Experimental Designs Or Analyses 1:** Lack of Single Agent Framework Specification: The paper does not specify the framework employed for the single-agent baseline models.\n\n**Answer:** Sorry for the missing details. We employ vLLM library to deploy LLMs. For all baseline methods, we run their official open-sourced codes.\n\n---\n\n&nbsp;\n\n**Experimental Designs Or Analyses 2:** Cost Comparisons and Metric Justification. I have reservations regarding the cost comparison metric used in this experiment. To the best of my knowledge, standard practice in evaluating LLM cost involves assessing token cost and time cost to quantify both financial and temporal overheads. Therefore, I recommend that the authors augment this section by providing token cost and time cost data for MAS-GPT and all baselines across the benchmarks used. Furthermore, the methodology for performance measurement in this cost comparison section, as well as the source of the experimental data, are not clearly delineated. Please provide this essential supplementary information.\n\n**Answer:** Thanks for the recommendation! Following your advice, we have reported the token consumptions in addition to the number of LLM calls in the following table. In our initial experiments, we did not record token consumptions, so we need to conduct additional experiments during rebuttal. Due to limited time, we report the following results . We will report more in our revision. From the table, we see that our MAS-GPT achieves the best performance with the least inference costs (both call times and token consumptions).\n\nThe performance measurement uses accuracy as metric, which is averaged on all benchmarks in Table 2. And the specific numbers are taken from Table 2. We will include all of these in our revision.\n\n|              | AgentVerse    | DyLAN         | MAS-GPT                       |\n| ------------ | ------------- | ------------- | ----------------------------- |\n| LLM Calls    | 12.05 (70B)   | 12.96 (70B)   | 1 (32B) + 6.44 (70B)          |\n| Tokens       | 8610.07 (70B) | 4874.22 (70B) | 1133.18 (32B) + 2126.98 (70B) |\n| Accuracy (%) | 59.36         | 60.54         | 64.47                         | \n\n\n---\n\n&nbsp;\n\n**Experimental Designs Or Analyses 3:** Scaling Effects of Data Size and Supervised Fine-tuning: The \"Scaling effects of data size\" analysis appears inconsistent with established supervised fine-tuning methodologies. It is widely understood that fine-tuning a 32B parameter LLM with only 100 data samples is generally insufficient to achieve meaningful adaptation. Consequently, the experimental results presented in Figure 5(a) may not adequately support the conclusions drawn in this section.\n\n**Answer:** Thanks for this valuable comments. We would like to answer from two perspectives.\n\n(1) The base model here is a 32B-Instruct model, which has coding capability at some levels. Therefore, training on 32B-Instruct models to learn to generate MAS code may have difference compared to training 32B-pretrained models to learn instruction-following capabilities.\n\n(2) Despite that Figure 5 (a) shows that using 100 training samples could significantly reduce the error rate of execution, it does NOT indicate that using 100 samples is sufficient. Please note that in Figure 5 (b), using 100 training samples results in significantly lower perfomance compared to using 10000 samples. That is, using 100 samples may successfully teach the LLM to generate executable MAS code, but fails to teach it to generate appropriate MAS, which is essential in ensuring good performance.\n\nWe truly believe in this new paradigm. With more diverse and high-performance MAS being included, we believe that MAS-GPT will be further advanced in a way similar to the advancement of ChatGPT: with better data and training techniques, the models become better.\n\n---\n\n&nbsp;\n\nOverall, we hope that our responses can fully address your concerns and will be grateful for any feedback.", "claims_and_evidence": "The claims in this paper are largely consistent with the experimental results and avoid overclaiming.", "methods_and_evaluation_criteria": "The method proposed in this paper, or more precisely, the overall pipeline for training MAS generator, effectively addresses the relevant problems and offers significant reference value.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "The experimental design in this paper is largely sound and effectively demonstrates the capabilities of MAS-GPT. However, several points require further clarification and consideration:\n\nLack of Single Agent Framework Specification: The paper does not specify the framework employed for the single-agent baseline models.\n\nCost Comparisons and Metric Justification: I have reservations regarding the cost comparison metric used in this experiment. To the best of my knowledge, standard practice in evaluating LLM cost involves assessing token cost and time cost to quantify both financial and temporal overheads. Therefore, I recommend that the authors augment this section by providing token cost and time cost data for MAS-GPT and all baselines across the benchmarks used. Furthermore, the methodology for performance measurement in this cost comparison section, as well as the source of the experimental data, are not clearly delineated. Please provide this essential supplementary information.\n\nScaling Effects of Data Size and Supervised Fine-tuning: The \"Scaling effects of data size\" analysis appears inconsistent with established supervised fine-tuning methodologies. It is widely understood that fine-tuning a 32B parameter LLM with only 100 data samples is generally insufficient to achieve meaningful adaptation. Consequently, the experimental results presented in Figure 5(a) may not adequately support the conclusions drawn in this section.", "relation_to_broader_scientific_literature": "This paper is insightful for building LLM-based multi-agent systems.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "See Experimental Designs Or Analyses.", "other_comments_or_suggestions": "No.", "questions_for_authors": "See Experimental Designs Or Analyses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper is concerned with constructing multi-agent systems of LLM agents in an automated way. The idea is to construct a system that can take in a user query and turn it into a multi-agent system that can then be simulated to accomplish some purpose. The authors achieve this by creating a dataset of multi-agent systems suitable for particular user queries. They then adapt their multi-agent system construction LLM by fine tuning on this dataset. The reviewers and I were all in agreement that this is a novel, important, and interesting paradigm. I have myself, in the context of my own work, wondered if such a thing would be possible so it's very nice to see that the authors have gone and done it. 

During the review process, a significant issue initially appeared to arise: several reviewers believed that the experimental evaluations in the paper were contaminated by an issue akin to 'training on the test set'. Different reviewers had different versions of this critique. However, it looks to me like the rebuttal responses contained convincing answers to all of them. It is my view that the authors were able to show in the rebuttal that their experimental results did not rely on training on the test set. This conclusion is based on my understanding of a few of the debates that arose between the authors and the reviewers. One debate hinges on the soundness of the authors' "shoulders of giants" argument. I am willing to accept this argument. It seems the real cause of the dispute with some reviewers on this point was more likely a poor explanation for this choice in the paper, so I would recommend the authors work on clarifying this logic in the main text. The other main debate hinged on whether zero-shot generalization was tested and whether the comparison to the baselines was like for like. I believe the authors adequately answered this concern in the rebuttal phase. They pointed specifically to the table in question containing the relevant results.
Decision: Accept (poster)