MAS-Zero: Designing Multi-Agent Systems with Zero Supervision

## Abstract

Abstract Multi-agent systems (MAS) leveraging the impressive capabilities of Large Language Models (LLMs) hold significant potential for tackling complex tasks. However, most current MAS depend on manually designed agent roles and communication protocols. These manual designs often fail to align with the underlying LLMs’ strengths and struggle to adapt to novel tasks.
Recent automatic MAS approaches attempt to mitigate these limitations but typically necessitate a validation set for tuning and yield static MAS designs lacking adaptability during inference. We introduce MAS-Zero , the first self-evolved, inference-time framework for automatic MAS design. MAS-Zero employs meta-level design to iteratively generate, evaluate, and refine MAS configurations tailored to each problem instance , without requiring a validation set. Critically, it enables dynamic agent composition and problem decomposition through meta-feedback on solvability and completeness. Experiments across math, graduate-level QA, and software engineering benchmarks, using both closed-source and open-source LLM backbones of varying sizes, demonstrate that MAS-Zero outperforms both manual and automatic MAS baselines, achieving a 7.44% average accuracy improvement over the next strongest baseline while maintaining cost-efficiency. These findings underscore the promise of meta-level self-evolved design for creating effective and adaptive MAS.

## 1 Introduction

While standalone large language models (LLMs) have demonstrated strong performance across numerous tasks deepseekr1; ke2025demystifying; schick2023toolformerlanguagemodelsteach; vu2024foundational, many problems remain too intricate for a single model to solve effectively wang2024rethinkingboundsllmreasoning; guo2024large. To tackle these challenges, the exploration of multi-agent systems (MAS) composed of multiple LLM agents has gained increasing traction among researchers ke2025surveyfrontiersllmreasoning.(^1^11Agents in a MAS can interact with external environmental tools (e.g., search tools jiang2024rag; shi2024generate) or collaborate with other agents to address tasks requiring diverse capabilities or multiple steps liang2023encouraging; chen2023reconcile. This work focuses on the latter scenario, where each agent within the MAS is an LLM communicating with other LLM agents.) These agents often assume distinct roles, such as generator or verifier reflexion_shinn2024, engage in debates offering varied perspectives qian2025scalinglargelanguagemodelbased; wang2024mixtureofagentsenhanceslargelanguage, and perform assigned subtasks li2025agentorientedplanningmultiagentsystems.

A fundamental challenge in MAS lies in designing an effective connection and configuration of these agents to solve a given problem. Initially, MAS were handcrafted, with humans designing both agent roles and inter-agent communication protocols. However, MAS composed entirely of such manually designed configurations have faced issues such as poor problem specification and inter-agent misalignment cemri2025multiagentllmsystemsfail; qiao2025benchmarkingagenticworkflowgeneration, especially when the MAS agents are not specifically trained with such configurations.

These shortcomings are understandable, as manually specifying agent roles, communication protocols, and problem decomposition is difficult when the human designer and the underlying LLMs are not well aligned. Moreover, manual approaches do not scale well to novel problems, especially as the problems become more complex. Recent work has explored automatic MAS design, often framing it as a pruning problem given a fixed and fully connected set of human designed MAS zhuge2024languageagentsoptimizablegraphs; zhang2024aflowautomatingagenticworkflow; zhang2025multiagentarchitecturesearchagentic or a generation problem, where the MAS is generated by a central *meta agent* that orchestrates system design hu2025automated; zhang2024aflowautomatingagenticworkflow.

However, these automatic MAS design systems have significant limitations. Most rely on a “training” phase with labeled validation sets to tune configurations, which are often unavailable in real-world scenarios and may not generalize. This training, based solely on outcome correctness, provides limited insight into the system’s internal dynamics. Furthermore, the training often yields a fixed architecture (i.e., one for the entire problem set) which lacks per-problem adaptability at test time.(^2^22zhang2025multiagentarchitecturesearchagentic uses sub-networks for per-problem adaptivity, but the core MAS structure remains fixed.) While these limitations might be less evident on simple tasks such as problems in GSM8K cobbe2021gsm8k
and HumanEval chen2021codex, commonly used for evaluation hu2025automated; zhang2025multiagentarchitecturesearchagentic; zhang2024aflowautomatingagenticworkflow. Figure [1](https://arxiv.org/html/2505.14996v2#S1.F1) demonstrates that existing systems falter on complex problems requiring multi-step planning and task decomposition. Alarmingly, many methods show little to no improvement over simple CoT with the base LLM, meaning the integrated system does not even outperform a single component.

Figure: Figure 1: Scatter plots comparing the Pareto fronts of various GPT-4o-based systems on three benchmarks. Manually designed MAS baselines are marked in purple and automatic MAS methods in blue. MAS-Zero is highlighted as an orange star. MAS-Zero delivers high performance at lower cost than comparable automatic MAS methods, establishing a new frontier for accuracy vs. cost trade-off.
Refer to caption: x1.png

To overcome these limitations, we argue that an effective MAS should satisfy two core desiderata: (1) automatic design of agent structures *aligned with the underlying LLM’s capabilities*; and (2) adaptivity to the specific problem. In this work, we propose a novel automatic inference-time MAS optimization framework, called MAS-Zero, which designs MAS with *zero* supervision, while satisfying all the aforementioned desiderata. In particular, MAS-Zero introduces a meta-agent that iteratively understands the limitations of individual agents and their combinations, and refines the MAS design accordingly. This process operates entirely at test time without relying on a validation set, allowing for unique MAS designs on per-problem level and overcoming the limited adaptivity of prior work.

To achieve this, MAS-Zero tasks a meta-agent with multiple roles. Figure [2](https://arxiv.org/html/2505.14996v2#S1.F2) illustrates a conceptual overview and contrasts MAS-Zero with both automatic and manual MAS designs. Specifically, MAS-Zero involves two key steps:

- •
Meta-iterations: The meta-agent iteratively performs two functions:
–
Decomposition of the given task and generation of a corresponding MAS implementation (as executable code), allowing the system’s structure to adapt dynamically to each new problem.
–
Evaluation the generated MAS design for informative meta-level feedback, assessing two key criteria: solvability (i.e., can the current MAS solve the problem?) and completeness (i.e., does the current MAS cover all necessary steps?).
Each iteration of this process produces a candidate solution to the problem.
- •
Self-verification: In this final step, the meta-agent selects the most suitable outcome from the set of all candidate solutions generated throughout the meta-iteration process.

Figure: Figure 2: Conceptual comparison of MAS-Zero, existing automatic and manual MAS designs. In manual MAS design, “P” indicates the problem. “C” denotes the CoT generator, “A” denotes the majority aggregator in CoT-SC, “J” denotes the final judge in a debate system, and “R” denotes the reflector in the self-refine system. MAS-Zero avoids tuning MAS configurations on supervisions from a validation set by iteratively designing and evaluating problem-specific MAS configurations at inference time.
Refer to caption: x2.png

Evaluations across three datasets covering diverse domains (math, graduate-level QA, and code) and multiple LLM backbones (including both closed- and open-source models of varying sizes, such as 32B, 70B, and GPT-4o)
show that our model outperforms manual and other automatic MAS baselines by a minimum of 7.44% on average. Furthermore, MAS-Zero consistently lies on the Pareto frontier of accuracy and cost (Figure [1](https://arxiv.org/html/2505.14996v2#S1.F1)),
effectively gaining a deeper understanding of both strengths and weaknesses of the current MAS design over iterations to enhance future designs.

MAS-Zero also provides insights into MAS design. For example, off-the-shelf LLMs, struggle to design effective MAS when used directly, but with MAS-Zero, the MAS performance improves significantly. Additionally, by utilizing a verifier instead of validation set tuning, MAS-Zero is poised to leverage future improvements in verification. Despite achieving state-of-the-art (SoTA) with a relatively weak self-verifier, our analysis indicates significant potential for further gains as stronger verifiers become available.

These findings suggest that it is promising to design MAS with zero supervision. The proposed self-evolved, inference-time method opens new avenues for research in automatic MAS optimization. In summary, our key contributions are:

- •
We introduce MAS-Zero, a new framework for automatic MAS design that, to our knowledge, for the first time, performs meta-design in a fully self-evolved way by learning from the behavior of the underlying LLM agents via inference-time, enabling dynamic, per-problem MAS construction without requiring a validation set.
- •
We present a new SoTA automatic MAS system that achieves substantial performance gains over both manually designed and strong automatic baselines, while also being cost-efficient and Pareto-optimal across multiple LLMs and domains.
- •
We evaluate MAS-Zero on various domains and LLMs, presenting key insights from the self-evolving process, such as the difficulty off-the-shelf LLMs face in designing MAS, how MAS support improves performance, and how verification amplifies this benefit.

## 2 Related Work

##### Manual MAS Design

Building on the success of single-agent systems (e.g., CoT wei2022chain, self-consistency (CoT-SC) wang2023selfconsistency), studies have shown that grouping multiple LLM agents into a MAS can substantially improve individual agent performance. To this end, a variety of human-designed MAS approaches have been proposed xu2025expertpromptinginstructinglargelanguage; zheng2024stepbackevokingreasoning; lu2025intelligentgoexplorestandingshoulders. Representative work includes LLM debate du2023improvingfactualityreasoninglanguage, and self-refine (or generate-evaluator systems) madaan2024self.
However, manually designed MAS are often limited in adaptability and scalability. More importantly, human designers typically do not fully understand the capabilities or limitations of the LLM agents, and may apply subjective preferences that do not align with optimal agent behavior. This makes effective MAS design highly challenging, if not impossible, without automation.

##### Automatic MAS Design

Some prior work treats prompt optimization for individual agents as part of MAS design. Examples include PromptBreeder fernando2024promptbreeder and DsPy khattab2023dspycompilingdeclarativelanguage.
More recently, this idea has been extended to broader automatic MAS design, where prompt optimization is included either as an additional design step or as part of the search space. We broadly categorize recent work on automatic MAS design into two families: pruning-based and generation-based approaches.

Pruning-based.
In this family, a pre-defined graph architecture is given, where each node typically represents a single LLM-based agent or a human-designed foundational building block (e.g., CoT), and each edge represents information flow, i.e., the output of the source node is fed into the input of the target node. The graph is usually fully connected at initialization and the main objective of the algorithm is to prune the graph to identify the most effective set of connections.
This line of work has evolved quickly zhang2024cutcrapeconomicalcommunication; zhang2025gdesignerarchitectingmultiagentcommunication; hu2024selfevolvingmultiagentcollaborationnetworks. Earlier examples include GPTSwarm zhuge2024languageagentsoptimizablegraphs which optimizes graph structures via reinforcement learning but struggles to represent workflows with conditional state due to limitations of static graphs. DyLAN liu2024dynamicllmpoweredagentnetwork uses message passing to dynamically activate agent compositions, estimating each agent’s contribution based on a primary trial, AgentSquare shang2024agentsquare leverages a verifier as a performance predictor to guide the pruning. More recent work includes MASS zhou2025multiagentdesignoptimizingagents uses rejection sampling based on validation set performance, while MaAS zhang2025multiagentarchitecturesearchagentic extends MASS with a question-wise masking mechanism, allowing different questions to activate different subnetworks. While these methods are typically easier to train, they are heavily constrained by the pre-defined structure, which is suboptimal for many tasks.

Generation-based. In this family, a meta-agent LLM generates MAS from scratch, offering greater flexibility in defining novel agents and architectures compared to pre-defined structures. However, this expanded design space presents significant learning challenges. Earlier methods like AutoAgents chen2024autoagentsframeworkautomaticagent and AgentVerse chen2023agentversefacilitatingmultiagentcollaboration relied on human-engineered pipelines. EvoAgent yuan2024evoagent later employed evolutionary algorithms to optimize these pipelines. More recent efforts, including ADAS hu2025automated and AFlow zhang2024aflowautomatingagenticworkflow, represent MAS directly as code, framing MAS generation as a code-generation task. ADAS stores and searches historical designs based on validation performance, while AFlow enhances this with Monte Carlo Tree Search. Our framework also adopts a code-based representation for MAS design. Critically, unlike prior work that uses a potentially unreliable validation set, MAS-Zero uses self-evolve approach
to infer agent capabilities for meta-level design at inference-time.

## 3 MAS-Zero Framework

Figure: Figure 3: MAS-Zero overview. The human defines the task by providing question and building blocks as the seed set of MAS. MAS-Zero solves the task by self-evolving and producing the final answer. MAS-Zero operates in two key steps: First, in the meta-iteration step, a meta-agent designs sub-tasks and their corresponding MAS networks, while also providing meta-level feedback on solvability and completeness. Second, the self-verification step involves the meta-agent verifying the final outcome from a set of candidate answers.
Refer to caption: x3.png

As shown in Figure [3](https://arxiv.org/html/2505.14996v2#S3.F3), MAS-Zero takes a question and a seed set of MAS building blocks as input, ultimately producing the final answer. These building blocks, which can be established human-designed blocks (e.g., CoT-SC wang2023selfconsistency) to define the design space, is processed by the central meta-agent. The meta-agent orchestrates both the meta-iteration (Sec. [3.1](https://arxiv.org/html/2505.14996v2#S3.SS1)) and self-verification (Sec. [3.2](https://arxiv.org/html/2505.14996v2#S3.SS2)) stages. Importantly, the meta-iteration is functioned without prior knowledge or internal details of the underlying LLM agents.
Figures [3](https://arxiv.org/html/2505.14996v2#S3.F3) and [4](https://arxiv.org/html/2505.14996v2#S3.F4) offer a visual overview and detailed illustrations of the prompts and workflow within MAS-Zero. Appendix [A](https://arxiv.org/html/2505.14996v2#A1) contains the pseudocode algorithm, and Appendix [B](https://arxiv.org/html/2505.14996v2#A2) provides a visualization of the generated MAS.

### 3.1 Meta-Iterations Step

Initially lacking knowledge of the underlying LLM agents’ internal capabilities or limitations, the meta-agent learns the component agents’ potential by
observing both system-level and finer component-level performance.
MAS-Zero achieves this through an iterative process: As shown in Figure [3](https://arxiv.org/html/2505.14996v2#S3.F3), each iteration comprises two main functions: (1) meta-design (Sec. [3.1.1](https://arxiv.org/html/2505.14996v2#S3.SS1.SSS1)), where the meta-agent decomposes the given question into sub-tasks
and proposes a MAS, based on the building blocks and any available feedback from prior iterations,
to address each sub-question; and (2) meta-feedback (Sec. [3.1.2](https://arxiv.org/html/2505.14996v2#S3.SS1.SSS2)), where the meta-agent reviews the proposed MAS and sub-tasks to generate feedback. Such feedback is based on intermediate outputs and reveals how the current MAS performs across various sub-question decompositions and configurations, acting as a valuable signal to guide future iterations.
Through the iterative cycle of meta-design and meta-feedback, the MAS is progressively refined to better solve the question (an example is provided in Appendix [B](https://arxiv.org/html/2505.14996v2#A2)).
In each round, the meta-agent can adjust the decomposition or MAS configurations (e.g., architecture, LLM agent temperature) based on the feedback received.

#### 3.1.1 Meta-Design Function

Rather than tackle complex problems all at once, MAS-Zero explicitly decomposes questions into sub-tasks and assigns specially designed MAS to solve each sub-question.
In this function, the meta-agent is responsible for decomposing the original question into manageable yet interdependent sub-tasks, then generating or assigning one (or more) agents, referred to as a sub-MAS, to solve each sub-question. During this function, the meta-agent not only decomposes questions, but specifies necessary coordination logic between agents, ensuring specialized yet aligned agents. While the quality sub-tasks and sub-MAS inherently depends on the capabilities of the underlying LLM-powered meta-agent, MAS-Zero is designed to iteratively evaluate and refine both the decomposition and corresponding sub-MAS designs, which can correct initially imperfect designs, as shown in Fig. [B.1](https://arxiv.org/html/2505.14996v2#A2.F1) (Appendix [B](https://arxiv.org/html/2505.14996v2#A2)).

Constraining the search space with building blocks.
A key component of MAS-Zero is an iterative approach that improves upon itself. As such, it is important for designs to not vary extremely between iterations. Rather than direct the meta-agent via prompting, MAS-Zero deliberately constrains the search space. Namely, MAS-Zero uses code structures as a representations for each agent. While prior work has explored generating wholesale MAS from code hu2024ADAS; zhang2024aflowautomatingagenticworkflow, our approach introduces a crucial constraint: Limiting changes to either updating questions or updating the sub-MAS with respect to the given building blocks, a set of fixed sub-MAS. After performing question decomposition, the meta-agent is restricted to modifying connections between different building blocks or adjusting building blocks’ parameters (e.g., temperature, number of debate rounds, etc.). This deliberate constraint, informed by our preliminary experiments, balances exploration with improvement: The meta-agent should not invent “new” agent roles that closely resemble existing ones, nor should it simply pruning a fixed architecture. Rather, it is given freedom to analyze and decompose questions then assign new seed-informed sub-MAS to solve subquestions. This work focuses on using CoT wei2022chain, Self-consistency (CoT-SC) wang2023selfconsistency, LLM debate du2023improvingfactualityreasoninglanguage, and Self-refine madaan2024self as the building blocks.

Code template and sanity check.
Generating correct code for MAS can be challenging. We provide a code template and utility functions so that the meta-agent only needs to fill in a specific forward function. We also perform sanity checks on the generated code, including syntax validation and field consistency. More details can be found in Appendix [E](https://arxiv.org/html/2505.14996v2#A5).

Figure [4](https://arxiv.org/html/2505.14996v2#S3.F4) gives an illustration. Given a question, the meta-agent is prompted to decompose the task and propose a MAS based on the question and building blocks (see Appendix [D](https://arxiv.org/html/2505.14996v2#A4) for the detailed prompts). The meta-agent then outputs a MAS in the form of code, which can be executed by an external compiler to obtain intermediate and final outputs for the sub-tasks and agents.

Figure: Figure 4: Example pipeline for the functions meta-design and meta-feedback in MAS-Zero.
Refer to caption: x4.png

#### 3.1.2 Meta-Feedback Function

A key challenge in MAS design is that the meta-agent only has access down to the agent-level: It does not have access to an agent’s internal memory or knowledge. Unlike conventional evaluation strategies that rely only on final outcome signals, in MAS-Zero, we propose to use system-level and agent-level signals, i.e., the inputs and outputs to sub-MAS and agents, as proxies for agent capabilities and limitations. By determining how effective the current MAS is by scrutinizing interactions between agents and sub-MAS components, the meta-agent obtains a richer look into both strengths and weaknesses in the overall MAS. To facilitate this, the meta-agent must first obtain intermediate MAS outputs, from which it can perform this fine-grained, intermediate evaluation. MAS-Zero then centers two key criteria: solvability (whether the generated sub-tasks are sufficient to reach the final answer to the original question) and completeness (whether the MAS is capable of solving these sub-tasks). The feedback resulting from these criteria is more actionable than feedback from the final outcome, as it directly informs whether the decomposition should be refined or the MAS should be made more sophisticated to handle the assigned sub-tasks.

Obtaining intermediate output.
To obtain such fine-grained feedback, we execute the MAS. This involves solving each subquestion by executing each sub-MAS, which itself is comprised of one or more agents, producing a candidate answer to the original question. MAS execution grants the meta-agent access to two levels of intermediate input-output pairs: One at the sub-question (sub-MAS) level, which we denote $\{(x_{\text{sub}},y_{\text{sub}})\}$, and one at the agent-level, which we denote $\{(x_{\text{agent}},y_{\text{agent}})\}$.

Solvability and completeness as meta-reward.
Given the sub-question samples and agent samples, MAS-Zero evaluates two key metrics: solvability and completeness. In our self-evolved framework, the meta-agent is given agency in determining each metric, which serve as valuable meta-rewards that guide the meta-agent in improving the MAS design.

- •
Solvability requires that each sub-question be independently and completely solvable by its corresponding sub-MAS. This ensures that every sub-question yields reliable outputs. If the meta-agent determines a sub-question not solvable, the meta-agent must either modify the sub-MAS to better solve the sub-problem, or further decompose the sub-problem into easier steps. Since MAS-Zero operates in a self-evolved setting, the meta-agent itself is responsible for assessing solvability. To aid the meta-agent, we allow each agent to output a special token, [TOO HARD], if it determines that the assigned sub-question is beyond its current capabilities. The ability to self-supervise at the meta-agent level and at the agent level (in recognizing a problem is too difficult via [TOO HARD]) ultimately depends on the ability of the underlying LLMs to recognize and abstain, which may be imperfect. However, we show via ablations in Sec. [4.2](https://arxiv.org/html/2505.14996v2#S4.SS2) that such signals serve as reasonable proxies for solvability and contribute meaningfully to overall performance.
- •
Completeness requires that the complete set of sub-tasks covers all necessary information from the original input, ensuring that the answers to the sub-tasks can produce a correct and comprehensive aggregated answer to the original task. While an individual sub-question may address only part of the necessary content, all critical information must processed and used at some point in the MAS.

Generating feedback.
Based on the meta-rewards of solvability and completeness, the meta-agent then generates targeted feedback for specific aspects of the MAS that may require revision. If the meta-agent identifies a sub-question as not solvable, then during the next Meta-Design function, it can either further decompose the sub-question or update the corresponding sub-MAS; solvable sub-tasks and corresponding sub-MAS are left untouched. If the meta-agent finds that the union of sub-tasks misses necessary information, it can refine how the original problem is decomposed to incorporate any identified missing information. Overall, this feedback serves a guide for subsequent Meta-Design functions, allowing the overall system to iteratively converge toward an effective decomposition and MAS. Crucially, this entire process occurs without any evaluation of the final answer produced when executing the MAS, relying only on analysis of intermediate outputs.

In Figure [4](https://arxiv.org/html/2505.14996v2#S3.F4), we can see that after performing meta-decomposition and executing the MAS code via a compiler, both the resulting MAS and its intermediate outputs (sub-question samples and agent samples) are used as input to the meta-feedback function. We prompted the meta-agent to review the solvability and completeness. Based on this evaluation, which serves as meta-reward, the meta-agent generates feedback and updates its design in the next iteration.

### 3.2 Self-Verification Step

At each meta-iteration, the MAS is executed to obtain intermediate output information. A byproduct of this execution is a candidate answer. After multiple rounds of meta-iterations, MAS-Zero must determine which candidate answer is the most reliable and complete. Relying on the last iteration (or any single iteration) may be suboptimal, as the MAS evolves through varying levels of agent understanding, per-instance adaptivity, or general stochasticity. Instead, MAS-Zero formulates verification as a selection problem and tasks the meta-agent with selecting the most coherent and correct output from the set of candidate answers.

To achieve this, MAS-Zero collects the CoTs and final answers produced by each candidate MAS, and applies a structured verification procedure: It
first ranks the candidate answers based on their frequency across iterations. This acts as a prior favoring majority responses, a strategy shown to be effective in prior work wang2023selfconsistency. It then filters out clearly invalid answers (e.g., options not present in a multiple-choice question). Finally, the meta-agent selects the best answer from the remaining candidate CoTs and answers. This strategy of treating verification as a multiple-choice selection over a finite candidate set, is often more tractable than independently scoring each output gu2025surveyllmasajudge; zhou2025evaluating,
especially for challenging questions where correctness is hard to assess in isolation.

## 4 Experiments

Setup.
We consider both the closed-source GPT-4o gpt4o and the open-source LLMs of various sizes,
Llama3.3-70B grattafiori2024llama3herdmodels and Qwen2.5-32B qwen2025qwen25technicalreport. To fairly evaluate how well MAS-Zero performs relative to the underlying LLM used to construct the MAS system, we always use the same LLM for both the meta-agent and the individual LLM agents in the MAS.

Benchmarks.
We consider three benchmarks across various domains: AIME24 aime2024 in the math domain, GPQA-diamond (GPQA) rein2023gpqagraduatelevelgoogleproofqa in the graduate-level QA domain, and SWE-Bench-Lite-Oracle (SWE) jimenez2024swebench in the code domain. To fairly compare with baselines that rely on validation sets, we split each benchmark’s original test set into 20% for validation and 80% for testing. For manual MAS baselines and MAS-Zero, which do not rely on the validation set, we evaluate on the same 80% split.
Note that self-verification does not apply to SWE, as correctness in SWE is determined directly by the compiler. Detailed statistics and implementations can be found in Appendix [C](https://arxiv.org/html/2505.14996v2#A3).

Baselines.
We include four widely used manual MAS baselines: CoT wei2022chain, self-consistency (CoT-SC) wang2023selfconsistency, debate du2023improvingfactualityreasoninglanguage and generator-evaluator (self-refine) madaan2024self. These four also serve as the building blocks designs in MAS-Zero, allowing us to clearly observe how MAS-Zero improves upon the initial configurations. In addition, we include the SoTA automatic MAS design methods, including pruning-based methods MaAS zhang2025multiagentarchitecturesearchagentic and AFlow zhang2024aflowautomatingagenticworkflow and generation-based method ADAS hu2025automated. These methods have been shown to outperform other automatic MAS approaches significantly,
making them strong baselines for comparison.

Implementation details.
For MAS-Zero, we first execute each of the four building blocks designs before conducting 5 rounds of iterative refinement. This yields 9 candidate answers, from which our meta-agent selects one final answer. For fair comparison, we sample 9 independent outputs for CoT-SC and take the majority vote.
Similarly, both debate and self-refine are run for 9 rounds. All models are accessed through their respective APIs.(^3^33We use TogetherAI API ([https://www.together.ai/](https://www.together.ai/)) for Llama and Qwen.)
Temperature for meta-agent is set to 0.5. For baselines, we strictly use parameters found in original papers and provided code.

**Table 1: Overall results. “Avg.” denotes the weighted average, where weights are based on the number of samples across benchmarks. “—” denotes non-applicable (CoT-SC is not applicable to SWE). “$\times$” indicates ADAS achieved 0% accuracy, despite being tuned on the validation set. “${\color[rgb]{0,0.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,0}\uparrow}$” indicate the difference (improvement) compared to MAS-Zero. Highlighting indicates manual MAS design, automatic MAS design, and our method. Note that SWE is not included for Qwen2.5-32B due to its small maximum context length (32K). We report the average of three runs for experiments, following zhang2024aflowautomatingagenticworkflow; liu2024dynamicllmpoweredagentnetwork.**
| LLMs | GPT-4o | Llama3.3-70B | Qwen2.5-32B |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | AIME24 | GPQA | SWE | Avg. | AIME24 | GPQA | SWE | Avg. | AIME24 | GPQA | Avg. |
| CoT | 8.33 | 45.78 | 9.17 | 23.26 (↑ 12.55) | 16.67 | 50.60 | 2.92 | 22.09 (↑ 9.58) | 12.50 | 50.00 | 45.26 (↑ 3.69) |
| Debate | 4.17 | 46.99 | 12.50 | 25.35 (↑ 10.46) | 20.83 | 50.60 | 6.67 | 24.42 (↑ 7.25) | 8.33 | 49.40 | 44.21 (↑ 4.74) |
| CoT-SC | 16.67 | 43.37 | — | — | 29.17 | 51.20 | — | — | 16.67 | 49.40 | 45.27 (↑ 3.68) |
| Self-Refine | 4.17 | 46.39 | 11.67 | 24.65 (↑ 11.16) | 29.17 | 54.22 | 1.67 | 23.49 (↑ 8.18) | 16.67 | 50.60 | 46.31 (↑ 2.64) |
| MaAS | 12.50 | 43.37 | 10.00 | 23.02 (↑ 12.79) | 33.33 | 43.98 | 5.00 | 21.63 (↑ 10.04) | 20.83 | 46.99 | 43.68 (↑ 5.27) |
| AFlow | 20.83 | 46.99 | 16.25 | 28.37 (↑ 7.44) | 33.33 | 47.59 | 6.67 | 23.95 (↑ 7.72) | 33.33 | 48.80 | 46.84 (↑ 2.11) |
| \cdashline1-12<br>ADAS | $\times$ | 45.20 | $\times$ | $\times$ | 8.30 | 53.60 | $\times$ | $\times$ | 12.50 | 47.00 | 42.64 (↑ 6.31) |
| \cdashline1-12<br>MAS-Zero | 33.33 | 50.60 | 25.83 | 35.81 | 37.50 | 52.41 | 16.74 | 31.67 | 29.17 | 51.81 | 48.95 |

### 4.1 Overall Results

Performance.
Table [1](https://arxiv.org/html/2505.14996v2#S4.T1) shows the overall results across all LLMs and benchmarks. On average, MAS-Zero achieves the best performance across all LLMs and domains. Below, we summarize the additional takeaways from the comparison:

$\bullet$ MAS-Zero consistently outperforms all automatic MAS methods.
Across all LLM backbones and benchmarks, MAS-Zero outperforms SoTA baselines, beating the next best method, AFlow, by 7.44% in on average with GPT-4o as the backbone. The most sizable improvements come SWE, with 58% and 149% relative gains over AFlow. The only instance where MAS-Zero lags is on AIME24 with the Qwen backbone, where it underperforms AFLow by one sample (out of 24 total). Notably, we find that the generation-based ADAS is unable to perform certain tasks (0% accuracy), despite being given access to a validation set.
Overall, our results highlight the benefit of incorporating meta-design and the ability to adapt at inference time, the two capabilities absent in prior automatic MAS approaches.

$\bullet$ MAS-Zero outperforms all manually designed MAS baselines, while others do not.
MAS-Zero is the only method that always outperforms CoT and reliably outperforms other manually designed baselines. Because these manual MAS are also used as the building blocks for MAS-Zero, our results demonstrate that the meta-iteration and self-verification mechanisms effectively improve upon the initial human-designed setups. One caveat is the self-refine baseline on GPQA with Llama3.3, where correct candidates were generated but ultimately not selected, revealing weaknesses in the Llama backbone’s verification ability. AFlow, ADAS, and MaAS alarmingly all underperform the simple CoT baseline on multiple benchmarks, with both ADAS and MaAS on average exhibiting degradation from CoT performance. AFlow fares slightly better, but has relatively muted gains when compared to MAS-Zero. These results suggest that current automatic MAS methods struggle on challenging tasks where MAS-Zero offers substantive improvements.

**Table 2: Ablations on removing decomposition and meta-feedback with GPT-4o.**
| Methods | AIME24 | GPQA | SWE | Avg. |
| --- | --- | --- | --- | --- |
| MAS-Zero | 33.33 | 50.60 | 25.83 | 35.81 |
| - decompose | 20.83↓12.5 | 45.18↓5.4 | 23.75↓2.1 | 31.86↓4.0 |
| - meta-reward | 25.00↓8.3 | 42.17↓8.4 | 23.33↓2.5 | 30.70↓5.1 |

Cost-efficiency.
Figure [1](https://arxiv.org/html/2505.14996v2#S1.F1) shows the trade-off between performance and cost for GPT-4o across the three benchmarks. Cost is estimated using the latest OpenAI API pricing(^4^44[https://openai.com/api/pricing/](https://openai.com/api/pricing/)) and includes both “training” (if any) and test-time usage. We observe that MAS-Zero lies on the Pareto front across all three datasets. It is significantly more cost-efficient than AFlow, MaAS, and ADAS, with the lone exception of ADAS on GPQA, where the cost increase comes with a 12% accuracy improvement. Of automatic MAS frameworks, MAS-Zero delivers the highest performance at relatively low cost. While it is expected that automatic MAS methods incur higher costs than manual baselines, MAS-Zero delivers substantially better performance, making the trade-off highly favorable.

Figure: Figure 5: Performance gains (purple) over base performance (light blue) given an oracle verifier. Automatic MAS baselines (ADAS, AFlow, and MaAS) cannot integrate external verifiers, yielding zero improvement.
Refer to caption: x5.png

### 4.2 Ablation Study and Further Analysis

Ablations. To understand the contribution of individual components in MAS-Zero, we conduct ablations on two key parts of the meta-iteration step (other components are not independently removable): (1) problem decomposition (MAS-Zero (-decompose)) in the meta-design function. Here, we modify the prompt to ask the meta-agent to propose a MAS configuration without attempting to decompose the question into sub-problems; (2) meta-reward (MAS-Zero (-meta-reward)) in the meta-feedback function. We alter the prompt so that the meta-agent critiques the current MAS without analyzing the solvability and completeness of each sub-question or LLM agent.

Table [2](https://arxiv.org/html/2505.14996v2#S4.T2) presents the ablation results. We observe that removing either component leads to a significant drop in performance, indicating that both the problem decomposition and meta-reward mechanisms are critical to the overall effectiveness of MAS-Zero.

Figure: Figure 6: Performance vs. meta-iterations with GPT-4o.
Refer to caption: x6.png

Establishing a performance upper bound.
MAS-Zero eschews validation-set tuning for self-verification, tasking the meta-agent with selecting a final answer from the set of candidate responses. However, MAS-Zero can easily accommodate external verifiers. This versatility sets MAS-Zero up to take advantage of future improvements in verification, unlike existing automatic MAS frameworks, which have no straightforward way of integrating a verifier. Fig. [5](https://arxiv.org/html/2505.14996v2#S4.F5) illustrates these potential future gains. Specifically, when given access to an oracle verifier that outputs outputs “correct” or “incorrect” based on the ground-truth answer, the performance gap between MAS-Zero and both manually designed and automatic MAS widens, with GPQA performance approaching 95%.

Gains from meta-iteration.
We investigate whether performance improves across meta-iteration rounds. Figure [6](https://arxiv.org/html/2505.14996v2#S4.F6) shows how performance changes with the number of iterations. Overall, we observe a consistently increasing trend, indicating that the meta-iteration is beneficial. These results suggest that off-the-shelf LLMs struggle to design effective MAS when used directly. However, with our proposed inference-time self-evolved method, MAS-Zero, performance improves over meta-iterations due to its ability to understand the capabilities of the LLM, decompose problems, and adapt to each question.

## 5 Conclusion

We introduced MAS-Zero, a novel framework for automatic MAS design that performs meta-level design in a self-evolved manner at inference-time.
Unlike prior approaches that rely on fixed architectures or tuning on validation sets, MAS-Zero iteratively constructs and refines MAS configurations for each input using a meta-agent that observes and learns from its own execution traces.
Experiments across diverse benchmarks and LLM backbones show that MAS-Zero outperforms both manual and state-of-the-art automatic MAS baselines, while remaining cost-efficient.
These results highlight the promise of MAS-Zero, but also reveal limitation: MAS-Zero relies on self-evolved and inference-time signals, which occasionally result in noisy feedback. While iterative refinement helps mitigate this, future work should focus on improving verification mechanisms and/or meta-agent specific-training for meta-design and meta-feedback.

## References

- (1)
Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica.
Why do multi-agent llm systems fail?, 2025.
- (2)
Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F. Karlsson, Jie Fu, and Yemin Shi.
Autoagents: A framework for automatic agent generation, 2024.
- (3)
Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal.
Reconcile: Round-table conference improves reasoning via consensus among diverse llms.
arXiv preprint arXiv:2309.13007, 2023.
- (4)
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.
Evaluating large language models trained on code.
2021.
- (5)
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou.
Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors, 2023.
- (6)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.
Training verifiers to solve math word problems.
arXiv preprint arXiv:2110.14168, 2021.
- (7)
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong
Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu,
Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
- (8)
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch.
Improving factuality and reasoning in language models through multiagent debate, 2023.
- (9)
Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel.
Promptbreeder: Self-referential self-improvement via prompt evolution.
In Forty-first International Conference on Machine Learning, 2024.
- (10)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang,
Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur,
Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher,
Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh
Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna
Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim
Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan,
Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi
Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma.
The llama 3 herd of models, 2024.
- (11)
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo.
A survey on llm-as-a-judge, 2025.
- (12)
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang.
Large language model based multi-agents: A survey of progress and challenges.
arXiv preprint arXiv:2402.01680, 2024.
- (13)
Shengran Hu, Cong Lu, and Jeff Clune.
Automated design of agentic systems.
arXiv preprint arXiv:2408.08435, 2024.
- (14)
Shengran Hu, Cong Lu, and Jeff Clune.
Automated design of agentic systems.
In The Thirteenth International Conference on Learning Representations, 2025.
- (15)
Yue Hu, Yuzhu Cai, Yaxin Du, Xinyu Zhu, Xiangrui Liu, Zijie Yu, Yuchen Hou, Shuo Tang, and Siheng Chen.
Self-evolving multi-agent collaboration networks for software development, 2024.
- (16)
Jinhao Jiang, Jiayi Chen, Junyi Li, Ruiyang Ren, Shijie Wang, Wayne Xin Zhao, Yang Song, and Tao Zhang.
Rag-star: Enhancing deliberative reasoning with retrieval augmented verification and refinement.
arXiv preprint arXiv:2412.12881, 2024.
- (17)
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan.
SWE-bench: Can language models resolve real-world github issues?
In The Twelfth International Conference on Learning Representations, 2024.
- (18)
Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, Caiming Xiong, and Shafiq Joty.
A survey of frontiers in llm reasoning: Inference scaling, learning to reason, and agentic systems, 2025.
- (19)
Zixuan Ke, Yifei Ming, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty.
Demystifying domain-adaptive post-training for financial llms.
arXiv preprint arXiv:2501.04961, 2025.
- (20)
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts.
Dspy: Compiling declarative language model calls into self-improving pipelines, 2023.
- (21)
Ao Li, Yuexiang Xie, Songze Li, Fugee Tsung, Bolin Ding, and Yaliang Li.
Agent-oriented planning in multi-agent systems, 2025.
- (22)
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu.
Encouraging divergent thinking in large language models through multi-agent debate.
arXiv preprint arXiv:2305.19118, 2023.
- (23)
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang.
A dynamic llm-powered agent network for task-oriented agent collaboration, 2024.
- (24)
Cong Lu, Shengran Hu, and Jeff Clune.
Intelligent go-explore: Standing on the shoulders of giant foundation models, 2025.
- (25)
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al.
Self-refine: Iterative refinement with self-feedback.
Advances in Neural Information Processing Systems, 36, 2024.
- (26)
OpenAI.
Gpt-4 technical report.
arXiv preprint arXiv:2303.08774, 2023.
- (27)
Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun.
Scaling large language model-based multi-agent collaboration, 2025.
- (28)
Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen.
Benchmarking agentic workflow generation, 2025.
- (29)
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report, 2025.
- (30)
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman.
Gpqa: A graduate-level google-proof qa benchmark, 2023.
- (31)
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
Toolformer: Language models can teach themselves to use tools, 2023.
- (32)
Yu Shang, Yu Li, Keyu Zhao, Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li.
Agentsquare: Automatic llm agent search in modular design space.
arXiv preprint arXiv:2410.06153, 2024.
- (33)
Zhengliang Shi, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, and Zhaochun Ren.
Generate-then-ground in retrieval-augmented generation for multi-hop question answering.
arXiv preprint arXiv:2406.14891, 2024.
- (34)
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.
Reflexion: Language agents with verbal reinforcement learning.
Advances in Neural Information Processing Systems, 36, 2024.
- (35)
AIME Team.
Aime problems and solutions, 2024.
- (36)
Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung.
Foundational autoraters: Taming large language models for better automatic evaluation.
arXiv preprint arXiv:2407.10817, 2024.
- (37)
Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou.
Mixture-of-agents enhances large language model capabilities, 2024.
- (38)
Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song.
Rethinking the bounds of llm reasoning: Are multi-agent discussions the key?, 2024.
- (39)
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.
Self-consistency improves chain of thought reasoning in language models.
In The Eleventh International Conference on Learning Representations, 2023.
- (40)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
Advances in neural information processing systems, 35:24824–24837, 2022.
- (41)
Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao.
Expertprompting: Instructing large language models to be distinguished experts, 2025.
- (42)
Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Dongsheng Li, and Deqing Yang.
Evoagent: Towards automatic multi-agent generation via evolutionary algorithms.
arXiv preprint arXiv:2406.14228, 2024.
- (43)
Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang.
Multi-agent architecture search via agentic supernet, 2025.
- (44)
Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen.
Cut the crap: An economical communication pipeline for llm-based multi-agent systems, 2024.
- (45)
Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, Tianlong Chen, and Dawei Cheng.
G-designer: Architecting multi-agent communication topologies via graph neural networks, 2025.
- (46)
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu.
Aflow: Automating agentic workflow generation, 2024.
- (47)
Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, and Denny Zhou.
Take a step back: Evoking reasoning via abstraction in large language models, 2024.
- (48)
Han Zhou, Xingchen Wan, Ruoxi Sun, Hamid Palangi, Shariq Iqbal, Ivan Vulić, Anna Korhonen, and Sercan Ö. Arık.
Multi-agent design: Optimizing agents with better prompts and topologies, 2025.
- (49)
Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, and Shafiq Joty.
Evaluating judges as evaluators: The jetts benchmark of llm-as-judges as test-time scaling evaluators.
arXiv preprint arXiv:2504.15253, 2025.
- (50)
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber.
Gptswarm: Language agents as optimizable graphs, 2024.

## Appendix A MAS-Zero Algorithm

In Section [3](https://arxiv.org/html/2505.14996v2#S3), we provide examples of the prompt and workflow of MAS-Zero. Algorithm [1](https://arxiv.org/html/2505.14996v2#alg1) presents the detailed algorithm. Note that we merge meta-feedback and meta-design in meta-iterations into a single Meta_Feedback_Update function in the implementation for simplicity.

Figure: Algorithm 1 MAS-Zero: Self-Design Multi-Agent System

## Appendix B Visualization of Resulting MAS

Figure: Figure B.1: Visualization of the resulting MAS from MAS-Zero. “P” indicates the problem, while “$P_{i}$” indicates sub-question $i$. “C” denotes the CoT generator, “A” denotes the majority aggregator in CoT-SC, “J” denotes the final judge in a debate system, and “R” denotes the reflector in the self-refine system (the same LLM as the CoT generator).
Refer to caption: x7.png

We visualize one example question from AIME24 using GPT-4o in Figure [B](https://arxiv.org/html/2505.14996v2#A2). We present naive CoT and two selected meta-iterations (the first and the last) to illustrate how the meta-agent refines the task decomposition over time.

We can observe that the naive CoT approach fails to break down the problem into well-defined sub-problem, but jumps directly from a vague geometric interpretation to a numerical conclusion. For example, assuming without justification that ‘The sum of the lengths of the triangle formed by the extensions of the sides of the hexagon is equal to the perimeter of the hexagon.’.

In contrast, MAS-Zero decomposes the problem more deeply. While meta-iteration 1 tried to decompose the task, it is still very vague and difficult to solve. After 4 rounds of meta-iterations, in meta-iteration 5, we can observe that MAS-Zero improved the decomposition by including useful hints in the sub-questions, such as angle analysis, symmetry, and vector geometry. These additions make the sub-tasks more solvable and provide better guidance for the MAS. Notable, the answer for $P_{3}$ can directly capture relations between ‘The sum of the lengths of the triangle formed by the
extensions of the sides of the hexagon‘ and ‘the perimeter of the hexagon’.

MAS-Zero also proposes using multiple building blocks such as CoT, CoT-SC, and debate, to solve different sub-questions. Since the entire process is designed by the meta-agent based on self-supervision signals on solvability and completeness, the resulting MAS is more aligned with the behavior of the underlying LLM. Moreover, as an inference-time method, it is adaptive to each individual problem and scalable to novel tasks.

## Appendix C Benchmarks and Baselines Details

##### Benchmarks

Table [C.1](https://arxiv.org/html/2505.14996v2#A3.T1) shows the detailed statistics for each dataset. We evaluate SWE using its official code available at [https://github.com/SWE-bench/SWE-bench/](https://github.com/SWE-bench/SWE-bench/).

##### Baselines

For the manual MAS baselines, we adapt the implementation from ADAS. For the automatic MAS baselines, we use the official implementations of ADAS ([https://github.com/ShengranHu/ADAS](https://github.com/ShengranHu/ADAS)), AFlow ([https://github.com/FoundationAgents/MetaGPT/tree/main/examples/aflow](https://github.com/FoundationAgents/MetaGPT/tree/main/examples/aflow)), and MaAS ([https://github.com/bingreeky/MaAS](https://github.com/bingreeky/MaAS)).

**Table C.1: Data size for each split in each dataset.**
| Split | AIME24 | GPQA | SWE |
| --- | --- | --- | --- |
| Validation | 6 | 32 | 60 |
| Test | 24 | 166 | 240 |

## Appendix D Prompt Details

In Section [3.1](https://arxiv.org/html/2505.14996v2#S3.SS1) and Figure [4](https://arxiv.org/html/2505.14996v2#S3.F4), we use prompts to implement meta-design and meta-feedback. Here, we provide more details of the prompts in Figures [D.1](https://arxiv.org/html/2505.14996v2#A4.F1) and [D.2](https://arxiv.org/html/2505.14996v2#A4.F2). As mentioned in Appendix [A](https://arxiv.org/html/2505.14996v2#A1), we merge meta-feedback and meta-design in meta-iterations into a single Meta_Feedback_Update function. As a result, there are two sets of prompts: one for the initial Meta_Design, and another for Meta_Feedback_Update used in the meta-iterations.

Figure: Figure D.1: Simplified prompt for Meta_Design.

Figure: Figure D.2: Simplified prompt for Meta_Feedback_Update.

## Appendix E Code Template

In Section [3.1](https://arxiv.org/html/2505.14996v2#S3.SS1), we use a code template to constrain MAS code generation to filling in a specific forward function. Figure [E.1](https://arxiv.org/html/2505.14996v2#A5.F1) shows how the utility code is provided. Figures [E.2](https://arxiv.org/html/2505.14996v2#A5.F2), [E.3](https://arxiv.org/html/2505.14996v2#A5.F3), [E.4](https://arxiv.org/html/2505.14996v2#A5.F4), and [E.5](https://arxiv.org/html/2505.14996v2#A5.F5) show the implementations of each seed MAS.

Figure: Figure E.1: Utility code.

Figure: Figure E.2: Implementation of seed MAS (CoT)

Figure: Figure E.3: Implementation of seed MAS (CoT-SC)

Figure: Figure E.4: Implementation of seed MAS (debate)

Figure: Figure E.5: Implementation of seed MAS (self-refine)

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: - In Table 1, how do you control the input/output token budget to be the same? 
- How did you implement the baselines? 
- In Figure 1, is it inference cost or optimization cost? If it's one of these two, what's the other?
soundness: 2
strengths: - Comprehensive baselines.
- Relatively clear presentation.
summary: The work proposes an optimization framework that optimizes a meta-agent which initializes and designs the multi-agent framework. It gathers experiences during the training time and can directly generate well-performing multi-agent during the inference time. It's evaluated on two reasoning tasks, compared against broad baselines, and shown to achieve improved performance and higher cost-efficiency.
weaknesses: - The system seems to need quite some manual design, for example, the modules in the `MAS-INIT`, and prompts in Appendix J. It's not clear how the system is robust w.r.t these manual design and how much replies on human instructions.
- Following the above reason, the novelty might be lacking because the lack of automation.
- Some implementation details to ensure fair comparison is not clear, see questions.
- The title doesn't seem to be precise? Specifically, "ZERO SUPERVISION" holds in the sense that " support adaptivity at inference time, so that MAS designs can betailored per problem instance without relying on training or validation sets. ", but it still requires supervision during meta-agent training.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. The cost-efficiency analysis (Fig. 1) is useful. However, for a practical application, what is the inference-time latency comparison? MAS-ZERO runs 9 total candidate generations (4 init + 5 evolve). How does this end-to-end time/cost compare to a single inference pass from a baseline like AFlow or MAS-GPT on the same test problem?

2. Figure 4B shows a massive performance leap with an oracle verifier, implying the MAS-Verify step is a significant bottleneck. Since MAS-Verify is also an LLM call, have the authors experimented with using a stronger, dedicated model (e.g., O3) only for the final verification step, even when the agents and meta-agent are weaker (e.g., GPT-4o)?

3. The system is initialized with four building blocks (CoT, CoT-SC, Debate, Self-Refine). How sensitive is the final performance to this specific set? For instance, what happens if it is only given CoT and Debate? Does the "MAS-Evolve" step successfully reinvent a Self-Refine-like process, or is its creativity constrained to what it is initially given?
soundness: 2
strengths: 1. The framework's primary strength is its ability to adapt its design to each specific problem instance at inference time, without the need for a pre-tuned configuration or a validation set. This is a compelling alternative to static, validation-set-tuned systems.

2. The expanded ablation studies in Section 4.2 provide a clear and valuable breakdown of the system's performance. The ablations on MAS-Init, MAS-Evolve, meta-design, and meta-feedback effectively demonstrate that all components contribute meaningfully to the final result.

3. The method shows consistent and sometimes substantial improvements over strong baselines across multiple challenging domains (AIME24, SWE-Bench), demonstrating the practical effectiveness of the self-evolving design.
summary: The authors propose MAS-ZERO, an inference-time-only framework for automatically designing Multi-Agent Systems (MAS) without supervision or validation sets. The core idea is to use a "meta-agent" that operates in a three-stage process: 1) MAS-Init, which runs a set of predefined "building block" strategies (like CoT, Debate) to gather initial solutions; 2) MAS-Evolve, an iterative loop where the meta-agent decomposes the problem, designs a novel MAS by composing the building blocks, executes it, and then critiques its own design based on "solvability" and "completeness" feedback; 3) MAS-Verify, which selects the best final answer from the pool of candidates generated by both MAS-Init and MAS-Evolve. The authors demonstrate that this self-evolving, per-instance approach outperforms both manual and prior automatic MAS baselines on tasks in reasoning, coding, and agentic benchmarks.
weaknesses: 1. While the empirical results are now much stronger, the core conceptual framework can be viewed as a very sophisticated and well-executed combination of existing ideas (problem decomposition, self-refinement loops, agent routing) orchestrated via prompt engineering. The "meta-design" component, while effective, is fundamentally a well-crafted heuristic rather than a wholly new paradigm.

2. The entire system's success hinges on the reasoning capability of the meta-agent to correctly decompose problems and accurately critique its own designs. The paper acknowledges this the ablations and show the components are necessary, but the system is still vulnerable to meta-agent flaws. A failure in meta-feedback could send the iterative process in a useless direction.

3. For Table 5, the current experiment only swaps the meta-agent or the agents. A more powerful extension, which the framework seems suited for, would be for the meta-agent to dynamically assign sub-tasks to different agent models based on their known strengths (e.g., assign a reasoning-focused sub-task to o3-mini and a coding sub-task to a coder model), rather than using a homogeneous pool of agents.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: Address my weeknesses.
soundness: 3
strengths: 1. Originality: a new zero-supervision, inference-time self-evolving MAS framework, breaking validation set dependence and supporting dynamic task decomposition/complexity switching.
2. Quality: Comprehensive experiments across domains/models, comparisons with 11 baselines, ablation studies validating core modules, and planned open-sourcing ensuring reproducibility.
3. Clarity: Clearly describes the framework (3 key steps) and details (code templates, prompts), with complete appendices reducing understanding barriers.
4. Significance: Outperforms baselines on high-difficulty tasks (AIME24, SWE) and reveals the new insight that "simpler systems excel in some scenarios," with practical and guiding value.
summary: This work proposes MAS-ZERO, a self-evolved, inference-time framework that requires no validation set and can iteratively optimize Multi-Agent System (MAS) configurations through meta-level design, which enabling dynamic task decomposition and agent composition, simplifies the system when appropriate, and outperforms both manual and existing automatic MAS baselines across multiple tasks and different Large Language Models (LLMs).
weaknesses: 1. From my understanding, this paper focuses on prompt engineering and involves no model training. Its performance upper bound is constrained by the capabilities of the underlying model, and its effectiveness bears similarities to test-time scaling. I consider its contributions limited, that is, had the authors proposed a training paradigm or a data framework to explore ways of pushing the model’s performance ceiling, the work would have been far more impactful.  
2. Drawing parallels to test-time scaling, increasing the inference budget will undoubtedly yield performance gains. I believe presenting a table that quantifies the trade-off between effectiveness and efficiency (e.g., latency, token consumption) would be highly valuable. It appears that time consumption surges with the system’s complexity, which poses a relatively significant challenge for user experience and real-world robotics deployment.  
3. The experiments lack in-depth analysis. For instance, there is no exploration of the specific patterns or scenarios where different methods fail.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. How does the performance gap between MAS-ZERO and simpler baselines (e.g., CoT-SC) evolve as the underlying LLM's capability increases? Does the sophisticated design process yield diminishing or increasing returns with more powerful models?

2. Did the self-evolution process uncover any surprising or counter-intuitive MAS designs that challenge conventional human intuition? Are there examples of novel agent compositions or problem decompositions that were consistently discovered for specific task archetypes?

3. As single models become more capable and integrate complex reasoning and planning internally, how do you see the role of external multi-agent orchestration frameworks evolving? Will their functions be absorbed into the base models, or will systems like MAS-ZERO remain essential?
soundness: 2
strengths: 1. The motivations are very great. For example, this paper wants to eliminate the need for a labeled validation set.
2. The framework's explicit mechanism for breaking down complex problems into manageable sub-tasks and designing tailored sub-MAS for each is a robust approach to tackling intricate challenges.
3. The paper can be easily understood.
summary: This paper introduces MAS-ZERO, a novel framework for automatically designing Multi-Agent Systems at inference time without requiring a supervised validation set. The core of the system is a meta-agent that iteratively refines an MAS configuration for each specific problem instance. The process involves three stages: MAS-Init, which generates baseline solutions from a set of predefined "building block" strategies; MAS-Evolve, where the meta-agent decomposes the problem, designs an MAS, and refines it based on self-generated feedback on sub-task solvability and completeness; and MAS-Verify, which selects the best final answer from all generated candidates. The authors demonstrate that MAS-ZERO achieves state-of-the-art performance across challenging reasoning, coding, and agentic benchmarks, establishing a new Pareto frontier for the accuracy vs. cost trade-off.
weaknesses: 1. The framework centralizes a heavy cognitive load on the meta-agent, which must simultaneously evaluate agent capabilities, refine system architecture, and verify final answers. This creates a potential single point of failure and places a high floor on the required capability of the underlying LLM.

2. While the framework avoids a traditional validation set, it replaces it with an online, self-referential validation loop (Meta-Feedback and MAS-Verify). This raises concerns about evaluation blind spots or self-reinforcing biases, as the "judge" is the same model responsible for the design.

3. The system's creative scope is limited by the initial, manually-provided set of building blocks. The meta-agent primarily optimizes the composition and parameters of these existing modules rather than inventing fundamentally new agentic behaviors or interaction patterns.

4. The iterative, multi-candidate generation process incurs substantial computational cost at inference time for every single problem. While presented as a favorable trade-off, this per-instance expense could be prohibitive for many practical, large-scale applications.