Title: Multi-Agent Collaboration via Evolving Orchestration
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Model

Autonomous Agent

Multi-Agent Collaboration

Evolving Orchestration
Abstract: Large language models (LLMs) have achieved remarkable results across diverse downstream tasks, but their monolithic nature restricts scalability and efficiency in complex problem-solving. While recent research explores multi-agent collaboration among LLMs, most approaches rely on static organizational structures that struggle to adapt as task complexity and agent numbers grow, resulting in coordination overhead and inefficiencies. To this end, we propose a puppeteer-style paradigm for LLM-based multi-agent collaboration, where a centralized orchestrator ("puppeteer") dynamically directs agents ("puppets") in response to evolving task states. This orchestrator is trained via reinforcement learning to adaptively sequence and prioritize agents, enabling flexible and evolvable collective reasoning. Experiments on closed- and open-domain scenarios show that this method achieves superior performance with reduced computational costs. Analyses further reveal that the key improvements consistently stem from the emergence of more compact, cyclic reasoning structures under the orchestrator’s evolution. Our code is available at https://github.com/OpenBMB/ChatDev/tree/puppeteer.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a “Puppeteer” framework that dynamically orchestrates multi-agent collaboration through a centralized controller trained with reinforcement learning. This approach seamlessly integrates dynamic agent selection, sequential decision-making, and adaptive evolution, leading to improved solution quality and computational efficiency across diverse tasks.\n\nSTRENGTHS AND WEAKNESSES:\n1. **Strengths**\n\n     1. The topic of multi-agent collaboration is interesting.&#x20;\n\n     2. The paper is well-written and well-structured. The authors use the metaphor of a puppeteer to introduce and explain their approach, making complex concepts more accessible.\n\n     3. The experimental section of the paper is  well-structured, providing convincing evidence of the proposed framework's efficacy.&#x20;\n\n  2. **Weaknesses**\n\n     1. Some key experimental details are incomplete or missing. Could the authors please provide more detailed information on the following aspects, **as this is crucial for understanding and reproducing the work?**\n\n        1. How many agents are initialized for the whole decision process? What is the initial distribution of these agents ? (For example, tool-use agents and reasoning agents)\n\n        2. The orchestration part is the most crucial part of this work. However, discussions and details about this part are limited. In **Appendix B.2,&#x20;**&#x74;he authors claim that they use a pre-trained language model combined with an MLP model as the orchestration part. I am a little bit confused and quite interested in:\n\n           1. Please provide more details on the integration and collaboration between the orchestration part and the optimization process of agent selection and topology change.\n\n           2. Is it feasible to use a fine-tuned LLM as the orchestration part? Why do you use Single-MLP instead?\n\n     2. In the introduction part, this paper proposes that existing MAS works are limited in scaling. In the experiment part , the paper conducts experiments on token expense efficiency, which is convincing. However, **I am quite interested in the other part of scaling&#x20;**--  How does Puppeteer perform compared with existing MAS works when we increase the number of initialized total agents?\n\n     3. Recently, there have been many popular papers on multi-agent collaboration and co-evolution. If we treat tools as a specific type of agent, there will be more similar work. Specifically, could the author describe the **key** differences between Puppeteer and the following works? (Even if some of these works are in the same period.\n\n     &#x20;        1\\. OTC: Optimal Tool Calls via Reinforcement Learning?  https://arxiv.org/abs/2504.14870v1\n\n     &#x20;        2\\. OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation. https://arxiv.org/abs/2505.23885\n\n     &#x20;       3\\. MASS: Multi-Agent Simulation Scaling for Portfolio Construction https://arxiv.org/abs/2505.10278\n\n     &#x20;       4\\. ReTool: Reinforcement Learning for Strategic Tool Use in LLMs  https://arxiv.org/pdf/2504.11536\n\n     &#x20;       5\\. Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies https://arxiv.org/abs/2502.02533\n\n\n\n     We are willing to raise the score if the author can address our questions and clarify our concerns.\n\nQUESTIONS:\nPlease see weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the organizational structure of multi-agent collaboration for LLMs. The authors argue that prevailing static collaboration topologies suffer from limitations in adaptability and efficiency. To overcome this, they propose a \"puppeteer-style\" paradigm where a centralized policy, optimized via reinforcement learning, dynamically selects which agent to activate at each timestep based on the evolving task state. The approach formulates the collaboration process as a sequential decision problem, aiming to dynamically generate flexible and efficient reasoning graphs. Experimental results on several closed- and open-domain tasks show that this method improves performance while reducing computational costs compared to a range of baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper provides a clear and compelling motivation, identifying the rigidity of static collaboration in LLM-based multi-agent systems as a significant and timely problem.\n\n2. The *puppeteer* analogy is highly intuitive, and framing the complex collaboration problem as a centralized, learnable sequential decision process is conceptually clean.\n\n3. The reward function, which considers both task performance and computational cost, is a major strength, guiding the system towards more practical and economical solutions.\n\n4. The analysis of hyperparameters like chain depth and exploration width provides practical guidance for applying the method.\n\n\n**Weaknesses**\n\n1. The idea of a central controller or router agent is not entirely new in the multi-agent space. The paper needs to better articulate the fundamental conceptual leap from prior dynamic orchestration works.\n\n2. Centralized Bottleneck. The orchestrator itself can become a computational and decision-making bottleneck as the number of agents and the complexity of the state space increase.\n\n3. The use of the basic RL algorithm, coupled with a small number of optimization samples. It raises concerns about the stability, convergence, and reproducibility of the training process. It might be prone to overfitting.\n\n4. The efficiency analysis focuses only on inference-time token consumption. The significant computational cost of the RL training phase itself is not reported, which is a key part of the total cost compared to static methods.\n\n5. Simpler Dynamic Baseline: The authors should conduct a comparison against a simpler, non-RL dynamic baseline (e.g., using a single LLM call to choose the next agent). This would help isolate the specific benefits gained from the complex RL framework.\n\n6. The paper doesn't adequately argue why a centralized approach is inherently superior to dynamic strategies. It assumes a god's-eye view for the orchestrator, which may be unrealistic due to the information bottleneck of the aggregated state.\n\n7. The core assumption that agent selection requires a complex reinforcement learning solution is not sufficiently challenged. Simpler dynamic approaches, like supervised routing or greedy selection by a powerful LLM, are not explored as baselines, making it hard to justify the added complexity of RL.\n\n8. The specific logarithmic form of the cost penalty is an arbitrary design choice without theoretical or empirical justification.\n\n9. The paper claims that emergent compact and cyclic structures are the reason for performance gains. However, this establishes a correlation, not causation. The argument lacks rigorous proof.\n\n10. This design is incomplete. Only the orchestrator learns and adapts, while the agents themselves remain static. A truly adaptive system might be expected to evolve both its organizational structure and the capabilities of its individual components.\n\n11. **Misleading Reproducibility Claims**: The authors explicitly state in their checklist that *The code and data are provided in the supplementary materials under the file name code.zip.* However, is absence of this file in the actual submission. It is impossible to reproduce the experiments and verify the paper's central claims, severely impacting the work's credibility.\n\nQUESTIONS:\nSee above.\n\n1. Could you please elaborate on the specific formulation and aggregation method for the global state? How does the system manage the growing context history from numerous agents and steps into a fixed-size vector for the policy network, and how does this approach scale?\n\n2. Have you explored more advanced RL algorithms like PPO, and would you expect them to yield different results?\n\n3. Will a decentralized or hierarchical approach become more effective at a certain scale?\n\n4. The paper states that the reward considers *computational resource consumption at each step*, but Equation 6 defines the step-wise cost purely as a logarithmic function of the step number t, not the actual tokens or FLOPs consumed in that specific step. Could you clarify this discrepancy?\n\n5. The multi-agent baselines like MacNet and EvoAgent are presented in a mono-model configuration. Does this not provide your method with a significant unfair advantage by having access to a more diverse set of capabilities?\n\n6. Increasing exploration width or depth from the default setting leads to worse accuracy, not just higher cost. This is counter-intuitive. What is your hypothesis for this performance degradation? Does it suggest a fundamental issue with credit assignment in your RL algorithm when faced with a larger action space.\n\n7. The method section specifies that for the policy network, a pretrained language model's parameters are kept frozen, and only a small MLP head is trained. What was the rationale behind freezing the entire base model? This will limit the ability of the puppeteer.\n\n8. In the ALFWorld generalization experiment, the exploration width was constrained to 1. This effectively reduces the orchestrator's role to generating a single, linear sequence of actions rather than dynamically managing multiple reasoning paths.\n\nLIMITATIONS:\nYes.\n\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInstead of wiring up a brittle, pre-built tree of chatbots and hoping for the best, the authors of “Multi-Agent Collaboration via Evolving Orchestration” teach a reinforcement-trained “Puppeteer” to glance at the shared scratch-pad, pick whichever specialist LLM seems most useful at that moment, and then file away a mental note about whether the detour was worth the tokens. Over a few thousand trial-and-error runs the policy quietly sidelines agents that don’t pull their weight, trims bloated reasoning chains, and even discovers tight feedback loops where two or three “puppets” bounce ideas until the answer sticks, lifting success rates on GSM-Hard, MMLU-Pro, software debugging and creative writing while doing so. Crucially, those gains arrive hand-in-hand with a 20–40 percent drop in token spend because the reward function values thrift as much as quality. When the dust settles, the learned chat graph is noticeably tighter—messages swirl around a couple of hub agents in short cycles—which the authors argue makes the whole affair cheaper to run and kinder to GPUs. And since one knob lets practitioners trade a bit of accuracy for a bit of frugality, the study hints that a well-drilled team of modest models, steered by this lean conductor, can rival a lone heavyweight LLM without breaking the budget.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The scheduler demonstrates impressive, cost-aware improvements: across four diverse benchmarks, it boosts accuracy by 6–12 percentage points while significantly reducing token usage, showing that thriftiness is truly embedded in its reward function.\n- Visual analyses highlight how the system gravitates toward a small set of high-value “hub” agents and benefits from rapid feedback loops—offering a clear explanation for simultaneous gains in both accuracy and efficiency.\n- The findings indicate that a carefully coordinated ensemble of smaller models can match the performance of a single, massive LLM like Titan, but at only a fraction of the inference expense—an especially practical result for teams wrestling with soaring token costs.\n- Unlike previous approaches that either use a fixed graph structure (e.g., GoT) or train each agent in isolation (e.g., Reflexion), this work introduces a learnable metacontroller that dynamically selects between different LLMs at each step, which represents a genuinely novel contribution.\n\n**Weaknesses**\n- While the paper quantifies inference savings, it fails to disclose how many GPU-hours were required to train the scheduler itself; without this information, it’s difficult to assess the true net efficiency.\n- It remains unclear what portion of the observed gains stems from the RL-based policy versus simpler strategies, such as token-cost thresholds or fixed priority lists.\n- The appendix briefly mentions a “Bradley-Terry + 3-layer MLP” architecture with an 8,000-dimensional input, yet offers no rationale for this design choice and omits details on the policy network’s parameter count.\n\nQUESTIONS:\n- I’d like to see a clearer picture of the RL scheduler’s computational footprint: specifically, how many wall-clock hours were spent training it, which GPU models were used, what peak memory requirements were, and the total number of tokens it processed.\n- It would be valuable to know whether this scheduling strategy holds up outside of purely text-based benchmarks—for example, on code-generation tasks like HumanEval or in interactive environments such as WebArena.\n- The paper reports using a single cost-weight λ for each task but doesn’t explain how it was chosen. Could the authors discuss how sensitive their results are to different values of λ, as well as the constants F and φ in Eq. (6)?\n\nLIMITATIONS:\nThe Discussion section currently only alludes to “future work” and skips over key limitations and risks. At minimum, the authors should:\n- report GPU-hours, hardware specs, and energy use for the RL scheduler.\n- discuss issues like mis-coordination or deceptive agreement and note that the method hasn’t been tested in interactive or safety-critical settings.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
* Aims to learn a dynamic orchestrator for LLM-based multi-agent collaboration, using a "puppeteer-style" paradigm

STRENGTHS AND WEAKNESSES:
Strengths
* I like the idea of evolving an orchestrator; I agree that many paradigms for coordination are too rigid
* The paper is well-written; clear and easy to follow; the illustrations are very useful as well
* The evaluation tasks seem reasonable, and the performance improvements are clear
* Extensive interpretable analysis of experiments

Weaknesses
* I can't help but feel that there must be much more literature on this kind of orchestration that is not referenced in the paper (see my question below); especially pre-LLM multi-agent RL (MARL) literature

QUESTIONS:
* I'd like to know how this work is related to works in MARL coordination; e.g. ALMA [1], which is a similarly hierarchical approach which also performs "orchestration" (task allocation) dynamically; it seems that there's probably some overlap of ideas (if not with ALMA, then more broadly in that literature)

[1] https://arxiv.org/abs/2205.14205

LIMITATIONS:
yes, Appendix E; appears comprehensive, and encapsulates my thoughts
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-07