Multi-Agent Collaboration via Evolving Orchestration
Authors: Yufan Dang
Chen Qian
Xueheng Luo
Jingru Fan
Zihao Xie
Ruijie Shi
Weize Chen
Cheng Yang
Xiaoyin Che
Ye Tian
Xuantang Xiong
Lei Han
Zhiyuan Liu
Maosong Sun
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Autonomous Agent
Multi-Agent Collaboration
Evolving Orchestration
Primary area: deep_learning

Abstract: Large language models (LLMs) have achieved remarkable results across diverse downstream tasks, but their monolithic nature restricts scalability and efficiency in complex problem-solving. While recent research explores multi-agent collaboration among LLMs, most approaches rely on static organizational structures that struggle to adapt as task complexity and agent numbers grow, resulting in coordination overhead and inefficiencies. To this end, we propose a puppeteer-style paradigm for LLM-based multi-agent collaboration, where a centralized orchestrator ("puppeteer") dynamically directs agents ("puppets") in response to evolving task states. This orchestrator is trained via reinforcement learning to adaptively sequence and prioritize agents, enabling flexible and evolvable collective reasoning. Experiments on closed- and open-domain scenarios show that this method achieves superior performance with reduced computational costs. Analyses further reveal that the key improvements consistently stem from the emergence of more compact, cyclic reasoning structures under the orchestrator’s evolution. Our code is available at https://github.com/OpenBMB/ChatDev/tree/puppeteer.

Multi-Agent Collaboration via Evolving Orchestration
Yufan Dang*
Chen Qian
Xueheng Luo*
Jingru Fan*
Zihao Xie*
Ruijie Shi*
Weize Chen *
Cheng Yang*
Xiaoyin Che
Tian %
Xuantang Xiong
Lei Han 2
Zhiyuan Liu*
Maosong Sun *
Tsinghua University Shhnnghii Jiao Tong University
Beijing University of Posts and Telecommunications
Siemens Tencent Robotics X
dangyf25emails. twwngnua..du....om qiancesjtu. edu. cn
lizzy@tsinghua edu. smsetsinghua. edu.
Abstract
Large language models (LLMs) have achieved remarkable results across diverse downstream tasks, but their monolithic nature restricts scalability and efficiency in complex problem-solving While recent research explores multi-agent collab- oration among LLMs, most approaches rely on static organizational structures that struggle to adapt as task complexity and agent numbers grow, resulting coordination overhead and inefficiencies. To this end, we propose a puppeteer-style paradigm for LLMM-baeed multi-agent collaboration, where centralized orchestra- tor ("puppeteer") dynamically directs agents ("puppets") in response to evolving task states. This orchestrator is trained via reinforcement learning to adaptively sequence and prioritize agents, enabling flexible and vvole collective reasoning. Experiments on closed- and open-domain scenarios show that this method achieves superior performance with reduced computational costs. Analyses further reveal that the key improvements consistently stem from the emergence of more compact, cyclic reasoning structures under the orchestrator's evolution. Our code is available at hhttps: thub. com cctm//ppeeBBBB/haaaaeeeereeeeppp teppeterr
Puppersor Lx Pollcy
Manup looe Le, Oirhheedndn
Puppet Ln, Agent
Envronmont Lo, Pentbnang
000000
1O000
Figure 1: Overview the proposed multi-agent collaboration framework. A central policy ("pup- peteer") dynamically orchestrates which agent ("puppet") should reason at each step based the evolving state of the task. As the task progresses, the orchestrator adaptively promotes more effective agents while removing those that less useful, analogous puppeteer learning to skillfully pull or cut strings to direct a performance.
Equua Contribution.
1 "Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Large language models (LLMs) 69 48 28 have achieved remarkable advances actoss diverse natural language processing tasks, demonstrating strong capabilities in planning 622 46), reasoning [70]
53
33
and decision-making [3
64 1. As the ambition to tackle ever more complex, multi-
faceted problems- such as tool-augmented inference [47 51
86 55 and collaborative problem- in open environments continues grow, the limitations of monolithic LLMS
solving 42 29 12
are becoming increasingly apparent 771 8 9
To address these challenges, recent research has drawn inspiration from human teamwork, exploring multi-agent systems (MAS) comprising diverse LLMM-ased agents with specialized skills [45 52].
personalized reasoning patterns 770 76 43], and external tool integrations 885 47. However, many existing approaches rely on predefined or statically generated agent topologies 661 44 that lack flexibility and scalability. This rigidity often leads increased coordination overhead, degraded system performance, and inefficiencies as the variety of tasks [81 23 and the number of agents scale [45]. Especially in large-scale scenarios, the absence of principled and dynamic coordination can further result in redundant computation (77), ineffective communication
13], and diminished
collective problem-solving effectiveness 66
Can dynami Frrcesiatrocestattiion simultaneously maximize collaborative effectiveness and computational efficiency? Answering this question is crucial for building scalable, robust, and practical collective intelligence suitable for complex real-world settings. To this end, we propose a novel paradigm for constructing flexible and continually evolving multi-agent system. Drawing inspiration from puppet shows where a central puppeteer skillfully directs multiple puppets behind the scenes- reconceptualize multi-agent collaboration puppeteer who dynamically selects and sequences agent activations based on evolving task states, implicitly coordinating collaboration within the group. As tasks progress, the orchestrator learns to prioritize effective agents and suppress less efficient ones, continually driving the system toward higher overall performance and efficiency.
reasoning process orchestrated by a centralized
Specifically, our framework advances multi-agent reasoning by introducing two key innovations: (i) Dynamic Orchestration: Moving beyond static collaboration patterns, we employ dynamic orchestrator that routing agents at each step based on the current contexts. This process is formulated sequential decision problem, effectively yielding implicit inference graph and supporting flexible, scalable agent coordination. (ii) Adaptive Evolution: To maximize efficiency and minimize redundancy, we employ reinforcement learning to continuously update the orchestrator's policy by leveraging feedback from completed tasks. Over time, the orchestrator learns to emphasize strong agent trajectories and prune less effective ones, enabling the system evolve toward greater efficiency progressively.
Empirical results on both closed- and open-domain scenarios demonstrate that our approach consis- tently yields more effective solutions while requiring less computational overhead. Analyses further reveal that, although the evolved topologies are not fixed across different tasks, the key improvements consistently stem from the emergence of more compact, cyclic reasoning structures.
2 Method
We propose a unified framework for multi-agent reasoning that organizes diverse LLM-based agents via orchestrating their collaboration dynamically using centralized policy, and continually optimizes their collaboration process through reinforcement learning.
A LLM-based agent can be abstracted in its minimal form as a = (m, !). where m denotes the foundation model, r represents the reasoning pattern or prompting strategy, and / is the of available extemaa tools. The agent space A enumerates all possible agents formed by combinations of these components, i.c., A {m, r, }. encompassing both intrinsic and cool-augmented reasoning. Each agent thus represents an atomic reasoning behavior participating in task solving.
For example, mesh structured multi-agent systems with 50 nodes can require up to hours to develop software comprising only few hundred lines code.
"hii paper refers to the dynamic organizational structure during sal-time reasoning.

---

For multi-agent collaboration, following [4], MAS is naturally formalized as directed graph (V,E), where each node v V corresponds to an agent E A, and each directed edge (y) E encodes dependency or information flow, conveying intermediate context from agent aj to agent aj. Typically, this graph presents a single-source, single-sink configuration: the source node represents the input task, while the sink node yields the unified task output (i.e., artifact). This formalism underlies a unified and temporal modeling framework for LLM-based reasoning systems and is analogous to "graph-of-thought" [
that captures the deep thinking process.
2.1 Dynamic Orchestration
Centralized Puppeteer A challenge multi-agent reasoning is achieving efficient orchestration as task complexity and agent diversity increase. In previous approaches, each agent autonomously selects collaborators, but this incurs coordination overhead and poor scalability, particularly as agents increase or change 88 45 Instead, inspired by centralized coordination (e.g.. puppeteer managing multiple puppets), we model the system as driven by a centralized orchestrator. This orchestrator dynamically selects which agents to activate in each step, based on the dynamic task state, and delegates reasoning to the selected agents. Such centralization decouples agent selection from internal agent behaviors, greatly enhancing adaptability and scalability without extensive parameter retraining.
Serialized Orchestration Another challenge stems from the combinatorially large space of possible collaboration topologies among agents. Exhaustive search is infeasible, thus prior work focuses only on canonical graphs (egg, chains, trees, graphs) 766 70 2] Instead, we propose to serialize the collaborative reasoning process: rather than searching the entire topological space, the orchestrator "unfolds" the graph into reasoning sequence guided by a topological traversal strategy. By main- taining a topological ordering, the reasoning steps follow the partial order implied by the problem dependencies. It is important to note that although the orchestration of agents appears be serialized and "unfolded", when this episode is restored through folding, it can be reconstructed as directed graph (with agents as nodes and orchestration partial order relations as edges).
Building on the two concepts above, we formalize the multi-agent collaboration as sequential decision process governed by a centralized policy æ. At each time step t, the orchestrator selects single agent a; E A to activate, conditioned on the current global system state S₁ and the task specification т. The global state S, consists of all relevant agent states and aggregated contextual information up to step t:
a, a<,m(S,,7)) = P(u| S,,T)
where the policy is a function mapping the observable context-such as the current state and task description-to distribution over candidate agents 773 74] (e.g., via neural scoring, embedding-
based models, or Bradley-Terry style approaches (68).
Upon activation, agent a, receives its state sy (extracted from S2) and generates its output by generative reasoning mapping fa., after which the system state is updated (4) as:
0, fae (a.), St), S o()
(2)
The process continues iteratively: at each step, the orchestrator observes the updated system state S and selects the next agent 0,+1 to activate, again using the policy π conditioned only on S and r. This sequential agent selection process explicitly satisfies the Markov property [17
P(ar+1
T) PP(ee+1 S.+1,
The process terminates when a stopping criterion is met (e.g.. when the selected agent is a designated terminator when the task-solving resource is exhausted). At that point, final aggregation function Fagg combines all agent outputs across timesteps to yield the overall solution:
o²
=
Fags ((oo, o1....,r) or = (Sr, or)
where T' denotes the total number of reasoning steps.
2.2 Adaptive Evolution
While dynamic orchestration enables flexible agents' long-chain reasoning, naive implementations may invoke redundant or unhelpful agents, resulting in unacceptable inefficiency. To address this,
3

---

thanks to the Markov property, learnable policy is considered for continuously learning to make agent selection decisions adaptively. After each reasoning episode, the system receives feedback jointly evaluating solution quality and resource consumption, enabling the policy to learn which agent most valuable based on real-time task states.
Practically, this facilitates dynamic pruning of agents: the orchestration process adapts to increasingly favor compact reasoning chains by reducing reliance on agents that offer little incremental gain or incur excessive cost. Over time, the orchestrator policy evolves to organize more effective agent sequences, balancing expressive collaboration with computational efficiency. Thus, the evolvable puppeteer not only orchestrates agent collaboration, but also distills the reasoning process to its most essential components, enabling robust and scalable performance.
Policy Optimization To systematically optimize both the effectiveness and efficiency of collabo- ration, we employ REINFORCE
[60], reinforcement learning (RL) technique, as our underlying 73 [4]. By doing so, the orchestration policy learns from previous executions, adaptively refining agent selection and pruning strategies to achieve more robust and cost-efficient multi-agent agent reasoning.
optimization framework 40 31
Concretely, the optimization objective is to maximize the expected return over complete reasoning trajectories, where the return reflects both overall effectiveness and inference efficiency:
N
T
J(0) = E [(T)), VoJ(O) ~
E.
Σ
Vo log To (as(()))
R(T)
(5)
N ==1
with R(T) denoting the total reward accrued for trajectory T (So, dg 0o, 00, Si,..., ST, UT, OT), and the output generated by agent a at state S., N is the sample size, and 7' is the total number of steps in one trajectory. The orchestrator's parameters are updated iteratively via gradient ascent: 0 < 0 + aV))(()), with learning rate a. Through such RL-driven optimization, the orchestrator leverages accumulated cross-task experience to refine agent selection, dynamically suppressing unhelpful costly agents and converging toward more compact, high-performing collaboration structures.
Reward Design To effectively guide the orchestrator's optimization, we design reward function that jointly accounts for both solution quality and computational efficiency. Upon completion of each task trajectory, terminal reward is assigned: for tasks with ground-truth answers, {0, indicates correctness; for open- tasks, r E [0, 1] quantifies answer quality. The overall reward is defined recursively over time steps (59] At the terminal state (t T), the cumulative reward incorporates both solution quality and total computational cost:
--- CT. Rs
i ( 7 Re+1 - X.C,, C., it FF=F log 1+ (++1
A
where A controls the trade-off between accuracy and efficiency, 7 E (0, 1] is the discount factor. To encourage economical reasoning, we penalize excessive computational expenditure. Specifically, for each reasoning step f,, we define step-wise cost Ct based on FL.OPs or token-level metrics (50). denoted by F, and step factor normalized by the maximum step budget pp, i.e., t/yp. This composite reward formulation incentivizes the orchestrator to achieve correct and high-quality solutions while minimizing unnecessary computation, ultimately driving the MAS to discover reasoning structures that are both effective and cost-efficient.
3 Experiments
Datasets and Metrics To thoroughly assess our framework, we use a range of publicly available and logically demanding datasets covering both closed- and open-domain reasoning tasks:
These tasks require precise, objective reasoning and unambiguous answers, making them well-suited for evaluating core inference accuracy and mathematical rigor. GSM- Hard(18) features arithmetic problems involving unusually large numbers and complex multi-step calculations, challenging models' advanced mathematical reasoning and error-free execution. MMLU-Pro 67) is comprchensive benchmark spanning diverse subjects and difficulty levels,
Closed-domain Tasks:

---

using moultple-choice questions to assess both factual knowledge and logical deduction. Both benchmarks are designed to assess the model's ability in mathematical and commonsense reasoning and inference, with accuracy as the evaluation metric.
Open-domain Tasks: These tasks inherently creative and open-ended, requiring multi- dimensional qualitative evaluation. They rigorously assess agents' ability to integrate information, understand context, and generate novel solutions. SRDD[44] consists of real-world textual software requirements and tasks agents with building the corresponding software, demanding proficiency in requirement comprehension, system/design reasoning, code generation, and testing. Its official evaluation metric combines completeness, executability, and consistency [44), reflecting the practi- cal demands of real-world software development workftows. CommonGen-Hardl (39] challenges agents to generate coherent sentences that connect seemingly unrelated concepts, highlighting their abilities in commonsense reasoning, contextual understanding, and creative expression. Evaluation is based on an aggregate metric that incorporates grammar, relevance, logical consistency [32] as well as concept coverage [39] providing nuanced assessment of generative quality.
Baselines To mitigate performance interference- whhere strong models may overshadow the con- tributions of weaker ones and evaluate our method's adaptability across agents with varying capacities, we partition the agent pool based on the parameter scale of the underlying models. Specif- ically, we define a Mimas subspace (smaller models: Qwen-2.5-78 Qwen-2.5-144 14B, LLaMA-3.1-80 LLaMA-3.2-3B, Mistral-7B, Mistral-Nemo-122 and Titan subspace (larger models: GPT-4-Turbo, GPT-4o-Mini, Gemini--15-PPr
Demini-I.5--last Claude-3-Sonnee
Claude-3-Haike Qwen-2.5-728
11aMAA33.1440500 covering both closed- and open-source families. All experiments are performed under both the Titan and Mimas subspace settings. To ensure the rigor and credibility of our experi- mental study, we select suite of representative and recent baselines that comprehensively span the spectrum from straightforward LLM generation to advanced agentic paradigms:
This category evaluates foundation models in the absence of explicit agent struc- turing or workflow orchestration, serving as a baseline for generative inference performance. For Mimas, competitive open-source models include LLaMA-3.1-8% Mistral-Nemo-124 and Qwen-2.5-14B For Titan, options such as 1LLMMA-31140550 GPT-4o-mini, and GPT-4-turbo are considered.
Pure Models:
This category explores paradigms where reasoning is performed by a single agent using a specific reasoning pattern or workflow. Self-refine [39 exemplifies iterative, self-corrective reasoning within a feedback loop, whereas AFlow [81] enhances agent reasoning efficiency by utilizing Monte Carlo Tree Search to optimize code-represented agent workflows.
Single-Agent Methods:
Multi-Agent Methods: We benchmark against the latest multi-agent reasoning systems, show- casing saate-oofthh--eee capabilities in leveraging agent heterogeneity and dynamic collaboration. MacNet [45] orchestrates agents within topologically static directed acyclic graphs, facilitating collective intelligence driven by one foundation model to enhance reasoning performance. EvoA- gent (78 adopts evolutionary algorithms to automatically generate and optimize diverse multi-agent systems, adaptively improving collaboration and performance without manual intervention.
Implementation Details Different agents are equipped with distinct reasoning patterns- such as task decomposition. reflection, refinement, critigue, modification, summarization, and termina- tion ciablina flexible problem-solving 43] 21
82 39]. External tools like WebViewer, WikiSearch, BingSearch, arXivSearch, Code Interpreter, and File Reader also integrated [47). Dynamic collaboration uses majority voting for output aggregation. The policy is initialized with a variant of Llama-3. F using default settings: episode length 4, parallel exploration up to 3, X 0.1, and 0.99. All baselines are rerun under identical settings.
3.1 Does Our Method Lead To Elevated Performance?
Many prior studies multi-agent systems have employed the same base model to drive agent behavior 44 36 87). To enable a more meaningful comparison and account for model heterogeneity, method, referred to as Puppeteer, introduces two distinct configurations within each agent
1 Other smaller scale models in same series (e.g.. Qwen-2.0-7B) have also been experimentally validated, and their performance is generally weaker than that of the larger-scale model within the same series. https::/huuginggf ace. http:aaac....c//////e//eeeeeeaae co.nviiiialllaaaaaaaaaaaaaaaaaaa 3. Nenn 700rronnn------0 008-Reward-Ht
5

---

subspace: the Mono setting, wherein all agents are driven by the same model, and we use LLaMA- 3.1-405B for Titan subspace and LLaMA-3.1-8B for Mimas subspace, and the default setting, wherein agents are driven by a diverse set of models. As Puppeteer undergoes online reinforcement learning. we categorize its learning process into two distinct phases: an initial phase characterized by unrefined behavior, and an evolved phase marked by reinforced performance.
Table 1: Performance comparison of different methods across various datasets in Titan and Mimas subspaces respectively. For each dataset, the highest scores are highlighted in bold and the second- highest underlined.
Minia
GSM-Hard
MMLU-PPr
SRDD
CommonGGn-Haar
AVG.
L.ama-3.1-8B
0.0000
0.5250
0.4615
0.6992!
0.4214
Mistal-Nemn-I1e
0.0350
0.4500
0.2097
0.7146
0.3523
Qwen-2.5-1488
0.0450"
0.38007
0.58911
0.3972
Sell Aclf-Refinszawa-1..
0:4750
0.2600
0,5412
0.6018
0.4695
APlowuare 1.a
0.2900)
0.500)
0.6362
0.7194
0.5364
MacNotuuaatidd
0.0000
0.2000
0.2017'
0.2862
Evo.genf
0.12507
0.5000
0,2510
0.7167
0.3981
Initialized
Evolved
Initialized
Evolved
Initialized
Evolved
Initialized
Evelved
Initialized
Evolved
Soapetter-MMitliaesse Puppeteer
0.24677
D.4800
0.5200
0.6983
0.6323
0.5068
0.6147
0.5600
5.5400
0.5700
0.6300
0.6653
0.6266
0.7139
0.7333
0.273
0.6324
Titan
GSM-Hard
MMLLU-Po
CommonGen-Hard
AVG.
Llama-3 1.1-0S
0.1350
0.7600
0.6061
0.8116'
0,5781
GPT-4o-Mini
0.1050/
0.5950
0.6822!
0.66911
GPT-4-Tarbo
0.2750/
0.68001
0.62441
0.7632!
0.5856
Self Refineuuunuuee
0.5250
0.50007
0.6345
0.7033
0,51577
AFlowuara-11--
0.5400
0,7500
0.6478
0.8218
0 0.6899
0.2905
0.4800
0.4228
0.8817!
0.5187
BvoAgeettamusst-ee
0.4250
0.5400
0.1730
Initialized
Evalved
Initialized
Evolved
Initialized
Evolved
Initialized
Evalved
Initialized
Evolved
Puppeteer Mono 1.00
0.5400
0.6100
0.6910
0.7600
0.6264
0.7697
0,8111
0.8417
0.6671
0.7453
Puppeteer
0,6600
0.7000
0,7400
0,8300
0.6191
0,7637
0.7381
0.7987
0.6893
0,7731
As detailed in Table 1 Puppeteer consistently achieves superior average performance during the evolved phase across all evaluated tasks, irrespective of domain type or model space size. Similarly, Puppeteer-Mong demonstrates robust performance across both large- and small-scale models. These results collectively underscore the exceptional capability of our centralized orchestrator in coordinat- ing both heterogeneous and single-model-driver agents to form highly effective MAS, highlighting its robustness in managing diverse agent configurations.
Compared to various agent workflows and multi-agent baselines using the same base model, Puppeteer-Mono consistently outperforms competing methods across nearly all evaluated tasks. This result highlights the efficacy of our centralized orchestrator in coordinating single-model-drive agents with optimized reasoning strategies and strategic tool utilization, surpassing alternative frame- works. It underscores that superior performance stems from sophisticated organizational design and context-aware dynamically constructed multi-agent topology. Notably, despite Puppeteer-Me Mono leveraging near-optimal models within its respective subspaces, Puppeteer consistently achieves superior performance, likely benefiting from complementary interactions among heterogeneous agents driven by diverse models. Additionally, the expanded space in Puppeteer enables broader exploration of the solution landscape, thereby enhancing optimization opportunities.
To illustrate Puppeteer's capability in organizing effective MAS, we compare performance between the initial and evolved phases. The results show that continued optimization yields substantial gains-for example, Puppeteer in the Titan subspace improves from 0.6893 to 0.7731 on average, with a similar trend observed in the Mimas subspace. These findings highlight the critical role of continual optimization in enhancing coordination and task specialization, and further suggest promising directions for advancing beyond traditional, static agent paradigms toward more adaptive and collaborative agent systems.
3.2 Does Performance Gain Come at the Expense of Efficiency?
Recent research in non-learnable multi-agent systems highlighted trade-off: performance gains achieved through agent collaboration are often accompanied by increased overall token consump-

---

tion 777 45] To investigate whether our approach exhibits a similar pattern, we visualize the average token consumption and the number of orchestrated agents throughout the training process.
As shown in Figure 2 the token metric consistently de- creases over the course of learning across almost all set- tings. This demonstrates that our system's 's performance improvements do not come the cost of increased compu- tational overhead; on the contrary, our approach achieves simultaneous advances in both effectiveness and efficiency. This result is primarily attributed to our reward design, which balances accuracy and efficiency via a tunable weighting factor A, enabling adaptable trade-offs tailored to different application needs, with higher values of À indi- cating greater emphasis on minimizing cost, shown in the Figure Specifically, the reward is designed to encour- age the orchestrator to: (i) prioritize agents that complete tasks with reduced token usage while preserving perfor- mance, and (ii) terminate reasoning early by invoking the Terminator agent, thereby fostering efficiency through the exclusion of redundant or lowtcontri agents. This mechanism enables the orchestrator to optimize both overall performance and resource consumption. In extreme cases, if the efficiency factor is entirely omitted from the reward design, the system naturally degenerates into a traditional large-scale collaborative framework, albeit with potentially further improvements in performance.
Token Count Evolution
Figure 3: Token consumption and agent count orchestrated in the Titan subspace for Puppetecr-Mone on MMLU-Pro.
More specifically, in the Titan setting, the number of active agents notably decreases over the course of learning, suggesting that the orchestrator progressively learns to terminate the reasoning process earlier for more efficient problem-solving. the number of orchestrated agents remains relatively stable, indicating the orchestrator's caution in prematurely halting the reasoning process due to the comparatively limited capabilities of the agents. In this case, reductions in token consumption are primarily achieved through the preferential selection of lower-cost agents rather than shorter reasoning chains. This contrast between Titan and Mimas arises from fundamental differences in agent capacity: Titan agents can solve tasks more efficiently, enabling earlier stopping without quality loss, whereas Mimas agents often require longer, more elaborate reasoning processes to ensure reliable completion.
contrast, in the Mimas setting (see Appendix for results),
3.3 How Does Organizational Topology Evolve During Reinforcement?
To elucidate the emergent organization of multi-agent collaboration, we systematically examine the evolution of agent interaction topologies throughout the learning process. Multi-agent reasoning can be abstractly modeled as dynamic orchestration governed by our centralized orchestrator, yet empirical evidence reveals an untrained "initialized" MAS system often results in a highly sophisticated and adaptive organizational structure. Instead of relying on static structure, our Puppeteer dynamically constructs intricate topological motifs such as trees, graphs, and cycles- by selecting the next agent to activate at each reasoning step based on the current reasoning state. Thus, the topology emerges incrementally during reasoning, embodying a flexible, context-aware organizational paradigm.
GSM-Hard
MMLU-Pro
CommonGen-Hart
SRDD
Lit Parks
Moris
U99 di
Loi Trooss
Learr Trocest
Leaning Traa
Figure 2: Evolution of token consumption and the number of orchestrated agents per task along the training process. Trends are fitted using LOWESS (Locally Weighted Scatterplot Smoothing) [14]
7

---

Chain Topology
Tree Topology
Graph Topology
From Chain to Tree
From Tree Graph
Branching Expantion
Pordia Tragectoyy
Crose
Secktrackrg
Sel-loop
Agent
Figure 4: The organizational topology evolves as gen- eral directed graph, shaped by the unconstrained orches- tration.
Edge between Agents
Figure 5: Examples of topology evolution.
Although the simplest form of multi-agent collaboration is often represented as chain structure 44 (70), the Puppeteer's dynamic orchestration naturally fosters tree-strue serustuuutured interactions by enabling the selection of one or multiple agents at each step. This mechanism supports branching behavior and parallel pathways, thereby enhancing scalability as the number of agents grows. However, despite an initial resemblance to tree-like expansion 76 driven by branching. the resulting topologies are This property arises from the flexible orchestration, which permits repeated agent activations, leading to cycles and feedback loops. Moreover, cross- branch connections emerge organically, underscoring the system's capacity to generate rich, expressive, and adaptive interaction patterns. Representative examples of these emergent motifs, including cycles, backtracking, and bross-branch links, are illustrated in Figure
inherently graph- graph-structured 45 87 2
cross-branch
As the puppeteer evolves over time, its orchestrating behavior changes accordingly, leading to distinct behaviors the resulting MAS. Here, we present specific sample, selected from both the initial and evolved phases, to demonstrate the emerging optimization effects. As shown in Figure 3 the initial phase features multiple disjoint chains, reflecting exploratory organization; after evolution, paths become fewer and cycles appear, indicating more stable and coordinated interactions. Additionally, the initial phase features two-agent communication with higher overhead; as evolution progresses, the structure condenses to single agent handling continuous reasoning, reflecting more efficient coordination and decision-making.
Eedieod
3.5
Toitbl avg. 1,0829
Evolved Big.. 1.4479
Evolved
0.62
Erahid ANp 1.40
Evolvds
Fvcheed 0.00
2.5
Density
1.5
0.5
cycle length=1
cycle length=2
cycle length=3
cycle length=4
(a) Graph Density Distribution
(b) Cycle distribution
Figure 6: The compaction and cyclicality dynamics in the evolution of multi-agent organizational structures.

---

Empirical observation reveals that Puppeteer's 's dynamic orchestration- fosters graph- structured topologies with diverse inter-agent connections gives rise two key structural phenomena: com- puction and eyclicality. The evolving interplay between densely clustered agents and frequent communication cycles marks a significant transformation multi-agent systems, shifting from loosely organized, exploratory interactions to tightly coordinated, specialized collectives.
Compaction. We observe marked trend toward structural compaction the course of learning (Figure 6a). As optimization proceeds, graph density quantiiinnn the degree of inter-agent Connectiity-steaddee increases, with organizational structure evolving toward highly interactive and tightly coupled subnetworks. Communication becomes progressively concentrated among subset of recurrently active 'hub' agents, forming dense subgraphs characterized by frequent and focused information exchange. This phenomenon is particularly pronounced in the Titan (large-model) subspace, where the orchestrator preferentially routes decisions through small cohort of strong agents, thereby intensifying iterative reasoning and collaborative consensus formation. Ultimately, the system transitions from diffuse, exploratory interaction patterns to highly synergistic and focused multi-agent reasoning.
Cyclicality. In conjunction with compaction, we document significant rise in cycle formation as learning progresses (Figure (b). Cyclic topologies- in which agents repeatedly revisit previous collaborators via closed-looy routes facilitate the re-circulation of intermediate results, mutual verification, and continual refinement. Unlike strictly hierarchical or acyelic networks, this cyclical structure supports recursive critique and sustained internal debate, much like what is seen in reflexive multi-agent paradigms (e.g.. Reflexion (54)). As cycles become more prevalent, system exhibits deeper internal feedback, more efficient reuse of information, and increased resilence-hhallmrrkk of mature, self-reeerential collaborative reasoning.
3.4 How to Further Use Hyperparameters Control Performance and Efficiency?
While reward shaping provides a di- rect mechanism for efficiency con- trol, the collaboration structure it- self is also crucial. As the or- chestrator organizes agent collabo- ration, upper-bound constraints on copology-speciffcallyy, chain depth and exploration width-are essential to prevent unbounded scaling and inef- ficiency. Depth refers to the length of orchestrated agent chains, while width captures the number of parallel explo- rations. As shown in Figure 7 there a clear monmonnotoiic relationship: the default setting achieves the best trade-off for our purposes. whereas increasing depth or width leads to re-
0.32
6500
0.50
VIID
0.33
WMD2
5002
300
5500
5502
0.30
0.30
5000
5002
6.25
4502
0.29
4502
0.26
5365
2500
2502
Accuracy
Takens
Accuraty
Tokeens
Figure 7: Impact of topology constraints on token consump- tion and accuracy (WaDy denotes exploration with width and depth y; W4D2 is the default).
dundancy, higher computational costs, and possible performance degradation. In general, enhancing accuracy tends to increase token consumption, and vice versa, suggesting that carefully balancing depth and width can help maintain both efficiency and effectiveness.
Related Work
The rapid advancement of LLMs 69
3
48
28
31 4 has spurred the development of autonomous
LLM-based agents [7]
42
29
72
58 which exhibit strong capabilities in planning [66] 21 24
memory 42 25 41
and tool usage 51
47
5
75).
These agents demonstrate growing proficiency
in addressing complex tasks 19
80
20
adapting to dynamic real- wollw environments 83
38 84], and exhibiting human-lite like behaviors such as collaboration and decision decision-making [10 42 52
Given that a single L.LM-based agent may struggle to handle the diverse and complex range of
real-world tasks [16. 29 64 recent research has increasingly focused on constructing L.MM
multi-agent systems
12 for software development 44 22 social simulation
medical treatment 61 30 scientific discovery [79].

---

Early MAS designs relied on fixed, handcrafted structures, e.g., mirroring software engineering These static approaches led rigid coordination
paradigms like waterfall models 42
87
45, limited workflow flexibility [23 X1 and suboptimal agent composition 15
37
35
To
address these issues, more adaptive orchestration methods have emerged: network-style organizations
dynamically select agents (Dylan (6)) as optimizable graphs enable prompt refinement and better
cooperation (GPT-Swarm [87) MacNet [45]); and code-based representations allow modeling of dynamic, task-specific processes. Recent approaches employ code-space search (ADAS [23]. AFlow (8I)) or train LLMs to generate MAS configurations on demand (MAS- [77)). Beyond
LLM-based MAS, classical MARL (Multi Agent Reinforcement Learning) works [26 65 34 63
have long explored coordination and role specialization, offering key inspirations for our RL-driven
orchestration in LLM-based multi-agent systems.
5 Conclusion
We proposed a novel framework for adaptive multi-agent LLM collaboration inspired by puppet show orchestration, where centralized, learnable "puppeteer" orchestrator dynamically activated agents within a directed topologies, our approach incorporated Unlike previous methods with static or manually designed orchestration and reinforcement learning- driven policy adaptation, enabling more principled and efficient collaboration. Experiments on diverse tasks showed that our method achieved superior solution quality and reduced computational cost. Analyses further revealed that the orchestrator promoted compact, cyclic reasoning structures, underpinning the performance improvements. We hope this work can constitute valuable step toward dynamic and scalable coordination in multi-agent collaboration.
Acknowledgement
The work was supported by the Tencent Rhino-Birr Focused Research Program.
References
[1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan. Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee. Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jomell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can and not as say: Grounding language in robotic affordances. In S15iv:2200401699) 2022.
[2] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michat Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyezyk, and Torsten Hoefler. Graph of thoughts: solving elaborate problems with large language models. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourleenth Symposium on Educational Advances in Artificial Intelligence (AAAI), 2024.
[3]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[4] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Zhang. Sparks of Artificial General Intelligence: Early experiments with GPT-4, In arXiv:2303. 12712 2023.
10

---

[5] Tianle Cai. Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large Language Models as Tool Makers. In The Twelfth International Conference on Learning Representations (ICLR), 2023.
[6] Mert Cemri, Melissa Pan, Shuyi Yang. Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia. Joseph E. Gonzalez, and lon Stoica. Why Do Multi-Agen L.M Systems Fail? In arXiv:2503. 13657, 2025.
[7] Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. AlphaMath Almost Zero: Process Supervision without Process. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[8] Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, lon Stoica, Matei Zaharia, and James Zou. Are More LLM Calls All You Need? Towards the Scaling Properties of Compound AI Systems. In The Thirty-eigtth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[9] Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Matei Zaharia, James Zou, and Ion Stoica. Optimizing Model Selection for Compound AI Systems. In 14815 2025.
[10] Liting Chen, Lu Wang, Hang Dong, Yali Du, Jie Yan, Fangkai Yang, Shuang Li, Pu Zhao, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. Introspective Tips: Large Language Model for In-Context Decision Making. In arXiv:2305.11520 2023.
[11] Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[12] Weize Chen, Ziming You, Ran Li, yitong guan, Chen Qian, Chenyang Zhao, Cheng Yang, Ruob- ing Xie, Zhiyuan Liu, and Maosong Sun. Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
[13] Weize Chen, Jiarui Yuan, Chen Qian, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Optima: Optimizing effectiveness and efficiency for llm-based multi-agent system. In arXiv:2410.08/55 15, 2025.
[14] William S. Cleveland. LOWESS: A Program for Smoothing Scatterplots by Robust Locally Weighted Regression. In The American Statistician, 1981.
[15] Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing aersona-assigned language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics (EMNLPP), 2023.
[16] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving Factuality and Reasoning in Language Models through Multiagent Debate. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.
[17] S. N. Ethier and T. G. Kurtz. Markov Processes: Characterization and Convergence. In Wiley Series in Probability and Statistics, 1986.
[18] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: pideram-aided language models. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023.
[19] Yinggiang Ge, Wenyue Hua, Kai Mei, Jianchao Ji, Juntao Tan, Shuyuan Xu, Zelong Li, and Yongfeng Zhang, OpenAGI: When LLM Meets Domain Experts. In arX1v:2304.0433 2023.
11

---

[20] Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Mareus Makowski, Rickmer Braren, Georgios Kaissis, and Daniel Rueckert. Evaluation and Mitigation of the Limitations of Large Language Models in Clinical Decision-making. In Nature Medicine, 2024.
[21] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang. Daisy Zhe Wang, and Zhiting Hu. Reasoning with Language Model is Planning with World Model. In The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
[22] Sirui Hong, Mingehen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng. Jinlin Wang, Ceyao Zhang. Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[23] Shengran Hu, Cong Lu, and Jeff Clune. Automated Design of Agentic Systems. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
[24] Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng. Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. Uncertainty of Thoughts: Uncertainty-Awe Aware Planning Enhances Information Seeking in LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[25] Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yinggiang Ge, Libby Hemphill, and Yongfeng Zhang. War and Peace (WarAgent); Large Language Model-based Multi-Agent 2023.
Simulation of World Wars. In arXi::2311
[26] Shariq Iqbal, Robby Costales, and Fei Sha. Alma: Hierarchical learning for composite multi- agent tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[27] Jikun Kang, Xin Zhe Li, Xi Chen, Amirreza Kazemi, Qianyi Sun, Boxing Chen, Dong Li, Xu He, Quan He, Feng Wen, Jianye Hao, and Jun Yao. MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time. In arXiv:2405. 16265, 2024.
[28] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. In arXiv:2001.088: 083611, 2020.
[29] Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative Agents for "Mind" Exploration of Large Language Model Society. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[30] Junkai Li, Siyu Wang, Meng Zhang, Weitao Li, Yunghwei Lai, Xinhui Kang, Weizhi Ma, and Yang Liu. Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents. In arXiv:2405.02957 2024.
[31] Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, and Yuke Zhu. Pre-trained Language Models for Interactive Decision- In Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS), 2022.
[32] Zhongyang Li, Xiao Ding, and Ting Liu, Generating Reasonable and Diversified Story Ending using Sequence to Sequence Model with Adversarial Training. In International Conference on Computational Linguistics (COLING), 2018.
[33] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations (ICLR), 2023.
[34] Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Animashree Anandkumar. Coach-player multi-agent reinforcement learning for dynamic team composition, 2021.
12

---

[35] Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh R N, Yihao Feng. Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, Ran Xu, Phil L Mui, Huan Wang. Caiming Xiong, and Silvio Savarese. BOLAA:
Benchmarking and Orchestrating LLM Autonomous Agents. In ICLR 2024 Workshop on Large Language Model (LJM) Agents, 2024.
[36] Zijum Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang, A Dynamic LLM-Powered Agent Network for Task- -rie-Oriette Agent Collaboration. In First Conference on Language Modeling (COLM), 2024.
[37] Do Xuan Long, Duong Ngoc Yen, Anh Tuan Luu, Kenji Kawaguchi, Min-Yen Kan, and Nancy F. Chen. Multi-expert Prompting Improves Reliability, Safety and Usefulness of Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024.
[38] Kaixin Ma, Hongming Zhang, Hongwei Wang. Xiaoman Pan, and Dong Yu. LASER: LLM Agent with State-Space Exploration for Web Navigation. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023.
[39] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uni Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. SELF- REFINE: iterative refinement with self- feedback. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS), 2023.
[40] Lina Mezghani, Piotr Bojanowski, Karteek Alahari, and Sainbayar Sukhbaatar. Think Before You Act: Unified Policy for Interleaving Language Reasoning with Actions, In Workshop on Reincarnating Reinforcement Learning at ICLR, 2023.
[41] Ali Modarressi, Ayyoob Imani, Mohsen Fayyaz, and Hinrich Schuetze. RET-LLM: Towards a general read-write memory for large language models. In ICLR 2024 Workshop: How Far Are We From AGI, 2024.
[42] Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), 2023.
[43] Zhenting Qi, Mingyuan MA, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual Reasoning Makes Smaller LLMs Stronger Sobllmm-ollvr, In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
[44] Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Communicative Agents for Software Development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), August 2024.
[45] Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling Large Language Model- based Multi-Agent Collaboration. In The Thirleenth International Conference on Learning Representations (ICLR), 2025.
[46] Shuofei Qiao, Runnan Fang, Ningyu Zhang, Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Agent planning with world knowledge model. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[47] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. TooILLM: Facilitating Large Language Models to Master 16000+ Real-World APIs. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[48] Alec Radford, Jeffrey WU, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language Models Are Unsupervised Multitask Learners. In OpenAl blog, 2019.
13

---

[49] Toran Bruce Richards. AutoGPT. In
2023.
[50] Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: accounting for inference in language model scaling laws. In Proceedings of the 41st Interna- tional Conference on Machine Learning (ICML), 2024.
[51] Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language Models Can Teach Themselves to Use Tools. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
[52] Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role play with large language models. In Nature, 2023.
[53] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang. Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. In arXiv:2402.03330, 2024.
[54] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[55] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk. Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[56] Mohit Shridhar, Xingdi Yuan, Mlee-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In International Conference on Learning Representations (ICLR), 2021.
[57] Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023.
[58] Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths. Cognitive Ar- chitectures for Language Agents. In Transactions on Machine Learning Research (TMLR), 2024.
[59] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning. In Cognitive Neuroscience, 1999.
[60] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Advances in Neural Information Processing Systems (NeurIPS), 1999.
[61] Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning. In Findings of the Association for Computational Linguistics (ACL), 2024.
[62] Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the Planning Abilities of Large Language Models A Critical Investigation. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
[63] Alexander Sasha Vezhnevets, Yuhuai Wu, Remi Leblond, and Joel Z. Leibo. Options as responses: Grounding behavioural hierarchies in multi-agent rl, 2020.
[64] Guanzhi Wang. Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An Open-Ended Embodied Agent with Large Language Models. In Transactions Machine Learning Research (TMLR), 2024.
[65] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. (RODE): Learning roles to decompose multi-agent tasks. In International Conference on Learning Representations (ICLR), 2021.
14

---

[66] Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang. Nebojsa Jojic, Eric Xing, and Zhiting Hu. PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization. In The Twelfth International Conference Learning Representations (ICI.R), 2024.
[67] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj. Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang. Rongqi Fan, Xiang Yue, and Wenhu Chen. MMI.U-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS), 2024.
[68] Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. HelpSteer2 Preffeence Complementing Ratings with Prefer- ences. In The Thirteenth International Conference on Learning Representations, 2025.
[69] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent Abilities of Large Language Models. In Transactions on Machine Learning Research (TMLR), 2022.
[70] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-off-hhoue prompting elicits reasoning large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022.
[71] Lilian Weng. LLM-powered Autonomous Agents. In lilianweng.githubbic Jun 2023.
[72] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversations. In First Conference on Language Modeling (COLM), 2024.
[73] Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game. In Proceedings of the 41st International Conference Machine Learning (ICML), 2024.
[74] Xue Yan, Yan Song, Xidong Feng, Mengyue Yang, Haifeng Zhang, Haitham Bou Am- mar, and Jun Wang. Efficient reinforcement learning with large language model priors. In arXiv:2410.07927, 2024.
[75] Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[76] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In shirrty-eventh Conference on Neural Information Processing Systems (NeurIPS), volume 36, 2023.
[77] Rui Ye, Shuo Tang, Rui Ge, Yaxin Du, Zhenfei Yin, Siheng Chen, and Jing Shao. MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems. In arXiv:2503.03668 2025.
[78] Siyu Yuan, Kaitao Song. Jiangjie Chen, Xu Tan, Dongsheng Li, and Deqing Yang. EvoAgent: Towards Automatic Multi- Agent Generation via Evolutionary Algorithms. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), April 2025.
[79] Zheni Zeng, Bangchen Yin, Shipeng Wang, Jiarui Liu, Cheng Yang, Haishen Yao, Xingzhi Sun, Maosong Sun, Guotong Xie. and Zhiyuan Liu. ChatMol: Interactive Molecular Discovery with Natural Language. In Bioinformatics, 2024.
[80] An Zhang, Yuxin Chen, Leheng Sheng, Xiang Wang, and Tat-Seng Chua. On Generative Agents in Recommendation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2024.
15

---

[81] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hui Chen, Jiaqi Chen, Mingchen Zhuge. Xin Cheng, Sirui Hong, Jinlin Wang. Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating Agentic Workflow Generation. In The Thirleenth International Conference on Learning Representations (ICLR), 2025.
[82] Andy Zhou, Kai Yan, Michal Shlapentokh- Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning language models. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.
2024.
[83] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: Realistic Web Environment for Building Autonomous Agents. In arXiv:2307 13854, 2024.
[84] Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang. Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Xiangru Tang. Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan. Agents: An Open-source Framework for Autonomous Language Agents. In arXiv:2309.077 07800, 2023.
[85] Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A. Rossi, Somdeb Sarkhel, and Chao Zhang. ToolChain Efficient Action Space Navigation in Large Language Models with A Search. In The Tweifth International Conference on Learning Representations (ICLR), 2024.
[86] Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. ToolQA: A Dataset for LLM Question Answering with External Tools. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS), 2023.
[87] Mingehen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. GPTSwarm: Language Agents as Optimizable Graphs. In Forty-first International Conference on Machine Learning (ICML), 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper elaborates the limitations in section D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes].
Justification:
The paper provide the method details in Method part.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: The code and data are provided in the supplementary materials under the file name code.zip.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips...
public/guides//CoeeSuuuue www.bmiisionooolyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (ht
/ fttips.cc/pbbl.. htc/public/guiie cttegggiiee///ooeSuuu seteesuumisssionnn wswonsolicyyyso for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details are listed in the method part. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We performed significance testing on the data presented in our table [ and we use the symbol in the table denote statistically significant results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The token cost is specifically discussed in Fig and the full results are listed in A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p/s//eerrrr...cooooooooooca
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consider- ation due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents technical contribution without immediate real-worls deployment, and thus discussion of societal impacts not applicable this stage.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper has no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper cites all the papers related.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, ppaper swi thcode. com/ cona/aatsets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper focuses LLM-bbsed agents, and the use of LLMs is clearly described as an essential and original part of the core method.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy https////eeerrooocccooooooeee/es for what should or should not be described.
23

---

Appendix
This appendix complements the main paper by presenting extended evaluation results, detailed implementation configurations of the agents, and a discussion of current limitations.
A
Supplementary Evaluation
A.1 Token Cost Analysis
As discussed in Section 3.2, we present here the full set of cost-related results under both the mono- agent and multi-agent settings, covering the Titan and Mimas subspaces. While the main text only reports the results for the multi setting on the Titan subspace, we provide here the remaining three sets of results for completeness: (1) mono on Titan, (2) mono on Mimas, and (3) multi on Mimas. These supplementary results allow for a more comprehensive comparison across different agent configurations and subspace settings. As discussed Section 3.2, we focus on two key metries: token consumption and the number of orchestrated agents per reasoning trajectory. While the multi setting has already been analyzed in the main paper, here we focus on the mono setting.
In the Titan subspace, the observed reduction in token consumption across most tasks in Figure can be primarily attributed to shorter reasoning paths. However, for GSM-hard tasks, the number of agents involved remains relatively stable. This suggests that the cost reduction may stem not from fewer agents but from selecting agents that are prompted with fewer tokens, or from utilizing more concise prompts for the same agents. In the Mimas space, token consumption in Figure & decreases for certain tasks, while for others, no consistent downward trend is observed. This can be attributed to the fact that, under the mono-agent setting, all agents share the same base model, making it more challenging to identify agents that are significantly more token-efficient.
A.2 Performance-Cove Trade-off Improvement
To demonstrate the overall improvement in reasoning performance along with the associated reduction attributed to the effectiveness of our reward design we track the performance score for each sample throughout the entire optimization process. This score is defined cost
in token consumption
as the ratio between task performance (e.g., accuracy success rate) and token cost, serving comprehensive indicator that jointly reflects solution quality and reasoning efficiency.
In the evaluation presented in the main paper, we divide the optimization trajectory into two phases: the initial phase and the evolved phase. This division introduced solely for evaluation convenience, and does not correspond to any intrinsic change in model behavior. In practice, the online optimization process exhibits generally monotonic upward trend in performance apart from minor fluctuations caused by the stochastic nature of model generation. This consistent improvement further substantiates the effectiveness of our reinforcement learning approach and the benefits of our reward shaping strategy under both mono-agent and multi-agent settings.
Across most tasks, from Figure 9 we observe a clear upward trend in the performance/cost score, indicating that the optimization process leads both better solutions and more efficient use of tokens. This improvement reflects the ability of the system to identify more suitable agent organizations and reasoning pathways over time. In some cases, the score increases steadily throughout the process, suggesting smooth convergence toward optimal or near-optimal multi-agent coordination strategies. In other cases, the score initially drops before rising again, which implies that the optimization process explores some less effective multi-agent configurations in early stages but is ultimately able to recover and identify improved structures through continued exploration. This pattern highlights the importance of allowing sufficient exploration in the early optimization phase.
However, for a small subset of tasks, the score does not show a significant upward trend throughout the optimization. We consider two possible reasons for this. First, the number of optimization samples used in our current experiments is limited to 200, which may not be sufficient to fully explore the search space or discover high-quality MAS configurations, especially for tasks with more complex reasoning requirements. Second, the current hyperparameter settings, such as the maximum length of agent sequence allowed, may not be well-suited for tasks involving weaker base models. In such cases, achieving high performance may require larger MAS configurations or higher upper bounds on relevant parameters to allow for more extensive coordination and tool usage.
24

---

Multi-base Titan Subspace
GSM-Hard
MMLU-Pro
CommonGenHard
SRDD
Trien Narts
Idean Can Lut Harte
Token Ool Seso
Telen lakan Card dize
Tikes Coll Trentiee
Takan Cos frandin
Lange
Chii Lengit Paiiza
Chan
Multi-base in Mimas Subspace
GSM-Hard
MMLU-Pro
CommonGenHard
SRDD
Titen 201
Toler Eos EU1S PACEL
Card ELD
Takn Dod Des Natl
feed
Lengt Turrdie
10
10
Us in
10 15
Mono-base in Titan Subspace
GSM-Hard
MMLU-Pro
CommonGenHard
SRDD Touer Con ireddin
Touse Cost lendina
lakan Coil Trnniine
Toban Cart lerld
Cwch Length Trandine UA do
Langh bnncie the
Chai Langh Fied 1he
Mose-base in Mimas Subspace
GSM-Hard
MMLU-Pro Tiun Out Trandie
CommonGenHard
Tiken Oeta Norts Taken Caal Tracheee
Takes Ceal Trencire
Takes Car Troeche
Chan Lange Des 7080s Lange
Chars
Figure 8: Evolution of token consumption and the number of orchestrated agents per task for all settings. Trends are fitted using LOWESS (Locally Weighted Scatterplot Smoothing).
25

---

Evolution Puppetser_mane
Srealie
(a) Puppeteer Mono under Titan subspace
Parformance Cost Trade-off Evolution Puppeteer under Titan Space
SDM-Hard
(b) Puppeteer under Titan subspace
Performance-Cout Trade Puppetoor, mono Space
Foa polt
(c) Puppeteer Mono under Mimas subspace
Performance- Cost Trade-off Evolution Puppetoor under Mimas Space
500%
Sear
5010
(d) Puppeteer under Mimas subspace
Figure 9: Performance- Cost curves representing the ratio between task performance and token cost per agent chain across different subspaces and variants of Puppeteer. Trend lines smoothed using LOWESS (Locally Weighted Scatterplot Smoothing).
These observations suggest that while our reward design provides strong guidance for optimizing agent behavior and token efficiency, further gains may be achieved by increasing the optimization budget or dynamically adapting hyperparameters based on task difficulty or model capability.
A.3 Emergent Patterns in MAS Behavior
To further illustrate the behavioral diversity and coordination capabilitics our multi-agent system (MAS), present visualizations of several representative MAS trajectories guided by the puppeteer. As shown in the figure, these trajectories reveal a variety emergent patterns, including both previously validated structures and novel organizational behaviors that arise from agent interactions.
The visualizations in Figure 10 highlight how the policy organizes into distinet formations and adapt agents dynamically in response to high-level guidance. These patterns not only reflect the effectiveness of the puppeteer orchestrating agent collaboration, but also demonstrate the capacity of the system to generalize and generate new behaviors beyond those explicitly encoded in the training process. Such structural emergence further underscores the interpretability and robustness of our MAS design.
26

---

Beossringiger,
Temiiattrr
(a) Chain-of-Thouue
(b) Sell-Reflective Reasoning Chain
Reasoning then Termination
laracctggrt, diney--SSpp Meleetfarri Salmel
Cancuder Giten2277
(d) Two-Agent Communication
(e) Three-Agent Communication
(f) Four-Agent Communication
Pythnnigrt G1740mi
Mudthorkger, S1142m1s
OrasAgan Liilloh
Coedadentgnnn Caste
Withe Naadgen
Refleciggn Coecer
(g) Convergent Pattern
(h) Divergent Pattern
Complex Topology
Figure 10: Examples of MAS behavior patterns under the puppeteer's guidance.
A.4 Generalization to Embodied Environments
do not While our main experiments focus on tasks such as reasoning, coding, and involve interaction with external environments-wwe further demonstrate the applicability of our framework to embodied tasks. These tasks require agents to interact with dynamic environments and perform sequences of actions based on real-time feedback, thus posing additional challenges in planning. reasoning, and decision-mant makiin...
To showcase this capability, we employ ALFWorld (56) as the testbed. ALFWorld integrates simulated embodied environment with natural language interfaces, making it suitable benchmark that bridges textual reasoning with embodied interaction. In this setting, we configur guri the agent chain with maximum length of 50, as embodied tasks typically require multiple steps to complete. We also limit the number of exploration trajectories to 1, duc to environment constraints: execution cannot be parallelized, and only one admissible action is allowed per step. This experiment conducted under the mono setting, where all agents share the same model backbone (GPT-4o-mini).
To illustrate how our system operates embodied settings, we provide case study visualization using frames saved from the THOR enviromment (as shown in Figure IT]. This example showcases step-by-step execution
single task by the agent chain. The successful application of our framework in this setting not only underscores its flexibility across diverse task types, but also demonstrates extensibility to interactive environments, confirming its potential for integration into broader embodied agent systems that require real-time perception, planning, and execution.
27

---

Figure 11: The figures illustrate the implementation process of the task "look for the remote control under the floor lamp."
B Agent Implementation Details
B.1 Agent Configuration
We organize all agent actions into two categories: Tool-use Agent 44
and Reasoning For tool-use agents, their actions involve interacting with external environments, such querying the web invoking code interpreter. In contrast, reasoning patterns reflect the internal cognitive processes of LLMs. Drawing inspiration from prior studies human reasoning strategies 43 27 21,, revisit how humans approach complex reasoning tasks. In practice, individuals adopt diverse strategies: some decompose problems into sub-questions, others solve them directly, and some reframe the problem emphasize critical constraints. More importantly, humans dynamically adapt their strategies based on the evolving context and problem state. Motivated by these observations, we accordingly design set of specialized reasoning agents emulate diverse human strategies. Each agent equipped with a distinct cognitive role, contributing collaborative and adaptive reasoning process.
85 51
Agent [3] as seen Table 2
B.2 Agent Prompts
In multi-agent system, motivate each agent in the desired manner, we define its role through a role prompt and combine it with an action prompt to guide the agent in using tools or following specific reasoning patterns. As inllustrated in Fig 12 and Fig 13 role-conditioned prompts specify the agent's identity, domain expertise, and intended responsibilities, guiding the behavior of different agents. And we have designed set of structured prompts to guide the behavior of our prompt-based agent system, which can be categorized into two major types: tool-augmented action prompts and reasoning-mode prompts. Each prompt specifies a well-defined agent action, accompanied by template for language model invocation.
28

---

Table 2: Agent Tool-Use and Reasoning Pattems
Category
Name
file
search_arxiv
search_bing
Tool-Use
access_website
run_python
reasoning
critique
reflect
question
Reasoning Patterns
summarize
conclude
modify
planning
Poollaugmented action prompts in Fig 14 are designed to determine the parameters required for invoking external tools through the agent's s environment interface. The goal of these prompts not to execute the tool directly but to generate structured output that specifies the parameters for tool invocation. For example, such prompts may guide the generation of a search query (e.g., a paper title or keyword) when retrieving academic papers from arXiv aearch_arxiv or determine the URL to be accessed when invoking a web tool access seriss_webiite te. These prompts enable the agent to interface with external systems in modular and verifiable way by generating the precise input needed to trigger tool execution.
Reasoning-pattems prompts in Fig 15 and Fig 16 focus on internal cognitive processes, such as planning, reasoning, critique, reflection, or sub-question generation. These prompts do not rely on external tools but activate structured thinking patterns within the model, enabling complex problem-solving and decision-mate makiin through purely generative means.
C Computational Resources
Resource
Specification
GPU
NVIDIA A800, GPUs used 28.8-78, GB per GPU
Peak GPU Memory Usage
Training Time
2-6 hours
Variation Source
Benchmark differences and task complexity
Table Computational resources used for orchestrator training.
The table (Table 3) details the average computational resources used for orchestrator training. important to note that accurately measuring training costs is non-trivial. Our orchestrator is trained online, with parameter updates interleaved with multi-agent inference, making it difficult to isolate GPU hours solely for training. Moreover, several baselines (e.g., AFlow) rely LLM- based inference- sirrencce-iieeeeee search to construct workflows, whereas our method uses online gradient- optimization. Since these involve fundamentally different resource modalities, direct comparisons of computational cost are not appropriate.
D Limitations
Our work introduces a centralized mechanism, Puppeteer, to organize and optimize multi-agent systems (MAS), progressively improving reasoning performance, and efficiency. However, several limitations remain. The Puppeteer's optimization currently depends on gaasse-rrained rewards based only final outputs and token usage, lacking informative intermediate feedback. Incorporating fine-

---

grained supervision, such as step-level correctness, could enhance optimization efficiency. Moreover, the framework assumes fixed set of agents and tools, limiting adaptability and responsiveness task variations. Enabling dynamic agent tool modification during inference would improve flexibilityy and robustness. Finally, occasional mis-coordination or deceptive agreement among agents suggests the need for more robust interaction protocols and incentive designs. Future efforts may focus on reward shaping and adaptive mechanisms to refine both orchestration and agent-level behaviors, allowing the Puppeteer to make more context-aware and efficient decisions.
30

---

Figure 12: Tool-use Agent Role Prompts
FileAgent
You are expert in file handling, Responsible for reading files and extracting relevant information (read_file).
FileAgent:
ArxivAgent
You are an expert academic research. Responsible for searching relevant papers on arXiv (search_arxiv).
ArxivAgent:
BingAgent
You are expert in web search. Responsible for retrieving information via Bing (search_bing).
BingAgent:
WebsiteAgent
You expert in accessing and parsing websites. Responsible for extracting data from specific URLs
WebsiteAgent:
PythonAgent
PythonAgent: You are an expert in Python programming, Responsible for executing Python code and returning results (run_python).
31

---

Figure 13: Reasoning Agent Role Prompts
PlannerAgent
You are an expert in task decomposition and planning. Responsible for generating structured plans solve complex tasks (planning).
PlannerAgent:
ReasoningAgent
ReasoningAAentt You are an expert in logical reasoning. Responsible for synthesizing solutions to sub-problems (reasoning).
CriticAgent
You are an expert critique and verification. Responsible for identifying flaws and providing feedback on prior reasoning (critique).
CriticAgent:
ReflectAgent
You expert in metacognitive reflection. Responsible for analyzing the overall reasoning trajectory and proposing improvements (reflect).
ReflectAgent:
QuestionAgent
You are expert in problem decomposition. Responsible for generating clarifying or follow-up sub-questions (question).
QuestionAgent:
SummarizerAgent
SummarizerAgent: You are an expert summarization. Responsible for generating concise summaries of intermediate results (summarize).
AoneluderAgent
You are expert in synthesis. Responsible for producing the final conclusions based on collective reasoning outcomes (conclude).
ConcluderAgent:
ModifierAgent
ModifierAgent:
You are expert in error analysis and correction. Responsible for identify- ing errors and revising prior outputs accordingly (modify).
32

---

Figure 14: Tool-use Prompts
search arxiv
You have chosen search for academic papers arXiv. Please provide specific terms related to academic research, such as the title of paper, keywords, or topics in fields like physics, mathematics, computer science, or machine learning. Return json format. Example: ("action": "search_arxiv", "parameter": "quantum computing")
search wiki
You have chosen to search for information on Wikipedia. Please provide specific terms like concept, name, event, technical term for best results. Return in json format. Example: ("action": "search_wiki", "parameter": "Albert Einstein")
search_bing
You have chosen to search for information using Bing. Please provide descriptive phrases or keywords related to your query, including concepts, names, events, or specific questions to a broad range of results, including news, articles, and websites. Return in json format. Example: ("action": "search_bing", "parami amrtter: "latest advancements in AI"}
access website
You have chosen to access a website. Please provide the URL you want to access or the URL most relevant to the current question. Return in json format. Example: ("action": "access_websitt",, "parameter": "https:/ //wps..///ww.ee// www.eexampl.... com")
run_pyye umn_python
You have chosen to write and run Python code. Please write generic Python code in the parameter solve this type of problems using only standard python libraries. Make you use the print function for all output when relevant, Return in json format. Example: ("action": "run_python", "parameter": sprint('Helll) World!')")
read_file file
You have chosen to read file. Please provide the filename you want to read. Return in json format. Example: ("action": "read_file", "parameter": "data. txt")
33

---

Figure 15: Reasoning prasoning-pattern Prompts (Part 1)
planning
Decompose the question and plan the next steps to address the question. You should complete your planning using the following template:
REASONING RESULT: [YOUR REASONING RESULT]. *Your previous reasoning was:
().
Your planning should include:
reasoning
Now, you need to continue the reasoning get closer to the correct answer. You should finish your reasoning with the following template:
REASONING RESULT: [YOUR REASONING RESULT].
Finish your answer with:
FINAL ANSWER: [YOUR FINAL ANSWER]. *Your previous reasoning was: O).
You need to follow the direction of the reasoning path and go forward:
critique
You need to critique the previous reasoning. Complete your reasoning using: REASONING RESULT: [YOUR REASONING RESULT].
Conclude with:
FINAL ANSWER: [YOUR FINAL ANSWERJ. xYour previous reasoning was: O.*
Consider the following when critiquing: 1. Plausibility:
reflect
You will be provided with previous reasoning attempt where you had access to relevant context and were tasked with answering question. The attempt unsuccessful either due to an incorrect answer or a phrasing mismatch with the answer key. In few sentences, diagnose the potential cause of failure discrepancy, and outline new, concise, high-level plan to prevent the same issue. Use complete sentences.
Reflect on the current state of the task and propose the next steps. Conclude with:
REASONING RESULT: [YOUR REASONING RESULT].
FINAL ANSWER: [YOUR FINAL ANSWERJ. *Your previous reasoning was: O)*
question
Your task is to propose the next sub-question along with its answer. Ensure it logically follows from the previous reasoning and addresses any gaps.
Provide a well-reasoned answer, supported by evidence or logical arguments.
Conclude with:
REASONING RESULT: [YOUR REASONING RESULT].
FINAL ANSWER: [YOUR FINAL ANSWER]. *Your previous reasoning was: 0.*
34

---

Figure 16: Reasoning-patter Prompts (Part 2)
summarize
You need summarize previous results and provide some intermediate conclusions. Finish your reasoning with:
REASONING RESULT: [YOUR REASONING RESULT].
Then:
FINAL ANSWER: [YOUR FINAL ANSWERJ. *Your previous reasoning was: O.*
Summarize the reasoning paths and provide final conclusion.
conclude
You need conclude the task and provide final answer.
Finish with:
REASONING RESULT: [YOUR REASONING RESULTI T].
Then:
FINAL ANSWER:
[YOUR FINAL ANSWER]. *Your previous reasoning was: O
modify
You need to identify and correct errors in the previous reasoning.
Use this template:
REASONING RESULT: [Clearly state: 1. Which part of the previous reasoning incorrect 2. Why it was incorrect 3. What is the correct understanding).
Then:
FINAL ANSWER: [Provide the complete corrected answer]. *Your previous
reasoning was: ().* *
Please explicitly point out and correct any errors, misconceptions, inaccuracies.
35


Reviews:
[{"review_id": "rKKOLJvSGe", "reviewer": "Reviewer_dA6i", "summary": "* Aims to learn a dynamic orchestrator for LLM-based multi-agent collaboration, using a \"puppeteer-style\" paradigm", "questions": "* I'd like to know how this work is related to works in MARL coordination; e.g. ALMA [1], which is a similarly hierarchical approach which also performs \"orchestration\" (task allocation) dynamically; it seems that there's probably some overlap of ideas (if not with ALMA, then more broadly in that literature)\n\n[1] https://arxiv.org/abs/2205.14205", "limitations": "yes, Appendix E; appears comprehensive, and encapsulates my thoughts", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n* I like the idea of evolving an orchestrator; I agree that many paradigms for coordination are too rigid\n* The paper is well-written; clear and easy to follow; the illustrations are very useful as well\n* The evaluation tasks seem reasonable, and the performance improvements are clear\n* Extensive interpretable analysis of experiments\n\nWeaknesses\n* I can't help but feel that there must be much more literature on this kind of orchestration that is not referenced in the paper (see my question below); especially pre-LLM multi-agent RL (MARL) literature", "was_revised": true, "final_justification": "I stand by my initial review and therefore maintain my current score.", "rebuttal": "**Thank you for your insightful comment and kind recognition.**\\\nOur paper mainly includes related works that leverage reinforcement learning to optimize and orchestrate LLM-based agents. While our current work primarily focuses on LLM-based multi-agent systems, and thus does not emphasize pre-LLM MARL literature extensively, we fully acknowledge that many of the coordination principles and theoretical foundations established in the MARL community continue to inspire the design of LLM-based agents, such as ALMA[1], RODE[2], and other[3,4]. In the revised version, **we will incorporate relevant citations and explicitly highlight both the inspirations and distinctions between classical MARL and emerging LLM-based paradigms.**\n\n**References**\\\n[1] ALMA: Hierarchical Learning for Composite Multi-Agent Tasks. Conference on Neural Information Processing Systems (NeurIPS), 2022.\\\n[2] RODE: Learning Roles to Decompose Multi-Agent Tasks. International Conference on Learning Representations (ICLR), 2021.\\\n[3] Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition. International Conference on Machine Learning (ICML), 2021.\\\n[4] Options as Responses: Grounding Behavioural Hierarchies in Multi-Agent Reinforcement Learning. International Conference on Machine Learning (ICML), 2020."}, {"review_id": "i45m3ukpkh", "reviewer": "Reviewer_T1e8", "summary": "Instead of wiring up a brittle, pre-built tree of chatbots and hoping for the best, the authors of “Multi-Agent Collaboration via Evolving Orchestration” teach a reinforcement-trained “Puppeteer” to glance at the shared scratch-pad, pick whichever specialist LLM seems most useful at that moment, and then file away a mental note about whether the detour was worth the tokens. Over a few thousand trial-and-error runs the policy quietly sidelines agents that don’t pull their weight, trims bloated reasoning chains, and even discovers tight feedback loops where two or three “puppets” bounce ideas until the answer sticks, lifting success rates on GSM-Hard, MMLU-Pro, software debugging and creative writing while doing so. Crucially, those gains arrive hand-in-hand with a 20–40 percent drop in token spend because the reward function values thrift as much as quality. When the dust settles, the learned chat graph is noticeably tighter—messages swirl around a couple of hub agents in short cycles—which the authors argue makes the whole affair cheaper to run and kinder to GPUs. And since one knob lets practitioners trade a bit of accuracy for a bit of frugality, the study hints that a well-drilled team of modest models, steered by this lean conductor, can rival a lone heavyweight LLM without breaking the budget.", "questions": "- I’d like to see a clearer picture of the RL scheduler’s computational footprint: specifically, how many wall-clock hours were spent training it, which GPU models were used, what peak memory requirements were, and the total number of tokens it processed.\n- It would be valuable to know whether this scheduling strategy holds up outside of purely text-based benchmarks—for example, on code-generation tasks like HumanEval or in interactive environments such as WebArena.\n- The paper reports using a single cost-weight λ for each task but doesn’t explain how it was chosen. Could the authors discuss how sensitive their results are to different values of λ, as well as the constants F and φ in Eq. (6)?", "limitations": "The Discussion section currently only alludes to “future work” and skips over key limitations and risks. At minimum, the authors should:\n- report GPU-hours, hardware specs, and energy use for the RL scheduler.\n- discuss issues like mis-coordination or deceptive agreement and note that the method hasn’t been tested in interactive or safety-critical settings.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The scheduler demonstrates impressive, cost-aware improvements: across four diverse benchmarks, it boosts accuracy by 6–12 percentage points while significantly reducing token usage, showing that thriftiness is truly embedded in its reward function.\n- Visual analyses highlight how the system gravitates toward a small set of high-value “hub” agents and benefits from rapid feedback loops—offering a clear explanation for simultaneous gains in both accuracy and efficiency.\n- The findings indicate that a carefully coordinated ensemble of smaller models can match the performance of a single, massive LLM like Titan, but at only a fraction of the inference expense—an especially practical result for teams wrestling with soaring token costs.\n- Unlike previous approaches that either use a fixed graph structure (e.g., GoT) or train each agent in isolation (e.g., Reflexion), this work introduces a learnable metacontroller that dynamically selects between different LLMs at each step, which represents a genuinely novel contribution.\n\n**Weaknesses**\n- While the paper quantifies inference savings, it fails to disclose how many GPU-hours were required to train the scheduler itself; without this information, it’s difficult to assess the true net efficiency.\n- It remains unclear what portion of the observed gains stems from the RL-based policy versus simpler strategies, such as token-cost thresholds or fixed priority lists.\n- The appendix briefly mentions a “Bradley-Terry + 3-layer MLP” architecture with an 8,000-dimensional input, yet offers no rationale for this design choice and omits details on the policy network’s parameter count.", "was_revised": true, "final_justification": "The rebuttal addressed most of my concerns, and I will maintain my score.", "rebuttal": "**We sincerely thank you for your thoughtful review and these insightful questions, which greatly helped us improve the clarity and depth of our work. We will address your comments point by point below. And due to the strict length limitation,  we would be happy to provide further details or clarifications upon your response.**\n## To weakness 1 and question 1:\nThe following table details the computational resources used for orchestrator training:\n| Metric                     | Value                                           |\n|----------------------------|-------------------------------------------------|\n| GPUs Used                  | 8 × NVIDIA A800 80GB                            |\n| Training Time (Wall-clock) | 2–6 hours                                       |\n| Peak GPU Memory Usage      | 28.8 GB – 78.4 GB per GPU                       |\n| Variation Source           | benchmark differences and task complexity       |\n\nAccurately measuring the training cost is inherently difficult due to:\\\n(1) Online Training: As noted in lines 112–117 and 205–207, our orchestrator is trained online, interleaving parameter updates with multi-agent inference, making it hard to isolate GPU hours purely for training.\\\n(2) Incomparable Cost Modalities: Baselines like AFlow (lines 352–353) perform inference-time search using LLMs, whereas ours uses training-time optimization via gradients. These fundamentally differ in resource type and cannot be compared directly.\n## To weakness 2:\nEffective orchestration in LLM-powered multi-agent collaboration must jointly optimize both efficiency (e.g., token usage) and performance (e.g., reasoning quality). While heuristics such as token thresholds can provide partial cost savings, they fail to capture this trade-off adequately. Inference-time compute can improve individual agent outputs [1], and involving more agents often enhances collective problem solving [2], but both come at increased cost—necessitating more nuanced, context-aware strategies.\nStatic schedules (e.g., fixed agent priorities) are fundamentally limited due to the dynamic and context-sensitive nature of LLM interactions. An agent’s utility may vary across different task phases [3, 4], and LLM outputs are inherently stochastic and dependent on evolving context [5, 6], making static heuristics brittle across diverse scenarios.\\\nTherefore, robust orchestration requires adaptive mechanisms that dynamically adjust agent priorities based on contextual signals rather than fixed rules. In this light, **we argue that truly effective multi-agent orchestration cannot be static, it must be dynamic and context-aware.**\\\nTo enable such adaptability, the orchestrator must rely on rich contextual cues, such as intermediate outputs, task progress, and interaction history, which reflect the current state of reasoning [7, 8, 9]. These signals are essential for deciding when to continue, halt, or re-route agent actions. To leverage them effectively, we adopt reinforcement learning, which enables the orchestrator to interpret and act upon these nuanced states (lines 91–92). As a result, our orchestrator dynamically adjusts its strategy based on task requirements, reward signals, and the evolving reasoning states—capabilities fundamentally beyond the reach of static heuristics (lines 91-95, lines 106–117).\\\nAnd we evaluate our method against strong non-RL baselines, including AFlow (MCTS-based) and MacNet (static topologies). As shown in Table 1, our RL-based orchestrator consistently outperforms them across benchmarks, and our work empirically validates RL as an effective approach for orchestrating LLM-powered multi-agent collaboration.\\\n**References**\\\n[1] Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. arXiv preprint, 2024.\\\n[2] Scaling Large Language Model-based Multi-Agent Collaboration. ICLR, 2025.\\\n[3] Why Do Multi-agent LLM Systems Fail? ICLR Workshop, 2025.\\\n[4] Generative Agents: Interactive Simulacra of Human Behavior. UIST, 2023.\\\n[5] ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs. EMNLP, 2024.\\\n[6] Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning. NeurIPS, 2023.\\\n[7] Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game. ICML, 2025.\\\n[8] Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use. EMNLP, 2024.\\\n[9] Efficient Reinforcement Learning with Large Language Model Priors. arXiv preprint, 2024.\n## To weakness 3:\n**Clarification of Our Orchestration Policy Design**\\\nOur orchestration policy uses a pretrained language model under the Bradley-Terry framework—nvidia/Llama-3.1-Nemotron-70B-Reward-HF (see footnote of line 197)—paired with a lightweight three-layer MLP for agent selection. The model acts solely as a semantic encoder: we extract the final hidden state of the last token (an 8192-dimensional embedding) and feed it into an MLP with hidden sizes 512, 128, and 32 to output a score distribution over agents (see lines 997–1007). This setup leverages the model’s strong semantic representations while maintaining efficient, modular decision logic.\\\n**Why This Model and Architecture?**\\\nWe choose Llama-3.1-Nemotron-70B-Reward-HF because it is trained specifically for quality assessment in multi-turn conversations, making it highly effective at encoding fine-grained reasoning signals, and can effectively distinguish between high-quality and low-quality reasoning in generated text [1]. It also achieves state-of-the-art performance on multiple automatic alignment benchmarks (e.g., AlpacaEval 2 LC, Arena Hard, MT-Bench) as of October 2024, surpassing models such as GPT-4o and Claude 3.5 Sonnet [2], thus making it suitable to be our policy's language backbone.\\\nWe use the final hidden state of the last token—a well-established practice in decoder-only architectures like LLaMA/GPT—as a compact semantic summary of the input. This embedding is projected by a simple three-layer MLP (512 → 128 → 32), enabling dimensionality reduction and feature abstraction, and culminating in a distribution over candidate agents. Similar projection heads are widely used to transform high-dimensional contextual inputs into task-relevant decisions [3-6]. And empirically, this policy network consistently outperforms all baselines across benchmarks (Table 1), validating both our model choice and architecture design.\\\n**References**\\\n[1] HelpSteer2-Preference: Complementing Ratings with Preferences. ICLR, 2025.\\\n[2] NVIDIA. Llama-3.1-Nemotron-70B-Reward-HF. Hugging Face, 2024. \\\n[3] iVideoGPT: Interactive VideoGPTs are Scalable World Models. NeurIPS, 2024.\\\n[4] InstructEval: Instruction-Tuned Text Evaluator from Human Preference. ACL, 2024.\\\n[5] Defending against Indirect Prompt Injection by Instruction Detection. arXiv:2505.06311, 2025.\\\n[6] Hugging Face Transformers. LlamaForSequenceClassification Documentation. \n## To question 2:\nWe agree that testing generalizability beyond text-only tasks is important. To this end, we evaluate our method on both code-oriented and embodied interaction tasks. Specifically, we include SRDD [1], which requires generating implementation code from software specifications (see Table 1), and ALFWorld [2], which involves multi-step planning and state tracking in simulated environments (see Section C.4 in appendix). Our method performs well on both, demonstrating strong generalization from language to structured, action-driven domains.\\\n**References**\\\n[1] ChatDev: Communicative Agents for Software Development. ACL, 2024.\\\n[2] ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. ICLR, 2021.\n## To question 3:\n**For cost-weight λ:**\\\nThe cost-weight λ controls the trade-off between accuracy and efficiency. A larger λ emphasizes reducing computational cost, potentially at the expense of task performance, whereas a smaller λ prioritizes accuracy but may incur higher computational cost. In extreme settings, an overly large λ can lead to premature termination or under-reasoning, while an excessively small λ may cause unnecessary computation and neglect efficiency. \\\nOur sensitivity analysis (in Figure 3) shows (1) λ effectively controls this trade-off, (2) our method is robust within a reasonable range of  λ (e.g., 0.03 and 0.1).\\\nBased on these findings, we did not perform exhaustive tuning for λ in each task or setting. Instead, we adopted empirically reasonable values (see Table 3 and Appendix B.3) that ensure balanced performance. While λ was not optimized per task, we believe further task-specific tuning or broader hyperparameter search could enhance the orchestrator’s performance.\\\n**For F and φ:**\n- F represents the actual computational cost at each step, measured by real usage metrics such as FLOPs or token count. It is not a tunable parameter but is instead computed from the agent’s execution trace.\n> Typo Correction: We mistakenly referred to F as a “scale factor” in the manuscript — this was a typo. F denotes actual computational cost (e.g., FLOPs or token count). We will correct it in the revised version.\n- φ appears inside a logarithmic term and acts as the denominator of the step count. Typically, φ is set to the maximum episode length. This allows the logarithmic penalty to grow progressively, effectively penalizing prolonged trajectories: at the final step, it reaches log(2), while at step 1, it is near log(1 + 1/φ), which is negligible. Figure 7 analyzes φ’s impact on total cost and success rate. In our experiments, φ is chosen via empirical heuristics and a lightweight grid search (see Table 3 in Section B.3).\n## To limitation:\nThanks for the suggestion — we will include all the mentioned points in the revised version, including computation cost and potential limitations in multi-agent collaboration such as mis-coordination and deceptive agreement."}, {"review_id": "pgTngPLIZc", "reviewer": "Reviewer_XZCy", "summary": "This paper addresses the organizational structure of multi-agent collaboration for LLMs. The authors argue that prevailing static collaboration topologies suffer from limitations in adaptability and efficiency. To overcome this, they propose a \"puppeteer-style\" paradigm where a centralized policy, optimized via reinforcement learning, dynamically selects which agent to activate at each timestep based on the evolving task state. The approach formulates the collaboration process as a sequential decision problem, aiming to dynamically generate flexible and efficient reasoning graphs. Experimental results on several closed- and open-domain tasks show that this method improves performance while reducing computational costs compared to a range of baselines.", "questions": "See above.\n\n1. Could you please elaborate on the specific formulation and aggregation method for the global state? How does the system manage the growing context history from numerous agents and steps into a fixed-size vector for the policy network, and how does this approach scale?\n\n2. Have you explored more advanced RL algorithms like PPO, and would you expect them to yield different results?\n\n3. Will a decentralized or hierarchical approach become more effective at a certain scale?\n\n4. The paper states that the reward considers *computational resource consumption at each step*, but Equation 6 defines the step-wise cost purely as a logarithmic function of the step number t, not the actual tokens or FLOPs consumed in that specific step. Could you clarify this discrepancy?\n\n5. The multi-agent baselines like MacNet and EvoAgent are presented in a mono-model configuration. Does this not provide your method with a significant unfair advantage by having access to a more diverse set of capabilities?\n\n6. Increasing exploration width or depth from the default setting leads to worse accuracy, not just higher cost. This is counter-intuitive. What is your hypothesis for this performance degradation? Does it suggest a fundamental issue with credit assignment in your RL algorithm when faced with a larger action space.\n\n7. The method section specifies that for the policy network, a pretrained language model's parameters are kept frozen, and only a small MLP head is trained. What was the rationale behind freezing the entire base model? This will limit the ability of the puppeteer.\n\n8. In the ALFWorld generalization experiment, the exploration width was constrained to 1. This effectively reduces the orchestrator's role to generating a single, linear sequence of actions rather than dynamically managing multiple reasoning paths.", "limitations": "Yes.\n\nSee Weaknesses.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper provides a clear and compelling motivation, identifying the rigidity of static collaboration in LLM-based multi-agent systems as a significant and timely problem.\n\n2. The *puppeteer* analogy is highly intuitive, and framing the complex collaboration problem as a centralized, learnable sequential decision process is conceptually clean.\n\n3. The reward function, which considers both task performance and computational cost, is a major strength, guiding the system towards more practical and economical solutions.\n\n4. The analysis of hyperparameters like chain depth and exploration width provides practical guidance for applying the method.\n\n\n**Weaknesses**\n\n1. The idea of a central controller or router agent is not entirely new in the multi-agent space. The paper needs to better articulate the fundamental conceptual leap from prior dynamic orchestration works.\n\n2. Centralized Bottleneck. The orchestrator itself can become a computational and decision-making bottleneck as the number of agents and the complexity of the state space increase.\n\n3. The use of the basic RL algorithm, coupled with a small number of optimization samples. It raises concerns about the stability, convergence, and reproducibility of the training process. It might be prone to overfitting.\n\n4. The efficiency analysis focuses only on inference-time token consumption. The significant computational cost of the RL training phase itself is not reported, which is a key part of the total cost compared to static methods.\n\n5. Simpler Dynamic Baseline: The authors should conduct a comparison against a simpler, non-RL dynamic baseline (e.g., using a single LLM call to choose the next agent). This would help isolate the specific benefits gained from the complex RL framework.\n\n6. The paper doesn't adequately argue why a centralized approach is inherently superior to dynamic strategies. It assumes a god's-eye view for the orchestrator, which may be unrealistic due to the information bottleneck of the aggregated state.\n\n7. The core assumption that agent selection requires a complex reinforcement learning solution is not sufficiently challenged. Simpler dynamic approaches, like supervised routing or greedy selection by a powerful LLM, are not explored as baselines, making it hard to justify the added complexity of RL.\n\n8. The specific logarithmic form of the cost penalty is an arbitrary design choice without theoretical or empirical justification.\n\n9. The paper claims that emergent compact and cyclic structures are the reason for performance gains. However, this establishes a correlation, not causation. The argument lacks rigorous proof.\n\n10. This design is incomplete. Only the orchestrator learns and adapts, while the agents themselves remain static. A truly adaptive system might be expected to evolve both its organizational structure and the capabilities of its individual components.\n\n11. **Misleading Reproducibility Claims**: The authors explicitly state in their checklist that *The code and data are provided in the supplementary materials under the file name code.zip.* However, is absence of this file in the actual submission. It is impossible to reproduce the experiments and verify the paper's central claims, severely impacting the work's credibility.", "was_revised": true, "final_justification": "The paper overall gives the impression of being hastily completed, with issues such as declared files not existing and poor article quality. I believe the article still requires significant revision and optimization. Additionally, the author's response did not effectively address my concerns. Therefore, I tend to reject.", "rebuttal": "**We sincerely appreciate your thoughtful review. Below we address each point; due to length constraints, please feel free to ask for any further clarification.**\n## To w1:\nMost routing-based or central controller approaches perform one-shot, static orchestration, selecting a fixed agent or model based on the initial input [1,2]. In contrast, we propose step-wise, context-aware orchestration, where a learnable policy operates over the language state to make dynamic, fine-grained agent decisions throughout execution. While our design uses a centralized orchestrator, the core novelty lies out of centralization pattern, but in how orchestration is **contextually modeled and jointly trained via RL**, surpassing prior hand-crafted [3,4] or template-based [5] method.\n## To w2, w6 and q3:\nWe adopt a centralized orchestration policy not out of a theoretical claim, but for practical reasons. This has a classical trade-off in multi-agent: centralized control facilitates coordination but risks bottlenecks; decentralized control offers scalability but often suffers from instability and communication overhead [6-8].\nInspired by blackboard architecture [9], we employ a learnable centralized policy over a shared language state to:\n- Integrate Global Context: The language state encodes task semantics and agent outputs in a compositional manner, enabling information sharing with minimal loss.\n- Stabilize Training and Improve Scalability: Centralized control avoids the non-stationarity and synchronization issues common in decentralized learning, and scaling to more agents only expands the output space rather than system complexity.\n\nWe acknowledge that centralized orchestration may not suit all scenarios, and view our approach as a proof of concept for language-based coordination. Future work may explore decentralized extensions.\n## To w3 and q2:\nOur focus is not on analyzing the convergence or optimization of RL algorithms themselves, but on demonstrating the effectiveness of a language-based, RL-optimized orchestration mechanism in multi-agent collaboration. \nSo we adopt REINFORCE, which offers several advantages in our context:\n- It avoids confounding factors from auxiliary components like critic networks or clipping.\n- Its simplicity provides a clean testbed to isolate and evaluate the orchestration design’s contribution.\n-  REINFORCE has also shown effective in LLM post-training tasks [10], supporting its suitability for language-driven learning scenarios.\n\nAlthough we have not explored more advanced algorithms that may improve sample efficiency or training stability, we consider these enhancements **orthogonal** to our core contributions. Our results demonstrate that a basic RL algorithm enables to achieve robust performance across diverse tasks. We view more advanced algorithmic approaches as a promising direction for future work.\n## To w4:\nThe computational resources used for orchestrator training are as follows:\n| Metric                 | Value                     |\n|-|-|\n| GPUs                   | 8 × NVIDIA A800 80GB      |\n| Training Time (Wall-clock) | 2–6 hours             |\n| Peak GPU Memory Usage   | 28.8 GB – 78.4 GB per GPU |\n| Variation Source       | Benchmark differences and task complexity |\n## To w5 and w7:\nWe use RL to enable dynamic and learnable orchestration, in contrast to static approaches like greedy LLM routing or supervised prompting that cannot **adapt to evolving contexts or leverage trajectory-level feedback**. To avoid added complexity, we adopt a simple and transparent RL method, REINFORCE, to demonstrate an effective formulation for multi-agent orchestration. And our evaluation includes strong baselines spanning search-based (AFlow), evolutionary (EvoAgent), and static orchestration strategies (MacNet), which represent the main paradigms of multi-agent collaboration. Our method consistently outperforms all of them (see Table 1).\n## To w8 and q4:\n**Why logarithmic form?**\\\nWe adopt a logarithmic step-wise cost $\\log(1 + \\frac{t}{\\tau})$, which is monotonically increasing to penalize longer reasoning, bounded to prevent excessive penalties, and exhibits sublinear growth with diminishing marginal penalties, meaning the cost rises slower than linearly and each additional step adds progressively less penalty. This balance supports efficient reasoning without imposing a hard limit on trajectory length.\n\n**About the cost item**\\\nWe apologize for the confusion—F was incorrectly described as a “scale factor.” It actually denotes the computational cost at each step (e.g., FLOPs or token usage). We will correct this in the manuscript. Thank you for your careful reading.\n## To w9:\nWe clarify that our work does not aim to establish a causal link between emergent topological structures and performance gains, but rather reports descriptive patterns that arise during dynamic orchestration. And similar structures have been observed in prior work (e.g., Reflexion), emphasizing the role of cyclical information flow, and dense hubs. Our findings align with these observations, reinforcing the relevance of such patterns in emergent coordination.\n## To w10:\nWe acknowledge that jointly training both the orchestrator and agents is a promising direction. Unlike the traditional multi-agent RL, which needs end-to-end training, leveraging pretrained language models may offer a low-overhead and potentially distinct paradigm for optimizing LLM agent collaboration. Our work represents an initial step toward this approach, and we look forward to exploring co-evolutionary architectures in future research.\n## To w11:\nWe sincerely apologize for the missing code.zip file. We were finalizing our submission until the last moment and unfortunately failed to upload the file to system. Since these comments will be made public once paper acceptance, we hereby make a formal and public commitment to release the full code. We appreciate your detailed and insightful feedback.\n## To q1:\nThe orchestration policy processes only textual input, limited by the underlying language model’s attention window. The global state is formed by combining (1) the task-specific instruction and (2) the accumulated reasoning context and intermediate results from all agents into a single text sequence fed into the policy (lines 90-92). Specifically, the policy uses a pretrained language model, paired with a three-layer MLP for agent selection. The model encodes semantics by extracting the final hidden state of the last token (an 8192-dimensional vector), which is then passed through MLP layers to output a distribution over candidate agents. As such, the effective input length is constrained by the language model’s maximum context length (which is generally sufficient in our setting), rather than by the number of agents or their individual embedding dimensions. This design decouples the input size from the agent count, enabling the system to scale to many agents as long as the total context fits within the LLM’s context window.\n## To q5:\nWhile multi-agent baselines like MacNet and EvoAgent use a mono-model setup, we evaluate our method fairly under the same condition under the Puppeteer-Mono setting (lines 203–204), where all agents share a common backbone model. In this setup, our method also outperforms them across nearly all tasks (Table 1 and Figure 2).\n## To q6:\nWe attribute the performance drop under wider or deeper exploration to two factors: (1) Diminishing Marginal Utility — activating additional agents or extending reasoning chains yields limited gains beyond a certain point [5]; (2) Coordination Overhead — more agents increase the likelihood of misalignment and conflict, making effective collaboration and coordination harder [11].\n## To q7:\nWe deliberately freeze the base LLM and only train a lightweight MLP head, for:\n- Training stability: Fine-tuning large LLMs with sparse or noisy signals may cause forgetting or alignment drift. Freezing the pretrained language model preserves its high-quality alignment and semantic understanding.\n- Strong semantic representation: The LLM is already well-optimized for capturing fine-grained quality signals. Using the last token’s hidden state as a summary retains its strong semantic fidelity without further tuning.\n- Efficiency and modularity: The three-layer MLP is fast to train, and follows standard reward modeling practice, ensuring generalizability without losing performance.\n## To q8:\nIn the ALFWorld experiment, we limit exploration width to 1, so the orchestrator generates only a single linear action sequence. This is because ALFWorld does not support parallel environment instances, making parallel reasoning infeasible. As a realistic embodied environment, each action sequentially and irreversibly changes the world state, causing divergent states from different actions that cannot be merged or jointly reasoned about. Therefore, parallel speculative reasoning is not supported. For these reasons, we report this result in the appendix for completeness rather than as a main benchmark.\n## To Paper Formatting:\nThe \"2\" links to figure 2.\n## References\n[1] MasRouter: Learning to Route LLMs for Multi-Agent System. ACL 2025.\\\n[2] GraphRouter: A Graph-based Router for LLM Selections. ICLR 2025.\\\n[3] ChatDev: Communicative Agents for Software Development. ACL 2024.\\\n[4] MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. ICLR 2024.\\\n[5] Scaling Large Language Model-based Multi-Agent Collaboration. ICLR 2025.\\\n[6] Learning multiagent communication with backpropagation. NeurIPS 2016.\\\n[7] Multi-agent actor-critic for mixed cooperative-competitive environments. NeurIPS 2017.\\\n[8] Multi-agent reinforcement learning: Independent vs. cooperative agents. ICML 1993. \\\n[9] Nii, H. Penny. Blackboard Systems. No. KSL8618. 1986.\\\n[10] Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs. ACL 2024.\\\n[11] Why Do Multi-agent LLM Systems Fail? ICLR Workshop, 2025."}, {"review_id": "fwfuE7Szeg", "reviewer": "Reviewer_QYtQ", "summary": "The paper introduces a “Puppeteer” framework that dynamically orchestrates multi-agent collaboration through a centralized controller trained with reinforcement learning. This approach seamlessly integrates dynamic agent selection, sequential decision-making, and adaptive evolution, leading to improved solution quality and computational efficiency across diverse tasks.", "questions": "Please see weakness.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. **Strengths**\n\n     1. The topic of multi-agent collaboration is interesting.&#x20;\n\n     2. The paper is well-written and well-structured. The authors use the metaphor of a puppeteer to introduce and explain their approach, making complex concepts more accessible.\n\n     3. The experimental section of the paper is  well-structured, providing convincing evidence of the proposed framework's efficacy.&#x20;\n\n  2. **Weaknesses**\n\n     1. Some key experimental details are incomplete or missing. Could the authors please provide more detailed information on the following aspects, **as this is crucial for understanding and reproducing the work?**\n\n        1. How many agents are initialized for the whole decision process? What is the initial distribution of these agents ? (For example, tool-use agents and reasoning agents)\n\n        2. The orchestration part is the most crucial part of this work. However, discussions and details about this part are limited. In **Appendix B.2,&#x20;**&#x74;he authors claim that they use a pre-trained language model combined with an MLP model as the orchestration part. I am a little bit confused and quite interested in:\n\n           1. Please provide more details on the integration and collaboration between the orchestration part and the optimization process of agent selection and topology change.\n\n           2. Is it feasible to use a fine-tuned LLM as the orchestration part? Why do you use Single-MLP instead?\n\n     2. In the introduction part, this paper proposes that existing MAS works are limited in scaling. In the experiment part , the paper conducts experiments on token expense efficiency, which is convincing. However, **I am quite interested in the other part of scaling&#x20;**--  How does Puppeteer perform compared with existing MAS works when we increase the number of initialized total agents?\n\n     3. Recently, there have been many popular papers on multi-agent collaboration and co-evolution. If we treat tools as a specific type of agent, there will be more similar work. Specifically, could the author describe the **key** differences between Puppeteer and the following works? (Even if some of these works are in the same period.\n\n     &#x20;        1\\. OTC: Optimal Tool Calls via Reinforcement Learning?  https://arxiv.org/abs/2504.14870v1\n\n     &#x20;        2\\. OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation. https://arxiv.org/abs/2505.23885\n\n     &#x20;       3\\. MASS: Multi-Agent Simulation Scaling for Portfolio Construction https://arxiv.org/abs/2505.10278\n\n     &#x20;       4\\. ReTool: Reinforcement Learning for Strategic Tool Use in LLMs  https://arxiv.org/pdf/2504.11536\n\n     &#x20;       5\\. Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies https://arxiv.org/abs/2502.02533\n\n\n\n     We are willing to raise the score if the author can address our questions and clarify our concerns.", "was_revised": true, "final_justification": "This manuscript explores a highly engaging topic that holds significant relevance in the current research landscape. The overall structure is complete and the writing is fluent. Additionally, the experimental results presented are relatively rich. However , I think this paper DOES NOT make a insightful contribution on multi-agent collaboration, requiring further refinement. If all other reviewers lean toward acceptance, I will not object.", "rebuttal": "**We sincerely appreciate your thoughtful review. Below we respond to each point; due to length limits, we welcome any follow-up for further clarification.**\n## To weakness 1.1:\nWe define the agent space as a Cartesian product of two orthogonal dimensions: the capability (captured by either a reasoning pattern or a tool) and the base model:\n$$\\text{AgentSpace} = (\\text{ReasoningPattern} \\lor \\text{Tool}) \\times \\text{BaseModel}$$\nEach agent is instantiated by pairing a reasoning pattern or tool with a base model:\n- Reasoning/Tool dimension (Table 4, lines 1111–1121):\n  - Tools (5): read_file, search_arxiv, search_bing, access_website, run_python\n  - Reasoning Patterns (8): reasoning, critique, reflect, question, summarize, conclude, modify, planning\n  - Terminator (1): A special agent that solely emits a termination signal.\n- BaseModel dimension (details see lines 170–173):\n  - Mimas (small): 6 small-scale open-source LLMs\n  - Titan (large): 8 large-scale LLMs\n\nAgent counts:\n- Puppeteer (multi-model):\n  - Mimas: (5+8)×6 +1 = 79 \n  - Titan: (5+8)×8 +1 = 105 \n- Puppeteer-Mono (single model):\n  - Both Mimas and Titan: (5+8)×1 +1 = 14 \n\nThis design offers a diverse and configurable agent pool, varying in functionality (reasoning vs. tool use) and model size (small vs. large), enabling the orchestrator to dynamically assemble task-specific teams at inference time.\n## To weakness 1.2:\n### 1.2.1\nOur orchestration policy takes in a language-based state and outputs a probability distribution over candidate agents (lines 997–1005). By sampling from this distribution, the orchestrator dynamically selects agents for execution. This design enables:\n- Adaptive agent selection: Agents always yielding higher rewards are increasingly favored over time.\n- Dynamic agent count: The number of selected agents varies with the distribution’s entropy, fewer for confident states, more for uncertain ones.\n- Evolving topology: As selections vary step-by-step, the interaction structure grows into a multi-branch tree. And we impose no constraints on cross-branch communication and the reuse of previously invoked agents, the resulting interactions go beyond tree structures, leading to the emergence of directed acyclic graph (Figure 4).\n### 1.2.2\n**Is it feasible to use a fine-tuned LLM?**\\\n Conceptually, any model that satisfies the following criteria can be used to replace our current orchestration model:\n  1. Semantic Understanding: The model should be capable of capturing nuanced semantic and contextual information from language states, effectively distinguishing between good and bad reasoning contexts.\n  2. Agent Prediction: It should be able to produce accurate probability distributions over candidate agents based on the given context.\n  3. Adaptability: The model should be readily adaptable to new tasks. In particular, fine-tuning it for agent prediction should preserve its original capacity for understanding and encoding semantics, without performance degradation.\n \nTherefore, there are multiple design choices—for example, using a frozen pre-trained language model or embedding model as a stable backbone with a lightweight head, or directly fine-tuning an LLM if careful measures are taken to preserve its original capabilities. \n\n**Why we design using LLM+MLP architecture?**\\\nWe choose Llama-3.1-Nemotron-70B-Reward-HF as the LLM backbone because it is trained specifically for quality assessment in multi-turn conversations, making it effective at encoding fine-grained reasoning signals, and can effectively distinguish between high-quality and low-quality reasoning in generated text [1]. It also achieves state-of-the-art performance on multiple automatic alignment benchmarks as of October 2024, surpassing models such as GPT-4o [2], thus making it suitable to be our policy's language backbone.\\\nAfter the language backbone, we use the final hidden state of the last token, a well-established practice in decoder-only architectures like LLaMA/GPT, as a compact semantic summary of the input. This embedding is projected by a simple three-layer MLP (512 → 128 → 32), enabling dimensionality reduction and culminating in a distribution over candidate agents. Similar projection heads are widely used to transform high-dimensional contextual inputs into task-relevant decisions [3-6]. And empirically, this policy network consistently outperforms all baselines across benchmarks (Table 1), validating both our model choice and architecture design.\n\n**Reference**\\\n[1] HelpSteer2-Preference: Complementing Ratings with Preferences. ICLR, 2025.\\\n[2] NVIDIA. Llama-3.1-Nemotron-70B-Reward-HF. Hugging Face, 2024. \\\n[3] iVideoGPT: Interactive VideoGPTs are Scalable World Models. NeurIPS, 2024.\\\n[4] InstructEval: Instruction-Tuned Text Evaluator from Human Preference. ACL, 2024.\\\n[5] Defending against Indirect Prompt Injection by Instruction Detection. arXiv:2505.06311, 2025.\\\n[6] Hugging Face Transformers. LlamaForSequenceClassification Documentation. \n## To weakness 2:\nWhile prior multi-agent collaboration often equate scalability with involving more agents in computation and communication, our framework introduces a key distinction between initialized agents—those pre-created to enrich the decision space—and invoked agents—those actually selected to participate at runtime (lines 75-77). This decoupling enables Puppeteer to maintain a large and diverse candidate agent pool to support flexible and fine-grained orchestration, while smartly invoking only a minimal, task-relevant subset of agents during execution. \n\nIn contrast, prior works like MacNet [1] scale agent numbers under static topology directly, causing interaction costs to grow polynomially with diminishing returns. We illustrate this contrast via interaction steps $I$:\\\nFor Puppeteer:  $I_{\\text{puppeteer}} = \\text{Constant} \\perp n$\n\nFor static topology of agents like in MacNet [1]:  $$I_{\\text{chain}}, I_{\\text{tree}} \\propto n；I_{\\text{mesh}} \\propto n^2；I_{\\text{layer}} \\propto (\\frac{n}{k})^k, k \\geq 1；I_{\\text{random}} \\propto n^a, a \\in (1,n]$$\n\nEmpirically, our experiments cover a wide range of settings, with the number of initialized agents varying from 14 (Puppeteer-Mono setting) to 105 (Puppeteer setting under Titan subspace). Across this spectrum, Puppeteer consistently demonstrates strong performance while keeping token usage under control (see Table 1 and Figure 2). \nIn real-world applications, enriching the agent space with domain-specific tools, reasoning modes, or foundation models allows Puppeteer to more effectively select and orchestrate agents tailored to new tasks.\n\n**Reference**\\\n[1] Scaling Large Language Model-based Multi-Agent Collaboration. ICLR 2025.\n\n## To weakness 3:\nWe provide a table including the aspects from research focus, methodology and experimental tasks: \n| Work               | Motivation                              | Method                                                                 | Multi-agent?                                                      | RL Used?                                                      | Primary Tasks                                                                 |\n|-|-|-|-|-|-|\n| **Puppeteer**      | Multi-Agent Collaboration Design & Optimization | Orchestrates heterogeneous LLM agents using RL for dynamic collaboration | Yes — heterogeneous agents differing in reasoning strategy, tool usage, and model backbone | Yes — for agent orchestration and selection                   | General reasoning tasks across domains, including math, code generation, writing, and embodied environments |\n| **OTC**            | Single-LLM + Tool Use                   | Optimizing a single LLM’s tool call behavior via reinforcement learning to reduce tool usage cost | No — single base model (Qwen 2.5 series) with several tools       | Yes — to penalize unnecessary tool calls while maintaining performance | Math problem solving and open-domain question answering       |\n| **ReTool**         | Single-LLM + Tool Use                   | Teaching a single LLM when and how to invoke tools to enhance mathematical reasoning | No — single base model (Qwen 2.5 series or DeepSeek-R1) with several tools | Yes — to learn optimal reasoning strategies involving tool manipulation | Math reasoning                                                 |\n| **OWL**            | Multi-Agent Collaboration Design & Optimization | Designing a multi-agent framework (planner–coordinator–workers), and optimizing the planner for cross-domain generalization | Yes — structured as a planner–coordinator–worker framework        | Yes — used only to train the planner agent for better generalization | Multimodal reasoning and task automation                      |\n| **Multi-Agent Design** | Multi-Agent Collaboration Design & Optimization | Searching for optimal agent prompts and collaboration topologies offline | Yes — agents serve as configurable building blocks                | No — uses search-based methods to discover effective multi-agent collaboration | Reasoning, QA, and code generation tasks                      |\n| **MASS**           | Multi-Agent for Domain Applications     | Using multi-agent simulations to optimize the performance for portfolio construction tasks | Yes — a multi-agent simulation setup for financial domains        | No — employs multi-agent for simulation                        | Financial decision-making                                      |\n\nIn summary, our work (Puppeteer) is distinct in the following aspects:\n- Heterogeneous agent orchestration: It coordinates agents with diverse roles, tools, and base models, beyond single-model tool use (e.g., OTC, ReTool).\n- Adaptive multi-agent optimization: It dynamically learns collaboration strategies via reinforcement learning, unlike fixed (e.g., OWL) or search-based (e.g., Multi-Agent Design) schemes.\n- General-purpose task coverage: It handles diverse tasks, generalizing beyond domain-specific systems (e.g., MASS)."}]

Author rebuttal:

Meta-review: In this submission, the authors propose a puppeteer-style paradigm for LLM-based multi-agent collaboration whereby a centralized orchestrator dynamically directs agents as a response to task states that keep evolving over time. Trained using RL to adaptively order and prioritize the agent, the orchestrator provides flexible and evolvable collective reasoning. The authors provide empirical results on closed- and open-domain scenarios which show that the proposed method achieves superior performance with reduced computational costs.

The reviewers have noted several strengths of the paper, including impressive, cost-aware improvements over four diverse benchmarks. The reviewers agree that the paper provides a clear and compelling motivation behind the main idea, and that framing the multi-agent collaboration problem as a centralized & learnable sequential decision process is conceptually clean. During the rebuttal period the authors addressed many of the concerns raised by the reviewers. I believe that this is an interesting research direction with compelling results, and therefore I’m pleased to recommend it for acceptance.
Decision: Accept (poster)