OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation
Authors: Mengkang Hu
Yuhang Zhou
Wendong Fan
Yuzhou Nie
Ziyu Ye
Bowei Xia
Tao Sun
Zhaoxuan Jin
Yingru Li
Zeyu Zhang
Yifeng Wang
Qianshuo Ye
Bernard Ghanem
Ping Luo
Guohao Li
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
LLM-based Agent
Multi-Agent
Primary area: applications

Abstract: Large Language Model (LLM)-based multi-agent systems show promise for automating real-world tasks but struggle to transfer across domains due to their domain-specific nature.
Current approaches face two critical shortcomings: they require complete architectural redesign and full retraining of all components when applied to new domains.
We introduce **Workforce**, a hierarchical multi-agent framework that decouples strategic planning from specialized execution through a modular architecture comprising:
*(i)* a *domain-agnostic* **Planner** for task decomposition,
*(ii)* a **Coordinator** for subtask management, and
*(iii)* specialized **Workers** with *domain-specific* tool-calling capabilities.
This decoupling enables cross-domain transferability during both inference and training phases:
During inference, Workforce seamlessly adapts to new domains by adding or modifying worker agents;
For training, we introduce **Optimized Workforce Learning (OWL)**, which improves generalization across domains by optimizing a domain-agnostic planner with reinforcement learning from real-world feedback.
To validate our approach, we evaluate Workforce on the GAIA benchmark, covering various realistic, multi-domain agentic tasks.
Experimental results demonstrate Workforce achieves open-source state-of-the-art performance (**69.70%**), outperforming commercial systems like OpenAI's Deep Research by **2.34%**.
More notably, our OWL-trained 32B model achieves **52.73%** accuracy (**+16.37%**) and demonstrates performance comparable to GPT-4o on challenging tasks.
To summarize, by enabling scalable generalization and modular domain transfer, our work establishes a foundation for the next generation of general-purpose AI assistants.

*Our code is available at [Anonymous URL](https://anonymous.4open.science/r/annonymous-owl/), and our data is available at [Anonymous URL](https://huggingface.co/anonymous21016).*

OWL: Optimized Workforce Learning
General Multi-Agent Assistance for WoaldWorld Task Automation
Mengkang Hu2.4, Yuhang Zhou' Wendong Fan', Yuzhou Nie", Bowei Xia'
Tao Sun Ziyu Ye Zhaoxuan Jin Yingru Li", Qiguang Chen Zeyu Zhang
Yifeng Wang Qianshuo 9 Bernard Ghanem 10, Ping Luo Guohao Li
The University of Hong Kong Eigent.AI UCSB *CAMEL-AI
5 The University of Chicago Northwestern University 7 CUHK ANU University of Cambridge
10 KAUST
Abstract
Large Language Model (LLM) multi-agent systems show promise for au- tomating real-world tasks but struggle to transfer across domains due to their domain-specific nature. Current approaches face two critical shortcomings: they require complete architectural redesign and full retraining of all components when applied new domains. We introduce WORKFORCE, hierarchical multi-agent framework that decouples strategic planning from specialized execution through modular architecture comprising: (i) domain-agnostic Planner for task de- composition, (ii) a Coordinator for subtask management, and (iii) specialized Workers with domain- domain-specift specific tool-calling capabilities. This decoupling enables cross-domain transferability during both inference and training phases: During inference, WORKFORCE seamlessly adapts to new domains by adding modifying worker agents; For training, we introduce OPTIMIZED WOR TFRRCO LEARNING (OWL), which improves generalization across domains by optimizing a domain- agnostic planner with reinforcement learning from real-world feedback. To vali- date our approach, we evaluate WORKFORCE the GAIA benchmark. covering various realistic, multi-domain agentic tasks. Experimental results demonstrate WORKFORCE achieves open-source sfate-of-the-ar performance (9.70%) ), out- performing commercial systems like OpenAl's Deep Research by 2.34% % More notably, our OWL-trained 32B model achieves 52.73% % accuracy (+16.37 %) and demonstrates performance comparable to GPT-4o on challenging tasks. To summa- rize, by enabling scalable generalization and modular domain transfer, our work establishes foundation for the next generation of general-purpose Al assistants.
Introduction
Large Language Models (LLMs) have undergone a period of rapid advancement, evolving from simple text predictors into powerful autonomous agents capable of planning, tool use, and multi-step reasoning [OpenAI, 2024, Anthropic, 2025]. Recently, multi-agent systems (MAS) have emerged as a promising approach for complex real-world tasks, demonstrating that dividing tasks among specialized agents can enhance performance [Li al., 2023, Qian et al., 2023, Wang al., 2024].
Although current MAS have yielded impressive results, their designs are typically domain-specifie, severely restricting cross-domain transferability. This shortcoming appears in two forms: (i)
39th Conference on Neural Information Processing Systems (NeurIPS 2025).
OOr code is available at http://githab.com/mmeaaaaaai

---

Previous Inference
Workforce
Contralized
Decontraliad
Domain-Agnosse
Conrdinator
Planner
Task Chaunel
GMUA GATA Scoce
W Worker Nodes
Agunt
Agunt
Agrat
Chaln
Task- Independoni
Tasks Independent
Task-
Model
Independeat
Previous Training
Contest
Training
Optimized Workforce Learning
Train
Mullf-Agent System
Domain- A Agnostic Planner
Task
Train
Coordigator
Agent
Agent
Task Channel
22
Agent
Agent N
CALA
Need Re-train!!
Worker Nodes
Domatn
New Domain A daptation
Agent
Agent
Agent 3
Model
Figure 1 Overview of WORKFORCE and OPTIMIZED WORKFORCE LEARNING.
First, on the inference side, deploying system in new domain often entails a full redesign; for instance, MetaGPT [Hong et al., 2023] depends on Standard Operating Procedures tailored to software engineering. hindering extension to other fields. (ii) Second, on the training side, existing works often optimize every agent. MALT [Motwani et al., 2024], for example, follows fixed generator-verifier verifier--effenn pipeline, necessitating separate training for every component. Consequently, migrating such systems demands retraining the entire agent ensemble, sharply reducing flexibility. These brawscck underscore the need for generalized, modular multi-agent architecture that can be rapidly adapted to diverse domains with minimal retraining and redesign.
Building upon these observations, firstly, we introduce WORKFORCE, a hierarchical multi-agent inference framework that decouples strategic planning from domain- execution. As is shown Figure This modular design comprises three core components: (i) nomain-agnosie Planner: Generates abstract task decompositions based on high-level goals. (ii) Coordinator: Assigns subtasks to appropriate workers. (ii) Domain-specife Worker Nodes: set of specialized agents that perform tool calls to accomplish each subtasks. The decoupling of these components enables plug-and- play extensibility, allowing WORKFORCE to be seamlessly adapted to new domains by simply replacing or adding worker nodes. Furthermore, this modular architecture facilitates OPTIMIZED WORKFORCE LEARNING (OWL), a novel multi-agent training paradigm. OWL focuses on enhancing MAS's cross-domain transferability by training a generalizable domain-agnostic planner. Specifically, we employ a two-stage training strategy: Supervised Fine- Fine-Tuning (SFT) for planner initialization, followed by Reinforcement Learning [Rafailov al., 2023] further enhance the model's 's generalization capabilities.
We evaluate our approach the GAIA benchmark, rigorous suite for generalist AI assistants that spans diverse domains and demands multimodal reasoning, code execution, and live web search [Mialon et al., 2023]. WORKFORCE attains accuracy of 69.70% %, surpassing strong commercial proprietary baselines like OpenAl's Deep Research [OpenAI, 2024] (55.15%). To demonstrate the effecctiveness of OWL, we further post-trained strategic planner initialized by Owen2.55-3BBIInstuu [Yang et al., 2024] on a custom-curated training dataset without using any GAIA data. After training, the model reaches 52.73% score (+16.37%), outperforming models like GPT-4o-mi mini [OpenAI, 2024] (47.27%) and Swwe2.5-72B-IInsse (49.09%). These results confirm that our modular training strategy generalizes across domains while requiring minimal retraining.
Our key contributions are fourfold:
A New Flexible and Modular Multi-Agent Architecture. We propose WORKFORCE that is modular and scalable in both inference and training, enhancing cross-domain transferability.
State-of-the-Art Performance. Our system achieves open-source state-of-the-an performance on the GAIA benchmark, even surpassing proprietary systems like OpenAl's Deep Research.

---

3. Efficient and Effective Training Paradigm. OWL significantly enhances model capabilities with minimal overhead, enabling Qwen2.5-32B --Isttstt achieve 16.37 % performance gain and reach comparable performance to GPT-4o on challenging tasks.
4. Fully Open-Source.
We release all code, models, and data support open research.
2
Preliminary
Large Language Model (LLM)-based Agents are autonomous systems that perceive, reason, and act in various environments [Xie et al., 2023]. These agents operate in a prrceptionrccusoinngattii loop, where they observe their environment, process information through the language model, determine appropriate actions, and execute them to achieve goals.
Multi-Agent Systems extend this paradigm by enabling multiple LLMM-bssed agents to collaborate on complex tasks. Frameworks such as CAMEL [Li et al., 2023] and MetaGPT [Hong et al., 2023] have demonstrated that collaborative approaches can outperform single-agent systems on tasks requiring diverse expertise. However, existing multi-agent frameworks are typically constrained by domain-speciiic designs that limit their broader applicability. This paper aims to develop scalable and generalizable multi-agent frameworks WORKFORCE through strategic planning and task execution decoupling, enabling efficient coordination across diverse domains. (See complete formalization in Appendix C and more related works in Section 6.)
Generalist AI Assistant was first introduced by GAIA [Mialon al., 2023]. These systems designed to handle a diverse range of complicated tasks across multiple domains and modalities. As the first question answering benchmark evaluating generalist Al assistants, GAIA proposes to enable LLM--aased agents to gather information in real-world contexts, testing fundamental capabilities including multi-modal understanding, web browsing, reasoning, and complex problem-solving. Recently, numerous companies have released generalist Al assistant products (e.g.. OpenAT's Deep Research [OpenAI, 2024J). While open-source frameworks have made significant progress (e-g., Huggingface's Open Deep Research [Roucher et al., 2025a]), they still lag behind commercial solutions. In this paper, we aim close the gap between open-source and commercial proprietary agent frameworks. The proposed WORKFORCE surpasses OpenAl's Deep Research by 2.34%, while our training method OWL significantly improves Qwen2.5-333 3 performance by 16.37 16.37%.
3
Multi-Agent Inference: WORKFORCE
3.1 WORKFORCE
Motivation.
Contemporary multi-agent systems are critically limited by domain specificity and architectural rigidity, requiring complete redesign and retraining for each new application domain. We introduce WORKFORCE, which addresses this fundamental challenge through modular architecture, particularly the separation of domain-agnostic planning from domain specific execution. More details about WORKFORCE can be found in Appendix D.
(1) Task Planning
Task Charnel
(2) Task Solving
paper thout All misiatirs wich thoo Plyers duciile type socidyy Aaguz 2016?
Sudtink Smarc
mysiatinn..
Sabbei daxid eodigaed couds agped
Der
Tinks Stdtes DONE swcch agent
Coordraler
Subtadk Scarc Plyses usc Socady utide vatniibll
Descniig
Subbok south paper repciabas...
Task Seate: Jpree
Agoet
Planner
Scbtook Search Plysto
Subtzok
Sebtook 2 Sworth for the Plydis unal Seciity sttide
Socliy ariit...
worth fives Ue aepdaian....
and ldeetity keywont usd disctibe a typ of society
Stit:
Subteel: &t Compan the lotel wards Coos de soguation....
Subtakk & Troesfim de odgit pashilen iito ucpecal fistel.
Satenk Tranfore
Tanks States
OPEN
Figure 2: Overview of the WORKFORCE framework. The system consists of a Planner Agent for task decomposition. Coordinator Agent for orchestrating subtasks, and multiple specialized Worker Nodes equipped with domain -omain-specific toolkits to execute assigned tasks.

---

Architecture. As illustrated in Figure 2, WORKFORCE comprises three core components: (i) Planner Agent analyzes incoming tasks and decomposes them into subtasks based on worker capability registry; (ii) Coordinator Agent serves as the central orchestration mechanism, managing worker assignments and task dependencies while integrating intermediate results; (iii) Worker Nodes consist of one or more specialized agents equipped with specific capabilities and toolkits that execute assigned subtasks and post results. This modular architecture provides inherent flexibility, allowing the framework to be deployed across diverse applications by simply modifying the worker nodes while maintaining its core planning and coordinating mechanism.
Communication Mechanism. Communication in WORKFORCE operates through shared task channel that serves as a central hub. The coordinator posts tasks and assignments to this channel. Upon completion. workers post only their final results back to the channel, while the detailed execution context of tool calls remains isolated within each subtask's scope. This maintains a clean context for each worker, who only has access to the current subtask details and concise previous subtask results. This centralized approach simplifies system management and enhances scalability by eliminating direct agent-to-agent messaging.
Task Flow. The task processing workflow within WORKFORCE follows a structured pipeline: (i) The Planner analyzes incoming overall tasks and then decomposes them into a set of subtasks according to the capabilities of available worker nodes and the overall task's complexity; (ii) The Coordinator assesses the capabilities of available worker nodes and dispatches subtasks accordingly; (iii) The Worker Nodes perform the assigned subtasks using their specialized tools; (iv) Results from worker nodes are posted to the shared task channel; (v) The Coordinator manages task dependencies and integrates results, ultimately forwarding them the Planner, (vi) The Planner analyzes the results of each subtask and synthesizes the final output.
Replanning Mechanism. During task execution, workers self-assess whether the assigned subtask is failed. When a worker determines that subtask has failed, it posts failure information to the task channel. The task channel then detects this failure, and the planner will be prompted to generate new subtasks based on the feedback information. This replanning mechanism enables test-time scaling by dynamically adjust its approach to increasingly complex tasks, which validated in Section 5.
3.2 Generalist Multi-Agent Assistance
To construct generalist multi-agent assistant capable of processing diverse real-world tasks, instantiated WORKFORCE with three worker agents, each equipped with domain-specific toolkits: (i) A Web Agent, capable of performing web searches, extracting webpage content, and simulating browser actions; (ii) A Document Processing Agent, designed to process documents and multimodal data, including text, images, audio, video, spreadsheets, and various file formats; and (iii) A Rea- soning/Coding Agent, which handles analytical reasoning and code execution tasks. The details of worker agents and corresponding toolkits could be found in Appendix D.3.
3.3 Experiments
Baselines. We selected a comprehensive set of baseline systems, categorized into four main groups: (i) Proprietary frameworks establish upper-bound commercial performance, including commercial agentic systems like OpenAl's Deep Research [OpenAI, 2024], h2oGPTe Agent [H2O.ai, 2024], etc. (ii) Open-source frameworks reveal the community progress, including strong baselines like HuggingFace's ' Open Deep Research [Roucher et al., 2025a], Trase Agent [TTrase, 2024], etc. (iii) A Single Agent baseline utilizing multi-stpp tool calling. (iv) Role Playing Li et al. [2023] comprising two agents (user agent and assistant agent) that collaborate through structured dialogue to accomplish tasks. Note that to control experimental variables, both the Single Agent and Role Playing baselines utilize identical tool sets to WORKFORCE.
Implementation Details. For our implementation, we access all models via APIs, eliminating the need for GPUs To ensure reproducibility, we configure API inference with greedy decoding. The default replanning threshold is set to 2. For evaluation methodology, we employ pass@3 sampling for W WORKFORCE with GPT-4o [OpenAI, 2024] and pass @ for WORKFORCE with Claude-3.7- sonnet [Anhhropic, 2025]. Since some gold answers from GAIA had been leaked online, we blocked several websites to ensure fair comparison [OpenAI, 2024].

---

Table Performance comparison of agent frameworks on GAIA [Mialon al., 2023] validation set with accuracy score (%) as the evaluation metric. Scores of upen-source and proprietary frameworks were obtained from the official leaderboard. 'he best-performing proprietary and open-source frameworks are highlighted in bold.
Agent Name
Base Model
Level
Level 2
Level 3
Average
Proprietary Frameworks
DRP-val-v.1.0
56.60
48.84
15.38
46.06
omne
O1-Previewf
60.38
44.19
23.08
46.06
Barcelona v0.1
Mrude-3.5-Sonnee
62.26
50.00
26.92
50,30
Ormind v0.1
Claudel
69.81
54.65
26.92
55.15
desearch
GPT-4o
71.70
58.14
23.08
56.97
Anges
Claudel
66.04
65.12
30.77
60,00
h2oGPTe Agent v1.6.8 [H20.ai, 2024]
Moude-3.5-oommn
67.92
67.44
42.31
63.64
Deep Research [OpenAI, 2024]
03
78.66
73.21
58.03
67.36
Trase Agent v0.3 [Trase, 2024]
Claudel
83.02
69.77
46.15
70.30
Langfun Agent v2.1 [Google, 2023]
Claude-3.7-Sond
83.02
68.60
57.69
71.52
Open-Source Frameworky
FRIDAY [Wu ct al., 2024]
GPT-4-Turbo
45.28
34.88
11.54
34.55
Multi-Agent Exp v0.1 [Microsoft, 2024]
GPT-4-Turbo
54.72
38.37
11.54
39.39
HuggingFace Agents [Roucher al., 2025b]
GPT-4o
58.49
43.02
19.23
44.24
Magnetic-One [Fourncy et al., 2024a]
01
56.60
46.51
23.08
46.06
AutoAgent (TTng et al., 2025]]
Claude-3.5-Someet
71.70
53.49
26.92
55.15
Open Deep Research [Roucher et al., 2025a]
01
67.92
53.49
34.62
55.15
TapeAgents [Bahdanau al., 2024]
Claude-3.7-Sonnet
71.70
53.49
30.77
55.76
Single Agent
GPT-4o
52.83
34.88
15.38
37.58
Role Playing eL al., 2023]
GPT-4o
75.47
52.33
54.55
Role Playing* [Li et al., 2023]
GPT-4ol
81.14
54.65
23.08
58.18
Ours
WORKFORCE WORKFORCE
GPT-4o
81.14
58.14
26.92
60.61
Claude-3.7-Somme
84.91
68.60
42.31
69.70
Main Results. Several conclusions could be drawn from Table 1:
(i) Workforce achieves state-of-the-art performance among open-source frameworks. Our WORKFORCE achieves 69.70%. %,, consistently outperforming previous open-source frameworks across all difficulty levels. Under strictly controlled settings with the model and toolkits, GPT-4o- based WORKFORCE, achieves 60.61 % accuracy 23.03 % higher than the Single Agent and 6.06 % higher than the multi-agent baseline Role Playing-
(ii) Workforce demonstrates comparable or even superior performance relative to commercial proprietary frameworks. While previous open-source frameworks have exhibited substantial performance gap compared to closed-source alternatives, WORKFORCE considerably narrows this divide. To our knowledge, WORKFORCE is the first open-source system to exceed OpenAl's Deep Research, achieving a 2.34% improvement, and also establishes a new Level best by outperforming Langfun Agent v2.1 by 1.89%.
4
Multi-Agent Training: OPTIMIZED WORKFORCE LEARNING
4.1 Training Strategy
Motivation. The WORKFORCE architecture separates domain-agnostic planning from domain-specific execution, allowing us to adapt to new domains by simply adding or replacing Worker Nodes while preserving the core planning mechanism. We introduce OPTIMIZED WORKFORCE LEARNING (OWL), which focuses on enhancing generalizable Planner Agent capable of handling diverse
Indicates systems that multiple models, and only the primary model is listed. Indicates frameworks that have released early version but have not provided the latest version that can replicate these results.
Indicates an optimized version of role-playing where we introduced an LLM-based classifier that switches to 03-mini when tasks require more reasoning and coding, and uses GPT-4o otherwise.

---

Table 2: The statistics of the dataset. We use icons to repre- sent different agent capabilities: Web Browsing; @ Reasoning; +
(Owana with Fitering (ONL)
Wittort Jiltering
Document Processing; sD Coding: Rel Multimodal Handling.
Average Score (%)
52-23
More details of training data can be found Appendix D.4.
46.06
Dataset
# Total
# SFT Filtering
DPO Filtering
Agent Capahilities
41.21
HotporQA
998
495
354
40
Wiki WikTTableQqestions
577
311
X
36.36
Math-related
1100
297
Infinity-M
499
40
47
@
Base Model
SFT
SPT+DPO
Total/Average
3466
1599
1009
(a) Ablation study on trajectory filtering
real-world scenarios. This design significantly reduces training overhead since only the Planner Agent requires intensive optimization, while Worker Nodes can leverage existing domain -omain-specific tools with minimal adaptation. This "stable core, variable periphery" approach enables efficient knowledge transfer across domains and climinates the need to retrain entire systems for new applications, substantially reducing computational costs while maintaining consistent performance.
Implementation. More specifically. we adopt a two-phase training paradigm: (i) In the first phase, we employ supervised fine-tuning (SFT) initialize the Planner Agent with fundamental task decompo- sition skills derived from expert demonstrations. (ii) Subsequently, we utilize reinforcement learning to further optimize the SFF-iinitialized Planner Agent. We select Direct Preference Optimization (DPO) [Rafailov et al., 2023] as our optimization algorithm of choice, as this phase enhances the quality of decomposition strategies beyond mere imitation of demonstrations, enabling the Planner to develop more sophisticated decision-making capabilities.
4.2 Task Curriculum
Motivation. The core innovation of WORKFORCE is its architectural separation of domain-agnostic planning from domain-specific execution. For this design to be effective, the Planner Agent must possess robust generalization capabilities across diverse problem domains. This creates fundamental tension: the Planner must simultaneously maintain a deep understanding across disparate domains while avoiding overfitting specific task patterns or domains. To address this challenge, we developed a strategically balanced task curriculum that deliberately spans multiple capability dimensions required for general intelligence. Our curriculum design guided by two key principles: (i) Capability Coverage: Exposing the Planner to diverse reasoning patterns and problem structures. (ii) Transfer Learning: Prioritizing tasks that develop complementary cognitive skills which can knowledge.
transfer across domains, rather than domain-spectee
Implementation. More specifically, is shown in Table 2, carefully selected four datasets, each targeting distinct agent cognitive capabilities dimensions: (i) HotpotQA [Yang et al., 2018]: This dataset requires multi-hop reasoning based online information, challenging the Planner orchestrate complex information-seekinn
behaviors. (ii) Wik/TableQQestions [Pasupat and Liang, 2015]: This dataset requires the Planner formulate strategies for navigating, filtering, and operating over tabular information. (iii) Math-related Problems: This is a custom-curated collection of mathematical problems that require reasoning or coding to solve, covering various mathematical domains. They help the Planner develop skills in logical reasoning and computational problem- solving. (iv) InI [Gu et al., 2024]: As multimodal dataset, Infinity- challenges the Planner to orchestrate multimodal information processing., including vision, text, and structured data.
4.3 Trajectory Synthesis
Supervised Fine-tuning. We employed our WORKFORCE approach ($3.1) with GPT-4o-mini to synthesize expert trajectories, which consist of planner-generated subtasks and worker-generated execution traces. To filter out low-quality data, we applied different evaluation metrics across datasets: For HotpotQA and WikiTTableQuestionss, we used accuracy metrics; For Infinity-MM, we used text cosine similarity with 0.7 threshold between ground truth and generated answers; For math-related problems, we implemented LLM-as-a-juke judgee to compare ground truth answers with selkforre-generatee solutions. Ultimately, as shown in Table and 6, we obtained 1,599 filtered trajectories for supervised fine-tuning, with each trajectory containing an average of 3.41 subtasks.

---

Table Performance comparison between different planners. Each result is based on the Workforce framework with GP as the worker foundation model.
Planner
Level
Level 2
Level
Average
GPT-4o-mini [OpenAI, 2024]
64.15
45.34
19.23
47.27 .2
Owen2.5-72B-Instiie (Yang et al., 2024]
60.30
51.16
19.23
49.09
Claude-3.7-Sone [Anthropie, 2025]
81.13
53.49
34.61
59.39
GPT-4o [OpenAI, 2024]
81.14
58.14
26.92
60.61
Owen2.5-32B-Ine [Yang et al., 2024]
49.05
33.72
19.23
36,36
w: OWL (SFT)
56.60
+7.55
39.53 +5.81
15.38 -3.85
41.21 +4.85
w: OWL (SFT + DPO)
67.92
+18.87
51.16 +17.44
26.92 +7.69
52.73 +16.37
Reinforcement Learning. We use the SFT-initializer model to generate pair-wise trajectories for DPO [Rafailov et al., 2023]. Specifically, for each question in our collected dataset, we roll out n = distinct trajectories. The evaluation of these generated trajectories is the same in the SFT stage. Then, we construct preference pairs from the n trajectories generated for each question based on their evaluation outcomes. For Math, HotpoiQA, and WikiTableQuestions tasks, trajectories that correct are labeled as "chosen", while incorrect answers are labeled as "rejected". For Infinity-MM dataset, trajectories whose final text cosine similarity scores exceeded the same threshold as in the SFT stage (0.7) are labeled as "chosen". while those below the threshold are labeled as "rejected". As shown in Table 2, we collect 1009 filtered trajectories pairs.
4.4 Experiments
Baselines. We compare our approach against multiple proprietary and open-source models as baselines, including the GPT-4o series [OpenAI, 2024], Claude-3.7-Some [Anthropic, 2025], and the Qwen2.5 series [Yang et al., 2024]. These models represent the current state-of-the-art in language model capabilities across different scales and architectures.
Implementation Details. Our model training is conducted on a computing cluster equipped with NVIDIA H100 GPUs. We use the LlamaFactory [Zheng al., 2024] framework for managing and executing our training procedures. Specifically, for all the models we trained, the input sequences are truncated maximum length of 32,768 tokens and the learning rate set to 10 All models trained for a total of two epochs. To optimize memory usage and training efficiency, we use bfloat16 mixed- precissre training. The effective batch size is 12, achieved through 1 per-device batch size combined with 12 gradient accumulation steps.
Main Results. Table 3 reveals multiple important findings:
(i) OWL significantly enhances planner capabilities, enabling open-source models to outperform proprietary alternatives. The OWL-trained Qwen2.5-32B Instructt model demonstrates remarkable improvements with substantial 16.37% % gain. OWL enables open-source model reaches 52.73% % performance, surpassing both the proprictary GPT-4o-mini (47.27%) and the larger Qwen2.5-72B- Instruct (4..0%)) While GPT-40 (60.61 %) still maintains an advantage, our OWL-trained model achieves comparable performance to 4P (26.92%) on the more challenging Level 3 tasks.
(ii) Reinforcement learning significantly boosts planner generalization. While supervised fine- tuning (SFT) alone improves performance on simpler tasks (Level and 2), it shows a regression on the most complex tasks (Level 3), with performance dropping by 3.85%. However, when combined with DPO, our approach not only recovers this performance but significantly exceeds the base model across all difficulty levels, with a 7.69 % improvement on Level 3 tasks.
Ablation Study, We evaluated the impact of trajectory filtering on model performance. As shown in Figure 2a, models trained on filtered trajectories consistently outperform those trained unfiltered data, which underscores that data quality is more critical than quantity for effective planner training.
Analysis
Performance Across Capability Types. Each evaluation case may require one or more types of agent capabilities (e.g., Web Browsing, Coding, Multimodal Processing). As shown in Figure 3(a), our experimental results demonstrate that: (i) The WORKFORCE approach consistently outperforms both
7

---

Playtng
Workforce
Workforce swwenag-gatt-iutnne Warkforce Qwenaa 5-325--istrrrrd
GPT-40
(.2dl-3-7--5000 3.7-Sonnct
0.75
Average (%)
Average Score (9)
0.8
0.7
0.65
0.6
04
0.5
0.55
0.51
0.54
03-
0.5
0.45
0.43
0.1
Web Browsing
Multi Multi- Modality
biveess Filetpss Reading
Ressoning
Coding
Replanning Count
(a) Performance by Capability
(b) Test-time scaling
Figure 3: Left: Average scores on GAIA validation categorized by different agent capabilities. Blue compares different agent frameworks, while brown compares different planner models. Right: Changes in Workforce performance as the number of replanning iterations increases.
Plannen Frnor
E.tetted Toal Capabilitt
Stagle Agent
Robe Haylng
Worklorce (Ours)
Language/Questoon Anbiguity Limibed Model Capebility
Respending Eroor
Average Score (96)
59-46
62:34
55:36
Limited
Capabitity
51-34
Contoct Exceed
11.5%
40
38.36
3544
36.2
5.8%
19 angg
Subbank Anbiguitt
Capshilities
Number
Lggh
Ferformance across different capability requirements
Selection Ertor
5. ..6%
9.6%g
6%
Lannttne//Qunetto Aintiguity
Guem.y-pl--tttret O Worber
(Mano)
Reasoning Ervor
Responae Format
8
17.
Web Toelkn Fillure
Averaae
Multimedal Toolitt Fallure
Rncument Toolkit Falluze
Level
(a) Fallure Modes on the GALA Benchmark
(c) Performance comparason between different training contiguration
Figure 4: Manual error analysis and performance comparison of our multi-agent system.
Role Playing and Single Agent approaches across different capabilities, (ii) After OWL training, we observe consistent improvements across all capability types.
Test-time Scaling. As discussed in Section 3.1, we introduced a replanning mechanism, which enables test-time scaling in multi-agent systems. As shown in Figure 3(b), both GPT-4o and Claude- 3.7-Sonnet workforces demonstrate improved performance as the number of replanning iterations increases. Notably, the models have access ground truth in this process, which indicates that the workforce possesses inherent self-correction and self-evolving capabilities during test time.
Robustness Across Capability Requirements. We analyzed robustness by grouping tasks based on required capabilities (1, 2, or 2 3). Figure 4(b) shows that baselines degrade significantly on multi multi-capability tasks (Role Playing drops from 62.3% to 34.6%), while W WORKFORCE maintains consistent performance across all complexity levels. This stability comes from our modular design, where specialized workers handle focused subtasks, making complex tasks more manageable than when a single agent must handle everything. Other robustness metrics can be found in Appendix 1.1.
Planner vs. Worker Training. Our ablation study in Figure 4(c) reveals that training only the planner (45.45%) significantly outperforms training only workers (31.51%) %).. Training both components together offers minimal additional gains (46.6%%) while substantially increasing computational costs. This confirms our design choice to prioritize planner optimization, as effective task decomposition is more crucial than enhancing individual worker capabilities.
Error Analysis. To deeply investigate the failure modes WORKFORCE, we conducted manual error analysis on Claude 3.7 7-Sonnet results. As shown in Table 7, approximately half of the emrors

---

stem from limitations in the foundation models themself or from tool-related issues. Among agent- specific errors, the highest proportion is attributed to planner failures (21.15%), further highlighting the significance of planner optimization. Examples for each error category and detailed error distribution can be found in Appendix E. We also provide more qualitative analysis Appendix
Related Work
LLM-based Multi-Agent Systems. Recent work has explored architectures where multiple LLM- based agents cooperate, each with specialized roles, to tackle complex tasks beyond the ability of a single model [Phan et al., 2024, Yang et al., 2025, Guo et al., 2024, Chen et al., 2023b, Trivedi et al., 2024]. Early two-agent role-play systems such as CAMEL [Li et al., 2023] showed that dialog between agents can elicit step-wise reasoning. Recent frameworks extend this idea by assigning explicit roles to multiple agents: MetaGPT [Hong al., 2023] and ChatDev [Qian al., 2023] replicate software- -eggieernninniinggggg pipelines with manager, designer. and coder agents; Other works like Magnetic-Ong One [Fourney et al., 2024b] and AG2 [Wang et al., 2024] introduce a central orchestrator that assign tasks to workers. While effective, these systems hard-code role sets, limiting cross-domain transfer. In Pan et al. [2025], the authors find that major failure pattern of those works stem from system design issues. WORKFORCE instead decouples strategic planning, coordination, and task execution into independent agents, so new workers can be swapped in without touching the planner. The scalability and flexibility enables seamless domain transfer.
Post- Training for Agentic LLMs. Researchers have extensively explored post-training methods to enhance the performance of agentic systems. Prior works optimize the model on cool-augmented trajectories by supervised fine-tuning [Chen et al., 2024, 2023a, Zeng al., 2023, Hu et al., 2024, Qin et al., 2023] or reinforcement learning [Qi et al., 2024, Yu ct al., 2024, Wu et al., 2025, Wei al., 2025, Zhou et al., 2025, Jin et al., 2025, Zheng al., 2025, Jin et al., 2025, Feng et al., 2025] Although effective, full-trajectory training scales poorly and can over-fit the agent to a fixed tool suite. Another coccurent work MPO [Xiong al., 2025] sidesteps weight updates by iteratively refining high-level plans but still assumes task-specific reward design. In contrast, OWL trains only the domain- -annoic planner via reinforcement learning, enabling strong domain transferability.
Multi-Agen Training. Recent research has explored training methods where multiple LLM-based agents learn to collaborate through specialized roles and interactions [Ma et al., 2024, Slumbers et al., 2023, Subramaniam et al., 2024]. For example, MALT divides the reasoning process among generator, verifier, and refiner agent, and fine-tunes each role with off off-policy reward propaga- mon[Motwani et al., 2024]. Similarly, Subramaniam et al. [2025] propose a multiagent fine-tuning approach where LLMs initialized from the same base model are specialized through debate-drived data, enabling diverse reasoning and collective self-improvement beyond single-agent capabilities.
Unlike these approaches that train domain-specific multi-agent systems where each agent requires separate fine-tuning, OWL focuses on training a single generalizable domain-agnostic planner. This fundamental difference offers both effective domain transferability and training efficiency.
Conclusion
We introduced WORKFORCE, hierarchical multi-agent framework that decouples strategic planning from domain-specific execution, enabling cross-domain transferability without system redesign. WORKFORCE achieves 69.70% % accuracy the GAIA benchmark, outperforming both open-source alternatives and Deep Research. We also presented OPTIMIZED WORKFORCE LEARNING (OWL), which applies reinforcement learning and enhances the Qwen2.5-32B Instruct model by 16.37% on -nnsttsruttt the GAIA benchmark. We hope that by combining blug-ndd-play worker nodes with a generalizable planning core, our approach provides a scalable foundation for general- purpose AI assistants.
8 Acknowledgements
This work was carried out as collaborative open source research initiative at CAMEL-AL.rgg supported by funding from Eigent.AI. This paper is partially supported by the National Key R&D Program of China No.20227D00111000 and the General Research Fund of Hong Kong No. 17208825 and 17209324.

---

References
Authropic. Claude 3.7 sonnet and claude code, February 2025, URL ht httpps httphs//ww...ttttt///t// c.com//ew// cladde-laad--eee sonnet.. Accessed: 2025-05-13.
Dzmitry Bahdanau, Nicolas Gontier, Gabriel Huang, Ehsan Kamalloo, Rafael Pardinas, Alex Piché, Torsten Scholak, Oleh Shliazhko, Jordan Prince Tremblay, Karam Ghanem, et al. Tapeagents: a holistic framework for agent development and optimization. arXiv preprint 24Xiv:2412.08444 2024.
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning, arXiv preprint arXiv:2310.05915. 2023a.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprin! arXiv:2308. 10848, 2(4):6, 2023b.
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. fgant-flan: Designing data and methods of effective agent tuning for large language models. arXiv preprint 24X1v:2403.1288/ 2024.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504. 1153,, 2025.
Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang, Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, and Saleema Amershi. Magentic-one: generalist multi-agent system for solving complex tasks, 2024a. URL / //tp://aavvvvv//aa///aa/.aaaaaaa harg/abbs.
https:
Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al. Magentic-one: A generalist multi-agent system for solving complex tasks. arXiv preprint arXiv:2411 2...::411100468.. 2024b.
Google. Langfun. https: / ctm wwwggooeoggggggiiooo 2023. Version 0.0.1.
Shuhao Gu, Jialing Zhang. Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Infinity-mmi Scaling multimodal performance with large-scale and high-quality instruction data, arXiv preprint arXiv:2410. 18558, 2024.
Taicheng Guo, Xiuying Chen, Yaqi Wang. Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.00 01660, 2024.
H2O.ai. Autonomous agentic ai: execute multi-step workflows autonomously. https://h2o.at ipplat/oom/eeteerrsseeeeeeeee h2ogpte// tAgenticAI, 2024. URL htLhttsss////o.ee. hatformmeenterprisse-n222 h2ogpte/ httpr///eeeeeepppeee///tttaaaaaa #AgenticAI.
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang,
Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint 230.:230000035.. 3(4)66, 2023.
Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, and Saravan Rajmohan. Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation. arXiv preprint arXiv:2408.00 2000::20000006644, 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Ilms reason and leverage search engines with reinforcement learning. arXiv preprint 2rXiv:2503.095// 2025.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society. Advances Neural Information Processing Systems, 36:51991-52008, 2023.
10

---

Hao Ma, Tianyi Hu, Zhiqiang Pu, Liu Boyin, Xiaolin Ai, Yanyan Liang, and Min Chen. Coevolving with the other you: Fine-tuning 1lm with sequential cooperative multi-agent reinforeement learning. Advances in Neural Information Processing Systems, 37:15497-15525, 2024.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference Learning Representations, 2023.
Microsoft. Multi-agent experiment v0.1 msr ai frontiers (autogen team members). https:/ /atppps://aaa.. /aka. ms/ ms//aiaamml wttlllttainetttteeeeeeeee rrc__st, 2024. URL https::/ttaaktt..ee ms /ga a_mu diciagent_vl_marrcht
Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Rafael Rafailov, Ivan Laptev, Philip HS Torr, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. Malt: Improving reasoning with multi-agent IIm training. arXiv preprint arXiv:2412.01928, 2024.
OpenAL. Hello gpt-4o, May 2024. URL Accessed: 2025-05-13.
OpenAl. Introducing deep research. http::/://eeeettcccc//////a/aata p- researchh/ 2024. URL https: /ttp:///o/p////e/eeeeeeee htg.a//a..ooo/eeee////eooooonnno esearch/. Accessed: 2025-05-10.
Melissa Z. Pan, Mert Cemri, Lakshya Agrawal, Shuyi Yang. Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Kannan Ramchandran, Dan Klein, et al. Why Do Multiagent Systems Fail? In ICLR 2025 Workshop on Building Trust in Language Models and Applications, 2025.
Panupong Pasupat and Perey Liang. Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv: 150.00005, 2015.
Huy Nhat Phan, Tien N Nguyen, Phong X Nguyen, and Nghi DQ Bui. Hyperagent: Generalist software engineering agents to solve coding tasks at scale. arXiv preprint arXiv:2409. 16299, 2024.
Zehan Qi, Xiao Liu, lat Long long, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning. arXiv preprint arXiv:241 1.02337, 2024.
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. arXiv preprint arXiv:2307.07924 2023.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-worl world apis. arXiv preprint arXiv:2307. 16789, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https: / //openreview,...m/ net forum? dis.iisSSSaSSSSS
Aymeric Roucher, Albert Villanova del Moral, Clmentine Fourrier, Merve Noyan, and Thomas Wolf. Open-source deepresearch - freeing our search agents. https: httpp::/huuggiiiii httpg//ggggggccceee///// co/blog/ open-deep- resea rch, February 2025a. URL https: http://:///hggiggfcceeee cttb.cooooooooooo.oooooooooo en en-deep- research. Accessed: 2025-05-10.
Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismki.
'smolagents': a smol library to build great agentic systems. https: //gi thw.. com /huggi ngface/ watpceeesmolaggenn 2025b.
Oliver Slumbers, David Henry Mguni, Kun Shao, and Jun Wang. Leveraging large language models for optimised coordination in textual multi-agent reinforeement learning. 2023.
11

---

Vighnesh Subramaniam, Antonio Torralba, and Shuang Li. Debategpt: Fine-tuning large language models with multi-agent debate supervision. 2024.
Vighnesh Subramaniam. Yilun Du, Joshua Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse reasoning chains. arXiv preprint arXiv:2501.05707, 2025.
Jiabin Tang, Tianyu Fan, and Chao Huang. Autoagent: fully-automated and zero-code framework for Ilm agents. arXiv e-prints, pages arXiv-2502, 2025.
Trase. Meet trase systems, the ai agent platform. https: / www. www.stassssttsssscoomm com/, 2024.
URL httptttpp/// //www. trasesysteesss com/. Accessed: 2025-01-15.
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: controllable world of apps and people for benchmarking interactive coding agents. arXiv preprint arXiv:2407.18001, 2024.
Chi Wang, Qingyun Wu, and the AG2 Community, Ag2: Open-source agentos for ai agents, 2024. URL https::/githtii wwt.coo////ooo/ http//ggaaa/ggga ag2. Available at https://docs.ag22 ag..aa/
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing 11m reasoning via
reinforcement learning open software evolution. arXiv preprint arXiv:2502. 18449, 2025.
Junde Wu, Jiayuan Zhu, and Yuyuan Liu. Agentic Reasoning: Reasoning LLMs with Tools for the Deep Research. arXiv preprint arXiv:2502. 04644, 2025.
Zhiyong Wu, Chengeheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self seff-improvement. arXiv preprint 240.0240000045660 2024.
Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Jumning Zhao, Qian Liu, Che Liu, Leo Z. Liu, Yiheng Xu, Hongjin Su, Dongchan Shin, Caiming Xiong, and Tao Yu. Openagents: An open platform for language agents in the wild. CoRR, abs/2310.10634 48550/ (00/5/0///ii.... URL https:/ /tt.//oiioo//// org /0t.///...50000///1
2023. doi: 144.555/AAXII.21110006666.. 10634.
Weimin Xiong, Yifan Song, Qingxiu Dong, Bingchan Zhao, Feifan Song, Xun Wang, and Sujian Li. MPO: Boosting LLM Agents with Meta Plan Optimization, arXiv preprint arXi::2503.02682 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
Yingxuan Yang, Huacan Chai, Shuai Shao, Yuanyi Song, Siyuan Qi, Renting Rui, and Weinan Zhang. AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systems. arXiv preprint arXiv:2504.00587, 2025.
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint S00::1800009000 2018.
Yuanqing Yu, Zhefan Wang, Weizhi Ma, Zhicheng Guo, Jingtao Zhan, Shuai Wang, Chuhan Wu, Zhiqiang Guo, and Min Zhang. StepTool: A Step-grained Reinforcement Learning Framework for Tool Learning in LLMs. arXiv preprint arX1v:2410.07745, 2024.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for 1ls.. arXiv preprint arXiv:2310.1287) 2023.
12

---

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URLhttp: http::/aatt httt:///rxxi.orrt org/ abs 13372.
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai. Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforeement learning in real-world environments. arXiv preprint arXiv:2504.03166, 2025.
Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks. arXiv preprint arXiv:2503. 2025.
13

---

Appendix
A
Limitations
Despite our modular architecture's plug-and-play capabilities, performance remains dependent the availability of high-quality domain-specific tools. In domains lacking reliable toolkits, the system may face execution bottlenecks. Additionally, reinforcement learning from real-world feedback is time-consuming due to latency issues (such as delays in online searches).
B
Contributions
Paper Writing, Mengkang Hu and Yuhang Zhou finished most part of the paper. Mengkang Hu, Yuhang Zhou, Bowei Xia, and Yuzhou Nie finished the appendix part. Mengkang Hu and Ziyu Ye collaborated the related work part. Yingru Li and Ziyu Ye helped refining the abstract and introduction sections for multiple rounds. Yingru Li, Ziyu Ye, and Qiguang Chen carefully reviewed the paper and gave helpful feedback.
Experiments. Mengkang Hu, Yuhang Zhou, and Bowei Xia conducted experiments of Table 1 (including baseline experiment). Mengkang Hu and Yuhang Zhou conducted analysis experiments. Mengkang Hu, Yuhang Zhou, Yuzhou Nie, and Ziyu Ye conducted experiments Table 3, including model training and evaluation.
Dataset Construction. Mengkang Hu designed the data collection schema and pipeline. Mengkang Hu, Yuhang Zhou, Bowei Xia, Tao Sun, Yuzhou Nie, and Zeyu Zhang collaborated on data collection, with each one responsible for an individual part of the dataset.
Code Implementation. Mengkang Hu and Yuhang Zhou implemented the code base of the OWL project and experiment pipeline based on the CAMEL [Li al.. 2023] framework. The agent infrastructure by CAMEL sped up the implementation. Yuzhou Nie, Mengkang Hu and Yuhang Zhou collaborated on the implementation of the model training pipeline. Mengkang Hu, Yuhang Zhou, and Yuzhou Nie designed and implemented the data collection pipeline, including SFT and DPO datasets. Mengkang Hu and Yuhang Zhou designed and implemented the essential toolkits for generalist ai assistants, including browser toolkit, document processing toolkit, multimodal analysis toolkit (for audio, video, and image), excel toolkit, and so on. Qianshuo Ye helped with refining the multimodal analysis toolkit. Zhaoxuan Jin implemented the initial version of the workforce. Mengkang Hu and Yuhang Zhou collaborated on refinement (including adding new features and debugging), instantiation, and implementation of the general multi-agent assistant.
Open-source Maintenance. Wendong Fan (Lead), Mengkang Hu (Co-Lead), Yifeng Wang. Tao Sun, Qianshuo Ye, Yuhang Zhou, Bowei Xia, Guohao Li, and other open-source community members helped maintain the open-source implementation OWL. Their efforts in improving code stability, adding auxiliary features, and managing community feedback have helped enhance the usability and visibility of the project. While the research, design, and writing of this paper were conducted independently, the continued maintenance work by the team played a supportive role in sustaining the project's public engagement.
Error Analysis and case study. Mengkang Hu, Yuhang Zhou, Bowei Xia, and Guohao Li designed and conducted the error analysis, including the evaluation results of role playing and workforce.
Methodology Proposal. The initial ideation of the Workforce framework was led by Guohao Li. The Optimized Workforce Learning approach jointly proposed by Guohao Li and Mengkang Hu. Yuzhou Nie, Yuhang Zhou, Ziyu Ye, and Yingru Li also participated in the discussion regarding the methodology proposal.
Project Leadership. The overall direction of the project was led by Guohao Li and Mengkang Hu. Guohao Li also supervised the coordination and integration of all research and engineering components. Mengkang Hu led the implementation and experimental evaluation of the research project.
14

---

C Formalization
C.1 Large Language Model-based Agents
Formally, an LLM-based agent can be defined as tuple O,, .T), where M is the underlying model, O represents the observation space, T is the set of available tools, and O + T the policy function. The agent operates in penception-resooning action loop, where each step t, it receives an observation appropriate action d, = т (ox) T. and executes this action to achieve its goals. Multi-agent systems extend this paradigm by enabling multiple L.LM-based agents to collaborate. A multi-agent system can be represented as S = {-A1, A2, ...; An, C), where each A; an individual agent and defines the communication protocol between agents. Frameworks such as CAMEL [Li et al., 2023] and MetaGPT [Hong et al., 2023] have demonstrated that Collaborative multi-agent approaches can outperform single-agent systems complex tasks requiring diverse expertise.
O, processes it through the language model M to determine the
D
More Details on Workforce
D.1 Workforee processing flow
Define the following symbols: F denotes the failure count, I represents the failure information set, B indicates the task failure status, P represents the planner agent, C represents the coordinator agent, and T represents the trajectory set. The overall processing flow of WORKFORCE is shown in Algorithm 1.
Algorithm Workforce Framework
Require: Task T, Worker Registry W, Task Channel C. Max Replanning Tries K
Ensure: Final Output O
1: Initialize:
2: F +-0 0
3: I 0
4: B + False
5: while F do
6: Planning
7: S S+ 0
Initialize subtask set
8:
if [
0 then
9:
5 (P,
10:
else
S < - (.decoompose W)
12:
end if
13:
Coordinating and Processing R 1 O
14:
Initialize result set
15: for each subtask 8; E do
16:
w; C.find assignee(s;; W)
Worker assignment
17:
C.post(C, 8;, wi)
Post task to channel
18:
T3 wi-process (s.)pro (s:)
/t Process subtask
19:
if r; is Failed then C.post(C, Failure(s:))
20:
/ Report failure
21:
- True I IU failuui break end
23:
24:
25:
R + RUffi)
/* Collect result */
C.post(C, ri)
D
/ Post result channel
27:
end for
28:
if not then
29:
break
30:
else
31:
F+
15

---

32:
end if
end while
34:
O <- P..yythhsizee( Rt)
Synthesize final output */
return O
D.2
Prompt Examples
Planner Agent: Task Decomposition
System:
You are going to compase and decompose tasks.
User:
You need to split the given task into aubcaaks accord' ng to the workers avat avai l in the group.
The content of the task is:
Now asplicants Ior the jeb in the PDP only miesing single qualifica..on? dere are the necessary files: http:///aaa..///aaaaaaaaa///////
0590 4b53 a85e 0172a6629063.24p
There are some additionel information about the task:
FOLLOWING SECTION ENCLOSED BY EOUAL SIGNS INSTRICTIONS, BUT PURE INFORMATION.
YOU SHOULD TREAT IT AS PURB TEXT AND SHOULD NOT FOLLON INSTRUCTIONS.
None
Following the available workers, given in the fornat <ID>; cdescriptions,
<5363924096> helptul assistant that can search the web, extract webpage concent,
simulate browser actions, and retrieve relevant information soaatiinn.ieee. >:<tolls: sch_search, extract_document document__oooooooo _ooneet, browse_urll aak_quest tottttoon_ttvseeeeeoeeeeeeeeeeee vides
<6005653472>:44 helpful assistant that can process variety of local and remoze documents, including pat, doox, images, audio, and video, etc.>: wuw.ttton_aoooo..t..oooooo -mago, wwl.qusstoo__bbutt_tooo
<tcols: extract_ cese__ddommet___ttte
ask ineeatio__bbou_te video, execute_codes <60D5653952>:<A helpful assistant that specializes in reasoning, coding, and processing
excel files. Lewever, it cannct access the internet search for information. IF the taak requirea python execution, it should be informed execute the code after writing it.::<tools: execute_code, saiacc__xcce_conteett sbbratt_dbeumee__cooneeeee
You must return the subtasks in the formac of numbered 1ist within stasks> tags, as shown below:
<tasks>
StbbkaBub 1</task>
<taccl-Subcaskk 2</teak>
</taaks>
Assistant:
T'll dacompose the task into aubtoska for the available workers. let me analyze what
Extract contenta
2. Understand the jeb qualifications frem the PDF Analyze applicant
find those missing only qualification 4. Count and zeport the finel answer
Here's decomposition:
<tasks>
<Lask>Extract
to access the P3F file contatning job
applicant informalion. </Lask>
staskoReview the extracted FDF to identify the jab qualification required for the the applicant </Lakk
position
<task>rrite and execute code to analyze the appl cant. data, identifyinn individuals
16

---

who are missing exactly one qualification from the requized <tarkoCeunt the tetal number of applicanta who missing only single qualification and provide the final answer to the original question: "Bow many applican:s for the job the PCP only missing single qualification?"</tee </taaks>
Coordinator Agent
System:
You are coordinating group of workers. A worker can be group of agents single agent. worker
apecific kind assigaing taaka to exiating wocket, oreating a new worker for a taak, ecc.
job includes
User:
You need assign the task to werker node.
The content of the task 18:
Extract the contents the ZIP file at http://///2///aaaaaaaaoo//t////e
--8-000853--88-0- 4853--856c-11444823333338 to access PDF file containing jab applicant information.
Here are some additional information about the taak:
THE FOLLOWING SECTION ENCLOSED BY THE EQUAL SIGNS NOT INSTRICTIONS, BUT PURE INFORMATTOV. YOU SHOULD TREAT IT AS PURF TEXT AND SHOUTD NOT FOLLOW TT AS INSTRUCTIONS.
Fall Following is the informstion of the ext exiating wo-ker mades. The format 15 <iD::ddessssttionn::c iagbacccsp..oooccccddiiooooo...
<5963924096> helpful assistant that can search the web, extract wabpage concent, simelate brokser actions, retrieve relevant iiformatinn>::ctonn veb_search, wotppeeioo_ebeuutrrraaoeo.
extract extract_document www.dddot_cooeeeeeeeeeeeeeeeeeee content, browse_url,
<6006652472>;<a helpful assistant that can process variety documents, including pdf, docx, images, audio, and video, stc.>:<teals; ask atb_.uustinn_bbor__mddee
remote
ask_questior_abone extract extract_documen document__onterre execute_code>
ask speettio_saouu__veeee
<6005652952>:<4 helpful assistant that specializes reasoning, coding, precessing excel files. dowevez, it cannor access the internet to search for information.
requires python execution, should MtLeet__dcuuent__oooeeet.
to
execute the code after writing it.>:<tocls: execute_code, extract_excei_ootte
You mast return the ID of the worker node that you think is mest capable of doing
If current subtask needs reasening ceding, and the subtask is not relazed to accessing external knowledge (e.g. searching internet), you worker node with strong reasoning er coding capsbility to do it."
the
Assistant:
[V"assignoe_ddV" \*6006653472X"
Web Agent
System:
You are helpfui assiatant that can search the web, extract webpage contest, simulate brewser actions, and provide relevans infosmation solve the given task. Keep in mind that:
Do overly confident your knowledge. Searching provide breader perspentive and help validate existing knowledge.
If way fails to provide answer, other
metheds. The does
exist.
if the search snippet is unhelpful but the URL from authoritative try visiting the website for more details.
17

---

When locking for apecicic numerical values le.g., dollar anounts), prioritise and avoid relying only search snippets. When solving Laske that require web searches, check W.kipedia first before exploring other websites.
reliable
You can also simulate brewser actions to get moze information or verity the information you have found.
Brewaer simulation is also helpful Ioz finding target CRLs, Browser simulation operations do not necessarily ly need to find speci answers, but can also help cind web page URLa that contain answers (usually difficult to rind thzough simple web searches). You can find the answer to the question by performing subsequen: operations on the URL, such as extracting the conteat of the webpage,
Do not solely re y document tools or browser simulation to find the anawer; you should combine document Lools and browser sinulation to cempeehensivell information. Some content may need browser simulation to get, or some content la rendered by JavaSeript.
process weo
In response, you should mention the URLs have vis'ted and processed.
Here are some tips that help you perform web search:
Never add too many keywords your search query! Some detailed results to perforn browser "nteraction to get, not vaing search toolk't.
the question answers. It is not likely to rind the answer direct using search toolkig only; concise focus finding official sources
complex,
results typically
provide precise
search query than df rect answers.
for example, the first National Geographic short YouTube that was ever released according to Montorey Bay Aquerium kabsito?, your coarse-grained like Naticnal Geographic Youlube to find the Youlube websize first, and try other fine-groined search terms step-by-step The results you return do not have to directly answez the original question) you only to collact relevant information.
question
maximum length metera
search
must
more URis.
User:
We are solving a complex Lack, and we have split the task into several subtasks.
nead process one given task. nont assume answer doea exist. If cant solve the task, please describe the reason and the result you have achieved in detail.
prohlem unsolvable.
The content of the task that you need to is:
<task>
Access the Collins Doanish-to-Anglise dietionary online and lezate the 1934 example
aa sentence for the identified spanish word
</task>
Sere is the overall task for reference, which containg some helpful information that help you solve the
coverall task>
The latin root of the Yola word gimlie shares apalling with a spanish word, What is title the example word is the Collins Spanish to English dictionary enline? Answez in plain text, punctuation.
Google tranalation
for that
</cveroll_tode tosks
sere results of some preroquisite results that can refer to (empty if chere 4 are no prerequtaite reaults!:
<dependency_res Bectdeeyyyreeu_____ weaglyrreuutt_eeeeeee teauite_infoo
The latin Root the Yola Word Gimlie
Stymology
According wiktionary entry gimlin, vola word has following elymology:
The Yola word ginlie (meaning chimney) Irom Middle English chineney, which turn cones from old French cheminee. The ultimete Letin zeot of the word is late Latin cominto.
Latin
The latin word caminta is the from which the Yola ward gimlis derives. This + Tatín root shares a ape apelling with the Spani sh word cam" rssa, which meana (logg) ramble Spanish.
walk, hike,
ELymology
Yola: giml ie: (meaning chimney!
18

---

Middle Bnglish; chineney
old French: cheminee
Late Latini caninta
Nodern Spanish cognate: caminate
Semantic Evolution
The Tatin root caminta is related to the Tatín word caminus [meaning 41 "ereplace forgel, which itself comes from Greek (kaminos, meening even or Surnace). The semantia shift from hesting stricture chimney occurred during the svolution
+ of these worda,
Conclusion
This research confirms that the Latin root of the Yola word gimlie is caminta, which spelling
medern Spanish word
Sources
3.ht:pps://ictiinn cambridge.. wwtp:///deeeionee./e//e /oddeiioaar//sppnnillle english/ caminata
insdict ierredictionaaee conaty.com. com/ diet Lonary/ /sppi/ssssssssseaeaaa -nng ish/ camnnamiaaaa http:://wwww....mmmciitieeeeee
h.http::/web.archige wwtp:///g////eeeeeeeooeeeteeettt
com: wwt.ai/toooccc//paaaaaa. www.eeenggglgaiiiiieeeeeeee Lan/caminaaa
Finding Spanish Nord that Shares the Same Spelling as the Latin Root of Gimlie
Answer: caminata
According to the dependency resulta provided, the Latin root of the Yola word gimlle caminta, which shares same spelling Spanish word (meening (leng) walk, hike, trek, or ramble in modern spanish).
verification Process:
1.Erom dependency results, confirmed t which evolved through this chein:
gimlie
gimlia (meaning chimney!
-Middle English: chimenay
French: cheminea
Late Latin: caminta
2. The modern Spanish
shores same spelling thouh mocron over the
4 1) caminata.
by checking which confirms that caminata is Indeed Spanish word meaning long walk, hike, ramble.
this
spantssh-nngl Diacionary,
Sources Consulted:
dependency results provided in the task
Collins Spanish Eaglish Dictionary via web web.archive.org cambridge Victienary
The spanish word that shares the spelling the Letin of gimlie is caminata.
</ cispendency_rouulta_oo
Sere are some additional information about the task (anly for referance, and may be emptyl:
codditional. lac_oanall_ffoaaoeee info>
</additional in infa>
Now plaase fully leverage the information above, try your best to laverage the exlating resulta and your toola solve the
Assisl)
"id"; "toolu_ "toolu_(12yee (12.yy/y////////tttttttt4tt
"Lype": "function", "Sunctionn:
"name"; "web_search",
"azgumenta"; "/\*question\"; \*Collins Dpanish-Bng.ies dinaro caminata 4 onlinel*j*
tool:
Here are some rei evant: resouTcea for the Col 1ins Span' sh-Rngl ish Di lonary online:
19

---

multimodal data, such 1. Collins EngLish Spenish Dietionacy +RUR [Colline English--panish Spanish
Dictionsty] http://wwwwlllnddittttcoooo www.oooorrrereeciiiooeeeeeee www///ittoonsrr/eggggllaaa spaniish))
** informationn this cnline dictionary provides complete knglish-Spente
translations with accurate promunciations in both languages, usage examples, and granmaz quidea.
sxCollins Spanish English en App
+*URie [Collins Spanish- on the App
Store] (hitps://apps htt.tpsss//pps.ppppe.... wttppe.com/ss/ap/////ppp//pppppp http:///////////lliaa spanish engLish/id29223999
*ninfrrmatoon This
offers dependable Spenish and English reference with over 130, tranalations,
conjugations, usage notes, and examples. It Functiona without requiring internet connection.
These resources should help you acceas
Spanish-English Dictionary
Document Processing Agent
System:
helpful assistent that can precess images, audlo, and video.
User:
We solving complex task, and have split the task into several subtasks.
You need to process given task. Don't asaume that the problem is unsolvable. The answer dees exist. IE you can't soive the task, please describe the reasen and the result have achieved in detail.
The content of the task that you need to is:
<task>
Review the extradted PDP to identify the job qualifications requized for the position and understand the formet of the applicans date.
</task>
Jere is the overall Lask for reference, which contains helpful inTormation that can help you solve the task:
soveral 1_task>
Now applicants Ior the job in the PDP only miesing single qualifica.ion? the recessary files:
httaaa/a//.///aaaaaaiioo/oeeeeee </overall_taag
0s90 4b53 a85e 0174
Tere reaults of some prerequisite results that you can refer to (empty chere are presaquisite results):
<dependency_ resuita_inoo Extraction Report
#1 Summary
have successfully extracted the contents of the ZIP file at
"data/gsta/2623/ee htt/://aaaaaaioonooo 1 1--6600-h533-880--00 2aM3--8800-11244559933.. 248869933.12211 The extraction was performed to directory named foxtrasted_files" files".
#4 Extracted Filas
contained files:
1. ++Job Listing.pdÉ-. EDF file that likely contains information about the
job requirenents
2. --xgpliconts..llew An Excel file that likely conteina information about the job applicants
Locations
The PCF file is located at "extracted Listing.
is
"extrrcttdd filee///geeieeeeeegeeeeettt
Steps
3ased on the overall task, which is to desermine "Bew many applicants for the job in are only missing single qualification?",
extracted files will essential for the next steps. The PCP file likely contains the job qualifications, while the Excel file likely contains the applicant informeticn and their qualifications.
# Status The cask has been completed successfully, and the files are new accessible for further processing.
</dapendene__ronute__imm
additional information about the task (anly for referance, may
+ emptyl:
20

---

cadditional_intoo
None
</ </addittional infos
Now please fully leverage the information above, try your best to leverage the existing results and your available tools solve the curzent task.
Assistant(tool.cam calls):
"id"; "Loolu_ 01WG6511110001180000045555555
"type*; "function",
"Function::
"name"; Textract_docurent htt____cueet_tootteee
"arguments": 11"documant__pacay \"xxeaated__ firrrctto__fles/06 ldsting.pdfV
i'query>" \*Wnat are the job qualiticationa required for the position?\"
tool:
3iclogist at ADC Biotech Research Company
Job Title: Riologist
Company: ABC siotech Research Co.
Job Type: Full-time
About the Company:
ABC Hiotech is leading biotechnology company dedicated to advancing
Reasoning and Coding Agent
System:
You are a helptul aasiatant that specializes in reasoning and coding, and can chink step by to solve the task. When necessary, you write python code solve the task. if you have written code, do not farget to execute the code, Nevez gencrate codas like 'example code', your code should on abla to fully salve the task, You can also leverage multiple libraries, such as requests, Reaut Sulpo processing excel files, you should write
re, pandas, elc, codes process them.
User:
We are solving complex Lask, and we have split the Lask into sevezal subtasks.
You nead to process one given task. Don't aasume that the problam am is unsolvable, The exist. If you 4 result you have achieved in detail. you need
task, please describe
Write and execute code to analyze the applicant data, identifying individuals who are missing exactly one qualification the sequiced list.
</task>
sere is the overall task for reference, which containa come helpful information that can help you solve the taak:
coveral task>
ноw many applicants for the icb in the PDP are only missing singla qualification? Reee are the neceasary tilea:
0693 4b53 a85c 0174a8629033.
</cverall task?
Here are results of some prerequisite results that you can refer to (empty if chere prerequiaite resultsl:
<dependency. resuita__nia infor
21e File Extraction Report
# Summazy
have successfully extracted the contents of the 2IP file
"data/gata/200 22023//222// idat i -00---0-0000--------000011110110 extraction performed directory "extracted_[iles""
The
21

---

#4 Extracted Piles
The 2IP file contained two files:
1. +.Job Listing-pdie A EDE file that likely contains information about the job requirements
An Excel file that likely conteins information about the job 2. .*Agplicants.
applicants
# File Locations
the PUF file is located at eatraccted Listing.pdf
Excel file is loested "extracted_ witps//ppppllpppppleaeaap xxnta.xxsxx
#$
aased on the overall task, which is to determine "Bow many applicents for the job in only misaing aingle qualification?"" theae extracted lea will be + essential for the next atepa. The PCP file likely contains the job giaaitictionn, likely containa the applicant qualifications.
the
thair
#4 Status
completed successfully, the are new acceasible
- proceasing,
Job qualifications and Applicant Data Analysis
sased on my review
extracted and the applicant data, I've job qualifications and the formas of the applicant data.
#4 Job Qualifications sequired for the #osition
From the *Job Listing.par" tile, I've identified the following qualifications required - for the siolagist position at ABC Biotech Research Company:
1. Eduuatioone:: Masters Degree or higher in biology, biochemistzy, or biotechnology sxperieece==:: 3+ years of experience
3. Training:.:; Training with leberatory equipment
4. ssublications::: 3+ publications in the field of biotechnelogy
5. +cCiiieeehiig-e: Citizenship
Countzy
6. **rregramingg<:: C-+, Ct,
experience
7. *=Laguugee 11 second lenguage
#4 Applicant Data Fornat
The "Applicants..kle xx1xx" file contains infermation about job applicants with the following structure:
1. Namemes Applicant's full
2. Degree Fielde+: Field study (Biclogy, Biochemietry, Biotechnology, etc.)
Degree Levele*: Education level (Hachelor, Kaster, Ph.D., etc.)
4. Experience (Years)*e: Number of years of experience
5. =sublicationse:: Number of publications
6. ++Lab Trained (X/3))++ Whether the applicant has laboratory treining Yes, N No)
7. ..Citizen (Y/N) **: Whether the applicant is a citizen (Y = Yes, N No)
**dregramingg Lengue: Programming language experience
9. .+Sacond Language**? Second language knowledge (NaN/missing valuaa indicate na aecond language)
#4 Observations
Excel containa
applicants
All columns are complate except for *Sacond language" which has 15 missing values missing values "Second Language* likely indicate applicants who don't 't have
+ secend language
job qualificationa, allowing direct comparison + between requirements and applicant qualifications
format aligns
This information will be essential for determining which applivants are missing only everell task objective.
single qualification the jcb,
Assistant(tool calls):
"id"; "tcolu_ "tcols_01B1DAYE LLBLDDABLL:::UUB22EESSS.. *Lype"; "function", "fnnction""
"nane"; "execute_code",
22

---

"azgumenta"; "[o"o"""";; \"import pandas as pdl lninport numpy as apVAn\ane
Load the Excel silellnapplieents_.de
("extracted__"]eee s. Aanaa Aniint
Define Tunction
applicent ckeck_reguiremssttttooo :\in requirements_met [11\n
requirement Wadee
regu []Vin Vin # Educations Masters Degree
higher in biology, biochemistry, or Mootechnoooyy/ie
valid_degrees
filidf ['Biology's 'Biochemistry" if rewl Level'] in valid_degress and cowl [owl"Degee DDDezee Field'] in felldd(ie
l'Kaster', 'Ph, V
"iooecchologyy
requirements_meet
append. aageee "dducation") Va
tool:
Executad Results:
Number of applicanta missing exactly one qualification: 17
Applicanta missing exactly one qualification:
Nebil Hates: Missing publications
BaEwan Sanders: Missing Education
D.3 Worker Implementation and Toolkit Design
WORKFORCE employs specialized worker agents, each equipped with specific tools to handle different aspects of complex tasks. The implementation follows modular design where each worker agent is instantiated with a tailored prompt, a foundation model, and set of tools optimized for its domain of expertise.
As shown in Table 4, when solving tasks from GAIA, WORKFORCE consists of three primary worker agents:
1. Web Agent: Specializes in web-based interactions, including search, content extraction, and browser simulation.
2. Document Processing Agent: Handles multimodal data processing including documents, images, audio, and video, etc.
3. Reasoning and Coding Agent: Focuses on logical reasoning, code execution, and structured data analysis.
The tools assigned to each agent are carefully selected to align with their specialized functions while avoiding unnecessary overlap.
As is shown in Table 5, our tool design is carefully configured based on downstream tasks. Each area implemented with appropriate safeguards and optimized for specific use cases:
Web Search:
Provides structured access to search engines, Wikipedia, and archived web pages with rate limiting and content filtering
Browser Simulation: Enables web page interaction through specialized dual-model approach where one model observes and acts while another plans navigation strategy
Document Processing: Handles various document formats with efficient parsing and information extraction capabilities
Multimodal Analysis: Processes different media types using specialized models for each modality
Code Execution:
Provides Python code execution using sandbox or subprocess module
This modular design allows for easy extension of the workforce with additional specialized agents and tools as needed, while maintaining clean separation of concerns between different functional domains.
23

---

Table 4: Worker Agent Implementation Details
Worker Agent
Tools
Model Backend
ggoole
search wiki
search wiki revisions
Web Agent
search.archived.wee.... webage
GPT-4o Seaude-3.7-oomme
extraet.doocmeen..cottti browsc_ur! eeguestion_aboutt _abouttvvdeoo
dorrrmeddcmment...mm content
usk_qqeetton_abee
www.aaabbbtasaaeee image
Document Processing Agent
ask_question_abon catiooonnauuttuuuuuaaa _audio
GPT-44o / Claude-3.7-Somme
ask ass..qustion_abee cabouutonabbttt..o video execute_com _ode
execute_code
Reasoning & Coding Agent
extract.excel.conter
03-mini / Claude-3.7- 3.7-Sonnet
extract_documeet__commnn
Table 5: Toolkit Design and Implementation Details
Toolkit Category
Tools
Toolkit Design and Implementation Details
scarch.google
search_wiki
Web Search
API-based web search services
search.wiki.revisionn. search_archivee search_archived_weerr
GPT-4o (For action execution)
Browser Simulation
browse url
O3-mini (For navigation planning)
extract document content
Document parsing API services and libraries
Document Processing
extract.excel.conte
e.g. html2text, Unstructured, Firecrawl, chunkr
ask question about about ask questtuonnnnnn about audio ask aak.question.abouu vidot
GPT-4o (For images)
Multimodal Analysis
Whisper-1 + O3-mini (For audio)
Gemini 2.0-Flash (For video)
Code Execution
execute_cod
Python Executor
In addition, the toolkit can also be extended to support more specialized agents and tools as needed according to the task requirements.
D.4 Synthesis Trajectories Statistics
Table 6 presents detailed statistics of our synthesis dataset. We collected 3,466 trajectories in total in four different datasets, with 1,599 remaining after data cleaning. On average, each trajectory contains 3.41 subtasks and 22.75 execution steps. Among these datasets, Math-related tasks require
Table 6: More details the statistics of our synthesis-base supervised fine-tuning dataset.
Dataset
Total
Filtering
Avg. Subtasks
Avg. Steps
Agent Capabilities
HotpotQA
998
495
3.49
26.92
WikiTableeQuetionns
869
577
3.03
20.81
Math-related
1100
487
3.87
23.32
Infinity-MM
499
40
2.88
16.56
Total/Average
3466
1599
3.41
22.75
24

---

Table Categorization and distribution of failure modes on GAIA benchmark.
Error Type
Explanation
Proportion (%)
Planner Error
Errors stemming from the planner agent.
21.15%
Incorrect Plan
Planner produced entirely incorrect task decomposition.
19.23%
Subtask Ambiguity
Plans were correct but ambiguous.
1.92%
Worker Error
Execution failures hy worker agents during subtask handling.
9.62 %
Tool Selection Eroo
Wrong tool selected for subtask execution.
9.62%%
Limited Tool Capability
Failures caused toolkits' limitations. Failures due to inability of web agent to complete tasks.
32.69%
Web Toolkit Failure
13.46%
Document Toolkit Failure
Specific document processing tools.
1.92%
Multimodal Toolkit Failure
Errors when handling multimodal inputs.
17.31%
Responding Error
Final response generation failed after successful subtasks.
Response Format
Output did not conform to expected format.
3.85%
Reasoning Error
Incorrect answer despite access correct information.
3.84%
Langunge/Question Ambiguity
Ambiguity in user input led to misunderstandding
9.62%
Limited Model Capability
Model- internalt limitations.
19.23%
Hallucination
Model fabricated non-existent facts.
1.92%
Context Exceed
Model failed due
length limitations.
5.77%
Limited Coding Capability
Model failed generate usable code.
11.54%
the most subtasks (3.87) while HotpotQA tasks involve the most execution steps (26.92). Each data set requires different agent capabilities, and reasoning is a common requirement across all data sets.
E
Error Analysis
E.1 Error Distribution
Table 7 shows the categorization and distribution of different error types encountered in the GAIA benchmark. The errors are broadly classified into six main categories: Planner Error, Worker Error. Limited Tool Capability, Responding Error, Language/Questte Capability. Among these, Limited Tool Capability accounts for the largest proportion (32.69%) of failures, particularly in web toolkit and multimodal toolkit operations. Planner Error the second most common type (21.15%), mainly due to incorrect task decomposition. Limited Model Capability represents 19.23% of errors, with coding capability limitations being a significant factor. Worker Error and Language/Questiin Ambiguity each account for 9.62% of failures, while Responding Error makes up 7.69% of the total errors.
Ambiguity, and Limited Model
E.2 Error Distribution by Levels
Table 8 presents detailed breakdown of errors across different difficulty levels the GAIA bench- mark. Level 2 tasks show the highest number of errors overall, particularly in web toolkit failures (6 cases) and multimodal toolkit failures (6 cases). Level tasks demonstrate increased complexity with more tool selection errors (4 cases) and context exceed emrors (3 cases) compared to lower levels. Level tasks have relatively fewer errors across categories, though they still show some issues with incorrect planning (2 cases) and limited coding capability (2 cases). This distribution suggests that error patterns vary significantly with task complexity, with higher levels generally showing more sophisticated failure modes.
25

---

Table 8: Detailed error counts by type and GAIA difficulty level.
Error Type
Level 1
Level 2
Level 3
Sum
Planning Error
Incorrect Plan
4
4
10
Subtask Ambiguity
0
1
1
Worker Error
Tool Selection Error
1
4
5
Limited Tool Capability
Web Toolkit Failure
6
0
7
Document Toolkit Failure
0
Multimodal Toolkit Failure
9
Responding Error
Response Format
2
0
2
Reasoning Error
I
I
2
Language/Questn Ambiguity
2
Limited Model Capability
Hallucination
00
0
1
Context Exceed
3
3
Limited Coding Capability
4
0
E.3
Error Cases Examples
Incorrect Plan: Error Case Level3 #12
was referencing each of the tables the file from papers that were cited by the "Trans fatty acid contents in chocolates and chocolate wafers in Turkey" paper. lost my own reference sheet and need know which of the papers each table came from. The file may not use the full table caption. If the references in "ren'sans fatty acid" paper bibliography were numbered starting with I, give me the numbers in the order that they would be used to fill the cells the Excel file from top bottom,
Question:
comma separated list.
Answer: 8, 29, 22, 1, 8, 26
Annotation steps:
Searched "Trans fatty acid contents in chocolates and chocolate www.eacchgec.eepppuuiittooo2---.
wafers in Turkey" on Google. 2.0pened
Trans fatty acid contents in chocolates and chocolate wafers in Turkey. 3.Opened the Excel file. 4.Searched each reference in the paper on Google. 5.Checked any free-to-access reference for a table similar to the titles in the Excel file. 6.Added the numbers of the references to the Excel file. 7.Copied the numbers into comma-separated list.
Incorrect Plan Error Analysis
Analysis: The subtask only lets the agent match the table content with the reference paper, without further extracting the content of the reference paper itself.
Agent Content:
("subtask": "Write and execute code analyze the Excel file content and match the tables with the references from the paper.".
"result": TTabl-to-Referene bibliography from the "Trans fatty acid contents in chocolates and chocolate wafers in Turkey" paper, I've matched each table its corresponding reference number.
Matching AnalysisBasel on my analysis of the Excel file and the
26

---

Subtask Ambiguity: Error Case Level3 #18
The year 2022. am at the National Air and Space Museum east of the Potomac River. want go to Fire Station 301 DCA ARFF using the metro. go in the wrong direction and end up at the station closest to Cleveland Elementary School. How many metro stations am away from my original destination if I don't change lines? Your answer should be numerical integer value.
Question:
Answer: 8
Annotation steps: 1. Google search "National Air and Space Museum". 2. Note there are two National Air and Space Museums. One in Virginia, the other Washington D.C. Google map search "Potomac River" and zoom out. 4. See that Washington DC is east of the Potomac River. 5. Determine that National Air and Space Museum refers to the one Washington D.C. 6. Google search "Metro Station National Air and Space Museum Washington D.C.". 7. Clicked on the first result: Getting Here - National Air and Space Museum, http://airaaddpaac.sictuuuu http:acc.ii.cuuiiiii/uuuuuuem dc/directions. 8. Read on the website, "The closest Metrorail stop is L' E'Enantt Plaza." Note this location. Google map search "Fire Station 301 DCA ARFF". Zoom out to look for nearby metro stations. 8. The closest station is Ronald Reagan Washington National Airport. 9. Google map search "Cleveland Elementary School". 10. The closest metro station to Cleveland Elementry School is on the second result: 2022 System Map. https://www.ww http://wwwwwwa.accc.mmm
Univ Station. 11. Google search "DC Metro Station Map". 12. Clicked
System-Map.poee 13. Locate L'Enfant Plaza station. is the transfer station for all color lines. 14. Locate Shaw-Howard Univ stations stops above L'Enfant Plaza station. 15. Locate Ronald Reagan National Airport station the blue/yellow line. 16. Recall the current location: Shaw- Howard Univ station's 's yellow/green line. 17. Since the question says no line changes, we deduce the line must be one that Howard Univ and Ronald Reagan National Airport stations have common: yellow line. 18. Begin at Shaw-Howard Univ station and follow the yellow line. 19, Count the number of stops until reaches Ronald Reagan National Airport station. 20. Final answer: 8.
27

---

Subtask Ambiguity Error Analysis
The subtask asks how many subway stations there are between the stations, but the question actually requires calculating how many subway stations you need to pass to reach your destination, which is different.
Analysis:
Agent Content:
{"subtask": "Calculate how many metro stations are between the station nearest to Cleveland Elementary School and the station nearest to Fire Station 301 DCA ARFF, without changing lines. Provide the final answer as numerical integer value.".
"result": "Calculating Metro Stations Between Shaw-Howard University and National Airport Based on the information provided in the dependency results and my analysis of the Washington DC Metro system, I need to calculate how many metro stations are between the station nearest Cleveland Elementary School and the station nearest to Fire Station 301 DCA ARFF, without changing lines. Identifying the Stations From the dependency results:- The station nearest to Cleveland Flementary School ** **Shha-Hooaa
University** on the Green Line The station nearest to Fire Station 301 DCA ARFF is **Ronald Reagan Washington National Airport** (also called "National Airport") on the Blue and Yellow Lines Determining the Connecting Licorccgg to the dependency results, the Yellow Line connects both stations This means a person can travel between these two stations without changing lines by using the Yellow Line, Calculating the Number of StationsI analyzed the Yellow Line stations from north to south and found:- Shaw-Howard University is station 9 on the Yellow Line- National Airport is station 17 on the Yellow Line stations between them (excluding the starting station) are: 1. Mount Vernon Square 2. Gallery Place 3. Archives 4. L L'Enfant Plaza 5, Pentagon 6. Pentagon City 7. Crystal City Therefore, there are **7** metro stations between Shaw-Howard University and National Airport on the Yellow Line without changing lines. Final Answer The number of metro stations between the station nearest Cleveland Elementary School and the station nearest to Fire Station 301 DCA ARFF, without changing lines, is **7**""
Tool Selection Error: Error Case Level3 #8
Question: was trying to remember how well the Cheater Beater performed in comparison to the Cheater when James tested it on his channel. know that the Cheater still outperformed the Cheater Beater in terms of CFM. Could you please look that up for me, and report the CFM of both the Cheater and the Cheater Beater? I'm not sure if he made any changes to his testing, but this was back in season 4, so just report the value from that season. Please format your response like this: CFM number for Cheater, CFM number for Cheater beater.
Answer: 101.376, 84.348
Annotation steps: 1. Using web browser, navigate to a search engine and conduct a search: "James Cheater Cheater Beater CFM Season 4" 2. Finding no relevant result, navigate to a search engine and conduct another search: "Cheater Beater Season 4" Navigate to the first search result,
4. Evaluate the YouTube page, noting that the video description identifies the video content comparing the performance of computer fans to fan referred to as the "cheater" 5. Follow the link to the YouTTube channel Major Hardware, https://www, www.yyytuuu.coo./oo MajorHardware 6. Navigate to the About tab link, https://www.j www.yyyuuuubcoooo @ MajorHarrdaare/bbouu 7. Evaluate the content, noting that the page identifies the operator of the channel as James 8. Navigate to a search engine and conduct a search, "James Major Hardware Cheater Beater" 9. Navigate to the first result, identical to the result from step 3 above, http:://www.youuttee www.:yyuuuubbooooooee www.wwwwo/wwwtccceeeccoo. http://wwcc/22233PPZ2B
http://www.youuubbcommm wot//watcch/s22gg330P22222
10. Search the page for CFM, finding result 11. Load the video content and review it 12. Note onsereen text element identifying a fan as "CALL SIGN: CHEATER BEATER" at timestamp 224 13. Note an onscreen table identifying the performance of various fans tested during season four, at timestamp 485 14. Evaluate the table content, identifying an entry for a fan named "Cheater" and a fan named "Cheater Beater" 15. Evaluate the table content, identifying that the data for both fans were recorded in season 4, S4E1 for Cheater, S4E6 for Cheater Beater 16. Record the data from the CFM column for the two fans, "Cheater: 101.376", and "Cheater Beater: 84.348" 17. Report the correct response to my user: "Cheater: 101.376 Cheater Beater: 84.348".
28

---

Tool Selection Error Error Analysis
Analysis: The agent traverses season4's videos, but only stops at searching and does not open the video page for searching and processing.
Agent Content:
{ (rroee: "assistant", "tool_co "tool_calls": [ (wq "type": "function",
"function": { ("name": wee__search",
"arguments": "("question": "Major Hardware Fan Showdown Season Episode Cheater Beater")) ) }.
{ "rolee: "assistant", "tool_calls"; [{
"type": "function",
"function": { "name": "web_ search".
"arguments": """qeestion": "Major Hardware Fan Showdown Season 4 Episode 7 Cheater Beater")" } 1.
{ ("role:: "assistant", "tool calls": [{
"type": "function",
"function": { ("name": "web search",
"arguments". "{" ("qqeestion": "Major Hardware Fan Showdown Season 4 Episode 8 Cheater Beater")" "aater"""}] }}}
Web Toolkit Failure: Error Case Levell #38
Question:
On Cornell Law School website's legal information institute, under the fifth section of federal rules alphabetically, what word was deleted in the last amendment to the first rule the article that has Witnessesin the most titles as of 2021?
Answer: inference
Annotation steps: 1. Searched "Cornell Law School legal information institute" on Google. 2.
Opened https://www.latt www.wwwwel....////////// 3. Clicked Get The Law Federal Rules ; Federal Rules of Evidence (fourth section down). 4. Found the article that has "witnesses" in the most titles (VII). 5. Opened the first rule (701). 6. Scrolled to the last amendment as of 2021 (2011 amendment). 7. Found the word that was deleted (inference).
Web Toolkit Failure Error Analysis
The question asked count the titles containing the exact word "witnesses". As result, the browser also counted the titles containing the singular "witness" (such as "Who May Impeach a Witness").
Analysis:
Agent Content:
{ (rrole: "assistant",
"tool_calls": [{ "type": "function".
"function": "nname": "browse url".
"arguments": ""* "tstart url": https://www.law.comme wooomell.cd//uule/sf//aaiii vi", "task prompt": "List all the rule titles in Article VI of the Federal Rules of Evidence count how many contain the word ""]tnesses""""]]]
{ ("role": "tool".
"content": "The task was to list all the rule titles in Article VI of the Federal Rules of Evidence and count how many contain the word "witnesses." " Here's the final answer:The following rule titles contain the word "witnesses": 1. Rule 605. Judges Competency as a Witness 2. Rule 606. Jurors Competency as a Witness 3. Rule 607. Who May Impeach Witness Rule 608. Witnesss Character for Truthfulness or Untruthfulness 5. Rule 611. Mode and Order of Examining
Witnesses and Presenting Evidence 6. Rule 612. Writing Used Refresh Witness 7. Rule 613. Witnesss Prior Statement 8. Rule 614, Courts Calling Examining a Witness 9. Rule 615. Excluding Witnesses. Total count: 9 rule titles contain the word "witnesses."".
29

---

Document Toolkit Failure: Error Case Level2 #44
Question: The work referenced in footnote 397 of Federico Lauria's 's 2014 dissertation is also the source for the titles of two paintings in the Smithsonian American Art Museum's collection, as of August 2023. What is the absolute difference between the chapter numbers of the chapters that the titles of these two paintings quote?
Answer: 8
Annotation steps: 1. Use search engine to search for "Federico Lauria's 's 2014 dissertation". 2.
Open the result from philarchive.cm orr and open the PDF file for the full paper. 3. Search for footnote
397 to find that the referenced work is Thomas Hobbes's s "Leviathan", 4. Use search engine search for "Smithsonian American Art Museum collection search".. 5. Go to the museum's s search webpage. 6. Enter "Hobbes Leviathan" into the search box and submit the search. 7. Open the two results, one by Jan Stussy ("Λ free man...") and one by Leon Karp ("Hereby it manifest..."). "). 8. Verify from the full titles of these works that the titles are quotes from "Leviathan", 9. Use search engine to search for "Thomas Hobbes Leviathan full text". 10. Open any result that contains the full text, like the Project Gutenberg version. 11. Search the text for the titles of each painting, using different substrings from the titles as needed to account for variations in spelling and punctuation. 12. Find that the "A free man..." quote is from Chapter XXI (21) and that the "Hereby it is manifest..." quote is from Chapter XIII (13). 13. Calculate the absolute difference of the chapter numbers: 21 13 = 8.
Document Toolkit Failure Error Analysis
Analysis: The PDF address of the paper has been determined, but the Page limit exceeded.
Agent Content:
{ ("role": "assistant", "tool calls": [ ("function": { ("name": "extract document content", "arguments"" "[ ""dcumenttppath "https:/phiilarchiiee.om www.:/aa.iii....ooo.../oooo Attp:/iie...rgggo......t/....... OUU.UUUOO Attp% heppiilpapees.og22aaaaaieeeeee
"query": "footnote 397 Moby
Dick chapter titles" }},
{ "rrole: "tool", "content": "(True, 'Error while processing document: Page limit exceeded")))
Multimodal Toolkit Failure: Error Case Level3 #17
Question: Which of the fruits shown in the 2008 painting "Embroidery from Uzbekistan" were served as part of the October 1949 breakfast menu for the ocean liner that was later used as a floating prop for the film "The Last Voyage"? Give the items comma-separated list, ordering them in clockwise order based on their arrangement in the painting starting from the 12 o' clock position. Use the plural form of each fruit.
Answer: pears, bananas
Use search engine to search for "2008 painting Embroidery from Uzbekistan", 2. Open the top result, a link to the painting's page on the Dayton Art Institute website, and verify that the painting has the specified title and year. 3. Identify the fruits in the painting as watermelon, pear, lemon, and banana, which can be verified by either watching the video on the page or reading its linked transeript. Use search engine to search for "ocean liner floating prop The Last Voyage". Note from the results that this ocean liner was the SS 00cele de France. 6. Use search engine search for "October 1949 breakfast menuSS Ö0cele de France". Go the result that shows the vintage SS OOcel de France breakfast menu for October 1949. 8. Search the menu for each of the four fruits from the painting, finding "Pear" and "Bananas" but no matches for "lemon" the pears come before the bananas in clockwise order starting from the 12 o clock position. 10. Format the final answer as specified using the correct ordering: pears, bananas.
Annotation steps:
"watermelon". 9. Check the positions of the fruits in the painting find that
30

---

Multimodal Toolkit Failure Error Analysis
Analysis: ImaggAnallyiiSoooliii recognizes that the banana is placed at 12 o' o'clock and the pear placed at 9 o' clockk but their actual positions are 11 o' cloock and 7 o' ''clock respectively.
Agent Content:
{ (subbtask": "Create the final answer by listing the matching fruits in a comma-separated list, ordered clockwise from the 12 o' o'clock position as they appear in the painting, using the plural form of each fruit.",
"result": 'I'll solve this task by creating final answer that lists the matching fruits in a comma- separated list, ordered clockwise from the 12 o' o'clock position as they appear in the painting, using the plural form of each fruit. Based on the dependency results, I need to: 1. Identify which fruits from the painting "Embroidery from Uzbekistan" were also served on the SS le de France's ' October 1949 breakfast menu 2. List these matching fruits in clockwise order starting from the 12 o' clock position 3. Use the plural form of each fruit From the comparison analysis in the dependency results, can see that the fruits that appear in both the painting and the breakfast menu are: - Banamas Pears Now, I need to arrange these in clockwise order starting from the 12 o' clook position as they appear in the painting. According to the dependency results, the clockwise order of fruits in the painting starting from 12 o clock is: 1. Bananas (12 o o'clock) 2. Watermelon (3 o' o'clock) 3. Pears (6 o' c'look) 4. Lemons (9 o' d'clock) Since only bananas and pears match with the breakfast menu, and they are already in plural form, the final answer is: bananas, pears This is the comma- sonmaa-seaaated list of matching fruits in clockwise order from the 12 o' c'oook position as they appear in the painting, using the plural form of each fruit.") }
Response Format: Error Case Level2 #85
Question: At the two two-minute mark in the YouTube video uploaded by the channel GameGrumps on May 14, 2017 as part of their playthrough of the game Mario Kart 8 Deluxe, the show hosts are competing one of the games racetracks. What was the world record time for that track in the games 150cc mode as of June 7, 2023? Express your answer in minutes and seconds, rounding the seconds the nearest hundredth, e.g. 1:01.001.
Answer: 1:41.614
Annotation steps: 1. Search the web for "gamegrumps mario kart deluxe may 14 2017". 2.
Click on the YouTube video result. 3. Navigate two minutes into the video. 4. Scroll further back until see the name of the racecourse, Yoshi Cireuit. 5. Search the web for "mario kart 8 deluxe yoshi circuit world record 150cc". 6. Scroll down until find a reliable world record listing site. 7. Navigate through the site until I find the record that meets the specified criteria. 8. Read to confirm that it applies the question's specified date.
the date the record
Response Format Error Analysis
Analysis: This question requires rounding the seconds to the nearest hundred, such 1:01.001, but the agent finally gives the result of **1:41.61**
Agent Content:
{ ("role": "assistant".
"content": "I need to analyze the results from all subtasks to determine the final answer to the original question Lookiig at the results: 1. The first subtask identified the correct GameGrumps video from May 14, 2017: "Mario Kart 8 Deluxe: The Grand Prix PART 7" https///www.outubbe.oo wwwpuuubb..oo//wtttlllaaaaaaaaee hwtp:aat//=vvvavvvkkW..
2. The second subtask determined that the two-minute mark of this video, the hosts racing ** Circuit* 3. The third subtask found that the world record time for Yoshi Circuit in 150cc mode as of June 7, 2023 was #* * - 4. The fourth subtask formatted this time ** *#133..96** (rounding the nearest hundredth). 5. Howevr,,, the verification task found a different time: **1:41.614* from an archived version of the Mario Kart 8 Deluxe World Records website from May 29, 2023, There's a discrepancy between the two reported world record times. The verification task provides more specific details, including: The record holder: "Army" from France Date March 10, 2023
31

---

Character/kart setup used: Birdo (Black) with Mr. Scooty kart, Azure Roller tires, and Paper Glider The verification task also specifically mentions "GCN Yoshi Circuit," which is the full name of the track (GCN indicating it originated in GameCube's Mario Kart: Double Dise).Given that the verification task provides more detailed information and cites specific archive date (May 29, 2023) that is closer to our target date (June 7, 2023), I will consider this the more reliable result. Therefore, the final answer the original question is: 1:41.61(1 minute, 41.61 seconds) This is the world record time for Yoshi Circuit in Mario Kart 8 Deluxe's 150cc mode of June 7, 2023, expressed in minutes and seconds with the seconds rounded to the nearest hundredth.")
Reasoning Error: Error Case Level3 #25
Question: NASA's Astronomy Picture of the Day on 2006 January 21, two astronauts are visible, with one appearing much smaller than the other. As of August 2023, out of the astronauts in the NASA Astronaut Group that the smaller astronaut was a member of, which one spent the least time in space, and how many minutes did he spend in space, rounded to the nearest minute? Exelude any astronauts who did not spend any time space. Give the last name of the astronaut, separated from the number of minutes by a semicolon.
Answer: White; 5876
Annotation steps: 1.Use search engine to search for "NASA's Astronomy Picture of the Day 2006 January 21". 2. Open the link to the image. 3. Read the explanation to find that the image is of astronaut Charles "Pete" Conrad reflected in the helmet of astronaut Alan Bean. 4. Observe that the smaller astronaut in the image is the one reflected in the other's ' helmet, so the smaller astronaut must be Charles "Pete" Conrad. 5. Go the Wikipedia page for Charles "Pete" Conrad. 6. Search for "Astronaut Group" to find that Conrad was member of NASA Astronaut Group 2. 7. Open the Wikipedia pages for each member of NASA Astronaut Group 2. For those who are not deceased, go to View history and select the latest version of their Wikipedia page of August 2023. 9. Compare the times listed in the infobox of each astronaut's Wikipedia page under "Time in space", observing that Ed White has the least time in space with 4d 01h 56m, but also that Elliott See does not have a listed "Time in space". 10. Read through Elliott See's Wikipedia article to find that he died in an accident before his first space flight, so he should be excluded, making Ed White's 4d Olh 56m the least amount of time in space. 11. Convert 4d 0lh 56m to minutes: 4d 24h/d 60m/h + 1h 60m/h 56m 5,876m. 12. Format the final answer specified: White; 5,876.
Reasoning Error Error Analysis
Analysis: The accumulated time in space for NASA's "New Nine" had been correctly captured, but the people who had not been to space were incorrectly taken into account.
Agent Content:
{ ("subtask": "Research the space time duration for each astronaut in this group. Use web search find reliable sources about each astronaut's missions and total time spent in space. Create comprehensive list with their names and space time durations.".
"result": "Space Time Durations for NASA Astronaut Group 2 Members"
1. **Neil Armstrong** Total time in space: 8 days, 14 hours, 12 minutes Missions: Gemini 8 (1966), Apollo 11 (1969)
2. **Frank Borman Total time in space: 19 days, 21 hours, 35 minutes Missions: Gemini 7
(1965), Apollo 8 (1968)
3. ****Carles Pete Conrad** Total time in space: 49 days, 3 hours, 38 minutes Missions: Gemini
5 (1965), Gemini 11 (1966), Apollo 12 (1969), Skylab 2 (1973)
4. **Jim Lovell** Total time in space: 29 days. 19 hours, 5 minutes (715 hours, 5 minutes)
Missions: Gemini 7 (1965), Gemini 12 (1966), Apollo 8 (1968), Apollo 13 (1970)
5. Jamess M*Divit Total time in space: 14 days, 2 hours, 56 minutes Missions: Gemini 4
(1965), Apollo 9 (1969)
6. **Elliot *s Total time in space: 0 minutes Note: Selected but never flew; died before
Gemini 9 mission.
32

---

7. **Thomas P. Stafford** Total time in space: 21 days, hours, 43 minutes (507 hours, 43 minutes) Missions: Gemini 6A (1965), Gemini 9A (1966), Apollo 10 (1969), Apollo-Soyuz Test Project (1975)
8. **Ed White** Total time in space: 4 days, hour, 56 minutes Missions: Gemini 4 (1965); first
American spacewalk; died in Apollo fire.
9. **John Young** Total time in space: 34 days, 19 hours, 39 minutes Missions: Gemini 3
(1965), Gemini 10 (1966), Apollo 10 (1969), Apollo 16 (1972), STS-1 (1981), STS-9 (1983) }
Language/Question Ambiguity: Error Case Level2 #73
Question: I'm curious about how much information is available for popular video games before their release. Find the Wikipedia page for the 2019 game that won the British Academy Games Awards. How many revisions did that page have before the month listed as the game's 's release date on that Wikipedia page (as of the most recent entry from 2022)?
Answer: 60
Annotation steps: 1. Search the web for British Academy Video Games Award for Best Game 2019. 2. Find the answer, Outer Wilds. 3. Find the Wikipedia page for Outer Wilds. Go to the last revision from 2022. 5. Note the release date, May 29, 2019. 6. View the page history. 7. Count how many edits were made to the page before May 2019. 8. Arrive at the answer, 60.
Language/Question Ambiguity Error Analysis
Analysis: There is an ambiguity in interpreting the phrase 2019 game that won the British Academy Games Awards. One interpretation refers the game that won the BAFTA Best Game award the 2019 ceremony, regardless of its actual release year. The other interpretation requires the game have been both released in 2019 and won the BAFTA Game Award. This difference leads to inconsistent results, the agent prefers the former (based solely on the award year), while the ground truth requires both conditions (release year and award) to be met.
Agent Content:
{" ("subtask": "Search for the 2019 game that won the British Academy Games Awards (BAFTA) and identify the specific game.".
"result": "2019 BAFTA Games Awards Winner God of War Based the information from the 15tth British Academy Games Awards held on April 4, 2019, **God of War** (2018) was the winner of the Best Game award. Game Details ** Developer Santa Monica Studio *ubblisher Sony Interactive Entertainment Europe Platffrrm** **; PlayStation 4 - #*leease * April 2018 Awards Won (5 total) Best Game 2. Audio Achievement 3. Music 4. Narrative 5. Performer (Jeremy Davies for his role as The Stranger) Additional Information God of War received 10 nominations total, making the most nominated game the ceremony The 15th BAFTA Games Awards honored games released in 2018, though the ceremony was held in 2019")
33

---

Hallucination: Error Case Level2 #79
Question: If this whole pint is made up of ice cream, how many percent above or below the US federal standards for butterfat content is when using the standards as reported by Wikipedia 2020? Answer as or number rounded to one decimal place.
Answer: +4.6
Annotation steps: 1. Open the image. 2. Search "butterfat wikipedia" on Google search. Open the Butterfat Wikipedia page. 4. Click "View history" on the page. 5. Scroll down to the end of 2020 and click the last 2020 version of the page. 6. Check the ice cream requirement for fat content (10%). 7. Click "View history" on the page. 8. Scroll down to the beginning of 2020 and click the last 2019 version of the page. Check the ice cream requirement for fat content ensure it's the same (10%). 10. Calculate the fat percentage of the pint of ice cream from the image of the nutrition panel (21g fat per serving 144g cream per serving 14.6%). 11. Calculate the difference from the standard (14.6% 10% 4.6%)
Hallucination Error Analysis
Analysis: The agent subjectively assumes that 85% of the total fat is milk fat to estimate the actual milk fat content, but the question does not provide additional information.
Agent Content:
{ (suubtask": "Determine the actual butterfat content of the ice cream shown in the image described the task.",
"result": "Based my analysis, I have determined the actual butterfat content of the ice cream shown in the image. Methodology 1. First, I examined the nutrition facts on the ice cream container, which shows: Serving size: 2/3 cup (144g) Total fat per serving: 21g 2. calculated the total fat percentage in the ice cream: (21g fat 144g serving) 100 14.6% total fat 3. I analyzed the ingredients list to determine what portion of this fat butterfat: Cream the first ingredient (highest proportion by weight)- Other dairy fat sources include butter and butteroil Non-dairy fat sources include coconut oil and soy oil Based on the ingredients list with cream the first ingredient and the presence of butter and butteroil, I estimated that approximately 85 % of the total fat is butterfat, which is a standard industry estimate for premium ice cream with these ingredients. Result The estimated butterfat content of this ice cream is 12.4%. This was calculated as: 14.6% (total fat) 0.85 (estimated butterfat proportion) = 12.4% To account for uncertainty, I also calculated possible range: Lower estimate: 11 11.7% (assuming 80% of fat is butterfat) Higher estimate: 13.1 (assuming 90% of fat is butterfat) The final butterfat content 12.4% my best estimate based on the available information from the nutrition facts and ingredients list."
34

---

Context Exceed: Error Case Level3 #5
Question:
What animals that were mentioned in both Ilias Lagkouvardos's and Olga Tapia's s papers on the alvei species of the genus named for Copenhagen outside the bibliographies were also present in the 2021 article cited on the alvei species" Wikipedia page about multicenter, randomized, double-blind study?
Answer: mice
Annotation steps: 1. Searched "alvei copenhagen" on Google. 2. Opened
(bbctrimma) 3. Searched "Ilias Lagk-
ouvardos hafnia alvei" on Google. 4. Opened https:/www..mdpi.com//2
2607/1 2007/11////3/yppeccccek & hafnia alvei" on Google. 7. Opened animals mentioned in the first paper. 9. Searched each animal from the first paper in the second paper. 10. Noted the animals mentioned in both outside the bibliographies. 11. Went back to the Wikipedia article. 12. Opened the link the references to "'The Probiotic Strain H. alvei HA4597@ Improves Weight Loss in Overweight Subjects under Moderate Hypocalorie Diet: A Sroof-of-Concepp Multicenter Randomized, Double-Be Double-Blind Placebo-Controlles Study". 13. Opened the PDF. 14. Found the animals shared by all three papers.
Opened new tab. Searched "Olga Tapia www..em..ciiimmii.......
8. Found all
Context Exceed Error Analysis
Analysis:
The content of Olga Tapiaresearch has been extracted correctly, but the mice mentioned in the text have not been extracted from the content successfully.
Agent Content:
{ ("role": "assistant", "tool calls": {{
"type": "function""
"function":
"name": Eextraet_document.comm
"arguments": "I """document_patt""
"query":
"Olga Tapia research on Hafnia alvei with mentions of animals")"
}
35

---

Limited Coding Capability: Error Case Levell #36
Bob was invited to participate in a game show, and he advanced to the final round. The final round offered Bob the chance to win a large sum by playing game against the host. The host has 30 shiny prop coins, each of which is worth $1,000 if Bob manages to win them by playing the game. The host hides the coins in three different prize boxes and then shuffles their order. The only rule restricting the host's s coin placement is that one box must contain at least 2 coins, and one box must contain 6 more coins than another box. In order to play, Bob must submit three guesses, one guess for the number of coins in each box. The box is then opened and the number of coins is revealed. If Bob's guess is number greater than the number of coins in the box, Bob earns no coins. If Bob guesses a number equal to or less than the number of coins in the box, Bob wins a number of coins equal to his guess. If Bob plays uses the optimal strategy, what's the minimum amount of money he can win from the game?
Question:
Answer: 16000
Annotation steps: Step 1: Evaluate the problem statement provided by my user, storing the relevant information: 30 coins with a value of $1,000 distributed between 3 boxes. Each box must contain at least 2 coins. One box must contain 6 more coins than another. Step 2: Evaluate the base distribution: 2-8-20, noting that two boxes must contain at least 8 coins. Step 3: Evaluate the most even allowable distribution: 8,8,14, noting that two boxes must contain least 8 coins. Step 4: Evaluate a case where Bob guesses 8 for each box in the outlier distributions. Step 5: For the worst case 2-8-20 distribution, Bob wins 0+8+8= 16 coins. Step 6: For the 8-8- 14 distribution, Bob wins 8+8+8 = 24 coins. Step 7: Convert the worst-case coin count to prize value, 16*$1,000 $11,0000 Step 8: Report the correct answer my user: "$16,000"
Limited Coding Capability Error Analysis
Analysis: The agent ignored the hard constraint that each box must contain at least coins when writing the code, and mistakenly considered the distribution of O and I coins (such as (0,6.24), (1,7,22), etc.) as legal.
Agent Content:
{" ("subtask": "Implement solution using code systematically explore all valid coin distributions, calculate Bob's optimal guesses for each distribution, and determine the minimum guaranteed winnings.".
"result": "Optimal Strategy for the Game Show Problem I've implemented a solution to systemat- ically explore all valid coin distributions, calculate Bob's optimal guesses for each distribution, and determine the minimum guaranteed winnings. Let me explain my approach and findings. Solution Approach 1. ** **enerate Valid Distributions created function to generate all valid distributions of 30 coins among 3 boxes that satisfy the constraints: Total of 30 coins One box must contain at least coins- One box must contain more coins than another box 2. **Calculate Winnings**, : For each distribution and guessing strategy, I calculated the maximum possible winnings by trying all permutations of the guesses (since Bob can choose which guess to apply to which box). 3. **FFnnd Optimal Strategy** I implemented systematic search to find the strategy that maximizes the minimum guaranteed winnings across all possible distributions. Key Findings 1. ** Valid Distributions** The code identified 12 valid distributions of 30 coins that satisfy all constraints. 2. **Optimal Strategy The optimal strategy for Bob is to guess #*10, 6, 12]** for the three boxes. 3. **Minimum Guaranteed Winnings# With this strategy, Bob is guaranteed to win at least **$18,000** regardless of how the host distributes and shuffles the coins. 4. **Worst Case Scenario** *+; The worst-case distribution for Bob is **(0, 6, 24)**, where one box has coins, one has 6 coins, and one has 24 coins. 5. ** **Veriiication **; I verified that with the optimal strategy against the worst-case distribution, Bob indeed wins exactly $18,000. Comparison with Other Strategies I also tested several other strategies mentioned in the dependency results: Strategy (6, 6, 6): Minimum guaranteed 2,0000 Strategy (2, 6, 10): Minimum guaranteed $16,000- Strategy (1, 5, 12): Minimum guaranteed $1 $17_000 Strategy (7, 7, 7): Minimum guaranteed $7,000- Strategy (8, 8, 8): Minimum guaranteed $8,000 Strategy (6. 6, 7): Minimum guaranteed $ $13,000 - Strategy (6. 7, 7): Minimum guaranteed $13,000 This confirms that [0, 6, 12] is indeed the optimal strategy, guaranteeing $18 $18,000 in winnings. How the
36

---

Optimal Strategy Works With the [0, 6, 12] strategy: 1. Bob guesses 0 for the box with the fewest coins 2. Bob guesses 6 for the box with the middle number of coins 3. Bob guesses 12 for the box with the most coins This strategy works because: two boxes Even in the worst-case distribution (0, 6, 24), Bob match his guesses optimally win $18,000 No other strategy can guarantee more than $18,000 across all possible distributions Therefore, $18,000 is the minimum amount Bob win using the optimal strategy." }
ensures Bob always wins from at least
F
Case Studies
The real environment is often more complex than the pre-built simulation environment. The GAIA benchmark is built in real-life environments, which brings many challenges. Below we will introduce the challenges that agents may face in solving problems in real environments.
F.1 Differences in Information Sources
In real-world scenarios, multiple information sources may provide different answers the same query. This discrepancy creates a challenge where tasks might have several valid responses, each considered "correct" depending the reference source consulted.
To illustrate this phenomenon, consider the following task:
Task from GALA Benchmark
the authors (First M. Last) that worked on the paper "Pic Menus or Linear Menus, Which Is Better?" in 2015, what was the title of the first paper authored by the that had authored prior papers?
Question:
Answer: Mapping Human Oriented Information to Software Agents for Online Systems Usage.
Annotation steps: 1. Searched "Pie Menus or Linear Menus, Which Is Better?" on Google. 2. Opened "Pie Menus Linear Menus, Which Is Better?" on https://oda.oo wwtaaaaoooooooooooooooooo xmlui/handdle/1064223 162. 3. Clicked each author's 's name. 4. Noted the name that had no other papers listed. 5. Searched "Murano, Pietro" on Google. 6. Opened http://www.pictromuraam org/
7. Clicked "Publications". 8. Found the earliest paper he contributed to.
as shown in Figure 5, consulting the author's ' personal homepage yiclds "Mapping Human-Orienten Oriented Information Software Agents for Online Systems Usage" the answer. However when referencing Google Scholar, the first publication listed "A new software agent 'learning' algorithm" as shown in Figure 6. This inconsistency demonstrates how different authoritative sources lead to divergent yet equally defensible answers.
F.2 Outdated information
Outdated information is another common issue, especially when retrieving data via web search and information retrieval tools. The agent might encounter outdated or incorrect content that misguides its reasoning process, ultimately impacting the quality and effectiveness of task execution. For example, the original task may require visiting someone's 's YouTube channel or searching for some content in the channel. However, by now, the content may have been modified by the channel user, which will directly lead to error in the ground truth.
Here is an example from GALA benchmark:
37

---

Murano, Piatr (2008) ddttrocomomnic Va Non Anw.rooooemehhs Software Interiaee Foodback for Oniine Factual Delisry PDC 7th Intemational Conference on infomationn Visualiaatoo (IV 2000) Intemational Conference on Computer Visuilisatoo and Graphics Appllcations London, England, 18-18 Juyy 2003, (6)
Murano, Pictro (2002) dctrooomomtii: Vs Hon wwt.rooccccommiiicccccoo Solware Interiace Foodbark for Onling Systome, Viare -PEE 7th European Research
(ERCIAM)
Spedial
Paris(handdyy
October Published Leclure Notes Computer Science (G)
(2002)
Mapging
Proceedings
Inmeriionall Corference Informatio Technology interfsces, Ceviat, Croatis, 24-27 June 2010.
(2001)
Agent 'Laaming
People
Rooms, Codipits and Command Centres, LMSST UK IEE.
Utage
Conferance Human interaacss Contal Rooms, Cockpits Command Cantres, UMIST, UK, IEE.
INTERNATIONAL PRESENTATIONS
Richings, Tim. Pappuillis, George. Garwey, Davd Munen, Pietro. Shais, Wieles (2003) Aucuerticn Enduraments By Automeled Lasdaan and Tea Datacioon interational
on Human Computer intersction Symposium Human interiace (Japan)
Interational Conference Engineering Psychology Cogritive Ergonomics Intemational Conference Universal imtraction. 22 27 Juno 2003, Croio, Grocoo.
Effeciveness
Environmnns Engineering
73rd
Annual Conference Human intertaoe With Machinos, UCLANUK, Soptomber 2000.
Figure 5: Pietro Murano's personal homepage showing publication history (accessed on 2025-05)
Part IV-Novel Interaction Paradigmns-New Modaiíiess and Dielague Slyles-Anih Stytes-Anifroppomophhi Non-anttropomorphic Soltware
2003
Online Systams Usage
MMMrrru
Locture Notas Computer Science 2516, 339 340
Anthropomorphic vs. ren-anthhopomorphiic soflware interface feedback online systems
10
2002
usage
P Mureno
ERUM Workshop on User intortaces All, 339 349
Anthropomorphic Non-Anihrrpomorrhhic Soltware Interface Feedback for Online Systems
Usage, 7th European Research Consarlium for Informatica and Mathemalics (ERCIM)
Workshop
PMurroo
Parts (Chansily), France 25
Effecliveness mepping human-oriented information feedback from soltware interface Murono
16
2002
2012 Procoedings the internetonal Conterenca intormatin
Mapping human-oricnted information to soltware agonts for online systems usage I Murano
10
2001
Internaiional
A new softwaro agenttearningbagoriimm
16
2001
Mursna
Effactivenees of VRML Building Modele for Direction Finding
Manchester, Mackey
Figure 6: Pietro Murano's Google Scholar profile displaying a different publication order (accessed on 2025-05)
38

---

Task from GATA Benchmark
Eva Draconis has a personal website which can be accessed on her YouTube page. What is the meaning of the only symbol seen in the top banner that has a curved line that isn a circle portion of circle? Answer without punctuation.
Question:
Answer: War is not here this is a land of peace.
Annotation steps: 1. By googling Eva Draconis youtube, you can find her channel. 2. In her about section, she has written her website URL, orionmindproject.com. 3. Entering this website, you can series of symbols at the top, and the text "see what the symbols mean here" below it. 4. Reading through the entries, you can see a short description of some of the symbols. 5. The only symbol with curved line that isn't tt circle a portion of circle is the last one. 6. Note that the symbol supposedly means "War is not here, this is a land of peace."
According to the annotation, the agent should first go to Eva Draconis youtube channel, and check her personal website URL from about section. However, there's nothing in the channel now, and the personal webpage cannot be opened either.
F.3 Language Ambiguity
Language ambiguity occurs when the agent encounters unclear or imprecise language in the user query, leading to incorrect reasoning. For example, question might contain terms with multiple meanings or lack the necessary context to disambiguate the intended meaning.
For example, consider the following task:
Task from GAIA Benchmark
I'm curious about how much information is available for popular video games before their release. Find the Wikipedia page for the 2019 game that won the British Academy Games Awards. How many revisions did that page have before the month listed as the game's 's release date on that Wikipedia page (as of the most recent entry from 2022)?
Question:
Answer: 60
Annotation steps: 1. Search the web for British Academy Video Games Award for Best Game 2019 2. Find the answer, Outer Wilds 3. Find the Wikipedia page for Outer Wilds 4. Go the last revision from 2022. 5. Note the release date, May 29, 2019 6. View the page history 7. Count how many edits were made to the page before May 2019 8. Arrive the answer, 60
There are two possible interpretations of the question: (1) "2019 award winner": refers to the game that won the "Best Game" award at the 2019 BAFTA Games Awards (British Academy Games Awards), regardless of when it was actually released. (2) "2019 released and won the game": refers to the game that was released in 2019 and has won the BAFTA Game Award.
F,4 Network Instability & Permission Deny
Network instability is a critical factor impacting task evaluation. Since web search tools rely on internet connection, network instability can result in the agent being unable to access required resources or cause task execution to be interrupted.
Permission deny is also a critical factor, which refers to scenarios where the agent attempts to access resources that require (1) authentication or (2) human verification (such as downloading a paper from a journal that requires login access or human verification). In some cases, IP region restriction could also prevent agent from futher processing the task. This could cause the agent to fail in completing the task, even if it has found the correct URL containing the final answer.
39

---

G Broader Impacts
G.1 Data Pollution
Data pollution refers to situation where the agent might directly retrieve answers from source (such GATA's Huggingface) instead performing own reasoning. This issue may lead the agent "cheating" by relying pre-existing answers from the training data, rather than conducting the reasoning required for the task. simple way to solve this problem is restrict agent's access to certain sources (e.g., by filtering out urls containing "huggingface" to prevent the agent from retrieving answers directly from specific datasets).
G.2 Potential Dangerous Behaviors
While using tools, the agent might display dangerous behaviors, such accessing harmful content, pushing inaccurate information, generating inappropriate interactions, etc. If the agent can execute code/use the terminal locally, some operations of the agent may destroy some of the original local environment. These behaviors can pose a threat to user safety and system stability.
H
Statistical Significance Analysis
To validate the robustness of our empirical findings, we conducted Wilcoxon signed-rank tests to compare system-level performance across key baselines and planner configurations.
First, we observe that WORKFORCE significantly outperforms both the Single Agent and Role Playing baselines, with p-values of < 0.0001 and 0.0203 respectively. These results confirm that the observed improvements are statistically significant rather than due random variability.
Furthermore, we evaluate the impact of our proposed training method, Optimized Workforce Learning (OWL), by comparing OWL-trained Qwen2.5-32B 1..-suu--nntt against the base Swen2.5-32B-Instuee model. The Wilcoxon test yields a p-value of 0.0018, indicating significant improvement in task planning effectiveness after OWL-based reinforcement learning.
I
More Details on Analysis
L1 More Details Robustness
Number of Capabilities
Model
Mean
Std +
IQR 4
23
Single Agent
38.36
35.14
44.44
39.31
3.86
4.65
Role Playing
62.34
51.14
34.62
49.36
11.39
13.86
Workforce (Ours)
63.01
59.46
55.56
59.34
3.05
3.73
Table 9: Performance across different capability requirements, as well as overall robustness indicators including standard deviation and interquartile range (IQR)
The complete table of performance across different capability requirements is presented in Table 9.
1.2
Planner vs. Worker Training
Table 10 presents detailed comparison of different training configurations for our multi-agent system. The results clearly demonstrate that training only the planner (45,45% average accuracy) significantly outperforms training only the workers (31.51%), and even performs comparably to training both components together (46.68%) Interestingly. training only workers actually degrades performance below the baseline (36.36%), suggesting that worker specialization without proper task decomposition can be counterproductive. These findings strongly support our design decision to prioritize planner optimization, effective task planning and decomposition are more crucial than enhancing individual worker capabilities. The minimal performance gain from training both
40

---

Trainable Components
Variant
Level
Level 2
Level 3
Average
Planner
Worker
Ewvn-332-Innsrrud
X
49.05
33.72
19.23
36,36
Worker
43.39
30,23
11.54
31.51
W. Both
60.38
47.67
15.38
46.68
Planner (Ours)
X
60.38
45.34
15.38
45.45
Table 10: Performance comparison between different training configurations of planner and workers. Training the planner alone achieves nearly the same performance as training both components, while training only workers degrades performance.
components (only +1.23%) does not justify the substantial increase in computational costs, making planner-focused training the most efficient approach for improving overall system performance.
41

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As shown in Line 14-22 of the Abstract and Line 53-72 of the Introduction, we present our main claims and outline the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings. It is fine include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations explicitly in Appendix A. Despite the modular ar- chitecture and plug-and-play extensibility, our system's performance still depends on the availability of reliable domain-specific toolkits. In domains where such tools are absent underdeveloped, execution bottlenecks may occur. Moreover, reinforcement learning from
feedback remains time-consuming due to external latency, particularly in wol-in-thee-logy loop settings such as online search. These factors may hinder scalability in highly dynamic or under-resourced environments.
tool-in-
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "LLimitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
42

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our work is not a theoretical research work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 1. we have discussed the architecture in detail. Section 4. and Section 3.3, we have provided detailed description and analyses of experimental setups for all our investigations, including the inference and training hyperparameters and configuration. Furthermore, open-sourced our code at https: anonymous. 4open. science/ x / maanonymous-www and data at https: httt:::://ugggggaaaee co anonymous21 016 6 for replication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
43

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have open-sourced our code in
httpt:tt///// httg:::////noommuus.s.. 4open sc ience / r/ annonymonsss owls/, and data is at https: : htt.//ugggggggggeeeeeeo/ co ano aymous21016. And we have provided the corresponding instruction for reproducing the experimental results in the repo
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https: /ttsss///ippp//eee cc/
public /wp//guissse/eee/ /ooe/eoddeSsmm/ ss sssonoooicyy for more details.
While we encourage the release of code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nniss cc/
cc/ppcllcpcccccccccc /ppppullicc / des CodeSubm's ss onPoliion icy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: As shown in Section 4.1 and Section 3.3, have provided detailed descrip- tions and analyses of the experimental setups for all out investigations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
44

---

Justification:
We conduct Wilcoxon signed-ram -ank tests to assess the statistical significance of model performance differences. Detailed results are reported in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: shown in Section 3.3 and Section 4.4, we have provided detailed model compute resources used for model training.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type osf compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tt::///eeuur neurips wttpccccppccccccccccccccccccccct h/th csGuide nnss?
Answer: [Yes]
Justification:
We are convinced that we comply with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
45

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work introduces general-purpose multi-agent framework and training paradigm that significantly improves the performance and scalability of open-source Al assistants. The potential positive societal impacts include democratizing access to high- quality AI systems by reducing reliance on proprictary architectures, promoting transparency via full open-sourcing of models and code, and enabling more efficient deployment of assistive agents. However, as with any powerful AI system, there are risks of misuse such as generating misleading information, automation bias, or unauthorized tool use. We explicitly discuss such risks in Appendix G, including challenges related to data pollution, outdated information, and potentially harmful agent behaviors. As mitigation strategy, we have implemented safeguards in toolkits (e.g., content filtering and rate-limiting for web tools) and recommend restricted access for high-risk capabilities.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative upplications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
46

---

12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly cited relevant source materials and obtained necessary permissions for any third-party resources used this study. Specifically, the CAMEL framework is released under the Apache License 2.0 (https: /tpp::///ittt /ttp//gghhbb////// com / lig hta iime camel) 1), and we have cited the official repository and adhered to all license requirements. The 20wen22.5IInstruc model is provided under the Qwen Community License (https: /ans://huee Thuggingg face. co Qwen/ Qwe//wen22222 -ttp///weennn-nntttee IB-Instrcc,), and we have fully complied with its terms of and attribution guidelines.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assett are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papersw iirswiite th com dat aset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have written a detailed document to guide others on how to use our code at https://anonyynn 4open. www.sscieeeeeeeeeeeee /ct.ence////nnnnnnnuuuuueeeeeese
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
47

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We have provided detailed prompt examples of LLM usage in our algorithm in Appendix D.2.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (ht /tte:///eeuee /t///eurippp cocceeeeeeessss / 20255/ LLM) for what should or should not be described.
48


Reviews:
[{"review_id": "s6AH5NdbrD", "reviewer": "Reviewer_vpYJ", "summary": "This paper introduces a hierarchical multi-agent framework that decouples strategic planning from domain-specific execution to enable cross-domain transferability. The system comprises three core components: a domain-agnostic Planner for task decomposition, a Coordinator for subtask management, and specialized Workers with domain-specific capabilities. The authors also propose Optimized Workforce Learning (OWL), a training paradigm that focuses on optimizing the domain-agnostic planner through supervised fine-tuning followed by reinforcement learning. The approach is evaluated on the GAIA benchmark, achieving 69.70% accuracy and outperforming commercial systems like OpenAI's Deep Research by 2.34%.", "questions": "0. Why are you using JSON agent instead of Code agent?\n\n1. How does the system perform on domains that are significantly different from those in the training curriculum? Can you provide analysis on out-of-domain generalization?\n\n2. How does the system handle cases where multiple workers could potentially handle a subtask? What mechanisms prevent conflicts or redundant work?\n\n3. While you mention replanning mechanisms, how robust is the system to cascading failures where multiple workers fail sequentially?\n\n4. What is the computational overhead of the coordinator and communication mechanisms compared to direct single-agent approaches?\n\n5. How does performance degrade as you add more workers or increase task complexity? Are there theoretical or empirical limits to the approach?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n\n- Novel Modular Architecture: The separation of domain-agnostic planning from domain-specific execution is well-motivated and represents a significant architectural innovation. This design enables plug-and-play extensibility across domains without requiring complete system redesign.\n\n- Strong Empirical Results: The paper demonstrates impressive performance on GAIA, achieving state-of-the-art results among open-source frameworks (69.70%) and surpassing proprietary systems like OpenAI's Deep Research. The OWL training method shows substantial improvements (+16.37% for Qwen2.5-32B-Instruct).\n\n- Comprehensive Evaluation: The paper includes thorough ablation studies, error analysis, and performance breakdowns across different capability types. The test-time scaling analysis and robustness evaluation add valuable insights.\n\n- Reproducibility: The authors commit to releasing all code, models, and data, which supports open research.\n\n\n## Weaknesses\n\n- Tool Dependency: The system's performance is heavily dependent on the quality and availability of domain-specific tools. The paper doesn't adequately address how to handle domains with poor tooling.\n\n- Scalability Concerns: The paper doesn't thoroughly address how the approach scales with the number of workers or the complexity of coordination required for very large multi-agent systems.\n\nMissed References:\n\n[1] Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage.\n\n[2] From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions.", "was_revised": true, "final_justification": "My concerns have been addressed. I maintain the original score.", "rebuttal": "Thanks for your valuable comments. We will answer the following questions in detail one by one.\n\n**W1. The system's performance is heavily dependent on the quality and availability of domain-specific tools. The paper doesn't adequately address how to handle domains with poor tooling.**\n\nWe appreciate your thoughtful insights. We admit that tool design does require careful manual design. However, this paper's focus is mainly on modular **agent architecture**. One solution to this problem is to let the agent generate the tool itself, which we plan to explore in the future work. \n\nFurthermore, one emergent behavior of the current system is that, when faced with tasks that the current tool set cannot handle—for example, \"According to GitHub, when was Regression added to the oldest closed numpy.polynomial issue that has the Regression label in MM/DD/YY?\"—our system used the **code execution tool** to **write a function leveraging the GitHub API**, and correctly answered the question.\n\n**W2&Q6. Scalability Concerns: The paper doesn't thoroughly address how the approach scales with the number of workers or the complexity of coordination required for very large multi-agent systems. How does performance degrade as you add more workers or increase task complexity? Are there theoretical or empirical limits to the approach?**\n\nThanks for your insightful consideration. Existing figures show **robustness on multi‑capability tasks** (Figure. 4 (b)) and **positive test‑time scaling** (Figure 3(b)).\nBesides, We believe that the key here is whether the system can correctly identify which worker a subtask under the current task domain should be assigned to. In light of this, we conducted additional experiments on PMC-VQA[1] and PathVQA[2], two medical VQA benchmarks. We first use all default sub-agents: `reasoning_coding_agent`, `document_processing_agent`, and `web_agent`. Besides, to better handle image-related questions, we also include an `image_analysis_agent`. Here are the experimental results:\n\nPMC-VQA Results:\n| Model | Accuracy |\n|-------|----------|\n| Claude-3.7-Sonnet | 77% |\n| GPT-4O | 74% |\n\nPathVQA Results:\n| Model | F1 Score | Exact Match |\n|-------|----------|-------------|\n| Claude-3.7-Sonnet | 59.2 | 18.9 |\n| GPT-4O | 52.3 | 14.9 |\n\nWe observe that our multi-agents system typically only use `image_analysis_agent` and `reasoning_coding_agent` (only for output answer) in its planning, demonstrating planner's ability to adaptively select relevant domain-specific agents based on task requirements when increasing workers, showcasing its flexibility in handling domain-specific challenges.\n\nWe will add more analysis such as consistently scaling up worker numbers with accuracy/latency/cost curves in our final version of the paper. \n\n**Q1. Why are you using JSON agent instead of Code agent?**\n\nThanks for your valuable consideration. We will answer your questions from the following four points：\n\n1. While OWL is designed as a **general‑purpose agentic system**, executable‑code agents (e.g., CodeAct) — though strong on data/algorithm‑heavy subtasks — are **not universally superior** to JSON/function‑calling across diverse tasks.\n2. CodeAct’s reference setup spins up a Docker‑isolated Jupyter kernel per session—good for isolation, but it introduces **cold‑start latency** and **non‑trivial CPU/memory/storage overhead**, increases governance/ops burden, and makes outcomes sensitive to **environment drift** (runtime/deps/network).\n3. Meanwhile, the community’s dominant practice—and much instruction data for base models—centers on **structured JSON/tool calls**, so models are typically better calibrated to that protocol.\n4. Thanks to OWL’s **modular design**, a CodeAct‑style execution path can be added as an **optional Worker** without changing the architecture, allowing us to **benefit from both** when appropriate.\n\n**Q2. How does the system perform on domains that are significantly different from those in the training curriculum? Can you provide analysis on out-of-domain generalization?**\n\nThanks for your comments. Firstly, there's already a gap between our training curriculum and GAIA. For example, our training curriculum doesn't include training data related to **file processing** for Excel, Docx, and Python, whereas many GAIA use cases require the agent to perform file processing. Secondly, to verify the ability of generaliation, we conducted a supplementary experiment on **medical** domain, which is not covered in the training curriculum: We added an additional agent for medical scenarios based on the default worker set configuration, to test whether planner can handle medical tasks effectively. We used the dev set from the **MedMCQA**[3] benchmark (a question answering benchmark in the medical domain) and tested the planner before and after training (based on the Qwen2.5-32b-Instruct model). The experimental results showed an accuracy of **71% before training** and **75% after training**, demonstrating that the trained planner maintains **generalization capabilities in out-of-domain scenarios**.\n\n**Q3. How does the system handle cases where multiple workers could potentially handle a subtask? What mechanisms prevent conflicts or redundant work?**\n\nWhen multiple workers can handle a subtask, the **Coordinator** makes the unified decision about assigning the subtask. **Worker descriptions** (including capability boundaries) are crucial for the coordinator's decision-making (currently manually constructed). If a worker mistakenly assigns a subtask to a worker that lacks the required capabilities, a **test-time replanning** mechanism kicks in. The planner receives the reason for the downstream subtask failure and attempts to replan the subtasks to align them as closely as possible with the worker's capabilities.\n\n**Q4. While you mention replanning mechanisms, how robust is the system to cascading failures where multiple workers fail sequentially?**\n\nThanks for your insightful feedback. Workers perform self‑assessment and post structured failure reports (reasons and alternatives) to the shared channel, and then the planner re‑plans based on this feedback. This enables the system to identify issues earlier—rather than waiting until final execution to debug—thereby reducing cross‑worker failure propagation. In addition, Figure 3 shows that performance improves as the number of replanning iterations increases from 0 to 2—for example, the GPT‑4o workforce rises from **0.43** to **0.60**.\n\n**Q5. What is the computational overhead of the coordinator and communication mechanisms compared to direct single-agent approaches?**\n\nWhen starting a workforce to try to solve a task in GAIA, the total average cost of the coordinator is 3458.89 tokens. The coordinator consumes an average of 1033.53 tokens to assign a worker. Maintaining the task channel involves some communication, consuming an average of 734 tokens.\n\n**References**\n\n[1] Zhang, Xiaoman, et al. \"Pmc-vqa: Visual instruction tuning for medical visual question answering.\" arXiv preprint arXiv:2305.10415 (2023).\n\n[2] He, Xuehai, et al. \"Pathvqa: 30000+ questions for medical visual question answering.\" arXiv preprint arXiv:2003.10286 (2020).\n\n[3] Pal, Ankit, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. \"Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering.\" Conference on health, inference, and learning. PMLR, 2022."}, {"review_id": "6onhGwYSQk", "reviewer": "Reviewer_RWGZ", "summary": "This paper introduces WORKFORCE, a hierarchical multi-agent framework designed to address the domain-specific limitations of existing LLM-based systems. By decoupling strategic planning from domain-specific execution through a modular architecture (Planner, Coordinator, Worker Nodes), WORKFORCE enables cross-domain transferability with minimal retraining. The framework is validated on the GAIA benchmark, achieving state-of-the-art open-source performance and outperforming commercial systems like OpenAI’s Deep Research. Additionally, the Optimized Workforce Learning (OWL) training paradigm enhances generalization via SFT and reinforcement learning, boosting the Qwen2.5-32B model by 16.37% and demonstrating performance comparable to GPT-4o on complex tasks.", "questions": "+ The writing style resembles an engineering report rather than an academic paper, with somewhat simplistic descriptions of algorithmic process details.\n\n+ As shown in Fig. 4, the effectiveness of OWL is primarily constrained by tool capabilities, even more so than the planner. However, tool invocation requires meticulous manual authoring. To fully achieve automation, how can Worker nodes proactively modify and expand their capabilities? This is mentioned in line 107 of the paper, but specific details are absent.\n\n+ Line 102 states, \"Planner Agent analyzes incoming tasks and decomposes them into subtasks based on worker capability registry.\" Evidently, the Planner Agent relies on existing Workers for task decomposition. Thus, when Workers are modified, does the Planner need to be retrained? Can a fine-tuned Planner dynamically adapt to different Worker identities?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- Clear and excellent academic writing with well-designed figures  \n- The flexible and modular multi-agent architecture is crucial for swarm intelligence system applications  \n- Only requiring tuning of the planner reduces training overhead significantly  \n- State-of-the-art (SOTA) experimental results outperform current renowned frameworks  \n- Full open-source release of the code  \n\n\n**Weaknesses**\n\nSee `Questions`.", "was_revised": true, "final_justification": "In the rebuttal phase, the author addressed my concerns about the algorithm's scalability and generalization. So I increased my score accordingly.", "rebuttal": "We appreciate your valuable feedback, and address your concerns as follows:\n\n**Q1. The writing style resembles an engineering report rather than an academic paper, with somewhat simplistic descriptions of algorithmic process details.**\n\nThank you for your feedback. As this paper proposes both an agent framework and a training method, some content has been simplified due to page limitations. Additional details are provided in the appendix. In the final version, we will further refine this section to enhance its academic rigor, including adding formal definitions for the workforce and OWL components.\n\n**Q2. As shown in Fig. 4, the effectiveness of OWL is primarily constrained by tool capabilities, even more so than the planner. However, tool invocation requires meticulous manual authoring. To fully achieve automation, how can Worker nodes proactively modify and expand their capabilities? This is mentioned in line 107 of the paper, but specific details are absent.**\n\nWe sincerely appreciate your insightful comments. \n\n(1) Regarding tool capability expansion, while our current focus is on architectural design, we recognize the importance of automated tool modification. The workforce system can already extend its capabilities through dynamic tool generation – for instance, our agent system automatically created a tool using GitHub API to solve a specific task \"According to GitHub, when was Regression added to the oldest closed numpy.polynomial issue that has the Regression label in MM/DD/YY?\". \n\n(2) To further demonstrate the potential for automating worker design, we conducted an experiment in which a meta-agent powered by GPT-4o automatically generated worker designs. We evaluated this setup on 10 diverse tasks, using human-designed workers as the baseline for comparison. Quality was assessed using human ratings on a 1–5 scale in terms of prompt/toolkit. The results showed that the workers generated by the meta-agent achieved performance close to that of human-designed ones (average score: 4.3 ± 0.2 vs. 4.7 ± 0.1). We will include these results in the final version.\n\n**Q3. Line 102 states, \"Planner Agent analyzes incoming tasks and decomposes them into subtasks based on worker capability registry.\" Evidently, the Planner Agent relies on existing Workers for task decomposition. Thus, when Workers are modified, does the Planner need to be retrained? Can a fine-tuned Planner dynamically adapt to different Worker identities?**\n\nThanks for your insightful comments. To verify the ability of generaliation, we conducted a supplementary experiment on **medical** domain, which is not covered in the training curriculum: while retaining the GAIA worker configuration, we added an additional agent for medical scenarios. We used the dev set from the **MedMCQA** benchmark (a question answering benchmark in the medical domain) and tested the planner before and after training (based on the Qwen2.5-32b-Instruct model). The experimental results showed an accuracy of **71% before training** and **75% after training**, demonstrating that the trained planner maintains **generalization capabilities in out-of-domain scenarios**.\n\n**References**\n\n[1] Pal, Ankit, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. \"Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering.\" Conference on health, inference, and learning. PMLR, 2022."}, {"review_id": "ARejf8TtPx", "reviewer": "Reviewer_rqab", "summary": "The paper proposes Workforce, a multi-agent framework that is modular and scalable, featuring a domain-agnostic planner, a coordinator, and domain-specific worker nodes. The paper also introduces OWL, an optimized workforce learning, an efficient and effective multi-agent training paradigm. Both the proposed framework and the training paradigm yield impressive performance on the GAIA benchmark.", "questions": "- Why pass@3 sampling for GPT-4o while pass @1 for Claude-3.7-sonnet?\n\nAlso see seaknesses.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The performance is impressive.\n- The code is fully open-sourced.\n- The analysis provides insight for the community.\n- The presentation is clear.\n\nWeaknesses\n\n- As a modular framework, there is no ablation study on the contribution of each module. Providing this could make the framework design more convincing. It's now unclear why introducing a coordinator is necessary compared to letting the planner handle this role assignment. Figure 4(c) only discusses the training of the planner and the worker, also gives the impression that the coordinator is not necessary.\n- The design choice of using a shared task channel is not supported by experimental results. E.g., would it be even more helpful if the planner could access the context of the worker to better replan?\n- Though the centralized mechanism makes the management of context easier, it also limits the potential of the proposed multi-agent system. E.g., workers could not ask for clarification from the higher-level agents (planner here).\n- For the goal of generalist multi-agent assistance, it might be worth some discussion on how to select the worker agents, or even creating worker agents autonomously. It's not justified why the three worker agents in 3.2 are chosen for this goal. It seems more of a choice for the GAIA benchmark. Moreover, the generalizability of the trained planner needs to be verified with different worker agent selections if it's supposed to be utilized off-the-shelf by the community.\n- For baselines in 3.3, it seems confusing to have (iii) and (iv) as groups, while in Table 1, there are only two groups of (i) (ii).\n- It's worth noting that the idea of decoupling strategic planning from domain-specific execution is also explored extensively in the embodied agents literature, which should also be discussed in the paper. [1][2][3]\n\n[1] Building Cooperative Embodied Agents Modularly with Large Language Models. ICLR24\n\n[2] Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents. NeurIPS23\n\n[3] Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. ICML22", "was_revised": true, "final_justification": "I believe with the details and additional experiments added during rebuttal, the paper is worth sharing with the community at the conference. I keep my original recommendation of accept.", "rebuttal": "Thanks for your constructive review.\n\n**W1. As a modular framework, there is no ablation study on the contribution of each module. Providing this could make the framework design more convincing. It's now unclear why introducing a coordinator is necessary compared to letting the planner handle this role assignment. Figure 4(c) only discusses the training of the planner and the worker, also gives the impression that the coordinator is not necessary.**\n\nThanks for your valuable feedback. Regarding the coordinator, we have two main considerations in the design:\n- **Context growth and decision interference:** We decouple planning from assignment and enforce a shared task channel that surfaces only concise subtask outcomes, so the planner’s input scales with the number of subtasks rather than worker‑log length, preserving strategic reasoning and reliable replanning.\n- **Adaptation to changing worker pools:** The coordinator maintains a capability registry and live state, allowing worker/tool substitutions to be absorbed by registry updates without retraining the planner, thereby retaining a planner‑only policy that generalizes across domains.\n\nFurthermore, we conducted an additional ablation experiment to determine whether to use the coordinator for task assignment. The model uses gpt-4o. The settings of worker and tool are consistent with the second-to-last row in Table 1. The experimental results are the accuracy of GAIA pass@3. The experimental results are as follows:\n\n\n| Method | Level1 | Level2 | Level3 | Avg.|\n| -------- | -------- | -------- | -------- | -------- |\n| **Workforce** | 81.14 | 58.14 | 26.92 | 60.61 |\n| **Workforce wo. Coordinator** | 75.47 | 54.65  | 23.07 | 56.36 |\n\nThe experimental results show that not using the coordinator will slightly reduce the performance of the overall system, which confirms our experience.\n\n\n**W2. The design choice of using a shared task channel is not supported by experimental results. E.g., would it be even more helpful if the planner could access the context of the worker to better replan?**\n\nThanks for your valuable feedback.\n\n(1) Keeping information concise and focused is also critical for enhancing agent performance. One of the key motivations behind using a shared task channel is to strike a balance between retaining essential subtask outcomes and avoiding excessively long contexts. While giving the planner full access to each worker's task context during replanning might provide more comprehensive information, it often introduces a large amount of irrelevant content (e.g., long passages from multiple webpages), which can overwhelm the planner and impair its decision-making.\n\n(2) We conducted an experiment comparing our default setup with a variant where the full trajectory was placed into the shared task channel. The final performance in this setting dropped to 46.06, with most failures attributable to \"out of context\" issues caused by the noise and overload in the shared task history.\n\n\n\n**W3. Though the centralized mechanism makes the management of context easier, it also limits the potential of the proposed multi-agent system. E.g., workers could not ask for clarification from the higher-level agents (planner here).**\n\nThanks for your insights. Currently, workers do have a certain ability to provide feedback to the planner. Specifically, when a worker fails in its attempt to solve a downstream task, it generates failure information, including the cause of the failure and possible alternative solutions. This information is then incorporated into the planner's replanning context to assist the planner in replanning.\n\n**W4. For the goal of generalist multi-agent assistance, it might be worth some discussion on how to select the worker agents, or even creating worker agents autonomously. It's not justified why the three worker agents in 3.2 are chosen for this goal. It seems more of a choice for the GAIA benchmark. Moreover, the generalizability of the trained planner needs to be verified with different worker agent selections if it's supposed to be utilized off-the-shelf by the community.**\n\nThanks for your valuable insights.\n\n(1) Firstly, our worker design follows **two principles**: (1) The overlap in capabilities between different workers should be minimized, and (2) the boundaries of worker capabilities should be clearly defined to facilitate decision-making by the coordinator and planner. We designed our workers based on the sveral capabilities required for general-purpose tasks: web browsing, multimodal processing, file handling, code writing, and reasoning capabilities. Based on this, we designed three workers (Web Agent, Document Processing Agent, and Reasoning/Coding Agent). Of course, more workers can be added to complete other types of tasks, such as adding an terminal agent to handle OS-related tasks.\n\n(2) Secondly, to verify the generalization of the trained planner, we conducted a supplementary experiment: while retaining the GAIA worker configuration, we added an additional agent for medical scenarios. We used the dev dataset from the MedMCQA[1] dataset and tested the planner before and after training (based on the Qwen2.5-32b-Instruct model). The experimental results showed an accuracy of 71% before training and 75% after training, demonstrating that the trained planner maintains generalization capabilities in out-of-domain scenarios.\n\n(3) Regarding automated agent design, we carried out an experiment where a meta-agent, powered by GPT-4o, autonomously generated worker configurations. This setup was evaluated across 10 varied tasks, with manually designed workers serving as the benchmark. Human evaluators rated task performance on a 1–5 scale. The outcome demonstrated that the quality of meta-agent–generated workers was comparable to that of human-designed ones, achieving an average score of 4.3 ± 0.2 versus 4.7 ± 0.1. We plan to include the complete results in the final version of the paper.\n\n\n**W5. For baselines in 3.3, it seems confusing to have (iii) and (iv) as groups, while in Table 1, there are only two groups of (i) (ii).**\n\nWe appreciate your detailed insights. One common variant of agent systems is the toolkit-based design. The reason we separated (iii) and (iv) as distinct baselines is that these two were implemented by us with carefully aligned tool sets, allowing for a more controlled and fair comparison focused specifically on agent architecture design. We have clarified this point in the paper (Lines 145–146).\n\n**W6. It's worth noting that the idea of decoupling strategic planning from domain-specific execution is also explored extensively in the embodied agents literature, which should also be discussed in the paper.**\n\nThanks for your constructive suggestion. We will add more comparisons of OWL and embodied literatures in the related work section of our paper.\n\n**Q1. Why pass@3 sampling for GPT-4o while pass @1 for Claude-3.7-sonnet?**\n\nThank you for your question. (1) Since GAIA operates in real-world environments (e.g., retrieving information from the web), it is more susceptible to issues such as network errors; thus, we adopted pass\\@3 for greater robustness. (2) Due to the high cost of the Claude-3.7-SONNET API, we used pass\\@1 for evaluation to manage computational expenses.\n\n\n**References**\n\n[1] Pal, Ankit, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. \"Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering.\" Conference on health, inference, and learning. PMLR, 2022."}, {"review_id": "QKm9vqyfdn", "reviewer": "Reviewer_vyod", "summary": "This paper proposes a hierarchical method for multi-agent task execution using LLMs. The novelty of the work is that it uses a domain-agnostic planner and coordinator, which decomposes tasks and relays them to different agents, and domain-specific agent experts, which execute each part of the task. This enables to train only the experts on domain-specific data and maintain the same planner across domains and tasks. To train this system, the paper proposes OWL, which trains the domain agnostic planner via SFT on demonstratiosn from GPT-4o and RL on task success. Without any training, the approach obtains SOTA on the GAIA benchmark compared to other Open-Source frameworks, and is the first to outperform the closed-course Deep Research model. The training process also shows significant improvements, leading to 13% relative improvement with SFT and 46% relative improvement with DPO.", "questions": "In order of importance:\n- Clarify differences between results in Table 2 and Table 3. Particularly difference between  Table 3,row 3 different from those on Table 2, last row.\n- It would be great to test how the finetuned planner generalizes to unseen workers, and in particular whether there are still improvements wiht respect to the zero-shot version.\n- Why only training the high-level planner? Would there be gains from training the coordinator as well?\n- Clarify differences (besides the finetuning approach) with respect to works like Magnetic-One.\n- Results: if SFT is finetuned on GPT4o-mini, why is there such a significant gap between the first and second to last row in in Table 3.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Relevance: The paper addresses an important problem, which has gained important in the research community: performing complex reasoning tasks by orchestrating multiple agents. \n- Novelty: to my knowledge, this is the first work that learns a high level planner with task reward data, rather than training the actor agents. This seems like a powerful approach since there may be agents in the that cannot be finetuned (either because the provider does not allow it or because they are non-learned agents). This framework focuses on the high level planner which is agnostic to the agents and tasks. the finetuning approach obtains significant gains over the zero-shot approach.\n- Experimental details: Provide a detailed analysis on the pretraining datasets used to improve the high-level planner, motivating the selection of different datasets and describing an approach to filter low-quality data from the experts, with significant gains on SFT. It would be great to look at the contribution of each dataset in GAIA performance.\n- The proposed appraoch can be combined with different base models and obtains SOTA results on GAIA when compared to similar models. In the zero-shot setting, WorkForce with GPT-4o and Claude-3.7-Sonnet, outperforms all the other open source frameworks with similar base models by a wide margin, and are close to other closed-source frameworks. In the fine-tuning setting, the framework outperforms closed-source based models such as GPT-4o-mini and larger models. \n\nWeaknesses:\n- Clarity: The structure of the paper makes it somewhat hard to follow, there is a high overlap between preliminary and related work, and they could be moved as a single section. Similarly the results in section 3.3 and 4.4 should be comparable since the only difference seems to be on whether the Workforce approach is further pretrained or not. Making these in separate sections creates confusion on whether the evaluations (dataset metrics and baselines) are comparable or not.\n- Results: if SFT is finetuned on GPT4o-mini, why is there such a significant gap between the first and second to last row in in Table 3.\n- One of the main claims of the work is that the planner is agnostic to tasks and agents, and therefore can generalize to different types of agents. It would be great to test how overfit is the high-level planner to the specialized workers. What happens if you change the capacity of the workers or their specialization? Do you still get gains on the finetuned planner? \n- Related work: It would be great to clarify the novelty with respect to related work, particularly what is the difference between the orchestration in your work vs Magnetic-One? You have a special agent that is doing task decomposition and one that is doing task planning, which seems similar to the orchestrator in Magnetic-One.\n- Method and Clarity: The way the results are presented, I am confused about how table 3 compares to table 2. Are all results on Table 3 showing the workforce framework with different base planners? Why are the numbers for Claude-3.7-Sonnet in Table 3,row 3 different from those on Table 2, last row? Shouldn't it be the same method? It would be useful to see how the same exact planner model performs both when using the Workforce framework vs not. Ideally taking one of the frameworks from Table 2 and replacing them with the Qwen2.5-32B-Instruct base, this way it would be possible to see how much gain there is from the workforce framework vs the post-training.\n- While the approach is promising, it looks like one big limitation is in the necessity to define worker agents per task. The result of this is that, for new tasks, it is still necesary to write agents with domain-specific knowledge. It would be great if authors could comment on the details of how these workers were defined in the experimental section and how scalable this is to new tasks.", "was_revised": true, "final_justification": "The rebuttal clarified my remaining doubts, I am keeping my score as I believe this paper should be accepted", "rebuttal": "We appreciate your consideration of our work as an innovative solution to an important problem in the multi-agent field concerning complex reasoning tasks. Due to overlapping areas between some of the weaknesses and questions, we have merged the overlapping parts in our response to stay within the character limit.\n\n**W1. Clarity: The structure of the paper makes it somewhat hard to follow, there is a high overlap between preliminary and related work, and they could be moved as a single section. Similarly the results in section 3.3 and 4.4 should be comparable since the only difference seems to be on whether the Workforce approach is further pretrained or not. Making these in separate sections creates confusion on whether the evaluations (dataset metrics and baselines) are comparable or not.**\n\nThank you for the suggestion. We will revise the structure to clarify the comparability between sections and reduce redundancy between the preliminary and related work.\n\nThe reason we separated Sections 3.3 and 4.4 is that we aimed to highlight the individual contributions of the Workforce agent architecture and the OWL training method. While it's true that the two experimental tables could technically be merged—though at the cost of readability—the focus of each experiment differs in terms of conclusions, experimental settings, and implementation details. Keeping them separate allows us to clearly articulate the specific gains from architecture versus training improvements.\n\n\n**Q1.& W5. Clarify differences between results in Table 2 and Table 3. Particularly difference between Table 3,row 3 different from those on Table 2, last row.**\n\nThank you for your question and for pointing out the need for clarification. We believe your concern refers to the experimental settings related to Tables 1 and 3 (since tables 2 and 3 are quite different). As noted in the caption of Table 3, this is a controlled experiment where the workers consistently use GPT-4o as their base model, while we vary the planner’s base model to examine its impact. In contrast, the last row of Table 2 corresponds to a different setting where both the planner and the workers use Claude-3.7-SONNET as their base model, as specified in Table 1. We will make this distinction clearer in the final version of the paper.\n\n**Q2.& W3. (1) It would be great if authors could comment on the details of how these workers were defined in the experimental section**\n\nAs for the worker definition, our worker design adheres to two key principles: capability specialization and clear role definition. First, we minimize functional overlap between workers to ensure efficient task allocation and avoid redundancy. Second, we establish precise capability boundaries for each worker type to enable optimal decision-making by the Coordinator and Planner.\nIn the current instantiation of Workforce, we primarily target general-purpose tasks, which demand agents equipped with at least the following core capabilities:\n(1) web browsing for information retrieval,\n(2) multimodal file processing for diverse input handling, and\n(3) advanced reasoning for complex problem-solving.\n\n**Q2.& W3. (2) how scalable this is to new tasks: how the finetuned planner generalizes to unseen workers, and in particular whether there are still improvements wiht respect to the zero-shot version.**\n\nWe conducted a supplementary experiment to verify the generalization of the trained planner. Specifically, we added an additional agent for medical image analysis in medical scenarios. We used the dev dataset from the MedMCQA[1] dataset and tested the planner before and after training (based on the Qwen2.5-32b-Instruct model). The experimental results showed an accuracy of **71% before training** and **75% after training**, demonstrating that the trained planner maintains generalization capabilities in out-of-domain scenarios.\n\n\n**Q3. Why only training the high-level planner? Would there be gains from training the coordinator as well?**\n\nThanks for your valuable concern about the coordinator. In our initial experience, the benefits of using models with different capability levels for the coordinator are not very different. Therefore, we didn't choose to train the coordinator. To verify this, we conducted a simple experiment to evaluate the impact of different base models on the coordinator to assign workers. Specifically, we used **GPT-4o** to generate subtasks for each of the GAIA tasks, with 783 subtasks in total. We then fixed the worker set and **manually labeled** the workers that should be used for each subtask, marking them as **ground truth**. We used **GPT-4o and GPT-4o-mini** as coordinators, respectively, and analyzed the **accuracy of task assignment** using different base models. The experimental results showed that **GPT-4o** achieved an accuracy of **88.8%** and **4o-mini** achieved an accuracy of **90.1%**, which means that when the boundaries between different workers are clear enough, an LLM with general capabilities can already assign the correct worker to the current task well. \n\n\n**Q4.& W4. Clarify differences (besides the finetuning approach) with respect to works like Magnetic-One.**\n\nWe sincerely appreciate this opportunity to clarify the difference between our work and Magentic-One. Beyond fine-tuning, our system differs from Magentic‑One in terms of **architecture** and **communication**. We **decouple** strategic task decomposition (Planner) from subtask orchestration (Coordinator), and use a **shared task channel** where workers post only concise final results—keeping contexts clean and improving the signal-to-noise ratio for replanning. In contrast, Magentic‑One employs a **single Orchestrator** that centrally plans, tracks, and replans tasks.\n\nFurthermore, the experimental results demonstrate the advantage of our workforce over Magentic-One (workforce with GPT-4o: 60.01 vs. Magentic-One with o1: 46.06). In addition, we conducted an experiment where we merged the Planner and Coordinator into a single agent—similar to Magentic-One—and the result (60.01 vs. 56.36) further highlights the benefit of our decoupled design.\n\n\n**Q5.& W2. Results: if SFT is finetuned on GPT4o-mini, why is there such a significant gap between the first and second to last row in in Table 3.**\n\nThanks for your valuable feedback. Our **SFT** uses **1,599 filtered trajectories** synthesized by GPT-4o-mini and shows the known pattern of **gains on L1/L2 but a regression on L3 (−3.85%)**, indicating transferring planning style from GPT‑4o‑mini to **Qwen‑32B** is challenging and can overfit to simpler cases. Once we add **DPO**, the planner improves from **41.21% to 52.73%**, **surpassing GPT‑4o‑mini (47.27%)** and yielding **+7.69% on L3**, indicating reinforcement learning is required to learn robust strategies beyond imitation. \n\n**W6. While the approach is promising, it looks like one big limitation is in the necessity to define worker agents per task. The result of this is that, for new tasks, it is still necesary to write agents with domain-specific knowledge. It would be great if authors could comment on the details of how these workers were defined in the experimental section and how scalable this is to new tasks.**\n\n\nThe system has the potential to become significantly more scalable through automated agent design. To explore this, we conducted an experiment in which a meta-agent, powered by GPT-4o, automatically generated worker configurations. We tested this approach on 10 diverse tasks, using human-crafted workers as the baseline. Task performance was evaluated using human ratings on a 1–5 scale. The results showed that the meta-agent–generated workers achieved quality scores close to those of manually designed counterparts (average score: 4.3 ± 0.2 vs. 4.7 ± 0.1). We will include the full results in the final version of the paper.\n\n**References**\n\n[1] Pal, Ankit, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. \"Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering.\" Conference on health, inference, and learning. PMLR, 2022."}]

Author rebuttal:

Meta-review: All the reviewers agree to accept this paper, so I recommend acceptance as well. However, I don't think this work is significantly novel. Re-training planner alone is not unique in this work. Issues such as scalability along the subtasks are not handled, as pointed out by reviewers and admitted by the authors.  To be frank, I've found myself learned nothing after reading this paper.
Decision: Accept (poster)