Title: OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Model

LLM-based Agent

Multi-Agent
Abstract: Large Language Model (LLM)-based multi-agent systems show promise for automating real-world tasks but struggle to transfer across domains due to their domain-specific nature.
Current approaches face two critical shortcomings: they require complete architectural redesign and full retraining of all components when applied to new domains.
We introduce **Workforce**, a hierarchical multi-agent framework that decouples strategic planning from specialized execution through a modular architecture comprising:
*(i)* a *domain-agnostic* **Planner** for task decomposition,
*(ii)* a **Coordinator** for subtask management, and
*(iii)* specialized **Workers** with *domain-specific* tool-calling capabilities.
This decoupling enables cross-domain transferability during both inference and training phases:
During inference, Workforce seamlessly adapts to new domains by adding or modifying worker agents;
For training, we introduce **Optimized Workforce Learning (OWL)**, which improves generalization across domains by optimizing a domain-agnostic planner with reinforcement learning from real-world feedback.
To validate our approach, we evaluate Workforce on the GAIA benchmark, covering various realistic, multi-domain agentic tasks.
Experimental results demonstrate Workforce achieves open-source state-of-the-art performance (**69.70%**), outperforming commercial systems like OpenAI's Deep Research by **2.34%**.
More notably, our OWL-trained 32B model achieves **52.73%** accuracy (**+16.37%**) and demonstrates performance comparable to GPT-4o on challenging tasks.
To summarize, by enabling scalable generalization and modular domain transfer, our work establishes a foundation for the next generation of general-purpose AI assistants.

*Our code is available at [Anonymous URL](https://anonymous.4open.science/r/annonymous-owl/), and our data is available at [Anonymous URL](https://huggingface.co/anonymous21016).*


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a hierarchical method for multi-agent task execution using LLMs. The novelty of the work is that it uses a domain-agnostic planner and coordinator, which decomposes tasks and relays them to different agents, and domain-specific agent experts, which execute each part of the task. This enables to train only the experts on domain-specific data and maintain the same planner across domains and tasks. To train this system, the paper proposes OWL, which trains the domain agnostic planner via SFT on demonstratiosn from GPT-4o and RL on task success. Without any training, the approach obtains SOTA on the GAIA benchmark compared to other Open-Source frameworks, and is the first to outperform the closed-course Deep Research model. The training process also shows significant improvements, leading to 13% relative improvement with SFT and 46% relative improvement with DPO.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Relevance: The paper addresses an important problem, which has gained important in the research community: performing complex reasoning tasks by orchestrating multiple agents. \n- Novelty: to my knowledge, this is the first work that learns a high level planner with task reward data, rather than training the actor agents. This seems like a powerful approach since there may be agents in the that cannot be finetuned (either because the provider does not allow it or because they are non-learned agents). This framework focuses on the high level planner which is agnostic to the agents and tasks. the finetuning approach obtains significant gains over the zero-shot approach.\n- Experimental details: Provide a detailed analysis on the pretraining datasets used to improve the high-level planner, motivating the selection of different datasets and describing an approach to filter low-quality data from the experts, with significant gains on SFT. It would be great to look at the contribution of each dataset in GAIA performance.\n- The proposed appraoch can be combined with different base models and obtains SOTA results on GAIA when compared to similar models. In the zero-shot setting, WorkForce with GPT-4o and Claude-3.7-Sonnet, outperforms all the other open source frameworks with similar base models by a wide margin, and are close to other closed-source frameworks. In the fine-tuning setting, the framework outperforms closed-source based models such as GPT-4o-mini and larger models. \n\nWeaknesses:\n- Clarity: The structure of the paper makes it somewhat hard to follow, there is a high overlap between preliminary and related work, and they could be moved as a single section. Similarly the results in section 3.3 and 4.4 should be comparable since the only difference seems to be on whether the Workforce approach is further pretrained or not. Making these in separate sections creates confusion on whether the evaluations (dataset metrics and baselines) are comparable or not.\n- Results: if SFT is finetuned on GPT4o-mini, why is there such a significant gap between the first and second to last row in in Table 3.\n- One of the main claims of the work is that the planner is agnostic to tasks and agents, and therefore can generalize to different types of agents. It would be great to test how overfit is the high-level planner to the specialized workers. What happens if you change the capacity of the workers or their specialization? Do you still get gains on the finetuned planner? \n- Related work: It would be great to clarify the novelty with respect to related work, particularly what is the difference between the orchestration in your work vs Magnetic-One? You have a special agent that is doing task decomposition and one that is doing task planning, which seems similar to the orchestrator in Magnetic-One.\n- Method and Clarity: The way the results are presented, I am confused about how table 3 compares to table 2. Are all results on Table 3 showing the workforce framework with different base planners? Why are the numbers for Claude-3.7-Sonnet in Table 3,row 3 different from those on Table 2, last row? Shouldn't it be the same method? It would be useful to see how the same exact planner model performs both when using the Workforce framework vs not. Ideally taking one of the frameworks from Table 2 and replacing them with the Qwen2.5-32B-Instruct base, this way it would be possible to see how much gain there is from the workforce framework vs the post-training.\n- While the approach is promising, it looks like one big limitation is in the necessity to define worker agents per task. The result of this is that, for new tasks, it is still necesary to write agents with domain-specific knowledge. It would be great if authors could comment on the details of how these workers were defined in the experimental section and how scalable this is to new tasks.\n\nQUESTIONS:\nIn order of importance:\n- Clarify differences between results in Table 2 and Table 3. Particularly difference between  Table 3,row 3 different from those on Table 2, last row.\n- It would be great to test how the finetuned planner generalizes to unseen workers, and in particular whether there are still improvements wiht respect to the zero-shot version.\n- Why only training the high-level planner? Would there be gains from training the coordinator as well?\n- Clarify differences (besides the finetuning approach) with respect to works like Magnetic-One.\n- Results: if SFT is finetuned on GPT4o-mini, why is there such a significant gap between the first and second to last row in in Table 3.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Workforce, a multi-agent framework that is modular and scalable, featuring a domain-agnostic planner, a coordinator, and domain-specific worker nodes. The paper also introduces OWL, an optimized workforce learning, an efficient and effective multi-agent training paradigm. Both the proposed framework and the training paradigm yield impressive performance on the GAIA benchmark.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The performance is impressive.\n- The code is fully open-sourced.\n- The analysis provides insight for the community.\n- The presentation is clear.\n\nWeaknesses\n\n- As a modular framework, there is no ablation study on the contribution of each module. Providing this could make the framework design more convincing. It's now unclear why introducing a coordinator is necessary compared to letting the planner handle this role assignment. Figure 4(c) only discusses the training of the planner and the worker, also gives the impression that the coordinator is not necessary.\n- The design choice of using a shared task channel is not supported by experimental results. E.g., would it be even more helpful if the planner could access the context of the worker to better replan?\n- Though the centralized mechanism makes the management of context easier, it also limits the potential of the proposed multi-agent system. E.g., workers could not ask for clarification from the higher-level agents (planner here).\n- For the goal of generalist multi-agent assistance, it might be worth some discussion on how to select the worker agents, or even creating worker agents autonomously. It's not justified why the three worker agents in 3.2 are chosen for this goal. It seems more of a choice for the GAIA benchmark. Moreover, the generalizability of the trained planner needs to be verified with different worker agent selections if it's supposed to be utilized off-the-shelf by the community.\n- For baselines in 3.3, it seems confusing to have (iii) and (iv) as groups, while in Table 1, there are only two groups of (i) (ii).\n- It's worth noting that the idea of decoupling strategic planning from domain-specific execution is also explored extensively in the embodied agents literature, which should also be discussed in the paper. [1][2][3]\n\n[1] Building Cooperative Embodied Agents Modularly with Large Language Models. ICLR24\n\n[2] Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents. NeurIPS23\n\n[3] Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. ICML22\n\nQUESTIONS:\n- Why pass@3 sampling for GPT-4o while pass @1 for Claude-3.7-sonnet?\n\nAlso see seaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces WORKFORCE, a hierarchical multi-agent framework designed to address the domain-specific limitations of existing LLM-based systems. By decoupling strategic planning from domain-specific execution through a modular architecture (Planner, Coordinator, Worker Nodes), WORKFORCE enables cross-domain transferability with minimal retraining. The framework is validated on the GAIA benchmark, achieving state-of-the-art open-source performance and outperforming commercial systems like OpenAI’s Deep Research. Additionally, the Optimized Workforce Learning (OWL) training paradigm enhances generalization via SFT and reinforcement learning, boosting the Qwen2.5-32B model by 16.37% and demonstrating performance comparable to GPT-4o on complex tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Clear and excellent academic writing with well-designed figures  \n- The flexible and modular multi-agent architecture is crucial for swarm intelligence system applications  \n- Only requiring tuning of the planner reduces training overhead significantly  \n- State-of-the-art (SOTA) experimental results outperform current renowned frameworks  \n- Full open-source release of the code  \n\n\n**Weaknesses**\n\nSee `Questions`.\n\nQUESTIONS:\n+ The writing style resembles an engineering report rather than an academic paper, with somewhat simplistic descriptions of algorithmic process details.\n\n+ As shown in Fig. 4, the effectiveness of OWL is primarily constrained by tool capabilities, even more so than the planner. However, tool invocation requires meticulous manual authoring. To fully achieve automation, how can Worker nodes proactively modify and expand their capabilities? This is mentioned in line 107 of the paper, but specific details are absent.\n\n+ Line 102 states, \"Planner Agent analyzes incoming tasks and decomposes them into subtasks based on worker capability registry.\" Evidently, the Planner Agent relies on existing Workers for task decomposition. Thus, when Workers are modified, does the Planner need to be retrained? Can a fine-tuned Planner dynamically adapt to different Worker identities?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a hierarchical multi-agent framework that decouples strategic planning from domain-specific execution to enable cross-domain transferability. The system comprises three core components: a domain-agnostic Planner for task decomposition, a Coordinator for subtask management, and specialized Workers with domain-specific capabilities. The authors also propose Optimized Workforce Learning (OWL), a training paradigm that focuses on optimizing the domain-agnostic planner through supervised fine-tuning followed by reinforcement learning. The approach is evaluated on the GAIA benchmark, achieving 69.70% accuracy and outperforming commercial systems like OpenAI's Deep Research by 2.34%.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n- Novel Modular Architecture: The separation of domain-agnostic planning from domain-specific execution is well-motivated and represents a significant architectural innovation. This design enables plug-and-play extensibility across domains without requiring complete system redesign.\n\n- Strong Empirical Results: The paper demonstrates impressive performance on GAIA, achieving state-of-the-art results among open-source frameworks (69.70%) and surpassing proprietary systems like OpenAI's Deep Research. The OWL training method shows substantial improvements (+16.37% for Qwen2.5-32B-Instruct).\n\n- Comprehensive Evaluation: The paper includes thorough ablation studies, error analysis, and performance breakdowns across different capability types. The test-time scaling analysis and robustness evaluation add valuable insights.\n\n- Reproducibility: The authors commit to releasing all code, models, and data, which supports open research.\n\n\n## Weaknesses\n\n- Tool Dependency: The system's performance is heavily dependent on the quality and availability of domain-specific tools. The paper doesn't adequately address how to handle domains with poor tooling.\n\n- Scalability Concerns: The paper doesn't thoroughly address how the approach scales with the number of workers or the complexity of coordination required for very large multi-agent systems.\n\nMissed References:\n\n[1] Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage.\n\n[2] From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions.\n\nQUESTIONS:\n0. Why are you using JSON agent instead of Code agent?\n\n1. How does the system perform on domains that are significantly different from those in the training curriculum? Can you provide analysis on out-of-domain generalization?\n\n2. How does the system handle cases where multiple workers could potentially handle a subtask? What mechanisms prevent conflicts or redundant work?\n\n3. While you mention replanning mechanisms, how robust is the system to cascading failures where multiple workers fail sequentially?\n\n4. What is the computational overhead of the coordinator and communication mechanisms compared to direct single-agent approaches?\n\n5. How does performance degrade as you add more workers or increase task complexity? Are there theoretical or empirical limits to the approach?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}