Agent-Oriented Planning in Multi-Agent Systems
Authors: Ao Li
Yuexiang Xie
Songze Li
Fugee Tsung
Bolin Ding
Yaliang Li
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-Agent System; Planning
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Through the collaboration of multiple LLM-empowered agents possessing diverse expertise and tools, multi-agent systems achieve impressive progress in solving real-world problems. Given the user queries, the meta-agents, serving as the brain within multi-agent systems, are required to decompose the queries into multiple sub-tasks that can be allocated to suitable agents capable of solving them, so-called agent-oriented planning. In this study, we identify three critical design principles of agent-oriented planning, including solvability, completeness, and non-redundancy, to ensure that each sub-task can be effectively resolved, resulting in satisfactory responses to user queries. These principles further inspire us to propose AOP, a novel framework for agent-oriented planning in multi-agent systems, leveraging a fast task decomposition and allocation process followed by an effective and efficient evaluation via a reward model. According to the evaluation results, the meta-agent is also responsible for promptly making necessary adjustments to sub-tasks and scheduling. Besides, we integrate a feedback loop into AOP to further enhance the effectiveness and robustness of such a problem-solving process. Extensive experiments demonstrate the advancement of AOP in solving real-world problems compared to both single-agent systems and existing planning strategies for multi-agent systems. The source code is available at https://github.com/lalaliat/Agent-Oriented-Planning

Published as a conference paper at ICLR 2025
Agent-Oriented Planning in Multi-Agent Systems
Ao Li Yuexiang Xie Songze Li Fugee Tsung!? Bolin Ding
Yaliang Listt
'The Hong Kong University of Science and Technology (Guangzhou)
The Hong Kong University of Science and Technology *Alibaba Group 'Southeast University Engineering Research Center of Blockchain Application, Supervision and Management (Southeast University), Ministry of Education
aliat ust hk, songzeli@seu. edu, en, season hk, fyuexiang. xyx, bolin. ding, yaliang. 11 e ibaia-aaaaaaae -incinnn--- inm.com
ABSTRACT
Through the collaboration multiple LLM-empowered agents possessing diverse expertise and tools, multi-agent systems achieve impressive progress in solving real-world problems. Given the user queries, the meta-agents, serving as the brain within multi-agent systems, are required to decompose the queries into multi- ple sub-tasks that can be allocated to suitable agents capable of solving them, so-called agent-oriented planning. In this study, we identify three critical de- sign principles of agent-oriented planning, including solvability, completeness, and ron-redunn.ncy, to ensure that each sub-task can be effectively resolved, re- sulting in satisfactory responses to user queries. These principles further inspire to propose AOP, novel framework for agent-oriented planning in multi-agent systems, leveraging a fast task decomposition and allocation process followed by effective and efficient evaluation via reward model. According the eval- uation results, the meta-agent is also responsible for promptly making necessary adjustments sub-tasks and scheduling. Besides, we integrate a feedback loop into AOP to further enhance the effectiveness and robustness of such problem- solving process. Extensive experiments demonstrate the advancement of AOP in solving real-world problems compared to both single-agent systems and exist- ing planning strategies for multi-agent systems. The source code is available at https///iihubb.com/aaall/////// -rrriiitt//gget-t-eteeeeede Oriented-Planning Planning
INTRODUCTION
In recent years, large language models (LLMss) (Achiam et al. 2023 Chowdhery et al. 2023 Touvron et al.) (023) have achieved impressive breakthroughs in natural language understanding and generation, marking a critical advancement in the exploration of artificial general intelligence (AGI), As the capabilities of LLMs progress, LLM-empowered agents (Qin et al. 2024 Dong al.) 2024 Wang et al. 2024) emerge as key components for integrating expertise and tools to effectively translate these ad advancements into practical applications. Advancing this paradigm, multi-agen sys-
tems (Shen et al.] 2023 Gao et al. 2024 Wu et al.) [024), which involve multiple diverse agents, provide great flexibility and adaptability by leveraging and collaborating on the strengths of various agents, promoting comprehensive solutions to complex real-world problems.
Previous studies (Cai al.; 2023 Qian et al. 2024 propose to define suitable standard operating
procedures (SOPs) based on the insights and experiences of human professionals for solving specific tasks, such software development (Hong al. 2024) and simulating personality traits (Serapio- Gareía et al.) 2023). In these scenarios, multiple agents are assigned to execute tasks following the predefined SOPs, resulting in remarkable successes. However, for multi-agent systems designed to tackle a diverse range of complex real-world problems, there is critical need for central entity that can automatically generate task specific operating procedures and coordinate the activities of various agents (Wang et al. 2024).
This central entity, typically referred to as meta-agent (a.k.a. a controller or planner), carries two primary responsibilities. Firstly, the meta-agent needs to comprehend user queries and decompose
Work done as an intern at Alibaba Group.
1 Corresponning author.

---

Published as a conference paper at ICLR 2025
them into several sub-tasks, ensuring that each sub-task can be adequately addressed by single agent. Secondly, the meta-agent is also expected to assign these sub-tasks to the appropriate agents for execution, enabling the solutions of these sub-tasks to collectively provide comprehensive an- swers to the original user queries.
However, since the meta-agent cannot effectively associate sub-tasks with agents based on the pro- vided agents descriptions (Liu et al. 2024 Hong et al.] [024), the performance of task decompo- sition and allocations might be sub-optional. To tackle these challenges, we identify three design principles for an agent-oriented planning framework, including solvability, completeness, and non- redundancy. These principles further inspire us to propose AOP, a novel framework for Agent-
Oriented Planning in multi-agent systems.
Specifically, to resolve a user query, the meta-agent first performs fast task decomposition and allo- cation, serving as intermediate results that can be further modified and revised. The proposed AOP incorporates a reward model designed efficiently evaluate the solvability of sub-tasks without re- quiring actual agent calls. According to the evaluation results, some sub-tasks may be executed by the assigned agents, others may be deemed inappropriate and require re-planning, while the remain- ing sub-tasks are further assessed using the representative works mechanism, which helps determine whether they should be planned in detail or re-described for better aligning the ability of agents. Mcanwhile, detector is utilized to identify the missing key information or redundant content in the decomposed sub-tasks, and to provide suggestions to the meta-agent for enhancing the com- pleteness and non-redundancy. Besides, feedback loop is integrated into AOP promote ongoing enhancements for the problem-solving process.
Extensive experiments are conducted based on several reasoning datasets that require collaboration among multiple L.Meeppoweredd agents. Comparisons between AOP and baseline methods demon- strate the remarkable advancements achieved by the proposed framework. Furthermore, we conduct an ablation study to show the contributions of different components in AOP, and provide discussions on the potential for further enhancing agent-oriented planning within multi-agent systems.
2 RELATED WORK
Large language models (LLMs) (Brown et al. 2020 Achiam et al.
2023 Chowdhery et al. 2023
Touvron et al. 2023] Le Scao al. 2023 Zhang et al. 2022) have demonstrated remarkable capabil- ities in understanding and generating human language (Zhao et al. 2024). To enhance LLMs' ability to solve complex problems, some research focuses on improving the internal reasoning capabilities of LLMs, typically involving decomposing complex questions into sequential intermediate steps before generating the final responses, as exemplified by Chain-of- Thouuhh (CoT) prompting (Weil et al. 2022) and its variants (Kojima et al. 2022, Wang et al., 2023 Zhou et al. 2023 Li et al.)
2023
For further improvements, recent studies propose to adopt multi-agent systems (Wu et al,,
2024
Hong et al. 2024 Wu et al. 2025) which leverage the collective intelligence and specialized
skills of multiple LLM-empowered agents to solve tasks collaboratively.
Traditional cooperative multi-agent planning methods mainly rely symbolic methods rein- forcement learning-based methods (He et al., 2016). Yao et al.) (020), with MA-PDDL (Kovacs) et al. 2012 being key example, establishing standard specification language for multi-agent planning, However, these methods can be limited by their complexity and heavy reliance on human experts (Huang al., 2024). Recently, the development of LLMs helps simplify and optimize the planning process through the creation of meta-agents, which serve as controllers that manage the flow of operations, perform task decomposition (Huang et al. 2024). There are mainly two technical approaches: interleaved decomposition and decomposition first. Regarding the first kind of approach, React (Yao et al., 2023) generates reasoning and action in an interleaved manner, where reasoning helps the meta-agent update action plans and action enables interaction with agents. HUSKY (Kim et al., 2024) trains an action model to iterate between two stages: generating the next action to take and executing the action using expert models and updating the current solution state. Given that the excessively long trajectories might bring hallucinations in interleaved decomposi- tion. HuggingGPT (Shen a (202) addresses this issue by carefully designing prompts to instruct ChatGPT to decompose tasks, selecting models in HuggingFace and summarizing the generated responses. Chameleon (Lu et al. 2023) prompts the meta-agent with tool descriptions and usage
2

---

Published as a conference paper at ICLR 2025
examples to infer program composed of sequence of tools to execute for generating the final response.
Although remarkable progress has been made, plans generated solely through prompt-based meth- ods often fall short of meeting the principles of solvability, completeness, and non-redundancy is- sues that have largely been overlooked in previous studies, which motivates us propose novel agent-oriented planning framework for multi-agent systems.
3 PRELIMINARIES
Definition of Agent-Oriented Planning In this study, a multi-agent system involves series of LLM-empowered agents for solving real-world problems. These agents are constructed by pro- viding specialized tools and unique system prompts that set their identities and instructions, and are powered by LLMs for query understanding, tool use, and response generation. For example, a search agent can utilize search engines to retrieve up-to- information relevant to a given query, while code agent is capable of generating code and executing via code interpreter. The collaboration among these agents is facilitated by a meta-agent, which serves as the brain of the multi-agent sys- tem. When user query is submitted, the meta-agent sends the relevant requests to the appropriate agents, optimizing for both effectiveness and efficiency. The responses produced by these agents are finally aggregated and synthesized to generate comprehensive answer to the original user query.
Formally, given a user query Q, a meta-agent P needs to select a set of appropriate agents from a total of agents, denoted as A = [_A,-.., An). Each agent is associated with a description d that outlines its capability. We denote the collection of all agent descriptions as D = {d1.... .... d J. The user query Q can be decomposed by P into m sub-tasks, with each sub-task assigned to a specific agent based on their descriptions:
(1) After that, each selected agent E [m] produces a response to its assigned sub-task, denoted as rs ((a)). These responses fri,....Fmm }m) are then utilized to generate the final answer to resolve the original query Q. Such process is called agent-oriented planning.
P(Q,D,A) = ((g. l [m]].
Challenges The challenges of agent-oriented planning in multi-agent systems can be two-fold. Firstly, different from existing studies focused on task decomposition (Shen et al.) 2023 Lu et al.) 2023) or thain-of-thought reasoning (Wei et al. 2022), agent-oriented planning requires intentional decomposition of user queries to effectively associate sub-tasks with agents, which includes con- siderations of the description of sub-tasks, the granularity of decomposition, the format of the re- sponses, and so on. An example is illustrated at the higher left of Figure Given user query "How much tin (kg) with 100 kg copper can lower the mixture's 's melting point to 800 "C?", a naive decom- position might suggest "Determine the melting point of tin and copper" followed by "Calculate the amount of tin (kg) required to reduce the melting point of the mixture to 800 "C with 100 kg cop- per". However, when the sub-task of determining the melting points assigned to a search agent, it may not result in satisfied responses since the query involves two entities simultancously. In the context of agent-oriented planning, is important consider the capabilities of agents. As a result, such a sub-task should be further decomposed into individual entity searches, i.e., first determine the melting point of tin and then determine the melting point of copper.
Secondly, assigning sub-tasks to appropriate agents is non-trivial, as the meta-agent can only rely on the agents' descriptions to determine task allocation in most cases (Shen et al.) 2023). However, a concise and highly generalized description that adequately illustrates an agent's 's capabilities may not always be available, Ieading to suboptimal allocation. For example, if the description of a commonsense agent does not specify the extent of its knowledge base, the meta-agent may struggle to ascertain whether it is suitable to assign the sub-task of querying the melting points of tin and copper to that agent, as shown at the lower left of Figure 1
Design Principles We further identify three critical principles that guide the design of AOP for effective and efficient agent-oriented planning in multi-agent systems:
Solvability. Each sub-task q: Vi E [m] should be independently and completely resolvable by least one single agent within the multi-agent system, ensuring that the response for
3

---

Published as a conference paper at ICLR 2025
Meta agent
agent
agent
Commonsenne Spent
Salvability
Sub task Determine 1% New York's popuittion.
jeal with copper
malting and copper
X
Sub- took Calculate the amount of required to reduce the metting point the minure 100'C with 100 cooper
owe
300 population?
mettin point
Sub Determine New Yorics pepulation.
Sub-task Calculate 14 New York's populazion.
The meiting point oopser is 1,88"C. The mat point of iin mencianed the resperse..
oupper.
Completeness
bab-ttak Determine the number planes 1% of New York's popuration.
Sub-tack 1.1: Determine the matting do ni tin.
The meiting ofint tin "C.
X
passenger planes carry
Sub-tank 1.2: Determine the milting co cupper.
The meiting point copper is 1,001 "C
tork's popularion?
carry capactty population.
Non reduundadyyyy
Sub-tank Detarmine the melting mi and coppar.
?
Commorsamse apent: smserr the suns cuest on commomamee reasoning,
How much tn (aa) with
Sub-taakk 1: Determine the ne ling point tin and copper. Sub-baek Search for common mee ting points for tin-copper aloa.
100 lnaeer mature's meeltnng point morce
Figure I: Examples of agent-oriented planning in multi-agent systems, regarding two challenges (left side) and three design principles (right side).
each sub-task can reliable. a sub-task does not satisfy solvability, the meta-agent expected to take some modifications or further decomposition.
Completeness. The array of sub-tasks {q1.... Im should include all necessary information from the original user query Q, which ensures that the aggregation of responses of these sub-tasks can effectively yield comprehensive answer to the user query. While sub-task might include only part of the necessary information, it not allowable for any particular piece of critical information to be omitted from all sub-tasks.
Im should not include redundant ele- ments, avoiding those task executions that are either irrelevant to resolving Q or duplicated. The principle of non-redundancy promotes that the sub-tasks form minimal effective set necessary to address the user query, enhancing overall efficiency.
Non-Redundancy. The array of sub-tasks
Several examples provided the right side in Figure I for a better understanding of these design principles. Note that here we focus on the redundancy between sub-tasks that should be simplified, and believe that adding redundancy among agents necessary for fault tolerance as discussed in Appendix B especially considering unexpected changes in agent availability.
While these principles may appear general in nature, we propose an explicit specification and sys- tematic design that follows these principles in the domain of LLM-based multi-agent systems. We hope that these principles, along with the framework we have outlined the next section, can serve as good starting point and inspire further research in this field.
4 AGENT-ORIENTED PLANNING IN MULTI-AGENT SYSTEMS
In this section, we introduce the details of the proposed AOP framework, with the overall architecture illustrated in Figure
4.1 FAST DECOMPOSITION AND ALLOCATION
First of all, following existing studies Shen et al. 2023 Chen al. 2024). provide detailed instructions to the meta-agent to guide it in performing agent-oriented planning. What sets our ap- proach apart is that the provided instructions incorporate the following requirements: (i) Integration
4

---

Published as a conference paper at ICLR 2025
Query
Meta-agent
Agents
Responses
Derector
Redundent sub-tasks
Missing dependencies
Decomposirion
Execution
& Allocation
Plan
Refined Plan
Feotbock
1000
Reward model
Represenlative
Replan
Re-describe
Plan detail
Figure 2: Overall architecture of AOP. The framework begins with the mela-agent performing fast decomposition and allocation of the received user query, resulting in an initial plan. A detector is employed to improve the completeness and eliminate redundancy of the plan, while reward model provides scores to guide the meta-agent in refining the plan further, which involves operations such as replan, plan-in-detail, etc. The refined plan sent back to the multi-agent system for generating responses to the user query.
of the user query Q and all the agent descriptions D: We include both the user query and the agents' descriptions in the instructions, promoting the meta-agent to fully consider the capabilities of each agent and tailor the sub-tasks to align with these capabilities. (ii) Suggestions for assigned agents: We require the meta-agent to provide suggestions for the agents assigned to each decomposed sub- task. Although we tend to separate task decomposition and assignment into two independent tasks performed sequentially, the experimental observations indicate that combining these two tasks en- hances the effectiveness of agent-oriented planning. (iii) Structured decomposition of tasks: The decomposed tasks required to be structured in sequential manner, specifying any dependencies that may exist between sub-tasks, which ensures that the execution of tasks follows correct logical order. The adopted prompt can be found in Appendix A.I
The aforementioned process of agent-oriented planning, which called decomposition and al- location in this study, heavily relies on the capabilities of the meta-agent and meticulously designed system prompts. While such an end-to-end approach can achieve high efficiency, its success rate, namely fulfilling the three design principles mentioned above and providing reliable answers the original user query, and its stability, namely the meta-agent can follow instructions and produce formatted responses, might not be satisfactory.
Inspired by recent studies on inference time computing OpenAA 2024), we propose to design mech- anisms to guide the meta-agent toward more comprehensive reasoning processes. The results of fast decomposition and allocation should be viewed as intermediate outputs rather than final results, which need to be further evaluated to offer detailed revision suggestions for the meta-agent. More details on this proposed approach are provided in the next subsection.
4.2 IS A SUB-TASK COMPLETELY RESOLVED?
To determine whether sub-task can be resolved by a single agent, i.e., satisfying the principle of solvability, straightforward solution is to send the sub-task to every agent in the system. Each agent generates a response to the sub-task, allowing us to select the best answer and assess whether it completely resolves the sub-task. However, this approach is often impractical due to the unafford- able overhead. For user query decomposed into m sub-tasks, multi-agent system with n agents would need to execute total of strategy in scenarios with a large number agents and sub-tasks.
agent calls for just single trial, making it inefficient
To tackle this, introduce reward model that provides efficient evaluations of the solvability of sub-tase tub-tasks, which aims to predict the quality of the agents' responses to tub-tasks without necessitat- ing actual agent calls.
5

---

Published as a conference paper at ICLR 2025
Specifically, first prepare a dataset for training the reward model. For given user query Q, we follow the fast decomposition and allocation process introduced in Section 4.1 requiring the meta-agent to decompose Q into several sub-task: and select agents for each sub-task based on the agents' descriptions, which can be formally given as follows:
P(Q,D,44 = A1, A.d) i [m]}-
(2)
After that, we execute the plan, i.e., sending the sub-tasks to the assigned agents, and obtain all the responses from agents
fri. = Ai)(a)) | [m]]. E [4].
(3)
where the choice of I can be trade-off. A large leads to comprehensive responses from lots of agents for constructing the training dataset, while it might need more computation resources and affect the overall quality of the dataset. In this study, we set / to be half the number of agents in the multi-agent systems.
We utilize scorer S that evaluates agents' responses sub-tasks, i.c., S(g;r..) st,s which serves as annotations in the training dataset. The scorer S provides evaluations from three key aspects, including correctness, relevance, and completeness, and can be implemented by using LLMs or employing human annotators. We apply this scoring process across a diverse array of user queries Q {Qu]k= K and collect the results to form the training dataset, denoted as T [[TK,i, da, , sk.i.j) k [K], E [m=x].] [4]] More details on the construction of the dataset, such as the adopted prompts, are summarized Appendix
The reward model M. parameterized as 6, consists of embedding layers followed by fully con- nected layers. We obtain embeddings for both the sub-task and the agent's description separately, then concatenate these embeddings together and feed them into the fully connected layers for further processing. To provide evaluations without making actual agent calls, we design the training objec- tive aimed minimizing the discrepancies between the model's predictions and the annotations provided by the scorer, which can be formulated as follows:
K
mg
L(T,O)
BRRAJ Molgs.i d).
K
me
With the well-trained reward model, the evaluation of the results produced by fast decomposition and allocation can be summarized as follows. For each suggestion provided by the meta-agent, stating the assignment of sub-task qi to agent A'. the reward model predicts a score denoted as Me(g; d)). A sufficiently high predicted score, which implies that 9: can be completely resolved by agent A', leads to the acceptance of this suggestion. Conversely, if the predicted score falls below predefined threshold, which indicates that the assignment should be revisited by the meta-agent, the reward model provides a set of scores $1,5 = Ma(gi, dz) for all the agents A,, = =,..., From these scores, the meta-agent reports the optimal selection as Jmaz argmax, $t.j.
Note that there may be scenarios where the highest score 8i.imux is still extremely low, indicating that the sub-task qi does not satisfy the principle of solvability and that no single agent can resolve it independently. In these cases, the meta-agent P is required to perform replan on g. The prompt used for replan detailed in Appendix A.2
4.3 SUB-TASK MODIFICATIONS
In addition to the two scenarios mentioned in Section 4.2 where the assignment of sub-task q: to agent A/ is either predicted to be acceptable by the reward model or to be entirely inappropriate and requires a replan process, there also exists another situation where agent A', might not provide reliable response to sub-task 9i, even though the assignment seems reasonable according to the reward model.
We identify two intrinsic reasons for such a situation in agent-oriented planning. First, the descrip- tion of the sub-task may lack critical information necessary for solving the problem, such as the interpretation of contextual elements like pronouns other ambiguous terms. In such cases, while the agent possesses the capability to resolve the sub-task, the missing information can hinder from providing satisfactory response. Second, the sub-task qi might be too complex, therefore single

---

Published as a conference paper at ICLR 2025
agent only address parts of this sub-task based on its expertise and tools. In this subsection, design mechanisms to distinguish between these two cases and propose corresponding solutions.
To be more specific, for each agent, propose to construct set of representative works that record the sub-tasks has completely resolved. We would initially bypass the construction and update for these representative works (refer Section 4.5 for more details) and focus how to utilize them to make sub-tasks modifications here. Given a sub-task Qi we calculate its similarity with representative work gr following C
where & the embedding part in M and | 1 calculates the cosine similarity. We define the similarity as follows:
cos(u, v)
w-1
sim(g: 2,) = max{cos (E()), E 22},
(5)
where it can be denoted as sim,, for short.
A large max(simi. E [na]} indicates that the representative works of the agent corresponding to max(sime, j É [r]]] contain a sub-task similar to qi- Such a case should be attributed to the first reason mentioned above, and tend to request the meta-agent to perform re-describe qa according to the similar representative work.
On the other hand, when the calculated similarity does not meet the threshold, regard qi as too complex for any single agent in the system to solve (corresponding to the second reason above). As a result, the meta-agent P is requested perform plan-in-dette for further decomposing the sub-task into simple ones. To avoid creating redundant sub-tasks, we provide P with all the sub-tasks and explicit instruction avoiding overlapping sub-asks. The prompt used for sub-task modification can be found in Appendix A.3 and Appendix A.4
4.4 DETECTOR FOR COMPLETENESS AND ROD-RREUND
Following the idea fast decomposition and allocation, we attempt to incorporate detailed instruc- tions to the aggett to satisfy the principles of completeness and non-redundancy. However, empirical observations reveal that these instructions may not always work well. fact, more than 15% of user queries still exhibit such issues during decomposition in our initial experiments.
To tackle these issues, we involve detector, implemented by providing a role-play prompt to LLMs, to evaluate both the completeness and non-redundancy of the intermediate results provided by the meta-agent. For the principle of completeness, the detector extracts all key elements and require- ments from the original query and then matches these elements against each sub-task, determining whether the sub-tasks collectively address all essential aspects of the original task. To further ensure that each sub-task can be successfully executed, the detector is also required to determine whether there are any additional dependencies needed beyond those identified during decomposition (c.g., unforeseen dependencies). If such dependencies exist, the execution results of these dependent sub- tasks should also be provided as inputs.
Regarding non-redundancy, the detector is responsible for analyzing whether any two subtasks con- tain identical information or try solve the same problem, and whether there are unnecessary subtasks that do not contribute to the user query's solution.
When the provided results do not satisfy the principles of completeness or non-redundancy, the de- tector identifies the missing key information or redundant content and offers recommendations for refining the plan, such suggestions for supplementing missing details removing overlapping sub-tasks. The prompt used for the detector is provided in Appendix AA.5) From the experimental results shown in Table we observe that the detector effectively mitigates the issues of incomplete- ness and redundancy, leading to an effective and efficient agent-oriented planning process.
4.5 FEEDBACK LOOP
An automatic feedback loop is integrated into AOP, promoting the ongoing enhancement of the problem-solving process. As introduced in Section 4.2 the reward model can identify certain sub- tasks as being resolvable by specific agents. These identified sub-tasks are then collected to form the representative works of the corresponding agents. The training data for the reward model. which includes the sub-tasks and the ground truth scores of responses provided by the agents, utilized to initialize these representative works.
7

---

Published as a conference paper at ICLR 2025
Note that the representative works continuously updated. When resolved, the sub-tasks decomposed from that user query can be added to the representative works of the agents who are selected to provide responses these sub-tasks. To prevent redundancy in the representative works, we implement a similarity threshold. This threshold ensures that only new and sufficiently distinct sub-tasks are incorporated into agent's representative works, maintaining the diversity and relevance of the tasks that each agent has previously resolved.
query is completely
5 EXPERIMENTS
In this section, we provide empirical comparisons between AOP and the existing studies.
5.1 SETTINGS
Datasets & Evaluations We conduct experiments based on numerical reasoning dataset (Kim al. 2024). which necessitates the collaboration of multiple agents in resolving the queries. For example, a query can be "If Sarah wants to buy one BMW X5 and one Tesla Model 3, how much more would she need to pay to buy the BMW X5 compared to the Tesla Model 3? To resolve this query, we first need to search for the prices of the BMW X5 and Tesla Model 3, and then calculate the price difference between them. Following previous study (Kim et al., 2024), we adopt Husky- QA, which consists of 1,440 queries in the training data and 292 queries in the test data. Besides, also provide more experimental results on the decontextualized versions of a subset DROP (Dua et al. 2019) and IIRC (Ferguson et al. 2020 in Appendix D.I.
For quantification comparisons, we provide instructions to GPT-4o (OpenAl 2024) for assisting in judging whether the execution results align with the ground truth and calculate accuracy as the evaluation metric. The prompt used for evaluation can be found in Appendix A.6
Involving Agents In the experiments, the multi-agent system includes a meta-agent and several diverse agents, including a code agent, math agent, search agent, and commonsense agent. The descriptions of these agents, which are similar to those in previous studies (Kim et al., 2024) for a fair comparison, are shown below:
Code Agent: Generate code in Python for precise computations to solve the given task.
Math Agent: Answer math questions by reasoning step-by-step.
Search Agent: Call the Bing Search API to obtain information related to the given task.
Commonsense Agent: Answer the given question using commonsense reasoning.
The meta-agent is tasked with decomposing user queries into sub-tasks and assigning the most suit- able agents to execute those sub-tasks. We use GPT-4o the employed LLM for all agents, and the prompts used are provided in Appendix A.7 Besides, an investigation on the effectiveness of involving expert agents powered by task-specified LLMs, and the method for efficiently handling the coexistence of similar agents can be found in Appendix
We adopt L-- (Wang et al. 2020) as the embedding layers of the reward model, which maps text sentences or paragraphs into 384-dimensional dense vector space. The embeddings for both the sub-task and the agent's 's description are computed separately, and then are concatenated together form 768-dimensional dense vector. Following the embedding model, we add a three-layer multilayer perceptron (MLP) with output dimensions of 256,64, and 1, respectively. We freeze the embedding layers and only fine-tune the parameters of the MLP, making the training process efficient. The batch size is set to 32, and the learning rate is le-3. We train the reward model for 50 epochs on Tesla V100-SXM2-32GIB GPU.
Reward Model
Baselines For the single-agent systems, we utilize GPT-4o as a baseline method, providing user queries directly without any additional prompt engineering. Besides, we provide GPT-4o with Chain-of- Thought (CoT) (Wei et al. 2022) prompt and Zero-Shot CoT (Kojima et al. 2022) prompt to guide the LLM to perform reasoning processes, resulting in two baselines denoted as CoT and Zero-Shot COT
8

---

Published as a conference paper at ICLR 2025
Table 1: Comparisons between AOP and baselines.
Method
Accuracy (%)
Prompt Tokens (M)
Completion Tokens (M)
Time (s)
GPT-4o
33.3
0.02
0.09
1,968
CoT
35.6
0.07
0.08
1,375
Zero-Shot CoT
32,2
0.02
0.11
1,565
Meta-Agent
30.0
0.65
0.13
5,364
Meta-Agent: Traversal
35.2
3.07
0.69
23.175
REACT
37.6
2.47
0.19
11,510
HUSKY
39.6
0.83
0.15
10,394
AOP (ours)
43.7
1.12
0.38
11,869
Regarding the multi-agent systems, we instruct GPT-4o to serve as meta-agent responsible for performing fast task decomposition and allocation. We implement two baselines, denoted as Meta- Agent and Meta-Agent: Traversal, respectively. Meta-Agent denotes that GPT--4 selects one agent for each sub-task, while Meta-Agent: Traversal denotes that GPT-4o iteratively queries all agents for each sub-task and selects the best response. Besides, we compare AOP with REACT (Yao) jet al., 2023) and HUSKY (Kim et al. 2024), which are representative methods involving task de- composition and sub-task execution.
5.2 COMPARISONS AND ANALYSIS
The comparisons between AOP and the baseline methods are shown in Table 1 We adopt accuracy the metric for comparing effectiveness, and use prompt/completion tokens (refer the total cost on the whole test dataset) and the execution time for comparing efficiency.
Overall, the experimental results demonstrate that AOP achieves notable improvements compared to all baseline methods. To be specific, when comparing with single-agent systems including GPT-4o, COT, and Zero-Shot CoT, AOP outperforms these systems by 10.4%, 8.1% 1%, and 11.5% terms of accuracy, respectively. These improvements can be attributed to the collaboration among multiple diverse agents and the effective scheduling provided by the meta-agent.
It is not surprising to observe that the cost of AOP is significantly higher than that of single-agent systems, and at the same level compared to other multi-agent systems. These additional costs during the inference phase, which include task decomposition, allocation, and modifications carried out by the meta-agent, are affordable and can be worthwhile as long as they bring significant im- provements in accuracy and stability when applied to real-world applications. Such exploration aligns with a recent study (OpenAl (0024) in inference time computing, aimed at efficiently utilizing more tokens resolve challenging tasks.
Compared to systems that also involve a meta-agent for task decomposition and allocation, we can observe from the table that AOP achieves at least 4.1% improvement in terms of accuracy while maintaining the same level of computation costs and inference times. The results Meta-Agent and Meta-Agent: Iteration indicate that simply instructing GPT-4o perform task decomposition and allocation does not always yield satisfactory responses, even though the capabilities of LLMs recognized powerful. Existing studies in task decomposition and allocation fail consider the abilities and characteristics of the agents beyond their descriptions, and lack mechanisms for modifications and feedback within the multi-agent system, which can be hindered by the challenges as summarized in Section AOP is built upon these identified design principles and incorporates novel mechanisms for automatically evaluating intermediate results, allowing timely modifications and revisions, which guide and leverage the strengths of agents in the system effectively resolve user queries and enhance the overall system performance.
5.3 FURTHER DISCUSSIONS
Ablation Study We conducted ablation study to confirm the contributions of different compo- nents in AOP. Specifically, we disable the detector, the reward model, and the representative works of agents in separate experiments, with results reported in Table From these results,

---

Published as a conference paper at ICLR 2025
Table 2: Experimental results of ablation study.
Method
Accuracy (%)
Prompt Tokens (M)
Completion Tokens (M)
Time (8)
AOP
43.7
1.12
0.38
11,869
w/o Plan Detector
36.6
1,05
0.29
9,504
w/o Reward Model
38.7
1.17
0.37
11,651
w/o Representative Works
41.1
1.14
0.36
11,178
Table 3: Experimental results on the impact of scorer and reward model
Method
Accuracy (%)
Prompt Tokens (M)
Completion Tokens (M)
Time (s)
AOP
43.7
1.12
0.38
11,669
Manual Scoring
46.9
1,26
0.40
12,676
+ Full Parameter Tuning
47.9
1.28
0.41
13,063
observe that the detector has significant impact on exccution accuracy, as indicated by a notable increase in the incompleteness rate during the meta-agent's task decomposition phase. Both the ward model and the representative works are necessary for ensuring the solvability of sub-tasks and for selecting the most suitable agents for each sub-task. Overall, these results demonstrate that all three components are indispensable, working together to ensure the feasibility of task decomposition and allocation, leading to satisfactory responses to the original user queries.
Impact of the Scorer and Reward Model The training dataset is constructed based annota- tions provided by the scorer, which in previous experiments was implemented using LLMs. this section, we shift to using human annotators to manually provide scores for the responses, evaluating whether a reward model trained on this manually labeled dataset would further enhance perfor- mance. This approach is denoted as Manual Scoring. More detailed information about the manual scoring process can be found in Appendix C Besides, based on the manually scored training dataset, we investigate the setting of updating the embedding layers of the reward model, denoted Full Parameter Tuning.
The experimental results shown in Table 3 indicate that both manual scoring and full parameter tuning lead to improved performance. These results suggest the potential for further improving AOP by providing high-quality datasets for robust training- Although using guman-expert-bassee scorer can lead to further improvements, we predominantly adopt model-based scorer in the above experiments, as it more cost-effective and generalizable. Besides, we provide empirical studies on the generalization capability of the reward model in Appendix D.2] showing that the reward model trained on one dataset demonstrates good generalization to other datasets.
CONCLUSION
In this study, propose AOP, novel agent-oriented planning framework for multi-agent systems, following three critical design principles to ensure that the meta-agent can effectively decompose the user query into several sub-tasks for producing satisfactory responses. AOP utilizes a fast de- composition and allocation process, which relies on the ability of LLMs to generate an intermediate schedule efficiently. After that, reward model and the representative works mechanism are em- ployed to evaluate these intermediate results, resulting in executing the sub-task making necessary modifications to align the sub-task with agents, such as replan the sub-task, plan in detail, and re- describe. Extensive experiments demonstrate that AOP achieves significant improvements over both the existing single-agent and multi- baseline methods. We provide discussions on the contribu- tions of different components in AOP and the potential for improvements in agent-oriented planning. We hope AOP can contribute to advancing research in this field, inspiring further exploration and innovation in practical applications.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is funded by National Natural Science Foundation of China Grant No. 72371217, the Guangzhou Industrial Informatic and Intelligence Key Laboratory No. 2024A03J0628,, the Nansha Key Area Science and Technology Project No. 2023ZD003, and Project No. 2021JC02X191, and the Fundamental Research Funds for the Central Universities Grant No. 2242024k30059.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv: 087774, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in Neural Information Processing Systems, 13:1877-1901, 2020.
Yuzhe Cai, Shaoguang Mao, Wenshan Wu, Zehua Wang, Yaobo Liang, Tao Ge, Chenfei Wu, Wang You, Ting Song, Yan Xia, et al. Low-code LLM: Visual programming over LLMs. arXiv preprint arXiv:2304.08800 2023.
Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Brje Karlsson, Jie Fu, and Yemin Shi. AutoAgents: A framework for automatic agent generation. In Proceedings of the Thirty- Third International Joint Conference Artificial Intelligence, pp. 22-30, 2024.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. PaI.M: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via ChatGPT. ACM Transactions on Software Engineering and Methodology, 33(7):1-38, 2024.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gard- ner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 2368-2377 2019.
James Ferguson, Matt Gardner, Hannaneh Hajishirzi, Tushar Khot, and Pradeep Dasigi. IRRC: A dataset of incomplete information reading comprehension questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp. 147, 2020.
Dawei Gao, Zitao Li, Xuchen Pan, Weirui Kuang, Zhijian Ma, Bingchen Qian, Fei Wei, Wenhao Zhang, Yuexiang Xie, Daoyuan Chen, Liuyi Yao. Hongyi Peng, Zeyu Zhang, Lin Zhu, Chen Cheng, Hongzhu Shi, Yaliang Li, Bolin Ding, and Jingren Zhou. AgentScope: flexible yet robust multi-agent platform. arXiv preprint 2402 14034, 2024.
He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Deng, and Mari Ostendorf. Deep reinforcement learning with natural language action space. In Proceedings the 54th Annual Meeting of the Association for Computational Linguistics, pp. 1621-1630, 2016.
Dan Hendryeks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, al. MetaGPT: Meta programming for multi- agent collaborative framework. In The Twelfth International Conference on Learning Represen- fations, 2024.
11

---

Published as a conference paper at ICLR 2025
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of LLM agents: A survey. arXiv preprint Xxt1::2422.0271... 2024.
Joongwon Kim, Bhargavi Paranjape, Tushar Khot, and Hannaneh Hajishirzi. Husky: A unified, open-source language agent for multi-step reasoning. arXiv preprini arXiv:24006 06469, 2024.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pp. 2219-22211, 2022.
Daniel L Kovacs et al. A multi-agent extension of PDDL3.1, In ICAPS 2012 Proceedings of the 3rd Workshop on the International Planning Competition, pp. 19-37, 2012.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman
Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. BLOOM: A 176b- parameter open-accest multilingual language model. arXiv preprint arXiv:221 2023.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem.
CAMEL: Communicative agents for "mind" exploration of large language model society. In Advances in Neural Information Processing Systems, volume 36, pp. 51991-52008, 2023.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic LLM-pooered agent network for task-oriented agent collaboration. In First Conference on Language Modeling, 2024.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language mod- els. Advances in Neural Information Processing Systems, 36:43447-43478, 2023.
OpenAl. Learning reason with LLMs, 2024. Accessed: 12 September 2024.
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang. Weize Chen, Yusheng Su, Xin Cong. Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Com- municative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp. 15174-15188 2024.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. TooILLM: Facilitating large language models to master 16000+ real-world APIs. In The Twelfth International Conference Learning Representations, 2024.
Greg Serapio-Garch Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. Personality traits in large language mod- cls. arXiv preprint arXiv:2307.00184. 2023.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hug- gingGPT: Solving AI tasks with ChatGPT and its friends in hugging face. In Advances Neural Information Processing Systems, volume 36, pp. 1811144-88880 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 1866):186345, 2024.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems, volume 33, pp. 3776-5788, 2020.
Deep self-
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi. Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of of-thuught prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pp. 24824 2483-24877728 2022.
Feijie Wu, Zitao Li, Fei Wei, Yaliang Li, Bolin Ding, and Jing Gao. Talk right specialists: Routing and planning in multi-agent system for question answering. arXiv preprini arXiv:2501.07815 2025.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversations. First Conference Language Modeling. 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.0061, 2024.
Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. Keep CALM and ex-
plore: Language models for action generation in text- games. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Prucessing, pp. 8736-8754, 2020.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
ReAct: Synergizing reasoning and acting in language models. International Conference Learn- ing Representations, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. OPT: Open pre-trained transformer language models. arXiv preprint mmXiv:2055.01066 2022.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223 2024.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learn- ing Representations, 2023.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, al. Deepseek-coder-v22 Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.1199 2024.
13

---

Published as a conference paper at ICLR 2025
A PROMPTS USED IN AOP
A.1 META-AGENT
The one-shot prompt for meta-agent's fast decomposition and allocation is shown Figure
meLa_agent__prong '..
You are a planning agent. You are responsible for dezempesing the given query into sub-tasks and choosing most scitable agent for each sub-task. Your main goal efficiently and accurstely corplete task planring based en the descriptions of agents provi ded, ensuring the coherence and quality the sub-tasks. output
corresponding agerta the fowiig JSON format: [(*task"; task dask_description "id"; task_id, "nane"; name_of_agent, *reason"; cour_detailed ressos_____o_tn_teteeeee cheicee, "dep" :depeadency taskecyyttts__sa_eeeeeaeee is the description of the sub-task, which will be used as the input of the chosen agent, "dep" denates the the ourrent sub-task.
"Lask"
previous sub-task
generates
resource ralied by
The available agents and the corresponding descriptions are: [sode_agent: Generate code in Python precise computat
given math_agents questione by reasening step by step, seacch_ageal: Call Ding Search API to obtain informetion regarding given task. commnnense_agentt cormansense reasoning.).
given question using
Here is example:
Jsez query: If plane can cerry 300 passengers and decides to fly from China to Indonesia, Indonesia
many full flights needed transport 11& population
output:
"Lask"; "Determine the population China.", "id": 1,
"name"; "seazch_ageal",
"reason": "The sesrch_agent can find the most recent and accurate population data for China.",
"dep": []
"cask": "Crlculate 18 the population of China.",
"name": "meth__gent",
"reason": math_agent can reason through caleulation step-by-step."t "dep": [1]
"cask": "Datermine the number of full flights needed to transport 18 of the papulation Indonesia, given that plane can carry passengers.",
"id": 3,
"name": "msth_agent",
"resson": "The math_ágent reason through the division and rouading process.", "dep": [2]
Given the user query, output the task plan with the format above directly. Kake sure all the imporcant informat avea as noura or numbers are included the sub-tasks. te you think the quecy be done with just one agent, you output only sub tack.
Figure 3: One-shot prompt for the meta-agent
A.2 REPLAN
The replan prompt is shown in Figure
14

---

Published as a conference paper at ICLR 2025
are a planning agent. are preliminarily responsible decomposing the given query into sub-tasks and choose the moat suitable agent for each aub-task according to the following jsen format: [("Lask"; Lask_description 'id"; task_id, "name"; hame_of_agent "dep" deppendeeey_ttstk.cooo format, "task" is the description of the sub-task, which will be used as the input of the chosen agent; "dep" derotaa the id of the previous sub-task which generates resource relied by the current sub-taak. available agents and
"reasen":
In this
corresponding descriptions are: [pode_agent: Cenerate Python for precise compitations questions by reasening step by step, search Call Bing Search API for cataining information regarding the given task. commonsense_agentt commonsense reasoning.].
solve the given task, math_agentss Answer math
the given question using
query: 4s, prel ininary task decomposition is:
But the sub Lask: à cannot be solved by any agent. Now you ace responsible for replaning this sub-task based on agentaf capahilities, Output the new ssb-task direc:ly.
format abova
Figure 4: Replan prompt.
A.3 PLAN IN DETAIL.
The prompt for planning in detail is shown in Figure
glan in_diiinnd_ii__eee prompt
You are a planning ageat. You are preliminerily zesponsible for decomposing the given quezy into sub-taske and choose the moat suitable agent for eaca aub-Lask according to the follswing jsen format: [("task": task_description, "id"; task_id, id, *name"; neme_of_agent, detaalle__resooe for_the_choice,
"reason":
in this the
formot, "task" the description of the sub-task, which will ba used the chosen agent; "dep" denotes the id of the previous aub-task waich generates new rescurce relied by current
The available agents and the corresponding descriptions are: [zode_agent: Generste code in Python for precise computations to solve the given Lask, math Answer math step-by-step. searci_agent:: Call Bing Search API for obtaining reasoning.).
questions regarding the given task. commonsens_aggent Ansxer the given question using information
Given the user query: 48, the preliminaty task decomposition is: $8. solved only this sub-task in detail and chaose the mast suitable agents based on agents' cepabilities. Output the aew sub tasks with the fornet above directly. Make sure that between sub-tasks and the given preliminery task
the sub-tesk: is cannot
agent
you are responsible
planning
there duplicate decomponition,
Figure Prompt for planning in detail.
A.4 RE-DESCRIBE
The prompt for re-describing the sub-task is shown in Figure 6
andescribe_auhteeee www.aattta_a_thaeeeeeeaaa pprtpattpppppp_... *.*
Sewrite the following sentence based on the given example, shile keeping the key informetior. unchanged. Resides, output the rewritten sentence in the form like **erovriteene*..
exemple:
Example sentence:
'Determine the pepulation of the United States in 2022.' Santence to be rewritten: "Assess the population of China 2022."
Rekritten sentence: x nntteraine the population of China in 2022." Output: Lettrmnnnn the populatien of china in 2022.'...
Example sentence: ts Sentence to zewritten:
Figure 6: Prompt for re-describing the sub-task.
15

---

any redundant part, Published as a conference paper at ICLR 2025
A.5 DETECTOR
The prompt for the detector is shown in Figure]
plan dstector reaponsible for analyzing the completeness and radundancy of the plan. Given the query and the plan formulated to solve the quesy, which involves several sub- sasks, you Detect whether the plan sasisfies the completeneas.... Evaluate whether the set of aubtaska covers all key aspects the original task inclading impoztant numbers and Specifically, check if each important element and pequicement from the original cask is addressed by at least one subtesk. Provide brief explanation if any key information miasing.
folloxing things:
2. ADstect whether the plan sacisfies the nen-redundacey.. vAz Evaluaze whether any two sub- information raquirements 11st and provide suggest for optimizing the plan.
tasks contain
For example:
Task: if a plane cen carry 300 passangers and flies from Brazil to Nigeria with a full load, only 754 capacity filled, many passengers in total cransported between the two countries in ane round trip?
returns
Subtask Determine the number of passengers tronsported
Nigeria one flight
with full load, Dependency: []
Subtask 2: Deternine the number of passengere tzansported from Nigeria to Brazil in one flight
with 754 cepacity filled. Dependency: []
Subtask 3: talculate the total number of passengars transported between Brazil and Nigeria in trip. Deperdency: [1,
Analysa: This plan does not satisfy completeness because the subtask loses the information of passergera'
original task. This pl plan satiafles non- redundancy because each subtask has unique Tocus and there is overlap in the information covered.
plane can carvy
Suggestions: Add the information aubtask 2.
carry 300 passengers' subtask 1 and
If there is no need to medily the plan, just return The plan satisfies completeness and nen- redundancy.".
Figure 7: Prompt for the detector.
A.6 EVALUATION
The prompt for comparing the response with the ground truth shown in Figure
evaluate_Fr evaluate_prampt prompt /**
You are CompareGPT, a msching to verify the correctneas of predictions. Answer with only yes/ no.
You aze given a question, the corresponding ground-tzuth answer and prediction from model. Compare "Ground-treth answer" and "prediction" to determine whether the contain extra information,
prediction correctly ansvera the question. The prediction but a correct predictior. includes the ground-truth answer, You can ansver "yes" if the prediction includes the graund-truth apecific detaila in the graurd-truth anawer that are not ioned in the pt If the prediction states something possibility, Lrest as definitive answer. that the errer within thres decimal places is negligible.
*aa"
any
Question: ka
Greund-truth znswer: 85 [Start of the pradiction)
is
[End prediction]
Figure 8: Prompt for comparing the response with the ground truth.
16

---

Published as a conference paper at ICLR 2025
A.7 AGENTS
The prompts for the code, math, search and commonsense agent are shown Figure 10 separately. Specifically, the commonsense agent generate commonsense information like the metal melting points and the atomic numbers of helium and hydrogen.
12
ande_agent__poone
used : computing Large numbers, fractions decimals. counting averaging long lists of numbers. performing date-related operations, such as counting the number of days between two dates. Write code in Pythan solve givan task with history, Give the in the following form directly.
ara agent.
example:
Task: Calculate the combined population of thina and india in 2022.
Mistory: enswer Determine the population. of Indla in 2022' 1s 1.41/B.
population of china
The onswer
Code:
"ppyhhon
Given populations
aopulattio_ccina..o
10..3 11.412
populattion_indi_22000 = 1.417 10.x3 # 1.417 billion
Calculate combined population
combi httpmmme_pppllatooooooomm
population, oowioonnnnaaaaeeeeeeeeeeeeeaeeee aachiaan2222_ 2022 popelation_tndda_220
Print the result
print (f"The combined populatien China people.*)
2022 ecorbined_populsssienn..
Taak: $8
History:
Code:
..r
zewrite cewrite_oode__ce poodde_gent_eeeeee prompt You are a rewrite agent. Given the input question, the code addressing this questien and the corresponding output, zewrite the output into complete sentence that integrates information from the question and the code output.
Question: 4s
Code output: ka Output: f,r
Figure Prompt for the code agent.
mathLegeet__prmme err are ageat. answer questions by reasoning step by-step with the data provided in the questien and history. Present the answer "ANs" to the subquastion in the box.
using the ansxer \bNx)d[ANS). any
Questions Miatozy: 88
Solutions
Figure 10: Prompt for the math agent.
17

---

Published as a conference paper at ICLR 2025
search_arg
You are a search agent. Write concise, informa:ive Ive Bing Search query for obtaining information regarding given teak.
Herr is an example:
Taak: Deternine the population or China in 2022.
listory: None
Search query: chine population 2022
Taak: 88
kistory: $5
Search
yeu rewrite agent. siven the search question, the response in the wab_pages from the bing saarch api, answer the search question with the information the response in concise words. Remove redundant informati 1a icrelevant to the questioa.
Questions 48
Answes_box: 38 Answer:
Figure 11: Prompt for the search agent.
commonser.o__ggen http:/ss__geet__rrrerree = You are commonsense agent. You can answer the given questior logical reasening,
knowl knowledge,
Question:
#istory: 85
Solutions r.*
Figure 12: Prompt for the commonsense agent.
B FAULT TOLERANCE: UNEXPECTED CHANGES AGENT AVAILABILITY
We categorize changes in agent availability into two situations: those occurring during non- execution periods and those during execution.
During non-execution periods: AOP allows for the addition removal of agents. The meta-agent can first broadeast a simple sync signal to determine agent availability. Only available agents provided the meta-agent for task allocation.
During execution: Changes in agent availability during execution indicate that an agent selected for a task may unexpectedly become unavailable, leading the meta-agent not receiving response to this sub-task. In such scenarios, the meta-agent is required to reas- sign the sub-task another agent with similar capabilities further decompose the task (i.e., plan-in-detail.
C CONSTRUCTING TRAINING DATASET
The prompt for the scorer is shown in Figure I3 The criteria for manual scoring are consistent with the standards outlined in the prompt provided to the scorer.
18

---

Published as a conference paper at ICLR 2025
soorer_
act impartial judge and evaluate quality to the usar task. Your evaluation should conat der three factors: correctness, relevance and completeness. Assign scuze of 0, or 2 for each fector and provide brlel explanation for your
response provided by the
The following is the grading criteria.
Correctneas
3: The reagonse contains severe errors and Ls completely inaccurate.
content generally correct 2: The reaponse 1a completely accurate and fully meeta the requirements of the task. Relevance
respense has same errors, but
3:
minimally relevant the task and completely off-topic, 1: The respense is somewhat relevant to the tesk but may include acme unrelated content.
reapanse highly relevant the uncelated content or deviation. Completeress
df addresaing
lasue without any
3: The reagonse Lacks necessary detail key information, resulting in Incomplete
understanding of the task.
1: while the response addresses part of the task, information concent is needed for completenass.
reaponse provides comprehensive information detailed explanations.
Realdes, summarize the rinal reault the form like CoCorrectteas:: score, Relevance: score, Compleleneas: scoreva' at the end of your respoase, where score be chosen from 0, and
Taak
[The Start of Agent's s Response]
[The End of Agent's Respense]
Figure 13: Prompt for the scorer.
We design mapping function that converts different combinations of correctness, relevance, and completeness values into final score, which is shown in Figure 14 Then, score of 5 or higher is considered sufficiently high, while score of lower is considered sufficiently low.
(2,
2, 21: B
(2, 21:
(2, 11: 6
(2, 11:
5,
(1,
1,
21:
3,
(1,
11:
(1,
21:
def soee_ssscreecrrrctteeee relevance, completeness):
return score_mep.get (trr:p.eet//correeeeee (ccorrectness, relevance, completeness),
Figure 14: Mapping function.
D EXPERIMENTS ON MORE DATASETS
D. PERFORMANCE ON DROP AND IIRC
In addition to Husky-QA Kim et al. 2024). also conduct experiments DROP Duu et al. 2019) and IIRC (Ferguson et al.) (020). The experimental results are shown in Table which in- dicate that AOP significanlly outperforms the baselines, achieving least a 5.0% improvement both DROP and IIRC. These experimental results further confirm the effectiveness and advance-
ments of AOP.
19

---

Published as a conference paper at ICLR 2025
Table Accuracy (%) comparisons between AOP and baselines on DROP and IIRC.
Method
DROP
IRCC
GPT-4o
23.0
33.0
CoTT
26.0
35.0
Zero-Shot CoT
24.5
33.0
Meta-Agent
25.5
32.5
Meta-Agent: Traversal
27.5
34.0
REACT
29.0
36.0
HUSKY
28.0
36.5
AOP
34.0
41.5
AOP (reward model trained on Husky-QA.
32.0
39.0
D.2 GENERALIZATION CAPABILITY OF THE REWARD MODEL
To evaluate the generalization capability of the reward model, we train a reward model based on Husky-QA and utilize experiments the DROP and IIRC datasets. As shown the bottom in Table the reward model trained on one dataset demonstrates good generalization to other datasets (though it experiences a slight performance drop compared the specifically trained reward model), achieving superior performance compared to baselines.
E
FURTHER INVESTIGATION OF EXPERT AGENTS
E. EXPERT AGENTS
To investigate the effectiveness of expert agents powered by LLMs fine-tuned task-specific datasets, we adopted Qwen2-Math-7B (Yang et al. 2024) as the backbone LLM for the math agent and DeepSeek-Codee Coder- V2 (Zhu et al. 2024) as the backbone LLM for the code agent. The experi- mental results show that integrating these expert agents results in an additional 1.8% improvement Husky-QA AA compared with using GPT-4. These results demonstrate the potential for further hancements of AOP by incorporating more powerful expert agents.
E.2 MULTIPLE AGENTS WITH SAME EXPERTISE
Based on AOP, we set an experiment involving 4 different math agents, using GPT-3.5, GPT-40, Qwen2-Math-7B, and Llama-3.2-3B, respectively. We instruct the multi-agent system solve the complex math problem from MATH (Hendrycks et al. 202T). With AOP, the queries are decom- posed into multiple queries and these agents run parallel to resolve them. The experimental results are shown in Table 5 demonstrating the effectiveness (at least 6% improvements) of AOP.
Table 5: Accuracy (%) comparisons MATH.
GPT-3.5
GPT-4o
Qwen2-Math-7B Instrruc
Llama-3.22 3.2-3B
AOP
MATH
43
62
66
36
72
F ADDITIONAL EXPERIMENTAL RESULTS AND DISCUSSIONS
F.1 QUANTITATIVE EVALUATION ON SOLVABILITY, COMPLETENESS, AND REDUNDANCY
We conduct quantitative evaluation to demonstrate the effectiveness of AOP in terms of solvability, completeness, and men-rndund.ncy. Specifically, we compare the decomposed sub-tasks provided by AOP to those provided by GPT-4o. These sub-tasks assessed by an L.LM- evaluator,
20

---

Published as a conference paper at ICLR 2025
providing binary scores for solvability, completeness, and non-redundancy, with scores of 1 for those that meet these principles and for not. The experimental results (averaged scores), as summarized in Table 6 show that AOP achieves significant improvements.
Table 6: Quantitative Evaluation on Solvability, Completeness and Non-redundancy.
Solvability
Completeness
Non-redundancy
GPT-4o
0.763
0.986 0.822 0.969
AOP
0.938
0..93
F.2 EFFECT OF DIFFERENT REPRESENTATION APPROACHES FOR AGENT DESCRIPTION
In this study, we employ combination of predefined natural language descriptions and represen- tative works the representations of agents descriptions. A natural language description can be manually provided or automatically generated, detailing the general and task iass-indeeendent abilities of agents. While providing comprehensive and detailed natural language description eficial, it also requires effective prompt engineering. The representative (please refer to Sec. 4.3 for more details) works complement the natural language descriptions and are often task-dependent, allowing for continuous updates during execution.
be ben-
We conducted experiments on Husky-QA compare the effectiveness of different representation approaches. The results, shown in Table 7 indicate that using natural language descriptions achieves significantly superior performance compared
using only representative works, which motivates the majority of existing studies to adopt natural language descriptions. Besides, incorporating task- specific representative works on top of natural language descriptions leads to further 1,9% perfor- mance boost, demonstrating the effectiveness of our proposed combined representation approach.
Table 7: Comparisons among Natural Language Descriptions and Representative Works.
Approaches
Accuracy (%)
Natural Language Descriptions
31.8
Representative Works Both
16.8
43.7
Besides, in this study, employ the natural language descriptions adapted from Husky Kim et al. (2024) to ensure fair comparisons. To further investigate the effect of natural language descriptions, provide two different versions: (i) LLM-ggenrrted We instruct the LLM to generate the natural language descriptions based on how we describe the agents in Section 5. (ii) Expert-written: The shown in Table 8 From the table, we can observe that using simple, fully automated natural language descriptions may somewhat affect the overall performance. Besides, incorporating more expert sights into the generation of natural language descriptions can provide an additional performance boost. These experimental results align with the current intuitive understanding of prompt engineer- ing.
natural language descriptions crafted by human expert. The comparisons on Husky-QA
Table 8: Comparisons among Different Natural Language Descriptions.
Accuracy (%)
Descriptions adapted from Husky
43.7
LLM--gnnrated descriptions
40.4
Expert-written descriptions
44.2
21

---

Published as a conference paper at ICLR 2025
F.3 THE EFFECTIVENESS OF DETECTOR HANDLING MISSING DEPENDENCIES
We conduct an experiment to evaluate the effectiveness of the detector in handling missing depen- dencies. To be more specifie, the detector is instructed determine whether there are any missing dependencies before a sub-task is assigned to an agent for execution. such dependencies exist, the execution results of these dependent sub-tasks must also be provided inputs. The adopted prompt is provided in Figure 15] If there exists any missing information, the meta-agent will add sub-tasks complete the dependencies. The experiments conducted on Husky-QA. show that, by enabling the detector to handle missing dependencies, AOP obtains 2.5% performance increase in accuracy (from 43.7% to 46.2%).
F.4 THE LIMITATIONS ON THE NUMBER OF AGENTS
AOP has inherent limitations the number of agents. However, the meta-agent is powered by LLMs, increasing the number of agents is subject to the contextual window length restrictions of LLMs (descriptions of large number of agents could exceed the LLM's output length limit, such as 128K tokens), and the effectiveness of LLMs could be affected by the ability to handle long contexts.
In practical applications, it is important to consider that as the number of agents increases, there may be redundant and functionally similar agents, which motivates us to apply some extension strategies to handle. For example, agents can be grouped according to their abilities. When the meta-agent allocates sub-tasks, can first select an agent group for each sub-task and then further choose the most suitable agent within the group. This strategy respects the LLM's context window limits and enhances the accuracy agent selection.
22

---

Published as a conference paper at ICLR 2025
deee_deeec___ee promptt !.r
You are intelligent detecter tasked with determining whether the provided dependency dependency informati
information sufficiert complete a geven task, insufficient, you will review all historical date find supplements emental information. if neither the dependency information nar the historical data is sufficient, you will identify
the missing information. Besidea, information in task given quezy can be viewed as available in the dependency information.
Inpet:
Taak: (Task deacription)
Dependeney informaiin: (Dependency information provided) ststorical hata: [Ristorical data (listed with serial rumbers) Available query: query information information.?
available inthe dependency
Requi rements:
1. Assess Dependency Infermation: Evaluate if the provided dependency information sufficient to complete the taak. If sufficient, ***, Besides, it the answer is **sYears,, there 1a no need to check the following requirements.
if ***Yos***;
***No
2. Review Historicel Data: If the ansver above is sasNoves, check the historical data to requirements. List serial
identify any relevant entries that con supplement
numbers of the relevant entries
""numbera""".
Identity Missing Information: If the historical data also cannot aupplement the miseing detaf axplicitly Hat information is still requi complate the task stSzequiced additional inCormation:555.
Output zormat (strictly follow this structure):
Ia dependancy information aufficient: "*""o=**</<<<oo....
2. Relevant information from historical data: "" """rumoers""" or """None".
3. Missing information: $88Specific misting information$$5 or $39NeneS85
Examples:
Input:
Taak: Caleul ate the total population Chinaa the Unittdd States 2022.
Dependency Infornation:
China's population 2022 is 1.4 billion.
#istoricsl Data:
1. China's pepulation in 2022 is 1.4 billion. United
populatien
million.
3. Total world population in 2022: billion.
Available query: If the populations of China and India were combined in 2022, how many
countries with population of 70,850, 000 each could be formed from this total population without leeving anyone out?
Output:
1. Is the dependency infermation sufficient: NesNosAs
Relevent information historical
"**2"""
3. Missing information: $8SNone8$5
Input:
Task: Calculate the totel populatien of China and the United States in 2022.
Dependency Information:
China's populetion in 2022 is 1.4 billion.
Ristorical
1. China'a population in 2022 1a 1.4 billion.
2. Total world population in 2022: billion. Available query: If the populations
China and India combined in 2022, how many
countries with a population of 70, 850, 000 ezch could be formed from this total population without leaving anyone out?
Output:
Ta dependency information auffisient: **.No..*
2. Relevant information from historical data: """None""
3. Missing information: 358The United States pepulation 2022848
Input::
2apendancy Information:
ts
Mistorical Data:
is
Available query: 4s
output:
Figure 15: Prompt for the detector in handling missing dependencics.
23


Reviews:
[{"review_id": "KjakhOV4AW", "reviewer": "Reviewer_4UEG", "summary": "The paper describes a method for solving queries using a multi-agent system, with multiple agents which are experts in at different sub-tasks. A meta-agent is then required to decompose the queries into multiple sub-tasks that are then allocated to suitable agents for solving them. The authors identified three critical principles for task completion:\n\nThat is Solvability - Each sub-task should be solvable by at least one of the available agents. \n\nCompleteness - The set of sub-tasks should include all necessary information from the original query. That is the aggregation of the sub-tasks should have a comprehensive answer to the original query.\n\nNon-redundancy – The set of sub-tasks should not include duplicate or unnecessary elements.\n\nFor each of these requirements, should it fail any, the meta-agent is required to revisit the task decomposition and/or task allocation.\n\nTo determine if a sub-task is completely resolved you would need to check if any of the agents can solve a sub-task, but due to the overhead required, they create a training dataset for a reward model using the fast decomposition process and a scorer, which evaluates agent response. This reward model is used to reduce the overhead of agent calls. It is used to approximate whether a sub-task is resolved by an agent by check if its score is sufficiently high (above a certain threshold). In cases where this score is not high enough for any agent, the sub-task is deemed to not fit the solvability criteria and the meta-agent replans.\n\nThey then explore when sub-tasks are lacking information/ambiguous or are too complex, and use a similarity calculation over the embeddings of the subtasks. If it is too high, there are tasks similar to it and it should be re-described according to the similar representative  sub-task. If it does not meet a threshold then the sub-task is considered too complex and is modified.\nA detector is added to check that all key elements of the original query exist in all the sub-tasks, and that no two sub-tasks resolve the same key elements. If either is failed a recommendation based on that key element is made to help remove overlapping subtasks or supplement missing details.\nThis is evaluated vs a number of baselines and experiments are done based on a numerical reasoning dataset. It is shown to outperform the baselines.\n\nAblations are done to show that each component adds meaningfully to the model.\n\nThe related work is only listed after all the results.", "questions": "The results only have one dataset to my understanding. Can this be extended to more datasets?\n\nI have concerns that the rewards model does not generalize outside of this dataset? Or would that simply require more training on bigger datasets?\n\nThe ordering of the sections, particularly the related work, and how it reads in context at the end, does not make sense, was this just compiled in the wrong place accidentally?\n\nAre there cases where the scorers are human experts? It was a bit ambiguous as to whether this was in fact the case.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The work shows three components which improve the ability of multi-agent LLM systems to improve the results of general LLM queries as compared to current methods.\n\nAll three components are shown to meaningfully add to the overall model, and directly tackle the presented critical principles for design.", "weaknesses": "The work is difficult to follow at times, I believe it can be made much clearer, especially with regards to Figure 2. It is difficult to understand this diagram without enough direct context. There is no clear direction that it takes. The detector does not seem to lead to a replan as described.\n\nThe ordering of the sections, particularly the related work, and how it reads in context at the end, does not make sense, was this just compiled in the wrong place accidentally?\n\nThe results only have one dataset to my understanding. Can this be extended to more datasets?\n\nI have concerns that the rewards model does not generalize outside of this dataset? Or would that simply require more training on bigger datasets?\n\nI believe this work is good and novel, however it could be improved upon with clearer writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lVrMW7S4uj", "reviewer": "Reviewer_LMUm", "summary": "The proposed system uses a LLM to generate a decomposition of a problem into an array of sub tasks that are then allocated to different agents. Then a reward model is used to improve the plan as it is being executed.", "questions": "* Why is it beneficial to use natural language over PDDL and other languages specifically designed for planning? There are papers on this already [2] so it would make sense to compare to such an approach.\n* Similarly, why not represent the agent capabilities in terms of planning operators or some other well-defined formalism rather than (what I presume) is natural language descriptions?\n* How do you formally verify that the generated plans are correct and solve the problem?\n\n[2] Translating Natural Language to Planning Goals with Large-Language Models by Yaqi Xie et al.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The problem of multi-agent planning is highly important with high potential impact.\n* How to decompose problems into sub-tasks based on the capabilities of the individual agents is often the key problem.\n* Learning the capabilities of other agents is a hard and important challenge.\n* The paper is easy to follow.", "weaknesses": "* The paper does not relate to the rich and long history of multi-agent planning. A starting point could be [1].\n* The design principles seem a bit backwards. Normally, the set of problems that can be solved by a set of agents is defined as the union of the problems that each agent can solve, then you have to check whether the particular problem is within that set. Normally, it is a hard computational problem to determine what problems an agent can solve. If the set of sub-tasks contain redundancy it is either a problem with the planner or there is reason, such as a need to deal with non-deterministic outcomes of actions. To have it as a design principle seems strange.\n* In the completeness principle it is mentioned \"The array of sub-tasks [...] should include all necessary information\" what does this mean? The first principled is defined in terms \"resolvable\" tasks. The second principle is defined in terms of \"necessary information\". What is the relation between \"necessary information\" and \"resolvable\"?\n* It is well known that LLMs cannot generate plans in the sense used in the planning community as there are no guarantees that the plan is correct nor that it actually solves the problem. My impression is thus that the proposed solution is more about learning a model of different agents capabilities than on planning. \n\n\n[1] Cooperative Multi-Agent Planning: A Survey. Alejandro Torreño, Eva Onaindia, Antonín Komenda, Michal Štolba. ACM Computing Surveys (CSUR), Volume 50, Issue 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZMzaPQ8kjZ", "reviewer": "Reviewer_vz1o", "summary": "The paper presents a multi-agent framework for problem-solving. In particular, the framework uses a meta-agent to decompose a task into subtasks and assign each subtask to downstream expert agents based on their expertise. Then, it uses a reward model to evaluate the performance of the expert agents and use the rewards as signals for re-planning. Additionally, the framework incorporates a feedback system to further improve the robustness and accuracy of problem-solving. Empirical analysis over a numerical reasoning dataset indicates how the proposed framework outperforms the benchmarks, such as direct query to GPT-4o and chain-of-thought. Additionally, a set of ablation studies necessitates each component, such as the plan detector and the reward model, within the framework.", "questions": "1. In Table 1, in my understanding, the prompt tokens and completion tokens refer to the AVERAGE NUMBER of tokens for EACH task, is it correct? Probably it's better to clarify this in the paper.\n\n2. Is this framework capable of collaborating with multiple agents with the same expertise and improving efficiency? For example, using multiple math agents to solve a complex math problem, while these agents run in parallel.\n\n3. In my understanding, the four agents in the experiment are GPT-4o with different input prompts, are there any additional fine-tuning to improve the expertise of each agent?\n\n4. The paper talks about solvability, completeness, and non-redundancy at the beginning of the paper. Are there any quantitative results showing that the proposed framework addressed these challenges?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The paper is well-organized, presenting a detailed and thorough explanation of the proposed framework. A particularly compelling aspect is the approach of breaking down tasks into smaller components and assigning them to specialized agents according to their areas of expertise. This decomposition allows each agent to handle tasks within its domain, enhancing overall efficiency and precision. Additionally, the integration of a feedback system significantly boosts the framework’s performance by refining the agents’ actions and ensuring adaptive learning.", "weaknesses": "There are several limitations of the work that the authors may consider to address:\n\n1. Expert agents: the difference between each expert agent is their input prompts, but they are using the same underlying model (GPT-4o). It could be more interesting to replace each expert agent with the current state-of-the-art model in their domain. I believe there are plenty of works to fine-tune the language models for code generation, solving math problems, etc.\n\n2. I don't see the significance of the commonsense agent. Feels like the meta agent is also doing some common sense reasoning (e.g., understanding and decomposing tasks). If possible, please provide some use cases where the commonsense agent is necessary.\n\n3. The cost (time and number of tokens) is high compared to direct querying GPT-4o, given that over 5x cost for only a 10 percent improvement. This probably can be solved by optimizing the expert agents (see point 1).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6wxHOz5xc7", "reviewer": "Reviewer_8PGi", "summary": "This paper introduces a framework for agent-oriented planning in multi-agent systems, where a central \"meta-agent\" decomposes complex user queries into sub-tasks, assigns these sub-tasks to specific agents, and evaluates their performance. The authors focus on three key principles, namely, solvability, completeness, and non-redundancy, to guide efficient task decomposition and assignment. The framework also includes a feedback loop, enabling the meta-agent to refine its planning over time.\n\nThe framework’s emphasis on structured decomposition and task dependencies is well-suited for complex queries that involve multiple agents with diverse capabilities. By ensuring each sub-task aligns with an agent’s expertise, the system promotes both efficiency and reliability.\n\nI have a few questions:\n- Task dependencies are identified during decomposition, but dependencies may also emerge during execution. How does the framework adapt when unforeseen dependencies between sub-tasks arise? Could this affect the overall solvability or efficiency of the task set?\n\n- The framework depends on structured task decomposition, yet real-world environments are often unpredictable and dynamic. How does this approach handle unexpected changes in agent availability or evolving task requirements without needing a complete re-plan?\n\n- Non-redundancy is a key focus, yet redundancy can be valuable for fault tolerance. How does the framework balance efficiency with the potential need for redundancy, particularly in scenarios where backup solutions could be essential?\n\n- The scalability of the approach isn’t clear to me. As the number of agents and interdependent tasks increase, how does the meta-agent manage complexity? Are there limitations on the scale or number of agents?\n\n- The meta-agent requires detailed knowledge of each agent's abilities for effective task allocation. But it isn’t immediately clear to me, how are these capabilities/descriptions in D represented? Can you provide more details?\n\n- Related to the comment above, can you discuss what representation is best suited for agent descriptions? Are these representations task dependent?\n\n- Another follow-up on the above comment, can you discuss about the process/requirements if these representations need to be verified or even updated through the process? How can the approach be adapted to handle such cases?\n\n- While the feedback loop allows for ongoing improvements, how does it prevent instability or oscillations in planning strategies? How would you introduce safeguards into the system or meta-agent to avoid fluctuating between different planning approaches?\n\nOverall, I'm happy with the contributions made in this paper and vote for a weak accept. I'd be happy to raise my score given mine and my fellow reviewers' comments are adequately addressed. Thanks!", "questions": "-- See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "-- See above", "weaknesses": "-- See above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JSxzy9Hj4s", "reviewer": "Reviewer_AMcZ", "summary": "This paper presents a new framework for multi-agent systems where meta-agents decompose user queries into sub-tasks, allocate these tasks to appropriate agents, and evaluate the results based on three key design principles: solvability, completeness, and non-redundancy. The meta-agent also uses a reward model to assess agent performance and includes a feedback loop for continuous improvement. The proposed framework demonstrates noticeable advancements in solving complex real-world problems by efficiently coordinating agents compared to existing methods.", "questions": "Could you elaborate on how the principles of solvability, completeness, and non-redundancy are uniquely adapted for LLM-based multi-agent systems in this framework?\n\nCould you clarify the unique aspects of your reward model and feedback loop in this framework? How do they differ from traditional surrogate models, and what specific contributions do they offer to agent-oriented planning in LLM-based systems?\n\nWhat strategies or guidelines do you suggest for fine-tuning the system’s parameters for new datasets or applications? Are there ways to simplify or automate this tuning process to improve usability?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper proposes an interesting framework for agent-oriented planning in multi-agent systems, including an integrated feedback loop and reward model, which appear to be novel.\n\nThis paper is well-structured, with a clear methodology and comprehensive experimental analysis.\n\nThis paper is clearly written and well-organized, making the complex concepts of agent-oriented planning accessible.\n\nThis paper addresses a crucial challenge in multi-agent systems—optimizing task decomposition and agent coordination, which has substantial implications for practical applications in artificial intelligence.", "weaknesses": "While this paper explored a very important problem of agent-oriented planning, I do have a few questions/concerns regarding the reported research work:\n\n1. The principles of solvability, completeness and non-redundancy appear to be very general in nature. They may not be entirely new to the field of multi-agent systems or task planning. The novelty may not lie in the principles themselves but in how they are implemented within the specific context of large language models (LLMs) and agent-oriented systems. While the paper integrates these principles into the implementation of a meta-agent, the principles alone may not represent a breakthrough. This paper could benefit from a more detailed discussion of how these principles are uniquely applied in the context of LLM based multi-agent systems. \n\n2. There seems to be a clear lack of technical details regarding how each agent is constructed, operates, and integrates with the meta-agent. While the paper describes agents such as the “search agent,” “math agent,” and “commonsense agent,” I don't quite understand their underlying architectures and decision-making processes (e.g., the underlying models used for each agent and their training processes). Additionally, the descriptions of agent capabilities seem to be vague. This lack of technical depth makes it difficult to assess the robustness, scalability, and adaptability of the agents, as well as their performance in diverse real-world applications. Without these technical details, the framework’s effectiveness remains partially speculative, relying more on theoretical design principles than on concrete, replicable methods.\n\n3. The reward model developed in this paper operates like a surrogate to predict the performance of using any specific task agent to tackle any given sub-task. The general idea of predicting agents' performance is not new. The concrete design of the network architecture and the training algorithm for the reward model may be new. However, the corresponding technical contribution was not clearly highlighted and justified. The same concern also applies to the feedback loop, which is commonly used to update the surrogate models in the literature. Its novelty under the newly proposed framework may need to be better elucidated. Potentially, this paper may benefit by providing a more detailed comparison with existing surrogate models or a clearer explanation of how their approach differs from standard feedback loops in the literature.\n\n4. This paper introduces multiple algorithm/system parameters, such as reward model thresholds, agent selection criteria, and similarity measures, which may be challenging to fine-tune. This complexity makes it potentially difficult to adapt the proposed multi-agent system to new datasets or applications, as optimal tuning may require deep domain-specific knowledge and extensive experimentation, further affecting the system’s usability. In line with this concern, it might be helpful for the authors to provide some concrete guidelines or heuristics for parameter tuning, or to discuss potential strategies for automating or simplifying this process for new datasets or applications.\n\n5. The experimental evaluation in the paper appears to be limited in scope, as it primarily focuses on a single numerical reasoning dataset. It remains questionable whether the newly developed multi-agent system can generalize well across diverse real-world tasks or more complex, domain-specific applications. Hence, the authors may need to discuss potential challenges in applying their system to more diverse or complex tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)