From an LLM Swarm to a PDDL-empowered Hive: Planning Self-executed Instructions in a Multi-modal Jungle
Authors: Kaustubh Vyas
Damien Graux
Yijun Yang
Sebastien Montella
Chenxin Diao
Wendi Zhou
Pavlos Vougiouklis
Ruofei Lai
Yang Ren
Keshuang Li
Jeff Z. Pan
Venue: ICLR 2025 Poster (2025)
Keywords: Deep Models
Planning
PDDL
Knowledge Graphs
Benchmark
Large Language Models
Primary area: applications to robotics, autonomy, planning
TL;DR: Introducing Hive: a powerful, explainable system for selecting models & planning atomic actions based on natural language instructions. Hive leverages PDDL to deliver complex multi-modal tasks while respecting user constraints.

Abstract: In response to the call for agent-based solutions that leverage the ever-increasing capabilities of the deep models' ecosystem, we introduce a comprehensive solution for selecting appropriate models and subsequently planning a set of atomic actions to satisfy the end-users' instructions.

Our system, Hive, operates over sets of models and, upon receiving natural language instructions, schedules and executes, explainable plans of atomic actions. These actions can involve one or more of the available models to achieve the overall task, while respecting end-users specific constraints. Hive is able to plan complex chains of actions while guaranteeing explainability, using an LLM-based formal logic backbone empowered by PDDL operations. We introduce the MuSE benchmark in order to offer a comprehensive evaluation of the multi-modal capabilities of agent systems. Our findings show that our framework redefines the state-of-the-art for task selection, outperforming other competing systems that plan operations across multiple models while offering transparency guarantees while fully adhering to user constraints.

Published as a conference paper at ICLR 2025
FROM AN LLM SWARM TO A PDDL-EMPOWERED HIVE: PLANNING SELF-EXECUTED INSTRUCTIONS IN
A MULTI-MODAA JUNGLE
Kaustubh Vyas!, Damien Graux!, Yijun Langg,/ébastier Montella', Chenxin Diao Wendi Zhou 1, Pavlos Vougiouklis', Ruofei Lai', Yang Ren Keshuang , Jeff Z. Pan i Huawei Technologies Ltd., UK University of Edinburgh, UK (f (firstname. lastname @ com {wendi zhou, J. z....ppa Bed.neeeddcc ac. uk
ABSTRACT
In response to the call for agent-based solutions that leverage the ever-increasing capabilities of the deep models' ecosystem, we introduce HIVE - a comprehen- sive solution for knowledge-aware planning of set of atomic actions to address input queries and subsequently selecting appropriate models accordingly. HIVE operates over sets of models and, upon receiving natural language instructions (i.e. user queries), schedules and executes explainable plans of atomic actions. These actions can involve one or more of the available models achieve the overall task, while respecting end-users specific constraints. Notably, HIVE handles tasks that involve mulli-modal inputs and outputs, enabling handle complex, real- world queries. Our system is capable of planning complex chains of actions while guaranteeing explainability, using an L.LM-based formal logic backbone empow- ered by PDDL operations. We introduce the MUSE benchmark in order to offer a comprehensive evaluation of the multi-modal capabilities of agent systems, Our findings show that our framework redefines the state-of-the-are for task selection, outperforming other competing systems that plan operations across multiple mod- els while offering transparency guarantees while fully adhering to user constraints.
INTRODUCTION
Within the past few years, the number of available models either through commercial paywalls or open-sourced-hoet has exploded both in terms of intrinsic performances and in terms of tasks handled
by them, ranging from text generation (Achiam et al., 2023 Anthropic 2023] Team et al.] [023]
He et al.) 2025) to more specific actions such code generation (Becker et al. 2023 Dong et al, 2024) or image generation (Wang et al., 2023b Zhu et al.) (023). This rapid growth has unlocked unprecedented potential for real-world applications, inspiring practitioners, especially in industry, to envision new use cases that leverage these powerful models (Liu et al.) 2023c [Shen et al. 2202aa)
Lu et al. 2024, Xing al. 20024). However, if creativity and possibility have been unleashed by such a surge, implementing pipelines that involve multiple models remains a complex and largely manual (and often cumbersome) process, particularly when addressing tasks beyond the original design of these models. This often leads developers to create ad hoc modules to manage these complexities. In addition, significant number of models available in the wild are either advanced proof-of-concept or very specialised ones, see e.g. the hundreds of thousands of models available on the HuggingFace platform! As a consequence of this abundance, navigating through this jungle to select the appropriate models for a set of tasks has become very challenging. This complexity arises both in terms of performance and compatibility. Connecting models' input and output formats is complex, the generated results often difficult control (Scholak et al.) 2021 Qin al. 2022). Moreover, planning and chaining tasks for real-world use-cases present a significant challenge too.
In this study, we present comprehensive solution tackle the aforementioned two challenges, i.e. (L) selecting appropriate models and then (II) planning a set of atomic actions to achieve the objectives in the end-users' instructions (i.e., user queries). Our knowledge-awan planning system, HIVE, takes natural language instructions (potentially involving multi-modal inputs and outputs)
Work done while at Huawei Technologies Ltd.
11 016 247 models on httph:: heatps://huggiiii www.:nnggggnaaac.co.c... coteade..coocmmeee of Octoher 1", 2024.

---

Published as a conference paper at ICLR 2025
and can effectively schedule, execute and explain plans composed of atomic actions. These plans may involve one or more models, carefully orchestrated to accomplish the overall task while adher- ing to users-specific constraints, such model size or licensing, to name a few.
One of our key contributions addresses the first challenge: the lack of machine- uutrhiii--undeetdddabbllee in- terface that consolidates comprehensive information about available models. To bridge this gap, we propose a Capability Knowledge Graph (C-KG), a knowledge graph (Pan et al.) (017) with all dimensions needed to for automated planning and execution. For each model, inter alia, C-KG cap- tures critical details such as supported tasks, performance metrics from state-of-the-arr benchmarks, and minimal code snippets for inference. Additionally, sequences with guaranteed explainability, we developed a novel planning approach which, instead of relying solely on L.LM reasoning capabilities (Pan et al. 2023), also employs formal logie to reach its conclusions. To achieve this, we took advantage of PDDL a formal language widely used in robotics for defining planning problems (Aeronautiques et al.) 1998), mapping the end-users' in- structions with atomic actions, thereby enabling the conversion of natural language instructions into a PDDL problem space. This approach allows us to formally plan before executing the tasks using code snippets from the C-KG. As a result, it has enabled us generate detailed reports that provide end-users with fine- graiggrinnd and reliable explanations.
enable the planning of complex action
In the absence of standard publicly available benchmarks for solving real-world tasks, we intro- duce MUSE and share as a Github repository a new evaluation benchmark of complex queries involving multi-modal inputs and outputs, assess our proposed framework. Using MUSE, reviewed the closest existing solutions, namely HuggingGPT (Shen et al. 2024a) and ControlLLM (Liu et al.) 2023c). which only tackle sub-problems of our broader objectives. The results indicate that HIVE not only surpasses these competitors but also consistently outperforms them across all benchmark dimensions. HIVE demonstrates a 30% higher accuracy in task selection and respects user constraints in 100% of cases, while being more reliable.
2 PRELIMINARIES
Planning Domain Definition Language (PDDL) (Aeronautiques et al., 1998) is a standardised lan- guage extensively used in the field of artificial intelligence (AI) planning to represent planning do- mains and problems. PDDL provides a formal syntax and semantics for defining the components of enables the clear
a planning task, including actions, predicates, objects, and their relationships. specification of the initial state, goal conditions, and permissible actions within a domain, facilitating the development and comparison of planning algorithms. In our work, PDDL plays a critical role in task decomposition and planning. By defining tasks as actions within PDDL domains, we leverage established planning techniques to generate coherent and feasible plans. The use PDDL allows us to formally model complex tasks, ensuring that the system can reason about the preconditions and effects of actions within a well-defined framework.
Let D {d1, de...., } be set of PDDL domains, Each PDDL domain dj E IB is associated di a As where j E [1, [D]] and A E N the num-
with set of PDDL actions s.t. ad ber of actions included within the PDDL. domain dj. Furthermore, let T be set of different tasks,
{ d, d
consisting of all PDDL actions across the available PDDL, domains E D, as follows:
U a
j=1
Finally, we define MI {m,, m2,... m)M) as the set of all models available for completing set of different tasks T. or combinations thereof.
3 HIVE - GENNRAL ARCHITECTURE
3.1 CAPABILITY KNOWLEDGE GRAPH
We extract model cards directly from HuggingFace and incorporate OpenlE extraction route for converting the textual descriptions from each model card into a structured representation. We align
htt.ass//athbb...oo/rrrr////teee https://huggingff http:///nwnccccococ/////////////
2

---

Published as a conference paper at ICLR 2025
Offline
User
Task Planner
Model
Query "*t Tr pc
PDDL Domain
Action
Rephrasinn
Classification
Selection
Merper
PapersWithCade
Plan Executiener
Preferences
Model Selection
G-K
Code Snippet
Extraction
2.1
lageni
Code Snippets
Response
-ul Tr
Argument Magping
Cade Execusion
405
Figure 1: HIVE modular architecture.
the models with Papers With Code which enables to collect information about how particular model performs across different benchmarks. Our goal is to build a Capability Knowledge Graph, denoted as C-KG, which a graph G (M, T, E), such that each model m, E M associated with one or more tasks from T along with its relevant performance. and different types of edges E E with various properties (e.g., number of parameters or supported languages). We search each model card collected by HuggingFace for potential arXiv paper ID. We make use of the arXiv paper ID in order to bridge models from HuggingFace with their corresponding performances in different tasks and benchmarks. When identify an arXiv paper ID within the knowledge base of Papers With Code, we extract all model versions and their performances across the different benchmarks.
Each different model version is represented as separate vertex within the C-KG. A model vertex my E C-KG is aligned with its model card if the model's ID can be matched (e.g., flan-t5-ba se in both the original model card and the ID of at least one of the retrieved records from Paper With Code. This enables us to build knowledge graph in which different models associated with different benchmarks and tasks. Apart from the relevant performance scores in the various benchmarks and the models' properties extracted from HuggingFace, model cards coding snippets that describe how each model can be loaded and executed. These ex- ecution snippets enable on-demand loading of chosen model and execution of its inference step based on the provided parameters, if selected by model selection pipelines (see Section 3.2.2)
extract from the
When processing the HuggingFace model card, we leverage combination of keywords (see Ap- pendix (D and regular expressions to identify coding blocks that showcase simplified examples of loading and executing a particular model my. After the coding blocks are extracted, we prompt (see Appendix D) an LLM that is proficient in code generation, such DeepScek-Code Cooeee] generate a suitable Python function invoking m, and running inference while taking into consideration input arguments of the originally extracted coding block. The resulting Python function, together with its signature (returned type, variable types and default values). is finally stored within the C-KG and connected with its corresponding model using an execut i on E. edge.
3.2 PLANNING MODEL ACTIONS
Having extracted and systematically structured the information pertinent to models associated with various tasks, we are now positioned to delineate the specific actions required to accomplish the objectives in the user query, depicted in Figure
https::/ppperrwiitt
comm
https://arxi/ org/
arriv papers may report results of different model variants, e.g. performance across different models sizes.
3

---

Published as a conference paper at ICLR 2025
3.2.1
TASK PLANNER
Parsing User Query User queries are often vague and unstructured, making it challenging for sys- tems to understand the user's s intent accurately. To overcome this, we introduce a persing-rephrasing stage that bridges the gap between the ambiguous query and the system's 's structured requirements. This initial step sets the foundation for the subsequent stages, enabling the system to extract relevant information.
We parse an input user query q into distinct components: instruction (i: strr,, input text (t: str), question (s: str), URL (u: str), data (x: dict), and categories (g: list), as follows:
P(q) = (i, t,8, g,
(I)
with P the parsing function. The ability of LLMs to parse user queries into structured formats. like JSON, has been highlighted across multiple research efforts (Petroni et al.] 2019; Wei et al. 2023). Using prompt engineering with few-shot setting Brown (2020) (see Appendix D) guide the LLM to convert an unstructured user input into structured data. Additionally, we ask the LLM rewrite the user instruction enhance clarity, simplifying complex directives and converting implicit information into explicit statements.
The instruction is crucial it is used in the later stages decompose the user query into smaller parts and determine objectives in the user query, q. By transforming vague queries into well-defined components, our system becomes more robust to handle diverse and complex user inputs.
Task Decomposition Given the resulting instruction, after processing the input user query q. we proceed to decompose it into smaller, manageable steps to identify specific plan attain the objectives (Wei et al., 2022, Yao et al. (023b) and understand how each part of the instruction is associated with achievable goals within the system's capabilities. We utilise an LLM classifier (Zhang et al. (024) in few-shot example setting (Brown 2020) (see Appendix D to identify the relevant domains from the original set of PDDL domains, D.
By providing the LLM with examples of instructions and their associated domains, we guide it to select the pertinent subset D' S D that aligns with the instruction. We prioritise recall in this classification step to ensure that all potentially relevant domains are considered, minimizing the risk of missing critical actions required to fulfil the user's objectives. This approach enhances the system's is robustness by accounting for a wider range of possible actions.
Once we determine the subset of relevant PDDL domains, D*, we leverage the predefined PDDL structures of each classified domain. These domain structures are then mergedto create unified PDDL domain file inclusive of all actions from ID*, s.t.
U a d, ED j=1
This method ensures that the combined domain file encompasses all necessary actions while main- taining consistency and comprehensibility. Next, we exploit the parsed instruction i and the com- piled set of actions from DD* a'", with an LLM to determine the specific actions required to achieve the instruction's 's objectives (see Appendix D as follows: af C P.
This allows to precisely map high-level user intents to concrete actions. Following this selection, the combined PDDL domain file (i.e. aP') and the identified actions set (i.e. a," help to reconstruct the corresponding PDDL problem. Finally, a Best First Width Search (Lipovetzky Geftner 2017) logical reasoner computes detailed Plan of Actions ordering the actions E aạ such that the system can execute step-by-step step, ensuring coherent execution in line with the user's intent.
This hierarchical approach not only enhances the system's robustness parsing and understand- ing diverse and complex user inputs but also guarantees the accuracy and feasibility of generated actionable plans by rigorously structuring them within established domain constraints.
3.2.2 MODEL. SELECTION
Following the task planning, the next stage is the selection of appropriate models capable of per- forming the specified actions in the plan. Utilising the information from our C-KG (see Sec- tion 3.1 our goal is identify model combination MI M that would satisfy set of conditions
4

---

Published as a conference paper at ICLR 2025
C= ................................ Got imposed by the user while offering guarantees that the selected model com- bination will be suitable to generate an output y that addresses the original imput query q, s.t.
M = arg max p (y/q, M, C).
M
For instance, if licensing is a concern, we filter out models that do not meet the required licensing terms. Similarly, if there are limitations on computational resources, we prioritise models that efficient in size and cost. Additionally, we consider performance scores from relevant benchmarks to guide our selection. By analyzing these performance metrics, we can choose models that have demonstrated high effectiveness on tasks similar to those required, ensuring that the selected models are not only compliant with user constraints but also optimal for the specific tasks at hand.
By balancing these factors: model capabilities, licensing requirements, resource constraints, and performance metries, we systematically select the most suitable model for each action. This ensures that the execution of the plan is aligned with both the technical requirements of the tasks and the practical considerations of the user. Ultimately, this careful selection enhances the system's 's effi- ciency and effectiveness, enabling it to perform complex tasks while adhering to user constraints.
3.2.3 PLAN EXECUTION
With the appropriate models selected for each action, we proceed to the plan execution phase. The execution involves retrieving the Python code snippets associated with execute relation with the chosen models, which are stored in the C-KG. More specifically, we map the arguments of the Python functions to the relevant components extracted from the parsed user query-(i, t, S, x, g} (cf. Eq. [ -using a complex similarity mapping algorithm. This mapping ensures that the models receive the correct imputs derived from the user's s query, accurately capturing the user's intent. In cases where the Python code snippet for the selected model is not available, we employ fallback strategy. We search for code snippets from other models that have been assigned the same task and possess similar functionalities. This approach leverages the semantic and functional similarities between models within the same domain, allowing us to substitute models when necessary without compromising the action's intended outcome. By orchestrating the retrieval of code snippets and the fine- mee-grii mapping of arguments, our system seamlessly transforms high-level plans into exe- cutable code, ensuring that the models operate on the intended data and parameters. This execution phase is critical as it bridges the gap between planning and action, which guarantees that each step of the plan is performed correctly and efficiently.
EXPERIMENTS
4.1 BASELINES
Addressing complex, multi-modal real-world tasks presents substantial challenges, and current so- lutions are limited, For our experiments, we compare our proposed method against the most relevant state-of-the-ar techniques: HuggingGPT (Shen et al. 2024a) and ControlLLM (Liu al. (023c). These represent significant advancements in integrating LLMs with task planning and execution frameworks.
We propose two innovative methods: HIVE and HIVE light HIVE leverages the advanced capabil- ities of ChatGPT for parsing user queries and decomposing tasks. To address the computational challenges associated with ChaiGPT-based systems, we designed HIVE light efficient alterna- tive that can be deployed on local servers. HIVE hight employs InterL.M2.577B-cha""" Ca et al. (2024) for task decompasition, both of which have
for parsing user queries and been subjected to 8-bit quantization. We selected a chat-oriented model for parsing, as conversa- tional models excel at understanding subtle queries. Additionally, an instruction fine-tuned model was chosen for task decomposition due to its capability to deliver precise instruction clarity. By employing this mual-method setup, ensure thorough performance evaluation, positioning HIVE and HIVE light as strong competitors to existing of frameworks.
chat
5

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of Task Selection, Flow of Tasks, and Output across all competitors.
Query Types
HuggingGPT
ControlLLM
HIVE light
HIVE
TS
FoT
TS
FoT
0
TS
FoT
O
TS
FoT
Single Task
0.47
0.47
0.83
0.74
0.74
0.74
0.80
0.80
0.72
0.88
0.88
0.79
Two Tasks
0.64
0.55
0.44
0.33
0.33
0.38
0.67
0.62
0.52
0.71
0.69
0.58
Three Tasks
0.42
0.42
0.30
0.36
0.36
0.33
0.57
0.43
0.33
0.67
0.67
0.46
Overall
0.57
0.51
0.53
0.43
0.43
0.47
0.69
0.64
0.55
0.74
0.73
0.62
4.2 MUSE MULTI-MODAL SUB-TAN TASK EXECUTION BENCHMARK
In the absence of standard publicly available benchmarks for solving real-world tasks, and recogniz- ing that HuggingGPT (Shen et al.) 2024a) did not release their evaluation dataset, we developed a new benchmark to assess proposed framework alongside state-of-the offtthe-rt methods like Hugging- GPT (Shen et al., 2024a) and ControlLLM (Liu et al., 2023c). Although ControlLLM (Liu et al. 2023c) released their benchmark, it utilises fine-tuned task decomposer, which could introduce bias if used for our evaluation. Therefore, to ensure a fair and unbiased comparison, we collaborated with experts from diverse linguistic backgrounds create set of 100 heterogeneous, real-world user queries (see Appendix Table ()) These queries are categorised into three types: Singlc-task, Two-task, and Three-task queries. In order to facilitate a fair comparison, we included only those tasks and models that are supported by all three systems. The benchmark is designed to cover vari- ous task domains, such as automatic speech recognition, question answering, and image generation, involving 15 models across different modalities This comprehensive benchmark enables us to rigorously evaluate the performance and generalisability of our framework.
Metrics. To assess our framework against state-of thethe- methods, we evaluate performance on three fronts, using binary metrics for simplicity and clarity:
Task Selection (TS): Determines whether the system accurately identifies the required tasks from the user's query. We assign a binary score of 1 the system selects the tasks correctly, and 0 it does not if it selected irrelevant tasks. Correct task selection is crucial it lays the foundation for successful execution and directly impacts the relevance of the final output.
Flow of Thought (FoT): We evaluate the logical sequence and integration of the selected tasks. A binary score is given based whether the system establishes the correct flow-1 for proper flow that respects task dependencies and order, and O for an incorrect sequence. This ensures that, especially in multi-task queries involving two three tasks, the system processes tasks in order that leads to the desired outcome.
Final Output (0): Assesses the correctness of the system's 's final response to the user's query. We adopt a binary evaluation-1 if the output fulfills the user's requirements, and 0 if it falls short. This includes evaluating the relevance of generated content, and the overall satisfaction of the user's intent. Note that do not evaluate the quality of the output, that is we do not judge if the output is accurate but only focus on whether the expected task has been performed.
By employing these binary metrics for each query, we simplify the evaluation process while effec- tively capturing the essential aspects of each system's performance.
4.3 OVERALL RESULTS
The results the experiment are presented in Table [ The scores highlight the effectiveness of our proposed approach in handling complex multi-modal tasks. HIVE consistently outperforms the baseline methods in overall performance across all evaluation metrics: TS, FoT, and O. This performance is evident in both single-task and multi-task scenarios. Remarkably, HIVE light which
MUSE involves 10 domains leading 10 distinct PDDL domain files covering 15 tasks. 67% of the queries multi-modal [text, image, audi also Appendix A and the supp" ementary material.

---

Published as a conference paper at ICLR 2025
Table 2: Cross-Modality Performances.
1.28
Single
10.97
In.
Out
Text ControlLLM
4.05
HuggingGPT
IIVE aghn
1,,02
Image
0.48
0.45
0.52
Two
Audio
0.25
0.25
0.67
4.31
6.00
Three
11.13
In]
Out
Image ControlLLM
6.59
HuggingGPT 0.36
HIVE
5,05
Text
0.T8
0.75
Overall
4.62
11:71
Audio
0.50
0.25
1.00
10
In|
Out
Audio ControlLLM 0.14
Hugging
ControlLLLM
IlIVE
IliVE
HuggingGPT 0.33
HIVE light
Text
0.71
Figure 2: Average times (s) before execution.
Image
0.80
0.00
0.00
based on 8-bit quantised models with only billion parameters, surpasses both HuggingGPT and ControlLLM in the overall evaluation. This underscores the effectiveness of our approach even with reduced computational resources.
In the case of single-task queries, all methods perform relatively well in generating correct final outputs. Though, HIVE stands out by achieving the highest performance in Task Selection and Flow of Thought, indicating more accurate and coherent understanding and reasoning process. Conversely, HuggingGPT attains the highest Final Output performance but performs poorly Task Selection and Flow of Thought metrics. This discrepancy arises because HuggingGPT tends to collect as many relevant tasks as possible, even query requires only single task, leading to over-selection. Despite this over-selection, its strong ChatGPT backbone enables it to produce high- quality final outputs. Nevertheless, this approach may reduce the trustworthiness of the results, as does not align precisely with the intended tasks (further discussed in Section 4.4.2). When it comes to multi-task queries, as expected, HIVE distinctly outperforms its competitors across all metrics. PDDL planning in HIVE light allows it surpass HuggingGPT and ControlLLM, demonstrating proficiency in task selection, task flow management and execution. In contrast, ControlLLM strug- gles considerably with multi-task queries, exhibiting the weakest performance among the evaluated methods. While HIVE, HIVE light and HuggingGPT employ prompt-based strategies to provide flex- ibility and adaptability, LoM relies on fine-tur munet task decomposer. This approach limits its capacity to generalise to queries that deviate even slightly from its training set, leading to significant performance declines in multi-task scenarios.
4.4 DISCUSSIONS
4.4.1
CROSS-MODALITY PERFORMANCES
To gain a better understanding of the multi-modality these systems' capabilities, we dig deeper into the Final Output (O) results from Table [ We divide this investigation into three distinct parts based on the output modality and analyze the performance when the other two modalities are in- volved in the input (Table 2
When it comes to text output, HIVE light demonstrates substantial lead over its competitors when the input includes any image or audio. This showcases HIVE light 's ability to integrate visual and auditory data to enhance text outputs. In the context of image output, HIVE light once again outperforms the other systems, illustrating its proficiency in converting textual and audio inputs into coherent visual responses. Lastly, although our system shows commendable performance in text-based audio generation, it falls short of achieving the desired objective in the maaae-to-audin scenario. indicating an area for potential improvement in future iterations.
4.4.2 TRUSTWORTHINESS
In order to review the connection between justifications (i.e. the conjunction TS and FoT) and outputs (O), group in Figure 3 the results based on (justification, output) scores which can
12
12T iff TS AND FoT both 1; I. in all other cases.

---

Published as a conference paper at ICLR 2025
58
50
HugginnggPP ControlLLM
42
40
33
HIVE light
II
HIVE
33
25
26
21
20
15
13
6
10
9
0
TT
TT
TT
Figure 3: Correlations between justificatin (TS AND FoT) and outputs (O).
respectively be correct T or incorrect 1. The failure cases Err are further discussed in Section 4.4.3 There are thereby four distinct cases. First, TT which corresponds to fully correct cases having both justifications and outputs, in this HIVE solutions (50+) outperform both HuggingGPT (25) and ControlLLMN (33). Then, T.L. means that the plans were correct but the execution did not go through. In this category the four reviewed systems perform similarly, ranging from 6 to 15 cases. On the right-end side of Figure 3 the plain-wrong case TT witnesses ControlLLM as the "worst" system (see results in Section 4.3 having the highest score. Finally, the critical IT case indicates lack of trustworthiness across all the baseline systems that may result in misleading outcomes. This case involves an incorrect plan or justification, despite the output being correct. Such cases have been recorded 9 times for ControlLLM and 13 times for Hug while being absent from HIVE and results produced by HIVE.
singleton in HIVE light showcasing the reliability
Overall, this discussion allowed us to highlight two aspects. One, unlike other solutions, Hugging- GPT exhibits results ranging from 13 to 25 in the four categories, meaning that it is hard to rely on it. Second, HIVE (s) tend not to fall in incoherent cases where results correct without the corresponding plan being correct-in other words, when results are good their explanations can be trusted well.
4.4.3 ROBUSTNESS
Since most of MUSE's queries require multiple models interact together compatible manner, we noticed that sometimes the tested systems fail at dealing with either the justification the outpur parts. In Table 3 we list the different cases encountered. The first point among the four systems. HuggingGPT is the less robust one by far: 22 Err against 10, 7 and 6 for the other solutions. Second, even more critical, that HuggingGPT, unlike the competition, able to generate correct results (T) while failing (Err) in its plan construction. This exacerbates the fact that its justifications cannot be trusted, as the executed actions tend in many cases not to be consistent with the compiled plan, using GPT-3.5 most places. This last finding is coherent with the LT discussion in Section
highlighted is that,
4.4.4 LATENCY FOR PLANNING
Lastly, in this discussion section, we analyse the time performances (in seconds) of the systems come up with a plan and select suitable models. As the chosen models may differ and en- forced rules such as "the quicker the better" (see Section 4.5 for discussion about model selection capabilities) were added, we measure the latencies up the model selection stage.
Figure 2 presents these latencies according to the split already presented in Section 4.3 as the number of tasks involved in the queries. First all, HuggingGPT and HIVE (s) share the same orders of magnitude whereas Con is magnitude slower (always 10+ seconds). Next, as expected, the more tasks within the query the slower the systems become until they reach an execution plan. On this, it is interesting to note that HuggingGPT's scaling law does not seem "linear" the slope increases greatly between two- and three-task queries. This behaviour is compatible with the internal implementation of HuggingGPT: when other systems rule-based (see the C-KG for HIVE (s) to select models), HuggingGPT needs to prompt (together with model descriptions) to select which models use for each task.
8

---

Published as a conference paper at ICLR 2025
Table 3: Failing case enumeration (Err), either as justifications (TS AND FoT) or outputs (O).
Failure Type
HuggingGPT
ControlLLM
HIVE light
HIVE
Err.
0
Err
3
-, Err
3
0
Err, Err 8
8
Overall
22
10
4.5 TAKING INTO ACCOUNT USERS' CONSTRAINTS IN TERMS OF MODEL SELECTION
As depicted in Section 3.2.2 once a plan of actions is established, HIVE selects the best models to realise them. Obviously, depending the cireumstances, the definition of what is "best" may vary a lot, egg. when resources are sparse, one may decide to use the smallest models possible even if the resulting quality is reduced, alternatively users might choose select models based on their respective (recorded) results for specific benchmarks. In order to respect these various cases, HIVE allows users to specify selection criteria. In this Section [3] we review the capabilities of HIVE against HuggingGPT when users want to force some conditions of their own in the model selection. Since ControlLLM has one-to-one mappings of models for each task, is de facto excluded.
Practically, we use the following query 14 "Transcribe the audio from audio.1 wav and find entity tokens". Regarding the task-model mappings, we let both HIVE and Hug- and vidi//prrakeet-mnt-I.. tively Apache-2.0 and CC-By-4.0 for licenses) for the ASR; and to dslim/bert-base-Ne license) for NER. We first run the query without any constraints (control run, see Appendix B): both systems, HIVE and HuggingGPT, were able to transcribe the audio file and perform NER (even though HuggingGPT result set was empty). We then applied the following model selection con- straints sequentially:
gingGPT have access to:
(having respec- (MIT
1. License restrictions: only use Openrail++ and Deepseek HIVE returned nothing which was the expected behaviour as the available models were not having the requested licenses; on the other hand, HuggingGPT performed the task in the control therefore infringing the restrictions (see Appendix D
2. Uses the "smallest possible" mode 15
HIVE complied with the user choice and used the smaller models whereas HuggingGPT kept using openai/whiisper--age--2 as in the control run
(see Appendix B).
3. Filter for the model having the best results at the speech recognition on common voice english benchmark Using the benchmark records from the C-KG, HIVE was able to select the correct model unlike HuggingGP'T which chose models like in the control run (see Appendix Table B)
Overall, HIVE answered each time while properly taking into account the given constraints. While HuggingGPT failed every time, misleading even the users with regards its justifications (refer to Section 4.4.2 for further justifications on this).
5 RELATED WORK
Automated Planning. The cognitive ability to organize and coordinate actions toward a specific goal is referred to as planning. While humans innately possess this capacity, machines lack such capability. Automated planning has garnered significant interest from researchers across various domains, including roboties (Guo et al. 2023), autonomous vehicles (Madridano et al., (022]. and dialogue systems (Wang et al. 2023a). The methodologies employed to devise sequences of actions have evolved considerably, particularly in light of recent breakthroughs in deep learning. Before the
13 See also Appendix B for an extensive result description. "A multimodal one involving two tasks: ASR and NER. We use the model disk footprint as a proxy for its size. Introdduee in "Common Voice: A Muulilily Speech Corpus" Ardila (019))

---

Published as a conference paper at ICLR 2025
advent of large language models (LLMs), planning frameworks such as STRIPS (Fikes & Nilsson 1971) or HTN (Erol et al. 1994) were developed to decompose tasks into a series of actions (or sub-tasks) leading to the desired outcomes (Sacerdoti [975). Building upon these frameworks, the Planning Domain Definition Language (PDDL) (Aeronautiques et al., 1998) emerged as a widely adopted standardized language for defining planning problems and domains. However, L.MSs have superseded those frameworks to stand as planner on their own (i.e. the LLM-as-planner paradigm). Multiple prompt engineering techniques (Liu et al. 2023b; Graux et al.) 2024) were designed to leverage in-context learning aiming to directly generate the multi-step problem solutions. More specifically, the Chain-of- Thouuhh (Wei et al.) (022) has revealed the promising reasoning capabil- ities of LLMs. and therefore new techniques were fashioned such as the self-consistency decoding strategy (Wang et al., 2022),
2023) or StraphoffThooughe Tree-of-Thought (Yao et al. (023a), Program-of-Thoughh (Chen et al., 2023c. Besta et al.] 2024). However, LLMs are still strug-
(Yao et al.
gling to produce acceptable and logical plans, especially as the complexity of the problem increases 2024 Zheng et al. 2024) Thus, numerous initiatives have
Valmeekam et al.) 2023 Xiao et al.,
therefore sought to integrate problem- specific languages like PDDL along LLMs to maximize their
effectiveness and leverage their full potential (Vyas al. 2025
Pallagani et al. 2023 Liu et al.,
2023a Oswald et al. 2024).
LLM-as-Agent. The genesis of large language models (LLMs) primarily stemmed from textual content, which initially narrowed the research focus to text generation. However, to address the
diversity of real-world scenarios, significant efforts have been directed toward developing vision or speech LLMs, thereby aligning with multi-modal paradigm (Zhu et al.) 2023 Wu et al. 2023] Wang et al. 2023b). Additionally, to expand the capabilities of LLMs, there has been an increasing
trend to integrate them with external tools incorporating retrieval (Shen et al. 2024c b or other tool
learning strategies (Wang et al.) 2024) Toolformer (Schick et al. 2024) pioneered the invocation of tool calls within generated sequences via special tokens giving rise to augmentedd LLMs (Qin) jet al. 2023a by Guo et al., 2024 Qu et al., 2024). Then, ReAct (Yao et al., 2023b) introduced such intermediate tool calls during the reasoning process by incorporating intermediate outcomes within the prompt to better guide the final resolution of the problem. In contrast to ReAct, Reflexion (Shinn) ct al. 2023) adds verbal feedback on those intermediate results to further assess and verify outcomes, In the meantime, a plethora of fine-tuned LLMs tailored for specific tasks has become ubiquitous on platforms such as Hugging Face Hub (Wolf et al.] [201)). alongside proprietary models such as GPT-4 (Achiam et al. 2023), Claude (Anthropic| 2023), and Gemini (Team et al. 2023) offering the opportunities to consider these L.Ms as distinct agents. Indeed, the gathering of technical details for each parametric model stands as a critical component in the reporting and tracking efforts underlined by the use of Model Cards Mitchell et al. (2019). HuggingGPT (Shen et al. 2024a), leverages such large pool of LLMs using ChatGPT the core controller. Following similar approach, ControlLLM (Liu et al. 2023c) and Chameleon Lu et al. (2024) explore task planning via prompt engineering and integrate more diverse pool of tools. While HuggingGPT, ControlLLM or Chameleon appoint appropriate models for each sub-task, however, their model selection process remains sub-optimal as they do not identify the most accurate model. Thus, if these frameworks can fulfil their plans, the resulting performance may be unsatisfactory if the best agent is not utilized. To the best of our knowledge, our work represents the first attempt to address this gap.
Lu
CONCLUSION
Our research introduces HIVE, an innovative and comprehensive knowledge-aware planning solu- tion designed to navigate the complexities of model selection and task planning using diverse set of deep learning models. By leveraging Capability Knowledge Graph and an LL.M-based formal logic planner, we transcend the limitations of the existing systems. HIVE stands out for its capability to plan and explain complex action chains while respecting user-specific constraints -thereby achiev- ing both high performance and full transparency. Empirical evaluations on our newly designed benchmark reveal HIVE'S superior performance, consistently outperforming competing platforms like HuggingGPT and LoM This breakthrough underscores HIVE'S potential to redefine the state-of-the the- in task selection and planning, ultimately facilitating more efficient and user- -rienfii applications of advanced deep models. HIVE thus advances the handling of multi-modal tasks.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.087771 2023.
Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins Sri, Anthony Barrett, Dave Christianson, et al. Pddl- the planning domain definition language. Technical Report, Tech. Rep., 1998.
Anthropic. Claude (oct 8 version). Accessed: 2023-10-08, 2023.
URL https://wwe
anthropic.com Large language model.
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. SpeechTS: Unified-modal encoder- decoder pre-training for spoken language processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7723-5738, May 2022.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A masiively-mutilingual speech corpus. arXiv preprint arXiv: arXiv: 191..6677, 2019.
Brett A. Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reill Reilly, James Prather, and Ed- die Antonio Santos. Programming is hard or at least it used to be: Educational opportunities and challenges of ai code generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, SIGCSE 2023, pp. 500-506, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394314,
doi: (0.11455/54494433569559)
URL wttph:::////oo/t http://///ooooo11..1111111 1145/ /34455555555555 3569759
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gian- inazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 38, pp. 17682-17690, 2024.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, al. Internlm2 technical report. arXiv preprint arXiv:2403. 17297 2024.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagonuyko. End-to-end object detection with transformers. CoRR, abs/2005. 2020.
URL wttpttts///axiv...tooo
2000 12872
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023.
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL
http:://axxi: org/
/llg///bbb. abs / 04805
Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. ACM Trans. Softw. Eng. Methodol, 33(7), September 2024. ISSN 1049-331X. doi: 10.1145/3672459
.org 10. 10.1141/3662453
Kutluhan Erol, James A Hendler, and Dana S Nau. Semantics for hierarchical task-network plan- ning. Citeseer, 1994.
Richard E Fikes and Nils Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence, ((3-4):199-20 1971.

---

Published as a conference paper at ICLR 2025
Damien Graux, Sébastien Montella, Hajira Jabeen, Claire Gardent, and Jeff Z Pan. [prompteng] first international workshop on prompt engineering for pre-trained language models. In Companion Proceedings of the ACM Web Conference 2024, pp. 1311-1312, 2024.
Huihui Guo, Fan Wu, Yunchuan Qin, Ruihui Li, Keqin Li, and Kenli Li. Recent trends in task and motion planning for robotics: A survey, ACM Compul. Surv, 55(13s), jul 2023, ISSN 0360-0300. doi: 10/1145/3883136 URL http::ttt///att
org/ 10. 1145/
Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models, 2024.
Jie He, Yijun Yang. Wanqiu Long, Deyi Xiong, Victor Gutierrez-Basulle and Jeff Pan. Evaluat- ing and Improving Graph to Text Generation with Large Language Models. In In the Proceedings of 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Com- putational Linguistics (NAACL 2025), 2025.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
//axxiv.org/ab//2231 06823
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to geuuenc-to-sequenee to-seqennce pre-training for natural language generation, translation, and comprehension, 2019a. URL https://
wtxiv.org/abs/// /91.. 13461
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequueee pre- training for natural language generation, translation, and comprehension. CoRR, abs/1910.13461, 2019b. URL /ttp::tttttpiiiiee wttp://.wrggaaa/.1....111
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified laniunee understanding and generation, 2022a. URL https:
7/ /aaxiiv.oo org.orggg.o...... /aob//2201... 12086
Junnan Li. Dongxu Li. Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image
pre-training for unified vision-language understanding and generation, 2022b. URL https:
12086
Nir Lipovetzky and Hector Geffner. Best-first width search: Exploration and exploitation in classical planning. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017. doi: 10.1609/aaiivv3lil..u.
URL httpp:///oss.oo.. /ttp:::://oss.a....... aaai.
php/
B. Liu, Yuqian Jiang, Xiaohan Zhang, Qian Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+p: Empowering large language models with optimal planning proficiency. ArXiv, abs/2304.114777, 2023a. URL. hhttps://ap) semant 258298051
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. ACM Comput. Surv. 55(9), jan 2023b. ISSN 0360-0300. doi: 10.1145/3560815. URL https: http:::/ddoi.rooo/ org. 10. 1145/35608155
Zhaoyang Liu, Zeqiang Lai, Zhangwei Gao, Erfei Cui, Zhiheng Li, Xizhou Zhu, Lewei Lu, Qifeng Chen, Yu Qiao, Jifeng Dai, and Wenhai Wang. Controlllm: Augment language models with tools by searching graphs. arXiv preprint arXiv:2305. 2023c.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: and-and-play compositional reasoning with large language mod- els. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Ángel Madridano, Abdulla Al-Kaff, David Martín, and Arturo de la Escalera. Trajectory plan- ning for multi-robot systems: Methods and applications. Expert Systems with Applications, 173: 114660, 2021. ISSN 0957-4174. doi:
URL https:
/ //www. Sc encedi rect. com wwwpeettcooscccccccccc
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pp. 220-229, 2019.
James Oswald, Kavitha Srinivas, Harsha Kokel, Junkyu Lee, Michael Katz, and Shirin Sohrabi. Large language models as planning domain generators. Proceedings of the International Confer- ence on Automated Planning and Scheduling, 34(1):423 431, May 2024. doi: 10.1609/capss.
v3421.31502.
URL sttps:://jjs aaai. org hwt.//inexx./oo php/ ICWP/CCPP/SAA
31502
Vishal Pallagani, Bharath Muppasani, Biplav Srivastava, Francesca Rossi, Lior Horesh, Keerthiram Murugesan, Andrea Loreggia, Francesco Fabiano, Rony Joseph, and Yathin Kethepalli. Plans- former tool: Demonstrating generation of symbolic plans using transformers. In Edith Elkind (ed.), Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pp. 7158-7162. International Joint Conferences on Artificial Intelligence Organiza- tion, 8 2023. doi: 1444966//iiii2223///333 URL /ttp:/:://o.....r//0..222333/ wttp:////......222...2 2023 839 Demo Track.
Jeff Z. Pan, Simon Razniewski, Jan-Christoph Kalo, Sneha Singhania, Jiaoyan Chen, Stefan Di- etze, Hajira Jabeen, Janna Omeliyanenko, Wen Zhang, Matteo Lissandrini, Russa Biswas, Gerard de Melo, Angela Bonifati, Edlira Vakaj. Mauro Dragoni, and Damien Graux. Large Language Models and Knowledge Graphs: Opportunities and Challenges, pp. 1-38, 2023.
J.Z. Pan, G. Vetere, J.M. Gomez-Peree Perezz and H. Wu (eds.). Exploiting Linked Data and Knowledge Graphs for Large Organisations. Springer, 2017. ISBN 978-3-319-45652-22
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv: weeei/109000000 2019.
Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy- based constrained text generation with langevin dynamics. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 9538-9551 Curran Associates, Inc., 2022. URL http::///pooeeeii.ss..eeosi.ss.s http://pppeer_eeppppppeeeeeeee fites//apeeee/////////// htte:///eeeeeee660000000606eeeee Con.--aaappe--oofooeeeeeeeeeeee .ddf
Yujia Shengding Hu, Yankai Lin,..., and Maosong Sun. Tool learning with foundation models, 2023a.
Yujia Qin, Shihao Liang, Yining Ye, and Maosong Sun. Toolllm: Facilitating large language
models to master 16000+ real-world apis, 2023b.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with large language models: A survey. arXiv preprint arXiv:2405.17933 2024.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URL hhttps://arxip
org/abs/ 2212. 04356
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. CoRR, abs/2103.11413. 2021. httth:t/p/aaaxtt http///wwww..../////........ 13413
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Patlern Recognition (CVPR), pp. 106851 June 2022.
13

---

Published as a conference paper at ICLR 2025
Earl D. Sacerdoti. The nonlinear nature of plans. In Proceedings of the 4th International Joint Conference on Artificial Intelligence Volume I, IJCAI'75, pp. 206,,14, San Francisco, CA, USA, 1975. Morgan Kaufmann Publishers Inc.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. In NeurlPS EMC² Workshop, 2019.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. PICARD:
Parsing incrementally for constrained auto-regressive decoding from language models. In EMNLP,, pp. 9895-9901, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 1...8663/112022..mmlppp www.emmmmmnmmin7777 7799. URU/itppp.///accaataaooogg/// owtphhoooooooy...221111111
emr.p ma maiin. 779
Tim Schopf and Florian Matthes. Nip-kg: A system for exploratory search of scientific literature in natural language processing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 3: System Demonstrations), pp. 127-135, 2024.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugging- gpt: Solving tasks with chatgpt and its friends in hugging face. Advances Neural Information Processing Systems, 36, 2024a.
Zhili Shen, Chenxin Diao, Pavlos Vougiouklis, Pascual Merita, Shriram Piramanayagam, Damien Graux, Dandan Tu, Zeren Jiang, Ruofei Lai, Yang Ren, and Jeff Z. Pan. GeAR: Graph-enhanced agent for retrieval-augmenter generation, 2024b. URL iittps://arxive /ttp///arxii.o.rr//////1222
18431
Zhili Shen, Pavlos Vougiouklis, Chenxin Diao, Kaustubh Vyas, Yuanyi Ji, and Jeff Pan. Improv- ing retrieval-augmented text-to-SQL with AST-based ranking and schema pruning. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing, pp. 7865-7879 Miami, Florida, USA, Novem- ber 2024c. Association for Computational Linguistics. doi: 10. 10.18653/v1/2000 1/2024 main.449, URL https: /atc///clacc anthol ogy. 2024 emn lammnp-mmtl 449/
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Advances in Neu- ral Information Processing Systems, volume 36, pp. 8634-8652. Curran Associates, Inc., paper files//aper/2023/ Caaer--ooeeeeeneeeee pdf
2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language mod- els on planning and reasoning about change. In Advances Neural Information Pro- Curran Associates, Inc., 2023. URI.
cessing Systems, volume 36, pp. httipss//proceediinggg.... neur papppprr..ppppppppppppppppepp..e /ttppprrerrallsspppr//////////// 7 @29bcceeee8607afdd1157222aa22222 Papei Paper- Datasets and Benchmarks. pdf
Kaustubh Vyas, Damien Graux, Sébastien Montella, Pavlos Vougiouklis, Ruofei Lai, Keshuang Li, Yang Ren, and Jeff Z. Pan. An Extensive Evaluation of PDDL Capabilities in off-the-shelff LLMs,
2025. URL Stttts//aaxx...rr//a////3333333 20175
Hongru Wang, Minda Hu, Yang Deng. and Kam-Fai Wong. Large language models as source
planner for personalized knowledge-grone grounded dialogues. In EMNLP, pp. 9556-9569, Singapore, 2023a. Association for Computational Linguistics. doi: 10. 10.18653/v1/203. win.e//n....innn---ee findings-emnlp.64...
URL U Ihttps://ack /
antho horoooloyy..rr. ogy. find ngs- 641
14

---

Published as a conference paper at ICLR 2025
Hongru Wang, Yujia Qin, Yankai Lin, Jeff Z. Pan, and Kam-Fai Wong. Empowering Large Language Models: Tool Learning for Real-world Interaction. In SIGIR, pp. 2983 2986, 2024.
Xinyu Wang, Bohan Zhuang, and Qi Wu. Switchgpt: Adapting large language models for non-text outputs. arXiv preprint wrXiv:2309.07633 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Lc, Ed Chi, Sharan Narang, Aakanksha Chowdh- cry, and Denny Zhou, confisoonss improves chain thought reasoning in language models. arXiv preprint 2 //171 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. ofain--f--though prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2453 2022.
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. Zero-shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302. 10205, 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the-art natural language processing. arxiv. arXiv preprint arXiv:1910.03771, 2019.
Chenfei Wu, Sheng-Kai Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models, ArXiv, abs/2303.04671, 2023. (Rtpps:s:
Ruixuan Xiao, Wentao Ma, Ke Wang. Yuchuan Wu, Junbo Zhao, Haobo Wang, Fei Huang, and Yongbin Li. Flowbench: Revisiting and benchmarking workflow-guided planning for Ilm-based agents, arXiv preprint arXiv:2406. 2488:2440...4844 2024.
Mingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao. Understanding the weakness of large language model agents within a complex android environment. arXiv preprint arXiv:2402.005566 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In A. Oh. T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 11809 1822. Curran Associates, Inc., 2023a. URL htttpps://prrcccedngggsss new.. neurips cwpppppepppppppppppppppppppppe.e htt.apprr_filllssapee/22233333
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting language models. In International Conference on Learning Representations (ICLR), 2023b.
Yao Yao, Zuchao Li, and Hai Zhao. Beyond chain-of-thought, effective graph-of-thought reasoning in language models. arXiv preprint arXiv:2305. 16582, 2023c.
Yazhou Zhang, Mengyao Wang, Chenyu Ren, Qiuchi Li, Prayag Tiwari, Benyou Wang, and Jing Qin. Pushing the limit of Ilm capacity for text classification. arXiv preprint arXiv:2402.07470, 2024.
Yilun Zhao, Linyong Nan, Zhenting Qi, Rui Zhang, and Dragomir Radev. ReasTAP: Injecting table reasoning skills during pre-training via synthetic reasoning examples. In EMNLP, pp. 9006-9018, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL htt.ps:://allatthhlllyy..
2022 wwnnl-main..n..m. 615
Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Tze Cheng, Quoc V Le, Ed Chi, et al. Natural plan: Benchmarking Ilms on natural language planning. arXiv preprint arXiv:2406. 24iv:2406.04520, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
15

---

Published as a conference paper at ICLR 2025
A MUSE EXPERIMENTAL SETUP
provides a comprehensive overview of the domains and tasks encompassed within the MUSE benchmark, All three competing systems have access to the models associated with each task listed in Table 4 It should be noted that both HuggingGPT and ControlLLM utilise ChatGPT as their backbone model, leveraging it for the execution of certain tasks. Since ControlLLM permits only one-to-one mappings, to maintain an unbiased benchmark, we assign one model per task (for de- tails, please refer to the Supplementazy Material repository Moreover, to preserve the naturalness of the queries, we have refrained from making any grammatical or spelling corrections the dataset.
Table 4: Al Tasks and Associated Models.
Domain
Task
Model
Automatic Speech Recognition
openai/whisper--ure lurgee-- (Radford et 2022
Audio
Text to Speech
microsoftsspeechrS.... .tt (A
Image Generation
Text Image
subilityaistuble-diiiiioooo. 2-1 Rombuch al.
2022
Image Captioning
SalesforccMii-ingge--ettioiinnni MaseeLin
2022b
Image Text
Object Detection
facebook//derr-ressee-1000 (Caion ul. 2020
Visual Question Answering
Salesforrc//bii-vqaaabuu ct 2022a
Image to Image
Depth Estimation
(Ranfll et al.
2021
Machine Translation
Translation from xx to
mistralai/Mistral IB--nnstuutt-OO I (Jang et al.
2023
Answer based on Context
distilbert/dsstibertt-
bus-tt-uu-----eed------- wawprssdddd-sslll--uuueeeeeeeeee Sanh et al.
2019
Question Answering
Open QA
mistralai/Mistral-T -a-rall----nnstut---000 Jiang el 2023
Table QA
www.LL//Y/asaaaaaarreeeee et al.
Summarisation
Abstractive Summarisation
facebook/bart
2019h
Text Generation
Text Generation
misttaluiMiisrra-7B--nmstto----- Dian
2023
Text Classification
Text Classification
facebook/bart hunge--mi (Lewis
2019a
Token Classification
Named Entity Recognition
Astim/bertbbuc-NERRRDAe
2018
Table 5: Small sample of queries from the MUSE benchmark.
Domain
Queries
"object__ettction"" detection"
How many people the image? /datav/magess/mmag..c. htjjs///mmaa..p..gp........
"image_jo _oeeext"" generrtenttion"
Write sonnet about this image /dttafimage/mmage_l
"image t0 text", "eneeeggenrraiion
Build pictorial presentation of the flower after deciphering the audio Jdhta/audios/audio w.wav
"audio", "question answering"
audio transcription as context and answer ing the war whom the associated speech is about?' from Adata/audios/au- dio_6.wav
the country start-
"andio", "summarisation", "image generation"
Extract a summary from audio transcript Adata/andioow/audio 8 way, and generate image based on the summary
16

---

Published as a conference paper at ICLR 2025
B SCENARIOS DETAILS FOR MODEL SELECTION VALIDATION
H
No Constraints (Cold run)
HIVE light
HuggingGPT
more perfeet Union, establish justice, eusure domestic tranquiity, provide for the common defense, promote the goneral welfare, and vecure the bless- aags ourselves posterity, ordain establish
We. paople Stotes,
Based
results, trasserited andio file following text.
":Aaddi]"
people United Sones, in order to furms a more perfect union, esnablish jus- rice, ensare domestic tnnquility, provide for the common defense, pramote the ourselves United welfare,
blessings
America"
Foe the named entity socognition tisk, have used the model "ádimmbrrt base-NER However, the in infernce result for this task empty, indicating
"LOChs s
"Onitad, "onstad_acates" Bed Fa
anttty tokess foundi
Here the complese woekflow for your request:
"MISC"
"Conee
Automatic Speech Recognition (ASR) Task:
Model used: "upenali w hisper large x2"
Inference resalt: andio file", "Auddoo"" transerited, and resulting
Named Recognition (NER)
Comment: Saccesfflll
"ddlinbert-hso-NEE"
Inference resalt: No named estny takens were found the tminerrtee test.
Please note that the inference resulls are based on the models used, and there possibility
invemplete detection.
Comment: Uluacessful: system pass outpot the previous sask as input the rabsequent one, preventing nie model from idemifinng any emiries
4
Specifically asked use the smallest model available for each task
HIVE light
HuggingGPT
We, the people the United Stotes, is order form more perfect Union, establish justice,
Based on the inference results, have trasseritedd the audio file "[Audio]" and found the fellowing namad enity tnkens: "We people
for the ings of liberty ourselves and our posterity, do ordain and establish this Constitution for the United States of America.
common defense, promatc general rraquility,
sacure
States" ORGIANISATION "the United States America" -LOCATION To accomplish this, fellowed the following workflow:
Task Audio Audio Transcription Model; No suitable
"LoCs
model found for this tzok
"Onn Statee"" "Oniled BLateddBBBaaaeeeee Lo Jase
Inference Result Error: "audio sudio" found avail-
NNISC*;
2: Automatic Speech Recognition (ASR)
Model: used hopens//hispe--laggggg
Inference resalt: "We, the people of the United States, in order form perfect union, establish justice, ensore domestic imaquility, for
defense, promote the welfim, and sccure the blessings of liberty to curselves United Staee of America,"
Commant:: Snecouffl!!
Named Eatiiy Recognition (NER) Task: Mode: used the "dslim/bert base NER" model Inference resul:: No named entily tokens found the
apologine
able
named
transcrited text. you any farthe questions
assistrec with anything clse, please lor me know!
Comment: Unouveesffll the system foiled iork salestion and refected "andio undio" first which aner query. sysion failed ileenify any emitles
17

---

Published as a conference paper at ICLR 2025
44
Specifically ask the systems to openrail++ & deepseek licensed models
HIVE light
HuggingGPT
Tupologire myy cunfesion caused. Based the inference results, here is the detailed wurkflow for your request:
Comment:
1. Automatic Speech Recognition (ASR) Task:
there were models with openrailve and deepseek ASR NER model should selected
Task: Transeribing the andio sile "IAudiol".
Model used: whespe large Infereace result: transcriledd
follows: "We, the people order form more perfect union, establish justice, ensure docsestic tranguliit, peovide for the common defense, promose the general welfare and secure the blessings of liberty ourselves and posterity, ocdain and estabissi Coustitation
States,
United States America."
2. Named Eurity Recognition (NER) Task:
Task: Finding named eacity tokens the trunsenbeed
used: Saslinchrt-boss-NE--
Inference result: Unformanttely. there is no available inference result this task.
Zerosshot Classification
Classifying eatny tokens.
Model used: "gpt-3 1.5-urbo-0613
Inference result: spologize, but All language model, am unoble accoss trineerthee andio transcribed, cation took. Could you plense provide the tranccripiion of audio and I'll happy help you with the classification taak using the provided text and image information?
direcily. However.
you
classifi-
4. Test- to Image Tusk: Task: Generating images based on named eniity tokens. Model used: "sabilitytsable dillosion-v!"
Inference result: Unfortantelly, there
inderence process this task.
Please niee that the inference results aee hosol the maillbbe madels and their capubilities. there please know.
specifle requiremens modifications would
Comment: Ensucvessful; first system performed incorvect of misgsided the nser by selrcting models thot do not aligued with the wores liomer contioiitt
Specifically asked to choose the Automatic speech recognition model based best performance on speech recognition on common voice english H
HIVE light
HuggingGPT
people der to fomm more perfect union esrablish justice insure domestic tranquility provide the defense
Based Inference will directly
and provide detailed complete path or
workflow your speciffic malk
inference
resalt::
the gen- blessings city ourselves and our posterat oedain and exaablsh this constitution for the united
Transeribing the andio "[Audio]" Mode uerd: "opensitu "spemsitehisee--lppe
Inference Result: The tramcribed text as follows: "We, the people of the United Stiies, order form perfect union. establish justice, domestic tranuility. deffnnee
general welfare, liberty to ourselves United States America."
blessings
"LOO"s 1
entity
tramscribed
"unit Btates", runstad_states_000 americc"
Model used: "dslim/bert base NEER Inference result: No named entiy tokens were found the taaseribed toxt.
Zero-shot classiiication for the task of "named entity tokens":
"ISC"* Puooaed ion'
Inference "gpt-3.-turbo-4611 3-tmrbo-0613""
Model used: "ppt-
language model, dol't shility pencess Therefore,
perform zero-shot classification
the noned entity tokess.
Generuting a response the task selecting models based on scores from "speech- recugnition-en sm-common-nus vuice "nnllisss"""" Model used: "gpi-3 5-turbo-055
Comment: Suceesguul: to tent the model on benchmark performances, added a "purokect-mui" model mix as this sprcifi beachmark per- forms better the "whispor lorge nyk successfullt ASR wol
on input
response generated:
"To select models for speech recognition on Common Voice English, you cun use the scores obtoined from the speoch recognition systems trained on the Common Voice Ea- glish dataset. evalusting the performance of dificrent models allows
these
which models must
for
the best models for your specific speech recognition needs. Factors such as accuracy, robusiness and language coverage should be considered when selectnng these models. The scures provide valuuble metric for compuring and evaluting different models, ul- timately asssiin
making process."
the information provided hased inferesee results models know!
If you have my specific coocerns funter questicus,
Comment: Unsucessslui the rystem fails at the task selection state while also Ignoring the resuivenennts to selrcted mudel for the ASR based on the henclmark proformasce
18

---

Published as a conference paper at ICLR 2025
C HIVE COMPARED TO CURRENT TOP MONOLITHIC LLMS
The recent advancements in models have shown that foundational model capabilities progressed a lot over the past years. This surge could lead to believe that at some point in the future only having one-single-larg model to-do-everything would be enough. Such a hope, has two issues:
1. such powerful model does not exist yet, and no one knows when it could arise;
2. it is barely impossible that such a model could exist and cover all the "niche" use cases and tasks which are currently populating the model landscape (e.g. protein folding or other rare scenarios).
To go further with the first aforementioned point, we MUSE over variants of some of the best models currently available [8 ol ol-preview and GPT-4o both from OpenAl and DeepSeek-v3 from
the eponymous company. The results are as follows:
Since ol -prreviw does not support image and audio modality, we only test the cases whose input, output and any expected intermediate re- sult only contains text. Overall, ol ol-prvveww was able to properly answer (O) 24 % of the time and properly justify (TS FoT)) its choices 26% % of the time.
Table 6: ol-preview
Domains
FS
FoT
0
Support
One
11
11
10
12
Two
10
10
11
Three
5
5
3
5
Total
26
26
24
29
Table GPT-4o.
Once again, since GPT-4o only supports text and image input and textual output, we filtered out all the entries from MUSE involving unsup- ported modalities. Overall, GPT-4o was able to properly answer (O) 43 % of the time and prop- erly justify (TS & FoTT) its choices 44% of the time.
Domains
FS
FoT
0
Support
One
16
16
15
20
Two
24
24
23
24
Three
4
4
5
Total
44
44
43
51
Since DeepSeek-v3 handles modalities simi- larly to GPT-4o, we filtered MUSE the same way. After running, it exhibits good planning performances (45% and 47% for FS and FoT respectively) but as compared OpenAI tested models, output quality felt down to 25%
Table 8: DeepSeek-v3.
Domains
FS
FoT
O
Support
One
19
20
11
20
Two
20
21
11
24
Three
6
Total
45
47
25
51
Unsurprisingly, even if all models are able to plan and to perform over MUSE for some of its queries, they are not yet able to deal with the richness of real-world multi-modal scenarios as depicted MUSE.
More generally, when it comes to the second point about the "niche" tasks, the essence of the Capability-KC lies the fact that its richness allows to retrieve candidate models for large set of tasks (more than 100 at the moment) and does not limit users to envision instructions revolving around a handful popular tasks.
Finally, we would like to mention direction we are currently exploring. So far, the presented C- KG gathers information related to models, but structurally, nothing prevents us from collecting and structuring information related to other objects. For instance, the graph nature of the C-KG led us starting the exploration of having sub-nodes for adapters attached to main prineipal-nooeeeeeee addpter-nodes still carry the same type of information as the main nodes currently listed in the C- KG, gathering performance information for instance. In the future, we think this addition would also allow building complex instruction pipelines which would rely mainly on a single backbone model and this could be a solution for some specific actors having access to only one model.
these
More generally, we would wrap up emphasising on the fact that even if the big models available at the moment aren't 't yet capable like a group of selected specialised models, the overall technical architecture of HIVE is still appropriate to tackle scenarios where only one model and associated variants (through adapters for instance) is available.
In this setting, both planning and execution parts are performed by the tested monolithic models, which slightly differs from the HIVE setting which delegates the execution to specialized models.
19

---

Published as a conference paper at ICLR 2025
D HIVE'S PROMPTS
Usage Snippet Extraction
($3.1)
You are python programming expert, mainly used to convert python code snippets into python functions. Follow the rules:
1. You also make sure all the function variables have default value.
There should always be input variable with default value that takes the input for the model.
3. Take Model path as variable with default value of model name
Return the model response in the python function
Transform the code snippets in the text into one signed Python function including all the potential variables and default values for all. Only respond with code and in markdown format " "pyyhhn"" {code}
Parsing & Rephrasing
3.2
Task Decomposition stage: The Al assistant parse input into multiple inputs and fill the relevant keys in the following JSON.
("instruction". None, "input_text!l None, "question": None, "url": None, "data__ictt"" {}, "categories": []}
Example
User: What is the date mentioned in this audio www.google.c www.goo.occooo.iiii.oooooooooo file.mp3? Response: (("instruction": "Convert the audio text and then answer the question", "url": wwww.google.com/andio....o.. "input_text": "What is the date?"})
Example 5
User: Please transcribe the voice into text //audi//audio 1 1.mp3 and classify the transcribed text into categories such as 'movie", "music", 'painting', or 'Other'. Response: ("instruction": "Convert the audio to text, and perform text classification", "url": " "/audio/audio._mmm I.mp3", "categories": l'movie', 'music", 'painting', 'Other')] ]]
Based on the above example, parse the following:
User: {USER_INPU)
Response:
Only return JSON
The JSON keys are defined below: instruction: textitwhat are the tasks asked the question, it maybe one, two or three tasks. Try to find the implicit tasks as well input-text extract the original text or context the input. Do not generate on your own question: extract if there is any question asked url: extract url passed
user query
data_dict: extract dictionary passed in the user query
categories: extract ALL categories mentioned the user query
Do not generate anything other than a parsable JSON
20

---

Published as a conference paper at ICLR 2025
Domain Classification
($3.2)
You are professional in natural language processing task. Find which domains are re- lated with the provided user query? You should pick domains from the following list [domains). You MUST NOT output other domains not the provdied list. Here are the examples:
Example 1: Answer the following questions in detail and give me summarisation for the answer Domains: question answering; summarisation
Summarise the transcript of the audio and find entities in Domains: audio; summarisation; classsclasiiiiattion
Example 11:
Provided Query: {query). 'The order matters' Domains:
Action Selection
($3.2)
You action selector. Given Task and list of Actions, Select the crucial/mandatory actions. Follow the instructions below:
1. Pick the least amount of actions that do the job in Task
Only select actions that are REQUIRED and NECESSARY for Task
3. Focus on precision of selection
4. Do not select EVALUATION or SCORE actions unless explicitly asked for in Task
Example 1
Task: want to select a schema and then generate SQL query for the a question
Actions: Sichema-Selections Can you understand the requirements of the Task and select necessary actions from the Actions. Do not give any explanations, only return a list and nothing else. Select at most three diverse, yet relevant actions.
"generate SQL", "execute query", "validate "validate-SQL.]
Aetected_Actionss ["Schema-Sellctioon., "generate.SQL""]
Example 3
Task: Retrieve documents renewable energy advancements and summarise the latest
technologies
Actions:
trieve woostrreleant.doocueee l'query based summarization", "rank documentss"s" ctt.aarrriaaiioeeealltatoooooo "get abstractive summarization",
"re-
"keyphrase "Keyphraae_extraction""" exxraccioon",
"get extractive summarization", trieve multiple documents")
"re-
Can you understand the requirements of the Task and select necessary actions from the Actions. Do not give any explanations, only return a list and nothing else. Select at most three diverse, yet relevant actions.
Selected Actions: ["retrieve ["retrieve_mullille documents"", "get
Task: (user.instruction)
Actions: (actions)
Can you understand the requirements of the Task and select necessary actions from the Actions. Do not give any explanations, only return a list and nothing else. Select most three diverse, yet relevant actions.
21

---

Published as a conference paper at ICLR 2025
dortlegenee
Modals
Developers
Tasks
Lenguages
Livense
Benchmark
Metric
Score
Code Block
Figure 4: Capability Knowledge Graph in use for the MUSE Benchmark.
E MUSE CAPABILITY-KG VISUALISATION
As described in Section we introduced the MUSE benchmark in order to compare the per- formances of HuggingGPT. ControlLLM and HIVE. The latter was declined in two sub-versions: HIVE light having an 8-bit quantised 7B model for planning tasks and HIVE relying on GPT-3.5 for the same action. particular, MUSE comes with 100 multi-modal, multi-task, complex, natural- language querie
this Appendix, we provide, in Figure a snapshot of the Capability Knowledge Graph which corresponds to the models and their associated pieces of data used by HIVE tackle the MUSE benchmark 20 is an excerpt, containing 461 triples involving 381 entities, of the complete C-KG which contains 125k triples for 39k distinct entities. The complete C-KG was formulated by extract- ing metadata from over 600,000 models listed on HuggingFace, with 26,806 models retained based on popularity metries. In Figure 4 model nodes depicted in yellow and at the core of the knowledge graph, in the sense that it is around models that the C-KG building process was designed, starting from the HuggingFace model cards (see Section 3.1 for more details on this process).
Visually, this graph is twofold, indeed the two stable-diffusion models (xl-base-1.0 1.0 & 2-1) are dis- connected from the of the graph. The remaining 13 models, the other hand, making single piece of graph and organisation nodes (in red) or license nodes (in black) are often con- nection hubs. Interestingly, one could see that the main part of the C-KG is "bordered" by gray and purple nodes which corresponds to metrics and respective scores for the various benchmark (in bright green) data points retrieved from the paperswi wwwwthodee API. Finally, the image-to-tent block composed of siip-image-captionin main graph and the whisper-large-v2 which completely surrounded by the languages covers (light blue nodes).
and blip-vqa-bass is somewhat separated from the
'ee the supp1 ementary material & repository for all the query details and their associated data. AA Weh-interface explore the MUSE C-KG is available supp ementary material Ø too.
22

---

Published as a conference paper at ICLR 2025
F TAXONOMY CONSTRUCTION OF NLP TASKS IN THE CAPABILITY-KG
In this Appendix, we present how we group various tasks -coming from HuggingFace's Model Cards- into a comprehensive taxonomy of model actions, starting from seed taxonomy manually curated (see in Figure 5). This laxonomy can be used to structure the PDDL domain files and to have finer domain classification while planning.
Practically, in order to reach comprehensive and detailed taxonomy, with useful hierarchy, we started the process by gathering manually various major tasks in NLP. For instance (cf. Fig- ure 5 which represents only excerpt of the seed taxonomy used), we put summa r ristt on and pa r reaphraiing within the text genera too larger category. We then iteratively go through the tasks in the Capability-Koo themselves extracted from HuggingFace Model Cards tasks, and try to integrate them within the taxonomy.
Task from C-KG
Encoded Teak
Most Sknnler element from Tixoromy
"socning"
Similarity Score
dlstiiaata" Do "extagee
Tnferattion warkd"
Texonimy Index
"aeed antiyy acmerttte D.
score range00 8, 0.9)
"hore orr errurtant D4
"sweertuttar" Dx
CENOTS KLMANN KINSTTM
"Iat granetior"
"partaar preritaar: O-
"peech viale tnarsher
Figure Taxonomy extension protocol: enriching seed taxonomy with tasks from the C-KG.
The hep-by-step process (Figure 5) undertaken can be described as follows:
Encode all the elements of NLP taxonomy using Triiennce-Trafoormme
2. Extract all the unique tasks present in the Capability-KG into list
3. For each task, search for the most similar element from the taxonomy (and its similarity)
if simi wwwllllrssyooo
2 0 9, we consider that the task (or something very close to it) already present in the taxonomy so we ignore
0. > simill larity score > 0.8, 8, we consider the task to be relevant and add it to the taxonomy wiiiiimatttysooooo
C.8, we ignore the task considering it irrelevant
Furthermore, when new task meets the relevant threshold (0.9 similarity O. 8), it is integrated into the taxonomy under the most similar existing category. This hierarchical structure is traversed to locate the appropriate level, and the task is added sub-category while preserving the nested format. At the end, the taxonomy is flattened to level-3 to maintain a balance between granularity and usability.
Overall, the constructed taxonomy (Figure 6) comprises 420 concepts over 3 hierarchical levels. goes from high-level conceptual tasks, e.g. syntactic text process sing, to very specific tasks like morpho logical tagqing. To the best of our knowledge" we are not aware of such a fine- ann-gerieed taxonomy of NLP tasks focusing on practical applications.
Available a json file the supp1 ementary material
Schopf & Matthes (2024) explored the fields of study in NLP, and even built classifier; but it does not dive as deep and technical as our taxonomy, but rather uses it to order Academic articles and extract research trends.
23

---

Published as a conference paper at ICLR 2025
Soean 1S HOMDAnng
eroeer
NO RNG
Iet tipe langusage resdess (uri
and
Name semanti pertog
porer lerporg nomting
sustoms
cinggeye nutill
moall
CONNEI
sinilaity
coniuriy
class enmonlie
you atrus
Intent enorels
Lins cicafication ac0t/si
Face Recoy coppattion
civoritcaioo
oepectiaioi artrrao
Jert recton
Sentimmnt chorficction
15721h
detectionn
@ arubea
(877s USject
Lugge Daga 120m
Brtutkro
gent
Aequbeiic and copotter sting
hott Corruciio
145 5ag mn:
sho comncction
costurrityy poor
symtertc
Vodicll rottcattaaiion
WONOOEEE youre
PRATT enpportiin CCHAL
Che Ang 1e
COREONORRI on 2e0000
Figure 6: Full taxonomy of the NLP tasks included in the Capability- KGG
24


Reviews:
[{"review_id": "SbPEe3lMB5", "reviewer": "Reviewer_XAQH", "summary": "The paper presents Hive, an agent-based system that selects and coordinates models to execute complex tasks based on user instructions. Hive creates explainable plans using atomic actions across various models, guided by formal logic and PDDL operations, ensuring transparency and adherence to user constraints. The authors also introduce the MuSE benchmark to evaluate multimodal capabilities in agent systems. Results show that Hive outperforms other systems in task selection and transparency.", "questions": "Could you please test the pipeline or more baselines on more widely used benchmarks in multimodal domains?\n\nWould you provide more examples about how the KG looks like and an entire planning example from scratch?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The KG construction method is thorough, utilizing model metadata from multiple sources to enhance graph quality. The entire planning pipeline demonstrates a clear improvement over prior work, such as HuggingGPT. Additionally, the writing is well-structured and easy to follow, which supports reader comprehension.", "weaknesses": "This paper has several notable weaknesses:\n\nFirst, it feels like the approach relies heavily on leveraging a powerful LLM to decompose user tasks based on key attributes, selecting open-source models from Huggingface, and executing plans sequentially. In essence, the paper presents a tool-usage framework where the tools are Huggingface models. While there are some unique designs for processing and indexing these models, the contribution appears marginal. The problem-solving paradigm closely resembles prior works, such as HuggingGPT. Additionally, the paper lacks baseline comparisons, particularly in evaluating why their proposed KG construction method is optimal or how it could be further applied in other scenarios.\n\nThe experimental design raises concerns as well. Although the authors introduce a new evaluation benchmark, it is based on an in-house dataset. It’s unsurprising that the method performs well on this dataset, but it lacks validation on standard benchmarks, such as classical VQA datasets for multimodal evaluation.\n\nOn a broader level, the necessity of maintaining a Huggingface model graph is questionable. As more cheap and advanced models (e.g., GPT-5, LLaMA-4, lower-cost GPT-4o) emerge with capabilities for comprehensive task handling, the reliance on a graph of smaller, domain-specific models may become obsolete. While a graph of non-LLM-based APIs remains valuable (e.g., calculators outperforms LLMs in calculation), most small LLM functionalities in the current graph could eventually be covered by future, more capable LLMs. This brings into question the core motivation behind the paper’s studied topic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lwucYFv5Bo", "reviewer": "Reviewer_MPzp", "summary": "This paper proposes a new system, HIVE, that coordinates instructions and task executions across models. HIVE also explains steps being taken. HIVE incorporates a capability knowledge graph to organize and retrieve model capabilities and it uses PDDL, a formal logic approach to assist with planning. To thoroughly evaluate, they also developed a new multi-modal benchmark, MuSE, consisting of complex, multi-modal queries. They demonstrate that HIVE outperforms competitors such as HuggingGPT and ControlLLM on MuSE.", "questions": "-I suggest making the abstract more crisp and making the distinction of this work very clear right away.\n\n-Figure 2 needs to be cleaned up - time is in seconds? The numbers look crowded in the image.\n\n-How does this dataset compare with the function-calling datasets? Is the key difference that these queries require model calls as solutions (rather than API/function calls)?\n\n-Will MuSE be open source? \n\n-Are there multi-modal query-trajectory datasets in the computer vision community? If so, how do those compare with your multi-modal dataset?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\n-They claim to be the first multiple model approach to leverage PDDL. \n-Their approach consists of intuitive steps, achieved using new ideas such as PDDL for planning and C-KG for model selection.\n-The multi-modal benchmark data set appears to be a new contribution in the multiple model task domain. More clarity here would be great, see my comments below.\n\nQuality\n-The idea of using model cards systematically to leverage new models is very practical and useful. I can see this being helpful for doing this at scale (say you have 1000+ models from which to choose).\n-HIVE outperforms the competing models on their more challenging dataset (MuSE)\n\nClarity\n-Overall, the steps in their system: parse user prompt → plan → select models → execute is clear and intuitive.\n\nSignificance\n-They solve the problem of model selection based on many constraints: hardware (model size), licensing, and model performance. This is going to be increasingly important for multi-agent/multiple model systems.", "weaknesses": "-PDDL was not defined until later in the paper - define it in the abstract and/or introduction.\n-Not clear if MuSE will be openly available\n-The interaction between PDDL and LLM (ChatGPT) in task decomposition could be made clearer - what exactly is the LLM doing and how is it leveraging PDDL? I think since PDDL could be new for readers, being clear here is very important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kJdsuouthf", "reviewer": "Reviewer_HTvY", "summary": "The paper proposes HIVE, a PDDL-based system to fulfill with user requirements with a large set of foundational models. The system can perform complex actions within given constraints\nin an explainable manner.\n\nThe paper also proposes a MUSE benchmark to evaluate the multi-modal performance of such agent systems.", "questions": "+ How many models are selected from Hugging Face as the candidates? Since the community is growing rapidly, will the models from earlier years (with worse performance) be actually selected to perform actions? What's the distribution on model years?\n+ It's commonly agreed that recent API-based large models (e.g. GPT-4 series, Gemini) performs better than open-sourced models available on Hugging Face. Have those models been taken into account?\n+ Many models are very similar in expertise, since recent community mostly focus on *general*-usage foundational model. What's the importance of such system to choose models from a pool? How many models will be actually selected with common instructions (smallest, best)?\n+ Is there any insights why HIVE fails on image-to-audio task? The system should be general enough in modality given the wide availability of multi-modal models, and the 0.0 result can hurt much the overall assessment and actual deployment of HIVE or HIVE-based system.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The papers proposes a method to automatically extract models with capabilities of various modalities from Hugging Face, and explains in details the methodology.\n\nThe automatic system also ease the user query but automatically choose the correct model from the pool as the detailed requirement by user in natural language, which is quite meaningful for the rapid-growing multi-modal foundational models in the community.\n\nExperiments show that even the light version can beat the baselines much except for the image-to-audio case. Experiments on the trustworthiness and robustness also show the advantage of HIVE as an explainable system.", "weaknesses": "Although an automatic exaction system designed, the number of final candidates for HIVE are unknown. Appendix A only mentions a very small sets of models, most of them fall behind SOTA much (e.g. in image generation, machine translation, text generation). (also see questions)\n\nThe paper fails to mention the context why such a system is necessary given that most recent foundational models are for general purpose in its specific modality or multi-modality domain, and user can easily choose a model from public leaderboard or knowledge, if the candidates of HIVE is only a limited set of models which are not necessarily SOTA.\n\nHIVE fails completely (0.0 score) on image-to-audio, 1 of the 6 settings in Table 2, casting doubt on the generalizability of the system.\n\nIt's unclear how many modalities and multi-modality settings are covered in main text without looking into appendix. The list shall be small and should be put into main text to give readers a clearer view of the evaluation settings. There also lacks a concrete example of the possible user query HIVE is designed for until section 4.5, making the paper hard to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)