From an LLM Swarm to a PDDL-empowered Hive: Planning Self-executed Instructions in a Multi-modal Jungle

Authors: Kaustubh Vyas, Damien Graux, Yijun Yang, Sebastien Montella, Chenxin Diao, Wendi Zhou, Pavlos Vougiouklis, Ruofei Lai, Yang Ren, Keshuang Li, Jeff Z. Pan
Primary area: applications to robotics, autonomy, planning
Keywords: Deep Models, Planning, PDDL, Knowledge Graphs, Benchmark, Large Language Models

Abstract:
In response to the call for agent-based solutions that leverage the ever-increasing capabilities of the deep models' ecosystem, we introduce a comprehensive solution for selecting appropriate models and subsequently planning a set of atomic actions to satisfy the end-users' instructions.

Our system, Hive, operates over sets of models and, upon receiving natural language instructions, schedules and executes, explainable plans of atomic actions. These actions can involve one or more of the available models to achieve the overall task, while respecting end-users specific constraints. Hive is able to plan complex chains of actions while guaranteeing explainability, using an LLM-based formal logic backbone empowered by PDDL operations. We introduce the MuSE benchmark in order to offer a comprehensive evaluation of the multi-modal capabilities of agent systems. Our findings show that our framework redefines the state-of-the-art for task selection, outperforming other competing systems that plan operations across multiple models while offering transparency guarantees while fully adhering to user constraints.

Decision: accept (poster)
Reviewer scores: 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730071998487

thread_type:
official_review

<thread_object>
creator: reviewer_htvy

<content>
summary:
The paper proposes HIVE, a PDDL-based system to fulfill with user requirements with a large set of foundational models. The system can perform complex actions within given constraints
in an explainable manner.

The paper also proposes a MUSE benchmark to evaluate the multi-modal performance of such agent systems.

strengths:
The papers proposes a method to automatically extract models with capabilities of various modalities from Hugging Face, and explains in details the methodology.

The automatic system also ease the user query but automatically choose the correct model from the pool as the detailed requirement by user in natural language, which is quite meaningful for the rapid-growing multi-modal foundational models in the community.

Experiments show that even the light version can beat the baselines much except for the image-to-audio case. Experiments on the trustworthiness and robustness also show the advantage of HIVE as an explainable system.

weaknesses:
Although an automatic exaction system designed, the number of final candidates for HIVE are unknown. Appendix A only mentions a very small sets of models, most of them fall behind SOTA much (e.g. in image generation, machine translation, text generation). (also see questions)

The paper fails to mention the context why such a system is necessary given that most recent foundational models are for general purpose in its specific modality or multi-modality domain, and user can easily choose a model from public leaderboard or knowledge, if the candidates of HIVE is only a limited set of models which are not necessarily SOTA.

HIVE fails completely (0.0 score) on image-to-audio, 1 of the 6 settings in Table 2, casting doubt on the generalizability of the system.

It's unclear how many modalities and multi-modality settings are covered in main text without looking into appendix. The list shall be small and should be put into main text to give readers a clearer view of the evaluation settings. There also lacks a concrete example of the possible user query HIVE is designed for until section 4.5, making the paper hard to understand.

questions:
+ How many models are selected from Hugging Face as the candidates? Since the community is growing rapidly, will the models from earlier years (with worse performance) be actually selected to perform actions? What's the distribution on model years?
+ It's commonly agreed that recent API-based large models (e.g. GPT-4 series, Gemini) performs better than open-sourced models available on Hugging Face. Have those models been taken into account?
+ Many models are very similar in expertise, since recent community mostly focus on *general*-usage foundational model. What's the importance of such system to choose models from a pool? How many models will be actually selected with common instructions (smallest, best)?
+ Is there any insights why HIVE fails on image-to-audio task? The system should be general enough in modality given the wide availability of multi-modal models, and the 0.0 result can hurt much the overall assessment and actual deployment of HIVE or HIVE-based system.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730663282042

thread_type:
official_review

<thread_object>
creator: reviewer_mpzp

<content>
summary:
This paper proposes a new system, HIVE, that coordinates instructions and task executions across models. HIVE also explains steps being taken. HIVE incorporates a capability knowledge graph to organize and retrieve model capabilities and it uses PDDL, a formal logic approach to assist with planning. To thoroughly evaluate, they also developed a new multi-modal benchmark, MuSE, consisting of complex, multi-modal queries. They demonstrate that HIVE outperforms competitors such as HuggingGPT and ControlLLM on MuSE.

strengths:
Originality
-They claim to be the first multiple model approach to leverage PDDL. 
-Their approach consists of intuitive steps, achieved using new ideas such as PDDL for planning and C-KG for model selection.
-The multi-modal benchmark data set appears to be a new contribution in the multiple model task domain. More clarity here would be great, see my comments below.

Quality
-The idea of using model cards systematically to leverage new models is very practical and useful. I can see this being helpful for doing this at scale (say you have 1000+ models from which to choose).
-HIVE outperforms the competing models on their more challenging dataset (MuSE)

Clarity
-Overall, the steps in their system: parse user prompt → plan → select models → execute is clear and intuitive.

Significance
-They solve the problem of model selection based on many constraints: hardware (model size), licensing, and model performance. This is going to be increasingly important for multi-agent/multiple model systems.

weaknesses:
-PDDL was not defined until later in the paper - define it in the abstract and/or introduction.
-Not clear if MuSE will be openly available
-The interaction between PDDL and LLM (ChatGPT) in task decomposition could be made clearer - what exactly is the LLM doing and how is it leveraging PDDL? I think since PDDL could be new for readers, being clear here is very important.

questions:
-I suggest making the abstract more crisp and making the distinction of this work very clear right away.

-Figure 2 needs to be cleaned up - time is in seconds? The numbers look crowded in the image.

-How does this dataset compare with the function-calling datasets? Is the key difference that these queries require model calls as solutions (rather than API/function calls)?

-Will MuSE be open source? 

-Are there multi-modal query-trajectory datasets in the computer vision community? If so, how do those compare with your multi-modal dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730881813079

thread_type:
official_review

<thread_object>
creator: reviewer_xaqh

<content>
summary:
The paper presents Hive, an agent-based system that selects and coordinates models to execute complex tasks based on user instructions. Hive creates explainable plans using atomic actions across various models, guided by formal logic and PDDL operations, ensuring transparency and adherence to user constraints. The authors also introduce the MuSE benchmark to evaluate multimodal capabilities in agent systems. Results show that Hive outperforms other systems in task selection and transparency.

strengths:
The KG construction method is thorough, utilizing model metadata from multiple sources to enhance graph quality. The entire planning pipeline demonstrates a clear improvement over prior work, such as HuggingGPT. Additionally, the writing is well-structured and easy to follow, which supports reader comprehension.

weaknesses:
This paper has several notable weaknesses:

First, it feels like the approach relies heavily on leveraging a powerful LLM to decompose user tasks based on key attributes, selecting open-source models from Huggingface, and executing plans sequentially. In essence, the paper presents a tool-usage framework where the tools are Huggingface models. While there are some unique designs for processing and indexing these models, the contribution appears marginal. The problem-solving paradigm closely resembles prior works, such as HuggingGPT. Additionally, the paper lacks baseline comparisons, particularly in evaluating why their proposed KG construction method is optimal or how it could be further applied in other scenarios.

The experimental design raises concerns as well. Although the authors introduce a new evaluation benchmark, it is based on an in-house dataset. It’s unsurprising that the method performs well on this dataset, but it lacks validation on standard benchmarks, such as classical VQA datasets for multimodal evaluation.

On a broader level, the necessity of maintaining a Huggingface model graph is questionable. As more cheap and advanced models (e.g., GPT-5, LLaMA-4, lower-cost GPT-4o) emerge with capabilities for comprehensive task handling, the reliance on a graph of smaller, domain-specific models may become obsolete. While a graph of non-LLM-based APIs remains valuable (e.g., calculators outperforms LLMs in calculation), most small LLM functionalities in the current graph could eventually be covered by future, more capable LLMs. This brings into question the core motivation behind the paper’s studied topic.

questions:
Could you please test the pipeline or more baselines on more widely used benchmarks in multimodal domains?

Would you provide more examples about how the KG looks like and an entire planning example from scratch?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731711702020

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Note

comment:
Dear Reviewers,

**1)** We would like to express our gratitude for your time and careful consideration of our paper.

**2)** We have noted the common feedback regarding the clarity of the article's main motivation and claims. To facilitate further discussion, here is a clarification:

> The primary aim of HIVE is to provide a unified and explainable solution for responding to natural language (NL) instructions that may involve multi-modal inputs and expect multi-modal outputs. The system respects user conditions across a variety of metrics, not solely performance-based ones. This necessitates distributing sub-instructions to potentially different and specialized models. To achieve this, we structured the landscape of available models to select the most suitable one for each task, utilizing our C-KG. Addressing the explainability challenge, we chose to shift the planning actions from the model space—where competitors focus—to the formalism of PDDL, a planning language widely used in robotics. This transition enables us to clearly understand the steps HIVE takes to formulate an answer.

> Additionally, we introduced a new benchmark, MuSE, because existing baselines do not provide a shared benchmark. MuSE enables the evaluation of complex, integrated user queries encompassing multiple tasks, crucial for real-world applications. Recognizing the lack of standard public benchmarks for such tasks, we constructed MuSE around some of the most popular tasks (15 in total). While not holistic, it effectively assesses our proposed framework, and we welcome future contributions to expand and enhance the benchmark.

As recommended by `Reviewer MPzp`, we intend to revise the Introduction and abstract to emphasize our motivation more effectively.

**3)** To provide comprehensive responses, we have organized our feedback into dedicated posts addressing each reviewer’s comments and questions.

Thank you again for your insightful feedback.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732707466351

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Article

comment:
Dear ACs, SACs, PCs, and Reviewers,  

We appreciate the opportunity to address the reviewers' feedback and hope our responses have clarified any concerns raised. Below, we highlight the key strengths of our work as acknowledged by the reviewers and outline the improvements made in the revised version of our article.  

---

### **Key Strengths of Our Method:**  

1. **Innovative Knowledge Graph Construction:**  
   Our approach integrates model metadata from diverse sources, significantly enhancing the quality and depth of the resulting knowledge graph. The proposed planning pipeline demonstrates substantial improvements compared to previous works, such as HuggingGPT.  

2. **Novel task planning:**  
   The systematic application of PDDL for planning with LLMs, combined with the C-KG framework for model selection, represents a novel contribution in the multi-model task domain. Reviewers particularly highlighted the originality of using PDDL in this context, along with the introduction of a comprehensive multi-modal benchmark dataset.  

3. **Clear and Intuitive System Structure:**  
   The system’s structured approach—encompassing user prompt parsing, planning, model selection, and task execution—efficiently addresses complex model selection challenges. By incorporating constraints like hardware availability, licensing, and performance, our method advances the state of multi-agent systems. Additionally, leveraging model cards for integrating new models ensures scalability, which is crucial when navigating extensive model libraries.  

4. **Robust Experimental Results:**  
   Experiments conducted using MuSE demonstrate that even a lightweight version of our system outperforms existing state-of-the-art solutions in most scenarios. The HIVE framework, in particular, has shown notable trustworthiness and robustness.  

---

### **Revisions Made:**  

In response to the feedback received, we have revised our manuscript and uploaded the updated version. The key modifications include:

- Rephrased the **Abstract** and **Introduction** for clarity and conciseness.  
- Added a bit of more background related to **PDDL** in the introduction.  
- Enhanced **Footnote 10** to include the multi-modality settings of MuSE.  
- Restructured **Appendix B** for better readability.  
- Added section references in **Appendix C**.  
- Included **C-KG metadata** in Appendix D.  
- Redesigned **Figure 1** and improved **Figure 2** for better visual representation.  

---

We sincerely appreciate your time and effort in reviewing our work. Your valuable feedback has been instrumental in strengthening our contribution.

Thank you and best regards,  
—  
The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a method to use many small (possibly domain specific) models to execute low level actions in an environment. While this concept of high level planning to low level actions with LLMs is not new by itself, the paper does appear to situate itself well with the existing literature and proposes a seemingly novel method with reasonable results.

Many of the reviewers' concerns that are not directly addressed by the authors are rather ideological in nature ("why bother with this line of work using many domain specific models if hypothetical models like GPT-5 or Llama 4 can just do it all one day" is one example that stood out to me) and are either invalid or not relevant to the topic of this paper's contributions to the current state of AI.