Tool-Planner: Task Planning with Clusters across Multiple Tools

Authors: Yanming Liu, Xinyue Peng, Jiannan Cao, Shi Bo, Yuwei Zhang, Xuhong Zhang, Sheng Cheng, Xun Wang, Jianwei Yin, Tianyu Du
Primary area: generative models
Keywords: Tool Learning, Task Planning, LLM agent

Abstract:
Large language models (LLMs) have demonstrated exceptional reasoning capabilities, enabling them to solve various complex problems. Recently, this ability has been applied to the paradigm of tool learning. Tool learning involves providing examples of tool usage and their corresponding functions, allowing LLMs to formulate plans and demonstrate the process of invoking and executing each tool. LLMs can address tasks that they cannot complete independently, thereby enhancing their potential across different tasks. However, this approach faces two key challenges. First, redundant error correction leads to unstable planning and long execution time. Additionally, designing a correct plan among multiple tools is also a challenge in tool learning. To address these issues, we propose Tool-Planner, a task-processing framework based on toolkits. Tool-Planner groups tools based on the API functions with the same function into a toolkit and allows LLMs to implement planning across the various toolkits. When a tool error occurs, the language model can reselect and adjust tools based on the toolkit. Experiments show that our approach demonstrates a high pass and win rate across different datasets and optimizes the planning scheme for tool learning in models such as GPT-4 and Claude 3, showcasing the potential of our method. Our code is public at
 https://github.com/OceannTwT/Tool-Planner.

Decision: accept (poster)
Reviewer scores: 5, 8, 5, 6

Reviews and discussion:
<thread>
creation date:
1729872385277

thread_type:
official_review

<thread_object>
creator: reviewer_uyda

<content>
summary:
This paper introduces Tool-Planner, a framework for task planning that uses clustered toolkits as nodes in tree search.

strengths:
1. The idea of using toolkits as nodes in tree-search is novel and intuitive, providing a more structured approach to handling API failures.
2. The experimental results are comprehensive, covering multiple datasets and comparing against various strong baselines.

weaknesses:
1. The method's performance heavily depends on the quality of the clustering algorithm and model. While Table 5 shows end-to-end performance comparisons of different clustering models, there's a lack of intermediate metrics to evaluate clustering quality.

2. The selection of the optimal k value for k-means clustering remains challenging. As is shown in Figure 3, determining an appropriate k value requires exploration and this is actually impractical when facing new datasets.  

3. The fundamental contribution of Tool-Planner can be viewed as introducing a prior into conventional tree-search-based methods - specifically, the assumption that when an API fails, the model should first attempt similar APIs. While this intuition is reasonable and the experimental results are promising, the paper has two key limitations in this aspect:

    **a.** Lacking theoretical analysis or justification for why clustering tools into toolkits leads to better performance. While empirically effective, understanding the theoretical underpinnings could provide insights into when and why this approach works better than alternatives.

    **b.** The paper doesn't explore whether similar benefits could be achieved through simpler means. Notably, the same prior (trying similar APIs first when one fails) could potentially be incorporated directly into LLM prompting. It remains unclear whether the additional complexity of explicit clustering and toolkit-based search is necessary, or if similar performance gains could be achieved by simply instructing the LLM to prefer similar APIs when handling failures.

questions:
See weaknesses. Further:

1. Have the authors considered using more sophisticated clustering methods that don't require pre-specifying the number of clusters?
2. How does Tool-Planner perform when the available APIs change frequently? Is reclustering needed, and if so, how often?

I would consider revising my rating based on the authors' responses to these questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730365240626

thread_type:
official_review

<thread_object>
creator: reviewer_hrkh

<content>
summary:
This paper introduces the Tool-Planner method, a toolkit-based task planning framework designed to enhance the task planning and execution efficiency of LLMs in the context of tool learning. Tool-Planner achieves this by clustering APIs with similar or identical functionalities into toolkits, enabling LLMs to plan across these toolkits. This approach facilitates rapid adjustments and optimization of planning pathways when encountering tool-related errors, thereby improving overall performance.

strengths:
- The proposed method is both intuitive and effective, as it enhances the accuracy and efficiency of LLMs in tool invocation through the clustering of tools and planning at the toolkit level.
- Experimental results on the ToolBench and APIBench datasets substantiate the efficacy of Tool-Planner in improving both success rates and competitive performance.

weaknesses:
- Certain aspects of the methodology require further clarification. For instance, in Section 3.2, the statement "In solving problems for specific states, the model will choose any API within the toolkit for invocation" raises concerns regarding its potential oversimplification. Additionally, since each API necessitates a specific parameter structure for input, how does the model determine the corresponding parameters for an arbitrarily selected API?
- The performance of Tool-Planner is significantly contingent upon the accuracy of the clustering process. Inadequate clustering may hinder the model's ability to identify suitable tools, thereby affecting task completion. Careful tuning of the clustering parameter is essential to achieve optimal performance, which may necessitate additional experimentation and adjustments.
- In the section on "Planning Exploration on Solution Space," it is important to quantify the number of tasks that progress to the "Task Replanning Across Toolkits" stage. Furthermore, what are the corresponding token consumption and computational time for these tasks?
- The ablation study requires further elaboration on the actual configuration of the baseline with Toolkit Integration. Specifically, how is the integration achieved, and what criteria are used to select the toolkit before invoking a specific API? Additionally, an adaptive adjustment method for the number of clusters should be described, along with experimental results validating the use of clustering methods such as DBSCAN for adaptive clustering.
- In the "Efficiency Evaluation" section, it is noted that the runtime of Tool-Planner is significantly higher than that of other algorithms outside of DFSDT. I would like to see a comprehensive analysis of the number of LLM invocations and token usage statistics associated with different methods. This would provide a more thorough assessment of method efficiency. Furthermore, analyzing the impact of cluster quantity selection on execution efficiency would be a valuable addition to the discussion.

questions:
Please refer to the Weaknesses section for details. I’m open to discussion and increasing my score if my comments are addressed satisfactorily.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730526950921

thread_type:
official_review

<thread_object>
creator: reviewer_twku

<content>
summary:
This paper introduces a new framework, Tool-Planner, which formulates LLM planning with API tools as a tree-search over clusters of tools, or 'toolkits', generated using SimCSE. Experiments and ablations show the soundness of this approach over its closest relative, DFSDT, which does a tree-search over single tools.

strengths:
$Originality:$ This paper builds upon DFSDT and introduces a toolkit which provides better performance.

$Quality$: The experiments cover domains with many APIs, relevant baselines, and ablations which showcase the importance of the design choices.

$Clarity$: The paper is presented very clearly, especially methodology. My only nitpick is Figure 2 - my first  impression was that this was another Toolchain [1] paper because the search seems to be over single APIs rather than a selection over toolkits and then the most relevant API in the toolkit.

$Significance$: LLM abilities are empowered as they gain access to new tools but they also perform poorly in reasoning tasks with large action spaces. This paper provides a clever trick to cluster tools together, reducing the LLM decision space and simplifying the planning task. This will ensure that LLM performance in domains with tools remains relatively high regardless of the future introduction of many new tools.

[1] ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Search (Zhuang et al. 2023)

weaknesses:
- Table 1's feature categories are vague (what is Tool Integration?) and don't make it obvious how your work  clearly contrasts from DFSDT. "Tool Clustering" distinguishes this work more from DFSDT.
- Figure 2 looks like DFSDT or ToolChain from a glance; it is not obvious that the nodes were chosen from toolkits. Adding more emphasis / text on the toolkits could make things clearer.
- Toolchain is compared against in Table 1 but missing from the experiments

questions:
- How were the "Explainable Planning" and "Tool Integration" categorizations decided?
- What does "Tool Integration" refer to?
- How often do you observe failures where prioritizing APIs within the same toolkit leads to unnecessary expansions (due to the appropriate API being in another toolkit)?
- Why is ToolChain compared against in Table 1 but not in the experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731086714273

thread_type:
official_review

<thread_object>
creator: reviewer_l3vz

<content>
summary:
This paper proposes clustering similar tool APIs via LLMs and embeddings to enhance LLM planning efficiency in tool use. The ToolPlanner framework effectively addresses tool planning and invocation tasks using toolkits. Empirical results demonstrate the framework's superiority in performance and efficiency compared to multiple baselines, while ablations confirm the effectiveness of tool clustering.

strengths:
1. The paper is well-written and easy to follow.
2. Comprehensive ablation studies on the core component, the toolkit.
3. Detailed appendix with experimental specifics and case studies.
4. Insightful error analysis.

weaknesses:
1. The paper could include more comparisons with tree-based inference methods in LLMs, e.g., [1][2] in related work.
2. Consider comparing with ToolChain*[3].

questions:
1. I suggest a name change for the framework to better reflect its toolkit-based planning, e.g., ToolkitPlanner.
2. Beyond Figure 4, what is the computation cost/wall time for clustering these APIs?
3. Line 217: How is another available API t′ selected within the same toolkit?
4. Line 259: Which LLM is used to assess Win Rate?
5. Line 262: Could the authors provide more details on detecting and measuring hallucinations?
6. The novelty is somewhat limited, as noted in Line 375, where the method is derived from DFSDT by replacing API nodes with toolkits. The authors could elaborate more in Sec 3.3 and include task replanning experiments to better distinguish this framework from DFSDT. While on the other hand, extensive results and ablations strengthen confidence in the framework's efficiency and effectiveness

 [1] Zhou, Andy, et al. "Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models." Forty-first International Conference on Machine Learning.

 [2] Feng, Xidong, et al. "Alphazero-like tree-search can guide large language model decoding and training." arXiv preprint arXiv:2309.17179 (2023).

 [3] Zhuang, Yuchen, et al. "ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Search." The Twelfth International Conference on Learning Representations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733172170823

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response for Readers(2/2)

comment:
|                                  | G1     | G2     | G3     | TorchHub | HuggingFace |
|----------------------------------|--------|--------|--------|----------|-------------|
| Avg. API execution Token Cost    | 243.78 | 282.51 | 260.98 | 158.43   | 131.22      |
| Avg. API execution Time Cost(ms) | 2610   | 2446   | 2371   | 4251     | 3348        |
| Avg. re-planning token Cost      | 553.22 | 520.91 | 597.12 | 498.33   | 503.41      |
| Avg. re-planning Time Cost(ms)   | 4210   | 3899   | 4732   | 3871     | 3688        |
| re-planning proportion           | 6.6%   | 7.3%   | 8.4%   | 4.2%     | 7%          |


> 3. Impact of Other Clustering Algorithms (e.g., DBSCAN)

DBSCAN remains robust and highly effective within our framework. Its advantage lies in accurately grouping tools with similar functions. Compared to k-means++, DBSCAN performs better under stricter constraints, maintaining strong performance. However, overly stringent constraints may reduce the number of tools meeting these criteria, potentially degrading performance to a level similar to DFSDT. Therefore, in performance-critical scenarios, DBSCAN achieves better results when the number of clusters is appropriate. Conversely, in scenarios emphasizing overall efficiency and consistency, k-means++ ensures stable performance and time efficiency throughout. The corresponding results are shown below:

Win Rate:

|                            | $$\epsilon$$ | G1-Inst  | G2-Inst  | G3-Inst  | TorchHub | HuggingFace | TensorFlow |
|----------------------------|----------|----------|----------|----------|----------|-------------|------------|
| Tool-Planner(k-means best) | -        | 63.3     | **72.3** | 67.5     | 77.6     | 75.4        | **72.4**   |
| Tool-Planner(DBSCAN)       | 0.001    | 57.5     | 68.3     | 66.3     | 75.1     | 72.8        | 65.2       |
| Tool-Planner(DBSCAN)       | 0.01     | 61.8     | 71.8     | 66.5     | 77.3     | 76.5        | 70.3       |
| Tool-Planner(DBSCAN)       | 0.02     | **64.3** | 70.3     | **68.0** | **78.5** | **77.2**    | 71.9       |
| Tool-Planner(DBSCAN)       | 0.04     | 63.5     | 69.8     | 68.3     | 77.0     | 73.1        | 69.7       |
| Tool-Planner(DBSCAN)       | 0.08     | 59.5     | 59.3     | 66.0     | 73.2     | 65.8        | 61.2       |
| Tool-Planner(DBSCAN)       | 0.16     | 50.5     | 45.5     | 57.5     | 65.1     | 56.9        | 55.9       |

Finally, we deeply appreciate the reviewers' efforts and the recognition of our work. If there are further questions or suggestions regarding our methodology, please feel free to response.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733172264561

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response for Readers(1/2)

comment:
Thank you all for your thorough review and valuable suggestions on our paper. We believe that these recommendations will not only contribute to improving the paper in the future but also have a profound impact on the adoption of the proposed method. Here, we aim to re-emphasize the advantages and unique aspects of our work while comprehensively addressing the common concerns raised by reviewers and readers:

> 1. Further Advancement in Tool Learning

Our method represents a significant step forward in the field of tool learning. Rather than simply providing tools to LLMs and allowing them to choose, we propose selecting the most representative toolkit from a broader range of options and utilizing it for problem-solving and planning. This approach enables efficient identification of usable tools during planning and maintains stability throughout the reasoning process. It minimizes error propagation in long reasoning chains, thereby ensuring higher output quality. This method extends the strategy-learning paradigm in planning while retaining efficient inference capabilities.

> 2. Clustering as a Planning Strategy in Tool Learning

Our clustering-based approach aligns well with realistic planning processes and addresses the inefficiencies seen in existing methods, such as DFSDT and AnyTool (representing strategy-learning approaches), and heuristic methods like ToolChain*. Specifically, it reduces the high latency caused by frequent calls to multiple tools in large action spaces. For scenarios involving thousands of tools, our method achieves an optimal balance between efficiency and execution time. This has been validated on the two most commonly used datasets for tool learning with the largest number of tools available. Furthermore, the clustering mechanism ensures scalability, limiting additional overhead even as the number of tools grows.

__Common Concerns from Reviewers__:

> 1. Performance Comparison with Heuristic Planning Methods (e.g., ToolChain*)

ToolChain* relies on the A* algorithm and performs multiple tool calls during path exploration. Experiments in the original ToolChain* paper were conducted in relatively small categories with a limited number of tools (only 10-100, as detailed in Section D.3 of their paper). While ToolChain* performs well in such scenarios, its performance degrades significantly, even becoming ineffective, when the number of tools increases—such as in real-world applications or comprehensive benchmarks like APIBench and ToolBench. Corresponding experimental results are provided below:

Win Rate: 
| Model  | Method       | Home Search | G1-Inst | G1-Tool | G1-Cat | G2-Inst | G2-Cat | G3-Inst |
|-|-|-|-|-|-|-|-|-|
| GPT3.5 | DFSDT        | 71.0  | 58.3    | 59.8    | 56.8   | 70.5    | 59.3   | 68.0    |
| GPT3.5 | ToolChain*   | 69.0  | 56.3    | 58.5    | 53.0   | 67.5    | 58.0   | 61.5    |
| GPT3.5 | **Tool-Planner** | 75.0  | 63.3    | 63.5    | 61.3   | 72.3    | 63.5   | 67.5    |
| GPT4   | DFSDT        | 73.0  | 65.8    | 69.3    | 65.3   | 72.0    | 56.8   | 81.5    |
| GPT4   | ToolChain*   | 76.0  | 38.5    | 70.5    | 64.0   | 73.3    | 54.0   | 75.5    |
| GPT4   | **Tool-Planner** | 79.0  | 75.5    | 75.8    | 71.8   | 79.8    | 70.3   | 92.0    |

Pass Rate

| Model  | Method       | Home Search | G1-Inst | G1-Tool | G1-Cat | G2-Inst | G2-Cat | G3-Inst |
|-|-|-|-|-|-|-|-|-|
| GPT3.5 | DFSDT        | 86.0        | 48.5    | 62.5    | 58.0   | 71.5    | 70.5   | 61.0    |
| GPT3.5 | ToolChain*   | 90.0        | 22.5    | 28.0    | 27.0   | 32.5    | 28.0   | 17.0    |
| GPT3.5 | **Tool-Planner** | 90.0        | 58.5    | 71.0    | 66.0   | 75.5    | 78.0   | 66.0    |
| GPT4   | DFSDT        | 87.0        | 57.0    | 72.0    | 64.5   | 77.5    | 69.5   | 71.5    |
| GPT4   | ToolChain*   | 91.0        | 31.8    | 33.5    | 27.5   | 41.0    | 31.8   | 19.5    |
| GPT4   | **Tool-Planner** | 92.0        | 66.0    | 78.5    | 75.0   | 83.5    | 77.5   | 83.0    |

> 2. Cost of Tool-Planner in Terms of Computation

This is a key advantage of our approach. Compared to DFSDT, our method significantly reduces the number of re-planning instances across toolkits. Since re-planning inherently relies on context, token consumption during this phase is inevitably high, making optimization essential. Large language models often exhibit a strong dependency on prior content during task planning, adding new information at the end or splitting intermediate steps. This leads to unnecessarily lengthy reasoning processes and increases the likelihood of errors in intermediate information. However, Tool-Planner effectively mitigates these issues, providing significant advantages in time and efficiency. These benefits are especially notable in reducing re-planning time and minimizing error propagation caused by re-planning. Relevant experiments are as follows:
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission introduces Tool(kit?)-Planner, an approach that helps LLMs with planning chains of tool use by first clustering similar tools into "toolkits" and then running search in the space of toolkits. The paper provides a useful error analysis (section 4.4)  and an empirical evaluation that shows the approach provides significant gains in task completion rates compared to search over the original space of tools.

Strengths:

- Simple and intuitive idea, easy to implement
- Insightful empirical evaluation 
- Potential for impact
- Clarity of the submission

Weaknesses:

- The dependence of performance on tool clustering quality, including the choice of the number of clusters

The metareviewer recommends this work for publication due to the practicality and effectiveness of its approach. Tool-Planner's dependence on clustering quality is indeed a conceptual issue, but in practice, after the clustering is tuned once from scratch for a given system, the clusters can be subsequently updated incrementally as new tools become available, which is generally a less error-prone process.