Venue: NAACL 2025

# EasyTool: Enhancing LLM-based Agents with Concise Tool Instruction

Siyu Yuan\({}^{1}\), Kaitao Song\({}^{2}\), &Jiangjie Chen\({}^{1}\), Xu Tan\({}^{2}\), Yongliang Shen\({}^{3}\), Kan Ren\({}^{2}\), Dongsheng Li\({}^{2}\), Deqing Yang\({}^{1}\)

Fudan University\({}^{1}\), Microsoft Research Asia\({}^{2}\), Zhejiang University

syyuan21@m.fudan.edu.cn, {kaitaosong, xuta, dongsli}@microsoft.com syl@zju.edu.cn, {jjchen19,yangdeqing}@fudan.edu.cn

###### Abstract

There has been a rising interest in utilizing tools in applications of autonomous agents based on large language models (LLMs) to address intricate real-world tasks. To develop LLM-based agents, it usually requires LLMs to understand many tool functions from different tool documentations. However, these documentations could be diverse, redundant, or incomplete, which immensely affects the capability of LLMs in using tools. Current LLMs exhibit satisfactory instruction-following capabilities based on instruction-following fine-tuning process. Motivated by this, in this paper, we introduce EasyTool, a framework transforming diverse and lengthy tool documentation into a unified and concise tool instruction to fully leverage instruction-following capabilities of LLMs for easier tool usage. EasyTool purifies essential information from extensive tool documentation of different sources, and elaborates a unified interface (_i.e._, tool instruction) to offer standardized tool descriptions and functionalities for LLM-based agents. Extensive experiments on multiple different tasks demonstrate that EasyTool can significantly reduce token consumption and improve the performance of LLM-based agents on tool utilization in real-world scenarios. Our code is available in supplemental materials. Our code is available at [https://github.com/microsoft/JARVIS/tree/main/easytool](https://github.com/microsoft/JARVIS/tree/main/easytool).

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding Authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote†: footnotetext: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding authors.


tended to widely deployed black-box LLMs, and lack flexibility in integrating external tools in a plug-and-play way. Another line of work (Song et al., 2023; Lu et al., 2023; Xu et al., 2023; Chen et al., 2023; Wang et al., 2024) retrieves and calls external tools by providing tool documentation and few-shot demonstrations of tool functionality. However, these methods struggle with limited context length and face difficulties when handling unusual tools, and thus hinder the development of an omnipotent LLM-based agent. Therefore, extensive effort is still required to efficiently and effectively improve the quality of tool utilization.

For tool utilization, tool documentation plays an indispensable component, which could include multiple meta information like tool descriptions, tool parameters, demonstrations and so on. However, as shown in Figure 1, we summarize the issues from existing documentation that could hinder the tool utilization of LLM-based agents:

* _Inconsistency_: Massive tools from different sources often have inconsistent and diverse documentation formats, posing new challenges for LLMs to understand;
* _Redundancy_: Tool documentation could encompass massive redundant and useless information, making it harder to grasp tool functionality and resulting in excessive token consumption in prompts;
* _Ineompleteness_: We expect the tool documentation to provide useful information to describe its functions, parameters and demonstrations for instructions. However, the absence of critical information in some tool documentations impedes effective tool utilization.

Overall, we regard the information provided by tool documentation as a critical element in instructing LLMs to use tools. However, the above issues in tool documentation bring some challenges to LLM-based agents to understand, especially considering the increasing of massive and diverse tools from different domains. Therefore, how to parse the documentation, extract the most essential information and provide a unified format has become a necessary topic to effectively use tools.

Recent LLMs, such as GPT-4 (OpenAI, 2023), ChatGPT (OpenAI, 2022), Vicuna (Chiang et al., 2023), and Mistral (Jiang et al., 2023a), demonstrate strong instruction-following capabilities due to their fine-tuning for this skill (Wang et al., 2023; Ren et al., 2024). Inspired by this, in this paper, we introduce EasyTool, an easy and effective method to create clear, structured, and unified instructions from tool documentations for improving LLM-based agents in using tools. High-quality tool instructions should follow two criteria: easy to 1) understand its functionality for selection and 2) predict its parameters for usage. To this end, we first collect massive tool documentations from different sources (_e.g._, RestBench (Song et al., 2023) and ToolBench (Qin et al., 2023)). Instead of directly using these various tool documentations with different complicated structures, we transform these documentations into a more concise and unified tool instruction, which includes standard _tool descriptions_ and guidelines for _tool functionality_. The converted tool descriptions can eliminate irrelevant content and only keep the core functionality of each tool for LLMs to attend to. Moreover, EasyTool provides detailed information for tool usage (_e.g._, its parameters with demonstrations generated by ChatGPT (OpenAI, 2022)) in tool functionality guidelines to instruct LLMs with tool usage.

Extensive experiments on multiple tool-usage benchmarks demonstrate these concise tool instructions generated by EasyTool can significantly reduce incorrect tool usage. Furthermore, we also prove that the capability of EasyTool can be generalized to open-source LLMs in a plug-and-play way and greatly improve their performance on tool utilization in different real-world tool-usage scenarios. Our contributions can be summarized as:

* We analyze and explore the limitations of current tool utilization in LLM-based agents and first point out the deficiencies of tool documentation that hinder LLMs in using tools.
* To address these issues, we propose EasyTool, which creates high-quality tool instructions from documentation to facilitate tool usage in LLM-based agents.
* Experimental results on three datasets from distinct domains show that our EasyTool effectively and efficiently improves the capability of LLMs in tool utilization.

## 2 Related Work

With the emergence of powerful LLMs (OpenAI, 2023; Touvron et al., 2023a,b), using tools has been 

considered a new trend to enhance the capabilities of LLMs A conventional strategy is to build synthetic data (Schick et al., 2023; Qin et al., 2023; Li et al., 2023; Patil et al., 2023; Shen et al., 2023) involved tool use and then fine-tune LLMs to generate text with tool invocation. However, these methods cannot be extended to some powerful closed LLMs, and lack the capability to use new tools. Although some methods Hao et al. (2023) attempted to fine-tune LLMs to obtain tool embeddings for plug-and-play usage, they still require additional data for training to get tool embeddings.

Therefore, there has arisen another branch (Shen et al., 2023; Song et al., 2023; Gravitas, 2023) that directly used LLMs as the controller and feed tool descriptions into prompts to instruct LLMs to understand and call tools. These methods do not need extra training and can use external tools in a plug-and-play paradigm, but they are limited to context sizes and the quality of tool documentation. As a result, these methods will lead to some failed or incorrect tool invocation (Zhang et al., 2023a). Some work (Hsieh et al., 2023; Xu et al., 2023) attempts to revise tool documentation to support a zero-shot tool utilization, but some inherent issues of tool documentation in real-world scenarios still hinder the effective and efficient usage of many tools. Besides, different from naive prompt compression (Mu et al., 2023; Jiang et al., 2023b), which is only suitable to compress plain prompt, the streamlined information from tool documentation should satisfy specific format and need to confirm the accuracy of tool invocation when processing user requests.

## 3 Preliminary

### Task Formulation

Motivated by previous works (Shen et al., 2023a; Song et al., 2023), just as shown in Figure 2, the pipeline of LLM-based agents for tool utilization can be summarized as a four-stage framework as:

* **Task Planning:** Agents analyze a user request \(T\) and decompose it into subtasks \(T=\{t_{1},t_{2},...,t_{n}\}\) with specific dependencies and execution orders, each optimized for execution with a single tool.
* **Tool Retrieval:** Here, the focus is on matching these subtasks with suitable tools from the tool inventory based on the similarity between the subtasks and tools. The aim is to select the top-K tools \(\{a_{1},a_{2},...,a_{K}\}\), that have the highest similarity to each subtask, forming a set of candidate tools for each.
* **Tool Selection:** In this stage, the most appropriate tool for each subtask from the set \(\{a_{1},a_{2},...,a_{K}\}\) is chosen based on its description. This stage also includes preparing the parameters for tool execution, as specified in its document.
* **Tool Execution:** After tool selection and parameter setup, the tool is executed. If a tool fails during execution, the process reverts to the tool selection stage for an alternative choice. This retry mechanism continues until successful execution or until the maximum trial \(R\) is reached.

After these stages, agents can orchestrate different tools and use their powers to generate the final answer for each user request.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Dataset** & \multicolumn{2}{c}{**Token\({}_{\text{Spec.}}\)****Token\({}_{\text{Dec.}}\)****Exp.**} \\ \hline RestBench (Song et al., 2023) & 58 & 3,881 & ✗ \\ Gorilla (Patil et al., 2023) & 88 & 284 & ✗ \\ ToolAipaca (Tang et al., 2023) & 567 & 7,661 & ✗ \\ ToolBench (Qin et al., 2023) & 744 & 2,530 & ✗ \\ HFmodels (Shen et al., 2023a) & 777 & 1,196 & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 1: The statistics of tool documentations in tool benchmarks. We report the average length of the tool description with parameters (\(\mathbf{Token_{pose.}}\) �), the average length of the tool documentations (\(\mathbf{Token_{pose.}}\) �) and whether the benchmarks have tool usage scenarios and example (**Exp.**).

Figure 2: The four-stage framework of LLM-based agents in tool-usage applications.

 



### Analysis

Previous studies typically adhere to the established paradigm, instructing LLMs with the tool documentation to use tools. However, relying on the tool documentation can hinder the performance of LLM-based agents due to its inherent limitations.

InconsistencyIn the real world, a wide variety of tools from different sources results in substantial diversity in terms of format, style, and guidelines. As a result, this diversity contributes to a mess of tool documentations without a cohesive and standardized structure, posing a significant challenge for LLMs to effectively use these tools.

RedundancyGenerally, the tool documents from different communities usually contain redundant information (_e.g._, URLs, IDs, etc.). In practical application, we just require LLMs to understand the core function of the tool and then decide whether to use and how to use this tool. As shown in Table 1, we analyze multiple tool-based benchmarks and the results reveal a high proportion of redundant information in many tool documentations. For example, the average length of tool documentations used in ToolBench is approximately 2,530 tokens in Table 1.1 This useless information can severely hinder LLMs from retrieving and selecting tools, leading to an incorrect tool invocation. Moreover, LLMs are constrained by a maximum context length, yet tool documentation is typically lengthy. This excessive length can limit the range of tool options available for LLMs to consider, posing a challenge for efficient tool selection.

Footnote 1: We adopt cl10@k_base encoding. The code is in [https://github.com/openai/tiktoken](https://github.com/openai/tiktoken).

IncompletenessPrevious work has demonstrated that LLMs may pass invalid parameters, leading to tool execution failure Song et al. (2023); Qin et al. (2023); Zhang et al. (2023a). As shown in Table 1, unlike human-oriented instruction manuals that provide usage scenarios and examples, existing tool documentation typically lacks such context, only offering example codes for tool invocation or results. This leads to LLMs struggling to know when and how to refer to the examples to pass the correct parameters, resulting in invalid parameters.

## 4 Method

As aforementioned, polishing, streamlining, and enhancing the tool documentation is important to improve tool utilization in LLVM-based agents. Currently, most of LLMs OpenAI (OpenAI, 2022, 2023; Jiang et al., 2023a; Chiang et al., 2023) exhibit satisfactory instrution-following capability due to instruction-following fine-tuning process Wang et al. (2023); Ren et al. (2024). Motivated by this, we introduce EasyTool, a simple method to condense tool documentation into more concise and effective tool instructions to improve LLM-based agents in tool utilization. The overall workflow is illustrated in Figure 1. EasyTool comprises two stages: the first stage is to re-organize the original tool documentation by eliminating the irrelevant information and only summarizing the multiple built-in function description of each tool (SS 4.1). For each tool, we further design a functional guideline instruction for LLMs and enable LLMs to further refine the tool documentation by providing parameters of each tool and simultaneously the examples to instruct LLMs for usage (SS 4.2).

### Tool Description Generation

As described above, tool documentation usually includes plenty of irrelevant information, making it difficult to understand practical usage for LLMs. Moreover, some tools that enable multiple built-in functions for different scenarios are not always comprehensively described. For example, Google Maps offers both distance calculations and coordinate provision, but its description might not cover all functionalities. To address this, we expect to use LLMs to polish and streamline these tool documentations and decode them into more concise and effective tool descriptions. Here, just as shown in Table 2 (I), we design an instruction and require LLMs (_i.e._, ChatGPT) to convert tool documentation to summarize its general purpose by following the designed instruction. Then, for each function in tool documentation, we ask the LLM to generate the functionality of it. We also add extra demonstrations to enhance the instruction-following of LLMs in parsing tool documentation.

### Tool Functionality Guidelines Construction

The tool descriptions generated in the previous step aid LLMs in tool retrieval and selection. However, we still need to predict the correct parameters of each tool for a successful execution. Previous work Qin et al. (2023); Zhang et al. (2023a); Xu et al. (2023) also confirms that many open-source LLMs are still inadequate in executing tools, re

sulting in parameter errors. Therefore, we further polish our tool descriptions in the first stage to supplement the parameters in the tool instructions. Here, we design another instruction that requires LLMs to extract parameters from tool documentation and then organize it into a structured output, thus facilitating LLMs to invoke tools. As shown in Table 2 (II), we use ChatGPT to create examples, including scenarios and parameter names with values to demonstrate how to input parameters for different scenarios and enhance LLMs to precisely use tools. To verify the quality of generated examples for the tool functionality guidelines, we input the parameters to execute the tools to confirm the correct input of parameters and the accuracy of results.

### Evaluation

To assess the quality of tool descriptions, we select 100 examples from ToolBench and employ three annotators to evaluate their accuracy. To assess the plausibility of the scenarios, we also sample 100 tool functionality guidelines from ToolBench and employ three annotators to evaluate the plausibility of the scenarios. To process conflicting annotations, we adopt a voting majority principle to determine the results. Finally, the evaluation results demonstrate that the correctness of tool description and tool functionality guidelines can reach a value of 100% accuracy, with Fleiss's \(\kappa=0.97\). This demonstrates that LLMs can summarize high-quality tool descriptions and generate plausible scenarios based on tool documentation, thereby highlighting the simplicity and effectiveness of EasyTool. The annotation details for tool instruction are shown in Appendix A. The evaluation on the robustness of the prompts is shown in Appendix D. The hallucination evaluation for tool instruction is shown in Appendix E. We also compare with prompt compression methods in Appendix F.

## 5 Experiment

In this section, we adopt EasyTool to three distinct tool-use applications to show that EasyTool can help LLM-based agents better answer real-world user requests through tool usage (SS 5.1), find correct tool solution paths (SS 5.2), and improve their tool utilization capabilities on complex math problems (SS 5.3).

### Real-World Question Answering

Since LLMs are still limited in their training data, it is essential for them to use external tools to access up-to-date information in response to user requests.

BenchmarkWe choose ToolBench Qin et al. (2023), a dataset containing diverse user requests

\begin{table}
\begin{tabular}{l c l} \hline \hline  & **I: Tool Description Generation** \\ \hline _/* I: Task prompt */_ & \\ Your task is to create a concise and effective tool usage description based on the tool documentation. You should ensure the description only contains the purposes of the tool without irrelevant information. Here is an example: _/* Examples */_ & \\ \{Tool Documentation\} & \\ Tool usage description: & \\ Tool name\} is a tool that can (General\_Purposes). & \\ This tool has (Number) multiple built-in functions: & \\
1. \{Function\_i\} is to \{Functionality\_of\_Function\_1\} & \\
2. \{Function\_2\} is to ... & \\ _/* Auto generation of tool description */_ & \\ Tool Documentation of ‘Aviation Weather Center’\} & \\ Tool usage description: & \\ _‘Aviation Weather Center’ is a tool which can provide official aviation weather data..._ & \\ \hline \hline \multicolumn{4}{c}{**II: Tool Function Guidelines Construction**} \\ \hline \hline \end{tabular}
\end{table}
Table 2: Examples of prompt for ChatGPT for tool description generation and tool function guidelines construction. _Green texts_ are generated by ChatGPT.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Dataset** & \multicolumn{1}{l}{\multirow{2}{*}{\(\mathbf{Token_{loc.}}\)}} & \multicolumn{1}{l}{\multirow{2}{*}{\(\mathbf{Token_{ins.}}\)}} & \multicolumn{1}{l}{\multirow{2}{*}{\(\mathbf{Reduce~{}(\%)}\)}} \\ \cline{1-1} ToolBench & 2,530 & 748 & 70.43\% \\ RestBench & 3,881 & 103 & 97.35\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: The average number of tokens in each tool documentation (\(\mathbf{Token_{loc.}}\)) and tool instruction generated by EasyTool (\(\mathbf{Token_{Ins.}}\)). We also report the reduced ratio (_i.e._ **Reduce (%)**) for reference.

 

with a massive set of publicly available REST API tools. The API tools in ToolBench are collected from RapidAPI, which requires additional individual subscriptions with payments. Therefore, we follow the strategy of many previous works to only select the subsets for evaluation (Zhang et al., 2023; Du et al., 2024; Liu et al., 2024). Other subsets of ToolBench only contain a single tool from the same category. Compared with other subsets of ToolBench, I2-Category and I3-Instruction contain complex user requests that need multiple tools from different categories to solve, which are more aligned with real-world user requests. Therefore, we only select I2-Category and I3-Instruction and conduct experiments on these 300 data samples. Each data of ToolBench consists of a user request with a ground truth toolset, and thus models only need to select and execute the tools from the ground truth toolset to complete the user request.

For evaluation, ToolBench designs two evaluation metrics based on ChatGPT: (1) **Pass Rate**, calculated by the proportion of instructions successfully completed within a limited budget; (2) **Win Rate**, measured by asking a ChatGPT evaluator to select its preference for two solution paths. Following Qin et al. (2023), we also measure **Success Rate**, which asks GPT-4 to check whether the responses can reasonably and accurately answer the user requests.2

Footnote 2: The prompt template for evaluating success rate is shown in Appendix B.1

BaselinesFollowing Qin et al. (2023), we select ChatGPT OpenAI (OpenAI, 2022), GPT-4, GPT-4o OpenAI (2023), Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct Dubey et al. (2024), and ToolLLaMA-7B as backbone models. ToolLLAMA-7B is fine-tuned from a 7B LLaMA model Touvron et al. (2023) on ToolBench. Both ReACT Yao et al. (2023) and DFSDT Qin et al. (2023) are common baselines in tool learning and have been implemented in the ToolBench as well. To make a fair comparison, we apply them as our baseline. We do not adopt TooLLaMA-7B on EasyTool due to its poor instruction-following capability.3 For the baselines, we follow the settings in Qin et al. (2023), which provides tool documentation for them

\begin{table}
\begin{tabular}{l l c r r r r r r r r} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{1}{c}{\multirow{2}{*}{**Method**}} & \multicolumn{5}{c}{**I2-Category**} & \multicolumn{5}{c}{**I3-Instruction**} & \multicolumn{5}{c}{**Average**} \\ \cline{3-10}  & & **Pass** & **Win** & **Success** & **Pass** & **Win** & **Success** & **Pass** & **Win** & **Success** \\ \hline \multirow{6}{*}{ChatGPIT} & ReACT & 39.0 & – & 18.0 & 23.0 & - & 1.0 & 31.0 & - & 9.5 \\  & DFSDT & 64.5 & 63.0 & 24.0 & 60.0 & 70.0 & 6.0 & 62.3 & 66.5 & 15.0 \\  & +EasyTool & **74.5** & **76.5** & **68.5** & 65.0 & **88.0** & 37.0 & **69.8** & **82.3** & **52.8** \\  & +EasyTool +Re. & 69.0 & 71.0 & 60.5 & **66.0** & 89.0 & **42.0** & 67.5 & 80.0 & 51.3 \\ \hline \multirow{6}{*}{ToolLLaMA-7B} & ReACT & 30.0 & 45.5 & 9.5 & 22.0 & 49.0 & 3.0 & 26.0 & 47.3 & 6.3 \\  & DFSDT & **66.0** & 55.0 & **24.0** & **56.0** & 56.0 & **6.0** & **61.0** & 55.5 & **15.0** \\  & +Re. & 57.0 & **60.0** & 11.5 & 54.0 & **69.0** & 2.0 & 55.5 & **64.5** & 6.8 \\ \cline{1-1}  & ReACT & 3.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.5 & 0.0 & 0.0 \\ \cline{1-1}  & DSFDT & 12.0 & 32.0 & 10.0 & 8.0 & 3.0 & 1.0 & 10.0 & 17.5 & 5.5 \\  & +EasyTool & **75.0** & **75.0** & **60.0** & **69.0** & 88.0 & 37.0 & **72.0** & **81.5** & 48.5 \\  & +EasyTool +Re. & 69.0 & 69.0 & 57.5 & 68.0 & **89.0** & **42.0** & 68.5 & 79.0 & **49.8** \\ \hline \multirow{6}{*}{Llama-3.1-70B-Instruct} & ReACT & 24.0 & 10.0 & 11.0 & 20.0 & 13.0 & 10.0 & 22.0 & 11.5 & 10.5 \\  & DFSDT & 45.0 & 40.0 & 27.0 & 35.0 & 56.0 & 21.0 & 40.0 & 48.0 & 24.0 \\ \cline{1-1}  & +EasyTool & **71.5** & **79.0** & **70.0** & 70.0 & **89.0** & **60.0** & **70.8** & **84.0** & **65.0** \\ \cline{1-1}  & +EasyTool +Re. & 70.0 & 71.0 & 65.0 & **71.0** & **89.0** & 54.0 & 70.5 & 80.0 & 59.5 \\ \hline \multirow{6}{*}{GTP-4} & ReACT & 67.5 & 53.5 & 27.0 & 40.0 & 71.0 & 4.0 & 53.8 & 62.3 & 15.5 \\  & DFSDT & 69.5 & 57.0 & 42.0 & 59.0 & 73.0 & 50.0 & 64.3 & 65.0 & 46.0 \\ \cline{1-1}  & +EasyTool & **76.5** & **78.5** & **76.0** & **69.0** & 89.0 & **64.0** & **72.8** & **83.8** & **70.0** \\ \cline{1-1}  & +EasyTool +Re. & 72.5 & 72.0 & 73.5 & **69.0** & **90.0** & 53.0 & 70.8 & 81.0 & 63.3 \\ \hline \multirow{6}{*}{GTP-4o} & ReACT & 66.0 & 56.5 & 30.0 & 42.0 & 71.0 & 5.0 & 54.0 & 63.8 & 17.5 \\ \cline{1-1}  & DFSDT & 72.5 & 63.5 & 54.5 & 60.0 & 80.0 & 63.0 & 66.3 & 71.8 & 63.8 \\ \cline{1-1}  & +EasyTool & **80.5** & **81.5** & **83.0** & **73.0** & **90.0** & **71.0** & **76.8** & **85.8** & **77.0** \\ \cline{1-1}  & +EasyTool +Re. & 76.5 & 79.0 & 81.0 & 69.0 & **90.0** & 67.0 & 72.8 & 84.5 & 74.0 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results of LLMs on ToolBench. Win rate (denoted as **Win**) is calculated by comparing each model with ChatGPT-ReACT. The win rate higher than 50% means the model performs better than ChatGPT-ReACT. Following Qin et al. (2023), apart from adopting retriever (_i.e._, +Re.), all methods use the ground truth toolset to select tools. The best results are **bolded**, and the second best ones are underlined.

 

to use tools.

Main ResultWe simplify the tool documentation from ToolBench into concise tool instructions with EasyTool.4 Each tool instruction consists of a tool description and functionality guidelines. As shown in Table 3, with EasyTool, replacing tool documentation with our tool instruction can greatly reduce the token cost of each tool. Especially in ToolBench, the token cost was reduced by 70.43%. Furthermore, results in Table 4 show that: _1)_EasyTool can help LLMs achieve state-of-the-art performance, indicating the superiority of tool Instructions over tool documentation in facilitating tool usage for LLM-based agents; _2)_ Tool instructions generated by EasyTool can help small models (_e.g._, Llama3.1-8B-Instruct) to better understand the usage of tools, even making them outperform the fine-tuned method, _i.e._, ToolLLaMA. _3)_ Llama3.1-7OB-Instruct outperforms Llama3.1-8B-Instruct with EasyTool, indicating that models with better instruction-following capabilities can achieve greater improvements with high-quality tool instructions.

Footnote 4: The data examples of ToolBench are provided in Appendix G.1.

EasyTool for Tool RetrievalIn real-world scenarios, asking users to manually recommend tools from a large pool for LLMs to select may not be practical. Therefore, ToolBench also provides a dense retriever based on BERT-base Devlin et al. (2019) to retrieve relevant tools for solving user requests, and claims that it outperforms text-embedding-ada-002, _i.e._ GPT Ada Ouyang et al. (2022), which retrieves tools based on the cosine embedding similarity between the subtasks decomposed by user requests and tool descriptions from original tool documentation in ToolBench. We argue that the poor performance of Ada may be due to low-quality tool descriptions, which often contain irrelevant details and lack clear functionality guidelines. Thus, we replace the original tool descriptions with the ones generated by EasyTool. Following Qin et al. (2023), we compare the performance of these retrieval methods using NDCG Jarvelin and Kekalainen (2002). NDCG is a ranking quality metric that compares rankings to an ideal order in which all relevant items are at the top of the list. Results in Table 5 show EasyTool can greatly improve the tool retrieval performance.

EasyTool for Tool SelectionWe utilize the I1-Instruciton of ToolBench, which comprises 100 user requests solvable by a single tool. We first obtain the golden tool from I1-Instruction and then retrieve other different tools based on cosine embedding similarity between the user request and tool descriptions as candidate tools. Then, we evaluate the selection accuracy of LLMs with varying numbers of candidate tools, using either original ToolBench descriptions or those generated by EasyTool. Figure 3 illustrates that EasyTool enhanced descriptions enable LLMs to select the correct tool more effectively from a larger pool.

EasyTool for Tool ExecutionFor each subtask in I2-Category and I3-Instruction, we retrieve the top 10 most similar tools using our tool descriptions and ask models to select and execute them.

Figure 4: Error rates of tool calls in different LLMs. The error rate is the number of two tool-related errors relative to the total number of tool calls. The results are evaluated by **human annotators**.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Method** & **I2-Category** & **I3-Instruction** & **Average** \\  & @1 & @5 & @1 & @5 & @1 & @5 \\ \hline BERT Retriever & 68.2 & 77.9 & **81.7** & 87.1 & 75.0 & 82.5 \\ Ada & 36.8 & 30.7 & 54.6 & 46.8 & 45.7 & 38.8 \\ + EasyTool & **73.4** & **82.7** & 80.1 & **88.5** & **76.7** & **85.6** \\ \hline \hline \end{tabular}
\end{table}
Table 5: The performance of different retrievers for two subsets in ToolBench. We report **NDCG@1** and **NDCG@5**.

Figure 3: The selection accuracy of LLMs on I1-instruciton of ToolBench.

 

As shown in Table 4, using these retrieved tools proves comparable, and sometimes even superior, to the ground truth tool set. The rationale is that EasyTool with retriever can retrieve similar tools with better functionalities to replace some tools in the ground truth tool set.

Error AnalysisWe follow Zhang et al. (2023) and define two types of error: Tool name error means models call non-existent tool functions that are not in the tool inventory, and parameter error means models pass invalid parameters. Both errors lead to unsuccessful tool execution. We sample 100 data from I2-Category and I3-Instruction and employ three annotators to manually examine the output of LLMs with tool documentation and tool instruction generated by EasyTool Fleiss's \(\kappa=0.91)\). The results in Figure 4 show that LLMs may generate non-existent tool names and pass invalid parameters to the right tool functions. However, EasyTool can significantly reduce these incorrect behaviors, leading to successful tool execution.5

Footnote 5: Annotation details are shown in Appendix A and further analysis about the number of examples is shown in Appendix B.3.

### Real-World Web Services

Real-world web services often need to execute tools following a specific order. We aim to explore the capability of LLMs to find correct tool solution paths.

Benchmark and BaselinesWe select RestBench Song et al. (2023), comprising tasks in real-world web service scenarios. We evaluate our method on a subset of RestBench, _i.e._, TMDB. TMDB is a movie information website that offers 55 official RESTful APIs as tools. Following the evaluation metric in RestBench, we use the correct path rate (CP%) to measure accuracy, which is the proportion of the tool path containing the gold tool path as a subsequence. We choose RestGPT Song et al. (2023) as our backbone model. The RestGPT has two versions, _i.e._, Vicuna-13B-based RestGPT and ChatGPT-based RestGPT. For Vicuna-13B-based RestGPT, we compare our method with ToolDec, a decoding algorithm to help LLMs invoke tools properly. For ChatGPT-based RestGPT, we compare our method with ReAct since ToolDec cannot be applied to close-sourced models.

ResultWe simplify the long tool documentation into concise tool instructions with EasyTool for LLMs to use.6 For comparison, we use the prompt from Song et al. (2023) containing original tool descriptions and four examples. Table 3 demonstrates that EasyTool significantly reduces the token cost. Figure 5 highlights the considerable improvement in the correct path rate, signifying EasyTool's effectiveness in aiding LLMs to find the correct tool solution paths.

Footnote 6: The data examples of RestBench are provided in Appendix G.2.

### Numerical Reasoning

We also explore whether EasyTool can endow LLM-based agents with better tool-utilization ability in complex math problems with incomplete tool documentation.

Benchmark and BaselinesWe adopt FuncQA Hao et al. (2023), which tests the numerical reasoning ability of LLMs on complex math problems involving 13 arithmetic operationsools. FuncQA has two subsets, _i.e._, **one-hop** and **multi-hop**. The 68 one-hop questions can be solvable with one tool. The 60 multi-hop questions average 2.78 tool uses per question. Apart from accuracy, we also measure tool error rate (_i.e._, **Error**), the proportion of tasks with at least one tool-related error. Following Hao et al. (2023), we select Vicuna-30B and ChatGPT and compare our method with 0-shot learning, Chain-of-thought (CoT) prompting and ReAct.

ResultUnlike the other datasets, FuncQA only provides the name and calling function of a tool as documentation, without any other tool descriptions for further usage demonstration. Therefore, by only leveraging the provided tool name and calling function, we can also apply EasyTool to generate tool descriptions with usage scenarios to construct

Figure 5: The correct path rate (CP%) on two versions of RestBench with different methods.

 

tool instruction for FuncQA.7 Results in Table 6 show that tool instructions generated based on our method (+ EasyTool) significantly improve the tool utilization ability of LLMs on complex math problems. Furthermore, a lower tool error rate of models with EasyTool indicates that concise and effective tool instructions can better guide models to select correct tools and pass valid parameters.

Footnote 7: The data examples of FuncQA are provided in Appendix G.3.

## 6 Conclusion

In this paper, we introduce EasyTool, an easy but effective method to enhance the tool utilization capabilities of LLM-based agents through the simplification and refinement of tool documentation into a clear, structured and practical tool instruction. Our comprehensive experiments demonstrate that EasyTool can effectively enhance performance in different real-world applications. We also expect EasyTool can facilitate the community to better develop efficient tool utilization.

### Limitations

First, this paper only focuses on tool documentation whose token length does not exceed the ChatGPT input limit. Documentation with token counts surpassing this limit cannot be processed by EasyTool without additional preprocessing. Second, our method is limited to single documentation, neglecting the dependencies among tools. Considering these dependencies in tool descriptions could significantly enhance the model's effectiveness in certain scenarios. Finally, EasyTool only works on models with instruction-following ability. Future work can focus on training specialized models using tool instructions generated by EasyTool, thereby improving their capability in tool utilization.

### Ethics Statement

We acknowledge that all authors are informed about and adhere to the ACL Code of Ethics and the Code of Conduct.

Use of Human AnnotationsOur institution recruited annotators to implement the annotations of tool descriptions and functionality guidelines. We ensure the privacy rights of the annotators are respected during the annotation process. The annotators receive compensation exceeding the local minimum wage and have consented to the use of tool instructions generated by EasyTool for research purposes. Appendix A provides further details on the annotations.

RisksThe tool benchmarks in our experiment are sourced from publicly available sources. However, we cannot guarantee that they are devoid of socially harmful or toxic language. Furthermore, evaluating the data quality of tool instructions is based on common sense, which can vary among individuals from diverse backgrounds. We use ChatGPT to correct grammatical errors in this paper.

### Acknowledgement

We thank the anonymous reviewers for their valuable comments. This work is partially supported by the Chinese NSF Major Research Plan (No.92270121).

## References

* Chen et al. [2023] Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, et al. 2023. T-eval: Evaluating the tool utilization capability step by step. _arXiv preprint arXiv:2312.14033_.
* Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* datapt quality.
* Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
* \\ + CoT & 13.33 & 4.00 &
- \\ + ReAct & 45.00 & 7.35 & 20.31 \\ + EasyTool & **65.00** & **11.76** & **10.15** \\ \hline ChatGPT & 55.00 & 9.00 &
- \\ + CoT & 48.33 & 17.64 &
- \\ + ReAct & 85.00 & 41.17 & 9.38 \\ + EasyTool & **91.66** & **48.53** & **2.34** \\ \hline \hline \end{tabular}
\end{table}
Table 6: The accuracy of Vicuna-30B and ChatGPT on the FuncQA.
 

* Du et al. (2024) Yu Du, Fangyun Wei, and Hongyang Zhang. 2024. Any-tool: Self-reflective, hierarchical agents for large-scale api calls. _arXiv preprint arXiv:2402.04253_.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Gravitas (2023) Significant Gravitas. 2023. Auto-gpt: An autonomous gpt-4 experiment. [https://github.com/Significant-Gravitas/Auto-GPT](https://github.com/Significant-Gravitas/Auto-GPT).
* Hao et al. (2023) Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2023. Toolkengptt: Augmenting frozen language models with massive tools via tool embeddings. volume abs/2305.11554.
* Hsieh et al. (2023) Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. 2023. Tool documentation enables zero-shot tool-usage with large language models.
* Jarvelin and Kevalainen (2002) Kalervo Jarvelin and Jaana Kevalainen. 2002. Cumulated gain-based evaluation of ir techniques. _ACM Trans. Inf. Syst._, 20(4):422-446.
* Jiang et al. (2023a) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lelio Renard Lavaud, Marie-Anne Lacnaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023a. Mistral 7b.
* Jiang et al. (2023b) Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023b. LLMLingua: Compressing prompts for accelerated inference of large language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 13358-13376, Singapore. Association for Computational Linguistics.
* Li et al. (2023) Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 3102-3116.
* Liu et al. (2024) Xukun Liu, Zhiyuan Peng, Xiaoyuan Yi, Xing Xie, Lirong Xiang, Yuchen Liu, and Dongkuan Xu. 2024. Toolboxnet: Connecting large language models with massive tools via tool graph. _arXiv preprint arXiv:2403.00839_.
* Lu et al. (2023) Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-play compositional reasoning with large language models. volume abs/2304.09842.
* Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishzizi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 12076-12100, Singapore. Association for Computational Linguistics.
* Mu et al. (2023) Jesse Mu, Xiang Lisa Li, and Noah D. Goodman. 2023. Learning to compress prompts with gist tokens. _CoRR_, abs/2304.08467.
* Mundler et al. (2024) Niels Mundler, Jingxuan He, Slobodan Jenko, and Martin Vechev. 2024. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. In _The Twelfth International Conference on Learning Representations_.
* OpenAI (2022) OpenAI. 2022. Chatgtp.
* OpenAI (2023) OpenAI. 2023. GPT-4 technical report.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhnii Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In _Advances in Neural Information Processing Systems_.
* Parisi et al. (2022) Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. TALM: tool augmented language models. _CoRR_, abs/2205.12255.
* Patil et al. (2023) Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large language model connected with massive apis. _CoRR_, abs/2305.15334.
* Qin et al. (2023) Yuja Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaari Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. ToollmT: Facilitating large language models to master 16000+ real-world apis. _CoRR_, abs/2307.16789.
* Qu et al. (2024) Changle Qu, Sunhao Dai, Xiaochui Wei, Hengyi Cai, Shuiajiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2024. Tool learning with large language models: A survey. _arXiv preprint arXiv:2405.17935_.
* Ren et al. (2024) Mengjie Ren, Boxi Cao, Hongyu Lin, Liu Cao, Xianguei Han, Ke Zeng, Guanglu Wan, Xunliang Cai, and Le Sun. 2024. Learning or self-aligning? re-thinking instruction fine-tuning. _arXiv preprint arXiv:2402.18243_.
* Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. _CoRR_, abs/2302.04761 

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023a. Hugging: Solving AI tasks with chatapt and its friends in huggingface. volume abs/2303.17580.
* Shen et al. (2023b) Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. 2023b. Taskbench: Benchmarking large language models for task automation. _arXiv preprint arXiv:2311.18760_.
* Song et al. (2023) Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li. 2023. Restgpt: Connecting large language models with real-world restful apis.
* Tang et al. (2023) Qiaoyu Tang, Ziliang Deng, Hongyun Lin, Xianpei Han, Qiao Liang, and Le Sun. 2023. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.
* Team and Google (2023) Gemini Team and Google. 2023. Gemini: A family of highly capable multimodal models.
* Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Lacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. _CoRR_, abs/2302.13971.
* Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyi Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inanc, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Kotara, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Dinana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Mobbyog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Hungta, Kaylan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Purxin Xu, Zheng Yan, Iilyaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialam. 2023b. Llama 2: Open foundation and fine-tuned chat models. _CoRR_, abs/2307.09288.
* Wang et al. (2024) Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, and Yu Su. 2024. LLms in the imaginarium: tool learning through simulated trial and error. _arXiv preprint arXiv:2403.04746_.
* Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13484-13508, Toronto, Canada. Association for Computational Linguistics.
* Xu et al. (2023) Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. On the tool manipulation capability of open-source large language models.
* Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In _The Eleventh International Conference on Learning Representations_.
* Zhang et al. (2023a)Kexun Zhang, Hongqiao Chen, Lei Li, and William Wang. 2023a. Syntax error-free and generalizable tool use for llms via finite-state decoding.
* Zhang et al. (2023b) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023b. Siren's song in the ai ocean: a survey on hallucination in large language models. _arXiv preprint arXiv:2309.01219_.

 

Crowd-sourcing Details for Tool Instruction Evaluation and Error Analysis

For tool instruction evaluation, we have recruited a team of three undergraduates. We pay each annotator $8/h, exceeding the local minimum wage. The screenshots of the instructions and interface for tool description and functionality guideline annotation are shown in Figure 7 and Figure 8.

For error analysis, we have recruited a team of three undergraduates as annotators. We pay each annotator $8/h, exceeding the local minimum wage. We randomly sampled 100 data from I2-Category and I3-Instruction and asked them to manually examine the output of LLMs using tool documentation and tool instructions generated by EasyTool. Each sample is examined by these three annotators with Fleiss's \(\kappa=0.91\). In cases of disagreement among them, the decision of the majority was accepted. The screenshots of the instructions and interface for error annotation are shown in Figure 9.

## Appendix B Details of ToolBench

### Success Rate Evaluation

The prompt of the success rate evaluation is given in List 1.

``` Pleasecheckwhethertheresponsecanreasonablyandaccuratelyanswerthenquestion.Ifitcan,pleaseoutput'YES';Ifnot,pleaseoutput'NO'Youneedtogivereasonsfirstandthemdecidewhethertherresponsecanreasonablyandaccuratelyanswethequestion.YoumustonlyoutputinaparsibleJSONformat.Twoexampleoutputslooklike: Example1:("Reason":"Treasonwhyouthinkethresponsecanreasonablyandaccuratelyanswerthequestion","Choice":"Yes") "Example2:("Reason":"Treasonwhyouthinkethrespensecannotreasonablyandaccuratelyansweredthequestion","Choice":"No") Thisisthesures 

Thetoolinstructionconsistsoftwoparts:tooldescriptionandtoolfunctionguidelines.

Thetooldescriptiononlycontainsthepurposesofthetoolwithoutotherirrelevantinformation.Hereisanexample:'add_'returnsthemsofalltheargumentspassedtotint,normalizedto2decimalplaces.

Thetoolfunctionguidelinesintroducetherparameterswithexamplesthatcontainthescenarioadoptingtheparameters.Hereisanexample:"Usage":{ "required_parameters":[ {  "name":""input",  "type":"List"  } ],  "Example":{  "Scenario":"ifyourantotodd2t01.",  "Parameters":{  "input":[2,1]  } } Youshouldfirstgeneratethetool descriptionandthengivethetool functionguidelines.YoumustonlyoutputinaparsibleJSONformat.Two exampleoutputslooklike:("tool_description":XX,"Usage":XX) Thisisthetoolname:{tool_name}Thisisthefunctionoftool:{ tool_function} Output: ```
## Appendix D Robustness Evaluation

In this section, we aim to evaluate the robustness of the task prompts in Table 2. We ask ChatGPT to rewrite these task prompts three times and the new task prompts are shown in List 3. We sample 100 tool documentations from ToolBench and ask ChatGPT to generate tool descriptions and tool functionality guidelines based on the new task prompts. Then we ask two annotators to evaluate the quality of four results (one from our task prompts, and three from ChatGPT generated task prompts). The results in Figure 6 show that the changes to the task prompt, without altering the actual meaning, do not affect the quality of the tool description and tool functionality guidelines, thereby demonstrating the robustness of our prompts.
``` Prompt-1:- ToolDescription: Yourassignmentinvolvesdevelopingascinctandpracticaldescriptiononusingaspecifictool,assoutlineditsdocumentation.Thisdescriptionshouldfocussolelyonthetool'sfunctions,excludinganyextraneousdetails.-ToolFunctionGuidelines: Createacenaorthincorporatestheuseofagspecifiedtool,ensuringitutilizestheprovidedparameters.Receiveadescriptionofatool,includingitspurposeandallistofparameters.Designascenariothaffectivelyemploystheseparameters.Ifboth"required_parameters"and"optional_parameters"areabsent,format yourresponseas:("Scenario":XX,"Parameters":{}).Prompt-2:- ToolDescription: Yourassignmentinvolvescraftingaucinctandpracticaldescriptionofaqutool,usingitsdocumentationasreference.Focusonoutliningthetool'sfunctions,excludinganyextraneousdetails.-ToolFunctionGuidelines:Yourassignmentinvolvesdevelopingascenariosthatutilizesaspecifiedtool.Herearetheguidelines:Youwillreceireinformationaboutaqtool,includingitsintendedusemadalistofparameters.Yourscanarioshouldincorporatetheseparameters.Incaseswhereboth"required_parameters"and"optional_parameters"areabsent,formatyourresponsesafollows:("Scenario":XX,"Parameters":{}).Prompt-3:- ToolDescription: Yourassignmentinvolveswritingastuccinetandcleardescriptionofaquolo'susage,guidedbyitsdocumentation.Thissingschedcriptionshouldexclusivelyfocusonthetool'sfunctions,omitting 

anyextraneousdetails. - ToolFunctionGuidelines: Yourassignmentinvolvescraftinga scenariothatutilizesapecifictool. Here'showtproceeded: First,finalizurizeyourselfwiththe tool'sintendeduseanditsavailable parameters.Then,designacsenaridithatterfectivelyincorporatedestheseparameters. Incaseswhereboth"required_parameters"and"optional_parameters"areabsent,formatyourspensesafollowswers["Scenario":XX,"Parameters":{}].

## Appendix E Hallucination Evaluation

Following Zhang et al. (2023), we categorize hallucination within the context of LLMs as follows:

* **Input-conflictinghallucination**, where LLMs generate content that deviates from the source input provided by users;
* **Context-conflictinghallucination**, where LLMs generate content that conflicts with previously generated information by itself;
* **Fact-conflictinghallucination**, where LLMs generate content that is not faithful to established world knowledge.

To evaluate the occurrence of hallucinations in tool description generation, we select 500 tool descriptions generated by EasyToolfrom ToolsBench, and employ the two hallucination evaluation methods:

* **Human Eval**Min et al. (2023): we employ three annotators for the hallucination evaluation (Fleiss's \(\kappa=0.94\)).
* **LLM Judge**Mundler et al. (2024): we adopt GPT-4o to automatically detect the above hallucinations.

The results in Table 8 indicate that ChatGPT occasionally produces minor hallucinations when generating tool instructions, but such instances are rare. This demonstrates that our method is both simple and effective.

Moreover, we also evaluate the hallucinations produced by LLMs when utilizing tool documentation and our tool instruction. We select 200 samples from ToolBench and engage three annotators for evaluation (Fleiss's \(\kappa=0.97\)). The findings in Table 9 indicate that using tool instructions significantly decreases the occurrence of hallucinations compared to using tool documentation, demonstrating the advantages of EasyTool.

## Appendix F Prompt Compression Method

We also adopt LLMLingua Jiang et al. (2023), a prompt compression method, to identify and remove non-essential tokens in tool documentation. As shown in Table 10, employing a common prompt compression method like LLMLingua can be detrimental. This method identifies and removes non-essential tokens in tool documentation, potentially including crucial tokens in parameters and functions necessary for successful tool execution. This compression harms the model's understanding of the tool. However, as shown in Appendix G, using EasyToolto compress tool descriptions can enhance the model's comprehension of the tool. This improvement significantly boosts tool retrieval performance, enabling LLMs to more effectively select the correct tool from a larger pool.

## Appendix G Examples of Tool Instruction

### Data Examples of ToolBench

Table 11 presents some examples of tool instructions generated by EasyToolin ToolBench for a better understanding.

Figure 6: Comparison of our task prompts with ChatGPT generated task prompts. Percentage of wins, ties and losses are calculated.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Metrics** & **Input** & **Context** & **Fact** \\ \hline Human Eval & 1.2\% & 0.8\% & 1.0\% \\ LLM Judge & 0.6\% & 0.8\% & 0.8\% \\ \hline \hline \end{tabular}
\end{table}
Table 8: The hallucination evaluation of tool instructions.

 



### Data Examples of RestBench

Table 12 presents some examples of tool instructions generated by EasyTool in RestBench for a better understanding.

#### Data Examples of FuncQA

Table 13 presents some examples of tool instructions generated by EasyTool in FuncQA for a better understanding.



* [ "product_id": "api_b04d269d-c7bd-4bB4-8e17-ffba24d64d3d",  "tool_description": "Get Products from Ebay(Unofficial)",  "home_url": "[https://rapidapi.com/felixeschmittres/api/ebay32/](https://rapidapi.com/felixeschmittres/api/ebay32/)",  "name": "Ebay",  "title": "Ebay",  "prricing": "FREEMIIUH",  "tool_name": "Ebay",  "host": "ebay32.p.rapidapi.com",  "api_list":  {  "name": "Product Details",  "url": "[https://ebay32.p.rapidapi.com/product/195499451557](https://ebay32.p.rapidapi.com/product/195499451557)",  "description": "Get the product details for a given product id and a specific country.",  "method": "GET",  "required_parameters": [  {  "name": "product_id",  "type": "NUMBER",  "description": "ID of the product. Can be obtained from the url  of the product or by using the 'search' endpoint.",  "default": "195499451557"  }  ],  "optional_parameters": [  {  "name": "country",  "type": "STRING",  "description": "Valid country to return offers for.\nValid values are in description of this endpoint.\nDefault: "united  states^..",  "default": "germany"  },  {  "name": "country_code",  "type": "STRING",  "description": "Country code of the valid country to return  offers for.\nValid values are in description of this endpoint.\ n Default: "us",  "default": "de"  }  ]

}

 

Thanks for participating in this HITI Please spend some time reading this instruction to better understand our HITI In this bit, you need to first read the tool document and then check whether the tool description for the tool is correct.

Tool documents are mostly sourced from the internet and authored by the service providers. As a result, the original tool descriptions often contain extraneous information such as URLs, product IDs, and recommendation terms, and do not clearly convey the tool's purpose. Moreover, many tools in the inventory have multiple built-in functions that serve various purposes in different scenarios. For example, Google Maps can calculate distances between locations and provide coordinates. However, not all tool functionalities are comprehensively covered in its description. Therefore, we focus on create a concise and effective tool description for the tool based on the tool document, which focuses on the purpose of tools as outlined in the tool documents. Next, you need to manually examine the tool description and check whether the tool description for the tool is correct or not.

#### Tool Document

[] {  "product_id": "api_22b/7f3fd-dfte-4062-bsbc-69bdd5d4e5Q2",  "tool_description":"Unleash the Power of Kloppedia: Effortlessly Retrieve Shop and Product Information with Our API;  "home_url": [https://rapidapi.com/dytra/api/tokopedia-super-api/](https://rapidapi.com/dytra/api/tokopedia-super-api/)",  "name":"Tokopedia Super APT;  "title":"Tokopedia Super API;  "pricing":"FEREMIUMU;  "tool_name":"Tokopedia Super API;  "host":"tokopedia-super-api.npi.npi.capliqip.com",  "api_list":[  {  "name":"sortProductsMaster",  "url":"[https://tokopedia-super-api.npi.npiadapi.com/sortProductsMaster](https://tokopedia-super-api.npi.npiadapi.com/sortProductsMaster)",  "description":"the list of available sorting methods",  "method":"GET;  "required_parameters":[],  "optional_parameters":[],  "code":"import requests   url ="[https://tokopedia-super-api.npi.rapidapi.com/sortProductsMaster](https://tokopedia-super-api.npi.rapidapi.com/sortProductsMaster)"

#### Tool Description

[] **Textbox**

[] "Tokopedia Super API" can effortlessly retrieve shop and product information. This tool has IAPI;  1.'sortProductsMaster' can provide the list of available sorting methods.

Please check whether the tool description for the tool is correct or not.

### Correct

[] **Incorrect**

Figure 7: The screenshots of the instructions and interface for tool description annotation.

 

Thanks for participating in this HIT Please spend some time reading this instruction to better understand our HITI In this hit, you need to first read the tool functionality guidelines and then check whether the scenarios in examples for the tool is reasonable.

The tool functionality guidelines not only contain the required and optional parameters from the tool documents, but also contain examples generated by large language models (LLMs) to demonstrate how to input parameters, including scenarios and parameter names with values. We have already executed the tool to verify that the parameters are correctly input and yield accurate results. Next, you need to manually assess the plausibility of the scenarios generated by LLMs.

The parameters in tool functionality guidelines

**Textbox**

{  "name":"Product Details",  "description":"Get the product details for a given product id and a specific country.",  "required_parameters":[  {  "name":"product_id",  "type":"NUMBER",  "description":"ID of the product. Can be obtained from the url of the product or by using the  "search' endpoint.",  "default":"19549945IS57"  }  ].  ].  "optional_parameters":[  {  "name":"country",  "type":STRING",  "description":"Valid country to return offers for.",  "default":"germany"  },  {  "name":"country_code".

Example

**Textbox**

{  "Scenario":"If you want to know the details of the product with product ID 19549945IS57 in Germany",  "Parameters":[  "product_id":19549945IS57,  "country":"Germany",  "country_code":"de"  }  } ```

Please check whether the scenario in the example is reasonable or not.

Figure 8: The screenshots of the instructions and interface for tool functionality guidelines annotation.

 

The first two terms are the same as those that we have used in our analysis. The second term is a _local_, which means that the local information of the 

* [10]
**Tool Description: /* Example 1 */  {"TokopediaApi' can search and retrieve product details from Tokopedia. This tool has  2 APIs: 1. "Search Product' can search for products on Tokopedia based on a query string and action type. 2. "Get Product Detail' can retrieve detailed information about a product on Tokopediabased on its slug.  /* Example 2 */  'Tokopedia Super API' can effortlesslyretrieveshop and product information. This tool has 1 API: 1. 'sortProductsMaster' can provide the list of available sorting methods.

```
ToolFunctionGuidelines: /*Example1*/ {  "name": "SearchProduct",  "description": "Search The Product",  "required_parameters":[  {  "name": "query",  "type": "STRING",  "description": "",  "default": "Celana Jeans"  },  {  "name": "act",  "type": "STRING",  "description": "",  "default": "search"  }  ],  "optional_parameters":[],  "Example": {  "Scenario": "if you want to search for a product with the query 'Celana Jeans' using the 'search' action",  "Parameters": {  "query": "Celana Jeans",  "act": "search"  }  } )/* Example 2 */  {  "name": "sortProductsMaster",  "description": "the list of available sorting methods",  "required_parameters":[],  "optional_parameters":[],  "Example": {  "Scenario": "if you want to retrieve the list of available sorting methods  for products using TokopediaSuper API",  "Parameters": {}  } }
```

\begin{table} \end{table}
Table 11: The tool instruction of ToolBench generated by EasyTool.

 

* [10] M. C. Dickson, R. J. Thorne, and A. S. Thomas, "A new method for the calculation of the number of events in a given event using the same data set", _Nucl. Instrum. Meth. A_**564**, 397-428 (2000)

[MISSING_PAGE_POST]

 



\begin{table} \end{table}
Table 13: The tool instruction of FuncQA generated by EasyTool.