Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios (2024)

Abstract: The recent trend of using Large Language Models (LLMs) as tool agents in real-world applications underscores the necessity for comprehensive evaluations of their capabilities, particularly in complex scenarios involving planning, creating, and using tools. However, existing benchmarks typically focus on simple synthesized queries that do not reflect real-world complexity, thereby offering limited perspectives in evaluating tool utilization. To address this issue, we present UltraTool, a novel benchmark designed to improve and evaluate LLMs’ ability in tool utilization within real-world scenarios. UltraTool focuses on the entire process of using tools from planning and creating to applying them in complex tasks. It emphasizes real-world complexities, demanding accurate, multi-step planning for effective problem-solving. A key feature of UltraTool is its independent evaluation of planning with natural language, which happens before tool usage and simplifies the task solving by mapping out the intermediate steps. Thus, unlike previous work, it eliminates the restriction of pre-defined toolset. Through extensive experiments on various LLMs, we offer novel insights into the evaluation of capabilities of LLMs in tool utilization, thereby contributing a fresh perspective to this rapidly evolving field. The benchmark is publicly available at https: //github.com/JoeYing1019/UltraTool.

heading: 1 Introduction
text: Recent advancements in equipping Large Language Models (LLMs) (Du et al., 2022; Touvron et al., 2023; Chiang et al., 2023; Bai et al., 2023) with external tools (Patil et al., 2023; Schick et al., 2023; Qin et al., 2023a) have markedly improved capability of AI systems in solving complex realworld tasks. As this field evolves, it becomes cru-
* This work was done during the internship at Huawei Noah’s Ark Lab.
† Corresponding authors.
cial to conduct a comprehensive evaluation covering full aspects of tool utilization, particularly within complex real-world contexts. As depicted in Figure 1 (a), addressing real-world tasks often necessitates not only the planning and usage of multiple tools but also the creation of new tools, when the existing tools are not enough to meet all specific requirements. However, existing benchmarks (Tang et al., 2023; Xu et al., 2023; Patil et al., 2023; Li et al., 2023; Qin et al., 2023b; Huang et al., 2023) often focus only on limited dimensions of this entire process. Besides, the user queries in most existing benchmarks exhibit limitations in mirroring the complexity of real-world tasks and unreasonable dependency on pre-defined toolsets.
To tackle these challenges, we introduce UltraTool, a benchmark covering a wide range of evaluation dimensions. It is constructed on complex, real-world queries and involves evaluation of tool-independent natural-language planning and advanced tool creation capabilities. UltraTool comprises 5,824 examples, spanning 22 diverse domains and incorporating 2,032 tools, and it comprehensively evaluates the tool utilization process
4363
including six dimensions covering three aspects: ① Planning: decomposing the complex goal into logical sequence of simpler sub-tasks for effective problem-solving. Tool Creation comprises two dimensions: ② awareness - assessing whether existing tools suffice, and ③ creation - developing the necessary tools if existing ones are inadequate. Tool Usage involves three dimensions: ④ awareness - determining which sub-tasks require tools, ⑤ selection - choosing the appropriate tools, and ⑥ usage - specifying the input parameters for these tools. As demonstrated in Table 1, recent benchmarks in tool utilization tend to focus only on limited dimensions. For examples, Tang et al. (2023); Patil et al. (2023); Li et al. (2023); Qin et al. (2023b) focus on tool usage (⑥), while Huang et al. (2023) evaluates tool usage awareness (④) and tool selection (⑤). Comparing with them, we further evaluate the capability in planning (①) and tool creation (② and ③), which is crucial for LLMs to adeptly navigate complex and varied real-world user demands.
Moreover, the queries in UltraTool have better realism and complexity. The natural workflow of tool-augmented task solving includes: “user queries LLMs → LLMs pose a solution plan → LLMs create/select tools to solve sub-task within the plan”. Importantly, the query and the plan should not be constrained by pre-existing tools. In contrast, existing benchmarks (Qin et al., 2023b; Li et al., 2023; Huang et al., 2023) are typically constructed by collecting toolset, generating simulated queries with tools randomly selected from toolset, and then annotating the solution as shown in Figure 1 (b). Despite the collecting efficiency, the generated queries may not accurately mirror real user demands. Furthermore, the randomly selected tools may lack a coherent logical relationship, potentially leading to skewed influences on the subsequent process. To ensure the complexity and reality of queries, we collect real-world multidomain user queries with high complexity. Specifically, we engage experts from various domains to craft complex queries that reflect real-world needs and potentially incorporate the use of diverse tools.
A distinct feature of UltraTool is it explicitly evaluates natural language (NL)-based plan, which simplifies task solving by decomposing a complex goal into several simpler sub-tasks described in NL. Each sub-task is then solved by tool creation and usage. Therefore, prior planning overcomes the restriction brought by limited pre-existing tools.
We conduct experiments on various LLMs and make in-depth analyses about strengths and challenges in the tool utilization of LLMs. Contributions of this work are: (1) UltraTool is a comprehensive evaluation benchmark derived from complex real-world queries, covering six key dimensions in tool utilization. (2) UltraTool explicitly evaluates NL-based planning and advanced tool creation capabilities. (3) Extensive experiments uncover the limitations and inspire the future direction of LLMs in tool utilization.
2 UltraTool Construction
UltraTool begins by gathering real-world user queries and encompasses a comprehensive range of evaluation dimensions, including planning, tool creation and tool usage. As depicted in Figure 4, the construction process of UltraTool includes: (1) query collection (§2.2); (2) solution annotation (§2.3); and (3) manual refinement (§2.4). For the detailed prompts utilized in the UltraTool construction process, please refer to Appendix A.5.

heading: 2.1 Definition
text: To formalize, a triple (Q,P, T ) is regarded as a sample within UltraTool. Specifically, Q represents the user query, while P = [(s1, t1), (s2, t2), ..., (sn, tn)] is the NL-based plan, In this context, each element (si, ti) comprises a step si and an associated tool calling message ti. Furthermore, T = [tool1, tool2, ..., toolm] is the corresponding toolset. As depicted in Figure 2, P maintains a hierarchical tree structure , which includes both ancestral steps (e.g., 1., 2., ...) and child steps (e.g., 1.1, 1.2, ...). The child steps fall into two distinct categories: (1) the tool-free step, which the LLMs can infer without requiring tool
callings, distinguished by the tool calling message ti as "No tool required"; and (2) the tool-usage step, which necessitates calling tools to complete the task, indicated by the tool callings message ti specifying the tool name toolnamei and the necessary arguments [{argsnamej : argsvaluej }]ikj=1. Here, ik denotes the number of arguments, argsnamej is the name of an argument belonging to the defined argument properties of tooli, and argsvaluej is the argument value derived from the query context Q or the outputs of preceding tool callings.

heading: 2.2 Query Collection
text: To ensure comprehensive coverage across various domains and meet diverse requirements in realworld scenarios, we carefully selected more than 20 domains such as Alarm, Train, Flight, Hotel, etc, as shown in Figure 3. In detail, we collaborate with a wide array of domain-specific experts to gather complex, de-identified user queries that necessitate complex tool usage as depicted in Figure 4 (a). To better align each query with its corresponding tool usage, we also request that the experts suggest several potential tools for each query. These suggestions are formatted in accordance with the style of ChatGPT plugins 1, which aided in crafting an initial toolset for each query. The query collection guideline can be found in Appendix A.2.
As shown in Figure 4 (a), to create more diverse and challenging queries, we utilized GPT-4 to generalize and complicate the initial queries. The examples of the generalization and complication processes are listed in Appendix A.4. We then merge the original, generalized, and complicated queries into a comprehensive collection. Additionally, every query undergoes a manual review to guarantee quality and applicability, and queries that do not align with actual human needs are removed.
1https://platform.openai.com/docs/guides/ function-calling

heading: 2.3 Solution Annotation
text: Due to the complexity of the queries addressed by UltraTool, UltraTool employs a multi-step plan as the backbone of solution for effective problemsolving, wherein each step includes a corresponding tool calling message. By breaking down user queries into simpler tasks via this planning process, these tasks can subsequently be executed by the designated tools. This strategy is not only in line with the problem-solving logic of humans but also reduces the complexity of tool usage, thereby facilitating the handling of more challenging tasks. We introduce an automatic solution annotation approach utilizing GPT-4, including: (1) plan annotation; (2) tool creation and plan refinement; (3) tool calling message annotation; and (4) tool merge. Plan Annotation. As illustrated in Figure 4 (b), upon receiving a user query, GPT-4 is utilized to formulate an initial multi-step, tree-structured plan [s1, ..., sn]. This fundamental plan is deliberately crafted without considering a predefined toolset. Such a strategy ensures a primary focus on task decomposition and the structural integrity of the plan. Moreover, it adeptly avoids the potential limitations that could surface when the initial toolset does not fully meet the requirements of user query.
Tool Creation and Plan Refinement. The initial toolset may not meet all requirements within the complex user queries in UltraTool, and the processes of generalization and complication may lead to emergence of new tool usage demands. Therefore, as depicted in Figure 4 (b), to accommodate the need for tools not present in initial toolset, we utilize GPT-4 to discern whether need to create new tools and create new tools in prescribed format.
Subsequently, we implement an automatic refinement process to enhance the quality of the NLbased plan. Given that the existing toolset has already fulfilled all the requirements of the user query, we provide GPT-4 with the query, the initial plan, and the comprehensive toolset (which includes both the initial and the newly created tools). This allows GPT-4 to refine the plan further in terms of the comprehensiveness of information included and the compatibility with tool callings. Tool Calling Message Annotation. As shown in Figure 4 (b), we utilize GPT-4 to annotate the tool calling messages [t1, ..., tn]. In detail, given the query, the refined plan, and the comprehensive toolset, GPT-4 initially differentiates between the tool-free steps and tool-usage steps. Subsequently, for each tool-usage step, GPT-4 selects the most appropriate tool from the comprehensive toolset. Moreover, utilizing the user’s query and the contextual information within the plan, GPT-4 generates the necessary arguments for the chosen tool.
It is also noteworthy that UltraTool incorporates nested tool callings, wherein the output from one tool serves as a requisite parameter for another subsequent tool. This feature significantly enhances our benchmark’s alignment with the complexities encountered in real-world scenarios.
Tool Merge. Due to each tool being created separately for the queries, there may be similar tools that exist in the whole benchmark. To address this issue, we manually merge groups of tools that have similar functions into a single tool. Specifically, we combine the arguments and results fields of similar tools and remove any duplicate properties, then we write an appropriate name and description for the merged tool. The remained toolset of UltraTool contains 2,032 distinct tools.

heading: 2.4 Manual Refinement
text: To guarantee and further enhance the data quality within UltraTool, we conduct a careful manual refinement process for all samples, as illustrated in Figure 4 (c). This process includes eliminating redundant steps, supplementing missing steps for coherent planning logic, rectifying tool usage demands and inappropriate tool selections, completing and refining the tool calling messages, and deleting data of substandard quality. We employ six experts endowed with specialized knowledge to refine the data, and we ensure that all the polished data undergoes a double-check process to maintain high-quality standards. The refinement guideline is provided in Appendix A.3.

heading: 2.5 Data Summary
text: Ultimately, we construct the UltraTool, which encompasses 22 domains, 2,032 tools, and 5,824 samples, detailed statistics are presented in Appendix A.1. And the source language of our collected data is Chinese, to broaden the scope of evaluation, we translate it to English version through GPT-4 and manual refinement. Therefore, UltraTool supports two languages: Chinese and
English, represented by the Chinese-dataset and the English-dataset.
3 UltraTool Evaluation
In this section, we introduce the metrics (§3.1), and the definition and measurement of each evaluation aspect, including: planning(§3.2), tool creation(§3.3) and tool usage(§3.4).

heading: 3.1 Metrics
text: Three evaluation metrics are employed within UltraTool, comprising: (i) Multi-Dimensional Point-Wise LLM-as-Judge Method (Liu et al., 2023), which utilizes a LLMscorer as automatic evaluator. By defining taskspecific evaluation dimensions M = [d1, d2, ...dm] and providing the query Q, the model’s response p, and the reference answer y, the LLM-scorer is prompted to provide a multi-dimensional score, along with an overall score:
S = LLM-scorer(M, [Q, p, y]), (1)
where S = [sd1 , ..., sdm , soverall] contains dimension scores sd1 , ..., sdm and the final overall score soverall, and all the scores are ranging from 1 to 10. The detailed prompts are listed in Appendix B.2. (ii) Key-Value based Accuracy and (iii) KeyValue based Levenshtein Distance, which are variants of traditional accuracy and Levenshtein distance (Miller et al., 2009) metrics. Given the keyvalue format model’s response (pk, pv) and reference answer (yk, yv), where the keys pk and yk represent steps and values pv and yv denote taskspecific results, these metrics compute the accuracy or normalized Levenshtein distance between the values when the keys match:
S = { F (pv, yv) if pk = yk 0 if pk ̸= yk , (2)
where S is the computated score and F represents the calculation function, which can applied by accuracy or normalized Levenshtein distance.
The multiple-step nature of the plan resulting in multiple predictions for a sample, so these metrics may be computed at global level that evaluates multiple predictions together or local level that evaluates each prediction separately. More illustration of computation level can be found in Appendix B.1.

heading: 3.2 Planning
text: Given a query Q, LLMs need to decompose the user query and generate a step sequence in hierarchical form [pPL1 , p PL 2 , ..., p PL n ] , to serve as the plan. Here, each pPLi represents a step. We utilize the Multi-Dimensional Point-Wise LLM-as-Judge Method, applied at a global level, to evaluate LLMs’ planning ability. The generated plan is evaluated across six score dimensions including Accuracy, Completeness, Executability, Syntactic Soundness, Structural Rationality and Efficiency, detailed dimension definitions are listed in Appendix B.3.1.

heading: 3.3 Tool creation
text: Since many real-world demands cannot be addressed by existing real-world tools, we design our tools as tool skeletons that contain all the necessary information for calling, as illustrated in Figure 2. Despite these skeletons are not specific implementations, they serve as simulated tools that accurately represent the tool’s functionality and provide guidance for the future development of actual tools.
Furthermore, given the golden plan P and corresponding toolset T , to evaluate whether the LLM can accurately be aware that the provided toolset is sufficient and effectively create the tools that are lacking, we construct an alternative toolset T̂ that may not contain all the necessary tools. Awareness. Given plan P and toolset T̂ , LLMs need to determine whether each tool-usage step in the plan can be matched with an appropriate tool from T̂ . The output is a predicted sequence [(si, p TCA i ), ..., (sj , p TCA j )], where p TCA i ∈ {0, 1} indicates the matching availability of a suitable tool within T̂ for step si. The evaluator of tool creation awareness is conducted through the KeyValue based Accuracy metric, which is calculated at both the global and local levels. Creation. Given plan P and toolset T̂ , LLMs are required to create the lacking tool in required format for those tool-usage steps that can not match a suitable tool in T̂ . The output is a predicted sequence [(si, pTCi ), ..., (sj , p TC j )], where p TC i denotes the created tool for step si. The Multi-Dimensional Point-Wise LLM-as-Judge Method, calculated at the global level, is adopted as the evaluator for tool creation. We evaluate the newly created tool across five score dimensions including Format Compliance, Accuracy, Content Reasonableness, Executability and Richness, corresponding dimension definitions can be found in Appendix B.4.1.

heading: 3.4 Tool Usage
text: Awareness. Given the plan P , LLMs need to determine whether child steps require the usage of tools and output a predicted sequence [(si, p TUA i ), ..., (sj , p TUA j )], where p TUA i ∈ {0, 1} indicates whether step si need to use tool. The tool usage awareness evaluator is Key-Value based Accuracy calculated at both global and local levels. Selection. Given the plan P and toolset T , we construct an augmented toolset T̄ by incorporating additional interference tools. Then LLMs are required to select the most appropriate tool from T̄ for each tool-usage steps and generate a predicted sequence [(si, pTSi ), ..., (sj , p TS j )], where p TS i denotes the name of the chosen tool for step si. The Key-Value based Accuracy calculated at both global and local levels is the evaluator for tool selection. Usage. Given the plan P , the toolset T , and the names of the required tools for each tool-usage step, LLMs are tasked with generating property arguments for each tool-usage step and outputting a predicted sequence [(si, pTUi ), ..., (sj , p TU j )]. Here, pTUi = [{argsnamej : argsvaluej }]kij=1 denotes the generated arguments for step si. The evaluation of tool usage awareness is conducted using Key-Value based Levenshtein Distance, which is calculated at the local level. We choose this metric for the consideration that the argument values argsvaluej may be expressed in many different manners.

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: We divide all data into two parts: a test set comprising 1,000 samples for constructing evaluation datasets, and a development set with 4,824 samples, which are not used for training and can optionally serve as a development set. For both the Chinese-dataset and English-dataset, we construct six evaluation datasets, each comprising 1,000 samples and corresponding to one of the six evaluation dimensions. The toolset sizes for T̂ in tool creation awareness and T̄ in tool selection are both set to 8.
We evaluate closed-source and open-source LLMs on UltraTool for both Chinese-dataset and English-dataset, aiming to provide comprehensive analyses for current LLMs. For closed-source LLMs, we select two representative models: GPT3.5 and GPT-4 from OpenAI. The version for GPT4 is gpt-4-1106-preview, and for GPT-3.5 is gpt-3.5-turbo-1106. For open-source LLMs, we choose a wide spectrum of models, including:
LLaMA2 (Touvron et al., 2023), ChatGLM3 (Du et al., 2022), Baichuan2 (Yang et al., 2023), Vicuna (Chiang et al., 2023), Qwen (Bai et al., 2023) and Mistral (Jiang et al., 2023). And All experiments are run on NVIDIA V100 GPUs.
To assist LLMs in grasping the desired output format, we provide a few-shot example as a demonstration for each evaluation task. For further details, the prompts are listed in Appendix B.6.

heading: 4.2 Main Results
text: The main results are illustrated in Table 2. As seen, we can have the following observations: GPT-4 achieves the best performance. Among all evaluated LLMS, GPT-4 demonstrates superior performance, achieving an overall score of 76.04% on Chinese-dataset and 74.58% on Englishdataset, and with particularly impressive results in the realms of tool creation and tool usage, setting the pilot of acting as skillful tool agent. Besides, Mistral-7B, GPT-3.5, and Qwen-72B also get competitive performance, and Mistral-7B gets the best of two worlds between exhibiting good tool utilization abilities while keeping efficient model scale. The larger the model scale, the better the tool utilization ability. Regarding open-source LLMs, we evaluate models at three scales, approximately 7B, 13B, and 70B parameters. The results indicate that tool utilization ability improves as the model scale increases. This finding aligns with prior research, which has shown that increasing of the model parameters bolsters the capabilities of LLMs (Chung et al., 2022; Wei et al., 2022). While most open-source LLMs demonstrate competitive planning performance, a notable gap still persists in tool creation and usage compared to closed-source LLMs, especially for smaller-scale models. This discrepancy may stem from the more intricate JSON output format requirements inherent in the evaluation dimensions of tool creation and usage. These dimensions are critical areas to focus on for enhancing LLMs’ abilities in tool utilization. Language affects the tool utilization ability. Although most of LLMs support both English and Chinese, their capabilities in these two languages differ. For open-source LLMs, in most cases, the Chinese-orientated LLMs (e.g. Qwen, Baichuan2) show better tool utilization ability on Chinese-dataset, while English-orientated LLMs (e.g. LLaMA2, Vicuna) perform better on Englishdataset. We attribute this discrepancy to variations in multilingual understanding resulting from differ-
ences in the proportions of training data. However, despite their English orientation, GPT-3.5 and GPT4 actually exhibit slightly weaker performance on English-dataset. We speculate that this could be due to heightened safety and alignment constraints in their English versions.

heading: 4.3 Fine-grained Analysis of Planning and Tool Creation
text: To offer a more refined analysis of the planning and tool creation capabilities of current LLMs, we present the multi-dimensional scores of 4 representative models under different model scales from the Multi-Dimensional Point-Wise LLM-as-Judge Method. More detailed multi-dimensional scores for planning and tool creation can refer to Appendix B.3.2 and Appendix B.4.2, respectively.
As illustrated in Figure 5 (a) and (b), we can observe that the GPT-4 and Qwen-70B are adept at breaking down complex goals into logically or-
dered, simpler sub-tasks. And the plans generated by smaller open-source LLMs showcase commendable Syntactic Soundness, yet reveal evident shortcomings across the remaining five score dimensions. This suggests that while many existing opensource LLMs demonstrate proficiency in generating grammatically accurate content, they still lack holistic language comprehension, particularly in fully grasping query requirements and structuring language to effectively break down tasks.
As shown in Figure 5 (c) (d), it is evident that GPT-4 and Qwen-70B are significantly ahead in all score dimensions, demonstrating their proficient tool creation capabilities. Conversely, the smallerscale models notably trail behind them across all score dimensions, signaling ample room for improvement in enhancing the abilities of many opensource LLMs concerning comprehensive query understanding, adherence to prescribed output structures, and the innovation of new tools.

heading: Executability

heading: Richness
text: Accuracy CompletenessSyntactic Soundness Executability
Structural Rationality Efficiency
60 65 70 75 80 85

heading: 4.4 Alignment of GPT-4 Scoring with Human Evaluation
text: To validate the effectiveness of the GPT-4 based Multi-Dimensional Point-Wise LLM-asJudge Method, we present the alignment of this metric with human evaluation on the UltraTool dataset. Specifically, for the evaluation of planning and tool creation, we randomly selected 140 evaluation samples for human evaluation. This selection incorporates the results of all 14 evaluated LLMs to ensure diversity and mitigate bias. By providing score dimensions, inputs, and references, we request human experts to evaluate these samples and provide a multi-dimensional score, along with an overall score, following the same scoring methodology as the Multi-Dimensional Point-Wise LLM-as-Judge Method. Subsequently, we calculate the Pearson correlation coefficient (Freedman et al., 2007) between human evaluation and GPT-4 evaluation for planning and tool creation. The results, depicted in Figure 6, indicate a high correlation between the scores of all dimensions and the overall scores from human evaluation and GPT-4 evaluation. This suggests that the Multi-Dimensional Point-Wise LLM-as-Judge Method is suitable for adoption in UltraTool evaluation, as it demon-
strates a strong alignment with human evaluation.

heading: 4.5 Error Analysis
text: We conduct analyses of the errors observed in evaluated LLMs, identifying five primary types: Not Following Instructions. Not following instructions is a major error type, causing a phenomenon of unanswered questions. The Error examples can refer to Appendix B.5.1. Hallucinations. LLMs often suffer from hallucinations (Ji et al., 2023), leading to outputs that include content outside the intended definition. The error examples are listed in Appendix B.5.2. Redundant Outputs. Redundant outputs indicate that outputs contain unnecessary/meaningless content. Error examples can refer to Appendix B.5.3. Incomplete Outputs. Incomplete outputs refer to outputs that lack necessary content. Corresponding error examples can be found in Appendix B.5.4. Incorrect JSON Format: In UltraTool, the expected output for planning is string format, while the other five dimensions require outputs in various JSON formats. Despite thorough post-processing, errors still persist due to incorrect JSON formatting. We calculate the proportion of outputs that adhere to the correct JSON format, namely the JSON format correct rate, and analyze its correlation with the
overall performance of UltraTool, as illustrated in Figure 7. It is evident that there exists a positive correlation between this ability and tool utilization, underscoring the importance of possessing strong skills in ensuring format compliance. For detailed results regarding the JSON format correct rate and examples of errors caused by incorrect JSON formatting, please refer to Appendix B.5.5.

heading: 5 Related Work

heading: 5.1 Tool Learning
text: The integration of external tools enhances the capabilities of LLMs to transcend the limitations of their training data, resulting in problem-solving abilities that are not only accurate and reliable but also highly specialized (Qin et al., 2023a). The approaches to LLMs’ tool learning can broadly be divided into two categories: tool-oriented learning and tool-augmented learning. The former involves directly fine-tuning the LLMs to master tool usage (Parisi et al., 2022; Hao et al., 2023; Xu et al., 2023; Shen et al., 2023; Schick et al., 2023), while the latter enhances LLMs with the ability to utilize tools through the provision of in-context tool descriptions and demonstrations (Mialon et al., 2023; Hsieh et al., 2023; Patil et al., 2023; Ruan et al., 2023). There are also works that explore the tool creation of LLMs (Qian et al., 2023; Cai et al., 2024), but still lack systematic evaluation of the tool creation abilities of current LLMs. With the rapid evolution of tool learning, conducting a comprehensive evaluation that encompasses all aspects of tool utilization has become vital, which is precisely the aim of UltraTool.

heading: 5.2 Tool Utilization Benchmark
text: Effective benchmarks allow for pinpointing limitations and charting the course for future developments. LLM tool utilization comprises three core aspects: planning, tool creation, and tool usage. Predominantly, existing benchmarks primarily focus on evaluating the model’s proficiency during the tool usage phase. This particular phase involves tool usage awareness of query, selection of the appropriate tool for each subtask (Huang et al., 2023) and the execution of specific tools (Xu et al., 2023; Tang et al., 2023; Li et al., 2023; Qin et al., 2023b; Ye et al., 2024). Unlike prior work, we conduct a comprehensive evaluation of tool utilization, which additionally evaluates capabilities for planning and tool creation, providing a more comprehensive and fine-grained analysis of tool utilization capabilities.

heading: 6 Conclusion
text: In this paper, we introduce UltraTool, a comprehensive evaluation benchmark derived from realworld complex queries, aimed at evaluating the tool utilization capabilities of LLMs across six dimensions that encompass three critical aspects: planning, tool creation, and tool usage. UltraTool excavates the necessary evaluation dimensions from actual tool utilization processes and pioneers in explicitly evaluating the NL-based planning and tool creation abilities. Our extensive analyses reveal that many current LLMs still have significant potential for enhancing their tool utilization abilities. We hope that UltraTool, coupled with our detailed experiments, will offer valuable insights and stimulate further research into the real-world application of LLMs in tool utilization.

heading: Acknowledgments
text: This work was partially supported by the National Natural Science Foundation of China 62176076, Natural Science Foundation of GuangDong 2023A1515012922, the Shenzhen Foundational Research Funding JCYJ20220818102415032, the Major Key Project of PCL2021A06, Guangdong Provincial Key Labo-ratory of Novel Security Intelligence Technologies 2022B1212010005.

heading: Limitations
text: This work contributes a comprehensive evaluation benchmark UltraTool for tool utilization based on real-world complex queries and covering comprehensive evaluation ranges. While promising, the
tools in UltraTool are not executable ; rather, they are tool skeletons that represent the tools’ functionalities, as many real-world demands cannot be fully addressed by existing real-world tools. Despite these tool skeletons are not specific implementations, they serve as simulated representations that accurately depict the tool’s functionality and offer guidance for the future development of actual tools. In the future, we plan to explore simulating the execution of our designed tools or gathering real-world data involving only executable tools.

heading: Ethics Statement
text: All data in UltraTool are de-identified and safeguarding privacy concerns. Our data construction and human evaluation processes are conducted by proficient experts, comprising postgraduate students from Chinese universities and employees of Chinese technology companies, all of whom receive fair payment for their contributions.

heading: A Data Details

heading: A.1 Data Statistics
text: UltraTool supports 2 languages including Chinese and English, and encompasses 22 domains, 2,032 tools, and 5,824 samples. Detailed statistics of the UltraTool are presented in Table 3. All samples within the UltraTool involve at least one tool calling. In particular, 5,318 samples comprise multiple tool callings, whereas 506 samples consist of single tool calling, and the percentage of nested tool callings in UltraTool is 39.61%. On average, each sample’s plan includes 12.27 steps and 2.72 tool callings, with each tool calling requiring an average of 3.05 arguments. And the existence of ancestral steps and tool-free steps results in a relatively low proportion of tool-usage steps among all steps.

heading: A.2 Query Collection Guideline
text: To ensure the high quality of collected user queries, we have developed a guideline that outlines the query collection standards for experts, which is illustrated in Figure 14.

heading: A.3 Manual Refinement Guideline
text: We also provide a comprehensive manual refinement guideline for experts, as depicted in Figure 15.

heading: A.4 Generalization and Complication Examples
text: We offer examples of the generalization and complication processes, as shown in Figure 8.
A.5 Prompt Template for UltraTool Construction
During the construction of UltraTool, we employed GPT-4 to perform a series of automated
tasks, which include query generalization (Figure 16), query complication (Figure 17), plan annotation (Figure 18, Figure 19), tool creation (Figure 20), plan refinement (Figure 21, Figure 22), and tool calling message annotation (Figure 23).

heading: B Evaluation Details

heading: B.1 Metric Computation Level
text: Due to the multiple-step nature of the plan P within UltraTool, it is common to make several predictions [(si, pi), ..., (sj , pj)] for the multiple steps of a given sample across distinct evaluation dimensions. To this end, we evaluate the performance in different levels, comprising both the global level and local level, where the former considers the sample as a whole and computes the metric based on all prediction results [pi, ..., pj ] in a sample, while the latter focus on individual steps and compute the metric based on a single-step prediction result pi.

heading: B.2 Prompt Template for Multi-Dimensional Point-Wise LLM-as-Judge Method
text: We utilize GPT-4 to apply Multi-Dimensional PointWise LLM-as-Judge Method for planning and tool creation evaluation. The prompt templates for Chinese-dataset and English-dataset can be found in Figure 24 (planning evaluation on Chinesedataset), Figure 25 (planning evaluation on Englishdataset), Figure 26 (tool creation evaluation on Chinese-dataset), Figure 27 (tool creation evaluation on English-dataset), respectively.

heading: Evaluation Task Model Example

heading: B.3 Planning Evaluation Details

heading: B.3.1 Score Dimension Definitions
text: The designed six score dimensions of MultiDimensional Point-Wise LLM-as-Judge Method for planning evaluation are: (1) Accuracy: The generated plan must be aligned with the user query’s objectives. (2) Completeness: The plan should encompass all tasks and constraints mentioned in the user query, ensuring no elements are omitted. (3) Executability: Every step in the generated plan ought to be logical and executable, forming a sequence that enables the progressive fulfillment of the user’s request. (4) Syntactic Soundness: The language of the generated plan should be grammatically sound. (5) Structural Rationality: The plan should exhibit a well-organized, tree-like hierarchical structure. (6) Efficiency: The plan must be concise and efficient, avoiding unnecessary complexity.

heading: B.3.2 Multi-Dimensional Scores
text: We provide the detailed multi-dimensional scores of Multi-Dimensional Point-Wise LLM-as-Judge Method on planning evaluation, as depicted in Table 4.

heading: B.4 Tool Creation Evaluation Details

heading: B.4.1 Score Dimension Definitions
text: The designed five score dimensions of MultiDimensional Point-Wise LLM-as-Judge Method for tool creation evaluation are: (1) Format Compliance: The created tool must be fully consistent with the standard answer in terms of format. (2) Accuracy: The created tool must align with the objectives of the user query and accurately address the user’s needs. (3) Content Reasonableness: The content within each field of the created tool should be reasonable.
(4) Executability: The tool name and description in the created tool should accurately express its function, including a comprehensive list of parameters and complete return results. (5) Richness: The created tool should encompass rich information, depth, contextual considerations, and diversity.

heading: B.4.2 Multi-Dimensional Scores
text: The detailed multi-dimensional scores of Multi-

heading: Dimensional Point-Wise LLM-as-Judge Method on
text: tool creation evaluation are also provided, as illustrated in Table 5.

heading: B.5 Error Examples
text: Through comprehensive observation, we categorize errors into five primary types: (1) not following instructions; (2) hallucinations; (3) redundant outputs; (4) incomplete outputs; and (5) incorrect JSON format.

heading: B.5.1 Not Following Instructions
text: The error examples for not following instructions are shown in Figure 9. Such errors signify outputs that deviate from the instructions’ requirements, potentially rendering them meaningless or irrelevant.

heading: B.5.2 Hallucinations
text: The error examples for hallucination are illustrated in Figure 10. This type of error indicates that the output includes content that does not within the intended definition, such as tools hallucination for argument names hallucination.

heading: B.5.3 Redundant outputs
text: The error examples for redundant outputs are demonstrated in Figure 11. This type of error suggests that the output includes unnecessary or nonsensical content, such as redundant meaningless texts or redundant predictions.
B.5.4 Incomplete outputs The error examples illustrating incomplete outputs are depicted in Figure 12. This type of error indicates that the output lacks essential content, such as incomplete predictions or fragmented tool structures.
B.5.5 Incorrect JSON Format In the UltraTool, the expected outputs for evaluation dimensions other than planning require diverse JSON formats, presenting more challenges due to the potential rendering of data unusable with even minor formatting errors. After thorough postprocessing, the proportion of outputs conforming to the correct JSON format among the evaluated LLMs, referred to as the JSON format correctness rate, is presented in Table 6. It is evident that as the
model scale increases, so does the JSON format correct rate.
Furthermore, error examples resulting from incorrect JSON formats are depicted in Figure 13. Such errors typically stem from incomplete or illegal JSON structures.

heading: B.6 Prompt Template for Inference
text: During the inference of our experiments, we utilize one-shot example for all LLMs. The prompt templates can be found in Figure 28 (planning), Figure 29 (tool creation awareness), Figure 30 (tool creation), Figure 31 (tool usage awareness), Figure 32(tool selection), Figure 33(tool usage), respectively.

heading: Evaluation Task Model Example

heading: Evaluation Task Model Example
text: Query Collection Guideline Assuming you are a user of an artificial intelligence assistant, based on your experience in the {domain} field, you are required to write a set of unique queries for the AI assistant. The requirements are as follows:

heading: 1. Authenticity: The queries should be based on real-world needs, reflecting scenarios that users might encounter. Please construct the queries based on real cases you are aware of or appropriately modified scenarios.

heading: 2. Complexity: The queries should be sufficiently complex to reflect the multiple decision points that may be needed to solve a problem, including the potential use of various tools. The best queries would require the use of multiple tools to obtain an answer.

heading: 3. Specificity: The queries should pertain to concrete needs within the {domain}, rather than being general or vague. Ensure that the queries are closely related to the specified domain and contain enough detail to understand their complexity.

heading: 4. Privacy: Avoid using real personal information. Construct the queries using pseudonyms and hypothetical scenarios.
text: 5. Diversity: The queries should cover a wide range of real-life needs. We encourage you to construct queries based on the needs of different user groups to reflect the diversity of the real world. For example, consider the needs that users of different ages, occupations, and cultural backgrounds might have.
After constructing each query, you will also need to list the potential tools required, including the tool's name, the parameters needed to call the tool within the query, and the expected return value of the tool.

heading: To assist you in understanding the type of queries required, here are some examples of queries and tools: {query example} {tool example}
text: Please note that these are only examples. We encourage you to provide a more diverse and specific set of complex queries, and ensure that the entire set of queries you provide is as non-repetitive as possible.
Figure 14: Guideline for query collection process.
Manual Refinement Guideline Given you a data for resolving user queries from an LLM (Language Model) perspective, which involves the calling of tools. This data includes the user's query (in the "question" field), a plan tailored to the user's query (in the "plan" field, where each step contains information about tool usage, indicated by the "tool" field), a list of tools involved in the plan (in the "tools" field), and a complete set of all the tools (the toolset). Your task is to perform detailed refinement on this data:
Definitions: 1. The plan includes three types of steps: tool-free steps (where the "tool" field value is "no tool required"), tool-usage steps (where the "tool" field specifies a particular tool calling), and some parent operations that are not of concern (where the "tool" field value is "null"). 2. When invoking a tool in a tool-usage step, the parameters are passed in the form "parameter name=specific value". 3. Refine the plan with minimal changes to the original plan, avoiding over-optimization and excessive modifications, while maintaining the sequential order of plan numbers.
Specific Refinement Guidelines: 1. Ensure the overall logic of the plan is coherent (each step should be based on the previous one and contribute to the overall goal, without contradictions or repetitions), the sequence of steps in the plan is reasonable (if not, adjust them to a logical order), there are no redundant steps (remove if any), and no missing steps (add if any). If adjustments involve renumbering, ensure the sequence remains orderly. 2. If a tool-free step actually requires the calling of a tool to solve the issue for the LLM, modify it to a toolusage step: find an appropriate tool from the list of tools or select one from the toolset (if no suitable tool is in the list), and populate it with the appropriate parameters, ensuring the completeness of the used parameters. 3. If a tool-usage step can actually be resolved without invoking a tool for the LLM, modify it to a tool-free step. 4. Ensure that the parameter values used in tool-usage steps have a source, which can be the user query, the result of pre-ordered tool calling, or some user-related information (such as name, identification card, account number, etc.). If the source of the parameter value used in a tool calling is the result of another tool calling XX (specific parameter name=XX), it should be modified to "specific parameter name=<XX>" to indicate that the parameter value is the result of a specific tool calling. 5. If the tool selection in a tool-usage step is inappropriate, select a more suitable tool from the toolset and populate the corresponding parameters. 6. If the format of the parameters used in a tool-usage step does not comply with "parameter name=specific value," it needs to be corrected. If any parameters are missing, they should be supplemented. If there is a clear mismatch between the used parameter names and their meanings and the content of the parameter values, modify them to appropriate values or select other reasonable parameter names defined by the tool. 7. If you feel that the data quality is indeed poor, you can also choose to delete the data directly.
To assist you in understanding the refinement requirement, here are some examples of data refinement: {example}
Figure 15: Guideline for manual refinement process.
Chinese Prompt 你是⼀个经验丰富的问题重写助⼿。给定⼀个问题，你的目标是将给定的问题进⾏重写，要求如下： 1. #重写后的问题#必须仍然是与#给定的问题#相似类型的问题。 2. #重写后的问题#要多样化，主题、格式、风格以及涉及的数字等都不能跟#给定的问题#⼀样。 3. #重写后的问题#必须是合理的，并且必须能被⼈类理解和回应。 4. 根据#给定的问题#重写2个不同的问题，#重写后的问题#对比#给定的问题#要有多样性。
以下是给定的问题： #给定的问题#: {query}
按如下格式输出： #重写后的问题#: {{ "问题1":"..." "问题2":"..." }}
English Prompt You are an experienced query rewriting assistant. Given an query, your goal is to rewrite the given query with the following requirements:

heading: 1. The #rewritten query# must still be a query of a similar type to the #given query#.

heading: 2. The #rewritten query# should be diversified; the theme, format, style, and any numbers involved must not be the same as the #given query#.

heading: 3. The #rewritten query# must be reasonable and must be capable of being understood and responded to by humans.

heading: 4. Rewrite 2 different queries based on the #given query#, ensuring that the #rewritten queries# demonstrate diversity when compared to the #given query#.
text: Below is the given query: #given query#: {query}
Output in the following format: #rewritten queries#: {{ ”query1":"..." ”query2":"..." }}
Figure 16: Prompt for query generalization process.
Chinese Prompt 作为⼀名资深的问题构建专家，你的目标是通过添加新的操作步骤或者约束条件，将输⼊的问题转化 为跟复杂的问题。请遵循以下指导原则： 1. 在#给定的问题#的基础上增加⼀到两个相关的条件约束或者能够通过⼯具API调用执⾏的操作步骤。 这些新增的操作步骤/条件约束应该与#给定的问题#逻辑相关联，并能够增加⼤语⾔模型解决问题所 需的推理和⼯具调用难度。 2. 避免在#重写后的问题#中使用“如果/若...则...”这类与选择(if-else)相关的复杂逻辑。 3. 避免#重写后的问题#形式为“请设计⼀个脚本/程序，该脚本/程序能够...”或“编写⼀个简短的脚本...”， 即不要求进⾏脚本编写。 4. 避免在#重写后的问题#中使用"给定问题"、"重写后的问题"等直接的原始术语，⽽是以自然流畅的 ⽅式表达。 5. 确保#重写后的问题#在难度上超越#给定的问题#，要求执⾏者采取更多的策略和处理更复杂的任 务。 6. 在增加复杂性的同时，避免不必要的冗长描述，尽可能保持问题的精炼、可读和可操作性。 7. 根据#给定的问题#重写2个不同的问题，#重写后的问题#对比#给定的问题#要有多样性，同时在 难度上超越#给定的问题#，要求执⾏者采取更多的策略和处理更复杂的任务。
对于以下给定的问题，请构建⼀个更为复杂的版本： #给定的问题#： {query}
#重写后的问题#： {{ "问题1":"..." "问题2":"..." }}
English Prompt

heading: As a seasoned query construction expert, your goal is to transform an input query into a more complex one by adding new operational steps or constraints. Please follow the guidelines below:
text: 1. Add one or two relevant conditional constraints or operational steps that can be executed through tool API calls, based on the #given query#. These added operations/steps or constraints should be logically associated with the #given query# and increase the difficulty of reasoning and tool invocation required for a large language model to solve the problem.

heading: 2. Avoid using complex logic related to selection (if-else) in the #rewritten query#, such as "if/then..." statements.

heading: 3. Avoid forms like "Please design a script/program that can..." or "Write a short script..." in the #rewritten query#, meaning that script writing is not requested.

heading: 4. Refrain from using direct terms such as "given query" or "rewritten query" in the #rewritten query#, but express it in a natural and fluent manner.

heading: 5. Ensure that the #rewritten query# is more challenging than the #given query#, requiring the executor to adopt more strategies and handle more complex tasks.

heading: 6. While increasing complexity, avoid unnecessary lengthy descriptions and maintain the conciseness, readability, and operability of the queries as much as possible.

heading: 7. Rewrite two different queries based on the #given query#. The #rewritten queries# should be diverse compared to the #given query# and more challenging, requiring the executor to adopt more strategies and handle more complex tasks.
text: For the following given query, please construct a more complex version: #given query#: {question}
#rewritten queries#: { "query 1": "..." "query 2": "..." }
Figure 17: Prompt for query complication process.
English Prompt
As a professional task planning assistant, you are capable of understanding complex user inquiries and
developing a structured, executable multi-step plan to fulfill the user's tasks. Specifically, the plans you create
should branch out like a tree into multiple subtasks, which are sequentially executed by underlying task
processors powered by a state-of-the-art Large Language Model (LLM) capable of handling many subtasks
and invoking external tools. An example of the output format is as follows:

heading: 1. Task 1
text: 1.1 Subtask 1.1
1.2 Subtask 1.2
1.2.1 Subtask 1.2.1
...

heading: 2. Task 2 ...
text: In the example, Task 1 includes Subtasks 1.1, 1.2, etc., while Subtask 1.2 further includes Subtasks 1.2.1, 1.2.2,
and so on. Each task must contain a description of the task. If a task contains multiple subtasks, the
relationship between different subtasks also needs to be explained, such as whether any subtask can be
completed, or all subtasks need to be executed in sequence to succeed. Important tips:
1. Tasks at the same level should have comparable complexity. Higher-level tasks can be more abstract. If a
task is too complex or not specific enough, it should be broken down into multiple executable subtasks. This
process should be recursively executed until a subtask can be addressed by a task processor that can invoke a
certain tool.

heading: 2. Always develop feasible and efficient plans to ensure the successful completion of tasks. Minimize the depth of the task tree and the number of subtasks, but ensure that all necessary goals are included.

heading: 3. Do not create new subtasks that are similar or identical to existing ones. For subtasks with similar objectives, try to merge them into one subtask, listing a series of sub-goals, rather than splitting them into multiple subtasks.

heading: 4. Please output the plan directly in the format shown above without generating additional explanations, as your response will be parsed and executed directly.
text: Your plan will be evaluated in the following aspects, so please pay attention to them: 1. **Completeness**: The plan should cover all aspects of the user's needs without omitting any constraints or fabricating requirements. Upon completion, the plan should indeed fulfill the user's task. 2. **Logical Coherence**: The plan should maintain logical consistency. That is, each step should be based on the previous one, and each step should contribute to achieving the overall goal. There should be no
contradictions or duplications in subtasks. The relationship between tasks and subtasks should be logical. 3. **Feasibility**: The plan should be clear enough to be understood. Higher-level tasks should be concise and in line with human task decomposition habits. The underlying tasks to be executed should be specific enough
and include all necessary aspects and details so that the task processor knows how to execute them. 4. **Format Compliance**: Only the plan itself, which meets the format requirements mentioned above, should be output without generating any additional redundant responses, to facilitate subsequent plan parsing
and execution.
Here are some examples of user questions and corresponding plans:
{example1}
{example2}
{example3}

heading: Below is the user's question: {question}
text: Next, please develop a plan based on the user's question, output format:
Plan: ...
Figure 19: English prompt for plan annotation process.
Chinese Prompt 你是⼀名专业的⼯具创造助⼿，给定⼀个#计划#和⼀个#现有⼯具#集合，你的任务是判断#现有⼯具# 是否⾜够完成任务，若不⾜够完成，需要进⾏相应的⼯具创造。要求如下： 1. 在#计划#中明确区分需要使用⼯具的步骤和不需要使用⼯具的步骤。 2. 对于需要使用⼯具的步骤，判断在#现有⼯具#是否能找到合适的⼯具，如果#现有⼯具#不⾜够完成 任务，需要创造对应的新⼯具。 3. 创造的新⼯具应遵循最小必要原则，具有通用性，并且参数化，以适应不同情况下的相似操作。 4. 创造的新⼯具的格式与#现有⼯具#中的⼯具格式相同。
以下是⼀些输⼊#计划#和#现有⼯具#，输出新⼯具的样例： {example1} {example2}
请根据以下#计划#和#现有⼯具#，参照样例格式，以json格式⽣成可能需要的新⼯具，注意保证json字 符串格式的正确性，不要输出其他任何内容。 #计划#： {plan} #现有⼯具# {tool} #输出#：
English Prompt You are a professional tool creation assistant. Given a #plan# and a collection of #existing tools#, your task is to determine whether the #existing tools# are sufficient to complete the task. If they are not, you will need to create the appropriate tools accordingly. The requirements are as follows:

heading: 1. In the #plan#, clearly differentiate between steps that require the use of tools and those that do not.

heading: 2. For steps that require the use of tools, assess whether suitable tools can be found within the #existing tools#.

heading: If the #existing tools# are insufficient to complete the task, new corresponding tools must be created.

heading: 3. The newly created tools should adhere to the principle of minimal necessity, be versatile, and parameterized to accommodate similar operations in different scenarios.

heading: 4. The format of the newly created tools should be consistent with the format of the tools within the #existing tools# collection.
text: Below are some examples of input #plans# and #existing tools#, and output of new tool samples: {example1} {example2}
Based on the following #plan# and #existing tools#, please refer to the sample format and generate the potentially required new tools in JSON format. Ensure the correctness of the JSON string format and do not output any other content. #plan#: {plan} #existing Tools#: {tool} #output#:
Figure 20: Prompt for tool creation process.
English Prompt You are a professional plan refinement assistant. Given an #user query#, with a corresponding #plan# and #toolset#, your task is to refine the provided tree-like #plan# to form an #refined plan#. Ensure that the leaf nodes in the #refined plan# are atomic operations, which represent the smallest indivisible steps in the execution plan. These atomic operations should match actions directly executable by an LLM (such as information retrieval, decision-making, reasoning, selection, text generation, etc.), or operations that interact with external systems (i.e., requiring tool invocation). The requirements are as follows:

heading: 1. Carefully review the #user query# and the provided #plan#. Formulate a preliminary #refined plan# for the #user query# by (1) removing redundant steps; (2) splitting, merging, refining, or adjusting existing steps to form more appropriate atomic operations; (3) adding new reasonable atomic operations.

heading: 2. If atomic operations include concrete nouns, abstract them in the form "(abstract noun: concrete noun)" to facilitate the passing of concrete nouns as parameters during the tool invocation process.

heading: 3. Clearly distinguish between atomic operations in the #refined plan# that require tool usage and those that do not.
text: 4. For atomic operations that require tool usage, there must be preceding atomic operations (that do not require tool usage) for information retrieval to acquire all necessary parameters for tool invocation. Following the tool invocation, there should be atomic operations (that do not require tool usage) for observing/using the results of the tool invocation. If these are missing, additional steps must be supplemented. 5. Atomic operations requiring tool usage should focus solely on the tool invocation and not on parameter retrieval (if so, separate them into a new information retrieval operation arranged beforehand). Ensure that all parameter values required by the tool come exclusively from the concrete nouns in the "(abstract noun: concrete noun)" format of preceding information retrieval atomic operations. If a parameter's concrete noun has not appeared in prior information retrieval steps, adjust the preceding steps or add corresponding information retrieval steps. However, avoid repetitive retrieval of the same concrete noun.

heading: 6. Ensure the #refined plan# maintains a structured, orderly tree form.
text: Below are examples of inputs for #user query#, #plan#, and #existing tools#, and outputs for the #refined plan#: {example1} {example2}
Please generate a #refined plan# in JSON format based on the following #user query#, #plan#, and #existing tools#, referring to the example format. Ensure the correctness of the JSON string format and do not output any other content. #user query#: {query} #plan#: {plan} #existing tools# {tool} #output#:
Figure 22: English prompt for plan refinement process.
English Prompt
As a professional assessment expert, your task is to objectively assess the quality of the provided data in reference to the standard answer, based on the given assessment dimensions. Given a user instruction, the standard answer, and a tool created in response to the user instruction, please score the quality of the created tool according to the following assessment dimensions:
1. Format Compliance: The created tool should be completely consistent with the standard answer in terms of format, fully including the basic components such as the tool name ("name" field), tool description ("description" field), list of arguments ("arguments" field, with "type" and "properties" fields within "arguments") and return values ("results" field, with "type" and "properties" fields within "results").

heading: 2. Accuracy: The created tool should align with the objectives of the user instruction and accurately address the user's needs.

heading: 3. Content Reasonableness: The content within each field of the created tool should be reasonable, including clear expression and solid grammar in the natural language description fields, as well as sensible types for each defined argument and return value.

heading: 4. Executability: The tool name and description defined in the created tool should appropriately express its function, with a comprehensive list of parameters and complete return values.

heading: 5. Richness: The created tool should include rich information, depth, contextual considerations, and diversity.
text: Overall, the higher the quality of the model answer, the higher the score. As an example, the standard answer can score 8 points in each dimension and in total. Compare the standard answer and step by step score each of the above assessment dimensions, then provide an overall score based on all dimensions. The specific criteria for the overall score are as follows: The total score must be 1 point if the model answer is irrelevant to the question, contains essential factual errors, or generates harmful content. The total score should be 2 to 3 points if the model answer is of low quality without serious errors and is harmless but does not meet user needs. The total score can be 4 to 6 points if the model answer generally meets user requirements but performs poorly in some dimensions and is of mediocre quality. The total score should be 7 to 8 points if the model answer's quality is close to the standard answer and performs well in all dimensions.

heading: A score of 9 to 10 points is only achievable if the model answer significantly surpasses the standard answer, fully resolves the user's issue and all requirements, and approaches a perfect score in all dimensions.
text: You must provide your assessment results in the following format: [ {{"Scoring Reason": <Provide reasons for scoring against the definition of format compliance and the standard answer>, "Format Compliance Score": <Assign a score between 1 to 10>}}, {{"Scoring Reason": <Provide reasons for scoring against the definition of accuracy and the standard answer>, "Accuracy Score": <Assign a score between 1 to 10>}}, {{"Scoring Reason": <Provide reasons for scoring against the definition of content reasonableness and the standard answer>, "Content Reasonableness Score": <Assign a score between 1 to 10>}}, {{"Scoring Reason": <Provide reasons for scoring against the definition of executability and the standard answer>, "Executability Score": <Assign a score between 1 to 10>}}, {{"Scoring Reason": <Provide reasons for scoring against the definition of richness and the standard answer>, "Richness Score": <Assign a score between 1 to 10>}}, {{"Scoring Reason": <Provide reasons for scoring against all assessment dimensions and the standard answer>, "Total Score": <Assign a score between 1 to 10>}}, ]
Below are the given user instruction, standard answer, and the created tool to be evaluated: User instruction: {question} Standard answer: {reference} Created tool: {answer}
Based on the above assessment dimensions and comparing against the standard answer, score each dimension for the created tool to be evaluated, then provide an overall score. The final output should be in the form of a JSON string, without any additional content. Output:
Figure 27: Prompt for Multi-Dimensional Point-Wise LLM-as-Judge Method on English-dataset tool creation evaluation.
Chinese Prompt 你是⼀个专业的计划制定助⼿，给你⼀个用户问题，你的任务是充分理解用户问题，并制定⼀个合理 的、可执⾏的多步计划以完成用户的任务，具体⽽⾔，你制订的计划应像⼀棵树⼀样分为多个⼦任务。 输出格式为字符串（内容是使用换⾏符分隔的⼀系列⼦任务），例如：1. 任务1 \n 1.1 任务1.1 \n 1.2 任 务1.2 \n 1.2.1 任务1.2.1 \n ... \n 2. 任务2 \n ...
你应该严格遵守输出格式要求，不要输出其他任何内容。 样例： 问题：{input} 输出：{reference}
让我们开始吧！ 问题：{input} 输出：
English Prompt You are a professional planning assistant. Given a user's question, your task is to fully understand the user's question and create a reasonable, executable multi-step plan to complete the user's task. Specifically, your plan should be like a tree with multiple subtasks. The output format is a string (content is a series of subtasks separated by newline characters), for example: 1. Task 1 \n 1.1 Task 1.1 \n 1.2 Task 1.2 \n 1.2.1 Task 1.2.1 \n ... \n 2. Task 2 \n ...
You should strictly follow the output format requirements and not output any other content. Example: Question: {input} Output: {reference}

heading: Let’s Begin! Question: {input} Output:
text: Figure 28: Prompt for planning inference on Chinese-dataset and English-dataset.
Chinese Prompt 你是⼀个专业的⼈⼯智能助⼿，给你⼀个计划和相应的⼯具集，计划中有些步骤带有'tool'字段，你的 任务是判断这些步骤能否在给定的⼯具集中找到合适的⼯具，如果⼯具集中没有合适的⼯具（需要创 造新的⼯具）则'tool'字段为'1'，否则为'0'。输出格式为元素是dict的list，每个dict包含'step'（给定计划 中有'tool'字段的步骤）和'tool'（判断，'0'或'1'）字段，例如：[{"step": "2.1 步骤2.1", "tool": "1"}, {"step": "4.2 步骤4.2", "tool": "0"}, ...]
你应该严格遵守输出格式要求，不要输出其他任何内容。 样例： 计划：{input} ⼯具集：{toolset} 输出：{reference}
让我们开始吧！ 计划：{input} ⼯具集：{toolset} 输出：
English Prompt You are a professional AI assistant. Given a plan and a corresponding set of tools, some steps in the plan have a 'tool' field. Your task is to determine whether it is possible to find an appropriate tool in the given toolset for these steps. If there is no suitable tool in the toolset (requiring the creation of a new tool), then the 'tool' field should be '1', otherwise '0'. The output format is a list of dicts, each dict contains 'step' (the step in the given plan that has a 'tool' field) and 'tool' (judgment, '0' or '1'), for example: [{"step": "2.1 step 2.1", "tool": "1"}, {"step": "4.2 step 4.2", "tool": "0"}, ...]
You should strictly follow the output format requirements and not output any other content. Example: Plan: {input} Toolset: {toolset} Output: {reference}
Let’s Begin! Plan: {input} Toolset: {toolset} Output:
Figure 29: Prompt for tool creation awareness inference on Chinese-dataset and English-dataset.
Chinese Prompt 你是⼀个专业的⼯具创造助⼿，给你⼀个计划和⼀个⼯具集，计划中有些步骤带有'tool'字段，你的任 务是参照给定⼯具集中的⼯具的格式，为这些步骤创造对应的⼯具，并在'tool'字段填⼊所创造的⼯具。 输出格式为元素是dict的list，每个dict包含'step'（给定计划中有'tool'字段的步骤）和'tool'（创造的⼯具， dict格式）字段，例如：[{"step": "2.3 步骤2.3", "tool": {"name": ..., "description": ..., "arguments": {"type": ..., "properties": {...}}, "results": {"type": ..., "properties": {...}}}}, ...]
你应该严格遵守输出格式要求，不要输出其他任何内容。 样例： 计划：{input} ⼯具集：{toolset} 输出：{reference}
让我们开始吧！ 计划：{input} ⼯具集：{toolset} 输出：
English Prompt You are a professional tool creation assistant. Given a plan and a toolset, some steps in the plan have a 'tool' field. Your task is to create corresponding tools for these steps, referring to the format of tools in the given toolset, and fill in the 'tool' field with the created tool. The output format is a list of dicts, each dict contains 'step' (the step in the given plan that has a 'tool' field) and 'tool' (the created tool, dict format), for example: [{"step": "2.3 step 2.3", "tool": {"name": ..., "description": ..., "arguments": {"type": ..., "properties": {...}}, "results": {"type": ..., "properties": {...}}}}, ...]
You should strictly follow the output format requirements and not output any other content. Example: Plan: {input} Toolset: {toolset} Output: {reference}
Let’s Begin! Plan: {input} Toolset: {toolset} Output:
Figure 30: Prompt for tool creation inference on Chinese-dataset and English-dataset.
Chinese Prompt 你是⼀个专业的⼈⼯智能助⼿，给你⼀个计划，计划中有些步骤带有'tool'字段，你的任务是判断完成 这些步骤是否需要使用⼯具，如果需要使用⼯具则'tool'字段为'1'，否则为'0'。输出格式为元素是dict的 list，每个dict包含'step'（给定计划中有'tool'字段的步骤）和'tool'（判断，'0'或'1'）字段，例如： [{"step": "1.1 步骤1.1", "tool": "0"}, {"step": "2.3 步骤2.3", "tool": "1"}, ...]
你应该严格遵守输出格式要求，不要输出其他任何内容。 样例： 计划：{input} 输出：{reference}
让我们开始吧！ 计划：{input} 输出：
English Prompt You are a professional AI assistant. Given a plan, some steps in the plan have a 'tool' field. Your task is to determine whether tools are needed to complete these steps. If tools are required, the 'tool' field should be '1', otherwise '0'. The output format is a list of dicts, each dict contains 'step' (the step in the given plan that has a 'tool' field) and 'tool' (judgment, '0' or '1'), for example: [{"step": "1.1 step 1.1", "tool": "0"}, {"step": "2.3 step 2.3", "tool": "1"}, ...]
You should strictly follow the output format requirements and not output any other content. Example: Plan: {input} Output: {reference}

heading: Let’s Begin! Plan: {input} Output:
text: Figure 31: Prompt for tool usage awareness inference on Chinese-dataset and English-dataset.
Chinese Prompt 你是⼀个专业的⼯具参数填充助⼿，给你用户问题，问题对应的计划和相应的⼯具集，计划中有些步 骤带有'tool'字段（⼯具名）和'param'字段，你的任务是先通过'tool'字段在⼯具集中找到对应⼯具，然 后为这些步骤的'param'字段填⼊调用对应⼯具所需要的参数（参数格式为"参数名=参数值"，多个参数 用','分隔），参数名来自于对应⼯具的'arguments'中的'properties'字段（不需要全部使用），参数值来 源于用户问题和每个步骤的前序步骤中的信息（若参数值来自于前序⼯具调用的返回值，用<>标识）。 输出格式为元素是dict的list，每个dict包含'step'（给定计划中有'tool'字段的步骤），'tool'（步骤对应的 'tool'字段）和'param'（填充的参数，dict格式）字段，例如： [{"step": "3.2 步骤3.2", "tool": "⼯具名1", "param": {"参数名1": "参数值1", "参数名2": "<参数值2>", ...}}, ...]
你应该严格遵守输出格式要求，不要输出其他任何内容。 样例： 问题：{question} 计划：{input} ⼯具集：{toolset} 输出：{reference}
让我们开始吧！ 问题：{question} 计划：{input} ⼯具集：{toolset} 输出：
English Prompt You are a professional tool parameter filling assistant. Given a user's question, the corresponding plan, and a set of tools, some steps in the plan have a 'tool' field (tool name) and a 'param' field. Your task is to first find the corresponding tool in the toolset through the 'tool' field, and then fill in the 'param' field for these steps with the parameters required to call the corresponding tool (parameter format is "parameter name=parameter value", separate multiple parameters with ','), where the parameter names come from the 'properties' field in the 'arguments' of the corresponding tool (not all need to be used), and parameter values come from the user's question and information from the previous steps (if the parameter value comes from the return value of a previous tool call, use <> to indicate). The output format is a list of dicts, each dict contains 'step' (the step in the given plan that has a 'tool' field), 'tool' (the corresponding 'tool' field of the step), and 'param' (the filled parameters, dict format), for example: [{"step": "3.2 step 3.2", "tool": "Tool Name 1", "param": {"parameter name1": "parameter value1", "parameter name2": "<parameter value2>", ...}}, ...]
You should strictly follow the output format requirements and not output any other content. Example: Question: {question} Plan: {input} Toolset: {toolset} Output: {reference}
Let’s Begin! Question: {question} Plan: {input} Toolset: {toolset} Output:
Figure 33: Prompt for tool usage inference on Chinese-dataset and English-dataset.

heading: Future Work
text: Despite these tool skeletons are not specific implementations, they serve as simulated representations that accurately depict the tool’s functionality and offer guidance for the future development of actual tools. In the future, we plan to explore simulating the execution of our designed tools or gathering real-world data involving only executable tools