ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
Authors: Yujia Qin
Shihao Liang
Yining Ye
Kunlun Zhu
Lan Yan
Yaxi Lu
Yankai Lin
Xin Cong
Xiangru Tang
Bill Qian
Sihan Zhao
Lauren Hong
Runchu Tian
Ruobing Xie
Jie Zhou
Mark Gerstein
dahai li
Zhiyuan Liu
Maosong Sun
Venue: ICLR 2024 spotlight (2024)
Keywords: Large Language Model
Tool Use
API Use
Primary area: generative models

Abstract: Despite the advancements of open-source large language models (LLMs), e.g., LLaMA, they remain significantly limited in tool-use capabilities, i.e., using external tools (APIs) to fulfill human instructions. The reason is that current instruction tuning largely focuses on basic language tasks but ignores the tool-use domain. This is in contrast to the excellent tool-use capabilities of state-of-the-art (SOTA) closed-source LLMs, e.g., ChatGPT. To bridge this gap, we introduce ToolLLM, a general tool-use framework encompassing data construction, model training, and evaluation. We first present ToolBench, an instruction-tuning dataset for tool use, which is constructed automatically using ChatGPT. Specifically, the construction can be divided into three stages: (i) API collection: we collect 16,464 real-world RESTful APIs spanning 49 categories from RapidAPI Hub; (ii) instruction generation: we prompt ChatGPT to generate diverse instructions involving these APIs, covering both single-tool and multi-tool scenarios; (iii) solution path annotation: we use ChatGPT to search for a valid solution path (chain of API calls) for each instruction. To enhance the reasoning capabilities of LLMs, we develop a novel depth-first search-based decision tree algorithm. It enables LLMs to evaluate multiple reasoning traces and expand the search space. Moreover, to evaluate the tool-use capabilities of LLMs, we develop an automatic evaluator: ToolEval. Based on ToolBench, we fine-tune LLaMA to obtain an LLM ToolLLaMA, and equip it with a neural API retriever to recommend appropriate APIs for each instruction. Experiments show that ToolLLaMA demonstrates a remarkable ability to execute complex instructions and generalize to unseen APIs, and exhibits comparable performance to ChatGPT. Our ToolLLaMA also demonstrates strong zero-shot generalization ability in an out-of-distribution tool-use dataset: APIBench.

Published as a conference paper at ICLR 2024
TOOLLLM: FACILITATING LARGE LANGUAGE MODELS TO MASTER 16000+ REAL-WORLD APIS
Yujia Shihao Liang',, Yining Ye!, Kunlun Zhu', Lan Yan', Yaxi Lu', Yankai Lin Xin Cong', Xiangru Tang', Bill , Sihan Zhao', Lauren Hong', Runchu Tian', Ruobing Xie Jie Zhou', s Mark Gerstein', Dahai Li Zhiyuan Liu't, Maosong Sun 'Tsinghua University "ModelBest Inc. RRemmin University of China "Yale University "WeChat AI, Tencent Inc. ZZhhu Inc. yujiaginl 60 gmail. com
ABSTRACT
Despite the advancements of open-source large language models (LLMs), e.g., LLaMA, they remain significantly limited tool-use capabilities, i.c., using exter- nal tools (APIs) to fulfill human instructions. The reason is that current instruction tuning largely focuses on basic language tasks but ignores the tool-use domain. This is in contrast to the excellent tool-use capabilities of state-of-the-aa (SOTA) closed-source LLMs, e.g., ChatGPT. To bridge this gap. we introduce TooILLM, a general tool-use framework encompassing data construction, model training, and evaluation. We first present ToolBench, an instruction-tuning dataset for tool use, which is constructed automatically using ChatGPT. Specifically, the con- struction can be divided into three stages: (i) API collection: we collect 16, 464 real-world RESTful APIs spanning 49 categories from RapidAPI Hub; (ii) instruc- tion generation: we prompt ChatGPT to generate diverse instructions involving these APIs, covering both single-tool and multi-tool scenarios; (iii) solution path annotation: we use ChatGPT to search for a valid solution path (chain of API calls) for each instruction. To enhance the reasoning capabilities of LLMs, we develop novel depth-first search-based decision tree algorithm. It enables LLMs evaluate multiple reasoning traces and expand the search space. Moreover, to evaluate the tool-use capabilities of L.MMs, we develop automatic evalu- ator: ToolEval. Based on ToolBench, we fine-tune I.LaMA to obtaín an I.I.M TooALLaMA, and equip it with neural API retriever recommend appropriate APIs for each instruction. Experiments show that TooILLaMA demonstrates a remarkable ability execute complex instructions and generalize to unseen APIs, and exhibits comparable performance to ChatGPT, Our TooILLaMA also demon- strates strong shot generalization ability in out- df-tiitribttiinn tool-use dataset: APIBench. The codes, trained models, and demo are publicly available at httpp::/gi ttt.:/gtttuuu...eee com Bttp://b...com/m/.M/MMM/////
Bench
INTRODUCTION
Tool learning QQin al. 2023b) aims unleash the power of large language models (LLMs) to effec- tively interact with various tools (APIs) to accomplish complex tasks. By integrating LLMs with APIs, we can greatly expand their utility and empower them to serve as efficient intermediaries between users and the vast ecosystem of applications. Although open-source LLMs, e.g., LLaMA (Touvron et al., 2023a). have achieved versatile capabilities through instruction tuning TTaori et al.) 2023] Chiang al. 2023), they still lack the sophistication in performing higher-level tasks, such appro- priately interacting with tools (APIs) to fulfill complex human instruction. This deficiency is because current instruction tuning largely focuses basic language tasks, with relative neglect of the tool-use domain. On the other hand, current state-of-the-art (SOTA) LLMs (e.g., ChatGPT (OpenAI
Indicates equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2024
Data Construction Train & Inference
API Collection
Instruction Generation
Solution Path Annotation
API Retriever
Retrieved APls
Instruction
Instructions relevant
ToolBench
Final Answer
XN
APp
SFT
Rapidari
RapidAPI
TooiLLaMA
LLaMA
ToolEval
Figure 1:
Three phases of constructing ToolBench and how we train our API retriever and TooILLaMA. During inference of an instruction, the API retriever recommends relevant APIs to LaMA,LaMA, which performs multiple rounds of API calls to derive the final answer. The whole reasoning process evaluated by ToolEval.
2022) and GPT-4 (OpenAl 2023)), ) which have demonstrated impressive competencies in utilizing tools (Bubeck et al. (202)), are coseed-ouuce with their inner mechanisms opaque. This limits the democratization of AT Technologies and the scope of community-drive innovation and development. In this regard, we deem it urgent to empower open-source LLMs to skillfully master diverse APIs.
Although prior works have explored building instruction tuning data for tool use (Li et al. 2023a Patil et al. 2023 Tang et al., 2023 Xu et al., 2023b), they fail to fully stimulate the tool-use capabilities within LLMs and have inherent limitations: (1) limited A they either fail to in-
volve real-world APIs (e.g., RESTAPI) (Patil et al. 2023 Tang et al. 2023) consider only
small scope of APIs with poor diversity (Patil et al. 2003 Xu et al.
2023b et al. 2023a);
(2) constrained scenario: existing works are confined to instructions that only involve one single tool. In contrast, real-world scenarios may require that multiple tools in- terleaved together for multi-roum tool execution to solve complex task. Besides, they often assume that users manually specify the ideal API set for a given instruction in advance, which is infeasible with large collection of real-world APIs; (3) inferior planning and reasoning: existing works adopted either CoT (Wei et al. (023) or ReACT (Yao et al. (2022) for model reasoning, which can- not fully elicit the capabilities stored in LLMs and thus fail to handle complex instructions. In addition, some works do not even execute APIs to obtain real responses (Patil et al. 2023 Tang et al., 2023), which serve as important information for subsequent model planning.
OPTE
0.8
Win Rate
ChanPt RWAKT
da RWAGT
RANGI
0.0
Vicana S.lpscs
0.8
Pass Rate
Pass rate (f) and win rate (1) of different methods tool-use evaluation. For win rate, compare each method with ChatGPT-ReACT DFSDT is our improved reasoning strategy over ReACT. LaMA surpasses Dax--Davnnci-033, Claude-2, and almost performs on par with ChatGPT.
Figure 2:
To facilitate tool-use capabilities within open-source LLMs, we introduce ToolLLM, a general tool-use frame- work including data construction, model training, and eval- uation. As illustrated in Figure we collect high-qualit, instruction tunning dataset ToolBench. is constructed automatically using ChatGPT (gpt-3.5-turbo- 16k), which has been upgraded with function call (link) capabilities.
The comparison between ToolBench and prior works is listed in Table Specifically, the construction of ToolBench entails three phases:
API Collection: we gather 16,464 representational state transfer (REST) APIs from RapidAPI (link), platform that hosts massive real-world APIs provided by developers. These APIs span 49 diverse categories such as social media, e-commerce, and weather. For each API, we crawl detailed API documents from RapidAPI, including the functionality descriptions, required parameters, code snippets for API calls, etc. By comprehending these documents to learn to execute APIs, LLMs can generalize new APIs unseen during training;
Instruction Generation: we first sample APIs from the whole set and then prompt ChatGPT to generate diverse instructions for these APIs. To cover practical scenarios, curate instructions
2

---

Published as a conference paper at ICLR 2024
ToolBench (this work)
APIBench
API-Bank
ToolAlpaca
ToolBench al. 2023b)
Resource
et al. 2023
2023a
Tang et 2023
Real-world API?
Real API Cespnees
X
Multi-tool Scenario?
X
API Retrieval?
V
X
X
Multi step Reasoning?
X
Number of tools
3451
3
400
Number of APIs
164664
1645
53
400
232
Number of Instances
126486 469585
17002
271
3938
27-16
Number of Real API Calls
0
568
3926
Avg. Reasoning Traces
1.0
1.0
2.1
1.0
5.9
Table 1: A comparison of our ToolBench to notable instruction tuning dataset for tool learning.
that involve both single-tool and multi-tool scenarios. This ensures that our model learns not only how to interact with individual tools but also how to combine them to accomplish complex tasks;
Solution Path Annotation: each solution path may contain multiple rounds of model reasoning and real-time API calls to derive the final response. However, even the most sophisticated LLM, i.e, GPT-4, achieves low pass rate for complex human instructions, making annotation inefficient. To this end, we develop novel depth-first search-based decision tree (DFSDT) to bolster the planning and reasoning ability of LLMs.. Compared with conventional ReACT, DFSDT enables LLMs to evaluate a multitude of reasoning paths and make deliberate decisions to either retract steps or proceed along a promising path. In experiments, DFSDT significantly improves the annotation efficiency and successfully completes those complex instructions that cannot be fulfilled using ReACT.
To assess the tool-use capabilities of LLMs, we develop an automatic evaluator, ToolEval, backed up by ChatGPT. comprises two key metrics: (1) pass rate, which measures LLM's ability to successfully execute an instruction within limited budgets, and (2) win rate, which compares the quality and usefulness of two solution paths. We demonstrate that ToolEval achieves a high correlation with human evaluation and provides a robust, scalable, and reliable assessment for machine tool use.
By fine-tuning LLaMA on ToolBench, we obtain TooILLaMA. After evaluation based on our ToolEval, we derive the following findings:
ToMILLaMA demonstrates a compelling capability to handle both single-tool and complex multi- tool instructions. As depicted in Figure 2 TooILLaMA outperforms Text-Davinci-003 and Claude-2, achieves comparable performance the "teacher model" ChatGPT, and is only slightly inferior to GPT4. Besides, TooILLaMA exhibits robust generalization to previously unseen APIs, requiring only the API documentation adapt to new APIs effectively. This flexibility allows users to incorporate novel APIs seamlessly, thus enhancing the model's practical utility.
We show that DFSDT serves general meeeiionnmmaknn strategy enhance the reasoning capabilities of LLMs. DFSDT broadens the search space by considering multiple reasoning traces and achieves significantly better performance than ReACT.
We train a neural API retriever, which alleviates the need for manual selection from the large API pool in practice. As shown Figure [ given an instruction, the API retriever recommends a set of relevant APIs, which are sent to TooILLaMA for multi-round decision making to derive the final answer. Despite sifting through large pool of APIs, the retriever exhibits remarkable retrieval precision, returning APIs closely aligned with the ground truth.
TooIL.LMA exhibits strong generalization performance on an out-of -ntt--i-iitriittionn (OOD) dataset APIBench (Patil et al. 2023). Despite not training any of the APIs instructions on APIBench, TooILLaMA performs on par with Gorilla, a pipeline specifically designed for APIBench.
2 DATASET CONSTRUCTION
We introduce the three-stage construction process of ToolBench: API collection ($ 2.11.). instruction generation ( (.2). and solution path annotation ($ (.3).. All procedures are based on ChatGPT (gpt-3.5-turbo-1okk requiring minimal human supervision and can be easily extended to new APIs.
3

---

Published as a conference paper at ICLR 2024
R Rapid
Single tool Instructions
Category
Collection
Category
Tool
Finance
Movies
Code
API
category multi-ton instructions
Intra Intra-colieen col.-colllccinnnnooo multi-tool instructions
Wars Characters
Streaming Avaiiaility
Search
Categury/ Cullection
Sampled
AP1
GET Services
GET
APls & API
Sampled
Documontation
APIL
API Documentation
API Name. Search By Titl API Description: Seorch movics and series by bill, Ilqquired Parameters: [3) titl (stning, itte to search foel, (2) country jatiing ..)
AP12
Sampled
Manoo
Optional Parameters (1) show type (string. Type of shows to ioclude in the results, othher "novie", "series, "oo" Defustt "20"), 12) ousput language istring... www.rt//teetttttttttoo wow.nnyyruuurruroboooooo Example Response: "movie" Uille Belman" p poesino: "lapanene...
Indrraction
plann
right the mountsilss Can
Code Snippets:
Instructions & Relevant APIs
STA suggett
Relevam APC
AMLL AF12 APSS..
Figure 3: The hierarchy RapidAPI (left) and the process of instruction generation (right).
2.1 API COLLECTION
We start by introducing RapidAPI and its hierarchy, followed by how we crawl and filter APIs.
RapidAPI Hub RapidAPI is leading API marketplace that connects developers with thousands of real-world APIs, streamlining the process of integrating diverse services into applications. Developers can test and connect with various APIs by registering only a RapidAPI key. All APIs in RapidAPI can be classified into 49 coarse-grained categories (link), such as sports, finance, and weather. The categories associate API with the most relevant topic. Additionally, the hub also provides 500- fine teeiigr categorization called collections (link). e.g., Chinese APIs and database APIs. APIs in the same collection share common characteristic and often have similar functionalities or goals.
Hierarchy of RapidAPI As shown in Figure 3 each tool may be composed of multiple APIs. For each tool, crawl the following information: the name and description of the tool, the URL of the host, and all the available APIs belonging to the tool; for each API, we record its name, description, HTTP method, required parameters, optional parameters, request body, executable code snippets for API call, and an example API call response. This rich and detailed metadata serves as a valuable resource for LLMs to understand and effectively use the APIs, even zero-shot manner.
API Filtering Initially, we gathered 10, 853 tools (53, 190 APIs) from RapidAPI. However, the quality and reliability of these APIs can vary significantly. In particular, some APIs may not be well-maintained, such as returning 404 errors or other internal errors. To this end, we perform rigorous filtering process (details in appendix II.I to ensure that the ultimate tool set of ToolBench reliable and functional. Finally, only retain 3, 451 high-quality tools (16, 464 APIs).
2.2 INSTRUCTION GENERATION
Different from prior works, specifically focus on two crucial aspects for instruction generation: (1) diversity: to train LLMs to handle a wide range of API usage scenarios, thereby boosting their generalizability and robustness; and (2) multi-tool usage: mirror real-world situations that often demand the interplay multiple tools, improving the practical applicability and flexibility of LLMs. To this end, instead of brainstorming instructions from scratch and then searching for relevant APIs, we sample different combinations of APIs and craft various instructions that involve them.
Generating Instructions for APIs Define the total API set as SAPI- each time, sample few APIs: 5%uh {API,, APIN) } from SAPI. We prompt ChatGPT to understand the functionalities of these APIs and then generate (1) possible instructions (Inst,) that involve APIs in Sib and (2) relevant APIs (Stel C ggh) for each instruction (Inst,), i.e.. [[TT] Insty], ... [Stel Insty []. where denotes the number of generated instances. These (instruction, relevant API) pairs will be used for
4

---

Published as a conference paper at ICLR 2024
Reasoning Chains Different Methods
Usen: want give my friend birthday surprie. Know har favorite actress tie Steiniald. Help me pleasel
ReACT
DFSDT [ours]
Instruction
Instruction
Selected
Thought: will first get vome information about Halle Steintels.
APt Names iotra, character detalls Argumentic I"name" "oilee Strinfeld""
Normal
Normal
Error
Observation:
("Spiden--tamm Spider ..
Ropidar
Error
Error
Normal
Normal
Obscrvation:
("message": : "server
Is not avaliante."
Error
+
Firish Final
Success
Arguments:
recently Spider Woman "Sprdor Marn Acrus Spider-Venaa You buy Sptoen-Wooman mast as gift, and she would probably like i.)
Normal
API Call with Normal Response
Succoss
Final Answer
Error
API Call with Error
Fail
Give Up
Name: Frich Slving Up
A comparison of our DFSDT and conventional CoT or ReACT during model reasoning (left). We show part of the solution path annotation process using ChatGPT (right).
Figure 4:
training the API retriever in $ 3.1 We use different sampling strategies (introduced later) to cover all APIs and most their combinations, thus ensuring the diversity of our instructions.
The prompt for ChatGPT is composed of (1) general description of the intended instruction genera- tion task, (2) comprehensive documentation of each API in 8 which helps ChatGPT understand their functionality and interplay, and (3) three in-context seed examples (seed,, seed2, seeda). Each seed example is an ideal instruction generation written by human experts. These seed examples are leveraged to better regulate ChatGPT's behavior through in-context learning. total, we wrote 12 36 diverse seed examples (Seeed) for the single-tool / multi-tool setting, and randomly sampled three examples at each time. Detailed prompts for instruction generation are described in appendix A.7 Overall, the generation process can be formulated as follows:
ChatGPT (API) .Ply) Ea (seeds on seody CBaad
([[S)) Insti]." ... [at Insty
| []PH .......... AAII, seed,, ................................ seedg).
Sampling Strategies for Different Scenarios As shown in Figure 3 for the single-tool instruc- tions (I1), we iterate over each tool and generate instructions for its APIs. However, for the multi-tool setting, since the interconnections among different tools RapidAPI are sparse, random sampling tool combinations from the whole tool set often leads to series of irrelevant tools that cannot be covered by a single instruction in a natural way. To address the sparsity issue, we leverage the RapidAPI hierarchy information. Since tools belonging to the same RapidAPI category collection are generally related to each other in the functionality and goals, we randomly select 2-5 tools from the same category / collection and sample most APIs from each tool generate the instruc- tions. We denote the generated instructions as intra-category multi-tool instructions (12) and intra-collection multi-tool instructions (I3), respectively. Through rigorous human evaluation, we find that instructions generated in this way already have a high diversity that covers various practical scenarios. We also provide visualization for instructions using Atlas (link) to support our claim.
After generating the initial set of instructions, we further filter those with the hallucinated relevant APIs by assessing whether they exist in sath Finally, we collect nearly 200k qualified (instruction, relevant API) pairs, including 87413, 84815, and 25251 instances for 11, 12, and I3, respectively.
2.3 SOLUTION PATH ANNOTATION
As shown in Figure given an instruction Inst,, we prompt ChatGPT to search for valid action sequence: {ag...... ax sation for ChatGPT. At each round t, the model generates an action a, based on previous interactions, i.e., ChatGPT(a.) (a " ,0t-1, rt-1) Inst,), where denotes the real API response. For each
Such multi-step decision-making process is cast as multi-round conver-
5

---

Published as a conference paper at ICLR 2024
des ChatGPT should specify its "thought", which API to use, and the specific parameters for this API, ie., at has the following format: "Thought: ..., API Name: .., Par ameters: .....
To leverage the function call feature of ChatGPT, treat each API special function and feed its API documentation into ChatGPT's function field. In this way. the model understands how to call the API. For each instruction Inst,, feed all the sampled APIs SN ChatGPT'ss as available functions. To let ChatGPT finish an action sequence, we define two additional functions, i.e., "Finish with Final Answer" and "Finish by Giving Up". The former function has a parameter that corresponds to a detailed final answer to the original instruction; while the latter function is designed for cases where the provided APIs cannot complete the original instruction after multiple API call attempts.
Depth First or ReACT Yao al. [022) has inherent limitations: (I) error propagation: mistaken action may propagate the errors further and cause the model to be trapped in faulty loop, such as continually calling an API in a wrong way or hallucinating APIs; (2) limited exploration: CoT or ReACT only explores one possible direction, leading to limited exploration of the whole action space. Hence even GPT-4 often fails to find valid solution path, making annotation difficult.
Decision Tree In our pilot studies, we find that CoT (Wei et al. 2023
To this end, we propose to construct decision tree to expand the search space and increase the possibility of finding a valid path. As depicted in Figure our DFSDT allows the model to assess different reasoning paths and choose to either (1) proceed along promising path (2) abandon an existing node by calling the "Finish by Giving Up" function and expand a new node. During node expansion, to diversify the child nodes and expand the search space, we prompt ChatGPT with the information of the previously generated nodes and explicitly encourage the model to generate a distinct node. For the searching process, we prefer depth-first search (DFS) instead of breadth-first search (BFS) because the annotation can be finished as long as one valid path is found. Using BFS will cost excessive OpenAI API calls. More details are described in appendix AA.8| We perform DFSDT for all the generated instructions and only retain those passed solution paths. Ultimately, we generate 126, 486 (instruction, solution path) pairs, which are used train TooILLaMA in $32
3 EXPERIMENTS
In this section, we investigate the performance of TooILLM framework. We first introduce the evaluation metric and evaluate the efficacy of API retriever and DFSDT in 3.1 Then we present the main experiments in $ 3.2 followed by generalization experiment in 3.3
3.1 PRELIMINARY EXPERIMENTS
ToolEval Considering the API's temporal variability on RapidAPI and the infinite potential solution paths for an instruction, is infeasible to annotate a fixed ground-truth solution path for each test instruction. Considering that human evaluation can be time- time-consuming,
follow AlpacaEval (Li) et al. 2023b) to develop an efficient evaluator ToolEval based on ChatGPT, which incorporates Two evaluation metrics (details in appendix A.5): (1) Pass Rate: it calculates the proportion of successfully completing an instruction within limited budgets. The metric measures the executability of instructions for an LLM and can be seen as basic requirement for ideal tool use; and (2) Win Rate: we provide an instruction and two solution paths to ChatGPT evaluator and obtain its preference (i.c., which one better). We dre-ne set of criteria for both metrics and these criteria are organized as prompts for our ChatGPT evaluator. We evaluate multiple times based on ChatGPT to improve the reliability. Then we calculate the average results from the evaluator.
Through rigorous testing (details in appendix A.S
we find that ToolEval demonstrates high agreement of 87.1% 1% pass rate and 80.3% in win rate with human annotators. This shows that ToolEval can reflect and represent human evaluation to large extent.
Efficacy of API Retriever The API retriever aims to retrieve relevant APIs to an instruction. We employ Sentence- BERT (Reimers & Gurevych] 2019) to train a dense retriever based on BERT- BASE (Devlin et al. 2019). The API retriever encodes the instruction and API document into two embeddings, and calculates their relevance with embedding similarity. For training, regard the relevant APIS of each instruction generated in $ 2.2 as positive examples and sample a few other APIs negative examples for contrastive learning. For baselines, we choose BM25 (Robertson et al.

---

Published as a conference paper at ICLR 2024
Method
I1 NDCG
12
I3
Average NDCG
NDCG
NDCG
Method
11
12
13
Average
@1
@5
@
@5
@
@
@1
@5
ReACT
37.8
40.6
27.6
35.3
BM25
18.4
19.7
12.0
11.0
25.2
20.4
18.5
17.0
ReACT@N
19.4
49.4
34.6
44.5
Ada
57.5
36.8
30.7
54.6
46.8
49.6
45.4
DFSDT
58.0
70.6
62.8
63.8
Ours
84.2
89.7
68.2
77.9
81.7
87.1
78.0
84,9
Table 3: Pass rate of different reasoning strategies for three types of instructions (II, 12, 13) based on CharGPT.
Table 2: Our API retriever v.s. two baselines for three types of instructions (I1, 12, 13). We report NDCG@ and NDCG@5.
2009) and OpenAI's text-embedding--ade-ee (link). We evaluate the retrieval performance using NDCG (Jrvelin & Kekäläinen (2002). We train and evaluate our model on single-tool instructions (II), intra-category multi-tool instructions (12), and intra-collection multi-tool instructions (I3).
As shown in Table our API retriever consistently outperforms baselines across all settings, indicating its feasibility in real-world scenarios with massive APIs. Also, the NDCG score of 11 is generally higher than 12 and I3, which means single-tool instruction retrieval simpler than multi-tool setting.
Superiority of DFSDT over ReACT Before solution path annotation, we validate the efficacy of DFSDT. Based on ChatGPT, we compare DFSDT and ReACT using the pass rate metric. Since DFSDT consumes more OpenAl API calls than ReACT, for fairer comparison, we also establish "ReACT@N" N" bascline, which conducts multiple times of ReACT until the total costs reach the same level of DFSDT. Once a valid solution is found by ReACT@N, N. we deem it pass.
From Table 3 it can be observed that DFSDT significantly outperforms the two baselines in all scenarios. Since we only retain those passed annotations the training data, given the same budgets, using DFSDT could annotate more instructions. This makes DFSDT a more efficient way that saves the total annotation cost. We also find that the performance improvement of DFSDT is more evident for harder instructions (i.e., 12 and 13) than those simpler instructions (II). This means that by expanding the search space, DFSDT can better solve those difficult, complex instructions that are unanswerable by the vanilla ReACT no matter how many times it is performed.
3.2 MAIN EXPERIMENTS
We fine-tune LLaMA-2 7B model (Touvron et al. 2023b) using the instruction- solution pairs. The original LLaMA-2 model has sequence length 4096, which is not enough under our setting since the API response can be very long. To this end, we use positional interpola- tion (Chen et al. 2023) to extend the context length to 8192 (training details in appendix (A.3).
LoMILLaMA
Settings Ideally, by scaling the number and diversity of instructions and unique tools in the training data, TooILLaMA is expected to generalize to new instructions and APIs unseen during training. This is meaningful since users define customized APIs and expect TooMLLaMA to adapt according to
the documentation. To this end, we strive to evaluate the generalization ability of ToolLLaMA at three levels: (1) Inst.: unseen instructions for the same set of tools in the training data, (2) Tool: unseen tools that belong to the same (seen) category of the tools in the training data, and (3) Cat.: unseen tools that belong a different (unseen) category of tools in the training data.
We perform experiments on three scenarios: single-tool instructions (II), intra-category multi-tool instructions (12), and intra-collection multi-tool instructions (I3). For I1, conduct the evaluation for the aforementioned three levels (II IIst,,, I1-Too,, and I1-Cat.); for 12, since the training instructions already involve different tools of the same category, we only perform level 1 and level for the generalization evaluation (I2-Inst. and 12-Cat.); similarly, we only perform level 1 generalization for (I3tInst.) since already covers instructions that involve various combinations of tools from different categories (the tools in RapidAPI collection may come from different RapidAPI categories). For each test instruction, we feed the ground- truth (oracle) APIS Saub to each model. This simulates the scenario where the user specifies the API set they prefer.
Baselines We choose two LLaMA variants that have been fine-tuned for general-purport dialogue, ie., Vicuna (Chiang et al. 2023) and Alpaca (Taori et al. 2023). We also choose the "teacher model" ChatGPT, Text- Text-Davinci-O0 003, GPT-4, and Claude-2 as baselines, and apply both DFSDT and ReACT to them. When calculating the win rate, each model is compared with
7

---

Published as a conference paper at ICLR 2024
Model
Method
II-Inst.
II-Yool
II-Cat.
12-Inst.
12-Cat.
13-Inst.
Average
Pass
Win
Pass
Win
Puss
Win
Pass
Win
Pass
Win
Pass
Win
Pass
Win
ChaGGT
ReACT
41.5
44.0
44.5
42.5
46.5
DESDT
54.5
60.5
65.0
62.0
60.5
57.3
75.0
72.0
71.5
64.8
62.0
69.0
64.8
64.3
Clande-2
ReACT
5.5
31.0
3.5
27.8
5.5
33.8
6.0
35.0
6.0
31.5
14.0
47.5
6.8
34.4
DESDT
20.5
38.0
31.0
44.3
18.5
43.3
17.0
36.8
20.5
33.5
28.0
65.0
22.6
43.5
Text-Davinci003
ReACT
12.0
28.5
20.0
35.3
20.0
31.0
8.5
29.8
11.5
29.8
21.0
15.0
16.5
33.2
DESDT
43.5
40.3
44.0
43.8
46.0
46.8
37.0
42.0
43.3
46.0
63.0
43.1
46.3
GPT4
ReACT
53.5
60.0
50.0
58.8
53.5
63.5
67.0
65.8
72.0
60.3
17.0
78.0
57.2
61.4
DESDT
00.0
67.5
71.5
67.8
67.0
66.5
79.5
73.3
77.5
03.3
71.0
84,0
71.1
70.4
Vicuna
ReACT & DESDT
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Alpaca
RcACT & DFSDT
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
RoACT
25.0
45.0
42.0
33.0
47.5
50.8
25,
55.0
47.0
TooILLaMA
DESDT
57.0
55.0
61.0
55.3
62.0
54.5
77.0
68.5
27.0
58.0
68.0
69.0
66.7
60.0
DFSDT-Retiewe
64.0
62.3
61.0
59.0
60.5
55.0
81.5
08.5
68.5
00.8
05.0
73.0
67.3
63.1
Table 4: Main experiments of ToolBench. Win rate is calculated by comparing cach model with ChatGPT- ReACT. A win rate higher than 50% means the model performs better than ChatGPT-RcACT. Apart from DolLLLMA-DFFDD-RRttiiee all methods use the oracle API retriever (i.e., ground truth API).
Main Results
The results are placed in Table from which we derive that:
Both Vicuna and Alpaca fail to pass any instruction (pass rate & win rate = 0), which means their instruction-followe abilities do not cover the tool-use domain. This underscores the deficiency of current instruction tuning attempts, which largely focus on language skills;
2. For all LLMs, using DFSDT significantly outperforms ReACT in both pass rate and win rate. Notably, ChatGPT +DFSDT surpasses GPT-4+ReACT in pass rate and performs comparably in win rate. This underscores the superiority of DFSDT over ReACT in decision-making
3. When using DFSIDT, LaMILLaMA performs much better than Text-Dainci-003: and Claude-2, and achieves result almost on par with ChatGPT (the teacher model). In general, despite generalizing unseen instructions and tools, TooILLaMA +DFSDT demonstrates competitive generalization performance in all scenarios, achieving a pass rate second to GPT4+DFSDT.
Overall, these results demonstrate that ToolBench can sufficiently elicit the tool-use capabilities within LLMs and empower them to skillfully master even unseen APIs for various instructions.
Integrating A API Retriever with TooILLaMA In real-world scenarios, asking users to manually recommend APIs from large pool may not be practical. To emulate this practical setting, we feed the top 5 APIs (instead of the ground truth APIs SXL) recommended by our API retriever to Too.LLaMA. As shown in Table 4 using retrieved APIs even improves the performance compared the ground truth API set. This is because many APIs in the ground truth API set can be replaced by other similar APIs with better functionalities, which our API retriever can successfully identify. In other words, our retriever expands the search space of relevant APIs and finds more appropriate ones for the current instruction. It demonstrates the excellent ability of our API retriever to retrieve relevant APIs, especially considering the vast pool (16, 000+) of APIs from which our API retriever selects
3.3 OUT-OF-DISTRIBUU
(OOD) GENERALIZATION TO APIBENCH (PATIL ET AL. 2023
Settings We further extend TooILLaMA to an OOD dataset APIBench to validate its generalization ability. We equip ToA with two retrievers: our trained API retriever and the oracle retriever. We evaluate three domains of APIBench, i.c., TorchHub, TensorHub, and HuggingFace. We compare TooILLaMA with Gorilla, a LLaMA-7B model fine-tuned using the training data of APIBench. Following the original paper, we adopt two settings for Gorilla: shot setting (ZS) and retrieval- aware setting (RS). The latter means (RS) the retrieved APIs are sent to the model as part of the prompts; while the former (7S) does not incorporate the APIs in the prompts when training the model. We adopt the official evaluation metric and report the AST accuracy and the hallucination rates.
The results are shown in Table In general, TooILLaMA achieves remarkable OOD generalization performance on all three datasets, despite being trained on a completely different API domain and instruction domain. Specifically, TooILLaMA+our API retriever outperforms Gorilla+BM25 from both training settings (ZS/ / RS) in terms of AST accuracy on HuggingFace and TorchHub. With the same oracle retriever, TooILLaMA is consistently superior when compared to Gorilla-SS ZS. It should be noted that Gorilla model cannot be generalized to our ToolBench dataset due to our more complex settings, such the multi-tool use and multi-step reasoning.
Results
8

---

Published as a conference paper at ICLR 2024
HuggingFace
TorchHub
TensorHub
Method
Hallu. (4) AST (t)
Hallu. (L)
AST (t)
Hallu. (4)
AST (1)
TooILLaMA Our Retriever
10.60
16.77
13.70
51.16
6.48
40,59
Gorilla-7S + BM25
46.90
10.51
17.20
44.62
20.58
34.31
Gorilla-SS + BM25
6.42
15.71
5.91
50.00
2.77
41.90
TooILLaMA + Oracle
8.66
88.80
14.12
85.88
7.44
88.62
Gorilla-ZS + Oracle
52.88
44.36
39.25
59.14
12,99
83.21
Gorilla-RS+ Oracle
6.97
89.27
6.99
93.01
2.04
94.16
Table 5: OOD generalization experiments on APIBench. For the Gorilla entries, ZS RS means that Gorilla was trained in zero-shot/ / retrieval-awart setting on APIBench. We report hallucination rate and AST accuracy.
4 RELATED WORK
Tool Learning
Recent studies have shed light on the burgeoning capabilities of LLMs in mastering
tools and making decisions within complex environments (Nakano et al. [02T]] Qin et al. 2223a
Shen et al., 2023] Wu et al. 2023 Schick et al., 2023 Hao et al. |023] Qian et al. [023] Song et al.
2023:/Zuang et al., 2023 Gao et al. (023). p. Gaining access to external tools endows LLMs with
time-time factual lnggledge (Yang et al.j 2023), multimodal functionalities (Gupta & Kembhavi
2023), and specialized skills in vertical domains (Jin et al., (023). However, open-source LLMs still
lag far behind SOTA LLMs in tool use, and how tool-use ability is acquired by SOTA LLMs remains unclear. In this paper, we aim to bridge this gap and fathom the underlying mechanism.
Instruction Tuning Instruction tuning enhances LLMs in understanding human instructions and generating proper responses (Wei et al. |02 Bach et al. 2022). Since manual annotation is time- consuming, self-instruct (Wang et al. 22022 proposes to generate high-quality data from SOTA LLMs, which facilitates recent trend of data curation for multi turt dialogue (Taori et al. 2023 Chiang
et al. 2023 Xu et al., 2023a Ding et al., 2023). Compared with the dialogue, tool learning is more challenging given the vast diversity of APIs and the complexity of multi-tool instructions. As result, even GPT-4 often fails to find a valid solution path. However, the existing tool-learning dataset cannot effectively address real human needs as mentioned in $ Instead, ToolBench is designed for practical scenarios and improves the previous pipeline for tool- data construction.
Prompting LLMs for Decision Making Prompting facilitates LLMs decompose high-level tasks into sub-tasks and generate grounded plans (Ahn et al. 2022 Huang et al. [022a b
Ye et al.) 2023). ReACT (Yao et al., 2022) integrates reasoning with acting by allowing LLMs to give a proper reason for an action and incorporating environmental feedback for reasoning. However, these studies do not incorporate a mechanism for decision retraction, which becomes problematic as an initial error can lead to cascade of subsequent errors. Recently, Reflexion (Shinn et al.) 2023) mitigates this issue by asking LLMS to reflect on previous failures. Our DFSDT extends Reflexion to a more general method by allowing LLMs to assess different reasoning paths and select the most promising one. In essence, DFSDT shares a similar idea to one concurrent work: tree-of-fe tre-oof-houught (ToT) reasoning (Yao) let al. 2023). However, DFSDT targets general decision-making problems where the decision space is infinite, compared to ToT's relatively simple tasks that can be addressed by brute-force search.
CONCLUSION
To elicit the tool-use capabilities within LLMs, we present ToolBench, covering 16k+ real-world APIs and various practical use-case scenarios including both single-tool and multi-tool tasks. Moreover, we propose DFSIDT to reinforce the planning and reasoning ability of L.MMs, enabling them to navigate through reasoning paths strategically. For efficient evaluation of tool learning, we devise an automatic evaluator ToolEval. By fine-tuning LLaMA on ToolBench, the obtained model TooILLaMA matches the performance of ChatGPT and exhibits remarkable generalization ability to unseen APIs. Besides, we develop a neural API retriever to recommend relevant APIS for each instruction. The retriever can be integrated with TooILLaMA demonstrate the generalization ability of our pipeline to out-of di-tiitribttion domains. In general, this work paves the way for future research in the intersection of instruction tuning and tool use for LLMss.
more automated tool-use pipeline. In the experiments,

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The contributions are listed as follows: (1) API collection: Shihao Liang, Sihan Zhao, Kunlun Zhu, Yujia Qin; (2) instruction generation: Lan Yan, Kunlun Zhu, Shihao Liang, Yujia Qin; (3) solution path annotation: Yining Ye, Shihao Liang, Runchu Tian, Yujia Qin, Xin Cong; (4) model implementation: Shihao Liang, Yujia Qin, Kunlun Zhu, Lauren Hong, Yifan Wu; (5) system demonstration: Xiangru Tang, Bill Qian. Yujia Qin led the project, designed the methodology and experiments, and wrote the paper. Yankai Lin, Mark Gerstein, Dahai Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou advised the project. Yankai Lin, Xin Cong, and Ruobing Xie proofread the whole paper. All authors participated in the discussion. Yujia Qin is sponsored by the Baidu Scholarship.
The authors would like to thank Yifan Wu, Sun, Zheni Zeng, Chen Zhang, Yu Gu, Chenfei Yuan, Junxi Yan, Shizuo Tian, Mingxi Yan, Jason Phang, Chen Qian, and Weize Chen for their valuable feedback, discussion, and participation this project.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. ArXiv preprint, abs/2204.01691, 2022.
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Févry, et al. Promptsource: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 93-104, 2022.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712. 2023.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.1553 15595, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph Gonzalez. lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL httpss -vicuaaa
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186 Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10. (0.18653/v1/N11-14233, URL https:/7
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.1 14233, 2023.
Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. Assistgpt: A general multi-modal assistant that plan, execute, inspect, and learn. arXiv preprint urXiv:2306.086649 2023.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953-14963 2023.
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. arXiv preprint arXiv:2305. /554, 2023.
10

---

Published as a conference paper at ICLR 2024
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of PMLR, 2022a.
Proceedings of Machine Learning Research, pp.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. ArXiv preprint, 0e0/22077.5660 2022b.
Kalervo Järvelin and Jaana Kekäläinen. Cumulated based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422-446, 2002.
Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu. Genegpt: Augmenting large language models with domain tools for improved access biomedical information. ArXiv, 2023.
Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A benchmark for tool-augmented llms. arXiv preprint 2tti::33000082244, 2023a.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-follovinng models. https: / //gi thub. www.c/o/ttttseeeeec hatp//aaaaaaaaaaall 2023b.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. Webgpt: Browser-assisted question-answering with human feedback. ArXiv preprint,
2021.
OpenAl. OpenAl: Introducing ChatGPT, 2022. URL http::ttp://// htt.://tes///ppee......o///aaaap
OpenAL. Gpt-4 technical report, 2023.
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph Gonzalez, Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305. 15334, 2023.
Cheng Qian, Chi Han, Yi R Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. Creator: Disentangling abstract and concrete reasonings of large language models through tool creation. arXiv preprint arXivv2305. 14318, 2023.
Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang. et al. Webepm: Interactive web search for chinese long-form question answering. arXiv preprint arXiv:2305.06633
2023a.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. arXiv preprins SEtY::2304.03354 2023b.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends0 in Information Retrieval, 3(4):333-38, 2009.
Timo Schick, Jane Dwivedi- Yu. Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. ArXiv preprint, abs/2302.0476h 2023.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface, 2023.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023.
Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang. Ye Tian, and Sujian Li. Restgpt: Connecting large language models with real-world applications via restful apis. arXiv preprint arXiv:2306.00663 2023.
11

---

Published as a conference paper at ICLR 2024
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: General- ized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing Ilama model. https://github: cow con/tattu--leae wtt.aaab/staarrreeeea alpaca 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXhv:2302.13977 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307. 230.002800.0988. 2023b.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10562 2022.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint 10900:2/, 2021.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Lc, and Denny Zhou. Chain-of-thoughh prompting elicits reasoning large language models, 2023.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Vi- sual chatgpt: Talking, drawing and editing with visual foundation models. ArXiv preprint, abs/2303.046711 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models follow complex instructions, 2023a.
Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open- open-source large language models. arXiv preprint 265554 2023b.
Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. Chatgpt is not enough: Enhancing large language models with knowledge graphs for fact-aware language modeling. arXiv preprint arXiv:2306. 11489, 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv preprint, abs/2210.03629) 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.
2023.
Yining Ye, Xin Cong, Yujia Qin, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Large language model as autonomous decision maker. arXiv preprint arXiv:2308. 12519, 2023.
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang Toolqa: A dataset for Umm question answering with external tools. arXiv preprint vrXiv22306.13044 2023.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A IMPLEMENTATION DETAILS
A.1 DETAILS FOR FILTERING RAPIDAPI
We perform a rigorous filtering process to ensure that the ultimate tool set of ToolBench is reliable and functional. The filtering process is as follows: (1) initial testing: begin by testing the basic functionality of each API to ascertain whether they are operational. We discard any APIs that do not meet this basic criterion; (2) example response evaluation: response. Then we evaluate their effectiveness by response time and quality. APIs that consistently exhibit a long response time are omitted. Also, we filter out the APIs with low-quality responses, such as HTML source codes or other error messages.
make API calls to obtain an example
A.2 API RESPONSE COMPRESSION
When examining the response returned by each API, we discover that some responses may contain redundant information and are too long be fed into LLMs. This may lead to problems due the limited context length of LLMs. Therefore, we perform a response compression to reduce the length of API responses while maintaining their critical information.
Since each API has a fixed response format, we use ChatGPT to analyze one response example and remove unimportant keys within the response to reduce its length. The prompt of ChatGPT contains the following information for each API: (1) tool documentation, which includes tool name, tool description, API name, API description, parameters, and an example API response. This gives ChatGPT a hint of the API's functionality; (2) 3 in-context learning examples, each containing an original API response and a compressed response schema written by experts. this way, we obtain the response compression strategies for all APIs. During inference, when the API response length exceeds 1024 tokens, we compress the response by removing unimportant information. If the compressed response is still longer than 1024, we only retain the first 1024 tokens. Through human evaluation, we find that this compression retains important information contained in the API response and successfully removes the noises.
A.3 DETAILS FOR TRAINING TOOILLAMA
We train the model in a multi-rount conversation mode. For the training data format, we keep the input and output the same those of ChatGPT. Since it is unclear how ChatGPT organizes the function call field, we just concatenate this information into the input as part of the prompt for LaMA For the training hyper parameters, we use a learning rate of x 10-5, a warmup ratio of 4 x 10-2, total batch size of 64, maximum sequence length of 8192, and use position interpolation ratio of 2. We train the model for two epochs and select the model checkpoint with the best performance on the development set and then evaluate it on the test set.
A.4 DETAILS FOR DFSDT
In practice, it is essential to balance effectiveness with costs (the number of OpenAl API calls). Classical DFS algorithms generate multiple child nodes at each step, then sort all the child nodes, and select the gighest-scoring node for expansion. After greedily expanding to the terminal node, DFS backtracks to explore nearby nodes, expanding the search space. Throughout the algorithm, the most resource-intensiv part is the sorting process of child nodes. If we use an LLM to evaluate two nodes at a time, requires approximately O( log n) complexity of OpenAT API calls, where the number of child nodes.
In fact, we find empirically that in most cases, the nodes ranked highest are often the node generated at first Therefore, we skip the sorting process of child nodes and choose a pre-order traversal (a variant for DFS) for the tree search. This design has the following advantages:
If the model does not retract an action (e.g., for the case of simple instructions), then DFSDT degrades to ReACT, which makes it efficient as ReACT.
13

---

Published as a conference paper at ICLR 2024
After the algorithm finishes, the nodes explored by this method almost the those found by a classical DFS search. Hence, can also handle complex instructions that only DFS can solve.
Overall, this design achieves a similar performance DFS while significantly reducing costs.
should also be noted that ReACT can be viewed degraded version of DFSDT. Therefore, although ToAALLaMA is trained on data created by DFSDT, the model can be used either through ReACT or DFSDT during inference.
A.5 DETAILS FOR TOOLEVAL
We adopt two metrics for automatic tool-use capability evaluation: pass rate and win rate.
Details for Pass Rate To assess whether a solution path completes the tasks outlined in the original instruction and successfully passes it, we need to first consider the solvability of the instruction. In principle, an instruction can be classified as either (1) solvable: for example, at least one of the provided tools is potentially helpful in solving the original instruction; or (2) unsolvable: for example. all APIs are irrelevant to the instruction or the instruction provides invalid information such as invalid email address.
To determine whether a solution path is deemed passed or not, we need to consider whether the instruction solvable or unsolvable. our evaluation, three types of labels can given to each solution path, i.e., Pass, Fail, and Unsure. Specifically, we define different rules as follows:
the instruction is solvable:
1. If the model gives finish type "Finish by Giving Up".
(a) After trying all the APIs extensively during and receiving no helpful information from APIs, the solution path is deemed Pass.
(b) If the model only calls a few API or receiving valid information from the APIs, the solution path is deemed Fail.
2. If the model gives finish type "Finish with Final Answer",
(a) the APIs provide no valid information, and the model has tried all the APIs to retrieve useful information, but the final answer still does not resolve the original instruction or conveys refusal (such as "I'm sorry, but can't provide you with this, because the tools are unavailable"), the solution path is deemed Pass.
(b) If the tools provide valid information, and the final answer does not completely resolve the instruction or is a refusal, the solution path is deemed F Fail.
(c) If the final answer completely resolves the original instruction, the solution path is deemed a Pass.
(d) If is unable to determine the instruction is resolved based on the content of the final answer, the solution path is deemed an Unsure.
the instruction is unsolvable:
1. If the model gives finish type "Finish with Final Answer",
(a) If the final answer resolves an instruction that was initially considered unresolvable, the solution path is deemed Pass.
(b) the final answer is refusal, the solution path deemed a Pass.
(c) the final answer is hallucinated by the model itself and provides a false positive response (such "I've completed the task, the final answer *"), the solution path is deemed a Fail.
2. If the model gives finish type "Finish by Giving Up". (a) Under this case, the solution path is deemed Pass.
For every solution path, we instruct the ChatGPT evaluator to generate multiple (2 4) predictions and perform majority vote derive the final pass rate.
14

---

Published as a conference paper at ICLR 2024
Details for Win Rate Since pass rate only measures whether instruction is completed not, instead of how well it is completed, we adopt another metric: win rate. It is measured by comparing two solution paths for given instruction. We assume that passed candidate is better than failed candidate and only compare those solution paths that are both "Pass", , or both "Fa i eed" annotated by the ChatGPT evaluator. Note that compared with another solution path, one solution path will be annotated with one of the following: win, 1 lose, or t tie. We build rules for the evaluator's 's behavior decide which solution path is better, and the criteria are listed as follows:
1. Information richness: whether the final answer contains all the necessary information to answer the original instruction. A significantly richer answer is better, while a similar level of richness that is sufficient to answer the question ties.
2. Factuality: whether it accurately describes what has been done, and what failed in the end. A more accurate description in the final answer is better.
3. Reasoning:
whether a detailed and accurate reason for failure is provided if the query remains unresolved. A more detailed reason is better.
4. Milestone:
calculating the number of milestones reached during execution.
5. Exploration: whether more potentially useful APIs were attempted during the execution process. The use of greater number of APIs is better.
6. Cost: Having fewer repeated (redundant) API calls is better if the number of APIs used is the same.
For every solution path, we also generate multiple (2 4) predictions and then perform a majority vote to derive the final win rate. In Table for ease of reading, we split the ratio of t tie into two pieces and add them win and lose, respectively. In Table we report the original numbers as reference.
Comparing Human Evaluation and ToolEval To validate the reliability of ChatGPT evalua- tor in both pass rate and win rate, we sample among four different methods (ChatGPT+ReACT. ChaIGPTDDFSDT. ToOLLLMMADDFSDT and GPT4+DFSDT) to obtain solution pairs for 300 in- structions for each method. Then we engage humans to annotate the pass rate for ChaGGPTDDSDDT, TooILLMMA+DEDDT and GPT4+DFSDT, and the win rate among ChatGPT+ReACT and Chat- GPT+DFSDT. Our ChatGPT evaluator demonstrates high agreement of 87.1 % in pass rate and 80.3% win rate with human annotators. This result shows that our evaluator generates highly similar evaluation results to humans and can be viewed credible evaluator who simulates human evaluation on pass rate and win rate.
It should also be noted that the evaluation for tool learning is far more intricate than traditional tasks such as dialogue. The reason that there may exist infinite "correct" solution paths for each instruction. In our initial investigations, we surprisingly found that even human experts often disagree with each other in deciding which solution path is better, leading to a relatively low agreement. For instance, one may prefer solution path that uses only a few APIs derive the final answer quickly; while another may prefer a solution path that extensively tries all the APIS to cross-validate specific information. In this regard, we believe there is still long way to go for a fair evaluation of the tool-use domain, and we believe this work has paved the way for it. We expect more future works to explore this interesting research problem.
A.6 DETAILS FOR EXPERIMENTS ON APIBENCH
When generalizing TooILLaMA to APIBench, no training updates were made to TooILLaMA, but instead of treating each API in the prompt function call. We define one function that represents selecting an API, providing the code for invoking it, and describing the generated output in natural language. We do not consider the zero-shot setting of APIBench where the prompts do not contain any API descriptions because the APIs from the three tested domains were never encountered during training.
15

---

Published as a conference paper at ICLR 2024
Model
Method
1I-Inst.
11-Tool
H1-Cat.
12-Inst.
12-Cat.
13-Inst.
Average
Win
Tie
Win
Tie
Win
Tie
Win
Tie
Win
The
Win
Tie
Win
Tie
CharGPT
DFSDT
52.5
16.0
14.0
19.5
67.0
10.0
58.5
61.0
56.8
14.7
Claude-2
ReACT
27.0
8.0
24.0
7.5
29.5
8.5
32.0
6.0
28.5
6.0
43.0
9.5
30.7
7.5
DFSDT
31.
8.0
41.0
6.5
39.5
7.5
32.5
9.5
33.5
0.0
65.0
0.0
40.8
5.3
1ext-Davinci-003
23.5
10.0
28.5
13.5
27.0
26.5
6.5
25.5
8.5
41.0
8.0
28.7
DISDT
35.0
10.5
37.5
12.5
40.0
13.5
36.5
8.0
40.0
6.5
60.0
6.0
41.5
9.5
GPT4
ReACT
52.5
15.0
10.5
15.0
50.5
12.5
52.5
76.0
4.0
58.3
12.1
DESDIT
60.5
14.0
62.5
10.5
58.0
17.0
67.0
12.5
57.0
12:5
80.0
8.0
64.2
12.4
Vicuna
(ReACT & DESDT)
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Alpaca
(RaACT DFSDT)
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
ReACT
40.0
10.0
36.5
11.0
42.0
11.0
46.5
10.5
37.6
8.5
51.0
8.0
-2.1
9.8
TooILLaMA
DFSDT
48.5
50.5
10.0
62.5
12.0
52.0
68.0
2.0
53.2
Retriever
58.0
8.5
51.5
9.0
51.0
8.0
61.5
8.0
56.0
9.5
71.0
4.0
19.2
7.8
Table Win rate results before merging the tic label. Win rate is calculated by comparing each model with ChaGGPT-ReACT. A win rate higher than 50% means the model performs better than ChatGPT-ReACTI Apart from TooILLaMA-DFSDDRReiiee all methods use the oracle API retriever (i.e., ground truth API).
A.7 PROMPTS FOR INSTRUCTION GENERATION
Below we list the detailed prompt for instruction generation, which consists of four parts: task description, in-context learning examples, sampled API list, and other requirements.
Task Description of Single-tool Instructions:
You will be provided with a tool, its description, all of the tool's available API functions. the descriptions of these API functions, and the parameters required for each API function. Your task involves creating 10 varied, innovative, and detailed user queries that employ multiple API functions of a tool. For instance, if the tool 'climate news' has three API calls "get.all.climan diimatt_change..... news", bookkuu-climate.uu
and cittoriccl_llimate your query should articulate something akin to: first, determine today's 's weather, then verify how often it rains in Ohio in September, and finally, find news about climate change to help me understand whether the climate will change anytime soon. This query exemplifies how to utilize all API calls of 'climate news'. A query that only uses one API call will not be accepted. Additionally, you must incorporate the input parameters required for each API call. To achieve this, generate random information for required parameters such as IP address, location, coordinates, etc. For instance, don't merely say 'an address', provide the exact road and district names. Don't It just mention 'a product', specifyy wearables, milk, a blue blanket, a pan, etc. Don't 't refer to 'my company', invent a company name instead. The first seven of the ten queries should be very specific. Each single query should combine all API call usages in different ways and include the necessary parameters. Note that you shouldn's 't ask 'which API use', rather, simply state your needs that can be addressed by these APIs. You should also avoid asking for the input parameters required by the API call, but instead directly provide the parameter in your query. The final three queries should be complex and lengthy, deseribing a complicated scenario where all the API calls can be utilized to provide assistance within a single query. You should first think about possible related API combinations, then give your query. Related apis are apis that can be used for a give query: those related apis have to strictly come from the provided api names. For each query, there should be multiple related for different queries, overlap of related apis should be as little as possible. Deliver your response in this format: [Queryl: api2, a.urr. ....... related_apis apii''(ppi,, api5, api6...J, ....... rrelatedapis aapis':(ppplpl api7, ap.9........
'related Iapil,
Task Description of Multi-tool Instructions:
You will be provided with several tools, tool descriptions, all of each tool's 's available API functions, the descriptions of these API functions, and the parameters required for each API function. Your task involves creating 10 varied, innovative, and detailed user queries that employ API functions of multiple tools. For instance, given three tools 'nba_news", 'cat-facts', and "hotels': 'nba_news' has API functions 'Get individual NBA source news' and 'Get all NBA news", 'cat-facts' has API functions 'Get all facts about cats' and 'Get random fact about cats', 'hotels' has API functions "properties/get-detae (Deprecated)), 'properties/list (Deprecated)) and "locations/3//sarre query should articulate something akin to: 'I want name my newborn cat after Kobe and host
Your
16

---

Published as a conference paper at ICLR 2024
party to celebrate its birth. Get me some cat facts and NBA news to gather inspirations for the cat name. Also, find a proper hotel around my house in Houston Downtown for the party.' This query exemplifies how to utilize API calls of all the given tools. A query that uses API calls of only one tool will not be accepted. Additionally, you must incorporate the input parameters required for each API call. To achieve this, generate random information for required parameters such IP address, location, coordinates, etc. For instance, don't It merely say "an address', provide the exact road and district names. Don't just mention 'a product', specify wearables, milk, a blue blanket, a pan, etc. Don't refer to 'my company', invent company name instead. The first seven of the ten queries should be very specific. Each single query should combine API calls of different tools in various ways and include the necessary parameters. Note that you shouldn't I ask 'which API to use', rather, simply state your needs that can be addressed by these APls. You should also avoid asking for the input parameters required by the API call, but instead directly provide the parameters in your query. The final three queries should be complex and lengthy, describing a complicated scenario where all the provided API calls can be utilized to provide assistance within single query. You should first think about possible related API combinations, then give your query. Related APIs are APIs that can be used for a given query; those related APIs have to strictly come from the provided API names. For each query, there should be multiple related APIs; for different queries, overlap of related APIs should be little as possible. Deliver your response in this format: [Query I: ....... 'relaed_apis : :[[ooo name, api name], [tool name, api name], [tool name, api namee]..1,Quurry22 ................................ related_apis :[[tool name, 'related_apis [[ooo] name,
api name], [tool name, api name]. [tool name, api name I.uur,, api name]. [tool name, api name], [tool name, api name]...]......
In-context Seed Examples. In the following, we show one single-tool instruction seed example and one multi-tool instruction seed example.
For example, with tool ASCII Art, the given api names are 'figlet', 'list figlet styles', 'cowsay', "iist.oosaay_stylee 'matheq'.
Some sample queries and related_apis would be:
"Query": "Need to create an ASCII art representation of a mathematical equation. The equation constants. Help me generate the ASCII art for this equation.
is 'y mx + c', where m and Also please generate an ASCII art representation of the text 'Newton's Second Law of Motion'.", "related apis": l'figlet', 'list figlet styles', 'matheq']
"Query": "Working on a research paper on cows and need to include ASCII art representations of various cows. Can you first retrieve available ASCII art styles for cows? Then, can you generate ASCII art for cows like the Jersey, Holstein, and Guernsey? Finally, I want the cow to say 'Moo!' in the ASCII art.". "related apis": I l'figlet, 'list figlet styles', 'cowsay', 'list cowsay styles'] "Query": "I'm writing a blog post on ASCII art and need to include some examples. Can you generate ASCH art for the following strings: 'ASCII, 'art', and 'gallery'? ? You can first retrieve available figlet styles and then generate ASCII art for the strings using the styles.", "related_apis": ['figlet', "list figlet styles']
"Query": "Greetings! I'm putting together quirky slideshow about our furry friends and need your help to sprinkle some ASCII art goodness. Could you kindly fetch me the catalog of ASCII art styles available for animals? Also, I'm particularly keen on featuring ASCII art for creatures like pandas, cows, elephants, and penguins. And if they could say something cute like 'Hello!' or 'Hugs!' in the ASCII art, that would be purr-fect!", "related apis" ['figlet', 'list figlet styles', 'cowsay', "ist_coossy_ssyless]
For example, with tool ['Entrepreneur Mindset Collection', 'Random Words', 'thedigitalnews- feederapi', 'Chemical Elements'], the given api.names are (tool 'Entrepreneur Mindset Collec- tion") " Randomm Quote in JSON format', (tool 'Random Words') Get multiple random words', (tool 'Random Words') Get a random word', (tool 'thedigiallewssfedderra'"' gettrrippigggtgnggggggg articles', (tool "thedigitalnewsseeeerap"" " Cricket Articles', (tool "thedigitalnewsfeeder api'''getting specific news articles', (tool thedigitalnewssfeedraapi "Getting News Articles', (tool thhedigialnnwwsfccdennp") ''ettting all news articles", (tool 'Chemical Elements") "''nentt""'Getttt All Chemical Elements'.
getting specific cricket
Some sample queries and related apis would be:
"Query": "For my best friend's 's surprise birthday party, I require inspiration for party games and decorations. Kindly suggest some random words that can serve as themes for the party. Furthermore, I'm interested in gathering news articles about the latest party trends to ensure a modern celebration.
17

---

Published as a conference paper at ICLR 2024
Also, would appreciate details about the local hotels in my area for accommodation options. Your assistance is greatly appreciated.", "related.apis" [[" R'Rnaoom Words', "Get multiple random words') J. tthedigitanewssfeeeerrrpp "Getting News Articles', J, ['heddigtalnowsffcceeraaa 'Getting all news articles"]]
"Query": "In the midst of organizing team-building event for my esteemed company, eagerly seek your valued input for invigorating actívities. Might kindly request a collection of random quotes that encapsulate the essence of teamwork and motivation? Additionally, articles that showcase triumphant team- building events, as they serve as a wellspring of inspiration.", "edlate__pps"" [[ E'Ettrrpreneur Mindset Collection', 'Random Quote in JSON format') J], ('thedigi- aulnesfeeddrrpp', 'Getting News Articles'] ]] "Query": "I need specific cricket articles that discuss health benefits of sports for my research paper on exercise. also want to know which chemical elements are associated with exercising, like increased iron (Fe) and its impact on bone marrow.", "related_apic _apis": : [[ thaddigtalneessfcccrrrpp "getting specific cricket articles"], J. ['Chemical Elements', "Get All Chemical Elements"] ]]
keen on exploring news
"Query": "I'm starting a new business venture and need to make a speech announcing the new dawn. Provide some quotes and words for me to start with. would like gather news articles about successful entrepreneurs for inspiration.", ". "related apiis: [['Entrepeeneur Mindset Collection', 'Random Quote in JSON format') J, ['Random Words', 'Get multiple random words'], ['thedigital- newsfeederapi', 'getting specific news articles']]
These are only examples to show you how write the query. Do not use APIs listed in the above examples, but rather, use the ones listed below in the INPUT.
Sampled API List (An example)
"tool "tool_descripte description"
"EntreAPI Faker is used to dynamically create mock, demo, test and sample data for your application",
"name": "EntreAPI Faker", "api_list":
"name": "Longitute",
"url": "tttss:/eettzeepptfaak.. raapiappppppiiii..ooo cttp://rescsc//a/////aaa longitude",
"description": "Generate random longitude.",
"method"; "GET",
" "required [], "optional "rrrinnnlpprrmmterrrs"
"name": "max",
"type": "NUMBER",
"description": "Maximum va luee for latitude.", "default"; ".
),
"name": "min",
"type": "NUMBER", "description": "Minimum value for latitude.", "default": ""
),
"name": "precision",
"type": "NUMBER",
"description": "Precision for latitude.", "default";
"EntreAPI Faker",
"category_namee" "Data"
18

---

Published as a conference paper at ICLR 2024
{
"name": "Boolean",
"url":
/boolean",
"description": "Randomly generate boolean value.", "method": "GET"
srequirred_parmmtterss" [],
"optiionl__parantterrs"" [],
"tool_name": "EntreAPI Faker", "category_name"" "Data"
"name": "Past", "url": p.t.rreeepppaa....r....aaaaaee
past", "description": "Randomly generate date value in the past.",
"method"; "GET",
"required parameters": [], parameters"; [
"name": "refDate", "type": "STRING", "description": "Starting reference date", "default"; ""
)
"name": "years", "type": "NUMBER",
"description": "Number years for range of dates.", "default": ""
l, "tool_name", "EntreAPI Faker", "category_name", "Data"
"name": "Image Url",
"url":
imageUr1", "description": "Randomly generate an image URL.", "method"; "GET" parameters": [],
"name": "width",
"type": "NUMBER",
"description": "Width of the image. Default is 640.", "default"; ""
"name": "height",
"type": "NUMBER",
"description": "Height the image. Default is 480.", "default"; ""
19

---

Published as a conference paper at ICLR 2024
"name": "useRandomize",
"type": "BOOLEAN", " "description": "Add random number parameter to the returned URL.", "default": ""
"name": "category",
"type": "STRING",
"description": "The category for the image. Can be one: abstract, animal, avatar, business, cats, city, fashion, food, nature, nightlife, people, sports, technics, transport", "default": "
l.
"tool : "EntreAPI Faker", "category_namc" "Data"
"name": "Sentence", "url": pestps://entrreap--faeeee htw.eerrp.rappippi.coo/oooeeeeee sentence",
"description": "Randomly generate sentence of Lorem
Ipsum.",
"method": "GET"
"requí red_parameters"": [],
"
"name": "wordCount",
"type": "NUMBER",
"description": "Number of words in the sentence.". "default"; ""
l,
"tool_namer "EntreAPI Faker", "category_nant name": "Data"
{
"name": "Gender",
"url": hattps://entreapi-Eaaee paaprrepppp.akeeeppp.ppppipp..oo www.pppp....m/aaaaaaaaappppppape gender",
"description": "Randomly select gender.",
"method"; "GET"
"requireed_parrmeter: [],
"name": "useBinazy",
"type": "BOOLEAN",
"description"; "Use binary genders only.", "default"; ""
}, "EntreAPI Faker",
"category_name" "Data"
20

---

Published as a conference paper at ICLR 2024
{
"name": "Prefix",
"url":
prefix", "description"; "Randomly generate prefix (e.g., Mr., Mrs., etc.)",
"method"; "GET", "reeuired__paametersss"" [], Latriionnl_aaaamteersss [
"name": "gender",
"type": "STRING",
"description": "Optional gender.", "default"; ""
l,
"tool_name": "EntreAPI Faker",
"category_nani : "Data"
"name": "Array Element",
"url":
com/ random/
arrayElement",
"description": "Randomly select array element." .", "method"; "GET",
[], "aptional_parametti [
"name": "array", "type": "ARRAY",
"description": "The list of elements to choose from, Default [Vaa"", \"b\", \"e"")., "default";
}, "tool_name "EntreAPI Faker", "category_name"" "Data"
"name": "Number Value",
"url": "https://entreapi wwwereappp-aaeeeeeeeeee number", "description"; "Randomly generate number value.",
http::///etttrepttta........pppp www.appiprppiiipip.......... http:/dap..co//aaaaaaaaaaaa com/
"method"; "GET",
"required paramettrrr::: [],
"name": "min".
"type": "NUMBER", "description": "Minimum value.", "default"; ""
},
"name": "max",
"type": "NUMBER", "description": "Maximum value.". ", "default";
21

---

Published as a conference paper at ICLR 2024
"name": "precision",
"type": "NUMBER",
"descriptien "Precision of the number.", " "default"; "
1. "category_name", "Data"
"EntreAPI Faker",
"name": "URL",
"url": http://ttraaiiaaaeer..apaaaapapp
/uri", "description": "Randomly generate a URL.",
"method": "GET"
"requirred_aarameteer" [],
"optional__arannters"" [],
"tool_ne _ "EntreAPI Faker", saregorry_amee""" "Data"
Other Requirements:
Please produce ten queries in line with the given requirements and inputs. These ten queries should display diverse range of sentence structures: some queries should be in the form of imperative sentences, others declarative, and yet others interrogative. Equally, they should encompass a variety of tones, with some being polite, others straightforward. Ensure they vary length and contain wide range of subjects: myself, my friends, family, and company. Aim to include a number of engaging queries as long as they relate to API calls. Keep in mind that for each query, invoking just one API won't suffice; each query should call upon two to five APIs. However, try to avoid explicitly specifying which API to employ in the query. Each query should consist of minimum of thirty words.
A.8 PROMPTS FOR SOLUTION PATH ANNOTATION
We use the following prompt when searching for the solution path. When expanding the child nodes, use diversity Sucplyy-usepppppe prompt, showing the information of previous child nodes.
pystem_prroppe
You are Tool-GPT, capable of utilizing numerous tools and functions to complete the given task. 1.First, I will provide you with the task description, and your task will commence.
A.At each step, you need analyze the current status and determine the next course of action by executing function call.
3. Following the call, you will receive the result, transiticning you to new state. Subsequently, you will analyze your current status, make decisions about the next steps, and repeat this process.
4. Afteer several iterations of thought and function calls, you will ultimately complete the task and provide your final answer.
Remember:
1.The state changes are irreversible, and you cannot return to a previous state.
22

---

Published as a conference paper at ICLR 2024
Keep your thoughts concise, limiting them to maximum of five sentences.
3. You can make muitiple artempts. If you plan to try different conditions continuously, perform one condition per try. 4.If you believe you have gathered enough information, call the function "Finish: give_answer" to provide your answer for the task.
5.If you feel unable to handle the task from this step, call the function "Finish: give_ap_an_rrestattt." Let's Begin!
Task description: (task_description)
diversity_user_prome
This is not the first time you try this task, all previous trails failed.
Before you generate your thought for this state, I will first show you your previous actions for this state, and then you must generate actions that is different from all of them. Here are some previous actions candidates: [pzevious_candidatte]
Remember you are now in the intermediate state of a trail, you will first analyze the now state and previous action candidates, then make actions that is different from all the previous.
Finish httpsa__ffunttioodddrrittttooo
"name": "Finish",
"description": "If you believe that you have obtained result that can answer the task, please call this function to provide the final answer. Alternatively, if you recognize that you are unable to proceed with the task in the current state, call this function to restart. Remember: you must ALMAYS call this function at the end of your attempt, and the only part that will be shown to the user is the final answer, so it should contain sufficient information.",
"parameters";
"type": "object",
"properties":
creturn_type"!
"type": "string", "enum": ("give_answer", "give_uppand_resttrr"""
"iinal_asswer! "type": "string",
"description": "The final answer you want to give the user. You should have this field if \ return_typel Viggiaaaaee=siiivveeaaaasvee
"required"; ("return_type"",
23


Reviews:
[{"review_id": "wInYZBZ1HZ", "reviewer": "Reviewer_dyqS", "summary": "This paper introduces ToolLLM, a whole pipeline for creating and evaluating instruction-tuned language models that can use tools.\n\nThe authors created ToolBench, an instruction-tuning dataset by i) collecting a large number of real-world APIs from multiple categories; ii) creating instructions with seed demonstrations and tool sets; iii) annotate the api call solution paths with ChatGPT.\n\nThey fine-tuned several baselines on ToolBench and evaluated them with the proposed ToolEval pipeline, which also uses ChatGPT to evaluate whether the solution path is successful and whether the solution path is better than the one annotated by ChatGPT.\n\nAdditionally, they also propose to use an API retriever and a tree-search algorithm (DFSDT) during inference.", "questions": "How were ToolBench split into tuning and evaluation subsets? Could you explain more about this part?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The scale of ToolBench is unprecedented compared to previous tool learning datasets. It has more APIs, more tools, more task instances. This makes ToolBench much closer to real-world settings of tool-augmented language models.\n2. The authors conducted extensive experiments to show that their instruction-tuning pipeline enables language models to generalize to new tools and new domains without seeing them in the fine-tuning dataset.", "weaknesses": "### 1. The current annotation/evaluation pipeline is not rigorous and may lead to false impressions about model’s performance.\n\nToolBench is entirely annotated by ChatGPT. I understand that the annotations of the instruction-tuning data don’t have to be perfect, but I think as a benchmark for evaluation, it should be held to higher standards.\n\nToolEval solely relies on ChatGPT to evaluate whether a solution path “passes” and whether it “wins” the solution path from ChatGPT.\n\nWhile I appreciate the authors’ efforts in comparing ChatGPT’s evaluation with human subjects’, I think the current evaluation pipeline is problematic because of the following reasons:\n\n- Many tool-related tasks can have a definite set of correct answers or correct solution paths. For these tasks, the real correct answers should be used to judge the correctness of generated answers. Relying on ChatGPT’s annotation of win rates and pass rates can lead to over-confidence about wrong answers that look satisfactory.\n- ChatGPT (and other language models) as an evaluator is known to have order bias (gives preference to an option based on their order), egocentric bias (prefers its own outputs), length bias [1], selection bias [2]. It also chooses style over substance [3]. I would expect more discussion on if and how ToolEval mitigates these biases.\n- The evaluation rules for pass rates and win rates seem too complex even for human annotators to follow, which severely undermines my confidence in your human evaluation results. According to Appendix A.5, the rules to determine whether a solution path gets a “pass” form a 3-layer decision tree with as many as 10 leaves and each decision in the tree requires some non-trivial and thorough examination of the instructions, the available APIs and the solution path. I would really love to learn more about your human evaluation process. For example, did the annotators only submit the final flag of pass/fail/unsure or did they also submit all the decisions they made to get to the final results? Is there any evidence that can show that human annotators were faithfully following your rules instead of relying on human cognitive biases? If the human annotations themselves were not reliable, nor would the correlation between human and ChatGPT be good enough.\n- Win rates are computed by comparing model generation with annotations from ChatGPT+ReAct. Could inference algorithms that are more similar to the annotation pipeline (instead of better) lead to better results?\n\n**Reference**\n\n[1] Koo, Ryan, et al. \"Benchmarking Cognitive Biases in Large Language Models as Evaluators.\" *arXiv preprint arXiv:2309.17012* (2023).\n\n[2] Zheng, Chujie, et al. \"On Large Language Models' Selection Bias in Multi-Choice Questions.\" *arXiv preprint arXiv:2309.03882* (2023).\n\n[3] Wu, Minghao, and Alham Fikri Aji. \"Style over substance: Evaluation biases for large language models.\" *arXiv preprint arXiv:2307.03025* (2023).\n\n### 2. The evaluation results are constantly changing over time, making it very expensive and difficult to compare new methods and older baselines. This also ruins the reproducibility of the evaluation pipeline.\n\nI appreciate the authors’ consideration about the temporal variability on RapidAI.\n\n> *Considering the API’s temporal variability on RapidAPI and the infinite potential solution*\n*paths for an instruction, it is infeasible to annotate a fixed ground-truth solution path for each test instruction. Moreover, when comparing different models, it is crucial to ensure they employ the same version of APIs during evaluation.*\n> \n\nI think this is also part of the reason why they used ChatGPT as an evaluator instead of using ground truth annotation.\n\nHowever, I still think it’s problematic.\n\nFirst, people are not able to know roughly how good a method is by looking at the reported pass rates and win rates, because the reported results can only be compared with the results evaluated during the same time period.\n\nSecond, it makes evaluation much more difficult. As the authors pointed out, each evaluation run needs to use the same version of APIs to ensure fair comparison. This means every new method that needs evaluation on ToolEval must also run every baseline they compare to in a very short period of time. If some highly variable API were used (for example, APIs that query the availability of restaurants or realtime weather), this period could be as short as a few hours. This creates an extremely heavy burden for developers and researchers, because not only do they need to run all evaluation experiments multiple times, they also have to run them in parallel.\n\nTherefore, at the very least, I would expect some qualitative analysis of the tool set that can point out how many task instances involve these temporally variable tools and how much temporal variability impacts the evaluation results over time.\n\nI think a better solution to this problem might be creating a “snapshot” of the API call results at a certain time and release this snapshot with the evaluation suite. This way, it’s much easier to get reproducible results.\n\n### 3. The comparison between DFSDT and ReAct.\n\nWhy isn’t DFSDT always better than ReAct? According to your description, it seems that ReAct is a special case of DFSDT where the branching factor is 1. Therefore, I expected DFSDT to be better than ReAct in most cases. In other words, the win rate for ChatGPT+DFSDT against ChatGPT+ReAct should be close to 100%. According to the first two rows in Table 4, that is clearly not the case. Is there some explanation on why that happened?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u2EWfymWK8", "reviewer": "Reviewer_k5uq", "summary": "The paper collected a new instruction-tuning dataset (called ToolBench) for improving LLM's tool-use capability. To construct ToolBench, the author collected more than 16k REST APIs from RapidAPI and constructed synthesized instructions based on these APIs. To automatically obtain the desired behavior of the LLM, the author prompted prompt gpt-3.5-turbo-16k. By finetuning LLaMA on ToolBench, the author showed that ToolLLaMA achieves higher score according to a ChatGPT evaluator than Text-davinci-003 and Claude-2 in ToolBench.", "questions": "How does the models in Table 4 compare with each other if we adopt human evaluation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is valuable to those who are trying to build LLMs that can use tools. ToolBench is a well-engineered dataset and is shown to be more diverse than prior datasets in Table 1.", "weaknesses": "1. There are lots of papers that studied how to instruction-tune an LLM for tool-use, like GPT4Tools (in NeurIPS 2023), Gorilla, ToolAlpaca, etc. The techniques adopted in ToolLLM, such as constructing instruction-tuning samples by prompting ChatGPT, is very standard and has limited novelty. The paper is mainly about applying a popular synthetic instruction generation pipeline to distill knowledge from gpt-3.5 and has almost no research value.\n\n2. The author proposed \"Depth First Search-based Decision Tree\" (DFSDT) that seems to outperform ReAct prompting. However, the technique is closely related to self-consistency + CoT and Tree-of-thoughts. It is also unclear how DFSDT will perform for other types of LLMs not in the category of LLaMA / GPT / Claude.\n\n3. The evaluation metrics are based on ChatGPT, which is highly unreliable and may favor models tuned with ChatGPT-prompted datasets. The author mentioned in Appendix A.5 (Paragraph \"Comparing Human Evaluation and ToolEval\") that `Our ChatGPT evaluator demonstrates a high agreement of 87.1% in pass rate and 80.3% in win rate with human annotators. `. This shows that there is **close to 20% disagreement** between human evaluation and ChatGPT evaluation. This is a large discrepancy and should not be ignored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4tFKOGkjW9", "reviewer": "Reviewer_2o14", "summary": "- The paper introduces ToolLLM, a framework to facilitate tool use capabilities in open-source large language models (LLMs). It includes data construction, model training, and evaluation components.\n- A new instruction tuning dataset called ToolBench is constructed using ChatGPT. It contains over 16,000 real-world APIs from RapidAPI spanning 49 categories. The dataset covers both single-tool and multi-tool instructions.\n- An automatic evaluator ToolEval is developed to assess tool use capabilities. It incorporates pass rate to measure executability and win rate to compare solution quality.\n- By fine-tuning LLaMA on ToolBench, ToolLLaMA is obtained. Experiments show it matches ChatGPT's performance and generalizes well to unseen APIs. An API retriever is also trained to automatically recommend relevant APIs. ToolLLaMA demonstrates strong generalization on the out-of-distribution APIBench dataset, despite not being trained on it. This validates its capabilities on new domains.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Leveraging existing techniques in the literature, authors build a framework for developing models capable of tool use. This framework encompasses dataset building, model training, and model evaluation. The scope is comprehensive, and the execution is generally solid. \n\nThe authors do a good job documenting and elaborating design decisions such as dataset filtering and issues with prompting with a limited context window. \n\nThe authors make their artifacts (ToolBench, ToolEval, and model artifacts) publicly available so others can build on their work.", "weaknesses": "Most of the technical ideas in this work are from past works, with the exception of DFSDT, which is a simple application of DFS to prompting. In this sense, the current work's technical contribution is low.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zmyjMUNntp", "reviewer": "Reviewer_u7K9", "summary": "Tool utilization is an important capability of LLMs to extend their task scope. Although closed-source LLMs have achieved powerful performance by calling external tools, it is still important for open-source LLMs to enhance their capability in tool use. In this paper, the authors proposed ToolLLM to facilitate the large language model to master 16,000+ tools. Moreover, the contributions of this paper can be summarized in three parts: 1) ToolBench, a benchmark which generate instructions involving different tool utilization; 2) a depth-first search-based decision tree algorithm is introduced to enhance the capability of LLMs in tool utilization; 3) an evaluation platform called TaskEval. Experimental results demonstrate the effectiveness of the proposed method.", "questions": "1. In the design of solution path annotation, authors prefer to use DFS instead of BFS. However, DFS could also possibly backtrack and search multiple answers if it does not obtain a valid path. So, any experiments to prove the cost of DFS to find a valid path when compared with BFS?\n2. This paper mainly uses RapidAPIs for training. So is the fine-tuned ToolLLaMa suitable or adapted for APIs in other sources/formats (e.g., Pytorch/Tensorflow/HuggingFace in Gorilla, ChatGPT plugins)?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Although tool utilization has received much attention in LLM applications, building a standard benchmark is still challenging. To address this, this paper releases a high-quality instruction tuning dataset, called ToolBench, which covers many different tools.\n2. Based on the constructed ToolBench, this paper also designs a strategy for solution path annotation, which uses a depth-first search-based decision tree to search for a possible valid path.\n3. Experimental results validate that tuning LLMs with generated samples can effectively improve performance. \n4. The writing of this paper is good and easy to follow.", "weaknesses": "1. This paper introduces TaskEval, which encompasses two metrics, *Pass Rate* and *Win Rate*. Specifically, *Pass Rate* detects whether LLM can successfully execute user instructions, and *Win Rate* is designed to judge which solution path is better for a given instruction. However, to some degree, I think these metrics still have some deficiencies and cannot efficiently the capability of LLMs in tool utilization. For example, *Pass Rate* can only reflect LLM whether can execute user instruction. Sometimes, powerful LLMs (e.g., ChatGPT) can always generate answers for any user instructions without any tool use, and besides, the hallucination of LLMs can also let it generate some executed but counterfactual answers. Besides, *Win Rate* can only reflect the capability of different LLMs, not their ability of tool utilization. Even two LLMs without tool utilization (e.g., ChatGPT v.s. LLaMA-7b), will also demonstrate differences in performance. Therefore, I think the proposed metrics can only reflect the capability of LLMs but not their tool-use ability. Of course, I also admit that it is challenging to build such a metric since there have been no metrics in this area before.\n2. I appreciate that the paper releases a high-quality dataset in this area. However, it will be better to provide a detailed statistical of this dataset (e.g., distribution in domain, error analysis).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)