API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Authors: Minghao Li
Yingxiu Zhao
Bowen Yu
Feifan Song
Hangyu Li
Haiyang Yu
Zhoujun Li
Fei Huang
Yongbin Li
Venue: EMNLP 2023 Main (2023)
Keywords: Tool-Augmented LLMs
Large Language Model
Benchmark
TL;DR: This paper presents API-Bank, the first benchmark tailored for Tool-Augmented LLMs.

Abstract: Recent research has demonstrated that Large Language Models (LLMs) can enhance their capabilities by utilizing external tools. However, three pivotal questions remain unanswered: (1) How effective are current LLMs in utilizing tools? (2) How can we enhance LLMs' ability to utilize tools? (3) What obstacles need to be overcome to leverage tools? To address these questions, we introduce API-Bank, a groundbreaking benchmark, specifically designed for tool-augmented LLMs.  For the first question, we develop a runnable evaluation system consisting of 73 API tools. We annotate 314 tool-use dialogues with 753 API calls to assess the existing LLMs' capabilities in planning, retrieving, and calling APIs. For the second question, we construct a comprehensive training set containing 1,888 tool-use dialogues from 2,138 APIs spanning 1,000 distinct domains.  Using this dataset, we train Lynx, a tool-augmented LLM initialized from Alpaca. Experimental results demonstrate that GPT-3.5 exhibits improved tool utilization compared to GPT-3, while GPT-4 excels in planning. However, there is still significant potential for further improvement. Moreover, Lynx surpasses Alpaca's tool utilization performance by more than 26 pts and approaches the effectiveness of GPT-3.5. Through error analysis, we highlight the key challenges for future research in this field to answer the third question.

API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Minghao Li Yingxiu Zhao"; Bowen Yu'; Feifan Song Hangyu Li', Haiyang Yu', Zhoujun Li", Fei Huang', Yongbin Li' Allibaba Group, 'Hong Kong University of Science and Technology, Peking University, "Shenzhen Intelligent Strong Technology Co., Lid, (1mh397008, yubowen. ybw hangyu. {yifei yhy, f. f.huang shuide. Mbuidee.lybJeelibbb--ii com songffestu. pku. edu. www.accecccccccc....mmmm ust. hk 1izhoujun@aistrong.co com
com
Abstract
models exhibit increasingly human-like capabil- ities, such as powerful conversation, in-context learning, and code generation across a wide range of open-domain tasks (Bubeck et al., 2023).
Recent research has demonstrated that Large Language Models (LLMs) can enhance their ca- pabilities by utilizing external tools. However, three pivotal questions remain unanswered: (1) How effective are current LLMs in utilizing tools? (2) How can we enhance LLMs' ability to utilize tools? (3) What obstacles need be overcome leverage tools? To address these questions, we introduce API-Bank, ground- breaking benchmark, specifically designed for tool-augmented LLMs. For the first question, we develop a runnable evaluation system con- sisting of 73 API tools. We annotate 314 tool-use dialogues with 753 API calls to as- sess the existing LLMs' capabilities in plan- ning, retrieving, and calling APIs. For the sec- ond question, we construct a comprehensive training set containing 1,888 tool-use dialogues from 2,138 APIs spanning 1,000 distinct do- mains. Using this dataset, we train Lynx, ouglaaugeetted LLM initialized from Alpaca. Experimental results demonstrate that GPT-3.5 exhibits improved tool utilization compared to GPT-3, while GPT-4 excels plamning. How- ever, there is still significant potential for fur- ther improvement. Moreover, Lynx surpasses Alpaca's 's tool utilization performance by more than 26 pts and approaches the effectiveness of GPT-3.5. Through error amalysis, we highlight the key challenges for future research in this field to answer the third question". !
Despite their utility, LLMs are still constrained by training data (Brown et al., 2020; Zhao et al., 2023; Song et al., 2023). This data can quickly be- come outdated and may not cover all scenarios (Mi- alon et al., 2023). Consequently, there has been surge of research focused on enhancing LLMs by enabling them to leverage external API tools, such as accessing up-to-date information (Izacard et al., 2022) and interacting with third-party ser- vices (Liang et al., 2023). Traditionally. tool usage has been viewed as uniquely human behavior, with its emergence considered significant mile- stone in primate evolution (Ambrose, 2001). Draw- ing analogy to the timeline of human evolution, we argue that it is currently imperative address three pivotal questions: (I) How effective are cur- rent LLMs at utilizing tools? (2) How can we enhance LLMs' ability to utilize tools? (3) What obstacles still need to be overcome for LLMs to effectively leverage tools?
To tackle these inquiries, we present API-Bank, groundbreaking benchmark specifically designed for tool-augmented LLMs. In order to determine the users' needs regarding the utilization of tool- augmented LLMs, we initially conducted inter- views with 500 users. Taking into account their feedback, we establish the design principles for API-Bank. The evaluation scope of API-Bank must encompass three essential capabilities: planning, retrieving. and calling API tools, in order to ful- fill the users' requirements. Additionally, while constructing the benchmark, it is imperative to con- sider various aspects such domain diversity, API diversity, API authenticity, and evaluation authen- ticity. Subsequently, to answer the first research question, we implement the evaluation system of API- Pri-Bank, adhering to design principles. The sys-
1 Introduction
Over the past several years, significant progress has been made in the development of large language models (LLMs), including GPT-3 (Brown et al., 2020), Codex (Chen et al., 2021), ChatGPT, and impressive GPT-4 (Bubeck et al., 2023). These
Equal Contributions Corresponding author.
'The data and code are publicly available at ConvAT/
https://github.com
tree/main/api- -bank

---

tem comprises 73 commonly used APIs, along with 314 tool-use dialogues containing 753 API calls, all manually annotated, thus forming the first exe- cutable and authentic system capable of evaluating the effectiveness of LLM utilizing tools.
500 users who expressed interest in incorporating additional tools into LLM, we collected their re- quirements. Based on this foundation, we provide, for the first time, a definition to measure the abili- ties of tool-augmented LLM and the data standards for training and evaluating its abilities. We believe that these design principles, which reflect the ac- tual needs of users, can assist researchers in the field and contribute the future development of -ud-aneeeeted LLM benchmarks.
For the second question, we develop a com- prehensive tool-augmented LLM training dataset within API-Bank. This dataset comprises 2,138 distinct APIs and includes 1,888 dialogues with a total of 4, 4,149 API calls, reflecting three differ- ent levels of API usage abilities. However, an- notating thousands of APIS and the correspond- ing dialogues, adhering to our design principles, is not only prohibitively expensive but also time- consuming. In this work, we introduce Multi-agent, a novel method using LLMs instead of humans to automatically mass-produce oool-augeeted LLM training data. Multi-agent consists of five collab- orative agents that step-by-step generate domains, APIS, user queries, API calls & responses, while also ensuring the quality of the generated data aligns with our design principles. Multi-agent re- markably reduces the annotation cost by 98% com- pared human annotation. To verify the effective- ness of our training set, subsequently, we fine-tune a popular LLM Alpaca-7B (Taori et al., 2023), re- sulting in our own tool-augmented LLM Lynx.
2.1 Ability Grading
An ideal lool-augmented LLM should enable users to define the APIs they require in a private API Pool and request the LLM to invoke these APIs at the appropriate times to fulfill their needs. Based on our interviews with users, we have identified two dimensions that encompass the requirements for tool augmenedd LLM:
Few vs. Many APIs in Pool: Different users may define varying numbers of APIs. When users specify a small number of APIs (e.g., 2-3), the relevant information of all the spec- ified APIs (such as name, definition, input parameters, and output parameters) and the user's query can be provided as input to the LLM, allowing it to determine which APT to call to meet the user's needs. However, when users have defined a large number of APIS (e.g., hundreds), it is not feasible to input all the APIs the LLM due to input length strictions. In such cases, the LLM needs retrieve potentially suitable APIs fulfill the user's needs before calling them.
We conduct extensive experiments on API-Bank and obtained insightful results regarding the API tool usage capabilities of LLMs. Our findings re- veal that even smaller models like Alpaca-7B and ChatGLM-6.2B possess certain level of API call accuracy, about 20%. However, their proficiency in API retrieval and planning is negligible. In contrast, the larger-scale GPT-3 Davinc exhibits almost no capability in API usage, indicating that API usage might not be inherent feature of LLMs. Our Lynx, demonstrates an average improvement of 24% in the three API usage capabilities compared to Alpaca-7B. While approaches the performance of GPT-3.5, there remains a 21% gap from GPT-4. Additionally, we present a comprehensive exper- imental analysis that sheds light on the primary challenges encountered by GPT-4 and Lynx in uti- lizing APIs, addressing the third research question.
Single vs. Several API calls per Turn: Many user requirements are complex and necessi- tate the combination of multiple APT calls to achieve. Some users are willing to patiently decompose requirements into multiple queries and guide the LLM gradually to fulfill their needs, with each query involving single API call. But some users prefer to directly present complex requirements, expecting the LLM to autonomously perform step-by-step API calls within a single turn to meet their needs.
2 Design Principles of API-Bank
As shown in Figure 2, these two dimensions give rise four expected conditions: Few APIs, Single Call; Few APIs, Multiple Calls; Many APIs, Single Call; and Many APIs, Multiple Calls. During im- plementation, we discovered that the difficulty level
Due to the absence of authoritative ability defini- tions and benchmarks about tool-augmented L.M,, we conducted an extensive questionnaire survey in the initial phase. Through interviews with over

---

API 2ool
API Search
API
Retrieve
Call
Call
Query
Response
Call
Plan
Retrieve
Retrieve
Figure 1: The abilities to Call, Retrieve+Call, Call, and Plan=RRtrieve++all APIs of tool-augmented LLMs.
any errors during the call (such as incorrect API names, input parameter enrors, incorrect API call sequencing) can result in unfulfilled user require- ments. Therefore, for the benchmark construction, we need consider the following criteria:
Faw APTs in Pool
Many APTs in Pool
single
Few APIs, Single Call
Many APIs, Single Call Cetrieve=Call
API
Call
Call
Molti
Few APIs, Multi Calls
Many APIs, Multi Calls Plan-Retrieve-Call
The training and testing data should cover a wide range of domains as comprehensively as possible;
Domain diversity:
API
Call
calls
Figure Requirements of tool augmeneedd LLMs.
2. API authenticity:
The name, definition, in- put and output parameters of the APIs should closely resemble those in the real world;
the first two conditions is similar. Since planning the call of multiple APIs is straightforward when all the APIs are input, thus we merged the first two conditions. The three remaining conditions assess the following abilities:
3. API diversity: The benchmark should include a wide variety of API types and purposes.
1. Call: the ability to call APIs based on the given query when the APIs are known;
4. Evaluation authenticity: The evaluation
should incorporate functional system that enables real-time interaction with the LLM. The LLM offers an API call, which the system executes and subsequently returns the results to the LLM. The evaluation based on the impact of the execution on the system, assess- ing whether the LLM adequately responds to the user's requirements.
2. Retrieval+ Call: the ability to retrieve and call single API when the APIs are unknown;
3. Plan+ RetrieRetieal+CCCC. Call: the ability to continu- ously plan, retrieve, and call multiple APIs when the APIs are unknown.
We also present visual examples of each ability in Figure and Appendix.
Evaluation System of API-Bank
Based on the aforementioned principles, we first introduce how to construct the evaluation system of API-Bank, including the system implementation, the data annotation, and the evaluation metrics.
2.2 Data Standards
After grading the abilities to be assessed in the benchmark, another crucial aspect is ensuring the quality of the data. Calling APIs occurs within an open environment where we cannot predeter- mine the domains and functionalities. Morcover, API call is rigorous similar to mathematics, where
3.1 System Implementation
We have implemented 73 APIs in our system, in- cluding commonly used daily APIs such as weather

---

forecast, and accessing other Al models like Text- to-Image Generation. All APIs were implemented within the same framework by senior research and development engineers, with a total time in- vestment of 98 person-days. For APIs related to database operations, we establish the requisite databases and initialize them with initial entries, pivotal step in constructing dialogues. For APIs that access external information (e.g., search en- gines), we must ensure the retrieved information remains constant to ensure reproducibility. We track all queries for each API in test dialogues and record the retrieval results at a specific time point, hard-coding them within the API to maintain result consistency.
complex user requirement and decompose it into multiple simple queries, each of which could be fulfilled by executing a single API. To achieve this, we initially obtaine a set of APIs from the Pool, ranging from to 5, and ask annotators to deter- mine if they could collectively address a complex requirement. If so, they divide it into several simple queries. For each query, annotators label the API to be called by the LLM and provided the input parameters. They also label the response that the LLM should generate based on the system's 's output. Call is sim-
The annotation for ilar to that of Retrieval+ Call, with the difference being that annotators don't decompose the complex query into simpler ones. They are required to anno- tate sequential chain of API calls and the response derived from the execution of the last API.
Among them, we developed special APT called "API Search" to fulfill the evaluation requirements of both Retrieval+ Call and Plan+ Retrieeeriiaa+ Call abilities. Specifically, in these two scenarios, the LLM is unaware of the APIs available in the APT Pool in advance, so it needs to make use of the API Search to identify the potentially needed APIs according to the user query. In the input given the LLM, we provide the instructions of the API Search at the beginning, and an API Search is required before every other API call. When performing an API Search, the model should condense the user's demand into few keywords. The API Search obtains sentence embeddings from both the query keywords and all API meta information in the API Pool. It calculates the cosine similarity between keywords and all API embeddings, returns the meta information of the API with the highest similarity.
The introduction of APIs increases the difficulty of annotation. Despite recruiting computer science students for dialogue annotation, each dialogue re- quires discussion between two annotators to decide how to annotate. Additionally, two additional anno- tators are involved to ensure the quality of annota- tions, including the format, logical consistency, and reasonability of API calls, The average annotation cost per dialogue is $8. Out of the 400 dialogues annotated, we discard 21.5% due to various annota- tion issues. Ultimately, we retained 3 314 dialogues with a total of 753 API Calls.
3.3 Evaluation Metries
We evaluate model performance from two perspec- tives: the correctness of API calls and the quality of LLM responses. For the API call evaluation, we employ the Accuracy metric, which is calcu- lated as the number of correct predictions divided by the total number of predictions. In each evalua- tion, we start by initializing the evaluation system, ensuring that the databases for each API contain de- fault values. Then, compare the predicted API calls with the manually annotated API calls to de- termine their consistency. We define consistency whether the same database queries or modifications are performed and whether the returned results are the same. Regarding the evaluation of responses after API calls, we utilize the ROUGE-L metric.
3.2 Dialogue Annotation
Based on the grading abilities defined by de- sign principles, we annotate evaluation data for the abilities of Call, Retrieval+Call, and PlantRetrieval+Call Call APIs.
For the Call ability, we firstly randomly sample APIs from the API Pool. Then we instruct annota- tors to first imagine query that could be resolved by these APIs based on the API document. They then annotate the APT call and have the system execute it. Finally, they label the response based on the execution output. Please note that the Call data does not necessarily represent single-turn dia- logues. We also require annotators to ask multiple queries regarding the same set of APIs, providing both the dialogue history and the API call history. For the Retrieval+ + Call ability, we aim to obtain
Training Set of API-Bank
We hope API-Bank not only evaluate the effective- ness of cxisting LLMs in utilizing tools but also to enhance their performance in utilizing such tools.

---

The most direct approach achieve this goal is creating high-quulity training datasct tailored to tool-augmented LLMs. However, it is challenging to construct large-scale training dataset at a low cost while meeting the our design principles of do- main diversity and API authenticity. The manual annotation cost for each dialogue in our evaluation set reaches $8, making it expensive to create large dataset. Furthermore, it is challenging for the anno- tators to design diverse and authentic API pool. Our recruited annotators could only come up with 100 APIs, which cannot satisfy our need for di- verse training sets. Therefore, we propose a multi- agent data generation method to rapidly and cost- effectively construct tool-augmented LLM training data that aligns with the design principles.
number of errors still persist. Our investigation reveals that the errors stems from providing LLM with an overwhelming number requirements all at once, making it difficult to comprehend them effectively. Consequently, an intuitive approach emerges: can we alleviate this issue by decompos- ing the requirements into multiple simpler tasks, allowing LLM to exccute task at time?
We commence by providing overview of the elements that should be incorporated in the data based on our design principles: domain, API, query. ability, and API call&response. The domain deter- mines the functionality of the APIs, while the APIs and ability determine the types of queries they can handle. The combination of domain, APIs, query, and ability determines how the LLM makes API calls and generates an appropriate response.
To simulate the dependencies among these ele- ments, we propose the utilization of five agents as shown in Figure 3: (1) The first agent generates several domains, such as healthcare and fitness. (2) The second agent, considering the domain, gener- ates potential APIs. It is worth noting that during this stage, to ensure the authenticity of the simu- lated APIs, we add examples from Public APIs into the agent input. (3) The third agent randomly selects one or more APIs from the simulated APIs. Additionally, it chooses an ability outlined in our design principles. This information then used to create a query that matches the chosen ability and can be fulfilled by calling the selected APIs. (4) The fourth agent takes the domain, API, abil- ity, and query input. It is expected make the necessary API calls, simulate the execution of the APIs, and generate response that addresses the query. (5) Finally, we introduce the fifth agent, act- ing as a tester. This agent automatically verifies whether the generated data adheres to our design principles (it actually discard 35% instances). All five agents are implemented by providing specific prompts ChatGPT. Together, they collaborate in step-by-step manner accomplish the complex data generation. Multi-agent eliminates the need for human labelers, costing only S0.1 per dialogue and saving 98% compared to manual annotation.
Agent
Bequizenente
Qaezy
Resporse
Call
Generator
Teater
Figure Five agents collaborate to accomplish training data generation of tool-augmented L.MMs.
The emergence of LLMs has brought about a paradigm shift in data annotation, offering an al- ternative to human annotators for the automated generation of labeled data. One representative ap- proach is self-instruct (Wang et al., 2022). How- ever, we have observed that while self-instruct is effective in generating data for simple tasks such as writing, it encounters difficulties in generating tool- augmented L.LM data that aligns with our design principles. Specifically, we formulated complex instruction encompassing requirements for domain diversity, APT diversity&&auuhenniicyy and three spe- cific abilities about planning, retrieving and call- ing APIs. Nevertheless, the widely used ChatGPT struggles to generate data that fully adheres to our instructions, only 5% of the data is available. Even upgraded to the more powerful GPT-4, the available rate improves to 25%, but a significant
Benchmark Analysis
Statisties. In the end, we construct a bench- mark consisting of 1,008 domains, 2,211 APIs, 2,202 dialogues, and 6,135 turns. Among them,
hhips//gihhab.commpppiiaaapppppp

---

Statisties
Dialogue Type
Evaluation Type
Tool Usuge Ability
Benchmark
domains
APIs
Multi-mm
Multi-call
API Call
Response
Call
Retrieve
Plan
DATESET (Schick 2023)
1
x
x
x
x
x
APIBench (Patil et al., 2023)
90
1,645
x
x
x
V
x
ToolAlpaca (Tang al., 2023)
50
426
x
ToolBenchl (Qin et al., 2023b)
49
16.464
ToolBench2 (Xu al., 2023)
252
x
ToolQA (Zhuang ct al., 2023)
6
13
X
x
x
API-Bank (ours)
1,000
2,138
Table I: Comparison among API-Bank and existing benchmarks, where API-Bank comprehensively includes the most domains and APIs, covers both multi-tun turn and multi-call dialogues, evaluate both API call and response, and thoroughly consider three distinct tool usage abilities.
Statisties
Training
Evaluation
we discover that 78% of it does not adhere to our design principles as intended. It shows that the tester agent can process data well according to the given principle.
of Domains
1,000
# of APIs
2,138
73
# of Dialogues
1,888
314
Turns
5,221
914
- # of single call
3,147
363
Comparison. We have conducted comparative analysis of API-Bank in relation to recently re- leased benchmarks, as presented Table 1. To en- sure the comprehensiveness of this table, we have also incorporated some contemporaneous studies. Leveraging the design principles derived from our investigation of 500 user requirements, Bank stands out among all benchmarks currently available for its: (1) Highest diversity: encom- passing wide range of domains and APIs. (2)
of multiple calls
493
122
of Call
720
214
of Retrieve+Call
719
50
# of Plan=Retrieve+Call
449
50
avg. turns per dialogue
2.76
2.91
API-
Table 2: Statistics of API-Bank.
there are 934 dialogues in the Call category, 769 in the Retrieve+Call category, and 499 in the Reatieeervve+alll category. Each dialogue has 2.76 turns in the training set and 2.91 turns in the testing set. The training data is generated auto- matically by the LLM, while the evaluation data is manually annotated. Therefore, variations exist among them in terms of domain, API scope, and dialogue content. We aim to assess the generaliza- tion of models trained on our benchmark by testing their performance on distribution-shif evaluation set. Please refer to Table 2 for detailed statistical data. We also provide samples and running demo of our benchmark in the Appendix.
Highest realism: simulating multi-turn dialogues that occur in real-world scenarios, including cases where multiple APIS are called in each turn. (3) Highest coverage: incorporating API call and re- sponse annotations, taking into account the es- sential capabilities required for tool-augmented LLMs, such plamning, retrieving, and calling APIs. Therefore, we firmly believe that API-Bank represents the most comprehensive benchmark for tool-augmented LLM available at present.
Related Work
Quality. Each instance in the evaluation has been reviewed by four annotators, allowing us to primarily assess the quality of the training bench- mark generated by Multi-agent. We randomly se- lect 100 data samples from the training set and have the annotators evaluate their quality. The results demonstrate that the available rate of the Multi- agent generated data is 94%, which is an 89% im- provement compared to data generated solely by a single agent (self-instruct), indicating that our training set possesses a high level of quality and reliability. Furthermore, upon examining the data that was automatically filtered by the tester agent,
Recent research in language modeling has explored the use of external tools supplement the knowl- edge stored the model's weights (Qin et al.. 2023a). This approach allows for tasks such as exact computation or information retrieval to be offloaded to external modules such as a Python in- terpreter or search engine (Mialon et al., 2023). These tools can include other neural networks even the language model itself. Socratic Mod- els (Zeng et al.. 2022) is modular framework that allows for the composition of different pre-trained models on various modalities. Alternatively, natu- ral language knowledge can be retrieved from exter-

---

Call
Retrieve+Call
Plan-Retrieve+Call
Total
Ability
LLM
Correctness
Rouge
Correctness
Rouge
Correctness
Rouge
Correctness
Rouge
Alpaca-7B
24.06%
0.0204
5.19%
0.0019
0.00%
0.086
15.19%
0.0318
ChatGLM-6B
23.62%
0.2451
13.33%
0.2173
0.00%
0.1522
16.42%
0.2191
7ero-shot
GPT-3 Davinci
0.50%
0.35
1.48%
0.091
0.00%
0.0156
0.57%
0.0814
GPT-3.5-tubo
59.40%
0.4598
38.52%
0.3758
22.00%
0,3809
47.16%
0.4267
GPT-4
63.66%
0,3691
37.04%
0.351
70.00%
0.4808
60.24%
0,3910
Fine-tuning
Lynx-7B
49.87%
0.4332
30.37%
0.2503
20.00%
0.3425
39.58%
0.3794
Table 3: Main results of different L.LMs the API-Bank evaluation system.
nal sources, as demonstrated by WebGPT (Nakano et al., 2021) and ReAct (Yao et al., 2022) through the use of search APIs. Other approaches, such as Toolformer (Schick al., 2023), ART (Paranjape al., 2023), ToolkenGPT (Hao al., 2023) and TRICE (Qiao et al., 2023) leverage combination of search APIs, guestion-answeringg APIs, machine translation APIs, calculators, and other tools to solve various NLP tasks. ChatGPT Plugins and TaskMatrix.Al (Liang et al., 2023) demonstrate the potential for language models to integrate with thousands to millions of APIs. LATM((ai et al., 2023) and CREATOR (Qian et al., 2023) leverage GPT-4 to make API tools. Despite the promising demonstrations of these approaches, researchers have limited knowledge regarding three key issues: (I) How effectively can current LLMs utilize tools? (2) How can we enhance LLMs' ability to utilize tools? (3) What obstacles still need to be overcome for LLMs to effectively leverage tools? In this pa- per, we introduce API-Bank, the first benchmark specifically designed for vol-augmented LLMs, to address these three questions. As analyzed in Ta- ble API-Bank is also the most diverse, realistic, and comprehensive tool-augmented LLM bench- mark currently available.
LLMs in conjunction with tools.
7.1 Baselines
We evaluate the following models for analy- sis: GPT-3 Davinci (Brown et al., 2020), the first powerful variant of the GPT-3 family of models. GPT-3.5-turbo with the gpt-3.5-turbo-06. point; GPT-4, utilizing the gpt-4-0613 checkpoint; ChatGLM-6B (Du et al., 2022). bilingual chatbot with 6B parameters; Alpaca-7B (Taori et al., 2023), an instruction-tuned variant of LLaMA-7B, lever- aging 52K instruction-followinn data. Notably, Al- paca is currently recognized as the most widely used open-source LLM.
check-
7.2 Main Results
We present the experimental results of currently public LLMs and our trained Lynx model on the AP1- Pal-Bank evaluation system, shown in Table 3. As expected, the performance of each model grad- ually decreases with the increasing required ability. Given the API description, calling the APT can be seen as a slot filling task. Both the 7B Alpaca and 6B ChatGLM achieve about 20% accuracy in API call, indicating that basic language models possess some tool-utilization ability. Surprisingly, GPT-3 Davinci, despite being well-known 175B LLM, exhibits extremely low correctness in this task. We speculate that this is because API call requires strong understanding of instructions, which can only be unlocked through instruction tuning, a step omitted in GPT-3's training process.
7 Experiments
We proceed to fine-tune Lynx, model based on LLaMA-7B (Touvron et al., 2023), using our API- Bank training dataset. The fine-tuning process con- sists of three epochs, with a batch size of 256 and learning rate of 2c-5. Subsequently, we conduct a series of experiments on our API-Bame API-Bank evaluation system, wherein we benchmark our model Lynx against other LLMs. The prompts used for eval- uation are given in the Appendix. Through these experiments, aimed to identify the remaining challenges that hinder the effective utilization of
On the other hand, the instruction-tuned GPT- 3.5 demonstrates outstanding performance in this aspect, surpassing Alpaca-7B by 35 points in API call correctness and 0.44 in the response Rouge-L score. However, the effectiveness of GPT-3.5 de- creases by 21% when compared simple API calls in the Retrieve+ Call setting, and further 17% decrease is observed in the Retrieeetiee++CCau setting. This is because knowing which API to
https://oggaai.ommmbogggaaagpppp

---

call and planning how use the API both require certain reasoning abilities beyond instruction com- prehension. GPT-4, currently known as the most powerful LLM, shows a points improvement in API calls compared to GPT-3.5. Its performance in Retrieve+ Call is similar to GPT-3.5, but it achieves nearly 50% improvement in the most difficult Rearivetrieve+ Call setting. We hypothesize that this is due to GPT-4's emphasis reasoning and planning abilities, similar its performance in tasks involving mathematical reasoning.
Error Type
Rate
API Hallucination
61.38%
Has Exception
16.40%
Invalid Input Parameters
8.47%
False API Call Format
6.88%
No API Call
5.29%
Miss Input Parameters 1.59%
Table 5: Distribution of errors in the Lynx evaluation.
An exciting discovery is that the Lynx model, trained our API-Bank with Alpaca-7B initializa- tion, outperforms Alpaca-7B by 26 points in API Call correctness and 0.41 in the response Rouge-L score, approaching the effectiveness of GPT-3.5. This strongly indicates the satisfactory quality of training data generated through our Multi-agent automated construction strategy.
Error Type
Rate
Failed API Retrieval 67.86%
False API Call Format
17.86%
Invalid Input Parameters 7.14%
Miss Input Parameters 7.14%
Table 6: Distribution errors GPT-4 evaluation.
Here, we would like to emphasize the differences between developed API-Bank and the concur- rently released APIBench(Pati et al., 2023) and ToolAlpaca(Tang al., 2023) based on experimen- tal results. GPT-3.5 achieves an API usage accu- racy of 80% to 90% on their datasets. However, there is still significant room for improvement in our benchmark. This is because our evaluation set is manually constructed, considering design princi- ples and closely resembling real-world scenarios. In contrast, their evaluation set is generated through self-instruct by the LLM and lacks diversity due to its narrow domain focus.
duction in the Lynx model following fine-tuning with the API-Bank training dataset. This observed improvement suggests that disparities between the patterns of API calls in Alpaca's training dataset, which is constructed using 52,000 instruction data, and the actual API calls may contribute to this prob- lem. It is plausible that the original Alpaca model encounters challenges in comprehending the API calls instructions in the evaluation system.
Another noteworthy issue we encounter is the presence of "False API Call Format." Given that our testing prompts format, the model has to rely solely on guidance about making API calls in instructions. This liance on instruction-based guidance may cause difficulties for the model learning the correct API call format. The application of fine-tuning re- sults in a substantial enhancement in this regard, underscoring the significant benefit of finc-tuning in improving the model's 's capacity to generate ac- curately formatted API calls.
presented in zero-shot
7.3 Error Analysis
Error Type
Rate
No API Call
36.77%
API Hallucination
15.93%
Invalid Input Parameters
7.96%
False API Call Format
23.65%
Miss Input Parameters 1.17%
The analysis of the Lynx model results given in Table 5, where the most significant type of error is the API name mismatch, accounting for 61% of the total errors. This occurs when the model makes API calls that do not match the annotated API calls in Ground Truth. In some cases, Lynx generates fake APIs that unrelated the user's intent, even if they are not provided in the test prompt. This issue arises due to Hallucination in Lynx, where it incorrectly calls APIs that it has encountered during training.
Table Distribution of errors in the Alpaca evaluation.
In this study, we systematically categorize six primary error types, with detailed definitions pro- vided in the Appendix. Notably, the primary issue identified during our evaluation of the original Al- paca model, as depicted in Table 4, the frequent occurrence of "No API Call." However, it is worth mentioning that this issue exhibits a significant re-

---

Number of training samples
Accuracy (Call)
Rouge (Call)
ToolAlpaca Lynx (Ours) 6,184
10,366
53.88
39.75
54.64
39.80
Table 7: Results of the fine-tuned Alpaca on the ToolAlpaca dataset and Lynx.
The second major category of errors is related problematic input parameters, which can lead three types of errors: triggering an Exception within the API, being identified as an invalid input parameter, and generating an API call that fails to be parsed. These errors collectively contribute to 32% of the overall errors. Representative examples of problematic input parameters include passing placcholder a value, using illegal format (such not following the given date format, resulting in parsing failure), missing input parameters, and misunderstanding the requirements of the input parameter (c.g., requesting a stock code but passing in a company name). In addition, a few other issues exist, such as forging user or Al utterances instead of making an API call.
We eagerly anticipate investigating this possibility.
7.4 Dataset Comparison
To assess the quality of the dataset created, we con- ducted experiments to fine-tune Alpaca using the training dataset of ToolAlpaca. ToolAlpaca auto- matically creates a tool-use corpus, which contains 3938 tool-use instances from more than 400 real- world tool APIs spanning 50 distinct categories. We convert the training set of ToolAlpaca into the training format consistent with ours, and finally ob- tained 10366 training samples. Since ToolAlpaca does not involve API Retrieval in its process, our evaluation focused only on its ability to Call APIs, ensuring fairesss in the comparative analysis.
The results, presented in Table 7, reveal that our model Lynx, fine-tuned on the API-Bank dataset, outperforms the fine-tuned Alpaca on the ToolAl- paca dataset, even with fewer training data. This observation underscores the superior data quality of our constructed dataset and validates the efficacy of the multi-agent data synthesis methods.
As shown in Table 6, the primary issue with GPT- 4 is its inability to effectively use APT search for retrieving specific APIS, which accounts for 68% of the overall problems encountered. This problem does not occur in the evaluation of the fine-tuned Lynx model. Fine-tuning appears to facilitate teach- ing the model the Retrieve+ +Cal pipeline, unlike In-Context Learning, which makes challenging to control the model's API calls as expected. The sec- ond common problem with GPT-4 is the API calls it gives cannot be parsed by the evaluation system. This issue arises because GPT-4 sometimes makes multiple simultaneous API calls, which violates the tested prompt's instructions.
Conclusion
This paper presents API-Bank, the pioneering benchmark for tool-augmented LLMs comprising 2,202 dialogues involving 2,211 APIs from 1,008 domains, and provides three key findings: (1) Cur- rent LLMSs demonstrate some ability to use APIs, but there still room for improvement in pro- viding reliable services to users consistently. (2) Building diverse and high-quality training data is promising approuch to enhance LLMs' ability to use tools. (3) Proper API calls and strict adher- ence to API documentation pose significant chal- lenges for LLMs in utilizing API tools. Overall, the API-Bank benchmark provides a groundbreaking resource for evaluating and advancing the state-of- the-art coll-uugmetedd LLMs. We believe that this research will inspire future studies and pave the way for developing more advanced Al systems that can intelligently integrate external resources to fulfill human requirements.
Therefore, aim to highlight three potential directions worth exploring: (I) Improved API call- ing methods: The direct generation of API calls, similar to Toolformer (Schick et al., 2023). lim- ited by the number of available APIs Introducing additional API retrieval tools is challenging due to issues such as hallucination and difficulties in accu- rately calling retrieval. (2) Enhanced APT decoding algorithms: It is essential to demand strict adher- ence to the input parameter definitions when gener- ating API calls with LLMs. (3) Larger-scale train- ing data: While Lynx has demonstrated promising API usage capabilities, we wonder if scaling up the training data further could yield even better results.

---

9 Limitations
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint WnXiv2207.03337.
API-Bank is a pioneering benchmark for tool- augmented LLM. While we have followed com- prehensive design principles, it does have few limitations. Firstly, API-Bank focuses solely on English, and we plan address data construction and model evaluation for other languages future work. Secondly, we have only fine-tuned Lynx-7B on API-Bank and have not explored larger-scale models, although Lynx-7B performs comparably to GPT-3.5. Lastly, we have trained a commercially viable tool auo-euegeetee LLM based on a larger LLM within our company. However, due to anonymity reasons, we are unable to report and analyze the results of the online model. We aim to include this information in future versions of our work.
Zhengxiao Du Yujie Qian, Xiao Liu, Ming Ding. Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregres- sive blank infilling. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320-335.
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2023. Toolkengpt: Augmenting frozen lan- guage models with massive tools via tool embeddings. arXiv preprial arXiv:2305. 1154..
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.03299
10 Ethical Statement
This paper constructs a new benchmark for tool- augmented LLM, and discuss some related eth- ical considerations here. Firstly, during interviews regarding their needs for tool-augmented LLMs, participants are informed in advance that their feed- back will be used for product development and po- tentially published in the form research paper, without disclosing any personal private informa- tion. Secondly, all API tools in our dataset are original implementations and do not infringe upon any existing commercial software. Lastly, for data annotation in the benchmark, we hire four annota- tors. The hourly pay is set to 15 US$ per person, higher than the local statutory minimum wage.
Yaobo Liang, Chenfei Wu. Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji Shaoguang Mao, et al. 2023. Taskmatrix. ai: Com- pleting tasks by connecting foundation models with millions of apis. arXiv preprint arXiv:2303.16434
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et 2023. Augmented language models: a survey. arXiv preprint arXiv:2302.07%A.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: assistr-dssisted question- answering with human feedback. arXiv preprint arXiv:2112.09332
References
Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannanch Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multi- step reasoning and tool-use for large language mod- els. arXiv preprint arXiv:2303. 3000009
Stanley H. Ambrose. 2001. Paleolithic technology and human evolution. Science, 291(5509):1748 1753.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1
Shishir G Patil, Tianjun Zhang, Xin Wang, and
Joseph E Gonzalez. 2023, Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305. 15334.
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712
Cheng Qian, Chi Han, Yi Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. 2023. Creator: Disentan- gling abstract and concrete reasonings of large lan- guage models through tool creation. arXiv preprint 2rXiv:2305.1433
Shuofei Qiao, Honghao Gui, Huajun Chen, and Ningyu Zhang. 2023. Making language models better tool learners with execution feedback. arXiv preprint arXiv:2305.1300
Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. 2023. Large language models as tool makers. arXiv preprini arXiv:2305.1 177.1

---

Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen. Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang. Chaojun Xiao, Chi Han, et al. 2023a. Tool learning with foundation models. arXiv preprint arXiv:2304.08354
Yingxiu Zhao, Bowen Yu, Binyuan Hui. Haiyang Yu. Fei Huang, Yongbin Li, and Nevin L Zhang. 2023. A preliminary study of the intrinsic relationship be- tween complexity and alignment. arXiv preprint .rXiv:2308.06666.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang. Bill Qian, al. 2023b. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprini
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. Toolqa: A dataset for I1m ques- tion answering with external tools. arXiv preprint wrXiv2230613304
Timo Schick, Jane Dwivedi- Yu,, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom, 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint 202.v2230..
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang. Yongbin Li, and Houfeng Wang. 2023. Pref- erence ranking optimization for human alignment. arXiv preprint arXiv:2306. 202223666.7722..
Qiaoyu Tang, Ziliang Deng. Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. 2023. Toolalpaca: Gener- alized tool learning for language models with 3000 simulated cases. arXiv preprint arxfr:2306.05311.
Rohan Taori, Ishaan Guirajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-followee llama model. https:// gi thub. wwt.oo///ttaat watatsslla/ssaafotttoooo pacca..
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 202.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprint arXh::22
Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023, On the tool manipulation capability of open-source large lan- guage models.
Shunyu Yao. Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint ar 10.iv22210.03629
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski. Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vin- cent Vanhoucke, et al. 2022. Socratic models: Com- posing zero-shot multimodal reasoning with lan- guage. arXiv preprint 22405:2004.00598,

---

A
Appendix
No API Call: There no call detected the prediction.
Missing Input Parameters: Necessary input parame- ters are missing from the prediction.
Generate an API request in the format of (Aey- 'vauu1'' key2='value2? vallue'l....) ...)] based on the previous dialogue context. The current year is 2023.
A.2 Implement Details
The evaluation prompts consisted of two parts: for API testing (Figure 4) and the other for response testing (Figure 5). We kept the prompts as concise as possible to showcase the models' fundamental capabilities.
Input:
User: User's utterence
Al: Al's response
A.3 Examples
Expected output:
Because the huge amount of the domains in the training set, is impossible to give a table statisties, so we only count the distribution of the API in the test set on the domain and give the following table:
API-Request: [ApiNamee(eey==valleel)) key2=\value22",...
API descriptions:
Domains
Numbers
Figure 4: The prompts used for API call evaluation.
Account Management
Information Query and Processing
22
Generate response as an Al assistant
Health Management
8
Schedule Management
19
based on the previous dialogue context and API request.
Smart Home
6
Finance Management
6
The current year is 2023.
Others
5
Input:
Table 8: The domain statistics for the test set.
User: User's utterence
Al: Al's response
Here are some examples of the training and testing set domains along with their associated APIs:
API-Request: [ApiNamme((eyI==Valleee)) key2='value2", ...]]
Expected output:
Training Set Domains:
Al: Al's response
Mental Health Hotline and Support
Dental Procedure Cost Estimate
API descriptions:
Nutrition Planning
Training Set APIs:
Figure 5: The prompts used for response evaluation.
SearchDoetors
GetPrice
A.1 Error Definitions
RecordMaintenacce
We briefly outline the definition of each type of errors:
Testing Set Domains:
Account Management
API Hallucination: API name in ground truth does not match name the prediction.
Hcalth Management
Entertainment
Has Exception:
The prediction triggers custom built-in Python exception, which should not be present in the ground truth.
Travel
Testing Set APIs:
Invalid Input Parameters:
The prediction contains
BookHottel
invalid input parameters.
DeleteAlarm
False API Call Format: The prediction's 's API call for- mat is not parscable.
SearchFngine

---

Prompt
Meccate in A2I reqpeat zt the Ecenat Aegilessoogeraaaall segelozicess, and) coaed as poeriess dialopa conte.. ourress yeur 2032.
Ber Boett
51% repprte
tot-top Dhyilioe Day shapteety tmphotattet mil
fily furteet prraectrns (aatotarty Prypess toceerppein feveeess falled1111
Thereefoeeeee "ineriipti rhetts casll wit most Fistand Redt sll relee, fire
unumae "aasorase"s "sppes pacecatess" Cataaaa's aas "ascaaata"" viscegt Faseoeed tor tis cal,, aises gec code, tase eaaia "Muiiiiation daane puRsSE "saccaa""" ("spart
rerificacion to de Late 1ay casded ise aucoos parsoot ('pper "acst, "Sasessttes" *ON au passoued of de (oppers "sastestiia"" "aezouss
Way
mer/"lly feceper
"descrtpe totf the
Phoyeets
Torrper forper_pernneeet persneters (Peckesti Pheppati faxes, "Meeetpeten" the toler the e35l
forgot my possword, con you help me reset it?
LIM
Response:
Suze,
derinitely help you. your usernane get
My username is feo and my email is fao@example.com
User
LLM
Respoose: Alright.
RoI-Reguent: [Morgot ratatoo 'ooggtt Paseward', ueermamee'ooo', erail~ Eso8exarple.com
veridication Ploase provide with reast your password.
verification code 970420
User
LIM
Response: What's your pasaword?
Myy new password is newpazsword.
User
Responee: Great.
API- ARI-Rutua
Fooggt seword utse uv-- fortion Code'', verificos fie code- 970450', passwarde (aanpasseed"))
observation:
Success
Response:
Your password has besa auccessiully reset.
Figure 6: One example of Calling APIs.

---

Prompt
irgut:
forr Wor's uctesono
3t. 43% exposse
Eepested
dascriptions:
(hranohy "ialderrders Nerce iptioerty tearikeo for velisenn trala Library lireed in keyesrass, figgt peranetevst ("sepyooda": M"sppet "acs", rowsstipriept, riee seyvood to swarch toe/flls "ourpet. paw.aeccess"" ("tert,paccia", I"tpp": Thise Istat lnlct diets" "sserryytinn: *tee beat naces tool(s), (10/
you help me reminder for my meating ot 3pm on Jonuary 5, 2023 with the topic "sales neport"?
User
LLM
API-Request: [Tool Secrcher (keyserds=" add reminder")]
Observation:
"iopur paranatere''
wss Uil, "ouigut patannlers" Hoahen's "ype'' face, docription" The Loian
Ideas for saaoch": teat sse tobandas the uear toban, it cheaes": "addesttdee"', "descrtptianss fidd a pestnder Apt Che Lakes thre percesters "tokentul Beteentent and toaee The "Lokes parsecer calera to Ue waer's tosm and desorigtion los the resiider sarinder should be tzigpered.* "sepet pacoratarz's P'tabants ('typets tase", "Seccriptien"s "Dpeets (teypelt
the 'costerr' paraeeter reters
trise" parareter speoiiies the thes
l'typet
"duccription's
pazsaesazats
tatz'. "Seecrionien" e
fatledtalal
Response:
Pound
Now, you your use chame, password?
you please de with
My username "newuser', password is "hewpass".
User
LLM
Response:Thank Now I'm going to get ozer taken.
API-Request:
[GetUserToken jusernane- 'newuser', password "azaword-'newpaza")
Observation:
(1 Loken''s (19k0]7h6g5fAdd3e
Response:
Gol
16
Now
at me add the reni nden for you.
Reg-att
[AddRemi MaidRnninder: ttcker=
19k001/6g5ttddde :deeee content- tsales zeport', time- 2020-20233-00 01-05 15:00:00"]]
Observation:
'stattu's 'euccess")
Response: You should receivv vonfi conttiin shortly.
Figure 7: One example of Eetrieving+Callinng APIs.

---

Prompt
will be Lested your obi cbility wake rulliple ArI walls to feltill repuiteeent based single seruencee
You vill be çlies an API bax thas inctudas sot of Mila sest ss csicuiator, tcansistor, WikiSasrch, ecc. Ibas you vart del, you have anarer
the searet cogire using seyoords: rry describs elcs these keyaonds. the tool search engine ther return you the noet releusot information about the tool papt naw, Stscripeion, tupeb/octgut pascresass).
call, based ressice, your
salt fior will intaro conalt 32t and cornats arplanatory tests which
call
with eves with
sace is in scaopie of a seze siace clacurr sopesseria you and ARL sopenserca the perser. valen,
Caquicenans: calcriate pozzlt 13135r and sur MISS
caloslatad,
valid
eccople.
Nectpot peneesters Prorette rreppats wrreet,, cestreperne safy the soeete the formiss:"11
ctacorc: weceilateittseeeee (2-5)169))
("roule"
ctusors paatelaats focnela (64011)
descriptiors
"spiCstett thoil Seandere,, Vescrrptionn: "Gurrtee for seteeet trols Hereayy based the hepordts" "penensters"; Peepernael (toyatt "atety Aeccoription tost. ehe keyerre to vernes fiy "respoos"" Pheorrattess Plypeet Nosion (aanteies siets", "Secciiptiee" 1ihe best zatch secstan.
Calculate the monthly salory after taxes for Financial Andlyst
User
L.M
Reiquet::::
[7oc1Searche: (xeyworda- totteeeoottooottotttttoooooo www.oooppptooooooaaaaraoooo
Observation:
(Topf, nanets 'ToolBearcher', 'ingut" I'kegsords's "ettccopstionnalaryy "ostpueti ("home") SeetSeorsstionda ary', 'desaripcion' 1897 far querying salary
'Lngut, ('accupation'; ("typet: "écoeditionn" libe oecupattion, paraaettratt' l'salary'i ("typa") "eleat', "deseription"
the 'Il,
decopation.
UPI-Raquatt: (Ge LOoccupt t So" any Joccupati fowe Pinanciel Ana lyst !Y;
"Gttecupationiollayyy observation:
["api_rame"
('scoupation': 'Financial Analyst'l> 'output": ("salory's 100003), "excoption's Nonn)
API-Request: [Tool] Searcivr (keywords= TexCslculator")
Observation:
sapi_name'; "toolGesrcher', 'ingut': keywords'; 'TaxCslculator" 'output' for calculating tax deductiona baaed on the given salary.'s "input, parensters's l'salary': ("type's 'eloat', 'description': parinaters's ('eslary_atcer_tate taxtt ('sype'; "floet', 'sescription's "The ralary after tax deduct lons. (1)), 'exception's None)
("nane' : "raxCatoulator", 'deseription's
eslculate seductions
API- ODI-Request:
Texxcelouletooceelllee. "10000
Observation:
caalary_attee tex" 70000.0;
Response:
For P'inacciel Anslyst with salacy of 5100, C00, the nonthly salacy after taxes 378,
Figure One example of Plamiing-Rerrevvinnn-Clliinn APIs.


Reviews:
[{"review_id": "PJT6rQbTnU", "reviewer": "Reviewer_ZmrS", "summary": "", "questions": "A. The reliance on ChatGPT for multi-agent data generation raises questions about the approach's resilience given ChatGPT's limitations. Is additional filtering necessary to maintain data quality?\n\nB. Regarding human annotations, what measures were taken for quality control and agreement assessment among annotators? How was the overall annotation quality upheld?\n\nC. The paper mentions a larger commercial tool-augmented LLM, can you share more details?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Based on your comments, we will make several improvements to our paper:\n\n1. (W1) Include a comparison of the data generation effects among other LLMs.\n2. (W2) Explore the fine-tuned performance of different model variants.\n\nMeanwhile, we hope our answers have well addressed your concerns.\n\n### **W1: Should explore other LLMs for data generation.**\nWe acknowledge the potential value of exploring other LLMs such as Claude2 for data generation. The core requirement in the process of training data as well as in the practice of multi-agent is the ability to complete the task according to the given prompt. Chatgpt and Claude2 are recognized as two powerful LLMs with the ability to follow instructions. We have verified in this work that the idea of using ChatGPT for training data synthesis and the overall synthesis results are as expected, so it can be considered feasible to use Claude2 as the core for training data synthesis. We will compare the effectiveness of different LLMs used for data synthesis in future work. \n\n### **W2: Should explore different model variants (larger models).**\nIn addition, it is also valuable to explore the performance of different model variants at different scales for tool use after fine-tuning, but due to the time and resource constraints during the rebuttal period, we were not able to complete the corresponding model training and experimental validation, and we will explore the fine-tuned performance of different model variants later (e.g. 13B and 65B).\n\n### **Q1: The reliance on ChatGPT for multi-agent data generation ... Is additional filtering necessary to maintain data quality?**\n\nIn the multi-agent data synthesis, there are two additional agents dedicated to ensuring data quality. One agent analyzes the correctness of synthesized dialogs, API parameters, and results, generating a check report. A second agent corrects the data based on this report. We've verified the efficacy of this process through random sampling and found significant improvements in data quality. After examining, the agent-corrected data, we found that 78\\% of the data that was automatically filtered by the tester agent did not adhere to our design principles as intended.\nTo further guarantee the quality of the training data, we randomly selected 100 samples from the training set and had the annotators check their quality. The results show that the available rate of the Multi-agent generated data is 94\\%, which is an 89\\% improvement compared to data generated solely by a single agent (self-instruct), indicating that our training set possesses a high level of quality and reliability.\n\n### **Q2: Regarding human annotations, what measures were taken for quality control ... How was the overall annotation quality upheld?**\nIn our study, we took several measures to maintain annotation quality and agreement. We recruited computer science students for annotation but ensured that each dialogue was reviewed by two annotators for consistent annotation. Moreover, two additional annotators were employed to verify the quality of annotations, including format, logical consistency, and API call reasonability. Each instance has been reviewed by four annotators to ensure its quality.\n\n### **Q3: The paper mentions a larger commercial tool-augmented LLM, can you share more details?**\nWe appreciate your interest in the larger commercial tool-augmented LLM mentioned in our paper. However, due to company policy, we are unable to disclose specific details. In general terms, this commercial LLM is built upon an internal base model within the company. It allows individual users or business clients to register APIs and utilize zero-shot capabilities for making API calls. We regret that we cannot provide more detailed information, but we hope this general description offers some insight into its functionality.\n\n### **Missing References & Typos Grammar Style And Presentation Improvements:**\nWe appreciate your suggestion to include a reference to the paper titled \"A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT\" in our manuscript. We have reviewed the paper, and it indeed provides valuable insights into prompt engineering techniques for enhancing interactions with ChatGPT. Multi-Agent synthesis of training data and bootstrapping of models for API calls is also essentially Prompt engineering, and has a relatively strong relevance to this paper. We'll add a citation to this paper in the camera-ready version and also correct the typos problem you mentioned.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work tackles the crucial task of enhancing LLMs with external tools and APIs, vital for their scalability. It introduces a comprehensive benchmark with clear metrics, realistic dialog evaluations, and an innovative multi-agent data generation method that cuts annotation costs. This technique could be valuable for complex LLM training data beyond this context. The model analysis offers unique insights into the current tool utilization challenges and potential research paths. By providing open access to benchmark data, code, and models, API-Bank bridges a notable resource gap in this field.", "reasons_to_reject": "The training data generation primarily depends on ChatGPT, which could have limitations. Exploring other LLMs like Claude2 could be valuable. The paper only focuses on one model variant, Lynx-7B; using larger models might yield further insights. However, I do not think any of these issues warrant rejecting the paper. The benchmark and analysis greatly contribute to advancing this NLP field. I recommend accepting the paper.", "questions_to_authors": "A. The reliance on ChatGPT for multi-agent data generation raises questions about the approach's resilience given ChatGPT's limitations. Is additional filtering necessary to maintain data quality?\n\nB. Regarding human annotations, what measures were taken for quality control and agreement assessment among annotators? How was the overall annotation quality upheld?\n\nC. The paper mentions a larger commercial tool-augmented LLM, can you share more details?", "paper_topic_and_main_contributions": "This paper presents API-Bank, a benchmark for enhancing large language models' (LLMs) proficiency in using external tools and APIs. Key contributions include:\n- Defining three core capabilities for tool-augmented LLMs: Call, Retrieve+Call, and Plan+Retrieve+Call APIs, forming a framework to assess tool usage.\n- Implementing a functional evaluation system with 73 APIs spanning diverse domains. It incorporates 314 manually annotated dialogues featuring 753 API calls to benchmark existing LLMs.\n- Introducing a training dataset generator using a multi-agent approach, significantly reducing annotation costs (98%) while maintaining diversity and authenticity.\n- Training LLM \"Lynx\" on the API-Bank dataset. Experimental results show progress over untrained models but also disparities compared to human performance.\n- Offering a comprehensive analysis of errors made by current state-of-the-art LLMs, unveiling challenges in API retrieval, decoding, and reasoning.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "References are adequate. Maybe a good idea to explore the prompt strategy you adopted and cite this paper: A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT", "Typos_Grammar_Style_and_Presentation_Improvements": "Should add highlight in Table 3"}, "extra_text": {}}, {"review_id": "SzfsgOsk5K", "reviewer": "Reviewer_1pDE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your constructive comments. We address your concerns as follows:\n\n**We have added more details to our paper:**\n1. (W1) Include statistical data distribution analysis.\n2. (W4) Provide specific explanations of error types.\n3. (W2) Revise all the inappropriate statements.\n\n**In the experimental section, we have added:**\n1. (W3/W6) Compare Lynx with Alpaca fine-tuned on other training data to validate the effectiveness of the training set.\n2. (W5) Compare the error types between fine-tuned Alpaca (Lynx) and non-fine-tuned Alpaca.\n\nMeanwhile, we hope our answers can sufficiently address your concerns, and if that is the case, we kindly ask you to consider increasing the score.\n\n### **W1: The distribution of the API calls is missing. ... This makes it difficult to evaluate if the benchmark is even useful.**\nFor the training set construction, the diversity of the domain determines the diversity of the generated tool-used APIs and dialogues. Given 1000 various occupations in the world, we instructed ChatGPT (gpt-3.5-turbo) to generate related domains and approporiate APIs that can be called to fulfill users' needs based on each occupation.\nThe training set obeys uniform distribution, consisting of 1000 domains and 2138 APIs (2 or 3 APIs per domain). The test set, also referred to our evaluation system, consists of 8 domains and 73 manually implemented APIs. \n\nHere is the statistics table for the test set.\n\n| Domains                          | Numbers |\n|----------------------------------|---------|\n| Account Management               | 7       |\n| Information Query and Processing | 22      |\n| Health Management                | 8       |\n| Schedule Management              | 19      |\n| Smart Home                       | 6       |\n| Finance Management               | 6       |\n| Others                           | 5       |\n\nWe also list some examples for the constructed domains, APIs for training and test sets.\n\n| Training set domains (example)                                                          | APIs (example)                             |\n|-----------------------------------------------------------------------------------------|--------------------------------------------|\n| Mental Health Hotline and Support,  Dental Procedure Cost Estimate,  Nutrition Planning | SearchDoctors, GetPrice, RecordMaintenance |\n\n| Testing set domains (example)                                | APIs (example)                       |\n|--------------------------------------------------------------|--------------------------------------|\n| Account Management, Health Management, Entertainment, Travel | BookHotel, DeleteAlarm, SearchEngine |\n\n### **W2: The writing of this paper is not up to the mark. The paper makes many strong statements without sufficient support.**\nYour feedback is very valuable for us, and we appreciate your effort in pointing out those strong statements in our paper.\nWe apologize for those statements that may have caused concerns. We understand and acknowledge the points you have raised, and we are committed to addressing all of them in the next version of our paper.\n\n### **W3: The work showcases ... of the dataset created.**    \n\nKindly note that we present the first comprehensive training dataset for multi-turn dialogues with multi-call APIs, while matching the three levels evaluation of LLM abilities (Call, Retrieve, Plan).\n\nConsidering your suggestion, we **conduct experiments to fine-tune Alpaca on the dataset build from ToolAlpaca**, ensuring a consistent prompt format with our API-Bank. Since ToolAlpaca does not involve api-retriever in its work, we only evaluate its ability of calling APIs to ensure the fairness for comparison.\n\nThe results are shown in the following table. We can notice that our model Lynx training on API-Bank dataset achieves better performance than ToolAlpaca even with fewer training data. This also indicates the better data quality of our constructed dataset.\n\n|             | Number of training samples | Accuracy (Call) | Rouge (Call) |\n|-------------|----------------------------|-----------------|--------------|\n| ToolAlpaca  | 10366                      | 53.88           | 39.75       |\n| Lynx (Ours) | 6184                       | 54.64           | 39.80       |\n\n\n### **W4: Section 7.3: Error analysis ... the standards of the work.**\n\nSorry for the inconvenience caused by our poor writing. We will definitely refine our paper in the next version. Below, we briefly outline the definition of each type of Error: \n1. **API Hallucination**: The API name in the ground truth does not match the name in the prediction.\n2. **Has Exception**: The prediction triggers a custom or built-in Python exception, which should not be present in the ground truth.\n3. **Invalid Input Parameters**: The prediction contains invalid input parameters.\n4. **False API Call Format**: The prediction's API call format is not parseable.\n5. **No API Call**: There is no API call detected in the prediction.\n6. **Missing Input Parameters**: Necessary input parameters are missing from the prediction.\n\n### **W5: The error analysis in section 7.3 ... However, this is also missing.**\n\nWe compared Alpaca's error analysis with (Lynx) and without fine-tuning on the dataset with the following results:\n\n| Error Type               | Rate   | Num |\n|--------------------------|--------|-----|\n| No API Call              | 36.77% | 157 |\n| API Hallucination        | 15.93% | 68  |\n| Invalid Input Parameters | 7.96%  | 34  |\n| False API Call Format    | 23.65% | 101 |\n| Miss Input Parameters    | 1.17%  | 5   |\n**Alpaca Error Distribution**\n\n| Error Type               | Rate   | Num |\n|--------------------------|--------|-----|\n| No API Call              | 5.29%  | 10  |\n| API Hallucination        | 61.38% | 116 |\n| Invalid Input Parameters | 8.47%  | 16  |\n| False API Call Format    | 6.88%  | 13  |\n| Miss Input Parameters    | 1.59%  | 3   |\n**Lynx Error Distribution**\n\nThe primary issue observed with the original Alpaca model in our evaluation was the occurrence of \"No API Call.\" This issue was significantly reduced in the Lynx model after fine-tuning on the API-Bank training dataset. We speculate that differences in the patterns of API calls between Alpaca's training dataset (constructed using 52K instruction data) and the actual API calls may have contributed to this problem. Understanding the instructions for API calls in the training data may have been challenging for the model.\n\nAnother issue we observed was \"False API Call Format.\" Since our testing prompts were in a zero-shot format, relying solely on instructions to guide the model to make API calls might have made it difficult for the model to learn the correct API call format. Fine-tuning led to a significant improvement in this aspect, highlighting the substantial benefit of fine-tuning in enhancing the model's ability to produce correctly formatted API calls.\n\n### **W6: Although the approach ... developing tool-augmented LLMs.**\n\nAt the time of our research, there were no existing works that included training datasets specifically designed for tool-augmented LLMs. We were among the first to explore the methodology of synthesizing training data for this purpose. We also compared the performance of Tool-Alpaca against Alpaca as a point of reference.\nWe would like to emphasize our key advantages:\n1. **Complexity of Training Data**: In our training dataset, we synthesized multi-turn dialogues, each may containing multiple API calls. This complexity in the data is a distinguishing feature. It allows our model to handle situations where a single user query may involve multiple API calls. This complexity is not present in many existing benchmarks.\n2. **API Call & Response Training**: Our training process not only involves teaching the model to make API calls but also to generate responses based on the API's return values. This comprehensive training approach ensures that our model has a deeper understanding of the entire API call process, including handling return values.\n\nWe believe that these unique aspects of our benchmark set it apart and make it particularly well-suited for developing tool-augmented LLMs. While we acknowledge the importance of comparing benchmarks, the absence of comparable benchmarks during the time of our work limited our ability to make such comparisons. We hope that future research in this area can build upon our findings and extend these comparisons.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The annotation scheme designed and implemented for the evaluation dataset is comprehensive and well thought out.\n\n* The comparisons made by the authors show that their dataset has a significantly higher diversity of domains and the number of distinct APIs. Their dataset also includes multi-turn dialogue with multiple APIs being called multiple times. Additionally, the proposed design principles of segregating API calls into bins for planning, retrieving and calling allow better diversity of complexity in the dataset. All these features strongly add to the value of the dataset in my opinion. It would be valuable to have such a comprehensive dataset for the development of tool-augmented LLMs.", "reasons_to_reject": "* The distribution of the API calls is missing. This would show if the dataset has any bias on the usage of the API calls. The training and test sets are not defined. The authors indicate the 73 tool APIs selected are commonly used but do not list the tools that this dataset is consisting of. This makes it difficult to evaluate if the benchmark is even useful.\n\n* The writing of this paper is not up to the mark. The paper makes many strong statements without sufficient support. A few of them for instance are: \n\n   - \"a ground-breaking benchmark\" is a strong statement on L009 and is repeated to be used in L636, \n\n   - \"comprises 73 commonly used APIs\" on L078 has no supporting evidence nor any list of the APIs provided in the paper, \n\n   - \"Our investigation reveals that the errors stems from providing LLM with an overwhelming number of requirements all at once, making it difficult to comprehend them effectively\" on L349 does not refer to any study given in the paper, \n\n   - \"GPT-4, currently known as the most powerful LLM\" on L537 is another strong statement, \n\n   - \"We hypothesize that this is due to GPT-4’s emphasis on reasoning and planning abilities, similar to its performance in tasks involving mathematical reasoning.\" on L542 is an unwanted statement on GPT4's mathematical reasoning which again lacks support. Although GPT4 is irrelevant to the author's work such instances reflect the low quality of grounded writing presented in this research article.\n\n   - \"In contrast, their evaluation set is generated through self-instruct by the LLM and lacks diversity due to its narrow domain focus\" on L564 refers to the comparison made with other benchmarks. This is insufficient as there is no explanation about the diversity and distribution of the 1000 domains that the dataset uses.\n\n* The work showcases finetuning performances on only a single model, Alpaca. The paper misses to compare the effect of finetuning Alpaca on other similar datasets and comparing their performance. This would have allowed assessing the quality of the dataset created.\n\n* Section 7.3: Error analysis begins with tables 4 and 5 based on some experimental setups that are not introduced in the writing earlier. This makes it difficult in understanding what exactly the errors correspond to. The errors mentioned in these tables are also not explained earlier. Later in the text, the writing explains some stats in those tables but lacks reference to the table. All these contribute to the bad writing quality dropping the standards of the work.\n\n* The error analysis in section 7.3 is only on the fine-tuned Alpaca (Lynx model) and GPT4. However, since GPT4 is an unknown model, the analysis of its errors adds no significant value to the analysis. The motivation to analyze only the two among the complete set of models that were evaluated is not clear and not explained. It would have made sense to compare the error analysis of Alpaca with fine-tuning (Lynx) and without fine-tuning on their dataset. That would have provided insight on the quality of their benchmark. However, this is also missing.\n\n* Although the approach for data creation is sophisticated and well-developed, the evaluation and analysis of the benchmark remains weak in terms of comparing with other benchmarks the effectiveness on the task of developing tool-augmented LLMs.", "questions_to_authors": "", "paper_topic_and_main_contributions": "* This work's main contributions are focused on improving tool-augmented large language models (LLMs) through benchmarking. The 3 main research questions addressed by this paper are through the following three proposed approaches: \n\n   (i) What is the state of current LLMs: Evaluation of current LLMs' capability of utilizing tools.; \n\n   (ii) How to enhance their capability of utilizing tools: Development of a large dataset to benchmark the performance. Proposed a method to accelerate the annotation of the required dataset. Propose their model Lynx that is finetuning of the Alpaca model using their dataset.;\n\n   (iii) Obstacles and gaps for further improvement in leveraging the use of tools by LLMs: Error analysis resulting in discussion of scope for future directions.\n\n* For the evaluation of current LLMs, the authors develop a dataset consisting of 73 tools on 314 dialogues making a total of 753 API calls. This is made manually by humans.\n\n* The evaluation is made on the following aspects: planning, retrieving and calling of the APIs. They incorporate this with the help of their design principles of the API-Bank. For this, they define two dimensions of evaluation: Few vs Many APIs in a pool & Single vs Several API calls. The presence of few APIs tests the LLM's ability to call; in the presence of many APIs the ability to retrieve+call can be assessed; finally, multiple calls in the presence of several APIs add to the assessment of planning as well.\n\n* The training dataset created has 1,888 dialogues from 1000 domains and 2,138 tools making a total of 4,149 API calls.\n\n* For annotation of the large training dataset, the authors propose and use a method using LLMs - Multi-agent. This method consists of 5 LLM agents collaborating to iteratively generate the domain, API call, select the user query, call API and respond.\n\n* The paper uses the performances of Alpaca, ChatGLM, GPT3.5 and GPT4 to make comparisons with their fine-tuned model Lynx.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos: \"obtaine\" on L269; \"a dn\" on L467;"}, "extra_text": {}}, {"review_id": "v3etF9Yr15", "reviewer": "Reviewer_FRbH", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Based on your comments, we will make several improvements to our paper:\n1. (W1) **Quality of queries**: We will add more details about how to ensure the quality of queries\n2. (W2) **Model replication**: We will include a section that provides a comprehensive overview of the test conditions, including the prompts used, the parameters used for model generation, and any relevant information that affects the test environment.\n\nMeanwhile, we hope our answers have well addressed your concerns.\n\n### **W1: Apart from APIs, a very important issue ... discusses this relatively little.**\n\nIn our data synthesis process, we take several steps to guarantee both the diversity and quality of user queries.\n\nFor the training set construction, the diversity of the domain determines the diversity of the generated tool-used APIs and dialogues. Given 1000 various occupations in the world, we instructed ChatGPT (gpt-3.5-turbo) to generate related domains and approporiate APIs that can be called to fulfill users' needs based on each occupation. \n\nAdditionally, in our multi-agent data synthesis, we employ two dedicated agents to enhance data quality. One agent performs an analysis of synthesized dialogs, API parameters, and results, generating a comprehensive quality check report. A second agent then corrects the data based on this report. Through random sampling, we have verified the effectiveness of this process and observed significant improvements in data quality, eliminating the need for additional filtering.\n\nFurthermore, for the evaluation set, each instance has been reviewed by four annotators to ensure its quality. To further guarantee the quality of the training data, we randomly selected 100 samples from the training set and had the annotators check their quality.\n\nWe believe these measures collectively contribute to the high quality and diversity of the queries in our dataset, and we appreciate the reviewer's attention to this critical aspect of our work.\n\n### **W2: Some details of model replication are not discussed in detail ... in terms of their scientic validity.**\n\nThe models under evaluation were categorized into two groups: those directly used for inference and those that underwent fine-tuning. Importantly, both groups used the same prompts during evaluation. This design choice was made to ensure fairness in the evaluation process.\n\nThe prompts themselves consisted of two parts, one for API testing and the other for response testing. We kept the prompts as concise as possible to showcase the models' fundamental capabilities. We acknowledge that future work can explore prompt variations, such as adopting the ReAct Prompt mode, to potentially improve performance. We intend to add these implementation details in the camera-ready version of the paper.\n\n**Prompt for API Call Testing**\n```\nGenerate an API request in the format of [ApiName(key1='value1', key2='value2', ...)] based on the previous dialogue context.\nThe current year is 2023.\nInput: \nUser: User's utterence\nAI: AI's response\n\nExpected output:\nAPI-Request: [ApiName(key1='value1', key2='value2', ...)]\n\nAPI descriptions:\n```\n\n**Prompt for Response Testing**\n```\nGenerate a response as an AI assistant based on the previous dialogue context and API request.\nThe current year is 2023.\nInput: \nUser: User's utterence\nAI: AI's response\nAPI-Request: [ApiName(key1='value1', key2='value2', ...)]\n\nExpected output:\nAI: AI's response\n\nAPI descriptions:\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Tool Learning is an essential topic in the era of LLM.\n\n2. The paper provides very valuable resources, especially the large number of APIs and the virtual evaluation environment.", "reasons_to_reject": "1. Apart from APIs, a very important issue is how to ensure the quality of queries (i.e., queries that can naturally and reasonably call multiple APIs), but this paper actually discusses this relatively little.\n\n2. Some details of model replication are not discussed in detail, such as how existing models are implemented to call APIs, which makes the evaluation results of the given models somewhat difficult to measure in terms of their scientific validity.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper constructs a system-level platform for evaluating LLM API calls, which includes over 2000 APIs, and establishes a set of environments for continuous evaluation. The only downside is that this platform is not accessible during the review period, making it difficult to assess its reproducibility and actual quality.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4q4D43nHz1", "reviewer": "Reviewer_vaMb", "summary": "", "questions": "- Although this paper is one of the very first works on creating an evaluation benchmark for evaluating tool-augmented LLMs, the authors at least need to mention or compare with the most recent tool-usage benchmarks for a complete EMNLP submission, including 2 ToolBench (https://github.com/OpenBMB/ToolBench, https://github.com/sambanova/toolbench) and 1 ToolQA (https://arxiv.org/abs/2306.13304). I know some of them might be concurrent work, but it would be better if you can make a comparison with them.\n- I noticed a very interesting number that the authors keep emphasizing in the writing: \"1008 domains\". But I could not find any explanations/illustrations of what are the domains. And it is still quite weird for me that 2202 dialogs are distributed on 1008 different domains. I am wondering if there is a visualization on the domain distribution or are there 2 dialogs on the same domain?\n- What are the differences between other tuned tool-augmented LLMs (ToolAlpaca, ToolLLaMA, Gorilla) and Lynx from a methodology angle? \n\nFrom the reviewer: I know it is not possible to compare all the recent methods/concurrent works. But as this direction is developing fast, it would be better if you can compare your work with the other works I mentioned in the question section. I will also take this into consideration when judging whether to change the score or not. Thanks!", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Based on your comments, we will make several improvements to our paper:\n\n1. (W1) Explore the impact of different prompt styles on performance.\n2. (W2) Reorganize the content for improved clarity.\n3. (W3) Provide more details on tool implementation.\n4. (Q1) Extend Table 1 to include relevant works.\n5. (Q2) Include statistical data distribution analysis.\n6. (Q3) Clarify the differences between our approach and other adapted tool-enhanced LLMs.\n\nMeanwhile, we hope our answers can sufficiently address your concerns, and if that is the case, we kindly ask you to consider increasing the score.\n\n### **W1: Not enough baselines ... The current experiments seem to be too weak for me.**\n\nWe appreciate the reviewer's comment regarding the need for a broader set of baselines in our evaluation. We want to clarify our approach and rationale.\n\nIn the context of the latest large language models, we recognize that prompt design is a crucial aspect of their development. While our primary goal was to assess the practical tool-usage abilities of these models, we also hope this benchmark that could serve as a guideline for evaluating different styles of prompts.\n\nTo maintain a focus on fundamental baselines, we deliberately refrained from incorporating specific examples or complex prompt techniques during evaluation. This choice was made to showcase the models' core capabilities without relying on intricate prompt engineering.\n\nWe acknowledge the existence of other tool-augmented LLMs like ReAct, Reflexion, Chameleon, and others, and we appreciate the reviewer's suggestion to include them in the evaluation. However, our intention was to provide a foundational assessment of tool-usage abilities, and we plan to explore comparisons with these models and their prompt styles in future work.\n\n\n### **W2: Although the paper is easy to follow, the writing still needs to be improved ... on what they really want to claim.**\n\nWe appreciate the reviewer's feedback on the paper's clarity and focus. We want to emphasize our core contributions:\n\n1. We develop an authentic and interactive evaluation system with 73 API implementations and 314 annotated tool-used dialogues. We are the first to propose design principles and measure the ability of large language models to use tools from three different levels of difficulty: call, retrieval, and planning.\n2. Utilizing the technique of multi-agent cooperation, we construct the first comprehensive training dataset for multi-turn dialogues with multi-call APIs, tailored for tool-augmented LLMs. This dataset consists of 2,138 APIs and 1,888 dialogue annotations.\n3. We finetune the constructed dataset with Alpaca-7B to build our tool-augmented LLM called Lynx. The experiment results verifiy the superior performance of Lynx over other open-resource LLMs and even comparable to GPT-3.5. Our extensive analyzes of experimental results also shed light on key challenges in the field of tool-augmented LLMs.\n\nWe apologize for any readability issues and appreciate the feedback. We will reorganize the content for improved clarity in the camera-ready version.\n\n### **W3: Lack of implementation details ... (e.g., retreiver, etc.).**\n\n\nWe appreciate the reviewer's attention to implementation details. We would like to provide more clarity on our tool implementation:\n\nRegarding the **73 APIs**: The entire interactive environment used for evaluation was implemented in Python and includes 73 runnable APIs. For APIs related to database operations (e.g., adding reminders - database insert, canceling meetings - database delete), we initialize the corresponding databases and generate initial entries for dialogue construction. For APIs that access external information (e.g., search engines), we must ensure the retrieved information remains constant to ensure reproducibility. We track all queries for each API in test dialogues and record the retrieval results at a specific time point, hard-coding them within the API to maintain result consistency.\n\nAs for the **retriever implementation**, we adopted a fundamental approach. The model generates calls in the format \"APISearch(keywords='[keyword1 keyword2 ...]').\" The \"keywords\" section consists of multiple keywords separated by spaces. The retriever first extracts a sentence embedding from the entire string (including spaces) using the model: \"sentence-transformers/paraphrase-MiniLM-L3-v2.\" Then, it extracts sentence embeddings for all API meta information (API Name + API description) in the API Pool. It calculates the cosine similarity between keywords and all API embeddings and returns the meta information of the API with the highest similarity.\n\nWe hope this clarification addresses the concerns about implementation details. Thank you for highlighting this, and we will ensure that these details are presented clearly in the final paper.\n\n### **Q1: Although this paper ... a comparison with them.**\n\n| Benchmark       | Statistics |        | Dialogue Type |            | Evaluation Type |          |\n|-----------------|------------|--------|---------------|------------|-----------------|----------|\n|                 | # domains  | # APIs | Multi-turn    | Multi-call | API Call        | Response |\n| API-Bank (ours) | 1,000      | 2,138  | Y             | Y          | Y               | Y        |\n| ToolBench1      | 49         | 16,464 | Y             | Y          | Y               | N        |\n| ToolBench2      | 8          | 232    | N             | Y          | Y               | N        |\n| ToolQA          | 6          | 13     | N             | Y          | Y               | Y        |\n\nIn our camera-ready version, we will extend Table 1 as indicated and include references to the relevant works for a more thorough evaluation and comparison with the mentioned benchmarks. Thank you for highlighting these benchmarks, and we will ensure their inclusion to enhance the completeness of our submission.\n\n### **Q2: I noticed a very interesting number ... on the same domain?**\n\nFor the training set construction, the diversity of the domain determines the diversity of the generated tool-used APIs and dialogues. Given 1000 various occupations in the world, we instructed ChatGPT (gpt-3.5-turbo) to generate related domains and approporiate APIs that can be called to fulfill users' needs based on each occupation.\n\nThe training set obeys uniform distribution, consisting of 1000 domains and 2138 APIs (2 or 3 APIs per domain). We achieve dialogue variety in our dataset by interchanging the order of these three APIs, which allows us to synthesize 2-3 distinct dialogues for each domain.\nThe test set, also referred to our evaluation system, consists of 8 domains and 73 manually implemented APIs. \n\n\n\nHere is the statistics table for the test set.\n\n| Domains                          | Numbers |\n|----------------------------------|---------|\n| Account Management               | 7       |\n| Information Query and Processing | 22      |\n| Health Management                | 8       |\n| Schedule Management              | 19      |\n| Smart Home                       | 6       |\n| Finance Management               | 6       |\n| Others                           | 5       |\n\nWe also list some examples for the constructed domains, APIs for training and test sets.\n\n| Training set domains (example)                                                          | APIs (example)                             |\n|-----------------------------------------------------------------------------------------|--------------------------------------------|\n| Mental Health Hotline and Support,  Dental Procedure Cost Estimate,  Nutrition Planning | SearchDoctors, GetPrice, RecordMaintenance |\n\n| Testing set domains (example)                                | APIs (example)                       |\n|--------------------------------------------------------------|--------------------------------------|\n| Account Management, Health Management, Entertainment, Travel | BookHotel, DeleteAlarm, SearchEngine |\n\n\n### **Q3: What are the differences between other tuned tool-augmented LLMs ... from a methodology angle?**\n\nThe differences between our approach and other adapted tool-enhanced LLMs (ToolAlpaca, ToolLLaMA, Gorilla) are as follows\n\nIn terms of data generation:\n- **ToolAlpaca**: It involves two phases - collecting tool names from the internet and synthesizing tool documentation using LLMs. The training data is generated through multi-agent interactions between users, assistants, and tool executors.\n- **ToolLLaMA**: This approach utilizes 16,464 publicly available RESTful APIs. It employs a depth-first search decision tree method to traverse feasible combinations of APIs across various scenarios, generating training data.\n- **Gorilla**: Gorilla uses a self-instruct method, starting with 18 seed instruction-API pairs to synthesize its training data.\n\nIn terms of training:\n\n- All approaches use LM training objectives combined with masking to train the models to generate the desired outputs.\n- The main difference lies in the fine-tuning prompts. API-Bank uses pure instruction zero-shot prompts. ToolAlpaca and ToolLLaMa use ReAct-style prompts and model output styles. Gorilla employs custom prompts that include API calls and explanations, along with the model's output style.\n\nWe hope this clarifies the differences in methodology between our work and the mentioned LLMs. Thank you for your question, and we'll ensure that these distinctions are made clear in the final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents API-Bank, the first benchmark tailored specifically for tool-augmented Large Language Models (LLMs). To the best of my knowledge, API-Bank is the first comprehensive benchmark in this direction.\n- The paper divided their data into different levels of difficulty and propose three different levels of data to answer three important questions in the tool-augmented LLMs research.\n- The paper is well-written and easy to follow. Good visualizations are also included to help with people understanding.\n- The paper conducts extensive experiments on both closed- and open-source LLMs, evaluating their efficacy of tool usage.", "reasons_to_reject": "- Not enough baselines. The authors only compare with the zero-shot closed-source LLMs (gpt-3, gpt-3.5, gpt-4) and zero-shot open-source LLMs(alpaca, ChatGLM). A lot of prompting-based tool-augmented LLMs are not tested, like ReAct (https://arxiv.org/abs/2210.03629), Reflexion (https://arxiv.org/abs/2303.11366v3), Chameleon (https://arxiv.org/abs/2304.09842), etc. Comparing not enough baselines limits the paper to only evaluating the tool-usage abilities of LLMs with no additional design. The current experiments seem to be too weak for me.\n- Although the paper is easy to follow, the writing still needs to be improved. From my perspective, the authors wanted to cover a lot of things and miss the emphasis on what they really want to claim.\n- Lack of implementation details. The details of tool implementation are missing in this paper and so are some important components (e.g., retreiver, etc.).", "questions_to_authors": "- Although this paper is one of the very first works on creating an evaluation benchmark for evaluating tool-augmented LLMs, the authors at least need to mention or compare with the most recent tool-usage benchmarks for a complete EMNLP submission, including 2 ToolBench (https://github.com/OpenBMB/ToolBench, https://github.com/sambanova/toolbench) and 1 ToolQA (https://arxiv.org/abs/2306.13304). I know some of them might be concurrent work, but it would be better if you can make a comparison with them.\n- I noticed a very interesting number that the authors keep emphasizing in the writing: \"1008 domains\". But I could not find any explanations/illustrations of what are the domains. And it is still quite weird for me that 2202 dialogs are distributed on 1008 different domains. I am wondering if there is a visualization on the domain distribution or are there 2 dialogs on the same domain?\n- What are the differences between other tuned tool-augmented LLMs (ToolAlpaca, ToolLLaMA, Gorilla) and Lynx from a methodology angle? \n\nFrom the reviewer: I know it is not possible to compare all the recent methods/concurrent works. But as this direction is developing fast, it would be better if you can compare your work with the other works I mentioned in the question section. I will also take this into consideration when judging whether to change the score or not. Thanks!", "paper_topic_and_main_contributions": "The paper introduces a new benchmarking platform, API-Bank, to evaluate the efficacy of the tool-augmented large language models (LLMs). API-Bank is a comprehensive benchmark that includes 3-levels of data, answering three key questions in tool-use domain. These resources are designed to thoroughly evaluate the ability of LLMs to plan step-by-step API calls, retrieve relevant APIs, and correctly execute API calls to meet human needs.\n\nThe authors made the following contributions:\n- The authors present the first benchmark tailored for tool-augmented LLMs. This benchmark includes a variety of API tools and annotated dialogues to evaluate the performance of LLMs.\n- The authors use API-Bank to evaluate the performance of different closed-source and open-source LLMs.\n- The authors provide a detailed error analysis and case studies to demonstrate the feasibility of tool-augmented LLMs for daily use, as well as to identify the primary challenges that future research needs to address.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "See them in drawbacks and questions.", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents API-Bank, a benchmark for enhancing large language models' (LLMs) proficiency in using external tools and APIs. As mentioned by the reviewers, the paper make several contributions. For the evaluation of current LLMs, a dataset consisting of 73 tools on 314 dialogues (a total of 753 API calls) has been developed. It also includes multi-turn dialogue with multiple APIs being called.
The paper introduces a training dataset generator using a multi-agent approach, significantly reducing annotation costs while maintaining diversity and authenticity. The paper also proposes their model Lynx, which fine-tunes Alpaca model using their dataset.

The reviewers pointed towards some issues with the writing, along with missing details. For instance, the details of tool implementation are missing in this paper and so are some important components The distribution of API calls is missing. The authors have provided these details in the rebuttal, and it should be added to the main paper.
Decision: Accept-Main