Title: API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Benchmark: liAPIbankComprehensiveBenchmark2023
Summary: The paper introduces API-Bank, a benchmark designed to evaluate and enhance LLMs in utilising external tools. It addresses questions regarding the effectiveness of current LLMs in tool use, methods for improvement, and remaining obstacles. The benchmark includes a runnable evaluation system with 73 API tools and a comprehensive training set of tool-use dialogues, used to train a tool-augmented LLM called Lynx.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Ability of an LLM to plan, retrieve and correctly call external APIs (i.e., tool use).
Phenomenon defined by authors: Yes
Phenomenon definition: The paper explicitly defines three capacities:
1. Call: the ability to call APIs…
2. Retrieval + Call: the ability to retrieve and call a single API…
3. Plan + Retrieval + Call: the ability to continuously plan, retrieve, and call multiple APIs… 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Models must follow a user dialogue, decide which API(s) to invoke (possibly searching for them first), generate structurally correct calls, and finally answer the user with the API results.
Task item definition: A dialogue (single‑ or multi‑turn) annotated with the gold API call(s) and expected assistant response.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Dialogs simulate realistic user requests (e.g., booking hotels, resetting passwords) but are not collected from actual user logs.
Dataset size: 314
Dataset size detail: Evaluation dialogue count is 314, comprising 914 turns and 753 API calls.
The number of dialogues and API calls varies across the different abilities (Call, Retrieve+Call, Plan+Retrieve+Call) within both the training and evaluation sets.
Dataset size notes: 1888
Dataset metadata available: Yes
Dataset metadata detail: Each dialogue is tagged with domain, ability category and counts of API calls/turns.
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric subscores detail: Scores are provided separately for each of the three abilities: Call, Retrieve+Call, and Plan+Retrieve+Call.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: They provide data quality checks (four‑annotator review, 78 % rejection by “tester” agent) and discuss limitations such as language coverage.
Taxonomy: Agents / Tool Use
New taxonomy: General Application / Tool Use