Title: API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs
Benchmark: API-Blend
Summary: API-Blend introduces a benchmark and training dataset for constructing API calls with LLMs. It especially focuses on enhancing existing datasets to mimic real-world use cases for the tasks of tool detection, slot filling, and sequencing.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: API usage.
Phenomenon defined by authors: No
Phenomenon definition: LLMs should be able to perform simpler tasks with APIs such as (a) APIs detection: Based on a user query, correctly choose which API to call, (b) Slot filling: Given the API, extract either the slots/parameters from the user utterances or request from the user more information to fill the required parameters of the detected API, and (c) Sequencing: Given an utterance specifying a task, write the sequence of APIs that needs to be called to accomplish the task.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Purpose notes: Note, that while the sub elements' definitions are uncontested, the specific breakdown is somewhat more contested (but broadly matches the literature)
Task definition: The task is given a query to provide a list of predicted APIs with filled in slots.
Task item definition: Each item has a _query_, a gold set of API-calls, and an answer. 
Task source: Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: While the tasks are based on real queries, the setup is fairly artificial. 
Dataset size: 13355
Dataset size detail: Size of subcategory test sets range from 71 to 6095
Dataset size notes: Train: 159476, Dev: 17946
Dataset metadata available: No
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric definition detail: The n-gram is specifically longest common subsequence for sequential tool calls. 
Metric aggregation: Weighted Mean
Metric statistics: Weighted mean based on test samples.
Metric subscores: Yes
Metric subscores detail: By dataset
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Agents / Tool Use
New taxonomy: General Application / Tool Use