Title: METATOOL BENCHMARK FOR LARGE LANGUAGE
MODELS: DECIDING WHETHER TO USE TOOLS AND
WHICH TO USE
Benchmark: MetaTool
Summary: MetaTool proposes a benchmark for tool selection. It encompasses a diverse set of scenarios and four different settings (Similar tools, multi-tool, scenario, and reliability). The benchmark only focuses on tool selection and not actual execution.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Tool selection
Phenomenon defined by authors: Yes
Phenomenon definition: They break it down into tool awareness, i.e., "whether LLMs can resort to external tools when they encounter problems they cannot solve" and actual tool selection, which they define as a knowledge retrieval task given a set of tools and a query.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: The task is broadly to select the relevant tool(s) (if any) given a query. 
Task item definition: A query with a set of "correct" tools to use.
Task source: LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Tool descriptions are sourced from OpenAI plugins but the actual queries are LLM-generated
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The task is deliberately a narrow aspect of "real" QA tasks. Still, it's unclear how realistic the queries are.
Dataset size: 975
Dataset size notes: Train: 21127
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test, Train
Response format: Free response (e.g. summary paragarph)
Response format detail: The responses are specifically a set of tools
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: For multi-tool, it's reported for different strictness (e.g., "only one of two correct)
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Author validity detail: The do human validation of the benchmark and whether the queries reliably trigger tools, but no more than that. 
Taxonomy: Agents / Tool Use
New taxonomy: General Application / Tool Use