Title: ToolQA: A Dataset for LLM Question Answering with External Tools
Benchmark: ToolQA
Summary: This paper introduces ToolQA, which is designed to faithfully evaluate LLMs’ ability to use external tools for question answering as compared to just retrieving from memorization. ToolQA involved a scalable, automated process for dataset curation, along with 13 specialized tools designed for interaction with external knowledge in order to answer questions.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Question answering with external tool utilization. 
Phenomenon defined by authors: Yes
Phenomenon definition: Ability to answer a question using external tools to obtain information from a reference corpus without relying on intrinsic parametric knowledge
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Given a question, a reference corpora and tools, use the tools to retrieve information from the reference corpora that can be helpful in providing an answer
Task item definition: A single task item consists of a question, answer
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1530
Dataset metadata available: Yes
Dataset metadata detail: Difficulty
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric subscores detail: Subtask dataset and difficulty
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Agents / Tool Use
New taxonomy: General Application / Tool Use