Title: TaskBench: Benchmarking Large Language Models for Task Automation
Benchmark: TaskBench
Summary: TaskBench is a framework for evaluating how well large language models (LLMs) can automate complex tasks. It addresses three stages of task automation: task decomposition, tool selection and parameter prediction. It introduces Tool Graph - a novel representation of tools and their dependencies.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: task automation
Phenomenon defined by authors: Yes
Phenomenon definition: "task automation, which involves decomposing complex tasks described by user instructions into sub-tasks and invoking external tools to execute them, playing a central role in autonomous agents."
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: The task requires models to generate task steps and a tool graph based on the user instruction
Task item definition: A single item consists of a user instruction and the output contains detailed task steps and a tool graph with parameters.
Task source: Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The task simulates real user instructions that would be given to autonomous agents, but in a controlled environment.
Dataset size: 17,331
Dataset metadata available: Yes
Dataset metadata detail:  tool graph structure, number of tools, domain , tool names, tool dependencies, parameters required
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Correlation (Matthew's correlation, Pearson's r)
Metric aggregation: Simple Mean
Metric statistics: simple means to report F1 scores and ROUGE metrics
Metric subscores: Yes
Metric subscores detail: domain categories, different graph structure types, and complexity levels 
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: They include human evaluation of the dataset quality, asking experts to rate samples on naturalness, complexity, and alignment. They compare to existing baselines.
Taxonomy: Agents / Tool Use
New taxonomy: General Application / Tool Use