Title: AppBench: Planning of Multiple APIs from Various APPs for Complex User Instruction
Benchmark: AppBench
Summary: AppBench is a benchmark for complex API interaction with interweaving dependencies across multiple Apps. It leverages graph relationships between apps to construct more complex test cases than previous works. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Interdependent tool use
Phenomenon defined by authors: No
Phenomenon definition: Summary: The ability of a planning LLM to construct planning paths spanning potentially multiple different apps and APIs. 
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Not defined
Task definition: Given a set of Apps and APIs and an instruction, the LLM must construct a `planning path`, i.e., a list of APIs to call with associated arguments. 
Task item definition: A set of apps and associated APIs and an instruction (e.g., "book a hotel at EMNLP 2024")
Task source: Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: The tasks are based on human task-oriented dialogue datasets. LLMs are used to reformat and score the reformatted datasets. 
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 800
Dataset metadata available: Yes
Dataset metadata detail: complexity of Apps, complexity of APIs, number of APIs and Apps. 
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: The tasks are broken down into overall success, F1 of App selection, and F1 of API selection. Success requires specifying fully correct Apps, APIs and parameters. 
Metric aggregation: Simple Mean
Metric statistics: simple mean. 
Metric subscores: Yes
Metric subscores detail: Different complexity levels (single/multiple app/api)
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Agents / Tool Use
New taxonomy: General Application / Tool Use
Procedural notes: Note, the few shot is only in an ablation and not in the main results