Title: AppWorld: A Controllable World of Apps and People
for Benchmarking Interactive Coding Agents
Benchmark: AppWorld Benchmark
Summary: AppWorld introduces a benchmark and simulator for evaluating LLM-based agents in performing complex, real-world digital tasks across multiple apps via API calls.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Tool use, planning, natural language understanding, reasoning, decision-making (based on the enviornment).
Phenomenon defined by authors: Yes
Phenomenon definition: "Automating this not only requires operating multiple apps via APIs but also performing interactive reasoning and sequential decision-making, while handling unexpected hurdles, and avoiding undesirable outcomes." (page 1)

"Solving these tasks requires writing rich code [...] and the ability to plan, handle failures, adapt dynamically, and learn from the interaction." (page 2)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Task definition: Completing realistic, everyday digital activities e.g. managing emails, playlists, finances, or shopping - by writing executable Python code that uses APIs to interact with simulated apps and data in a stateful environment.
Task item definition: Natural language instruction from a user (the “supervisor”), an environment state (apps, users, data, timestamp), an evaluation that checks whether the desired changes in the environment were achieved without unintended side effects.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Tasks are not collected from real users (but mainly written as template by the authors) and are designed to closely simulate realistic everyday digital tasks (e.g., doing shopping and ordering items, sending emails) in a safe, controllable environment. They are representative of how autonomous agents could be used in real digital settings, but not deployed on real accounts or data.
Dataset size: 585
Dataset size detail: difficulty level, number of apps used, number of API calls, number of solution code lines, evaluation test count, scenario grouping
Dataset size notes: train: 105, validation: 60
Dataset metadata available: Yes
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragraph, executable code), Extended interaction (e.g. conversation, calling an API and processing the response), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Scores are reported separately for Test-N (normal) and Test-C (challenge) sets. Performance is also broken down by difficulty level, number of APIs used, lines of code, and other task characteristics.
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Authors validate the benchmark using end-to-end “validation solutions” for each task, ensuring they are solvable under the defined evaluation framework. They also perform extensive unit tests on APIs (98% coverage) and programmatic evaluation using state diffs to ensure robust, construct-valid scoring.
Taxonomy: Agents / Coding
New taxonomy: General Application / Coding Agents