Title: OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Benchmark: OSWorld
Summary: OSWorld introduces a scalable, executable computer environment supporting real OSs (Ubuntu, Windows, macOS) to evaluate multimodal agents on 369 open-ended real-world tasks. It includes complex setups, execution-based evaluation, and detailed analysis of LLM/VLM agents' capabilities and deficiencies.
Contribution: They put significant emphasis on that they are "The first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across operating systems." <- with emphasis on the first
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multimodal tool use, open-ended task execution in real OS environments,  reasoning (though they don't mention word reasoning even once, the tasks implicitly require reasoning)
Phenomenon defined by authors: No
Phenomenon definition: The phenomenon is implicitly defined through the benchmark’s design and the types of tasks it includes. Authors describe the benchmark as evaluating agents’ ability to complete open-ended, real-world computer tasks using multimodal perception and actions (such as screenshots, accessibility trees, mouse/keyboard inputs) across various applications and operating systems.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Not defined
Task definition: Open-ended computer activity, which is described in natural language, executed by an agent inside a real operating system environment. Each task includes an initial state setup, a goal instruction, and a custom execution-based evaluation script to determine success.
Task item definition: "Each example is carefully annotated with a natural language instruction, a setup configuration with corresponding files and setup actions for initialization of initial states upon our provided VM image, and a manually crafted evaluation script to check if the task is successfully executed." (page 7)
Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Modified from another benchmark (e.g. translation into another language)
Task source detail: They sourced tasks from an incredible number of sources, diverse between each other - there is a table (appendix B.3) taking almost two pages to just list program sourced.
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Task ecology detail: The benchmark involves full workflows from A to Z - such as software installation, document editing, web navigation, and multi-application coordination. These tasks are executed in real OS environments (Ubuntu and Windows) and they use apps/programs (e.g. Chrome, LibreOffice, VLC), to operate as general-purpose assistants.
Dataset size: 412
Dataset size detail: "OSWORLD benchmark [...] encompasses 369 real computing tasks defined and executed on Ubuntu. Additionally, we provide a set of 43 tasks for Windows built on the OSWORLD environment." (page 6)
Dataset metadata available: Yes
Dataset metadata detail: human difficulty, task feasibility, application domain, task type
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code), Extended interaction (e.g. conversation, calling an API and processing the response)
Metric definition: Exact Match (accuracy, F1, precision, recall), Execution-based evaluation scripts
Metric aggregation: Simple Mean
Metric statistics: Just simple mean, with occasional reporting of variance or distribution plots.
Metric subscores: Yes
Metric subscores detail: By application domain: OS, Office, Daily, Professional, Workflow
By OS: Ubuntu or Window
By task difficulty: Easy, Medium, Hard (based on human completion time)
By feasibility: Feasible and Infeasible tasks
By input modality: Screenshot, Accessibility Tree, SoM etc.
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Author validity detail: 1. Human Baseline
They conducted human evaluations across all tasks, showing that humans (without prior expousure to the tasks) achieved +/- 72.36% accuracy, while top models performed under 12.24%, showing that the tasks are achievable.

2. Realistic Task Design
The tasks are based on real-world scenarios, sourced from impressively large number of sources: from user forums, tutorials, and many others, to everyday computer workflows.

3. Execution-Based Evaluation
They designed 134 custom, deterministic evaluation scripts to assess functional correctness and objective and reproducible scoring.

4. Model Performance Analysis
The authors analysed how models fail - such as difficulty with GUI grounding, interaction noise, and poor generalisation across applications - they tried aligning that observed performance with the skills they intended to measure through the benchmark.

4. Comparative Difficulty
They compare OSWorld to other benchmarks like WebArena and show that OSWorld tasks take longer for humans to complete and are harder for models to solve to support the idea that OSWorld includes more complex tasks, which are supposed to be closer to the real-world abilities.
Taxonomy: Agents / Tool Use
New taxonomy: General Application / Tool Use