Title: Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
Benchmark: Spider2-V
Summary: This paper introduces Spider2-V, a benchmark that evaluates multimodal agents on end-to-end data science and engineering workflows by requiring them to write code and perform GUI actions across professional enterprise applications. Tasks are implemented in a real-time, executable virtual environment and derived from real-world tutorials, with custom setup and evaluation functions. It tests agents’ ability to ground actions in visual and textual observations and leverage documentation through retrieval-augmented generation.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: code writing, managing data and engineering workflows
Phenomenon defined by authors: Yes
Phenomenon definition: The ability of a multimodal agent to perform data-related tasks by writing code and managing GUI in enterprise data software systems, including data warehousing, data ingestion and integration, data transformation, data analysis and visualisation, and data orchestration.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task requires agents to complete realistic, multi-step data science and engineering workflows in a live virtual desktop environment by interacting with GUI elements and writing code.
Task item definition: Each task instance is represented as a JSON dictionary which contains the following fields: 
• id: globally unique id of the current task example.
• instruction: the task instruction which indicates the task goal.
• source: a list of referenced tutorial links to construct the current task.
• config: a list of dictionaries which defines the sequential operations to initialize and reset
the computer desktop. Each dictionary contains the function name (the “type” key) and its
parameters (the “parameters” key), indicating one environment setup function
• related_apps: a list of application names which should be used in the current task.
• tags: a list of tags denoting different categories for this task.
• evaluator: a dictionary containing 3 fields: func, result, expected. It defines how to
evaluate the final results once task completion.
(citation from appendix)
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: It is partial because it simulates realistic workflows and tool interactions in a controlled environment, rather than being a full, unconstrained deployment in a live operational setting.
Dataset size: 494
Dataset metadata available: Yes
Dataset metadata detail: difficulty level, interface type (CLI, GUI, CLI+GUI), number of required actions/apps, real account required (yes/no)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Extended interaction (e.g. conversation, calling an API and processing the response), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), execution-based verification, file-based comparison, information-based validation
Metric definition detail: Evaluation methods include file-based comparison, information-based validation (extracting/checking information using APIs or UI navigation), and execution-based verification (running a script and checking status/logs).
Metric aggregation: Simple Mean
Metric statistics: Simple mean/sum (Success Rate %)
Metric subscores: Yes
Metric subscores detail: By task categories (data warehousing, transformation, ingestion, visualisation, orchestration, traditional data processing, IT service management), difficulty levels (Easy, Medium, Hard based on action steps), tasks with/without authentic user accounts, and interface types (CLI, GUI, CLI+GUI).
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Authors argue that the benchmark's realism, grounding in authentic tasks, and robust evaluation logic offer a valid measure of agent performance.
Taxonomy: Agents / Coding
New taxonomy: General Application / Coding Agents