Spider2-V: How Far Are Multimodal Agents From Automating Data Science
  and Engineering Workflows?
Authors: name: Ruisheng Cao; name: Fangyu Lei; name: Haoyuan Wu; name: Jixuan Chen; name: Yeqiao Fu; name: Hongcheng Gao; name: Xinzhuang Xiong; name: Hanchong Zhang; name: Yuchen Mao; name: Wenjing Hu; name: Tianbao Xie; name: Hongshen Xu; name: Danyang Zhang; name: Sida Wang; name: Ruoxi Sun; name: Pengcheng Yin; name: Caiming Xiong; name: Ansong Ni; name: Qian Liu; name: Victor Zhong; name: Lu Chen; name: Kai Yu; name: Tao Yu
Published: 2024-07-15 17:54:37

Abstract: Data science and engineering workflows often span multiple stages, from
warehousing to orchestration, using tools like BigQuery, dbt, and Airbyte. As
vision language models (VLMs) advance in multimodal understanding and code
generation, VLM-based agents could potentially automate these workflows by
generating SQL queries, Python code, and GUI operations. This automation can
improve the productivity of experts while democratizing access to large-scale
data analysis. In this paper, we introduce Spider2-V, the first multimodal
agent benchmark focusing on professional data science and engineering
workflows, featuring 494 real-world tasks in authentic computer environments
and incorporating 20 enterprise-level professional applications. These tasks,
derived from real-world use cases, evaluate the ability of a multimodal agent
to perform data-related tasks by writing code and managing the GUI in
enterprise data software systems. To balance realistic simulation with
evaluation simplicity, we devote significant effort to developing automatic
configurations for task setup and carefully crafting evaluation metrics for
each task. Furthermore, we supplement multimodal agents with comprehensive
documents of these enterprise data software systems. Our empirical evaluation
reveals that existing state-of-the-art LLM/VLM-based agents do not reliably
automate full data workflows (14.0% success). Even with step-by-step guidance,
these agents still underperform in tasks that require fine-grained,
knowledge-intensive GUI actions (16.2%) and involve remote cloud-hosted
workspaces (10.6%). We hope that Spider2-V paves the way for autonomous
multimodal agents to transform the automation of data science and engineering
workflow. Our code and data are available at https://spider2-v.github.io.