OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse
  Task Synthesis
Authors: name: Qiushi Sun; name: Kanzhi Cheng; name: Zichen Ding; name: Chuanyang Jin; name: Yian Wang; name: Fangzhi Xu; name: Zhenyu Wu; name: Chengyou Jia; name: Liheng Chen; name: Zhoumianze Liu; name: Ben Kao; name: Guohao Li; name: Junxian He; name: Yu Qiao; name: Zhiyong Wu
Published: 2024-12-27 16:21:58

Abstract: Graphical User Interface (GUI) agents powered by Vision-Language Models
(VLMs) have demonstrated human-like computer control capability. Despite their
utility in advancing digital automation, a critical bottleneck persists:
collecting high-quality trajectory data for training. Common practices for
collecting such data rely on human supervision or synthetic data generation
through executing pre-defined tasks, which are either resource-intensive or
unable to guarantee data quality. Moreover, these methods suffer from limited
data diversity and significant gaps between synthetic data and real-world
environments. To address these challenges, we propose OS-Genesis, a novel GUI
data synthesis pipeline that reverses the conventional trajectory collection
process. Instead of relying on pre-defined tasks, OS-Genesis enables agents
first to perceive environments and perform step-wise interactions, then
retrospectively derive high-quality tasks to enable trajectory-level
exploration. A trajectory reward model is then employed to ensure the quality
of the generated trajectories. We demonstrate that training GUI agents with
OS-Genesis significantly improves their performance on highly challenging
online benchmarks. In-depth analysis further validates OS-Genesis's efficiency
and its superior data quality and diversity compared to existing synthesis
methods. Our codes, data, and checkpoints are available at
https://qiushisun.github.io/OS-Genesis-Home/{OS-Genesis Homepage}.