VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos
Authors: name: Dunjie Lu; name: Yiheng Xu; name: Junli Wang; name: Haoyuan Wu; name: Xinyuan Wang; name: Zekun Wang; name: Junlin Yang; name: Hongjin Su; name: Jixuan Chen; name: Junda Chen; name: Yuchen Mao; name: Jingren Zhou; name: Junyang Lin; name: Binyuan Hui; name: Tao Yu
Published: 2025-10-22 11:25:48

Abstract: Training computer-use agents requires massive amounts of GUI interaction
data, but manually annotating action trajectories at scale is prohibitively
expensive. We present VideoAgentTrek, a scalable pipeline that automatically
mines training data from publicly available screen-recorded videos at web
scale, eliminating the need for manual annotation. Our approach addresses a key
challenge: raw videos contain implicit demonstrations but lack explicit action
labels. To solve this, we develop Video2Action, an inverse dynamics module
(IDM) with two components: (1) a video grounding model that detects and
localizes GUI actions with precise temporal boundaries and context, and (2) an
action-content recognizer that extracts structured parameters like click
coordinates and typed text with high fidelity. Applied to 39,000 YouTube
tutorial videos, our pipeline generates 1.52 million interaction steps
automatically. We leverage this data through continued pretraining followed by
supervised fine-tuning. On OSWorld-Verified, our approach improves task success
rates from 9.3% (SFT-only baseline) to 15.8%, a 70% relative improvement. On
AgentNetBench, step accuracy increases from 64.1% to 69.3%. Our results
demonstrate that passive internet videos can be transformed into high-quality
supervision for computer-use agents, providing a scalable alternative to
expensive manual annotation.