AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web
  Tutorials
Authors: name: Yiheng Xu; name: Dunjie Lu; name: Zhennan Shen; name: Junli Wang; name: Zekun Wang; name: Yuchen Mao; name: Caiming Xiong; name: Tao Yu
Published: 2024-12-12 18:59:27

Abstract: Graphical User Interface (GUI) agents hold great potential for automating
complex tasks across diverse digital environments, from web applications to
desktop software. However, the development of such agents is hindered by the
lack of high-quality, multi-step trajectory data required for effective
training. Existing approaches rely on expensive and labor-intensive human
annotation, making them unsustainable at scale. To address this challenge, we
propose AgentTrek, a scalable data synthesis pipeline that generates
high-quality GUI agent trajectories by leveraging web tutorials. Our method
automatically gathers tutorial-like texts from the internet, transforms them
into task goals with step-by-step instructions, and employs a visual-language
model agent to simulate their execution in a real digital environment. A
VLM-based evaluator ensures the correctness of the generated trajectories. We
demonstrate that training GUI agents with these synthesized trajectories
significantly improves their grounding and planning performance over the
current models. Moreover, our approach is more cost-efficient compared to
traditional human annotation methods. This work underscores the potential of
guided replay with web tutorials as a viable strategy for large-scale GUI agent
training, paving the way for more capable and autonomous digital agents.