AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials

Authors: Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, Tao Yu
Primary area: foundation or frontier models, including LLMs
Keywords: data synthesis, gui agent, large language model

Abstract:
Graphical User Interface (GUI) agents hold great potential for automating complex tasks across diverse digital environments, from web applications to desktop software. However, the development of such agents is hindered by the lack of high-quality, multi-step trajectory data required for effective training. Existing approaches rely on expensive and labor-intensive human annotation, making them unsustainable at scale. To address this challenge, we propose AgentTrek, a scalable data synthesis pipeline that generates high-quality web agent trajectories by leveraging web tutorials. Our method automatically gathers tutorial-like texts from the internet, transforms them into task goals with step-by-step instructions, and employs a visual-language model (VLM) agent to simulate their execution in a real digital environment. A VLM-based evaluator ensures the correctness of the generated trajectories. We demonstrate that training GUI agents with these synthesized trajectories significantly improves their grounding and planning performance over the current models. Moreover, our approach is more cost-efficient compared to traditional human annotation methods. This work underscores the potential of guided replay with web tutorials as a viable strategy for large-scale GUI agent training, paving the way for more capable and autonomous digital agents.

Decision: accept (spotlight)
Reviewer scores: 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730642281632

thread_type:
official_review

<thread_object>
creator: reviewer_safq

<content>
summary:
The AgentTrek framework introduces a scalable pipeline for generating high-quality GUI agent trajectory data by utilizing web tutorials. It automates the collection of tutorial-like instructions from the internet, transforms them into structured tasks, and uses a visual-language model agent to simulate and execute these tasks in real digital environments. An evaluator model verifies the generated trajectories to ensure accuracy, reducing reliance on labor-intensive human annotations. Experimental results show that models trained with AgentTrek-generated data outperform those trained on existing datasets, especially in task planning and GUI grounding. This framework offers a cost-effective, automated solution for training GUI agents on a large scale.

strengths:
The AgentTrek framework leverages online tutorials to automatically generate high-quality GUI agent trajectory data, automating the data generation process and reducing the need for manual data collection. Through an evaluator model, it verifies the generated trajectories, with a multi-layered generation and evaluation mechanism to ensure data quality and effectiveness.

Agents trained on AgentTrek-generated data perform exceptionally well in several benchmark tests (such as ScreenSpot and Multimodal-Mind2Web), especially in task planning and GUI element recognition, significantly outperforming traditional datasets.

weaknesses:
Certain technical details, such as automatic labeling and tutorial filtering, are only briefly mentioned in the paper, lacking more comprehensive explanations.

The paper notes that the success rate of generating effective trajectories is only 39.1%, based on GPT-4o Mini. Although GPT-4o Mini is relatively cost-effective, achieving larger-scale data generation with the current success rate remains challenging. There should be some indications if experiments are conducted with alternative open-source models to assess the feasibility and effectiveness of data construction within this framework.

Another consideration is the broader generalization of the framework for more complex computer control tasks. Many tasks may lack corresponding web tutorials or have very limited resources, especially those requiring highly precise and complex control, which will also lead to the data/categories bais issue. Do you have any thoughts or attemptions on these situations?

questions:
See the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643163314

thread_type:
official_review

<thread_object>
creator: reviewer_fph1

<content>
summary:
The paper introduces AgentTrek, a data synthesis pipeline that generates web agent trajectories from online tutorials. It automatically collects tutorials, converts them into task sequences, and uses VLMs to simulate and evaluate these tasks. Results show that agents trained with AgentTrek data perform better in task grounding and planning than those using traditional datasets, providing a cost-effective solution for large-scale web agent training.

strengths:
A clear pipeline for generating complex agent trajectories from web tutorials.

weaknesses:
1. In the Mind2Web experiment, it is clear that there is overlap between the training data and those in Mind2Web-test, with likely overlap in tasks, websites as well as domains. This overlap should be clarified, as it undermines the intended out-of-domain evaluation of Mind2Web.
2. The total number of trajectories remains limited, staying within the same scale as Mind2Web (in thousands). And the effectiveness is not good enough compared to the training data of Mind2Web (when the data size is 4x).
3. The comparison on grounding is not very meaningful, as performance significantly lags behind recent work focused specifically on grounding. (And empirically, a significant gain is from changing the backbone to Qwen 2 VL, compared to SeeClick and CogAgent.)
4. The web results on ScreenSpot, especially for icons, are not strong, which raises further questions about possible overfitting to specific websites or tasks in Mind2Web.
5. Some writing issues:  
   - baseline results should clearly indicate their sources
   - duplicate entries in the reference
   - wrong reference
   - more details of the evaluation on Mind2Web should be provided. (This is not minor. As there are huge differences with respect to the settings in table 6)
   - the synthesized data is essentially web agent trajectories. No need to always overclaim it to GUI agent trajectory. It only confuses people.

questions:
1. The overlap issue to Mind2Web.
2. Is there any other way to support the effectiveness of the synthesized data?
3. If possible, show the effectiveness of the synthetic data when it is further scaled up.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718821606

thread_type:
official_review

<thread_object>
creator: reviewer_mdgs

<content>
summary:
1. The paper introduces AgentTrek, a scalable data synthesis pipeline that generates high-quality GUI agent trajectories
by leveraging web tutorials.
2.  The method collects web tutorials from the internet, transforms them into structured task goals with step-by-step instructions, and uses a visual-language model (VLM) agent to simulate their execution in a digital environment. ​ A VLM-based evaluator ensures the correctness of the generated trajectories.
3. The paper provides experimental results and analayis showing that agents trained with the synthesized data outperform those trained on existing datasets in both grounding and planning capabilities.
4. The authors emphasize that the method is more cost-efficient than traditional human annotation methods, making it a practical solution for large-scale GUI agent training

strengths:
1. It is a novel pipeline that leverages web tutorials to synthesize high-quality GUI agent trajectory data at scale. This is a valuable contribution to the field, addressing the scarcity of reliable and scalable trajectory data. The proposed pipeline significantly reduces the cost of data collection compared to traditional human annotation
2. The paper is well-structured and clearly explains the steps involved in the data filtering
3. The dataset is comprehensive, containing a wide range of task types, platforms, and web environments.

weaknesses:
1. The paper does not provide a baseline to compare how using trajectory data compares with just using textual data. It would be beneficial to see how much the different elements( DOM/HTML structures, AXTree data, intermediate reasoning steps, full video recordings, and corresponding screenshots for each action) contribute to the dataset effectiveness.
2. The paper deals with only web-based tutorials and shows evaluation on only 2 benchmarks. It would be beneficial to expand the evaluation by including additional benchmarks such as MiniWob.
3. The paper lacks an analysis of potential failure cases. For example, is the trajectory data still as effective when the number of steps increase.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732562535131

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updated Manuscript and Response to All Reviewers

comment:
We sincerely thank the reviewers for their thoughtful and constructive feedback, which has been invaluable in improving our work. We are particularly encouraged by the recognition of AgentTrek's ability to scalably synthesize high-quality agent trajectories without requiring human annotators. In particular, we appreciate the specific acknowledgments from reviewers: 

- **`MDGs`**, **`fPH1`**, and **`SAfQ`** for highlighting the scalability and cost-effectiveness of our data synthesis pipeline and its significant impact on improving agent performance.  
- **`MDGs`** for emphasizing the extensive diversity of our trajectories across multiple domains and task types.

We are delighted to share that we have further scaled and verified over **10,000 agent trajectories**, which we believe constitutes the **largest web trajectory dataset** available. This growing dataset includes multi-modal resources such as screenshots, HTML/AXTREE structures, videos, and intermediate reasoning processes, all of which we plan to release publicly. We are confident that these contributions will offer valuable resources to advance GUI agent research. 

Based on the valuable feedback, we have addressed all concerns in our manuscript and added comprehensive details and explanations in our Appendix (updates are shown in purple for clarity):

- **Scaling Study:** Expanded AgentTrek to a **10K trajectory dataset** and conducted new studies demonstrating the effectiveness of scaling on agent performance.  
- **Out-of-Domain Generalization:** Conducted an analysis of AgentTrek trajectory overlap with the Mind2Web evaluation dataset to confirm its performance in out-of-domain scenarios.  
- **More Benchmark Validation:** Verified AgentTrek's effectiveness on **MiniWoB++** and a self-hosted realistic ** WebArena**, showcasing the generalizability of our synthesized data in textual modalities.  
- **Pipeline Details:** Provided additional technical details about the AgentTrek pipeline, including expanded explanations of (pre)-filtering and evaluation processes.  

We believe these updates further underscore the potential of AgentTrek as a scalable trajectory synthesis pipeline for advancing GUI agent research. We hope the revised submission meets your expectations and demonstrates the value of our contributions.

Thank you for your constructive feedback and support!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The reviewers are overall positive with the work. The authors contributed AgentTrek, a system for synthesizing web agent trajectory data from online tutorials. The authors propose a pipeline to collect tutorials, convert them into structured tasks, and use VLMs to simulate and evaluate these tasks. Results show that agents trained with AgentTrek data perform better in task grounding and planning. That said the reviewers raised several concerns. Reviewer fPH1 is concerned about potential overlap between AgentTrek data and the Mind2Web benchmark, which could undermine the out-of-domain evaluation. The reviewers questioned the grounding performance and wanted more evidence that the performance gain. In addition, the reviewers raised the concerns such as overfitting and generalization to complex tasks, and asked a number of questions regarding technical details. Overall, the reviewers and authors engage in a constructive discussion. The reviewers raise valid concerns, and the authors adequately address them. The paper presents a promising approach to synthesizing web agent trajectory data, and the proposed pipeline has the potential to be a valuable tool for training web agents.

Additional references

Mining tasks from web tutorials reminded me of the work by Li et al. ACL 2020 "Mapping Natural Language Instructions to Mobile UI Action Sequences"
and the most recent work on learning by reflection by Wang et al.  "Devil's Advocate: Anticipatory Reflection for LLM Agents", EMNLP 2024. These works should be discussed in the revision.