AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials
Authors: Yiheng Xu
Dunjie Lu
Zhennan Shen
Junli Wang
Zekun Wang
Yuchen Mao
Caiming Xiong
Tao Yu
Venue: ICLR 2025 Spotlight (2025)
Keywords: Data Synthesis
GUI Agent
Large Language Model
Primary area: foundation or frontier models, including LLMs

Abstract: Graphical User Interface (GUI) agents hold great potential for automating complex tasks across diverse digital environments, from web applications to desktop software. However, the development of such agents is hindered by the lack of high-quality, multi-step trajectory data required for effective training. Existing approaches rely on expensive and labor-intensive human annotation, making them unsustainable at scale. To address this challenge, we propose AgentTrek, a scalable data synthesis pipeline that generates high-quality web agent trajectories by leveraging web tutorials. Our method automatically gathers tutorial-like texts from the internet, transforms them into task goals with step-by-step instructions, and employs a visual-language model (VLM) agent to simulate their execution in a real digital environment. A VLM-based evaluator ensures the correctness of the generated trajectories. We demonstrate that training GUI agents with these synthesized trajectories significantly improves their grounding and planning performance over the current models. Moreover, our approach is more cost-efficient compared to traditional human annotation methods. This work underscores the potential of guided replay with web tutorials as a viable strategy for large-scale GUI agent training, paving the way for more capable and autonomous digital agents.

Published as a conference paper at ICLR 2025
AGENTTREK: AGENT TRAJECTORY SYNTHESIS VIA GUIDING REPLAY WITH WEB TUTORIALS
Yiheng Xu
Dunjie Lu
Zhennan Shen
Junli Wang
Zekun Wang*
Yuchen Mao
Caiming Xiong
Tao Yu
University of Hong Kong
* Salesforce Research
1yhxu, @cs hku... hk wexiong@@alessorrcee
attps://agenttre.. git.uu......t...........
ABSTRACT
Graphical User Interface (GUI) agents automate complex tasks across digital environments, but their development is hindered by the scarcity of high-quality trajectory data for training. Existing approaches rely expensive human anno- tation, making them unsustainable at scale. We propose AgentTrek, a scal- able data synthesis pipeline that generates web agent trajectories by leveraging publicly available tutorials. Our three-stage method: (1) automatically harvests and filters tutorial-like texts from the internet using specialized classification model, (2) transforms these texts into structured task specifications with step-by- languaa model (VLM) agent to exe- cute these instructions in real environments, while VLM-based evaluator verifies trajectory correctness. The synthesized trajectories encompass multiple modali- ties, including text-based HTML observations with function-calling API actions, and vision-bases screenshot observations with pixel-level actions. This multi- modal data, enriched with chain-of-thougle reasoning, enables agents to achieve state-of-the-ard performance on both textual web browsing benchmarks (e.g.. We- bArena) and visual web grounding and browsing benchmarks (e.g.. ScreenSpot Web and Multimodal Mind2Web). Furthermore, our fully automated approach significantly reduces data collection costs, achieving cost of just $0.55 per high- quality trajectory without human annotators. Our work demonstrates that guided replay using web tutorials is practical and scalable strategy for training advanced GUI agents, paving the way for more capable and autonomous digital assistants.
step instructions, and (3) employs
INTRODUCTION
Graphical User Interfaces (GUIs) serve as the primary medium for human-computer interaction across digital platforms. Automating these inter- faces through intelligent agents promises significant productivity gains by enabling autonomous tasks completion using human-centric tools. This automa- tion also creates opportunitics for AI systems learn from interactive digital environments.
High Level Goal
description that provides overview primary objective the tutorial.
Observation
Complete, process, examples such as stop wise matehing image, video recording, and environment changes (e-g.. HTML updates).
Low Level Instruction
Recent advancements in large language models (LLMs) have demonstrated remarkable capabilities in understanding, reasoning, and decision-mand makkin skills for GUI agents operating across web (Zheng al.) 2024), desktop (Xie et al.] 2024), and mo- bile applications (Zhang et a 2023). Despite these breakthroughs, GUI agents still perform subopti- mally in real-world scenarios. This limitation stems from a fundamental mismatch: contemporary
A detalled step-by-ste by--teppppppppp instruction that provides the reasoring thought behind each grounding action.
Action Command
interaction performed by agent during singlo stop esecution the loop.
precise reproducible
Figure Our Agent Trajectory Schema
Equul contribution

---

Published as a conference paper at ICLR 2025
Treedim Secoedyy
Envegrt to Toael Patia
LLM ansstaaod
Taak descripton:
We Tuowial
Toom Jeril Enuag breee diww
Clsssllir
Vntraction:
Fal?
Pastuatrr
pistess reployy hoss
Snapshoot of the Intaree.
Tutoriala Ranad from bnternet
Oodd rapley with web sutorial in ene
model
ovrr ganera tonts
Part T: Automatic tutoriaas collection from Internet
Part 2: Trajectory data collection via guided replay
Part TrainJFT the model with replay data
Figure 2: Overview of the AgentTrek Pipeline: Our three-stage approach consists of: (1) Au- tomatic Tutorial Collection: We extract and filter tutorial data from internet sources using heuristic methods and FastText classifier, then transform raw text into structured tutorials using LLMs. (2) Guided Replay for Trajectory Generation: A VLM agent executes these tutorials in real web en- vironments while we collect high-quality trajectory data (observations, reasoning, and actions). A separate VLM evaluator assesses trajectory quality to ensure effectiveness. (3) Model Training: The collected trajectories are used to train and fine-tune GUT agent models, which demonstrate signifi- cant performance improvements across standard benchmarks.
LLMs are primarily trained on datasets optimized for generating informative responses (Oyang et al. 2022; OpenAl (2024), not for making the complex, sequential decisions required for GUI in- teraction. These decisions demand long-term observation. historical context integration, and precise action grounding grounding-capabillties that require specialized training with multi-step trajectory data.
High-quality agent trajectories comprise several critical elements: high-level goal, interleaved observations, natural language reasoning, and grounded actions (as shown in Figure [ Unlike text or images, such data is scarce online because requires complex situational reasoning and multimodal interactivity. Current approaches predominantly rely on human annotation to collect these trajectories (Deng et al. 2024 Rawles et al. 2023 et al. 2024 a process that both
costly and inherently unscalable.
To address this data scarcity challenge, data synthesis has emerged as a vital approach in AI de- velopment. However, synthesizing agent trajectories presents unique difficulties due to the need for seamlessly integrated natural language instructions, visual observations, and context-specific ac- tions that must be accurately grounded in GUI environments. While LLMs have shown promise in data synthesis pipelines Ye et al.] 2022 Peng et al., 2023; Qin et al. 2023), the multimodal and interactive nature of GUI trajectory synthesis remains particularly challenging.
In this work, we present TgentTrek, a scalable GUT agent trajectory synthesis pipeline. Our ap- proach begins by automatically harvesting and filtering tutorial-like text from the web that describes GUI tasks and workflows. These tutorials are then transformed into structured agent tasks with high-level objectives and detailed step-by-step instructions. Using visual-language model (VLM) agent, we execute these tasks in real environments, guided by the synthesized tutorials. An evaluator model subsequently verifies goal achievement, ensuring data quality. Through this comprehensive pipeline, we efficiently generate a large corpus of high-qualit, web agent trajectories.
Our experimental results demonstrate that models trained with synthesized trajectories significantly improve performance. Compared to traditional human-annotated data pipelines, our method is more cost-effective, highlighting the scalability and economic viability of the AgentTrek approach.
We introduce TrentTrek, fully automated pipeline that transforms web tutorials into high- quality agent trajectories at scale, bridging the critical gap between LLM capabilities and the complex, multi-step training data required for effective agents.
Our comprehensive experiments demonstrate that agents trained with AgentTrek's synthe- sized data significantly outperform those trained on existing datasets across multiple bench- marks, showing marked improvements in both textual and visualweb browsing capabilities.
We achieve a cost-per pertaper-rraectorryy of just $0.551 while maintaining high data quality, establishing new paradigm for scalable, cost-effective GUI agent training data synthesis.
2

---

Published as a conference paper at ICLR 2025
Table I: Comparison of AgentTrek with other trajectory datasets for training, For the calcula- tion of dataset size and average steps, see Appendix A
Average Steps
Intermediate Reasoning
Matching Screenshot
Task Inst.
Datasets
Size
HTML
AxTrree
Video
Website
Level
RUSS
Yes
No
No
No
No
Low
ScreenAgent
No
No
Yes
No
Yes
High
WebLINX
Yes
No
No
Yes
High Low
MM-Mind2Web
7.3
Yes
No
No
No
High
GUIAct
No
No
Yes
High
AgentTrek (Ours)
10398
12.1
Yes
Yes
Yes
Yes
Yes
127
High & Low
METHOD
We present Agent,Trek, comprehensive pipeline for collecting and synthesizing high-quality agent trajectories from web tutorials. Our approach addresses the critical challenge of training data scarcity for GUI agents through a systematic three-stage process (Figure 2
1. Automatic Tutorial Collection and Processing: We harvest web interaction tutorials from large-scale internet corpora, applying multi-stage filtering and standardization to identify and structure relevant content.
2. Guided Replay for Trajectory Synthesis: We deploy VLM agent to execute these struc- tured tutorials in real web environments, recording multimodal observations, reasoning chains, and actions create comprehensive trajectories.
3. Agent Model Training: We leverage the synthesized trajectories train both text-based and vision-based web agents, enhancing their ability to navigate and interact with complex web interfaces.
This end-to-end approach enables efficient, scalable generation of high-quality training data without extensive human annotation, significantly reducing the cost and effort.
2.1 AUTOMATIC TUTORTALSS COLLECTION FROM INTERNET
illustrated in Figure 3 the first stage of our pipeline transforms vast amounts of web content into structured, high-quality tutorials suitable for agent training. We extract web interaction tutorials from the RedPajama dataset (Computer) 2023) through a four-step process: pre-filtering, LLM- based labeling, FastText classification, and standardization.
Task description:
Heuristic Prefilter
Annotated
Create customize
GPT4o verified
Google Calender.
Preliminary Filtered
Train
gold tutorials
prerequisite
Google account logged
Instructions:
visit Google
Click the gear lcon
Rephrased by LLM
Snapshot of the Internet
Classified raw text of tutorials
Tutorials Internet
Figure 3: Overview of the tutorial filtering and classification pipeline. Starting with RedPajama, the data undergoes pre- pre-filtering based on keywords and structural features. A subset then anno- tated by an advanced LLM to create training data for FastText classifier. This classifier further refines the dataset, after which the selected tutorials are transformed into standardized format with task descriptions, prerequisites, and step-by-step instructions.

---

Published as a conference paper at ICLR 2025
Rediaaaam 20.88
Engilish data 15.28
Prefiltered 68.8M
FastText Filtered 54.0M
Duplicate removed
Up-to-date deta
18.8M
10.2M
Domain Filter
Web 2.24M
FastText
Duplicate Filter
Date Filter
PrefiterrKeyworcc)
Prefilter(Content length)
English Filter
PrefilterUUR))
Figure 4: The data flow during the early stages of our pipeline.
2.1.1 PREFILTER FUNCTION
Although GUI tutorials are widespread online, they represent a small fraction of web content, neces- sitating efficient pre-filtering We developed rule-based filter identifying potential tutorials based on: (1) Keywors word Matching of action verbs (e.g., "click," " "type"), UI elements (c.g., "button," "menu"), and platform terms (e.g.. "macOS," "Windows">y " (2) Length Analysis with thresholds (200-5000 words) to exclude overly brief or lengthy content; and (3) URL Format Evaluation prioritizing domains with tutorial-related patterns (e.g., "how-to," "guide"). ""uidt",""guidd""
To validate effectiveness, we tested against 285 manually labeled samples. The pre-filter achieved 92.69% recall on positive samples while maintaining reasonable precision, reducing the dataset from 20.8 billion to 68.8 million entries as illustrated in Figure 4
2.1.2 LLM LABELER
While the pre-filter significantly reduces the data volume, the resulting 68.8 million entries still contain many non-tutorial texts (false positives). To further improve quality, we implemented an LLM-based labeling approach using GPT-40 MINI. This model analyzes text content and classifies it as either "tutorial" or "non-tutorial" based on structural and semantic features.
We evaluated the LLM labeler against human annotations on the validation set, where it achieved an F1 score of 88.5%. Interestingly. in cases of disagreement between human and LLM annotations, manual review revealed that the LLM often correctly identified tutorial content embedded within longer texts that human annotators had missed. This suggests that GPT-40 MINI may actually out- perform human annotators in certain aspects of tutorial identification, particularly when dealing with mixed-contern pages. The LLM labeler processed subset of 90,000 pre- pre-filtered entries, generat- ing high-confidence labels that served as training data for the next stage of our pipeline. Example prompts and classification criteria provided in Appendix G
2.1.3 FASTTEXT FILTER
To scale our classification approach the full dataset of 68.8 million entries, we trained a FastText model (Joulin al. 2017) on the LLM-labeled data. FastText was selected for its efficiency with large text corpora and its ability to handle out-o--vocabulary words through n-gram representations.
We constructed a training dataset from the LLM--lbeled data, which contains 90,000 samples. Us- ing a 95:5 train-test split, we trained the FastText model which achieved 89.5% Fl score on the validation set. The trained FastText classifier processed the entire pre-filtered dataset, identifying approximately 18.8 million deduplicated entries as likely tutorials. This represents a 72.7% reduc- tion from the pre-filee rreftiteeedd set while maintaining high recall of genuine tutorial content.
1.1 TUTORIAL STANDARDIZATION TAG & PARAPHRASE
Following FastText-based filtering, we implement a structured standardization process ensure tu- torial consistency and quality. We employ GPT-40 MINI to perform dual functions of semantic tagging and content paraphrasing, optimizing for both computational efficiency and standardiza- tion fidelity. The model extracts and organizes content according to predefined template struc- ture: Platform and Target Environment (specifying operating systems and software versions), Task Description (concise problem statement), Prerequisites (required dependencies and back- ground knowledge), Step-by-Step Instructions (procedural guidance with command syntax), and Expected Outcome (verification criteria and success indicators).

---

Published as a conference paper at ICLR 2025
Succees?
Pull video
Tatoriate
HTML
Teal Description
Pr
AXtree
Evatuator
Instructions
Past actions & thoughts
Observation
Detaked Thognes
1R3P
Action
VJb
portaeeit nell
BETTT Mon
gifaDen see)
Playwright
operation
Figure 5: Overview of the Guided Replay Pipeline. The process begins with a VLM agent re- ceiving step-by-step tutorials. The agent then observes and interacts with real web environments, generating actions based on both the tutorial instructions and its observations. Throughout execu- tion, all observations, actions, and intermediate reasoning recorded comprehensive trajectory data. A separate VI.M evaluator assesses the final result to ensure trajectory correctness and quality.
To ensure high-quality standardization, we fine-tuned the prompting strategy using gold-standard examples that exemplify ideal tutorial structure. This approach effectively manages tutorial length variability while preserving critical instructional content. The standardization pipeline processes tutorials with GPT-4o MINI at total cost of approximately $0.89 per 1,000 entries, demonstrating both computational and economic efficiency for scale tutorial processing.
2.2
TRAJECTORY DATA COLLECTION VIA GUIDED REPLAY
2.2.1 TRAJECTORY DATA SCHEMA
The trajectory data generated by our pipeline enhances an agent capabilities by integrating high-level planning with low-level instructions and grounded operations. Each trajectory instance comprises: (1) Task Information with comprehensive metadata including platform specifications, task descrip- tions, prerequisites, step-by-step instructions, and expected outcomes; (2) Screenshots and Video Recordings capturing the complete interaction sequence; (3) Reproducible Native Trace with de- tailed technical logs via Playwright, encompassing DOM snapshots, HTML structure, network flow, and precise action sequences; and (4) Pust-processed Trajectory structured as task metadata, ob- servations, intermediate reasoning, and action sequences for model fine-tuning.
2.2.2 GUIDED REPLAY WITH TUTORIALS
While our pipeline has successfully collected high-quality tutorials, a significant gap remains be- tween these instructional materials and the rich trajectory data needed to train effective agent mod- els. To bridge this gap. we implement a guided replay mechanism using BrowserGym (Drouin et al., 2024), enabling VLM agents execute tasks based on the standardized tutorials.
BrowserGym provides a flexible environment for web task automation within Chromium, allowing VLM agents to perform complex based operations (Drouin al., 2024). our implementation, agents receive tagged tutorials and targer_wad www.wwwbbeo. jurl, which serve comprehensive guides through multi-step tasks with explicit instructions and success criteria.
During execution, the agent's observations primarily consist of viewport screenshots and the acces- sibility tree (AXTreee), deliberately excluding the full HTML. structure due to excessive size and limited relevance for visual agents. The agent performs actions using Playwright's API functions such as click, select.option, and clear, while our system records comprehensive traces including target elements, precise coordinates, sequential sereenshots, and DOM snapshots all synchronized with the agent's 's documented reasoning process.
Our analysis shows that token usage averages approximately 8,027 per interaction step and 86, 86.114 per complete task. Executing 1,000 tasks with GPT-40-08- incurs a cost of approximately $215. Detailed cost analysis is provided in Appendix C
5

---

Published as a conference paper at ICLR 2025
Task: Find the return policy for any men's football apparel en Under Armour's website.
Thoughts Acter
start by navigating to the Under Armour wetsshto
Neak, hover over the Mart menu to be up the drepdocm
3. loroosed by clicking Bhins & Topo" from the dropdown:
cBe4/203)
continue,
that
1: Navigato to UA website
2-3: Go Shirts & Tops
5-6: Soloct sport
ckea72892)
chock detalts and find return pollege
cice/1998)
Tiliic Use
"Fru Betirnna Enchangas" buttio view the polliyy"
(be0/9929)
News, con onn nouum pollay information the pagei
Send Meg
8: Click return policy
Click item
wien 10:000
Figure 6: Guided replay example. This example demonstrates an agent's 's execution of finding the return policy for men's 's football apparel, showcasing its actions alongside the inner thoughts.
2.2.3 EVALUATION OF TRAJECTORY
While our guided replay mechanism generates substantial trajectory data, it is crucial to identify and extract the segments that most effectively enhance agent performance. Recent research by Pan et al. (2024) demonstrates that VLMs can effectively evaluate trajectory data by analyzing recorded images and interaction patterns. These VLMM-based evaluators offer significant advantages in scal- abilityy. cost-effectiveness, and evaluation transparency. Building this insight, developed specialized VLM Evaluator to systematically assess and filter our trajectory data.
VLM Evaluator Design. We define trajectory effectiveness according to primary criteria: (1) adherence to the specified task instructions and (2) successful completion of all key task compo- nents. Our evaluator employs GPT-40 as its core engine, assessing trajectories through carefully structured prompt. The evaluation process takes as input the task description d, the complete action history {aj, 022, .... an and the agent's inner reasoning ments are organized in a sequential format: {d, 71, 02, 02 ........ T', ), as illustrated in Figure The VLM evaluator provides comprehensive assessment at three levels: an overall trajectory- evaluation, a seep----tteppp analysis, and identification of the earliest point of failure when applicable.
{r1, 72, . These ele-
Table Evaluator Accuracy Comparison
Table 3: Cost Breakdown
Trajectory
Evaluator
Acc.
Phase
Cost/Ik (S)
Model
Replayed Web Tutorials
GPT-4o
84.0%
T&P
0.89
gpt-4o-mini
Replay
215.36
gpt-40
GPT-4V
80.6%
Eval
3.10
gp
WebArena Results
Cap. GPT-4
82.1%
Cap. + Mixtral
74.4%
Total 219.35
Validation on Couman-Annotatee Set. To rigorously validate our VLM evaluator's effectiveness, conducted comprehensive human review of 1,081 trajectories, creating gold-standard valida- tion set comprising 558 samples with detailed human-annotated justifications.
As shown in Table 2 our VLM evaluator achieved robust performance. Notably, detailed anal- ysis in Appendix [D] reveals that the VLM evaluator frequently applies more stringent evaluation criteria than human reviewers, demonstrating its reliability in identifying truly effective trajectories while maintaining a conservative filtering approach.
2.3 TRAINING WITH TRAJECTORY DATA
Agent autonomously collects thousands of multimodal trajectories. including screenshots, accessibility trees, reasoning chains, and detailed actions. This comprehensive data particularly well-suited for fine-tuning both text-based L.MSs and vision-based VL for web agent tasks.

---

Published as a conference paper at ICLR 2025
2.3.1 VISION-BASED WEB AGENT
The vision-based agent operates solely on visual input, eliminating dependency on underlying UI source code. This approach offers significant efficiency advantages: high-resolution models like Qwen2-VL process 720p screenshot using only 1,200 tokens, compared to approximately 4,000 tokens required for HTML representation. The agent's action space is implemented through pyau- togui commands, which directly interact with visual UI elements based on pixel coordinates. We develop systematic mapping from playwright actions to pyautogui commands and implement pluggable action system to handle specialized interactions such as se select.opt on operations, ensuring comprehensive coverage of web interaction patterns.
2.3.2 TEXT-BASED WEB AGENT
Our fext-bbsee agent leverages the accessibility tree (AXTree) as its primary observation source, pro- viding semantic understanding of web element relationships and properties. This representation enables the agent to comprehend hierarchical structures and element attributes without processing raw HTML. The agent executes actions through playwright commands, which offer precise control over web elements identified within the accessibility tree. This approach excels in scenarios re- quiring structured interaction with complex web components such as forms, dropdown menus, and nested navigation elements, where semantic understanding of element relationships is crucial.
2.3.3 MODEL ARCHITECTURE AND TRAINING
For vision-based agents, we employ Qwen2-VL Wang et al. 2024) with NaVIT as the image en- coder, which provides dynamic resolution support (Dehghani et al. 2023 This architecture effi- ciently processes visual information at varying resolutions, making it particularly well-suited for GUI tasks that require mapping user intents directly to visual elements. We fine-tune the model using 10,000 trajectories from the AgentTrek dataset, focusing on enhancing visual grounding capabilities and multi-step planning for complex web navigation tasks.
For text-based agents, we fine-tune Qwen2.5 LLMs (Qwen et al. 2025) at various parameter scales (7B and 32B) using 6,000 agent trajectories from the AgentTrek dataset. These trajectories pair accessibility tree observations with corresponding playwright actions, creating a comprehen- sive training signal for web interaction. The fine-tuning process significantly enhances the model's ability to interpret structured web representations, reason about element relationships, and gener- ate contextually appropriate actions based on textual cues, resulting in improved task planning and execution capabilities across diverse web environments.
EXPERIMENTS
We demonstrate the effectiveness of our approach by evaluating agents trained on AgentTrek data across multiple established benchmarks.
3.1 EXPERIMENTAL SETUP
Text-based Web Agent Evaluation. To assess our text-based agent's capabilities, we use We- bArena (Zhou et al. 2023) our primary benchmark. WebArena simulates realistic web environ- ments based on actual websites, providing a comprehensive evaluation framework with multiple virtual environments and diverse assessment methods. This benchmark is particularly suitable for evaluating real-world task completion capabilities due to its focus on practical web interactions.
Vision-based Web Agent Evaluation. To validate the effectiveness of our dataset for vision-based agents, we evaluate performance improvements on two benchmarks. First, ScreenSpot (Cheng et al. 2024) provides GUI visual grounding benchmark containing 1,200 instructions with target element bounding boxes across mobile, desktop, and web environments. We focus specifically on web-based performance to align with our dataset's '' domain. Second, Multimodal-Mind2Wais (Deng et al.] 2224 Zheng et al. (024) extends the Mind2Web benchmark to evaluate generalization across three increasingly challenging settings: cross-task, cross-website, and cross-domain.

---

Published as a conference paper at ICLR 2025
3.2 MAIN RESULTS
WebArena Results. As shown in Table E fine-tuning with AgentTrek's textual tra- jectories yields substantial performance im- provements. Models trained on Agent Trek data significantly outperform both open-source baselines and GPT-4o, demonstrating the high quality of our synthesized trajectories. The strong performance on WebArena-ar out-of- distribution benchmark featuring self-hosted websites not seen during training-confimms confirms that Agent Trek data enables robust general- ization to novel domains.
Table 4: Comparison of task success rate We- bArena
Model
WebArena
I.I 1aMa3--chat- (On et al. 2024
3.32
Qwen2,5-7B, (ree2.5-7BBInsttrill
3.80
LLamm3- chaaa----ha---00 70B (Ou et 2024
7.02
GPT-4o (Zhou et 2025
13.10
GPT- et 2024
Synatra-oodh Lade (On al 2024
6.28
AutoWebGLM (OOD SFTy (aaaar dra 2024
8.50
Owen2.5-7B-Inae w/ AgentTrek
10.46
Swen2.5-32B-Instee w/ AgentTrek
22.40
ScreenSpot Results. Fine-tuning Qwen2-VL with the AgentTrek dataset significantly im- proved visual grounding capabilities. Performance more than doubled across both text-based and icon-based tasks compared to the baseline model. The fine-tuned model surpassed several competi- tive baselines on the ScreenSpot benchmark, highlighting AgentTrek' effectiveness in enhancing GUI element localization and interaction.
Table 5: Comparison of grounding performance ScreenSpot Web Grounding
Model
Text
Icon/Widget
Average
GPT-4 CCeeg al. 2024
9.2
8.8
9.0
GPT-4o (Cheng ct al 2024)
12.2
7.8
10.1
Qwen2-VL-ih
35.2
25.7
30.7
SeeClick CCeng et al.
2024)
55.7
32.5
44.7
CogAgent (Cheng et 2024
70.4
28.6
50.7
GPT-4 OmniParser (L.u et al.
2024)
81.3
51.0
67.0
Qwen2-VL-7B w/ AgentTrek
81.7
51.5
67.4
Mind2Web Results. Our experiments on the Mind2Web reveal several important findings. The baseline Qwen2- VL-7B model was excluded from comparison due to its insufficient grounding pabilities, which are essential for visual web agent tasks. Training with AgentTrek data sig- nificantly enhanced model performance and the combination of AgentTrek with Mind2Web training data yielded the strongest results across all metrics, demonstrating complementary ben- efits: AgentTrek provides grounded interaction data, while Mind2Web contributes specialized resources for complex web tasks. Additional details result sources are in Appendix T.2 Table 6: Performance comparison across different methods and evaluation settings. H. 'AT, "M2W" stand for HTML, Image, AgentTrek, Mind2Web
Obs
Model
Method
Cross-Task
Cross- Pross-Domain
Ele.Ace
Op.F1
Step SR
Ele.Ace
Op.F1
Step SR
Ele.Ace
Op.F1
Step SR
HTML
GPT-3,5 GPT-4
Choice Choice
14.9
56.5
14.1
40.8
63.1
32.3
30.2
61.0
27.0
35.4
61.9
29.7
GPT-4 GPT-4
Choice
46.4
73.4
40.2
38.0
67.8
32.4
42.4
69.3
36.8
H+I
Qwen2-Vi VL
Image
+ AT
Vision
45.5
84.9
40.9
40.8
82.8
35.1
48.6
84.1
42.1
M2W
Vision
89.5
52.9
83,9
44,9
86.8
47.7
AT M2W
Vision
60.8
88.9
55.7
$7.6
88.1
51.4
$6.0
87.5
52.6
4 ANALYSIS
The AgentTrek pipeline produces high-quality trajectory data distinguished by three core strengths:
diversity, realism, and comprehensiveness. These attributes collectively enhance the dataset's 's utility for training GUI agents capable of handling complex, morgzoorzon tasks. Below, analyze these strengths in detail and compare approach with existing methods.
8

---

Published as a conference paper at ICLR 2025
Diversity and Scale Our dataset exhibits significant diversity, encompassing wide range of do- mains and task types. Starting with the RedPajama corpus, we filter 23,430 tutorials, ultimately yielding 10,398 successful trajectories across 127 websites and 11 task categories (e.g.. e-commerce, productivity, and knowledge navigation). This breadth ensures that agents trained on Agent data encounter varied scenarios, fostering robust generalization. Figure X illustrates the distribution of websites and domains, highlighting the dataset's 's extensive coverage.
Effectiveness of Data Scaling To further explore the benefits of scaling up synthetic data, we system- atically assessed performance gains using increasing proportions of the Agent Trek dataset. Evaluation Maltimodda-Minnd2We
Cress Detriis Peffernaxe Scaliss
on the challenging mark showed steady improvements with data scal- ing, as shown in Figure 7 In particular, the cross- domain step success rate improved from 39.5% (20% data) to 45.0% (100% data), clearly demon- strating that additional synthetic trajectories enhance model generalization to novel domains. Importantly, when compared against the human-annotaie annotated trajec- tory of the Mind2Web training split, which achieved a cross-domain metric of 47.7%, our fully automated AgentTrek dataset, even without human annota- tion, approaches similar levels of performance as it scales. This finding underscores that automated synthetic data generation is viable strategy for closing the performance gap with numan-labeled data, highlighting substantial potential for future scalability.
bench-
Step Stocess Ruze
44
Syathesited Trjee CAgeotTroks
Hurron Annorase Troj (MintWW0))
Amoostt
Figure 7: Scaling performance comparison with human-labeled data.
Realism through Authentic Environments A key advantage of ArentTrek is its reliance on real web environments during data collection. Unlike synthetic or simplified settings. our pipeline replays tutorials on live websites, capturing authentic interactions that mirror real-world complex- ities. This realism is critical for training agents that must navigate dynamic, unpredictable GUIs. Moreover, the use of internet-sourced tutorials enhances execution quality. In controlled experi- ment with 400 tasks, agents following detailed tutorial instructions achieved a 52% success rate (208 effective trajectories), compared to a 15.78% success rate (63 effective trajectories) when guided only by high-level goals- a 23% improvement. This underscores the value of structured guidance in producing actionable, high-fidelity trajectories (see Appendix B for details).
Comprehensiveness and Cost-Efficiency The AgentTrek dataset captures both strategic and operational details, including DOM/HTML structures, AXTree snapshots, video recordings, screen- shots, and intermediate reasoning chains. With an average of 12.1 steps per trajectory, this multi- modal data supports training on complex, multi-step tasks. Despite its richness, our fully automated pipeline achieves exceptional efficiency at $0.551 per trajectory. Table 3 provides a detailed cost breakdown, with further analysis in Appendix
Comparison with Existing Datasets Table benchmarks AgentTrek against prior work (Niul et al. 2024: Lu et al.] [2024] Deng et al. 2024 Yao et al., 2022, Song et al. 2024: Wornow et al., 2024 Our dataset, with nearly 5,000 verified trajectories, surpasses most in scale and step complex- ity. Its comprehensive modalíties Sotaaiites-sspannnng text, vision, and reasoning it apart from datasets like Mind2Web (Deng al.) 2024) and WetShop (Yao al. (022). which focus on narrower observation-action pairs. While fully automated, Agent Trek maintains diversity and realism, ad- dressing scalability limitations of human- -nmatttnd datasets a fraction of the cost.
Research Challenges Generating large-scale trajectory data poses challenges, including ensuring data quality, handling noisy web content, and adapting to evolving GUIs. Agent mitigates these through robust filtering (FastText and LLM classification), structured standardization, and evaluation. However, future work could explore dynamic adaptation to website changes and broader task coverage beyond the current 11 categories.

---

Published as a conference paper at ICLR 2025
Tour Bookinng 6.8%
Others 15.2%
Routine & Transportation
Sreeeee Ti
mi people
Entertalement Media 10.0%
Goverament Publicc 10.6%
Education & Information
E-Commerce Shopping
5.2%
Social Media Blogs 9.7%
Health
16.0%
1.7% Technology Devaloper Toolc
1.5%
Data
Enghas
Figure 8: Distribution of websites and domains in the AgentTrek dataset, showcasing its diversity
across 127 websites and 11 task categories.
5 RELATED WORK
5.1 LLM-BASED AGENTS
Large language models (LLMs) have driven significant progress in autonomous GUI agents, abling them to interpret natural language instructions and execute complex tasks across web (Nakano) al., 202T). desktop, and mobile environments (Zheng al. 2024; He al.] 2024). Projects like SeeAAt (Zheng et al. 2024) and WebVoyager (He et al. (2024) exemplity efforts to generalize agent behavior to real-world interfaces. However, these agents often struggle with sequential decision- making due to lack of specialized trajectory data. Agent addresses this gap by synthesizing multimodal trajectories that enhance L.L.Mbbased agents' navigation and interaction capabilities, ad- vancing their practical utility.
5.2 AGENT TRAJECTORY DATA
The growing adoption of GUI agents has intensified the need for scalable training data. Existing datasets and benchmarks, such WehArena (Zhou et al. (023). Mind2Web (Deng et. (024). and WebShop (Yao et al. 2022) rely heavily on human annotation, limiting their scale and adaptability. Recent efforts like BAGEL (Murty et al. 2024) and NNetNav (Murty et al., (025) propose synthetic trajectory generation to overcome these constraints. AgentTrek builds on this trend, offering fully automated pipeline that produces diverse, realistic trajectories
low cost ($0.551 per trajectory), outperforming human-dependent approaches in scalability and efficiency.
5.3 AUTOMATED EVALUATION OF DIGITAL AGENTS
Automated evaluation increasingly critical for assessing agent performance. Vision-language models (VLMs) and LLMs have emerged as powerful tools, analyzing trajectories the task level (Pan et al.) 2024) and step-by-step adherence to instructions (Wornow et al., 2024). These methods span diverse environments, including web platforms and mobile OS (Pan al. 2024). In AgentTrek, we leverage GPT-4o as a VLM evaluator, assessing trajectories based on task descrip- tions, actions, and reasoning chains. This approach ensures scalable, transparent quality control, aligning with state-of-the-ar practices while supporting our pipeline's automation goals.
6 CONCLUSION
In this work, we introduce AgentTrek, an efficient pipeline designed to automatically generate comprehensive and cost-effective agent trajectory data. Additionally, we present a large and diverse dataset generated using this approach, which validate by training models and evaluating their performance with promising result.Our research establishes a novel and promising direction for future development of LLM agent, particularly in the automatic and low-cost synthesis of trajectory data. Agent Trek serves as a strong standard for agent data generation, setting the stage for future advancements in this field.
10

---

Published as a conference paper at ICLR 2025
ACRNOWLEDGEMENTT
This paper's authors received support from the ECS (27212023) provided by the RGC of Hong Kong.
REFERENCES
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiy- ong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. ArXiv preprint, abs/2401.10935,, 2024. URL https://axxiv.orrg wttp:/iwwwrr/bb////a//aa/ wttp///a////...103/..3
Together Computer. Redpajama: an open dataset for training large language models, 2023. URL https://github cttm///oggeteeerroopeeeeeeeeeeee a ama Data.
Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim Alabdulmohsin, Avital Oliver, Piotr Padlewski, Alexey Gritsenko, Mario Lučič, and Neil Houlsby. Patch n' pack: Navit, vision transformer for any aspect ratio and resolution, 2023. URL https::/arxiv.ont
abs/2307. 06304
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam Hadj Laradji, Manuel Del Verme, Tom Marty, L'eo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks? ArXiv preprint, abs/2403.07718, 2024. URL http:/aaa/aa..v.oog///o/a//////a http://wwwww..orr/////////////// 07718
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. ArXiv preprint, abs/2401.13919, 2024. URL httthhttp////t /ttp:::///aavvaaaaeeeeee borg/abss 2401 13919
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Volume 2, Short Papers. Association for Computational Linguisties, 2017. URL https://: http://claaaahloccoooooooooo wowhhhlooy...o org/ E1 2068
Hanyu Lai, Xiao Liu, lat Long long, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Autowebglm: A large language model-based web navigating agent, 2024.
03648
Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Camphell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents, 2024. URL https:
23667 03679
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent, 2024. URL httpt:t////aaa. httpa/////xxxxxxxx/aaae/ abag///b//at
aataabs/244.. /2008 00203
Xing Han Lå, Zdeněk Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi- turn dialogue, 2024.
Shikhar Murty, Christopher Manning, Peter Shaw, Mandar Joshi, and Kenton Lee. Bagel: Boot- strapping agents by guiding exploration with language, 2024. URL https://arxiv.ozr
abs/ / 2403. 08140
Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, and Christopher D. Manning. Nnetnav: Unsupervised learning of browser agents through environment interaction the wild, 2025. URL https:
/aaaxi.orr//aba.. 02907
11

---

Published as a conference paper at ICLR 2025
Reiichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser- assisted question with human feedback. ArXiv preprint, abs/2112.09332, 2021. URL htt.ps://arxitii wttp://rrrraa///.../0o0000
Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. Screenagent: A vision language model-driven computer control agent. 2024.
OpenAl. Gpt-Av(ision) system card, 2024. gpt www.ysste----eee card
URL
Tianyue Ou, Frank F. Aman
Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, and Shuyan Zhou. Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale, 2024. URL httph:://///// arxiv. org/ abs/2409.
15637
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv preprint, abs/2203.000 022155, 2022. URL http:: http/aaxiivv/./////iv//eaa/aaaat org/ abs 2203.0215 02155
Jiay Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. ArXiv preprint, abs/2404.06674, 2024. URL hetps:
7/arxive www.vorg/aaal. abs 2404 06474
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. ArXiv preprint, abs/2304.03277, 2023. URL
03277
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian. Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023. URL htt.t:::///// /tttttssss//aii...ttt//a//t/ org/ abs/ 2307 16789
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL htthstts::/ arxiv org/ abs /atss/2412.00 15115
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control, 2023. URL /tRlstttss//aaxiv.co abs/ / 2307. 10088
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization for Ilm agents. 2024.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Ke- Ka-Yang Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vasion-language model's perception the world at any resolution. ArXiv preprint, abs/2409. 2024. URL https: 7/axxiv.or//abst 2211/
Michael Wornow, Avanika Narayan, Ben Viggiano, Ishan S. Khare, Tathagat Verma, Tibor Thomp- son, Miguel Angel Fuentes Hernandez, Sudharsan Sundar, Chloe Trujillo, Krrish Chawla, Rongfei Lu, Justin Shen, Divya Nagaraj, Joshua Martinez, Vardhan Agrawal, Althea Hud- son, Nigam H. Shah, and Christopher Re. Do multimodal foundation models understand en- terprise workflows? a benchmark for business process management tasks. ArXiv preprint, abs/2406. .....113... 13264, 2024. URL|httss: //atpsss//axivi www.or/g/abb///aaeee 13264
12

---

Published as a conference paper at ICLR 2025
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Stheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. ArXiv preprint, abs/2404.07972, 2024.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chun yue Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. ArXiv preprint, abs/2310.11441, 2023. https::/aaxxiiiii http://gggabb//......oog/ 2abbb//230.114...
org/ abs
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35, 2022.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. ZeroGen: Efficient zero-shot learning via dataset generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computa- tional Linguisties, 2022.
wrt. wttp://2222emmmmmmmammmmm emnlp-main. 801
Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users, 2023.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. ArXiv preprint, abs/2401.01614, 2024. URLhhttss///xxxiv.ooo wwtp:/ivoorgos////o
24C1.01612 01614
Shuyan Zhou, Frank F Xu. Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. ArXiv preprint, abs/2307. 13854, 2023. abs 220//3307....83
13

---

Published as a conference paper at ICLR 2025
Table of Contents in Appendix
Calculation of Other Trajectory Datasets
15
Analysis of the Effectiveness of Tutorials
15
C Cost Details
15
D Evaluator Alignment
15
Action Mapping
16
Experimental Results Textual Data
16
Details in Collecting Tutorials
16
Prefilter Function
16
G.2 LLM Labeler Prompt
18
G.3 Tagging & Paraphrasing Prompt and Format
18
H
Examples of Failed Guided Replay Trajectories
20
Scaling up AgentTred
21
Evaluation Benchmarks
21
J.1 GUI Grounding Evaluation
21
J.2 Offline GUI Agent Evaluation
21
Detailed Description of guided replay
21
14

---

Published as a conference paper at ICLR 2025
A CALCULATION OF OTHER TRAJECTORY DATASETS
RUSS: Cited based on the data provided in the table from WebLINX Lu et al. 2024).
ScreenAgent: Statistics obtained from the dataset available htthts:
com/
niuzaisheng/ hwtp://eeeeeeeeeeeeeeeeeeeeeeeee hwtinnngnnntttteeeeeeeeee /ttp:///aat///eeeeeagttt//teteea
WebLINX:
Calculated based on the train set information from Table (Lu et al. 2024)
and data on HuggingFace (excluding the "say" actions), resulting in a total of 18,249 non- say actions with 969 demos.
Mind2Web:
Statisties derived from http::///ugginnffcc...o/ttttttte
baunlp/Mind2west specifically from the training subset.
Webshop (agent-eto); Data statistics sourced from https://hugginggacc...
datasets/ catcassts//geentt-ettttttoosssss
jectory
WonderBread: Calculations based on data presented in (Wornow et al. 2024).
B ANALYSIS OF THE EFFECTIVENESS OF TUTORIALS
Key factors contributing this improvement include:
1. Direct Access to Target URL:
Tutorials provide the target URL, allowing direct access to the
initial task state, reducing errors in locating the correct webpage.
2. Assisted Planning with Human Expertise: Tutorials aid in planning by providing steps in- formed by human experience, which tend to be reliable, thereby reducing the likelihood of errors during task execution and bridging the gap in the agent's knowledge for unknown tasks.
3.
Navigating Multi-Level Menus: Tutorials offer clear paths to hidden elements, preventing the agent from failing due to incorrect navigation through complex menus.
C COST DETAILS
In this part we provide the details of our cost in generating trajectory data with via our pipeline:
Phase
Cost per 1,000 Entries (USD)
Model Used gpt-4o-mini gpt-4o-2024-08-06 8p1-4o-2024-08-06
Tag and Paraphrase
0.886
Replay
215.359
Evaluator
3.104
Table 7: Cost breakdown for each phase in the process
Another two important factors are the ratio of web-related tutorials (0.275) and the Replay Success
Rate (39.9%). Using these, we can calculate the cost per verified effective trajectory as follows:
Cost per trajectory Tag and Paraphrase price Replay price Evaluate price
Web ratio
Replay Success Rate
The cost per 1,000 verified effective trajectories is 550.75
D EVALUATOR ALIGNMENT
In this part, we provide the details of metrics between the human and automatic evaluator.
Trajectory
Evaluator
Accuracy
Web Tutorials
VLM Evaluator
84.0%
GPT-4V
80.6%
Webarena
Captioner + GPT-4
82.1%
Captioner Mixtral
74.4%
15

---

Published as a conference paper at ICLR 2025
Success
Failure
Human Judge
Success 0.39
0.14
Failure-
0.02
0.45
VLM Evaluator Predict
Figure 9: Confusion Matrix of our VLM evaluator's 's performance on the human-annotated validation set, compared with evaluators across different scenarios.
E
ACTION MAPPING
Table 8: Mapping between Playwright and PyAutoGUI Action Spaces.
Category
Playwright Action
PyAutoGUI Action
page.click ()
pyautogui.clis ()
page.type ()
pyautogui write ()
Basic Actions
page.press ()
pyautogui.press
hage. ()
pyautogui. Io
page.scrol1
peautogui.acroll/
fage)
pyautogui.wriee (clearing)
page. ()
pyautopui.doublleliaa.
Advanced Actions
page.
pyautoguii.dragToo
page.clear ()
pyautogui.clickk/
pyautogui.hockeyyCeo (ctrl,
pyautogui..press (delete)
Plugin
playwright..e..oct.op...o
browscr.select
F EXPERIMENTAL RESULTS ON TEXTUAL DATA
To provide further supports for the effective of our AgentTrek data, we conducted an experiment to evaluate the performance of a pure textual agent using the textual version data of our AgentTrek trajectories. This allow to study the contribution of textual modalities of AgentTrek.
We fine-tuned the 1annn2.5-7--Intrruue model using AgentTrek trajectories that included accessibility tree as observations and playwright actions the agent's action space. We then evaluated the model on WebArena, an OOD web agent benchmark featuring self-hosted websites. These websites entirely out-of-domain (OOD) from the AgentTrek dataset, ensuring that the evaluation reflects the model's generalization capability.
We fine-tuned Bwven2.-7BBIInstrude results on WebArena and Miniwob++ as shown Table We observe that our fine-tuned model achieves the highest performance among open-souree web agents and approaches the performance of GPT-4o, demonstrating the effectiveness of AgentTrek data in improving real-world web agent capabilities and generalization across modalities.
AgentTrek's textual data and achieved the following
G DETAILS IN COLLECTING TUTORIALS
G.1 PREFILTER FUNCTION
Keyword Density: The web content must contain a minimum of 20 common keywords, ensuring sufficient topic coverage.
Keyword Diversity: The text must incorporate at least distinct common keywords.
16

---

Published as a conference paper at ICLR 2025
System Prompt
You are an expert in evaluating the performance of a web navigat ion agent. The agent is designed to help human user navigate website complete task. Given the user's s task goal, the agent's trajectory, your goal is to decide whether the agent's execution successful or not.
*Evaluation Criteria*
Whether the agent's trajectory is effective and corresponding to the goal
+Instructions
1. Review the agent's acti ons and reasoning processes step by step.
if the agent is stuck in the very first login stage, which fails to log into target website at the beginning, that's failure.
3. Determine if the agent has achieved the task goal based the trajectory. A task can be considered successful if most trajectory effective.
4. the agent sometimes can't stop after finishing a task and continue doing zepeated actions. these actions may be some failed attempt after a series of correct actions. the task should be regarded as successful if the correct actions almost reach the goal.
effective and
5. if the agent is stuck in the loop at the early stage of the task, which means they don't even get close to the goal before they get stuck in the loop, that's failure. for example, the agent begin to get stuck before third step.
6. when the task is to change the google account password, it can't be regarded as successful when agent finish at trying to click "manage your account".
7. if there are over 8 correct action in the trajectory, it
regard as a successful agent.
8. final saving action is not must. the task is successful if the agent does most things right and just forget to save the change at last.
if the original task has 2 subtasks, the agent only complete them, that's still a success. e.g. the task is to update name and birthday, but agent only update name, that's fine. 10. if the task is to post review, the agent can be considered successful when it finish writing the review and reach the step to post it, don't have to click the post button.
11. Since we don't t have a printer, some printing related task can be considered successful if the agent reach the step to click print button.
12. if the task is finished the initial state and the agent do nothing because of it, should also regarded as successful,
IMPORTANT*
the trajectory, an action always follows a corresponding reasoning, which shows the observation and thought of the agent. your response should be contain:
2.
Thoughts: <your thoughts and reasoning process> Status:
"success" or "failure"
User Prompt
The goal of the task: (task_des) trajectory: (trajectory)
Figure 10: Prompts to query the VLM Autonomous Evaluator.
Essential Keyword Frequency: At least one mandatory keywords must appear multiple times (minimum twice) within the content, demonstrating topic relevance.
17

---

Published as a conference paper at ICLR 2025
G.2 LLM LABELER PROMPT
To achieve more precise and context-aware labeling, we designed the following prompt guide the LLM in further assessing whether the given URL and context meet our requirements, as illustrated in Fig
System Prompt
You are assistant that classifies content based on specific criteria. Your task is to svaluate whether given piece content serves as a tutorial specifically related to graphical user interfaces (GVI), such as for web applications, desktop applications, operating systems.
Classification Criteria
The content qualifies as GUI-related tutorial if it meets the following conditions:
includes achieved.
description outlining what needs to be
2. It provides clear step-by-steg instructions for interacting with
GUI, such as:
Step 1: Open the application
step 2: Navigate to the settings menu
Given the URL and context, determine if the content is GUI-related tutorial or not. Output '1" if it is a GUI-related tutorial and '0' if it is not. Provide only the number as the output.
User Prompt
URL: {url}
Context: [context]
Figure 11: User Prompt for Classifying GUI Tutorials
G.3 TAGGING & PARAPHRASING PROMPT AND FORMAT
Here we present the prompt designed to utilize LLM to help do the tagging & paraphrasing of the identified GUI-tutorial related context.
18

---

Published as a conference paper at ICLR 2025
User Prompt
The following is tutorial from the website. It may contain several tutorials. PLease extract the first tutorial only and format the first tutorial according to the specified schema:
Text:
(context)
Schema:
"platform":
"Platform. category (choose from: macos, Windows (Default if not specified in the tutorial), Linux, Android, i0s)",
"target type":
"Type of platform (choose from: Web browser, PC app, Mobile app, PC operating system, Mobile operating system, where the tuterial's steps are performed). Tutorials that involve interacting with the browser software itself, such 'opening Chrome settings," should be classified as a PC app type.",
"target object":
"Specific name of the web browser (non web browser) applications or operating system where the tutorial's steps performed (e.g., Chrome browser (Default for browser and tutorial), Microsoft Excel (app name), Windows system settings)
"target web URL":
"The exact URL of the web page where the tutorial's actions take place, applicable only if the target object is None, https://maLL.googlk com, https://wwe www.wwaaaoo.c.ommmmmmm com, https://github.ce
browser (e.g.,
Be careful, the URL provided at the beginning is always not the URL where the tutorial's actions are about. For example, tutorial from https://abidakon. http://bicckko.ccoooootttttooo conn/how-tommake-goo
iee-lideeett about
changing Google slides, its target web URL should be setps::/docs.goooe wwwpogll..cooppeeeeee com/presentation....
"task description":
"Task description text (Provide concise summary in sentence, including essential details)",
"prerequisites":
"Prorequisite text describing necessary conditions before starting the task".
"instructions"
"Step 1: Instruction text describing the action to be taken", // Following instructions
"instructions steps":
"Total number of instructions steps",
"expected result";
"Text describing the expected result after following the instructions"
Figure 12: User Prompt for Extracting and Formatting GUI Tutorials
19

---

Published as a conference paper at ICLR 2025
H EXAMPLES OF FAILED GUIDED REPLAY TRAJECTORIES
Task
Find and reserve table a top restaurant Boston for April 22.
Step
Step 3
Step
Thought:ll have sucnosfully fully marigattd need follow instructions to
Thought: have successfally clicked on the location theld button (old Si), need Broter' into this per Step at the instructions The location should be and ready for Input.
Thought: I have successfully entered BBostee Into the lecation fleld. The nent select
hetpess/resyccom
reserve table a top restaurant for The click on the locatton field and type Boottnn
reservation, which should Apeil need find and click the date field oreen the cxlendar picker.
Action: clicky (Z]
Action: G0U61 (Sosion))
Action: dica
Step
Step
Step 6
Thought: I have clicked on the date fleld button (hiè open calienser for selecting the reservation date. The celendar should now he viehle and buttion for April 22 and click on it sett the devreed date reservation.
Thouught: have opened the date
Thought: previoucly clicked the 'Net months button to sovigate through the callendar. Huwwwer, neee go montits from Desoberr 2024 to reach April 2004, Elicking the Y (Nest montii" button not the sursect action. PPrevious montl:"
selection colendar, but it's rumently showing October
April 2024 select Apeil 22 for the rosevvatioo ien. The calendar disns't soom to have a disect specifie mosth,
navigate
erpanded,
should
use (Previous montis:
button Insoad..
mulktple times to reach April 2004.
Action: dicck 523)
Action:
Action: cluck 606)
317)
Step
Stap 9
Thought provious weiken click the (Prestous month:" button s00: resulted in an ciror, indicating that the element could not be found. button's bid has
Thought:
mussage, month""
scema that buttoe with bid (Sis is not avuilable on the current page. This could be besause the calendar view has changed button
suggests
**FAIL. 09
changed the calendar been updated. need to reansess the current state of the and find an navigate April
been updated,
need reastess the avallatle elements and tind an alternative say navtgate April 2001.
altemative
Action:
Action: elickrS10)
(ick
Figure 13: Replay agent was unable to complete booking before the actual date due to the tutorial expiration.
20

---

Published as a conference paper at ICLR 2025
Table 9: Multimodal Mind2Web Step SR across varying amounts of training data from AgnetTrek
(AT).
Data Amount
Cross- Task
Webss-
Cross-Domain
20%
36.1
35.5
39.5
40%
41.0
35.8
42.5
60%
41.6
37.2
42.8
80%
42.6
38.0
44.3
100%
42.6
37.5
45.0
SCALING UP AGENTTREK
In this section, we further scale up the data amount of AgnetTrek (more than 10K trajectories) explore the effectiveness of AgentTrek in large-scale size. We trained the model using varying proportions of the dataset (20% to 100%) and assessed its performance across three splits. The results are presented in Table We observe that the performance improves steadily more data is used, with the best results achieved when using the full dataset. This underscores the value of scaling up AgentTrek in improving model effectiveness.
Multimodal-Mind22
J EVALUATION BENCHMARKS
In this section, we introduce more details of evaluation benchmarks used in our work.
J.1 GUI GROUNDING EVALUATION
ScreenSpot. ScreenSpot Cheng et al. 2024 is benchmark developed specifically for GUI visual grounding tasks, featuring 1.2K sigle-step instructions along with the coordinates target elements. The dataset includes diverse grounding instructions tailored for mobile, desktop, and web platforms and categorizes elements into text and icons/widgets. Two distinct assessment scenarios utilized: (1) Original Instructions, where models directly execute grounding actions as per the provided instructions; and (2) Self-plan, where models are expected to formulate plans in natural language based the original instructions before carrying out the grounding actions.
J.2 OFFLINE GUI AGENT EVALUATION
We evaluated the offline planning capabilities of GUI agents on websites using the Multimodal-Mind2Wee benchmark (Zheng et al., 2024), which is an extension of the original Mind2Wat benchmark (Deng et al. 2024). Performance was measured using Element Accuracy AEle.Acc), Operation Fl (Op.Fl 1),, and Step Success Rate (Step SR).
Multimodal-Wind22eeh.
The GPT-3.5 and GPT-4 results for HTML and HTML+Image observation are derived from the SecAct (Zheng et al. 2024) method. For the Choice method, it employs DeBERTa-borl cross-encoder to rank the interactive elements on the current HTML page. The top 50 elements are selected as options, and the GPT-3.5/GPT-4 model then chooses one of these elements the answer. For the SoM method YYag et al. 2023) Zheng et al.] 2024). it renders a new webpage image by adding red bounding boxes and labels every HTML node in the source code, allowing GPT-4 to understand the webpage screenshot and identify the target action object by referring to the labels. For the Image observation, as detailed Section ??, we Qwen2VL (Wang al.) 2024) within pure vision framework. Specifically, Qwen2VL processes only the webpage screenshots and specifies the target action object by generating coordinates.
K DETAILED DESCRIPTION OF GUIDED REPLAY
In this section, detailedly describe example of model execution in guided replay.
21

---

Published as a conference paper at ICLR 2025
Observation Prior to Execution Before executing any actions in the task execution process, the model observes the current webpage within the BrowserGym environment. Each webpage provides rich data, including its HTML structure, accessibility tree (AXTree), and screenshots. The model uses the AXereee as the primary observation source, with each element in the rree uniquely identified by [bid]. This structured observation ensures accurate and consistent interaction:
Axtree with Element bid
[119] 1ink 'Magento Admin Panel'
[120] image 'Magento Admin Panel'
[121] navigation "'
[122] menubar " of
[124] link Aue604 DASHBOARD'
[127] link Nue50b SALES'
[614] banner "
[617] heading 'Dashboard'
[620] link Aue600 admin'
Figure 14: Observation Prior to Execution in Guided Replay
the information in tutorial the tutorial may provide detailed textual description of the target element. The model realize that the target element is menubar, which associates with bid( (122)
Axtree with Element bid Step 1: click the menubar to see the sales
Figure 15: Observation Prior to Execution in Guided Replay
the action executed When the model performs an action, it does not need to provide fine-grained target element information such as coordinates, only the action type and the target object's 's bid as follows:
Axtree with Element bid
click (119
Figure 16: Observation Prior to Execution in Guided Replay
We can retrieve the target elements the webpage through the bid and perform corresponding operations through playwright action.
22


Reviews:
[{"review_id": "YLu1tx3n7G", "reviewer": "Reviewer_MDGs", "summary": "1. The paper introduces AgentTrek, a scalable data synthesis pipeline that generates high-quality GUI agent trajectories\nby leveraging web tutorials.\n2.  The method collects web tutorials from the internet, transforms them into structured task goals with step-by-step instructions, and uses a visual-language model (VLM) agent to simulate their execution in a digital environment. ​ A VLM-based evaluator ensures the correctness of the generated trajectories.\n3. The paper provides experimental results and analayis showing that agents trained with the synthesized data outperform those trained on existing datasets in both grounding and planning capabilities.\n4. The authors emphasize that the method is more cost-efficient than traditional human annotation methods, making it a practical solution for large-scale GUI agent training", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is a novel pipeline that leverages web tutorials to synthesize high-quality GUI agent trajectory data at scale. This is a valuable contribution to the field, addressing the scarcity of reliable and scalable trajectory data. The proposed pipeline significantly reduces the cost of data collection compared to traditional human annotation\n2. The paper is well-structured and clearly explains the steps involved in the data filtering\n3. The dataset is comprehensive, containing a wide range of task types, platforms, and web environments.", "weaknesses": "1. The paper does not provide a baseline to compare how using trajectory data compares with just using textual data. It would be beneficial to see how much the different elements( DOM/HTML structures, AXTree data, intermediate reasoning steps, full video recordings, and corresponding screenshots for each action) contribute to the dataset effectiveness.\n2. The paper deals with only web-based tutorials and shows evaluation on only 2 benchmarks. It would be beneficial to expand the evaluation by including additional benchmarks such as MiniWob.\n3. The paper lacks an analysis of potential failure cases. For example, is the trajectory data still as effective when the number of steps increase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aQXLe4qkQM", "reviewer": "Reviewer_fPH1", "summary": "The paper introduces AgentTrek, a data synthesis pipeline that generates web agent trajectories from online tutorials. It automatically collects tutorials, converts them into task sequences, and uses VLMs to simulate and evaluate these tasks. Results show that agents trained with AgentTrek data perform better in task grounding and planning than those using traditional datasets, providing a cost-effective solution for large-scale web agent training.", "questions": "1. The overlap issue to Mind2Web.\n2. Is there any other way to support the effectiveness of the synthesized data?\n3. If possible, show the effectiveness of the synthetic data when it is further scaled up.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "A clear pipeline for generating complex agent trajectories from web tutorials.", "weaknesses": "1. In the Mind2Web experiment, it is clear that there is overlap between the training data and those in Mind2Web-test, with likely overlap in tasks, websites as well as domains. This overlap should be clarified, as it undermines the intended out-of-domain evaluation of Mind2Web.\n2. The total number of trajectories remains limited, staying within the same scale as Mind2Web (in thousands). And the effectiveness is not good enough compared to the training data of Mind2Web (when the data size is 4x).\n3. The comparison on grounding is not very meaningful, as performance significantly lags behind recent work focused specifically on grounding. (And empirically, a significant gain is from changing the backbone to Qwen 2 VL, compared to SeeClick and CogAgent.)\n4. The web results on ScreenSpot, especially for icons, are not strong, which raises further questions about possible overfitting to specific websites or tasks in Mind2Web.\n5. Some writing issues:  \n   - baseline results should clearly indicate their sources\n   - duplicate entries in the reference\n   - wrong reference\n   - more details of the evaluation on Mind2Web should be provided. (This is not minor. As there are huge differences with respect to the settings in table 6)\n   - the synthesized data is essentially web agent trajectories. No need to always overclaim it to GUI agent trajectory. It only confuses people.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rIgMh0U47g", "reviewer": "Reviewer_SAfQ", "summary": "The AgentTrek framework introduces a scalable pipeline for generating high-quality GUI agent trajectory data by utilizing web tutorials. It automates the collection of tutorial-like instructions from the internet, transforms them into structured tasks, and uses a visual-language model agent to simulate and execute these tasks in real digital environments. An evaluator model verifies the generated trajectories to ensure accuracy, reducing reliance on labor-intensive human annotations. Experimental results show that models trained with AgentTrek-generated data outperform those trained on existing datasets, especially in task planning and GUI grounding. This framework offers a cost-effective, automated solution for training GUI agents on a large scale.", "questions": "See the weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The AgentTrek framework leverages online tutorials to automatically generate high-quality GUI agent trajectory data, automating the data generation process and reducing the need for manual data collection. Through an evaluator model, it verifies the generated trajectories, with a multi-layered generation and evaluation mechanism to ensure data quality and effectiveness.\n\nAgents trained on AgentTrek-generated data perform exceptionally well in several benchmark tests (such as ScreenSpot and Multimodal-Mind2Web), especially in task planning and GUI element recognition, significantly outperforming traditional datasets.", "weaknesses": "Certain technical details, such as automatic labeling and tutorial filtering, are only briefly mentioned in the paper, lacking more comprehensive explanations.\n\nThe paper notes that the success rate of generating effective trajectories is only 39.1%, based on GPT-4o Mini. Although GPT-4o Mini is relatively cost-effective, achieving larger-scale data generation with the current success rate remains challenging. There should be some indications if experiments are conducted with alternative open-source models to assess the feasibility and effectiveness of data construction within this framework.\n\nAnother consideration is the broader generalization of the framework for more complex computer control tasks. Many tasks may lack corresponding web tutorials or have very limited resources, especially those requiring highly precise and complex control, which will also lead to the data/categories bais issue. Do you have any thoughts or attemptions on these situations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)