VideoAgentTrek: Computer-Use Pretraining from Unlabeled Videos
Authors: Dunjie Lu
Yiheng Xu
Junli Wang
Haoyuan Wu
Xinyuan Wang
Zekun Wang
Junlin Yang
Hongjin SU
Jixuan Chen
Junda Chen
Yuchen Mao
Junyang Lin
Binyuan Hui
Tao Yu
Venue: ICLR 2026 Poster (2026)
Keywords: VideoAgentTrek
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: VideoAgentTrek automatically mines GUI action trajectories from unlabeled screen-recorded videos via inverse dynamics,      enabling scalable computer-use agent pretraining with 70% relative improvement on OSWorld-Verified.

Abstract: Training computer-use agents requires massive amounts of GUI interaction data, but manually annotating action trajectories at scale is prohibitively expensive. We present VideoAgentTrek, a scalable pipeline that automatically mines training data from publicly available screen-recorded videos, eliminating the need for manual annotation. Our approach addresses a key challenge: raw videos contain implicit demonstrations but lack explicit action labels. To solve this, we develop Video2Action, an inverse dynamics module (IDM) with two components: (1) a video grounding model that detects and localizes GUI actions with precise temporal boundaries, and (2) an action-content recognizer that extracts structured parameters like click coordinates and typed text. Applied to 39,000 YouTube tutorial videos, our pipeline generates 1.52 million interaction steps. We leverage this data through continued pretraining followed by supervised fine-tuning. On OSWorld-Verified, our approach improves task success rates from 9.3% (SFT-only baseline) to 15.8%, a 70% relative improvement. On AgentNetBench, step accuracy increases from 64.1% to 69.3%. Our results demonstrate that passive internet videos can be transformed into high-quality supervision for computer-use agents, providing a scalable alternative to expensive manual annotation.

Published as a conference paper at ICLR 2026
VIDEOAGENTTREK: COMPUTER USE PRETRAINING FROM UNLABELED VIDEOS
Dunjie Lu Yiheng Xu!." Junli Wang", Haoyuan Wu', Xinyuan Wang' Zekun Wang'. Junda Junlin Yang', Hongjin Su', Jixuan Junda Chen', Yuchen Mao', Jingren Zhou', Junyang Lin?, Binyuan Hui Tao Yu 1
The University of Hong Kong
2Qwen Team, Alibaba Group
ABSTRACT
Training computer-use agents requires massive amounts of GUI interaction data, but manually annotating action trajectories at scale is prohibitively expensive. We present VIDEOAGENTTREK, a scalable pipeline that automatically mines training data from publicly available screen-recorded videos, eliminating the need for manual annotation. Our approach addresses a key challenge: raw videos contain implicit demonstrations but lack explicit action labels. To solve this, we develop ACTION, an inverse dynamics module (IDM) with two components: (1) video grounding model that detects and localizes GUI actions with precise temporal boundaries, and (2) action-content recognizer that extracts structured parameters like click coordinates and typed text. Applied to 39,000 YouTube tutorial videos, our pipeline generates 1.52 million interaction steps. We leverage this data through continued pretraining followed by supervised fine-tuning. On OSWorid-Verified, our approach improves task success rates from 9.3% (SFT-only baseline) to 15.8%, a 70% relative improvement. On AgentNetBench, step accuracy increases from 64.1% % to 69.3%. Our results demonstrate that passive internet videos can be transformed into high-quality supervision for computer-use agents, providing a scalable alternative to expensive manual annotation.
INTRODUCTION
Teaching machines computers like humans do (clicking buttons, typing text, navigating interfaces) represents fundamental challenge in AL. While recent advances in Dision-langure models have made computer-use agents increasingly feasible (Bai al., 2025 Qin et al. 2025 Team
et al. 2025 Wang et al. 2025b), their development remains bottlenecked by data availability. Training screenshots paired with exact action parameters like click coordinates (z, y) and typed strings. However, creating such datasets through manual annotation is extraordinarily expensive, making it impractical to achieve the scale necessary for robust generalization across diverse applications and operating systems.
these agents requires extensive trajectories that precisely document GUI interactions:
Meanwhile, the internet hosts millions of screen- recorded tutorials where humans demonstrate computer use, from Excel tutorials to software walkthroughs. These videos implicitly contain the supervision we need: they show where users click, what they type, and how interfaces respond. Yet this resource remains untapped because videos lack the structured action labels required for training. The cursor movements are visible but not tracked; the typed text appears but isn't 't extracted; the timing of actions is implicit but not annotated. We can learn to automatically extract structured action trajectories from raw videos by training specialized models to detect when actions occur and infer what their parameters are, effectively converting passive recordings into active training data.
We introduce VIDEOAGENTTREK, scalable pipeline that mines computer-ume use trajectories from pub- licly available unlabeled videos without manual annotation. Our approach employs VIDEO2ACTION an inverse dynamics module (IDM) with two stages: First, an action event detection model performs dense event detection, identifying action types and their precise temporal boundaries (e.g., c ck at [1.5, 2.0]s, type at [3.5, 5.5]s). Second, the action parameterization model, an action-content
"Equal contribution. Corresponding authors. Page: https: /v http:///dddcceeetttr.....i.bb

---

Published as a conference paper at ICLR 2026
collect
screen filtor
videos
You Tube
r' lalabl
Girajectory
Seasoning
Inverse Dynamic
Action
training
Rearoning
tags
Aition
Search
Video Collection
2 Video2Action
3 Agent Traing
Figure 1:
Overview of VIDEOAGENTTREK, (1) Video Collection: crawl ccreen-recorded tutorials
and filter GUI footage with SCREENFILTER. (2) Video2Action: an inverse dynamics module that first performs dense action-event detection localize clips and assign action types, then action pa- rameterization (e.g., click coordinates, typed text) to yield structured (screenshot, action, parameters) trajectories. (3) Agent Training: use the mined trajectories for continued pretraining and supervised finetuning of computer-use agents.
recognizer, analyzes these localized segments extract structured parameters (pointer coordinates for clicks, literal text for typing), yielding complete (screenshot, action, parameters) trajectories suitable for training.
VIDEOAGENT enables large-scale computer-use pretraining with unlabeled web videos. From 39,000 YouTube videos, we automatically extract 1.52 million interaction steps. This represents not just more data, but more diverse data: the trajectories span hundreds of applications across Windows, macOS, and web platforms, capturing interaction patterns that would be infeasible to annotate manually.
We validate VIDEOAGENTTREK with a two-stage training recipe: continued pretraining on the mined trajectories followed by supervised fine-tuning on curated dataset. This combination leverages the broad coverage from videos to learn fundamental GUI interaction patterns, while supervised fine-tuning sharpens task-specific performance. Our models achieve 15.8% task success OSWorld- Verified compared to 9.3% for baselines, a 70% relative improvement. The gains are particularly pronounced in online environments where robustness visual variation matters most. We summarize our main contributions and findings below:
We propose VIDEOAGENTTREK, an unsupervised approach to training computer-use agents that automatically converts receen-recorded videos into structured training data through learned inverse dynamics, thereby eliminating the need for manual annotation.
Our VIDEO2AcTION module implements inverse dynamics, combining action event detection with millisecond preccssio temporal localization and action parameter extraction. It enables accurate reconstruction of GUI interactions (clicks, typing....) from raw video without ground-truth labels.
Experiments demonstrate that our approach achieves 15.8% task success on OSWorld- Verified compared to 9.3% for SFT-only baselines (70% relative improvement), and improves step accuracy on AgentNetBench from 64. to 69.3%, validating that passive internet videos can provide effective supervision at scale.
We provide reproducible pipeline and training methodology that enables researchers leverage publicly available screen recordings for computer-use agent training. To facilitate future research, we release SCREENFILTER for efficient GUI filtering and VIdEO2ACTION for action extraction as open-source tools.
2
VIDEOAGENTTREK
We introduce VIDEOAGENTTREK, video-driven pipeline that turns web tutorials into training supervision for computer-use agents. Each trajectory is a sequence R = {[I, Tkk ag. F. k following Yao et al. 2023). where 7 is representative screenshot, r&k brief inner monologue,
2

---

Published as a conference paper at ICLR 2026
Seed Keywords
Evaluate
High-quality Videos
Videos
Videos
Excsl Tuterial
Related Videos
Uburtte
#
Video
Tags
BLina
High-quality Channels
80% High-quality?
Evaluate
Videos
from Channel
Screen Filter
Collect Filttrred Videos
VAT Data
Figure 2: Video candidate auto-discovery. From seed keywords and tags, we search and evaluate videos, expand to related videos and high-quality channels (280% pass), and iteratively collect GUI-containing videos for VAT.
as AA is the action type (e.g. click, type), and my is the action content (e.g., pointer (v,y) y) typed text). The pipeline has three parts:
Video collection and preprocessing. We crawl tutorial videos with seeded queries and tag expansion, apply human-in-the-loog screening, and use cursor-based based filtering to retain screen segments with GUI interactions (Section 2.1).
VIDEOZACTION From raw video, we recover stepwise supervision without manual labels: (i) dense event detection produces typed segments with tight start/end times; (ii) action identification infers parameters Tę (e.g., click coordinates, typed strings); and (iii) short inner monologue rk: makes the intent explicit. Assembling these per-clip steps yields ReAct tuples for training (Section 2.2
Agent training. We combine large-scale agentic data produced by the method with human demonstrations and targeted GUI grounding pairs, and train end-to-end agent in stages: interleaved video-text pretraining followed by instruction-style finetuning (Section 2.3
This structure scales supervision web-scale while preserving the stepwise semanties needed for robust computer-use policies.
2.1 VIDEO COLLECTION AND PREPROCESSING
2. VIDEO CANDIDATE AUTO-DISCOVIE
We employ a scalable pipeline for video collection that leverages channel coherence- observation that YouTube channels typically maintain consistent content types and quality. Starting from seed keywords such "Excel tutorial" and "How use Windows", we validate initial results and extend to entire channels when sampling indicates high quality (i.e., when > 80% of sampled videos meet our criteria). This channel-based expansion enables efficient scaling: validated channels become trusted sources for candidate videos, while their tags and metadata enable iterative discovery.
When we identify high-quality channels through seed validation, we include all their videos as candidates rather than individually vetting each one. This approach deliberately optimizes recall over precision, as the subsequent SCREFNFILTER stage ensures final data quality. The channel coher- ence property-wher content creators typically focus consistent topics-maal this expansion particularly effective.
Through iterative rounds of keyword search, channel expansion, and tag extraction, we transform small set of manually validated seeds into 55,000 candidate videos (~10,000 hours). This process requires minimal human oversight: initial quality validation seed videos and periodic verification of expansion effectiveness. The resulting candidate pool intentionally includes some non-GUI content (presentations, tutorials with mixed content), which filtering stage handles efficiently.
3

---

Published as a conference paper at ICLR 2026
EASY
TEXT
Video Subtitles
Action Event
Detection
Actin
Parameterize
Action
Actin
0000
Generate Thoughts
Thoughts
Figure 4: Overview of VIDEO2ACTION: Given a screen-capture video (with optional subtitles), the module (1) detects GUI action events and segments clips, (2) parameterizes each action (type and arguments), and (3) generates step-level thoughts, yielding training-ready sequences of {action clip, action, thought)
2.1.2 VIDEO PREPROCESSING WITH SCREENFILTER
Although keyword-Ne based searches typically retrieve relevant computer operations, they also include non-interactive segments, such as explanatory sections where the presenter uses PowerPoint or other presentation tools. Additionally, some of the videos retrieved through this method may not meet the standards for GUI interaction content.
To address this, developed SCREENFILTER, lightweight cursor detection model upon YOLOv8x (Reis et al. 2024) to efficiently extract video segments that focus exclusively on GUI interactions. Using the detection results, retain video segments where at least 80% of the frames contain cursor for 6 seconds or more, with 2-second merge gap for temporal smoothing. When applied to our corpus, SCREENFILTER successfully extracts 7.377 hours of verified GUI interactions from 10,000 hours of raw video. SCREENFILTER data construction and evaluation details are in Appendix
2.1.3 VIDEOAGENTTREK DATA ANALYSIS
Quality and relevance. We collected 55k screen-capture videos (about 10,000 hours) from 50+ channels. The corpus is predominantly clear (about 97% 720p higher) and most clips are min- utes long, yielding sustained, readable interactions suitable for our pipeline (Table D). A lightweight title/description audit groups videos into tutorials, background pieces, tech talks, and unrelated; tuto- rials dominate (69.6%), with the remainder used mainly for tag mining or removed during filtering (Table [TO). Together, these checks indicate that the collected data are both visnally clean and tonically aligned with computer-use supervision.
Data classification. We label each video daily, office, workflow, professional, operating-system (OS), or other using a lightweight GPT-4.1 pass over the title and short transcript snippet. The dis- tribution (Figure 3) is skewed toward OS-level operations (~36%)) followed by professional 1~1,),, daily (~18%) and office (~16%); workflow is smaller (~7%) with small remainder labeled as other (~4%). This indicates broad coverage with a bias toward system and professional use cases.
Others 4.42%
OS 35.70%
Workflow 7.11%
Office 16.21%
Professional
Daily
Figure 3: Domain distribution.
2.2 VIDEO2ACTION: INVERSE DYNAMICS MODULE
We develop VIDEO2ACTION, an inverse dynamics module (IDM) that extracts structured action supervision from unlabeled GUI videos. Following insights from robotics where inverse dynamics recovers actions from observations (Nguyen- Tuong & feterr][210) VIDEO2ACTION detects GUI events (clicks, drags, scrolls, typing) and infers their parameters directly from pixel changes. This yields training-ready (screenshot, action) pairs without manual annotation, forming the second core component of VideoAgenttThek toolkit.
4

---

Published as a conference paper at ICLR 2026
2.2.1 ACTION EVENT DETECTION
Task. Given an unlabeled screen-capture video v (length T), perform prompt-free, dense event detection: predict set of typed GUI interactions with tight temporal bounds,
fo(v) S {(ag, th ()) as E A, 0 s S.
Unlike query-based setups, our input contains only o; the output is multi-event set with both action types and start/end timestamps.
Approach. We equip VLM with video grounding that, given clip, it emits sequence of (ag, Eg tg.) for all GUI actions, reframing keyframe detection as multi-class temporal event detection with tight bounds. (1) Training data: We utilize the annotation tool provided by OpenCUA (Wang eet al. (0256) to obtain synchronized screen videos and timestamped GUI interactions (mouse/key- board events). These raw demonstration logs are then used to create temporal-groundee
supervision, allowing precise event detection without manual annotation. (2) Model training: We leverage Qwen2.5-VL. (Bai et al.) 2025) as the base model, benefiting from its multimodal understanding and fine-graines spatiotemporal capabilities. We perform full-parameter supervised fine- fine-tuning on the Bwve2.5-VVL-7B-mmm Instruct model enable
generate ordered, typed event spans directly from raw video clips. (3) Evaluation: We evaluate the detector in two phases. First, we check its performance on small curated subset from the source corpus, ensuring tight boundaries and full recovery of relevant GUI actions. Second, we apply the model to unseen web tutorials and conduct blinded manual review to assess its robustness and real-world usability. Details are provided in Appendix D
2.2.2 ACTION PARAMETERIZATION
Task. Given a detected action segment Uy = vlty : ti] with type ag E A. predict the action content (parameters) Te:
(vw) - Tk).
For example, a click segment yields (vw) + (click, (x, v)), while a typing segment yields h.(())) (type, (content)).
Approach. We build a recognizer hg that, for each detected segment Uk. predicts both the action type and parameters (âg, Tx). (1) Training data: We start from the OpenCUA raw demonstration logs, which pair screen-capture video with timestamped mouse and keyboard events. Each event is converted into type-specific parameter labels and temporally aligned to its clip, yielding prompt-free free supervision that captures the exact content of the interaction.
(2) Model training: Using Qwen2.5-VI VL (7B Instruct) as the base, we perform full full-parameter supervised fine-tuning so the model maps uk directly to (āg, Tk); when available, predictions. (3) Evaluation: Because ground-trun object boxes are unavailable, we evaluate only on unseen web tutorials via blinded manual review, assessing whether the predicted action type and parameters are correct and practically actionable. Details are provided in Appendix
optionally condition the detector's aą to stabilize type
2.2.3 INNER MONOLOGUE GENERATION
Dense event detection and action identification recover what happened on screen but omit the stepwise rationale. We therefore generate a brief inner monologue r& before each action to make explicit the intent, the local plan, and the expected state change (e.g., "type query into the search box reveal results," " "scroll bring the 'Settings' button into view"". Explicit rationales provide structured supervision for planning and credit assignment, tie cursor target grounding to goals and affordances, and improve robustness on long-horizon tasks via better error detection and recovery. Recent GUI-agent work that injects step-level "thoughts" or System-2 reasoning reports notable gains perception, grounding, and task execution, motivating our inclusion of r ReAct-style trajectories (Xu et al. 2025b Qin et al. 2025 Wang et al. 2025b).
We cast inner-monologue generation as conditional paraphrasing with GPT-5 Medium. For each step k. we build structured prompt that includes: (i) the detected action type ag; (ii) its parameters mi (e.g.. typed text, cursor coordinates); (iii) the screen state immediately before and after the action (keyframes or thumbnails); and (iv) short ASR transcripts spanning 1-minute window before the action, the during span [t), tg]. and 1 --minue window after. Conditioned on these inputs, the model outputs concise rationale ri: that states the intent, the local plan, and the expected state change
5

---

Published as a conference paper at ICLR 2026
(grounded to visible UI). Additional prompt templates and representative inner-monolouee examples are provided in Appendix
2.3 COMPUTER USE MODEL PRETRAINING
We demonstrate VIDEOAGENTTREX' effectiveness by training an end-to-end computer-use use agent with our video-driven data and high-quality supervised finetuning set. On this strong finetuning basis, VIDEOAGENTTREK improves performance on online and offline agent evaluations.
2.3.1 AGENTIC DATA COLLECTION
VideoAgentrrek Data. We apply VIDEO2ACTION the collected tutorial videos and convert them into agentic supervision. For each processed clip, we (i) run dense event detection to obtain typed, tightly bounded segments, (ii) infer action parameters with the oetion-identificationn recognizer, and (iii) generate brief inner monologue for intent and expected state change. We then assemble the resulting steps (I,, Tk, ag, Tx) into trajectories and serialize them for downstream training. In total, we processed 39,000 videos; each video produces average 39 steps, yielding approximately 1.52 million ReAct steps overall, and about 26 billion training tokens. Detailed data statistics and examples will be provided in the Appendix P
Human demonstrations Data We sample human-annotated trajectories from OpenCUA (Wang al. 2025b) and AGUVIS (Xu et al. 2025b), harmonizing formats and labels into single schema. The corpus spans Windows, macOS, and Android, contributing about 8B tokens to training.
GUI Grounding Data. We include focused subset of GUI grounding pairs from the OSWorld-G dataset (Xie et al. (025a) to strengthen pointer-target alignment and layout-aware perception. This contributes roughly 1B tokens training.
2.3.2 TRAINING STRATEGY.
Automatically mined trajectories, while large-scale, inevitably contain residual noise. Motivated by prior findings that decoupling perceptiin/grounddii from policy learning improves robustness (Xu al. 2025b; Wang et al. 2025b), adopt a two-stage schedule that first stabilizes grounding on broad but imperfect supervision and then consolidates policy on a clean subset.
Foundation Qwen2.5-VI--7B (Bai al. 2025 is general vanionelanguage model with superior vision understanding capability, but it is not sufficiently pretrained on computer-use tasks with an end-to-end success rate of 4.5% on OSWorld (Xie al. 2024), which makes it proper starting point (base) for evaluating the data generated by VIDEOAGENTTREK.
Stage 1 training. We train for one epoch over 26B tokens drawn from the VideoAgentTrek jectories, augmented with small number of GUI grounding pairs. Trajectories are formatted as interleaved vision-text sequences: frames (or rraieeequivalent images) appear inline with the step- wise textual outputs, preserving temporal order across the entire clip. Loss is masked to the textual portions only; images are conditioning context and are not predicted. Please refer to Appendix G for representative formatting examples and complete training configurations including hardware, batch sizes, and optimization details.
Stage 2 training. We continue training for 8B tokens curated set of clean, human-annotated trajectories. Here we reformat the data into chat template with user prompts and assistant responses that describe execute the next action. We apply standard supervised finetuning with loss computed only on the assistant turns, leaving user turns as pure conditioning. Representative formatting examples and training details are provided in the Appendix G

---

Published as a conference paper at ICLR 2026
OSWorld Verified
AgentNet Bench
Task SR(%)
SR(S)
15.78
16
lare
84.1
tranking
staget
10
stogc2
6.
60
base model
stago2
staget + stage2
base model
stageZ
staget + stage2
Figure 5: Experimental Results on OSWorld- Verified (Xie et al. 2025b) and AgentNetBench (Wang) al. 2025b). VideoAgentTrek demonstrates significant improvements over baseline models, with lest-time scaling providing additional performance gains
3 EXPERIMENTS
3.1 COMPUTER USE AGENT PERFORMANCE
3.1.1
EXPERIMENT SETUP.
We evaluate the performance of our model on two computer-use agent benchmarks: OSWorld- Verified (Xie et al.) 2025b 2024) for online settings and AgentNetBench (Wang et al. 2025b) for offline settings. Further protocol, metrics, and computational details are provided in Appendix H
SeWorld-Verifieee OSWorld (Xic et al. 2024 an online computer-use agent evaluation bench- mark that includes 369 human-cratted Ubuntu desktop tasks. OSWorld-Verifee (Xie et al. (025b) a more stable version, with updated evaluation scripts, environments, and clarified instructions, designed to measure CUA's task-solving capabilities in dynamic, real-world environments.
AgentNetBench. AgentNetBench is an offline benchmark is based on 100 representative tasks from the AgentNet dataset, covering wide range of applications and websites Windows and macOS. The tasks are manually refined and offer multiple valid action options for each step to reflect the variety of correct interactions.
3.1.2 MAIN RESULTS.
Video pretraining enhances performance on offline benchmarks. On AgentNetBBnchh, incor- porating VideoAgentTrek pretraining achieves a step success rate of 69.3%, representing 5.2 percentage point improvement over the SFT-only baseline (%) and substantial 30.8 percentage point gain over the base model (38.5%). This consistent improvement demonstrates that video pretraining effectively transfers knowledge to structured offline evaluation scenarios.
Video pretraining delivers greater improvements online henchmarks. On OSWorld-Verinne
our complete approach achieves task success rate of 14.13% demonstrating a 4.83 percentage point improvement (+52% relative) over SFT-only training (9.3%) and more than tripling the performance of the base model (4.5%). These gains are spread fairly evenly across different OSWorld- Verified application domains, rather than being concentrated in a single category.
Video pretraining enables effective test-time scaling for computer-use agents. In our setting, test-time scaling refers to increasing the allowed action-step budget during evaluation while keeping the trained agent fixed. On OSWorld- Verified, the model trained with Stage and Stage 2 improves from 14. 13% to 15.78% under test-time scaling, demonstrating that it can productively use additional interaction budget rather than simply taking more redundant steps. This benefit emerges specifically from video pretraining: models exposed to longer trajectories learn plan over extended horizons, while the SFT-only baseline shows no improvement when evaluated under the same test- time scaling setup (see Section 4.1)
7

---

Published as a conference paper at ICLR 2026
Action
Preds
GT
Precision
Recall
F1
Action type Samples
Accuracy
Click
12,222
14,247
0.88
0.76
0.82
Click
324
0.713
Drag
971
1,462
0.52
Drag
0.366
Press
177
842
0.40
0.08
0.14
Press
47
0,362
Scroll
1,448
1,691
0.93
0.80
0.86
Scroll
34
0.735
Type
1,480
2,040
0.89
0.64
0.75
Тyрe
73
0.671
Total
17,298
20,282
0.88
0.70
0.78
Overall
500
0.658
Table I: Action-even detector evaluation: held- out test-set results by action type.
Table 2: Action parameterization evaluation: manual in-the-wild assessment.
3.2 VIDEO2ACTION PERFORMANCE
3.2.1
ACTION EVENT DETECTION
We assess VIDEO2ACTION with a two-part protocol: held-out, annotated test set and an in-the-wild manual validation:
Held-out test set. We hold out 23 hours of screen-capture videos with 20,282 annotated GUI events. Each event is tuple (type, /9, 1²) A prediction counts hil iff its type matches and its interval has any temporal overlap with ground-truth event; unmatched predictions are false positives and unmatched ground truths are false negatives. We report per-type Precision/Recal//FI and micro/macro aggregates.
Manual validation (in-the-wild). On unseen YouTube tutorials, we apply the same overlap eriterion and estimate recovery rates by human review to assess robustness outside the curated set.
Results. As the results shown in Table Overall precision high (0.88) with solid recall (0.71). Pointer-centric actions (click, scroll) are reliably localized; keystroke-only actions show lower recall/precision due to subtle visual evidence. the manual study, the detector recovers 70% of actions under the same criterion, consistent with in-house results.
3.2.2 ACTION IDENTIFICATION
Evaluating action identification automatically is difficult because target-element boxes are unavailable. therefore apply the identifier in-the-wild videos and perform blinded manual assessment. An action is judged proper if, when executed, it would plausibly produce the observed on-screen transition (for example, the clicked control changes state, typed text appears in the focused field, or the page scrolls). We evaluate 500 predictions sampled across action types.
Results. Annotators review pre/post frames and verify whether predicted parameters explain the observed changes, with disagreements resolved through second-pass review. Performance varies by action type: pointer-based actions (click, scroll) achieve highest accuracy, typing shows moderate accuracy despite OCR noise, while drag/press actions struggle with subtle visual cues. Despite these challenges, the predicted parameters are accurate enough for trajectory construction and downstream training; detailed counts and validation rates appear in Table
ANALYSIS
Step
Performance Scaling
Tank
4.1 EFFECTIVENESS OF DATA SCALING
To assess the impact of Stage-1 1 data scale, we train models using 0%, 50%, and 100% of the video tokens, then apply iden- tical Stage-2 SFT to each variant. With increasing tokens, per- formance scales consistently across both benchmarks. Agent- NetBench step success rates increase from 64.1% to 68. 1% and 69.3%, while OSWorld- Verified task SR @5 grow from 9.3% 13.3% and 15.7% (Figure These findings establish clear
6AX
565
Figure 6: Performance Scaling
8

---

Published as a conference paper at ICLR 2026
relationship between pretraining data size and computer- agent performance, demonstratigg the benefits of scaling video pretraining data.
4.2 IMPROVING LONG HORIZON PLANNING
VIDEOAGENTTREK provides substantially longer trajectories than previous CUA corpora. As illustrated in Figure T3 42.1 % of trajectories exceed 20 steps, while 14.5% contain 50 or more steps. yielding an average trajectory length of 39.25 steps. Cross-dataset comparisons (Table [12 reveal that this average substantially exceeds those of established benchmarks, demonstrating that emphasizes supervision of complex, multi-step workflows rather than brief, single-interee interaction sequences.
TIDEOAGENTTREKCOM
The benefits of varying step budgets. On Dooordd- we observe a striking difference in how models respond to increased action budgets. The Stage2-only model shows no performance improvement when the step budget expands from 20 to 50 steps, remaining flat at 9.3% task success, indicating it cannot effectively plan beyond its training horizon recover from early mistakes. In contrast, after Stage- I pretraining on AideoAgentTrek' ' long video trajectories, the agent demonstrates true test-time scaling: task success rate increases from 14. 13% 20 steps 15.78% 50 steps, + +1.65 point absolute improvement (+11.7% 1.7% relative gain, Figure 5). This differential reveals that exposure to extended video demonstrations during pretraining teaches the model to decompose complex tasks into subgoals, persist through intermediate failures, and leverage additional computational budget for exploration and error correction, capabilities that supervised fine-tuning on shorter trajectories fails instill.
supervision become evident when evaluating planning capabilities under
RELATED WORK
Generating agent trajectories. Computer-use trajectories have been obtained through human annotation, programmatic synthesis in instrumented environments, and web-scale mining of pub- lie resources. Human annotation, often aided by instrumentation log pointer coordinates and keystrokes, yields precise labels but is costly and narrow in coverage (Qin et al. 2025 Wang et al. 2025a b). Programmatic synthesis inside headless browsers or scripted desktop flows generate large volumes with exact parameters, yet coverage is constrained by simulator APIs and may diverge from real-world UI variability (Su al. 2025 Sun et al. 2023). Web-scale mining taps tutorials, RPA logs, and screen recordings to obtain diverse trajectories. but typically lacks precise temporal boundaries or action parameters (Xu al.. 2025a Jang al. (005)).
Precise Event Grounding in Video. Temporal grounding approaches such as temporal action local- ization, moment retrieval, keyframe detection, and dense video captioning seek determine when events take place and provide corresponding descriptions (Lin et al. 2019 2 et al., 2025 Wasim
(ct al. 2024). Meanwhile, recent multimodal systems (c.g., Qwen2.5 VL (Bai et al. 2025). Gemini 2.5 Pro (Comanici al., 2025)) have advanced the field by enabling more detailed spatiotemporal understanding and long-horizon video reasoning. Nonetheless, most general-purpose -puppsse grounding frameworks focus primarily on semantic interpretation, rather than achieving the millisecond- precision and parameter extraction required to faithfully reconstruct GUI interactions.
Learning from unlabeled video to act in environments. VPT demonstrated that large-scale unlabeled videos can be converted into effective training signals (e.g.. via inverse-dynamics auto- labeling followed by behavior cloning), substantially improving agent's ability to act (Baker et al.) 2022).Bur Building on this idea, subsequent work leverages internet-scale human videos to distill human policy priors that transfer to interactive environments, including learning action-centric latent spaces without action labels YYe et al. 2025) and scaling to humanoid control (Mao et al. 2024).
6 CONCLUSION
We presented VideoAgentTrek, a scalable pipeline that transforms publicly available screen recordings into structured supervision for computer-use agents without manual annotation. By developing an inverse dynamics module that accurately detects GUI events and extracts action parameters from raw video, we demonstrate that the implicit supervision in tutorial videos can be effectively harvested at

---

Published as a conference paper at ICLR 2026
scale. Our experiments on 39,000 YouTube videos yielded 1.52 million interaction steps, enabling continued pretraining that improved task success rates by 70% on OSWorld- Verified (9.3% to 15.8%) and increased step accuracy AgentNetBenchh from 64.1% to 69.3%. These results establish that unlabeled internet videos, when processed through learned inverse dynamics, provide a viable and cost-effective alternative expensive manual annotation for training robust computer-use agents. The open-source release of our ScreenFilter and Video2Action tools enables the community to leverage this abundant resource for advancing GUI automation research.
ACKNOWIEDG
The authors of this paper were supported by the ECS (27212023) and Areas of Excellence Scheme (AoE/E-601/24-N) from RGC of Hong Kong.
10

---

Published as a conference paper at ICLR 2026
Reproducibility Statement We have made extensive efforts to ensure the reproducibility of our work. Our pipeline consists of three main components, each with detailed specifications provided in the appendix: (1) ScreenFilter: Training details including the 15,000 synthetic training images, model architecture, and performance metrics (89.58% F1 score) are provided in Appendix C The model processes videos 1-2 fps with cursor detection thresholds and temporal smoothing parameters fully specified. (2) Video2Action: The inverse dynamics module training on 154 hours of OpenCUA data is detailed in Appendix D including the complete action taxonomy, training configuration, and evaluation metrics. Action parametrization methods are described in Appendix E (3) Agent Training: Complete training configurations for both Stage-1 continued pretraining and Stage-2 fine-tuning are provided in Appendix G including learning curves, Megatron-LM framework settings, and tensor/pipeline parallelism configurations. The Video dataset construction process, including YouTube video selection criteria (Appendix [B, data statistics, and quality validation results are documented in Appendix J Upon publication, we will also release a dataset manifest containing YouTube Video IDs for all processed videos and full action metadata for all steps (action types, parameters, inner thoughts, timestamps), together with a reconstruction script that can automatically download and preprocess the corpus from this manifest. In addition, will provide small Creative Commons-licensed subset whose trajectories, screenshots, and metadata can be shared directly, enabling exact replication of Stage-1 pretraining on miniature scale. Evaluation protocols for World Verified and AgentNetBench benchmarks are specified in Appendix H We will release the ScreenFilter and Video2Action tools as open-source software with pretrained checkpoints upon publication. The processed VidcoAgentTrek dataset will be made available subject to YouTube's 's terms of service. All LLLM usage for data auditing and annotation generation has been fully disclosed with prompts provided (Appendix(F) (.)
Ethics Statement We carefully considered the ethical implications of our work in developing VideoAgentTrek. Data Collection and Fair Use: We accessed YouTube videos through public APIs in compliance with YouTube's 's Terms Service using content for non-commercial research purposes under fair use principles. Videos were processed in a streaming manner without permanent storage, and we respect content creators' rights by not redistributing raw video data. Privacy Protection: Our pipeline designed to focus exclusively on GUI interactions and application interfaces, not personal content. The ScreenFilter module specifically detects cursor movements and interface elements while avoiding processing of personal information such as emails, documents, or user credentials visible in recordings. For training, we only use text that is already visible in public tutorial videos and do not attempt to infer hidden content (for example, masked passwords). and for any released manifests or textual metadata we apply conservative PII filters (for example, removing email-like strings or obvious credentials) so that our artifacts do not expose more sensitive information than the original public videos. Potential Misuse and Dual-Use Considerations: We acknowledge that automation technologies can be misused for unauthorized scraping, bot creation, or circumventing application restrictions. To mitigate risks, emphasize that our tools are intended for research and accessibility applications, and we encourage users to respect application terms of service, obtain appropriate permissions for automation, and consider the impact human workers whose tasks might be automated. The released models include documentation on responsible use guidelines. Transparency and Reprodueibility: We provide full disclosure of LLM usage in data processing. Our open-source release enables scrutiny of methods while our evaluation includes manual validation of extracted actions to ensure quality. We believe the benefits of advancing computer-use agents for accessibility, automation, and human-compute computer interaction research outweigh the risks when appropriate safeguards are implemented.

---

Published as a conference paper at ICLR 2026
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL
https://arxiv: www.oogg//s.... 2502 13923
Bowen Baker, lige Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos, 2022. httph:// wwwpexiiii.orvvaooooooeo abs /t.bs/2206. 11795
Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Guozhi Wang, Dingyu Zhang, Shuai Ren, and Hongsheng Li. Amex: Android multi- multi-annotation expo dataset for mobile gui agents. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 2138-211111111 2156. Association for Computational Linguistics, 2025. doi: 10. 18.18653/v 1/2025. wttp:////2......nsaaaaaalll findings--acl.1 URL http://dx.doi. org/10 1ttp:///...11133/112/2
findings acl. 110
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang. Yue Zhao, Chongyi Wang. Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Guicourse: From general vision language models to versatile gui agents, 2025. URL
org/aba/ /406 11317
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, and Ice Pasupat.etc. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL BttLtt:s:///axiitt httptts:/aa.....r////aaaaa 2507. 06261
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web, 2023. URL URL ptetps://arxiv..r abs/2306 06070
Yunseok Jang, Yeda Song, Sungryull Sohn, Lajanugen Logeswaran, Tiange Luo, Dong-Ki Kim, Kyunghoon Bae, and Honglak Lee. Scalable video-to generation for cross-platform mobile agents, 2025, UKULhctpps///axxx..rrr////////// 12632 12632
Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Camphell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on ui control agents, 2024. URL htt.hhttt////rttt// org/ abs//bb//// 2406 03679
Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. Bmn: Boundary-matching network for 09702
temporal action proposal generation, 2019. URL
Quanfeng Lu, Wenqi Shao, Zitao Liu, Lingxiao Du, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, and Ping Luo. Guiodyssey: A comprehensive dataset for cross-app gui navigation mobile devices, 2025. //tte////22..../////////o//o//// 08451
Jiageng Mao, Siheng Zhao, Siqi Song, Tianheng Shi, Junjie Ye, Mingtong Zhang, Haoran Geng, Jitendra Malik, Vitor Guizilini, and Yue Wang. Learning from massive human videos for universal humanoid pose control, 2024. Uttpsttts:///rxiiti http:///awivvvorr/////////////// htwp://///////......///// 14172
Duy Nguyen- Tuong and Jan Peters. Using model knowledge for learning inverse dynamics. In 2010 IEEE international conference robotics and automation, pp. 2677-2682. IEEE, 2010.
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li. Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin I. Liu, and Guang Shi. Ui-tars: Pioneering automated gui interaction with native agents, 2025. URL wtt.htttps//// /ttp//raxvvv// v. or..orrgg.. org 2550/ 12326
Christopher Rawles, Alice Li. Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A scale dataset for android device control, 2023. URL https: :tattss:/aaae htt///axvvvvoo/// abs/ / 2307. 0088
12

---

Published as a conference paper at ICLR 2026
Dillon Reis, Jordan Kupec. Jacqueline Hong, and Ahmad Daoudi. Real-time flying object detection with yolov8, 2024.
www.i.t.gg...........ooo.. abs abt/abas../////// 09972
Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö. Ank. Learn-by- interact: A data-centric framework for self-adaptive agents in realistic environments, 2025. URL
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, and Zhiyong Wu. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis, 2025. htthhttps:/aartt http:///wwww...or////////2.1.../
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang. Hao Zhang, Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang. Hongcheng Gao, Huabin Zheng, Jiaming Li, Jianlin Su, Jianzhou Wang, Jiaqi Deng, Jiezhong Qiu, Jin Xie, Jinhong Wang. Jingyuan Liu, Junjie Yan, Kun Ouyang. Liang Chen, Lin Sui, Longhui Yu. Mengfan Dong, Mengman Dong, Nuo Xu. Pengyu Cheng, Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song. Tongtong Bai, Wei Song, Weiran He, Weixiao Huang. Weixin Xu, Xiaokun Yuan, Xingcheng Yao, Xingzhe Wu, Xinhao Li, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong. Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang. Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yuhao Dong, Yulun Du, Yuxin Wu, Yuzhi Wang, Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang. Zhiqi Huang, Zihao Huang, Zijia Zhao, Ziwei Chen, and Zongyu Lin. Kimi-vl technical report, 2025. URL httpsttps: //tpiivv//axiiv.aaaa org/ abs/ 2504. 07491
Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinghao Zhao, and Guang Shi. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning, 2025a. URL http::ttt////cctt /ttp:///aaiivvvvaa org/ abs abs.bb//g 2509. 02544
Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang. Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao. Yeqiao Fu, Dongchan Shin, Martin Shin, Jiarui Hu, Yuyan Wang, Jixuan Chen, Yuxiao Ye, Danyang Zhang, Dikang Du, Hao Hu, Huarong Chen, Zaida Zhou, Haotian Yao, Ziwei Chen, Qizheng Gu, Yipu Wang, Heng Wang, Diyi Yang, Victor Zhong, Flood Sung, Y. Charles, Zhilin Yang, and Tao Yu. Opencua: Open foundations for computer-use agents, 2025b. htt.:/tttt///aaiie www.:org///ba//o.... wtwpaabb/2000000 09123
Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Video-groundingdino: Towards open-vocabulary spatio-temporal video grounding, 2024. URL
htttss://arxti org/ abs 2211 00901
Michael Wornow, Avanika Narayan, Ben Viggiano, Ishan Khare, Tathagat Verma, Tibor Thompson, Miguel Angel Fuentes Hernandez, Sudharsan Sundar, Chloe Trujillo, Krrish Chawla, Rongfei Lu, Justin Shen, Divya Nagaraj, Joshua Martinez, Vardhan Agrawal, Althea Hudson, Nigam H. Shah, and Christopher Re. Wonderbread: A benchmark for evaluating multimodal foundation models on business process management tasks, 2024. URL https: arxiv. org abs////////// / 2406.13263
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. URL htttps//arxiv..ort 2404.07977
Tianboo Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, and Caiming Xiong. Scaling computer-use grounding via user interface decomposition and synthesis, 2025a.
13227
Tianbao Xie, Mengqi Yuan, Danyang Zhang, Xinzhuang Xiong, Zhennan Shen, Zilong Zhou, Xinyuan Wang, Yanxu Chen, Jiaqi Deng, Junda Chen, Bowen Wang, Haoyuan Wu, Jixuan Chen, Junli
13

---

Published as a conference paper at ICLR 2026
Wang, Dunjie Lu, Hao Hu, and Tao Yu. Introducing osworld-verified xlang. at, July 2025b. URL https:/ xlang. http://ooo//sooo1ddvvaifief...
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and
Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials, 2025a. URL
https://arxinn org.org//aa wtw//a/////oooooooooooo.//t..aoa 2412 09605
Yiheng Xu, Zekun Wang. Junli Wang.
Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction, 2025b.
URL https://arxiv org/ abs/ / 2412.44000 04454
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Confer- on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Seonghycon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Man- dlekar, Reuben Tan, Yu- Wei Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee, Jianfeng Gao, Luke Zettlemoyer, Dieter Fox, and Minjoon Seo. Latent action pretraining from videos, 2025. URL
/ 24tpg///./22211.11111
Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and for gui agents, 2024. https/
Duyu Tang. Android in the zoo:
wttpxii.or//////////2/ 02713
Weijun Zhuang, Qizhang Li, Xin Li, Ming Liu, Xiaopeng Hong, Feng Gao, Fan Yang, and Wangmeng Zuo. Grounding-md: Grounded video-language pre-training for open-world moment detection, 2025. /tttttts///aaa http///ix...../aaaa...a// 14553
14

---

Published as a conference paper at ICLR 2026
A LLM USAGE DISCLOSURE
Scope. We used large language models in three ways: (i) data auditing, (ii) dataset annotation, and (iii) drafting/formating text. All experimental results, figures, and quantitative claims were produced from our own training/evaluation runs and were reviewed by the authors.
Data auditing (metadata classification). A lightweight GPT 1 pass video titles and short descriptions was used to assign coarse content-form labels futoriaa/background/eer talk/other) and OSWorld-style domain tags. These labels supported corpus auditing and analysis and did not directly determine benchmark results.
Dataset annotation (inner monologue). For the inner-monologue component of ReAct trajectories, we prompted an LLM (GPT-5 Medium) with structured evidence (action type/content, pre/post keyframes, and short ASR snippets) generate step-level rationales. Prompts and examples are provided in Appendix F Outputs were constrained by explicit rules (e.g.. .. no coordinates, first-person reasoning) and were filtered by automatic checks (consistency with (ag. Tx). basic PII redaction) followed by human spot-review.
Manuscript drafting and LaTeX. LLM assistant was used to help reorganize and polish prose, and to format LaTeX snippets (e.g.. converting paragraphs to concise task definitions, producing wrapfig/wraptable layouts). All technical content, equations, numbers, and citations were written or verified by the authors; passages were accepted verbatim without human editing.
Coding and experiment orchestration. Training, data processing, and evaluation scripts were authored and run by the authors. LLMs were not used to autonomously orchestrate experiments or to fabricate results.
Human validation. Authors reviewed prompts, sampled model-generated annotations for fidelity to the visual/audio evidence, and checked all reported metrics and tables against logs. If an LLM was used only for grammar or wording, its suggestions were edited and verified by the authors.
No undisclosed assistance. Beyond the uses above, no additional LLM assistance was employed.
B YOUTUBE VIDEO QUALITY STANDARDS
To ensure consistency and usability in selecting high- high-quality instructional videos from YuuTube for research purposes, the following standards must be met:
Minimal Overlays. If overlays, such as face cams or titles, are present, they must occupy more than of the screen area avoid obstructing the primary content.
2. Primary Focus on Screen Recording. The video should predominantly feature clean screen recordings. Brief transitions to other scenes, such as PowerPoint slides or face capture, are permissible but should be limited to introductory or concluding segments.
3. Screen Recording Method. The video must consist of direct screen recordings rather than footage captured by external camera.
4. Language Requirement. The video must be in English to facilitate monolingual captioning in subsequent processing steps.
5. Stable Visual Presentation. Frequent zooming in should be avoided. The entire screen or application window must be visible for the majority of the video duration.
6. Caption Availability, The video must include captions, indicated by the availability of the closed caption (CC) icon in the bottom right corner of the player. Captions may be auto-generated or manually annotated.
7. Orientation. The video must be recorded in a horizontal format, vertical videos often fail to capture complete desktop screens, limiting their utility.
8. Recency. Videos must be no older than five years to ensure that the user interfaces depicted remain relevant and applicable.
These criteria ensure that selected videos are suitable for detailed analysis and processing in research contexts.
15

---

The Published as a conference paper at ICLR 2026
Table 3: Cursor Type Distribution in Training Data
macOS (39 types)
Windows (19 types)
Icon
Type
Icon
Type
%
Acro Move
2.0
Default
25.0
Move (IL)
2.0
t
Hand Point
20.0
Move (L)
2.0
I
Text
15.0
Size (II.)
1.0
Size (L)
1.0
Cell
5.0
Size 2 (II.)
1.0
+
Cross
5.0
Size (L)
1.0
Hand Grab
5.0
6-
Size (II.)
1.0
Hand Open
5.0
Size (L)
1.0
Size (II.)
1.0
Move
5.0
Size 4 (L)
1.0
31 others
15.0
+ 8 default others
86.0
C SCREENFILTER DETAILS
SCREENFILTER is trained on 15,000 synthetic images generated by compositing cursor sprites onto GUI screenshots from the GUIEnv (Chen et al. 2025) dataset. To enhance its generalization across different platforms, we incorporate various cursor patterns from both Windows and macOS. Specifically, collected 39 common cursor types from macOS and 19 common cursor types from Windows. Detailed cursor specifications and their distribution in the training data are provided in Table On the held-out test set, SCREENFILTER achieves F1 score of 89.58% with 90.64% precision and 88.54% recall, demonstrating its effectiveness in accurately separating computer-use content from unrelated material.
For video processing, SCREENFILTER operates at 1-2 frames per second to balance both accuracy and efficiency. The model retains segments where least 80% of the frames contain a cursor for minimum of 6 seconds, with 2-second temporal smoothing gap to merge frames. This design allows SCREENFILTER process approximately 840 hours of video per GPU-day, facilitating large-scale filtering.
D DENSE EVENT DETECTION
Our dense detector is trained on 154 hours of screen-capture video paired with raw interaction logs from OpenCUA (Wang et al. 20256). The logs contain complete demonstrations with precisely timestamped GUI interactions (mouse and keyboard). We convert these logs into prompt-free temporal grounding supervision by mapping low-level events to our action taxonomy, merging short consecutive micro-events into typed spans with start and end timestamps, and discarding segments without actionable GUI operations.
For training-set preparation, downsample videos to 4 fps, segment them into non-overlapping 10s clips, and align the interaction logs within each clip to obtain typed spans with start and end timestamps. We adopt temporal patch size of 2 frames for modeling efficiency. Label names normalized to our action taxonomy. We visualize the data sample in Figure
Model training. We perform full-parameen parameter supervised fine-tuning of training configuration and loss curve are shown side-by-side in Figure g
Evaluation protocol. We evaluate dense event detection the segment level on 0.5 temporal grid and use different overlap criteria for short and long actions. Concretely, we consider the following action types: click, key, write (type), scroll, moveTo, dragTo, doubleClick, rightClick, hscroll, hotkey, tripleclick, middleClick. For short, instantaneous interactions such click, key, hotkey, scroll, and doubleci ick,
16

---

Published as a conference paper at ICLR 2026
Action type
Count
Resolution
Clips
click
410,101
x280x
9,854
key
138,660
1212x 758
5,950
write
80,749
1276x718
1,865
scroll
46,597
1192x772
1,679
move
32,840
1172x782
852
dr
32,840
1188x
604
doubleClick
14,241
x 1440
574
rightClick
7,451
1188x774
471
hscroll
3,411
1192x770
417
hotkey
2,570
1468 x956
182
tripleClick
2,428
other
806
middleClick
Total
23,254
Total 771,945
Table 5: Resolution distribution of dense event detection training clips.
Table 4: Event distribution in the dense event detection training data (154 hours).
"action_type" type": "left_click", "start_tine": "1.500", timet
"action_type" type"" "lect_click", "start_tine" "4,500", "end_tixe": "5.888"
"action_type": "type", "start tine": *7.500", "end_tixe's "18.
Figure 7: Example training sample for the dense event detector.
the raw interaction duration is very brief and the exact delay between input and visible UI change ambiguous. We therefore represent these as short segments aligned the 0.5 grid and count prediction as correct if the predicted segment has any non-zero temporal overlap with the ground-trutt -ruuh segment. For longer actions, especially text-input events (write/ty) (tpe). we require that the predicted segment cover at least 80% of the ground-truth duration, i.e., |intersection|/ so that a reflects capturing most of actual typing episode. will release the evaluation scripts so that future work reproduce and compare event grounding performance under the same cemporal-overlap criteria.
> 0.8,
E
ACTION IDENTIFICATION
Training data.
Our dense detector is trained on 512,000 screen-capture clips paired with raw interaction logs from OpenCUA (Wang et al., (0256). To preprocess action segments, we adopt dynamic frume-rate policy that caps frames per clip 20 while preserving short, fast interactions. For segment of duration At (seconds), we set
= (30, max(4, /00 }t]
then sample frames uniformly within [tg. (g]. This yields, for example, f=30 for brief clicks/scrolls (At zs 15 frames), f=20 for At 1.0 (20 frames), and f=4 for extended typing segments 5 20 frames). We visualize the data sample in Figure
17

---

Published as a conference paper at ICLR 2026
Framework
Megatron-LM
Hardware
32x H100 GPUs
Tensor parallelism
=P=4
Pipeline parallelism
PP=1
Global batch size
256
Training iterations
2000
LR decay iterations
2000
wwwp//////////////////// wwwp www.ereeeeaaaarerreeeeeoo www.eeeeeeeeweeeeeeeeoo
0.5
Wall-clock time
~15 h
523
Figure 8: Dense event detector: training loss (left) and training configuration (right).
"chatit, m
"role": "system",
contentt:
text": are provided with signatures within <tools><</tools> tags'
text": "Here a video clip Gu Interacttio,,, please identify the Interaction"
"role. "assistant". Gontent":
"tott: "ont::"ctoolcallle tattrol.cllllan("ammttttee hwoop//mmmmm---tootteetteeeeee uuett, l'argumentof":
(factionn" ("wetion(-1"ypael ('typll,
('textt: Titt:tttettiiiylloo2222222/oooo
Figure 9: Example training sample for the action parametrization model.
Stage-1/ / Stage-2 consistency handling. For each detected event, Stage outputs action type and temporal span (adet, 18, (g,. while the Stage 2 parametrization model, given the corresponding clip, predicts full tool call with its own action type aaara and parameters (for example, coordinates or text). In post-processing, prediction as the final action; if they disagree, we treat the case as potentially noisy, re-run Stage with the type fixed dett to obtain an alternative candidate, and then ask gpt-5 -itt-5-medinne um, given the pre-/post-action frames and both candidates, which action better explains the observed state transition. The chosen candidate is used as the final label, allowing Stage 2 to benefit from joint type+parameter prediction while using Stage as stabilizing prior in ambiguous cases.
compare these two: if ak para matches adet, directly take the Stage
F
INNER MONOLOGUE GENERATION
Prompt Content.
Inputs:
Action type: a Parameters: mi
Before/after keyframes: 7pre. 7post
ASR windows: [-60s, 0), [t.. 1. [o, 60s]
18

---

Published as a conference paper at ICLR 2026
Instruction (to model):
You are generating inner-monologue annotations for dataset of GUI agent trajectories built from in-the-wild screen recordings.
End-to-end setting.
Source: real GUI screen recordings from the wild.
Extraction: each GUI interaction (an action) is automatically detected from video/au- dio.
For every detected action, you receive three kinds of evidence:
Action details: {action type) and {action content).
action content may contain: coordinates (absolute normalized) and/or bbox; typed text; pressed keys; scroll amount/direction; drag start/end; and similar specifics.
Keyframes: start screenshot and, if available, an end screenshot right after the action executes.
Surrounding transcripts: short snippets of narration speech immediately before, during, and after the action.
Action validation (optional): brief validator description summarizing what occurred.
Your task. For each action, output exactly one JSON object with two fields: action deesoriienni and thought.
Field definitions (strict).
action action description: concise natural-language description of what I do in the UI this step. Name the target UI element if inferable (button, menu, tab, field); otherwise describe by role/label/relative position. Mention the immediate visible outcome only it is clearly observable. Forbidden: raw coordinates, code, function/method names, automatiin tokens, key-value argument lists.
thought: first-person inner monologue (4-8 sentences) as the demonstrator (use "T", "me". "my"". ). Provide substantive reasoning. Include: (a) what aim accomplish and why now; (b) how the speech context informs my intent (weave naturally); (c) brief summary of what likely changes from start end both frames exist; (d) a short breakdown of the atomic actions in this step (e.g., type press) and why each is needed: (e) what expect to verify or do next. Prefer present tense when natural.
General rules.
The thought must in first person; never switch to third person.
Evidence priority: prefer visual evidence from start/end keyframes; treat speech
weak hint for why. If they conflic,, prefer visuals.
Weave evidence naturally without naming "transcripts" "frames."
For coordinate-based actions, red hollow circle may mark the interaction point; do not mention the marker, describe the target element instead.
If only a start keyframe is available, focus on intent; if an end keyframe exists, you may include the immediate visible result.
When a step bundles multiple atomic actions, reason across them as one coherent operation.
Keep action action_description concise; let thought carry the details; avoid hedging and boilerplate.
exactly one valid JSON object with only actionndescription and thought; no extra keys or commentary.
Output format:
Output:
Tx: monor-mono JSON with fields action description and thought.
Qualitative analysis. To further sanity-check the reliability of the generated inner monologue, we conducted a small-scale human review on fixed validation subset of 100 step-wise samples spanning diverse scenarios (for example, macOS Mail, Windows Calculator, macOS Preview, Chrome browsing). For each action step, we examined whether (i) the action description matches the underlying tool call type and target element, (ii) the thought correctly captures the local intent of
19

---

Published as a conference paper at ICLR 2026
the step, and (iii) the expressed intent is reasonably consistent with how the narrator conveys in the transcript. On this subset, 71% steps satisfied all three criteria. Among the remaining 29%, around 7% were due to incorrect action recognition, 12% corresponded to inherently noisy or exploratory operations whose scene transitions were not meaningfully goal-directed, and 10% came from -on-goal-oriented segments with unclear narrator intent, where the inner monologue showed mild hallucination but remained acceptable for pretraining. Overall, 81 % of the reviewed steps were judged qualified for Stage-l pretraining.
Sample inner monologue. Figure 10 shows a representative example from our 20-trajectory valida- tion subset: given a pre-action frame, the automatically detected action, and the post-acting frame, the model generates a step-wise inner monologue that explains the user's 's intent (for example, opening specific photo event in iPhoto), the immediate UT operation, and the ex expected outcome.
las unarl spar
Aller elie dioe oloogg kn oulerie Gumalh
selert one or mole toin us an unok Hesr pioce
Action:
dealnn uheees
Tool call:
Faction: wwarddaaaa 5282 1201)
Figure 10: Example of generated inner monologue for single GUI action. Left: pre-action frame with the target element highlighted. Middle: generated thought and natural-language action description, together with the structured tool call. Right: post-action frame showing the resulting UI state.
G COMPUTER USE AGENT TRAINING
Training Data. visualize the training data samples in stage-1 and stage-2 training in Figure
11
Stage-1 I training. We perform full-parameter continue-pretrainnng Qwen2.5-VL-77--nnnte The training configuration and loss curve are shown side-by-side in Figure 12
Effect of Stage-1 pseudo-label noise on grounding. Stage-1 video continued pretraining designed to inject long-horizon computer-use use knowledge, but it is trained on automatically extracted pseudo- trajectories whose parameters are not perfectly clean. To quantify how such noise manifests in low-level skills, we evaluate all model variants on OSWorld-G. dedicated grounding benchmark that directly probes the ability to localize and select the correct on-screen target. Table summarizes the results alongside oSWorid-eerified and AgentNetBench.
Model variant
OSWorld-G
OSWorld- Ver.
AgentNetBench
WindowsAA
Qwen2.5- Swen2.55VL-78 VL-7B (base)
31.40
4.50
38.5
7,80
Stage-2 SFT only
31.56
9.30
64.1
15.38
Stage- CPT only
26.24
Stage-1 CPT + Stage-2 SET
30.50
15.78
69.3
17.39
Table 6: Effect of Stage-1 video continued pretraining and Stage-2 supervised fine-tuning ground- ing (OSWorld-G) and end-to-end performance across benchmarks (OSWorld- Verified, AgentNet- Bench, Windows Agent Arena with 50-step budget).
Stage-2 training. We continue a full- nallmparrmeter supervised fine-tuning on the stage- 1 1-trained check- point. The training configuration and loss curve are shown side-by-side in Figure 13
H COMPUTER USE AGENT EVALUATION
Evaluation Setting. We follow the OS OSWorld-Veern Verified protocol: the agent interacts with live desktop given a natural- instruction and the full history of prior states and actions. At each step, the policy conditions on the instruction and bounded visual context of up five recent screenshots
20

---

Published as a conference paper at ICLR 2026
Typn'' Tohatee
"souree: videoagenttrek; "lang" 'ee" ddo "d00,42788"; ccontent:]
Natel "apideer
Neath "wtzel Techsroriee evy
"pamind"
Beot": Please generate the next move according to the screenshot, instruction and previous actions.
"eet": "Thought: cllck Recet... button under...
"pastitent" "eerentt
"test" "Thaught The cank goa ia inarrs charting
Teatt coeal cobsindeaannt ('smpueer usaft
"Toought tclikk
cenntnaten 1828 S1i0Wowweee
(a) Stage-1 training
(b) Stage-2 training
Figure 11: Computer Use Agent Training Data (a) Stage-1 I training, (b) Stage-2 training.
Framework
Megatron-
Hardware
32x H100 GPUs
Tensor parallelism
TP=4
08
Pipeline parallelism
PP
Global batch size
512
66
Training iterations
6500
04
LR decay iterations
6500
Wall-clock time
~60 h
02
Figure 12: CUA Stage-1 Training: training loss (left) and training configuration (right).
(FIFO window) together with the action/rationale history, then emits next action. For the 20-step budget, we conduct three independent runs per model and report the average Task SR. For the 50-step budget, we perform a single run. All models use identical inference settings and action executors; no manual interventions allowed during evaluation.
AgentNet Bench summary. Overall step SR rises from 0.385 (base) to 0.641 with SFT-only and to 0.693 with Stage + Stage 2. These trends suggest that video pretraining notably strengthens grounding and multi-action control, especially for less frequent or harder motor primitives.
OSWorld-Verieee summary. Table 7 reports task success across turs and step budgets. With SFT-only (stage2 only), Task SR hovers around 9.1-9.4% at 20 steps and shows no improvement 50 steps (9.27%), indicating limited ability leverage longer budgets. Adding VideoAgent'rrek pretraining (stagel + stage2) raises Task SR to 13.6-14.7% at 20 steps and further to 15.78 % at 50 steps. Per-domain counts improve most for chrome/46 (up to 15 solved) and workflow/92 (up solved), with steady gains in os/24 and authoring apps (writer, impress). Across three 20-step runs, variance is modest, suggesting stable benefits from Stage 1. Overall, the results show that large- seael video pretraining yields higher step quality and makes the agent sensteet-sssiieeeeeeeeeeeeeeeeee to convert extra steps into additional task completions.
21

---

Published as a conference paper at ICLR 2026
Framework
Megatron-LMM
Hardware
64x H100 GPUs
Tensor parallelism
TP=4
09
Pipeline parallelism
PP=1
Global batch size
512
04
Training iterations
3000
077
LR decay iterations
3000
Wall-clock time
116
Figure 13: CUA Stage-2 Training: training loss (left) and training configuration (right).
Model
Eval
Task SR (%)
cale
chrome
gimp
vscade
writer
third
os
impress
workflow
vlc
tumi (20)
9.42
2.2
17.4
11.5
26.1
8.7
20.0
8.3
6.4
5.4
5.9
turn2 (20)
9.13
4.3
3.8
26.1
13.3
6.4
5.4
11.8
stage2 only
tum3 (20)
9.42
2.2
17.4
7.7
26.1
8.7
13.3
8.5
5.4
11.8
turn4 (50)
9.27
2.2
26.1
7.7
13.0
8.7
13.3
8,3
8.5
4,3
5.9
turni (20)
14.68
4.3
28.3
7.7
26.1
21.7
40.0
16.7
12.8
7.6
11.8
turn2 (20)
13.57
4.3
28.3
7.7
21.7
21.7
40.0
12.5
12.8
5,4
11.8
stagel + stage2
tur3 (20)
14.13
4.3
26.1
7.7
30.4
30.4
40.0
12.5
10.6
5.4
11,8
turdd (50)
15.78
2.2
32.6
7.7
26.1
26.
40.0
16.7
12.8
8.7
17.6
Table 7: OSWorld- WSrriie-Verified full results. Cells indicate per-application success rates (percent of solved tasks in each bucket).
VIDEOAGENTTREN DATA ANALYSIS
Resolution and scale. We downloaded 55,603 screen-capture videos (about 10,000 hours) from 50+ channels. The corpus is predominantly clear: 97% are 720p or higher (Table 2 Most videos are minutes long, providing sustained interactions suitable for dense detection and action identification.
based Title/description-baa based content classification. To quickly audit topical relevance at scale, we apply lightweight classifier each video's title and brief description.
Labels.
Autorial:: hands-on screen tutorials.
* Include: step-by-step demonstrations, cursor-drived walkthroughs, "how to to... tasks; fre- quent UI focus changes; imperative phrasing in titles ("Create..." "Install..." "Fix... Exclude:
talk-style narrations with few concrete on-screen actions; marketing teasers without real steps.
Signals: verbs tied to UI operations (open, click, type), timestamps/chapters per step, tool/app names plus action verbs.
B background: expository background with incidental screen use.
Include: market share reports, product overviews, concept explainers where the desktop appears only backdrop.
* Exclude: segments that actually show multi-step operations (move to A. .ttutorial..
* Signals: nouns like "overview, history, comparison, review," charts/stats title/descrip- tion, little or cursor interaction.
tech talk: talks or presentations with slides.
* Include: conference talks, webinars, lectures; slide navigation with limited live demos. Exclude: talks that transition into substantial live step-by-step demos (then split relabel Aututorial).
* Signals: "keynote, webinar, seminar, lecture," speaker names/affiliations, slide thumb- nails.
D. unrelateed: off-topic for computer-use learning.
* Include: content where a screen appears but no actionable computer-use use task is taught (e.g., pure entertainment, face-cam only).
22

---

Published as a conference paper at ICLR 2026
Model
Step SR
click
write
press
scroll
move
drugTo
botkey
dbClick
rClick
terminate
base
0.385
0.402
0.605
0.286
0.615
0.189
0.000
0.250
0.000
0.000
0.188
stage2 only
0.641
0.671
0.719
0.500
0.300
0.145
0.484
0.526
0.214
0.588
stagel
0.693
0.767
0.733
0.441
0.600
0.502
0.264
0.562
0.650
0.417
0.237
Table 8: AgentNet Bench: step success rate (overall and per action type). "_" indicates the metric was not applicable/recorded.
Resolution bucket
Count
High (1080p+)
2.322
Standard (720p-1080p)
49,589
Low (<720p)
1,464
Table 9: Resolution distribution of downloaded videos.
Exclude: any clip with consistent stepwise UI operations (move to Atttooral). Signals: lifestyle/vlog tags, gameplay without UI instruction, no app/task keywords.
Procedure. Single-pass GPT-4.1 classification with short instruction to choose exactly one of the four labels given the title and short description; no transcript or frames are used. We the result only for corpus auditing, tag mining, and optional down-weighting in later filters, not as hard accept/reject gate.
Limitations. Metadata-only classification mislabel borderline cases (e.g., talks that include substantial demos). Final training sets are still screened by cursor gating, license/PII checks, and downstream detectors.
Distribution. Class counts and shares are shown in Table 10 The majority are tutorials, indicating strong alignment with our target use case.
Action type
Count
Share (%)
left click
1,037,617
67.1
Label
Count
Share
type
214.816
13.9
key
145,860
9.4
tutorial
38,700
69.6%
scroll
111,203
7.2
background
12,900
23.2%
right_click
24,111
1.6
talk
2,391
4.3%
double_click
11,848
0.8
D Duunrelated
1,612
2.9%
mouse_men mooe
8,441
Total
55,603
100%
drag
6,372
0.1
hscroll
196
0.0
Table 10: distribution from title/decrription clas- sification.
Total
1,547,092
100.0
Table 11: Action distribution in the VideoAgent- Trek agentic dataset.
Action distribution. Table II summarizes action counts the VideoAgentTrek agentic data.
Cross-dataset comparison. We summarize reported average step counts (and task counts when available) for common CUA datasets and include our corpus for context.
Benchmark contamination analysis. Our pretraining corpus is mined from forg-form tutorial videos, which differ conceptually from the goal-conditioned tasks in OSWorld and AgentNettenchh tutorials typically demonstrate range of operations on ad-hoc content (for example, showing several Excel functions on randomly created tables) rather than executing a single explicit instruction toward precise target state. To visualize this gap, encode all tutorial video titles and all OSWorld task instructions into shared text-embedding space and project them into 2D (Figure 14). WoWlld instructions form several relatively compact clusters that cover only a small portion of the overall tutorial distribution, while the video titles are widely dispersed across many unrelated domains and workflows. We also verify that none of the titles contain the strings "OSWorld" or "AgentNetBBnch",
23

---

Published as a conference paper at ICLR 2026
Dataset
Tasks
Avg. Step
AndroidControl (Li al. 2024
15,283
5.5
AMEX (Chai al.
2,991
11.9
AitW (Rawles ct
2023
2,346
8.1
Proportion
AitZ Zhang ct al.
2024
1,987
6.0
GUI Oulsssy (Lu ci
2025
7,735
15.3
OS-Genesis (Sun al.
2025
2,451
6.4
WonderBread WWorow
2024)
598
30%
AgentTrek (Xu al.
2025a
10,398
12.1
Mind2Web (Deng el al.
2023
2,350
7.3
Stop num
GULAct Chen et 2025
2,482
6.7
AgentNet (Wang et
2025b
22,625
18.6
20
100
200
gineekgentTrek
39,000+
39.251
Table 13: VideoAgentTrek data distribu-
tion of step number
Table 12: Average steps across datasets (as reported in their papers). Estimatedd from 5,416-traj sample our corpus.
and a manual inspection of 200 videos nearest to OSWolld clusters reveals no obvious task duplication, suggesting that VideoAgentTrek primarily provides diverse, generic computer-use supervision rather than narrow memorization of benchmark tasks, although residual overlap cannot be entirely ruled out.
Figure 14: Text-embedding visualization of tutorial titles (gray) and OSWorld task instructions (colored) in shared 2D space. Left: topic clusters of the tutorial corpus. Right: OSWorld instructions overlaid on the same distribution.
J LIMITAION AND FUTURE WORK
Limitation. VideoAgentTrek is currently constructed from 2D desktop screen recordings with predominantly English user interfaces. While this setting already covers broad range of everyday computer-use workflows, it does not directly address mobile-only interaction patterns (for example, touch gestures and soft keyboards) or non non-Englise English UIs with different scripts, fonts. and themes. In addition, OSWorld-style benchmarks highlight that robust OCR and text-centric perception under diverse themes can further improve downstream performance. In this work, we intentionally focus on the iico-to-acctin pipeline itself, that is, how to transform large-scale unlabeled screen-capture videos into high-quality pretraining trajectories, and rely the underlying VLM/OCR backbone for text recognition, rather than explicitly optimizing OCR or eesolutton-speciine components. In principle, the same pipeline can be extended beyond our current scope by adapting the sereen filtering and action extraction modules (for example, replacing cursor-based filters with touch-indicator detection, enabling multilingual OCR/ASR, and incorporating mobile-specific GUI priors). Scaling
24

---

Published as a conference paper at ICLR 2026
VideoAgent'Tek mobile platforms and non non-English environments, together with more explicit OCR-enhanced variants, is an important direction for future work.
Future Work Our work opens up several clear directions for future research. First, would like to better understand the behavior of the current VideoAgen!Trek pipeline itself, for example by running synthetic corruption studies (jittering click coordinates or corrupting a controlled fraction of parameters), performing more systematic data and model ablations (such as isolating the contribution of inner monologue), and exploring confidence-aware filtering quantify and mitigate the impact of noisy supervision. Second, we aim to further improve the inverse dynamics module by going beyond purely visual signals, augmenting it with richer auxiliary cues such as ASR transcripts and keystroke audio, and potentially combining these with lightweight hybrid heuristics where they reliable. Third, we plan to broaden the scope of VideoAgentTrek by extending the video-to-action pipeline to mobile platforms and non-English user interfaces, and by applying VideoAgentTrek pretraining to additional vision language backbones when computational resources allow.
25


Reviews:
[{"review_id": "IdbShqA93e", "reviewer": "Reviewer_Rbqb", "summary": "The paper proposes VideoAgentTrek, a scalable pipeline to pretrain computer-use agents from unlabeled, publicly available screen-recorded videos. The key challenge is that raw videos  lack structured action labels (types, timestamps, and parameters). The authors address this by introducing an inverse dynamics module to generate the parameterized action tuples. With the collected dataset and the proposed multi-stage training strategy, the final performance of the GUI Agent achieved SOTA.", "questions": "1. See the weakness.\n\n2. Have you tried ensembling multiple parameter predictors or using confidence-based filtering to drop uncertain steps? How does selectively pruning low-confidence parameterizations affect Stage-1 benefits?\n\n3. Can you quantify how coordinate noise (e.g., ±k pixels) affects downstream success on benchmarks via a controlled perturbation study?\nImproving non-pointer actions:\n\n4. For press/type, did you explore fusing ASR transcripts or keystroke sounds to improve recall? Are there simple heuristics (e.g., stable cursor + text change) that boost detection without heavy supervision?\n\n\n5. Beyond releasing filter tools, can you also share:\n- Video ID lists and segment timestamps (without frames), with pass/fail reason codes.\n- A small, license-cleared demo subset (e.g., Creative Commons) to enable exact replication of Stage-1 on a miniature scale.\n\n6. Did video-pretraining improve performance uniformly across OSWorld app buckets (calc, chrome, vscode, etc.), or are gains concentrated in certain domains? \n\n7. Since mined text can include sensitive inputs (typed passwords, emails), how to deal with?\n\n8. Could you incorporate UI element detection (e.g., layout parsing) to normalize parameterization into element-centric actions rather than screen coordinates, improving cross-resolution robustness?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Interesting and reasonable reframing of GUI action recovery as inverse dynamics from raw videos without (great) manual labels.\n\n2. The filter system leverages cursor detection to automatically focus on GUI-heavy segments; channel-coherence expansion for source discovery is practical and scalable.\n\n3. Clear improvements over a strong SFT-only baseline on both OSWorld-Verified and AgentNetBench, plus analysis showing benefits grow with pretraining scale and planning horizon.\n\n4. Pipeline is well-structured and explained (collection → filtering → detection → parameterization → rationale → training).\n\n5. Practical tools and release plans (if they actually release finally) can catalyze broader community work.", "weaknesses": "1. While Stage 2 SFT mitigates noise (Parameterization accuracy, detection errors, etc), a more systematic study quantifying how parameter errors propagate to agent performance (e.g., via controlled corruption of parameters) would clarify robustness and failure modes.\n\n2. The dense detector shows weaker recall for keyboard/press actions, which can be crucial in many workflows. It would help to analyze how missing non-pointer actions affects task categories (e.g., auth flows, terminal usage) and to explore augmentations (ASR cues, keystroke audio hints) to close this gap. Also, since the author uses the pointer as one of the  video filters, I am very curious whether the trained model would bias to such pattern? Further discussion and analysis would help. \n\n3. More detailed release of manifests (video IDs, timestamps, filtering decisions) would further improve reproducibility within ToS boundaries.\n\n4.  The pipeline currently focuses on 2D screen recordings. Some OSWorld tasks involve subtle text/element detection under variable themes or require OCR robustness. An ablation with explicit OCR enhancements or higher-res sampling could clarify performance ceilings. Similarly, mobile platforms or non-English UIs are out-of-scope; discussing adaptation strategies would strengthen the broader impact story.\n\n5. In Table 9, the authors provided a cross-dataset comparison, which is good; But I do think there misses some important datasets. I would prefer a more complete comparison as done in AgentNet, Table 2. This will help the readers to localize. \n\n6. From the Figure, it seems the \"test-time scaling\" helps a lot, but I cannot find the further details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lbtq8kr1yM", "reviewer": "Reviewer_b4VU", "summary": "The paper introduces VideoAgentTrek, which synthesizes GUI interaction trajectories from actionless videos to replace costly manual annotations. The core is the Video2Action inverse dynamics module: it first localizes events, then infers action parameters, and finally synthesizes a chain-of-thought. Training proceeds in two stages: supervised fine-tuning on the full dataset, followed by fine-tuning on a human-verified subset. Experiments report absolute gains of +6% on OSWorld-Verified and +5% on AgentNetBench. The authors release the ScreenFilter and Video2Action tools but do not release the video dataset or the trajectory dataset.", "questions": "- Tutorial videos often contain a lot of noise (e.g., extraneous/meaningless mouse movements, hotkey usage). Is there any specific mechanism to handle this? For text-input actions, how does the model differentiate between a string-level typing action and individual key presses?\n- How are frames with no user action but visible UI changes handled? Does the Video2Action model have an explicit noop action?\n- Cursor icons can vary across operating systems and applications. What cursor types are supported by the cursor detection model?\n- Do event detection and cursor detection models require a standardized resolution/frame rate? How robust is the system across varying FPS and resolutions?\n- Can you provide an ablation on the effect of automatically generated inner monologue?\n- Is there a plan to release the full video dataset and extracted trajectories? Public release, especially of high-resolution videos with audio, would substantially strengthen the paper’s contribution. I would increase my score if this were to become available.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to follow. The data processing pipeline, training setup, and results are clearly presented.\n- Applying VPT-style ideas to GUI agents is intuitive and scalable for data collection, and the idea is well-executed.\n- Clear improvements on two popular benchmarks.\n- ScreenFilter and Video2Action are released to support the reproduction of the video annotation pipeline.", "weaknesses": "- While tools are open-sourced, the full video corpus and trajectory annotations are not available, which I believe could significantly increase the contribution of this work.\n- Potential data leakage is not quantified. The training data derived from public tutorials may overlap with OSWorld/AgentNetBench tasks, but the paper does not appear to include a rigorous deduplication or leakage analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l1AMhFR7n8", "reviewer": "Reviewer_wmub", "summary": "This paper presents VIDEOAGENTREK, a scalable pipeline that automatically extracts GUI-action trajectories from unlabeled screen-capture videos (39 k YouTube tutorials → 1.52 M steps) and uses them for large-scale pre-training of computer-use agents. A learned inverse-dynamics module (VIDEO2ACTION) first localizes actions in time (event detector, F1=0.78) and then infers their parameters (click coordinates, typed text, etc.; 65.8 % human-verified accuracy). Continued pre-training on the mined data followed by supervised fine-tuning improves task success on OSWorld-Verified from 9.3 % (SFT-only) to 15.8 % (+70 % relative) and step accuracy on AgentNetBench from 64.1 % to 69.3 %. The authors release SCREENFILTER and VIDEO2ACTION as open-source tools.", "questions": "This paper presents a timely and impactful contribution by introducing VIDEOAGENTREK, the first fully-automated pipeline that converts unlabeled, publicly available screen-capture videos into large-scale, training-ready trajectories for GUI agents. By equipping an inverse-dynamics module (VIDEO2ACTION) with dense event detection and parameter extraction, the authors bypass the expensive manual-annotation bottleneck and demonstrate clear downstream gains on both online and offline benchmarks. \nOne open question remains: will the complete codebase (SCREENFILTER, VIDEO2ACTION training & inference scripts, data-preparation pipeline) and the processed VideoAgentTrek dataset be publicly released?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novel, timely problem: Leveraging the enormous volume of passive screen-capture videos for GUI-agent training is an appealing idea that addresses the current data bottleneck.\nEnd-to-end pipeline: From raw YouTube crawl to executable (screenshot, action, parameters) tuples, the system is fully automated and scales to web size.\nStrong empirical gains: Clear, statistically meaningful improvements over a pure SFT baseline on two independent benchmarks, plus positive scaling curves with data volume and test-time compute.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y6azaAzfBu", "reviewer": "Reviewer_c9gy", "summary": "Paper introduces a scalable pipeline, VideoAgentTrek to automatically mine training data for use in training of computer-use agents.\nThe work aims to address the challenge costliness of manually annotating such training data at scale. It does so by exploiting the large amount of screen-recorded tutorials available online and introduces a means of extracting explicit structured action labels from such unlabeled videos.\nTo achieve this, the authors designed VIDEO2ACTION, a two component inverse dynamics module (IDM) consisting of a video grounding model and a action content recognizer.\nThe module first performs dense action-event detection to segment clips and assign action labels, then a action parameterization model analyzes these segments to produce structured parameters such as pointer coordinates, typed text etc.\nUsing 39000 YouTube tutorials, the pipeline generates 1.52M interaction steps for large-scale pretraining.", "questions": "Were the inner monologue used in the training?\n\nIn the first stage of action event detection, type a_k is predicted with the timestamps, why train the action parameterization model to again predict the type (line 245 - 247).\\\nit is mentioned in line 247 \"when available, we optionally condition on the detector’s $a_k$ to stabilize type predictions.\". How does this affect the performance of the parameterization model?\n\nFor the Stage 2 training, since it's stated that training was done on \"curated set of clean, human-annotated trajectories\" (line 310), were this the ones samples from Open CUA and AGUVIS.\nIf so how does the performance change if for stage 2, training data was drawn from only VideoAgentTrek generated trajectories and/or using a mix of human annotated and generated.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The scalability of the method, producing over a million structured steps providing a way to generate large amount of annotated data required for training.\n\nClear establishment of need for large scale data via performance scaling evaluation to support motivation and need for automated scalable data collection. \n\nInclusion of cognitive-style reasoning using inner monologue generation process to extract rationale for steps i.e explicit the intent, the local plan, and the expected state change, enhances model interpretability and could improve models reasoning.", "weaknesses": "It could be beneficial to show the performance across more vision-language models to see if the improvements are generalizable.\n\nFor the generation of inner monologue while effective, reliance on LLM-generated text raises concerns about consistency and reliability of the rationales. Were there any analysis on the outputs? \nQuantitative human study on a small subset could provide some insight.\n\nIn the performance evaluation of Action Event Detection, what is the threshold for temporal overlap to count as a hit?\n\nMinor comments not affecting rating:\nLine 266: \"ASR\" define abbreviations\\\nline 376 typo \"iff\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)