D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI
Authors: Suhwan Choi
Jaeyoon Jung
Haebin Seong
Minchan Kim
Minyeong Kim
Yongjun Cho
Yoonshik Kim
Park Yu Been
Youngjae Yu
Yunsung Lee
Venue: ICLR 2026 Poster (2026)
Keywords: embodied ai
vision-language-action models
inverse dynamics models
Primary area: foundation or frontier models, including LLMs
TL;DR: Desktop gaming data effectively pretrains embodied AI: 152× compression via OWA Toolkit, YouTube pseudo-labeling with Generalist-IDM, achieving 96.6% on LIBERO manipulation and 83.3% on CANVAS navigation with 1.3K hours of data.

Abstract: Large language models leverage internet-scale text data, yet embodied AI remains constrained by the prohibitive costs of physical trajectory collection.
Desktop environments---particularly gaming---offer a compelling alternative: they provide rich sensorimotor interactions at scale while maintaining the structured observation-action coupling essential for embodied learning.
We present D2E (Desktop to Embodied AI), a framework that demonstrates desktop interactions can serve as an effective pretraining substrate for robotics embodied AI tasks.
Unlike prior work that remained domain-specific (e.g., VPT for Minecraft) or kept data proprietary (e.g., SIMA), D2E establishes a complete pipeline from scalable desktop data collection to verified transfer in embodied domains.
Our framework comprises three components: (1) the OWA Toolkit that unifies diverse desktop interactions into a standardized format with 152× compression, (2) the Generalist-IDM that achieves strong zero-shot generalization across unseen games through timestamp-based event prediction, enabling internet-scale pseudo-labeling, and (3) VAPT that transfers desktop-pretrained representations to physical manipulation and navigation.
Using 1.3K+ hours of data (259 hours of human demonstrations and 1K+ hours of pseudo-labeled gameplay), our 1B-parameter model achieves 96.6\% success on LIBERO manipulation and 83.3\% on CANVAS navigation, matching or surpassing models up to 7$\times$ larger, such as $\pi_0$ (3.3B) and OpenVLA (7B).
These results demonstrate that sensorimotor primitives learned from digital interactions transfer effectively to real-world physical tasks, establishing desktop pretraining as a practical paradigm for embodied AI.
All resources are publicly available at https://worv-ai.github.io/d2e.

Published as a conference paper at ICLR 2026
D2E: SCALING VISION-ACTION PRETRAINING ON DESKTOP DATA FOR TRANSFER TO EMBODIED AI
Suhwan Choi' MAUMAI
Jaeyoon Jung MAUMAI
Haebin Seong" MAUMAII
Minchan Kim MAUMAAI
Minyeong Kim Stanford University
Yongjun Cho MAUMAAI
Yoonshik Kim MAUMAAI
Yubeen Park MAUMAA
Youngjae Yu' Seoul National University
Yunsung Leef MAUM.AI
ABSTRACT
Large language models leverage internet-scale data, yet embodied Al remains constrained by the prohibitive costs of physical trajectory collection. Desktop anviromments-pariiullaay gaming-offer a compelling alternative: they pro- vide rich sensorimotor interactions at scale while maintaining the structured observation-acton coupling essential for embodied learning. We present D2E (Desktop to Embodied AI), framework that demonstrates desktop interactions can serve as an effective pretraining substrate for robotics embodied Al tasks. Un- like prior work that remained domain-specifee (e.g.. VPT for Minecraft) kept data proprietary (c.g., SIMA), D2E establishes a complete pipeline from scalable desktop data collection verified transfer embodied domains. Our framework comprises three components: (I) the OWA Toolkit that unifies diverse desktop in- teractions into a standardized format with 152x compression, (2) the Generalist- IDM that achieves strong zero-shot generalization across unseen games through timestamp-based event prediction, enabling internet-scale pseudo-labeling, and (3) VAPT that transfers desktop-pretrained representations to physical manipula- tion and navigation. Using 1.3K+ hours data (259 hours human demon- strations and IK+ hours of pscudo-labeled gameplay), our IB 1B-parameter model achieves 96.6% success on LIBERO manipulation and 83.3% on CANVAS nav- igation, matching surpassing models up 7x larger, such To (3.3B) and OpenVLA (7B). These results demonstrate that sensorimotor primitives learned from digital interactions transfer effectively real-world physical tasks, estab- lishing desktop pretraining as a practical paradigm for embodied AI. All resources are publicly available atttpps: /ttp://wrvvaaa htt.:aaii.giaaata. github. io/dZe/
INTRODUCTION
Large-scale datasets have driven recent progress in large language models (LLMs) Kaplan et al., 2020 Hoffmann et al. 2022). where pretraining on internet-scale resources enables strong gener- alization across diverse downstream tasks. In contrast, embodied Al has yet to experience such scaling breakthrough. Unlike text, which can be collected from the web with minimum effort, em- bodied trajectories demand specialized hardware, costly human operation, and complex pipelines for annotation (Mandlekar et al.] 2019 Qin al.] [023) Fu et al. 2024] Cheng et al., 2024: Park al. 2024). As result, most existing datasets remain relatively small, domain-specific, and frag- mented across incompatible formats (Geng et al. 2025). preventing the emergence of a true "data flywheel" for embodied AI.
Desktop interactions screen, keyboard, and mouse offer a compelling alternative to scale vision- action learning (Baker al. 2022) Raad al.) 2024).. These interactions abundant, as desktop interfaces are well standardized and human- centric. Millions of users generate rich interaction tra- jectories through everyday digital activities. Crucially, desktop environments preserve the tight ob-

---

Published as a conference paper at ICLR 2026
Duta Scalability
Existing Approaches (e-g. DROID)
Desktop Embodied Al (Ours)
Finctune
Pretrain
Desktop
Robot
Expeasive
Low Diversity
Hard to Stale
Chrap
High Diversity
Easy Scale
OWA Toolkit
Generalist-IDM
VislonAcction Pretraining
Step Collect Human Domonetrations
Step Train Dynamics Model ASP e Non Elest Prodction wii Tespersi Ofter
Step Vision-A Foundation
NEP: Nazt-Ewer Prodiction
New Datasets (Uper World EPS, Platicmet,...
Teupprri Offici T
GEA
Ratibow Singe X
IDM with Strong OOD Generalicntion
Foundation Model with Strong Generalization
Step Generate Pacudo-Labels
Step
Downstream
Minecroft
YouTube Videos
LIBERO citen)
CANYAS Oimi
SoIOI (Real)
Presde Label-Labels (Keyboord Everts)
Cytrrpunk
OW Moop
Format
PURG
HDDM
Iub
Sart
High Quality Demonstratios
IDM--abbeled Videos
Models with Strong Devonstream Performannce
Figure 1: Overview of D2E framework. (1) The OWA Toolkit captures 335.6 hours of rich desktop demon- strations across 31 games, while the OWAMcap format achieves 152x compression compared to existing formats. (2) The Generalist-IDM uses next-event prediction with temporal offset (NEP-T) to achieve OOD generalization, enabling pseudo-labeling of IK+ hours of YouTube gameplay.
(3) Vision-Action Pretraining pretrained representations to embodied AL, achieving 96.6% success on LIBERO manipula- transfers desktop-pretrained
tion and 83.3% CANVAS navigation benchmarks which demonstrates desktop-to-robotics transfer.
servation-action coupling essential for embodied learning, while abstracting away hardware-specific constraints (Tang al. 2025 Shridhar al. 2020, Raad et al.) 2024). Gaming interactions, in par- ticular, exhibit complex sensorimotor patterns navigation, object manipulation, and strategic plan- ning-thi that mirror many embodied AI challenges, and are widely available internet scale through shared gameplay videos.
We introduce D2E (Desktop Embodied AI), a framework that systematically transforms desktop interactions into scalable pretraining substrate for embodied Al (Figure IT. D2E addresses two fundamental challenges: establishing a unified pipeline for high-quality desktop data collection and leveraging vast amounts of unlabeled internet videos beyond manual annotation.
Our first contribution, the Open- World Agents (OWA) Toolkit, provides the infrastructure for scalable desktop data capture. Built on Windows APIs and GStreamer (Microsoft Corporation] GStreamer Team), OWA's ocap recorder synchronizes multimodal treams--screen keyboard, and mousc-into time-aligned events, while our OWAMcap format achieves order-of-magnitude com- pression improvements over existing formats. Through OWA, we collected 335 hours of human demonstrations across 31 diverse games and applications, establishing foundation for desktop- based pretraining. Desktop data collection is substantially more cost-efficient than robotics data acquisition. 14 annotators collected our 335-hour corpus in one month, whereas DROID (Khaz-l aasky et al. 2024) required 50 collectors across 13 institutions over 12 months to assemble a dataset of comparable scale.
Beyond human demonstrations, we introduce the Generalist Inverse Dynamics Model (Generalist-IDM) demonstrate pathway toward internet-scale data collection. By reformulating action prediction as timestamp-awe next-event prediction (NEP-T, introduced in Section 4.2 our model achieves strong zero-shot generalization substantially outperforming specialist baselines on unseen games with minimal compute requirements. This generalization capability enables automatic pseudo-labeling of You'ubbe gameplay videos, expanding our dataset by over 1, 000 hours.
We demonstrate that desktop-pretrained representations transfer meaningfully to physical robotics through Vision-Action PreTraining (VAPT). Models pretrained on our combined desktop corpus show consistent improvements on standardized benchmarks: It achieves total success rate of 96.6% on LIBERO manipulation (Liu et al.) 2023) and 83.3% on CANVAS navigation (Choi et al. 2024). These results establish, for the first time, that the sensorimotor patterns learned from desktop in- teractions can directly enhance performance in embodied AI domains, validating desktop data as a practical alternative to costly physical data collection.
2

---

Published as a conference paper at ICLR 2026
Our contributions are threefold:
1. OWA Toolkit: A framework that contains ocap for synchronized event recording with FHD/QHD 60 Hz support, OWAMcap format for compact storage, and an optimized data pipeline for ML training training-achievinng achieving up to 152x compression and 41 x lower average disk read per image compared to TorchCodec; used to collect 335 hours of human demonstra- tions.
2. Generalist-IDM: An inverse dynamics model that outperforms game-specific Specialist IDMs, exhibiting out-of-domain generalization and in-context adaptation (e-g., calibrating mouse scale). Trained on OWA-collected data with around 192 H100-hours (~ $800), the strong generalization of Generalist-ID IDM allows us to pseudo-label over IK+ hours of YouTube gameplay.
3. VAPT foundation model: A vision-action pretrained model trained on 1.3K hours of desk- top data from OWA and Generalist-IDM pseudo-labeling, transferring desktop knowledge roboties. VAPT achieves 96.6% success on manipulation (LIBERO) and 83.3% on navi- gation (CANVAS).
RELATED WORK
Collecting Data for Vision-Action Pretraining. Large-scale vision-action (or vision-language- action) pretraining depends on multimodal corpora that pair perception with grounded actions across
diverse tasks (Kaplan et al. [2020] Hoffmann et al.) [022). Recent embodied agents unify perception
and control in a single model across heterogeneous domains (Reed al., 2022 Firoozi et al.) [202]]
Wen et a..). In robotics, resources are emerging: RT-1 (Brohan et al. 2022) and RT-2 (Zitkovich
et al. 2023) scale vision-language-actinn
to real robots; Open X-Embodiment aggregates heteroge- neous datasets to train RT-X models Neill et al. (024)): and LeRobot (Cadene et al. 2024) lowers the barrier to collecting and reusing real-world datasets. Despite this progress, assembling real-robot interaction at meaningful scale remains challenging because of fragmented tooling, hardware over- head, and safety constraints (Xing et al., 2025 Park et al., 2024 Gieng et al.] 2025). Similarly, desktop interfaces lack open, standardized corpora and toolkits, bottlenecking vision-action pre- training (Tagg et al. 2025 Chen et al.) 22025.. VPT (Baker et al. (022) offers human-annotated and pseudo-labeled Minecraft trajectories but remains single-domain, while SIMA (Raad et al., (024) demonstrates cross-game generalization through a unified interface yet keeps data proprietary. PLA- ICraft (He et 2025) advances multimodal Minecraft logging, but these efforts are environment- specific; broad cross applicappionnnonn generalization requires unified schemas that cover diverse desktop applications (McCarthy al., 2025). Unlike prior single-domain or proprietary efforts, we con- tribute an open, unified, multi-game (alliggammm desktop-action dataset (31 games; 335h) and an open-source toolkit, explicitly validated for transfer to embodied tasks.
Inverse Dynamics Models. Agents observe the states up to time - 1 and predict the action at time t. In contrast, Inverse Dynamics Models (IDMs) condition surrounding states- pastt and future-to infer the action taken at time t. IDMs have been pivotal for scaling imitation learning Internet-scale datasets, serving as pseudo-labelers for otherwise unlabeled action data (Ye al., 2024 Bjorck et al. 2025). In robot manipulation, UniPi (Du et al.) 2023) explores text-guided video generation couple language grounding with policy learning, and LAPA (Ye al.) (024) shows that latent action pretraining from videos can improve scalability and robustness. On the desktop side, VPT (Baker et al., 2022) trained a Specialist IDM on human-annotated Minecraft trajectories and used it to pseudo-label thousands of hours of Minecraft gameplay on YouTube We demonstrate the potential of Generalist-IDM. spanning multi-game, desktop-wide settings (McCarthy et al.) 2025), Our design also differs from common tick-based IDMs (Baker et al. 2022 Ye et al. 2024), which fix a prediction window (e.g., 50 ms) and thus must emit a prediction each tick inefficient in sparse-event regimes and coarse in temporal resolution. Instead, our IDM predicts the event and its timestamp, enabling event-driven modeling that avoids "no-op" ticks and makes more efficient use of inference context.
3

---

Published as a conference paper at ICLR 2026
ocsp: Open Desktop Recorder
OW.MMoap: Standardized Data Format
Vidao
OWAMesp File (maaa))
VPT Dataset
Stundardized Container
COMPRESSION
Audo
Mender Toyia
Thenseage Pracsie
Mevirge
-kobaad
JSONL 1.06 TIB
OWAMcuP
Masac
7.12 GIB
Referemocs
CS:GO Datasel
External Media Files Limko-pogg Optimized Storage
34,45x COMPHESSOO
Keybeam
1ila Frimso +laadha aas
Sounneet -eodboo High qaili
Andie -Somral
HDFS
OWAMcaa 20 GiB
Tiee 1900 0.5 0.0 0.5 031 035 039 035 64 045 690 C55 (8) 055 170 075
8.0 8.5 0:8 055 116
Figure 2: OWA Toolkit's recording and storage architecture. (Left) ocap recorder captures perfectly synchronized multimodal streams-vidos (60 FPS), audio, mouse events, keyboard inputs, and window states precise time alignment, enabling accurate reconstruction of desktop interactions. (Right) OWAM- cap format revolutionizes desktop data storage through its dual-layer architecture: standardized MCAP con- tainer for crash-safe metadata and event logging, paired with external media referencing for optimized video storage using H,265 codec (217x compression). This design achieves dramatic storage reduction-1522 for VPT dataset (1.06 TiB + 7.12 GiB) and 34,45x for CS:GO dataset (689 GiB + 20 GiB)-while maintaining event fidelity and enabling efficient random access for training.
OPEN- WORLD AGENTS TOOLKIT
We introduce the Open-World Agents (OWA) Toolkit alongside large-scale desktop data, estab- lishing both the infrastructure and data foundation for embodied Al research. The toolkit provides a unified interface (Zhang et al., 2024; 2025) for capturing interaction patterns across diverse ap- plications without Nomain--pecciie action space defimitions, while our data release demonstrates the practical scalability and diversity achievable through this standardized approach.
3.1 OCAP: SYNCHRONIZED DESKTOP RECORDER
Existing desktop recording tools lack critical features for desktop data collection. Content creation tools like OBS Studio (BBS Project) focus on streaming quality, while action modeling requires synchronized input event logging to capture the precise keyboard and mouse actions that caused visual changes. The ocap (Omnimodal CAPture) tool addresses this gap by capturing desktop signals in synchronized manner, recording video, audio, keyboard, and mouse interactions with high temporal precision. FigureZ](Left) illustrates an event timeline where these multimodal streams are well synchronized. By leveraging hardware acceleration using Windows APIs, we achieve real- time FHD/QHD recording at 60 Hz on consumer-grade GPUs with low overhead, ensuring that normal user activities remain unaffected and effectively lowering the hardware barrier for large- scale data collection. Implementation details are in Appendix A
3.2 OWAMCAP: STANDARDIZED DATA FORMAT
Prior desktop datasets suffer from storage inefficiency and poor random access capabilities. Ex- isting approaches (Baker et al.) 2022 Pearce & Zhu 2022) either store image-encoded frames in monolithic tables unsuitable for real-time recording, or use formats like JSONI. that lack proper introduce OWAMcap (Figure 2 Right), which extends the industry-standard MCAP format (Foxglove) 2022] 2022l-iidely widely adopted in robotics for multimodal sensor logging and providing efficient indexing, crash-safe writes, and broad ecosys- tem support- with two key deskkoop-specife additions.
indexing and crash-safety. To address these limitations,
First, we define standardized message schemas for desktop events (screen, keyboard, mouse) based on Windows APIs, enabling unified processing across different datasets without complex post- processing logic. Unlike other formats (e.g., RLDS (Ramos al. 202Th) that lack solid message
4

---

Published as a conference paper at ICLR 2026
definitions, our standardized schemas allow users to process identical message sets through single pipeline for foundation model training.
Second, MediaRef enables efficient video storage while maintaining MCAP compatibility. Raw video captures and image encoding approaches like PNG are prohibitively large for foundation model training, making efficient compression essential. MediaRef addresses this by enabling mod- em video codecs (H.265), achieving 217x x compression over raw captures and 68x over PNG while maintaining sufficient visual quality for agent training (Table TOP.
3.3 OPTIMIZED DATA PIPELINE
Training foundation models on OWAMcap data requires specialized data loading strategies to max- imize throughput, as I/O and data pipeline bottlnnecks have been identified as critical limitations in large-scale video model training (Zhao & Krähenbühl] 2023; Leclere et al.) 2023). We present four-stage optimized pipeline: (1) Media transcoding with optimized x264 parameters for sistent random access; (2) Event dataset conversion to HuggingFace datasets (Lhoest et al. 2021) format for efficient sequential and random access; (3) Fixed Sequence Length Dataset (FSa.Dataset) generation through tokenization and packing to maximize training throughput; (4) On-the-fly media loading with adaptive batch decoding that defers expensive media operations until training time. Our complete data pipeline optimizations are detailed in Appendix A.7 with comprehensive benchmark configurations provided in Appendix A.8
Fixed Sequence Length Dataset (FSLDataset) To optimize training throughput, we introduce FSLDataset that packs sequences to uniform lengths while preserving episode structure. Unlike conventional random concatenation, FSLDataset sequentially lists events within each episode up to the maximum sequence length, terminating at episode completion. This design enables consis- tent batch processing and converts fine-grained random access into coarse, coalesced patterns for improved I/O efficiency.
Adaptive Batch Decoding Strategy Video decoding requires seeking keyframe and then quentially decoding subsequent frames, since compressed video formats do not support independent decoding of arbitrary frames. To address this constraint, operates as follows: (1) seek to the target frame; (2) demux and decode until keyframe is encoun- tered; (3) upon hitting keyframe, resume seeking to the target frame. This strategy ensures stable performance across fine-grained, coarse-grained, and mixed access patterns.
adaptive batch decoding algorithm
Benchmarking Media Decoding on FSLDataset We evaluate our optimized pipeline on repre- sentative FSLDataset containing 64 episodes of 5-minute Minecraft gameplay 640x360 resolution and 20 Hz. The baseline uses single- frame- decoding per frame, while TorchCodec and our approach use batch decoding for frames within each FSt.Daasst sample. Throughput is measured as im- ages processed per second, while 1/O efficiency is measured as average disk read per image using isolated filesystem monitoring. Combining these optimizations-optiiiiiii
x264 parameters and adaptive batch decoding our complete pipeline achieves 119.16 img/s (10.2x over bascline) while reducing average disk read per image to 18.73 KB (3.4x less than bascline and 41x less than Torch- Codec (PyTorch Team 2024)). Table summarizes results across different configurations.
Intern Training Throughput Using the FSLDataset from the media decoding bench- mark, we benchmarked Intern training throughput on single H100 GPU. Our optimized pipeline achieves 4.77 it/s with dataloading worker, while the baseline requires 16 workers reach comparable throughput (4.55 it/s), demonstrating 16x efficiency gains. Moreover, the baseline per- formance saturates beyond 8 workers, indicating fundamental I/O bottlenecks that our optimizations successfully address (Table 2
3.4 COLLECTING HUMAN DEMONSTRATIONS SCALE
We collect desktop dataset that provides high -ualituultyt, synchronized multimodal signals for vision- action pretraining. While the OWA Toolkit can capture arbitrary desktop tasks (e.g., web surfing, productivity applications) with multimodal events incluuinn the screen, mouse, and keyboard-
5

---

Published as a conference paper at ICLR 2026
Table Media decoding benchmark on FSL- Dataset (Minecraft, 64x5 min, 640x360 @
20Hz).
18600
Configuration
Throughput (img/s)
Avg. Read (KB/img)
Baseline
11.68
63,46
optimized x264
24.25
41.69
+ TorchCodec
79.73
770.39
+ Ours
119.16
18.73
Table InternV3341 training throughput on FSLDataset.
Figure 3: Our FSLDataset design, coupled with a batched decoding API, converts fine-graine -re-ggained random I/O into coarse, coalesced random ac- cess, thereby avoiding the limitations of large- scale filesystems that are inefficient for small random reads.
Contiguration
Throughput (it/s)
Ours worker)
4.77
Ours (4 workers)
4.73
Baseline (8 workers)
3.79
Baseline (12 workers)
4.42
Baseline (16 workers)
4.55
focus on gameplay interactions. Gameplay data offer behavioral diversity while minimizing privacy concerns, which enables broad community contribution and data sharing. Using the ocap desktop recorder for efficient collection, 14 human annotators recorded the dataset. The dataset comprises 335 hours of newly collected human demonstrations across 31 games. It spans diverse genres, including 3D third-person games such as GTA V and Cyberpunk 2077. first-person games like Apex Legends and Minecraft, and 21) top-down games like Brotato and Stardew Valley. This variety captures wide range of visual environments and interaction styles, making it well-suited for vision- action pretraining. Further details on the dataset and collection process are provided Appendix
4 GENERALIST INVERSE DYNAMICS MODEL
Collecting large-scale action data through manual demonstrations is infeasible due to prohibitive costs. The OWA Toolkit (Section 3 closes the instrumentation gap and standardizes over 2.6k hours of synchronized trajectories (Table 12). yet human capture alone remains a bottleneck relative to the ocean of unlabeled gameplay available online. VPT (Baker et al. (022)) addressed this by leveraging Inverse Dynamics Models (IDMs) to pseudo-label You videos, but was limited to Minecraft, restricting generalization and dataset diversity. We train Generalist-IDM corpus collected via the OWA Toolkit, enabling generalization across heterogeneous interaction patterns. Our model can infer actions in ditt----iittiiutionn environments never seen during training. as demonstrated in Section 5.1 This capability enables pseudo-labeling of large-scale YouTube gameplay videos across diverse games, laying the foundation for internet-scale dataset collection.
multi-domain
4.1 TIMESTAMP- BASED EVENT TOKENIZATION
We represent desktop interactions discrete events, each serialized into short token sequence bounded by <EVENT_STARY and <EVENT_ENDS Observation events capture screen updates (Screen Events), while action events represent user inputs: Keyboard Events (key presses/releases) and Mouse Events (clicks, movements, scrolls). This event-level serialization unifies heterogeneous inputs into consistent sequential representation for transformer modeling (Vaswani et al. 2017). For example, the tokens emitted for a single event follow the format below:
JEVENT_STARI>
END>
While most existing IDMs adopt tick-based prediction (Baker et al. 2022 Ye al. 2024 pre- dicting actions at fixed intervals our design employs timestamp-bast based approaches that use fixed prediction window (e.g., 50 ms), IDM directly predicts both the event and its timestamp, preserving the asynchronous timing captured by ocap and converted corpora. This design provides two key advantages. First, it maintains cross-man modal alignment with-
prediction. Unlike tick-

---

Published as a conference paper at ICLR 2026
out resampling, allowing screen, keyboard, and mouse streams stay synchronized even when their natural cadences differ. Second, timestamp-basee prediction avoids generating empty ticks when no actions occur. By skipping unnecessary "no-op" tokens, approach makes more efficient use of the limited inference context, enabling denser packing of relevant information and improving the efficiency of both learning and inference. A detailed specification of the event tokenization process is provided in Appendix
4.2 NEP-7: NEXT-EVENT PREDICTION WITH TEMPORAL OFFSET
Once raw desktop interactions are converted to event token sequences, we train the Generalist-IDM with a next-event-pin rext-event-prediction objective. Given a trajectory consisting of observed states and actions (01, 01, 02, .2...... ,,),, where each action a, is taken state o and leads to state ot+1. the goal is to predict action ay based on all preceding observations and actions. This objective enables the model to learn mappings between observed states and actions while preserving temporal dependencies within the trajectory.
T Σ
NEP
E B(OLT
log Po(as 01: o12-1)
2025 which conditions on extended future trajectories to improve inverse dynamics, we adopt NEP-7, temporal-offset variant of NEP. Unlike IDM-K, which jointly encodes entire past and future trajectories, our method simply rearranges the (observation, action) sequences by shifting the observation window forward by T steps. This allows the model incor- porate future observations up to steps ahead without encoding entire future trajectories, enhancing temporal consistency. Formally, the objective is:
Inspired by IDM-K Tot et al.
log Po Po(as) on min((t,, T) (11-1)
LNEP- = B(OLT
(3)
We ablate 7 E {0, 50, 100, 150, 200} ms across six in-distribution games (Table 18). Without any offset (7 = 0), Pearson correlations collapse zero and keyboard accuracy drops sharply, con- firming that future context essential for resolving the current action. A small offset (r = 50 ms) recovers mouse prediction but remains suboptimal for keyboard accuracy. Performance stabilizes at 2 100 ms with only minor variation to 200 ms, showing that NEP-- is robust to the exact offset once sufficient future context provided. Based on these results, we adopt r 100 ms as the default in all experiments.
(ms)
Avg Pearson
Avg Pearson
Avg Scale
Avg Scale Y
Avg Keypress (Kbd)
Avg Keypress (Mouse)
0.108
0.050
25.66
22.30
0.386
0.961
50
0,803
0.734
1.17
1.48
0.492
0.948
100
0.796
0.783
1.31
0.730
0.957
150
0.891
0.818
1.08
1.14
0.767
0.965
200
01893
0.837
1.28
1.49
0.740
0.947
Table 3: Aggregate performance across all games for different temporal offsets down is provided in Appendix G.1
Per-game break-
4.3 PSEUDO-LABELING WITH YOUTUBE GAMEPLAY VIDEOS
We focus on pseudo-labeling gameplay videos because they are abundant, actively shared, and largely free of personally identifiable content, sidestepping the privacy concerns. YouTube game- play footage also exhibits consistent HUD layouts and frame rates, which align well with the OWA Toolkit's s event schema. Our pipeline first curates long-form gameplay uploads with permissive li- censes, retrieves them at 20 Hz, and converts the frames into Screen events so they can be fed through the same tokenizer used for human demonstrations. Building on this, we train the Generalist IDM using the Intern VL3-1B (Zhu et al., 2025) architecture with the NEP-7 objective. The Generalist- IDM then autoregressively produces Keyboard and Mouse events via the NEP-7 objective between consecutive screen frames, using a temporal stopping criterion and sliding context window (AI-

---

Published as a conference paper at ICLR 2026
gorithm D. after which we apply consisteney checks before materializing the pseudo-labels (Ap- pendix B
Applying this procedure contributes 1055 hours of additional trajectories across twenty publicly shared titles, as summarized in Table [4 complementing the curated corpus described in Table [22 and Section 3 Importantly, because our model is designed to be generalist, any filtering of domain-specific interfaces such as inventory menus or map screens. Instead, these heterogeneous visual contexts are naturally included as part of the pseudo-labeled demonstrations, broadening the scope of training data without additional heuristics. These pseudo-labeled trajecto- ries form the seed for scaling desktop vision-action pretraining to interet-scaaee data sources.
do not require
RESULTS
5.1
PERFORMANCE OF THE GENERALIST-IDM
In-Distribution Performance. We begin by evaluating the Generalist-IDM on six in-distribution video games spanning both 2D and 3D settings, comparing its performance to Specialist-IDMs trained individually on each game. We employ an autoregressive inference pipeline to generate actions and evaluate model performance across multiple metrics. Further details are provided in Appendix F As shown in Table 4 and Table 5 our Generalist-IDM achieves strong performance across all environments. Notably, it yields large gains in Pearson correlation (e.g., +39.5 points on Stardew Valley X) and Keyboard accuracy (e.g., +57.6 points on Brotato), demonstrating robust generalization over diverse control dynamics.
Came
Model
X
Pearson
Y
Scale Ratio
X
Y
Keypress Acc. Mouse
Kbd
Game
Model
Pearson
Scale Ratio
Keypress Ace.
Kbd
Mouse
Brotato
G-DM
65.92
73.65
82.03
1.04
1.29
59
Apex Legends
IDM
G-ID
85.27
1.29
1.13
76.55
99.33
99.67
Stardew Valley
IDM
G-IDM
43.47
82.98
75.57
63.69
1.19
1.13
1.17
1.18
09:35
74.35
91.90
96.43
GTA
G-IDM
IDM
63.64
79.44
81.08
83.89
1.39
1.09
1.23
1.42
60.83
53.13
94.65
94.11
Core Keeper
IDM
G-DM
48.03
77.25
62.09
64.55
1.15
1.43
1.17
1.51
70.00
69:42
92.33
94.01
Minecraft
IDM
G-IDM
80.29
59.83
63.83
78.38
1.24
1.20
1.22
1.27
60.97
53.54
91.65
82.48
Table 4: Evaluation results on 2D games
Table 5: Evaluation results on 3D games
Generalization. We evaluate the generalization of our Generalist-IDM on two unseen games: Battlefield 6 (3D) and Ogu and the Secret Forest (2D). Battlefield 6, the Generalist-IDM achieves 63% keyboard accuracy, matching or slightly outperforming the Specialist- IDM, indicating solid transfer to an unseen FPS similar to the training set. Moreover, when provided with few-shot prefix that fills the first 2048 tokens in our streaming inference, the predicted scale ratio improves adapt to mouse sensitivity. In Ogu and the Secret Forest, the Generalist-IDM more than doubles the Specialist-IDMss performance (from about 12% to nearly 28%), showing substantial gains even under large domain gap. Taken together, these results demonstrate that the GeneralistIIDM is capable of adapting across both familiar and substantially different environments.
that the Generalist-IDM exhibits an in-context ability to
5.2 TRANSFERABILITY TO DOWNSTREAM TASKS
To validate the transfer of useful knowledge from the desktop domain to the embodied AI domain, we evaluate our D2E framework on both robot manipulation and navigation tasks. For manipula- tion. we first assess performance in simulated environments using the LIBERO (Liu al.) (023) and Meta- (Yu et al. 2020) benchmarks, and then further verify effectiveness in the real world by following the evaluation protocol used in SmolVLA (Shukor et al. 2025). For navigation, we evaluate in simulation using the CANVAS benchmark (Choi et al. 2024). Collectively, these experi- mental results demonstrate that our D2E framework effectively transfers knowledge across domains, resulting in strong performance robotics downstream tasks.
For these experiments, we use the Intern VL3-11 IB model as our backbone, which is also the architec- ture used in our Generalist-IDM. We train this model under two different settings: VAPT without pseudo-labels (259 hours), which uses only the human collecee dataset, and VAPT with pseudo- labels (1.3K hours), which augments the human data with a pseudo-labeles dataset generated from
8

---

Published as a conference paper at ICLR 2026
Model
Pearson
Scale
Ratio Y
Keypress Acc.
Position
Y
X
Kbd
Mouse
8000
Ground Truhh
Battlefield 6
4000
the foo shot
IDM (FT)
57.28
61.74
1.00
1.00
62.44
94.55
G-IDM (ZS)
57.36
63.17
3.13
3.56
47.75
92.11
G-IDM (FS)
56.79
63.40
1.07
1.05
52.64
93.89
G-IDM (FT)
54.90
62.89
1.06
1.04
58.55
93.41
bicance
Ogu Forest
Mouse Position Over Time
2506
IDM (FT)
11.73
2000
G-IDM (ZS)
27.80
248
G-IDM (FS)
27.97
G-IDM (FT)
26.88
-006
-1000
Table 6: Out-of-distribution performance on unseen 3D and 2D games. Note that Ogu Forest uses only keyboard inputs.
Figure Trajectory of Battlefield
YouTube videos using the Generalist- Further training details can be found in Appendix and detailed experimental setups for each downstream task provided in Appendix
Transfer May Work. We identify three properties of desktop pre- training that may facilitate transfer embodied domains. First, action modality alignment: VAPT is trained on explicit vision-acto action trajectories rather than solely on image-tex pairs, encouraging the model internalize how visual observations correspond motor commands. Second, goal-directed sequential decision-making: desktop gameplay requires visual grounding, temporal reasoning, and long-range dependency modeling-capabllttees that translate directly into coherent robotic control. Third, high diversity: our 20-game corpus spans 2D and 3D environments with hetcrogeneous me- chanics, encouraging general-purpose control priors instead of domain speciffc shortcuts. Empiri- cally, training loss curves (Appendix G.2) corroborate these hypotheses: VAPT-initialized models converge immediately, whereas baseline models exhibit an initial plateau, indicating that desktop pretraining provides better-aligned representations for embodied control.
Why Desktop-to-Embonn
Robot Manipulation. For manipulation, we first evaluate our VAPT models on the LIBERO benchmark (Liu et al.] (2023). As shown in Table 7 the InternVL3-11 BB baseline performs rela- tively poorly. VAPT without pseudo-labels achieves a substantial improvement, reaching 96.6% on Total and 93.6% long-horizon tasks. These results comparable to of much larger models such as xo (3.3B) and Open (7B). Interestingly, incorporating pseudo- labels does not provide additional gains manipulation tasks. We attribute this the nature of manipulation tasks, where precise human supervision is more critical than data scale and diversity. Overall, our 1B-parameter model matches or outperforms significantly larger policies such To (3.3B) and Open VLA (7B), with particularly strong advantages on long-horizon tasks that require careful action sequeneing.
surpass those
Method
Params
VLA Pt
Spatial
Object
Goal
10 (long)
Total
Octo OOto Model Team
2024
93M
Yes
78.9
85.7
51.1
75.1
Open VLA (Km et 2024
7B
Yes
84.7
88.4
79.2
53.7
76.5
DiT Policy Dasan et al. 2025
115M
No
84.2
96.3
85.4
63.8
82.4
Black ct al. 2024
3.3B
Yes
90.0
86.0
95.0
73.0
86.0
SmoIVLA Shukor el al.
2025
2.25B
No
93.0
94.0
91.0
77.0
88.7
PI-KI (Driess et at. 2025
300M
Yes
98.0
97.8
95.6
85.8
94.3
OpenVLA-OFT (Kim al. 2025
7B
Yes
97.6
98.4
97.9
94.5
97.1
Baseline (IntemVL..31IB)
1B
No
94.4
97.0
93.6
54.2
84.8
+ VAPT w/o pscudo
18
No
95.8
98.4
98.6
93.6
96.6
+ VAPT w/ pseudo
1B
No
89.6
98.2
93.8
87.2
92.2
Table 7: Evaluation results Libero benchmark (success rates, %).
Next, we evaluate our VAPT models on Meta- World (Yu et al. 2020), a standard benchmark for multi-task robotic manipulation. We compare VAPT against the Intern VL.3-B baseline across tasks

---

Published as a conference paper at ICLR 2026
of varying difficulty. Even without cobotics-specifeg pretraining extensive hyperparameter tun- ing, VAPT consistently outperforms the baseline, showing an average success rate improvement of roughly 5% (a ~25% relative gain). The performance gap is most pronounced in the Hard and Very Hard categories (e.g., 8.0% vs. 20.0-24.0% on Very Hard), suggesting that the priors learned from desktop data particularly robust for complex manipulation challenges.
We further validate our approach with a real-world pick-and-place experiment using an SO101 robot arm, following the evaluation protocol of SmolVLA (Shukor al. 2025). The task requires grasping a blue cube and placing it in a white box, with the cube placed at five distinct initial positions. We collect 208 demonstration episodes and evaluate each trained policy over 30 rollouts (further details in Appendix H. As shown in Table g the baseline InternVL3 IB achieves 70% success rate, while both VAPT variants reach 80%, confirming that VAPT transfers effectively to real-world hardware.
Meta-Worke
SO101
Method
Easy
Medium
Hard
Very Hard
Avg
Pick&
Baseline (Intern VI.
55.4
14.5
1.7
8.0
19.9
70.0
VAPT w/o pseudo
53.6
18.2
8.3
20.0
25.0
80.0
VAPT w/ pscudo
52.1
16.4
6.7
24.0
24.8
80.0
Table Evaluation results on Meta- World and
S0101 pick-and-place (success rates, %).
Robot Navigation. For robot navigation, evaluate on the CANVAS (Choi al. 2024) bench- mark, which tests robustness to both misleading and precise instructions across diverse simulated environments. Compared to the baseline, VAPT framework shows clear gains: without pseudo- labels, performance matches the baseline (75.3%), while adding pseudo-labeled demonstrations increases performance to 83.3%, misleading instructions, as in sim_orre hep_orchard (86.7% vs. 53.3%) and fim_street_attt prreet_sdeealk (73.3% vs. 40.0%). whereas performance under precise instructions remains near ceiling. These results indicate that pscudo-labeling is particularly useful for navigation tasks, where success depends on high-level planning rather than precise low-level control.
8-point improvement. The benefit is especially large under
Gallery
Office
Orchard
Street (road)
Street (side)
Method
Total
Mis.
Prec.
Mis.
Prec.
Mis.
Prec.
Mis.
Pree.
Mis.
Prec.
Baseline (InternV1.3 1l)
53.3
100.0
100.0
100.0
53.3
40.0
94.4
100.0
40.0
73.3
75.3
+ VAPT w/o pseudo
33.3
93.3
93.3
100.0
53.3
53.3
88.9
91.7
53.3
93.3
75.3
VAPT w/ pscudo
53.3
93.3
100.0
100.0
86.7
60.0
88.9
100.0
73.3
80.0
83.3
Table 9: Evaluation results on CANVAS navigation benchmark (success rates, %). Mis. = mislead- ing instructions, Prec. precise instructions.
CONCLUSION
Embodied AI has long struggled with the prohibitive cost of collecting large-scale physical interac- tion data, limiting its ability to benefit from internet-scale resources. To address this challenge, we proposed using desktop interactions an abundant and low-cost substrate for pretraining. Our con- tributions are threefold: (1) the OWA Toolkit, which standardizes and compresses diverse desktop data into a scalable format; (2) the Generalist-IDM, IDM,, a timestamp-bras based inverse dynamics model that generalizes across unseen games and demonstrates pathway toward internet-scale pseudo-labeling representations robotics tasks.
and (3) VAPT, which explores the transfer of beskkoo--peetainee Leveraging 1.3K+ hours of human and pseudo-labeled data, our framework achieves 96.6% success on LIBERO manipulation and 83.3% on CANVAS navigation, demonstrating that digital sensori- motor patterns can directly improve embodied AI benchmarks. We release all our tools, datasets, and models publicly to enable the community to build upon this foundation and further investi- transfer. These results establish desktop data practical and scalable resource for advancing embodied intelligence, opening a new path toward general-purpose purpose agents without relying prohibitively expensive physical data collection.
gate cecktop-to-embodaee
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure full reproducibility of our work, we release comprehensive resources and documenta- tion. All source code for the OWA Toolkit (ocap recorder and OWAMcap format implementa- tion), GeneralistIIDM training, and downstream task fine-tuning is publicly available at https: including detailed installation instructions and usage exam- ples. The complete 2.6K hour desktop dataset (335 hours newly collected. 2.3K hours converted) and IK+ hours of pscudo-labeled data are accessible through the same repository with standard- ized OWAMcap format specifications described in Section 3 and Appendix [A] Pre-trained model weights for both Generalist-IDM and VAPT foundation models are provided along with training configurations. Hyperparameters and training schedules are detailed in Appendix [ including batch sizes, learning rates, and hardware requirements (8 H100 GPUs for IDM training). Data preprocess- ing pipelines. including temporal offset implementation (Section 4 and event tokenization schemes (Appendix C are fully documented with reference implementations. Evaluation protocols and metries are specified in Appendix F with corresponding evaluation scripts in the repository. For compute-constrained researchers, we release smaller dataset subsets and checkpoint models at vari- ous training stages facilitate partial reproduction and ablation studies.
7/github. com/ 1.tam//or--a0 poorvaai/De
ACKNOWLEDGMENTS
This work was partly supported by an Institute of Information & communications Technology Plan- ning & Evaluation (ITTP) grant funded by the Korean Government (MSIT) (No. 10021-11211300 Artificial Intelligence Graduate School Program (Seoul National University)), the Technology In- novation (egrram(RS2202-255567777
Development of a humanoid robot specialized chemical processes based on AI foundation model) funded by the Ministry of Trade, Industry and Resources (MOTIR, Korea).
REFERENCES
Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24699-24654, 2022.
Johan Bjorek, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Speneer Huang, al. GrOOt n1: open foundation model for generalist humanoid robots. arXiv preprint urXiv:2503. 14734, 2025.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. \mo: A lasion-anguage-acttinn flow model for general robot control. arXiv preprint arXiv:2410.2411 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishhan, Karol Hausman, Alex Herzog, Jasmine Hsu, al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068) 2022.
Remi Cadene, Simon Alibert, Alexander Soare, Quentin Gallouedec, Adil Zouitine, Steven Palma, Pepijn Kooijmans, Michel Aractingi. Mustafa Shukor, Dana Aubakirova, Martino Russi, Francesco Capuano, Caroline Pascal, Jade Choghari, Jess Moss, and Thomas Wolf. Lerobot: State-of atr machine learning for real-world robotics in pytorch. tittps://githir com/ 2024.
Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Huichi Zhou, Qihui Zhang, Zhigang He, Yilin Bai, Chujie Gao, Liuyi Chen, Yiqiang Li, Chenlong Wang, Yue Yu, Tianshuo Zhou, Zhen Li, Yi Gui, Yao Wan, Pan Zhou, Jianfeng Gao, and Lichao Sun. GUI-world: A video benchmark and dataset for multimodal GUl-oriented understanding. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https:///// http://////ee//iii.../ net/ forum?ic QarKTT5br2
11

---

Published as a conference paper at ICLR 2026
Xuxin Cheng, Jialong Li, Shiqi Yang. Ge Yang, and Xiaolong Wang. Open-television Teleoperation with immersive active visual feedback. arXiv preprint arXiv:2407.01512, 2024.
Suhwan Choi, Yongjun Cho, Minchan Kim, Jaeyoon Jung, Myunchul Joe, Yubeen Park, Minseo
Kim, Sungwoong Kim, Sungjae Lee, Hwiseong Park, et al. Canvas: Commonsense-aware navi- gation system for intuitive human-robot interaction. arXiv preprint crXiv:2410.01273, 2024.
Sudeep Dasari, Oier Mees, Sebastian Zhao, Mohan Kumar Srirama, and Sergey Levine. The in- gredients for robotic diffusion transformers. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 15617-15625. IEEE, 2025.
Danny Driess, Jost Tobias Springenberg. Brian Ichter, Lili Yu, Adrian Li-Bell, Karl Pertsch, Allen Z Ren, Homer Walke, Quan Vuong, Lucy Xiaoyang Shi, et al. Knowledge insulating vision- language-action models: Train fast, fast, generalize better. arXiv preprint arXiv:2505.23705, 2025.
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text- text-guided video generation. Advances in neural information processing systems, 36:9156-9172, 2023.
Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, et al. Foundation models in robotics: Appli- cations, challenges, and the future. The International Journal of Robotics Research, 2024. doi:
http://...oo//011117777776949229
Foxglove. Evaluation of robotics data recording file formats. Technical Report, October 2021. URL pdf Technical report; accessed 2025-08-23.
httpps::/maap..dv//ffeees
Foxglove. Introducing the mcap file format. Foxglove Blog, February 2022. URL https://
Foxglove.co httpe///////ll///t/tt/iiiigggaeo thapang--h----m----aae file- format Blog post; accessed
2025-08-23.
Foxglove Developers. Mcap: serializaaionnaanooiiic log container file format, 2024. URL https:
//mcap. dev Available from https:/ggihub.com//foogllveeeegp
Zipeng Fu, Tony Z. Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. In Conference Robot Learning (CoRI.), 2024.
Haoran Geng, Feishi Wang, Songlin Wei, Yuyang Li, Bangjun Wang, Boshi An, Charlie Tianyue Cheng, Haozhe Lou, Peihao Li, Yen-Jen Wang, form, dataset and benchmark for scalable and generalizable robot learning, arXiv preprint arXiv:2504. 18904, 2025.
Roboverse: Towards a unified plat-
GStreamer Team. Gstreamer: Open source multimedia framework. URLptttss://gstreea Freedesktop.orgg Official GStreamer Documentation.
Yingchen Hc, Christian D Weilbach, Martyna Wojciechowska, Yuxuan Zhang, and Frank Wood. Plaicraft: Large-scale time aligned iision-speech-actin dataset for embodied ai. arXiv preprint arXiv:2505. /2707, 2025.
Jordan Hoffmman,, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.15556 2022.
ITU-T. High efficiency video coding. Recommendation H.265, International Telecommunication
Union, 2024. Also published as ISO/IEC 23008-2.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Alexander Khazatsky, Karl Pertsch. Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty El- lis, et al. Droid: A large-scale in-the-will robot manipulation dataset. arXiv preprint arXiv:2403. 12945, 2024.
12

---

Published as a conference paper at ICLR 2026
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Anikait Srinivas, Jacky Liang, Pieter Abbeel, and Chelsea Finn. Openvla: An open-source vision- language-action model. arXiv preprint arXh::2406.0022 09246, 2024.
Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning Ssiio-language-action models: Opti- mizing speed and success. arXiv preprini arXiv:2502. 2025.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Alek- sander Madry. Ffev: Accelerating training by removing data bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12011-12020 2023.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Can- wen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL stttps://allanthooogy. org/ 2021. camnl--deme 21
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776-44791, 2023.
Ajay Mandlekar, Jonathan Booher, Max Spero, Albert Tung, Anchit Gupta, Yuke Zhu, Animesh Garg, Silvio Savarese, and Li Fei-Fei. Scaling robot supervision to hundreds of hours with robo- turk: Robotic manipulation dataset through human reasoning and dexterity. In 2019 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2019.
Robert McCarthy, Daniel CH Tan, Dominik Schmidt, Fernando Acero, Nathan Herr, Yilun Du, Thomas G Thuruthel, and Zhibin Li. Towards generalist robot learning from internet video: A survey. Journal of Artificial Intelligence Research, 83, 2025.
Microsoft Corporation.
Dxgi overview.
URL
https: //docs/
microsoff. wttp://ftt..coooonooss//aaaaaaaa d3d1 org000/rrppp.oospppprrremmmrrmre dxgi Microsoft Developer Documenta- tion.
OBS Project. Obs studio. URL htttss::/oosproojeesttt comp Open Broadcaster Software Studio Free and open source software for video recording and live streaming.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo. Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
Abby O'Neill, Abdul Rehman, Abhiram Maddukuri. Abhishek Gupta, Abhishek Padalkar, Abraham Lec, Acorn Pooley, Agrim Gupta, Ajay Mandlckar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and models: Open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6892-6903. IEEE, 2024.
Younghyo Park, Jagdeep Singh Bhatia, Lars Ankile, and Pulkit Agrawal. Dexhub and dart: Towards internet scale robot data collection. arXiv preprint arXhv:2411.02214 2024.
Tim Pearce and Jun Zhu. Counter-strike deathmatch with large-scale behavioural cloning, In 2022 IEEE Conference on Games (CoG), pp. 104-111. IEEE, 2022.
PyTorch Team. Torchcodec: A pytorch library for video decoding. https::/githhb. com/ pytorchh/torchcodee 2024. PyTorch video decoding library.
13

---

Published as a conference paper at ICLR 2026
Yuzhe Qin, Wei Yang, Binghao Huang, Karl Van Wyk, Hao Su, Xiaolong Wang, Yu-Wei Chao, and Dieter Fox. Anyteleop: A general vision-based dexterous robot arm- aom-hand teleoperation system. In Robotics: Science and Systems, 2023.
Maria Abi Raad, Arun Ahuja, Catarina Barros, Frederic Besse, Andrew Bolt, Adrian Bolton, Bethanie Brownfield, Gavin Buttimore, Max Cant, Sarah Chakera, et al. Scaling instructable agents across many simulated worlds. arXiv preprint arXiv:2404. 1011,, 2024.
Sabela Ramos, Sertan Girgin, Léonard Hussenot, Damien Vincent, Hanna Yakubovich. Daniel Toyama, Anita Gergely, Piotr Stanczyk, Raphael Marinier, Jeremiah Harmsen, Olivier Pietquin, and Nikola Momchev. Rlds: an ecosystem to generate, share and use datasets in reinforcement learning. 2021.
Scott Reed, Konrad Zoina, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06173 2022.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke 7ettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020.
Mustafa Shukor, Dana Aubakinova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zouitine, Michel Aractingi, Caroline Pascal, Martino Russi, Andres Marafioti, et al. Smolvla: A vision- lana--lgaagaattioon model for affordable and efficient robotics. arXiv preprint arXiv:2506.01844, 2025.
Gary J Sullivan, Jens- RaiseR Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (heve) standard. IEEE Transactions on circuits and systems for video technology, (1112)::649-111) 1668, 2012.
Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, et al. A survey on (m) Ilm-based gui agents. arXiv preprint 13865 2025.
Marko Tot, Shu Ishida, Abdelhak Lemkhenter, David Bignell, Pallavi Choudhury, Chris Lovett, Luis França, Matheus Ribeiro Furtado de Mendonça, Tarun Gupta, Darren Gehring, et al. Adapting a world model for trajectory following in 3d game. arXiv preprint arXiv: 12299, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Junjie Wen, Yichen Zhu, Minjie Zhu, Zhibin Tang, Jinming Li, Zhongyi Zhou, Xiaoyu Liu, Chaomin Shen, Yaxin Peng, and Feifei Feng. Diffusionvla: Scaling robot foundation models via unified diffusion and autoregression. In Forty-second International Conference on Machine Learning.
Youguang Xing, Xu Luo, Junlin Xie, Lianli Gao, Hengtao Shen, and Jingkuan Song. Shorteut learn ing in generalist robot policies: The role of dataset diversity and fragmentation. arXiv preprint a5Xiv:2508. 06426, 2025.
Qwen An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi-Chao Zhang, Yunyang Wan, Yuqi Liu, Zeyu Cui, Zhenru Zhang. Zihan Qiu, Shanghaoran Quan, and Zekun Wang. Qwen2.5 technical report. ArXiv, 2024.
Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Man- dlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos. arXiv preprint ar /758, 2024.
14

---

Published as a conference paper at ICLR 2026
Tianhe Yu. Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learning. pp. 1094-1100. PMLR, 2020.
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. Ufo: A ui-focused agent for windows os interaction. arXiv preprint DrXiv:2402.07939, 2024.
Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, et al. Ufo2: The desktop agentos. arXiv preprint arXiv:2504, 14603, 2025.
Yue Zhao and Philipp Krähenbühl. Training large video model on single machine in day. arXiv preprint arvi:22309. 0e0.30999666... 2023.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint 3xiv:2504.10470 2025.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-actoon models transfer web knowledge to robotic control. In Conference Robot Learning, pp. 2165-2183, PMLR, 2023.
15

---

Published as a conference paper at ICLR 2026
A
OWA TOOLKIT DETAILS
A. FORMAT COMPARISON
Prior desktop datasets commonly adopt one of two storage strategies. The LeRobot dataset dene et al.] 2024). CS:GO dataset (Pearce & Zhu 2022), and the CraftJarvis "minecraft-vla- dataset (He et al. 2025) store image-encoded frames directly in a single, monolithic table. While this layout sufficient for training, it is ill-suited for recording because long-table stores typically do not support efficient real-time appends. By contrast, the VPT dataset (Baker et al., 2022) packages each sample as MP4-JSONL pair. However, JSONL lacks the ability to interleave heterogeneous, typed streams with chunking and indexing. In practice, this limitation results in poor or unavailable topic-wise random seeking and reduced crash-safety, as writes are unreliable under unexpected ter- mination. Furthermore, datasets that rely on image encoding are substantially less storage-efficient compared to standard video codecs.
(Ca-
The robotics community has encountered similar multimodal logging challenges. Traditional ROS bags exhibit performance and extensibility limitations (Foxglove 202T), which motivated the de- velopment of the MCAP format (Foxglove 2022): an open-source container format designed with efficient indexing and compression. MCAP has since become the de facto logging standard for ROS 2 (Foxglove [2022; Foxglove Developers] 2024), demonstrating the benefits of specialized data for- mats for embodied Al research. However, no equivalent standard has been established for desktop datasets, motivating our introduction of the OWAMcap format.
A.2 COMPRESSION EFFICIENCY
OWAMcap achieves substantial storage savings across multiple datasets, demonstrating its effi- ciency and scalability. For the CS:GO dataset (Pearce & Zhu 2022). replacing the original HDF5 storage with OWAMcap (mkv+mcap) reduces the storage requirement from 689 GiB to 20 GiB-ii a 34.45x reduction. Similarly, converting the VPT dataset (Baker et al.) (022) from ISONL to OWAMcap (mcap format) shrinks disk usage from TiB to 7.12 GiB, achieving 152x reduc- tion. This significant compression arises from two different aspects: (1) from using video encoding instead of saving raw image buffer the CS:GO dataset's ' HDF5 and (2) from mcap's 's efficiency in representing/storing information on the VPT dataset's ' jsonl.
A.3 VIDEO COMPRESSION PERFORMANCE
Another advantage of OWAMcap MediaRef, a flexible system supporting storing media on (1) embedded or (2) external media. We support storing media in both external image files and ex- ternal video files. This flexible design provides the opportunity to acquire significant compression efficiency through video encoding, such as H.265/HEVC. To further evaluate the benefits of video encoding, we benchmarked video compression performance for various encodings. Table I0 shows that video encoding provides superior compression rates while maintaining visual quality, enabling large-scale storage without compromising data fidelity. ocap is storing all media in H.265 by de- fault and observed similar compression ratio for recorded files.
Format
Size per Frame
Total Size
Compression Ratio
Raw BGRA
5.97 MB
4.2 GB
1,0x (baseline)
PNG
1.87 MB
1.31 GB
3.2x
JPEG (Quality 85)
191 KB
135 MB
31.9x
H.265 (keyframe 0.5s)
27.8 KB
19.6 MB
217.8x
Compression performance comparison for various encoding on our recorded Minecraft video. Desktop screen capture at 1920x 1080 resolution, 12 seconds @ 60 Hz. H.265 en- coding uses nvd3dl 1h265enc for hardware acceleration. Video encoding yields significantly higher compression ratios than other formats. ocap is storing all media in H.265 by default and we ob- served similar compression ratio for recorded files. Note that size per frame for H.265 is an average over all frames, keyframes are larger.
Table 10:
16

---

Published as a conference paper at ICLR 2026
A.4 oCAP ARCHITECTURE
The implementation of ocap is designed maximize recording performance and reliability. ocap leverages Windows APIs, including DXGI (Microsoft Corporation) for capture, WASAPI for low-latency audio recording, and direct input event capture for precise key- board and mouse logging. The media pipeline is built on GStreamer (GStreamer Team) and employs H.265/HEVC encoding (TTU-T) 2024; Sullivan et al., 2012) achieve high compression efficiency while maintaining visual quality. The overall architecture, shown in Figure [S] integrates video, au- dio, and interaction streams within OWAMcap format while ensuring synchronized, crash-safe recording.
screen
owo.enndesktoop
twa etix get
Keyboourd Eventa
Mouse Events
Window Events
Screen Capture
Audio Cupture
Event Queue
Video/Aucio Pipcline
MCAP Writer
MIC Pipeline
In mosp
video
Figure 5: Architecture of ocap desktop recorder.
A.5 SCREEN CAPTURE PERFORMANCE BENCHMARKS
ocap employs H.265/HEVC encoding for video content and AAC encoding for audio streams, enabling real-time recording with minimal system overhead. Table compares the capture perfor- mance of ocap against existing alternatives, showing that implementation consistently achieves higher frame rates and lower CPU utilization while preserving recording fidelity.
Library
Avg. Time per Frame
Relative Speed
owa owa.env.ge gst
5.7 ms
1.0x (baseline)
pyscreenshot
33 ms
5.8x slower
PIL
34 ms
6.0x slower
MSS
37 ms
6.5x slower
PyOt5
137 ms
24x slower
Table 11: Screen capture performance comparison. Benchmarked on Intel i5- 1400 with GTX 1650. ocap achieves 6x faster performance than common alternatives through Windows API and GStreamer integration.
A.6 COMPARISON WITH EXISTING RECORDERS
To assess feature coverage and efficiency, compared ocap against commonly used desktop recording frameworks. As shown in Figure 0 ocap is the only system that provides synchronized multimodal recording, robust crash-safety guarantees, and efficient compression in single frame- work. These advantages make ocap a uniquely comprehensive solution for scage desktop interaction logging.
17

---

Published as a conference paper at ICLR 2026
Feature
ocap
OBS
wcap
pillow/mss
Advanced data formats (MCAP/MKV)
Yes
X No
X No
X No
Timestamp aligned logging
Yes
X No
X No
X No
Customizable event definition & Listener
Yes
X No
X No
X No
Single python file
X No
Audio Window Keyboard Mouse
Partial
X No
Hardware accelerated encoder
Yes
Yes
Yes
X No
Supports latest Windows APIs
Yes
Yes
Yes
X No (legacy APIs only)
Optional mouse cursor capture
Yes
Yes
Yes
X No
Figure 6: Comparison of key features between ocap and other desktop recording tools.
A.7 DATA PIPELINE OPTIMIZATIONS
Our data pipeline incorporates several key optimizations to address the limitations of conventional video processing approaches for foundation model training.
Baseline Video Properties To understand the limitations of default video encoding, we analyzed representative sample from dataset. The baseline configuration uses default x264 parameters, resulting in variable GOP structure that impacts random access performance. Frame type distribu- tion shows: I-frames 57 (0.9%), B-frames 3847 (64.1%) P-frames 2096 (34.9%). I-frame interval analysis reveals significant variability: minimum 1.35s, maximum 12.50s, average 5.32s, median 4.83s. This variable GOP size creates inconsistent seeking performance, motivating our optimized x264 parameters.
Optimized x264 Parameters Default video encoding with x264 creates variable GOP structures with unpredictable keyframe intervals, causing inconsistent random access performance during training. Our optimization fixes keyframe intervals to 30 frames (1.5 seconds at 20 Hz) and disables B-frames entirely. This creates predictable GOP structure: I-P-P-P...-PA. enabling consistent ran- dom access performance. The elimination of B-frames reduces decoding complexity during seeking operations, while fixed keyframe intervals ensure uniform seeking distances.
Dataat Construction FSLDataset preserves episode temporal structure during sequence packing. For each episode, we sequentially list all events (screen, keyboard, mouse) in chronologi- cal order, then concatenate episodes sequentially until reaching the maximum sequence length (e.g., 4096 tokens). When episode completes before reaching the maximum length, packing terminates immediately and remaining positions are padded. This approach maintains episode coherence while enabling uniform sequence lengths for efficient batch processing.
Adaptive Batch Decoding Strategy The baseline configuration uses single- frame decoding where each frame within an FSLDataset sample requires individual video seek and decode operations. For an FSL Datastt sample containing frames, the baseline performs separate video decoder calls,
18

---

Published as a conference paper at ICLR 2026
each involving: (1) seeking to the target frame position, (2) decoding from the nearest keyframe to the target frame, and (3) extracting the single target frame. This approach results in significant redundant I/O operations when multiple frames from the same video segment are needed.
Our adaptive batch decoding strategy processes all n frames within each FSLDataset sample through single batched operation, eliminating redundant seeking and keyframe decoding overhead. Both TorchCodec (PyTorch Team, 2024) v0.6.0 and our implementation use this per-sample batching approach: for each FSLDataset sample, we issue a single batched query that requests all images within the sample at once (no cross-sample batching or parallel workers).
A.8 BENCHMARK CONFIGURATION
To quantify the effect of our optimized pipeline, we conduct comprehensive benchmarks across different configurations and training scenarios.
Media Decoding Benchmark Setup The media decoding benchmark uses representative FSL- Dataset containing 64 episodes of 5-minute Minecraft gameplay recorded at 640x360 resolution and 20 Hz frame rate. The Dat.Dataset configured with fixed sequence length of 4096 tokens, where all sequences are tokenized and packed to this uniform length.
We measure performance using single-worker random-access iteration and report: (i) image throughput (img/s) calculated by dividing the total number of images by the time required to pro- cess all images during decoding, and (ii) average disk bytes read per image (KB/img) obtained by monitoring total bytes read during iteration divided by the number of images, capturing sceking and GOP decode overhead.
For 1/O efficiency measurement, create an isolated temporary filesystem and store all media data referenced by the FSLDataset in this dedicated path. During benchmarking, we monitor the total amount of data read from this filesystem obtain precise I/O measurements.
Progressive configurations test: (1) baseline with default x264 parameters and single-fraee ffraneeffemeae decod- ing, (2) baseline optimized x264 parameters, (3) optimized x264 + TorchCodec v0.6.0 batch de- coding, and (4) optimized x264 + our adaptive batch decoding strategy. All benchmark experiments were conducted three times to ensure result stability, with all runs showing consistent performance within measurement variance. The reported results represent the final experimental run.
InternVL3-1B Training Configuration Model training benchmarks use single H100 GPU with batch size 1, DeepSpeed Zerol for memory optimization, FlashAttention 3 for efficient attention computation, and context length 4096 tokens. The baseline configuration uses default x264 param- eters without batch decoding, while our optimized pipeline combines optimized x264 parameters with adaptive batch decoding API. We measure training throughput (iterations per second) across different numbers of dataloader workers to evaluate scalability and efficiency gains.
B DATASET DETAILS
B.1 COLLECTION AND QUALITY ASSURANCE
We collected the dataset using distributed approach supported by contributions from community volunteers. To ensure participant privacy, we applied automated detection techniques followed by manual review to remove any sensitive information. Quality assurance involved both automated and manual procedures. Automated validation checked for temporal alignment issues and corrupted recordings. while human annotators manually evaluated the realism and fidelity recorded behav- iors. The final dataset captures a wide range of desktop interaction patterns, including navigation behaviors, application switching, text input, menu interactions, and multi-step task exccution.
B.2 ANNOTATOR CALIBRATION AND PROTOCOLS
Before recording, contributors completed an ocap calibration wizard that verified refresh rate, dis- play resolution, cursor fidelity, and input-device mapping. Annotators-eille either modestly compen- sated participants or folntterss-followeed standardized game prompts covering navigation, combat,
19

---

Published as a conference paper at ICLR 2026
Game/Application
Category
Genre
External
Hours
Apex Legends
FPS
25.8
Euro Truck Simulator
ID
Driving
No
19.7
Stardew Valley
ID
Top-Down Sim
No
16.1
Cyberpunk 2077
ID
Open-World, World, RPG
No
14.6
Rainbow Six Siege
ID
FPS
No
13.8
Grand Theft Auto V
ID Open-World, Driving Simulation
No
11.7
Slime Rancher
ID Simulation, RPG No
No
11.1
Medieval Dynasty
ID Sandbox, Survival Survival, Co-op Factory-Building
10.7
Dinkum
ID
No
10.5
Raft
ID
No
10.3
Satisfactory
ID
No
10.1
Minecraft (SP 1.21.8)
Open- Open-World, Sandbox
No
10.1
Grounded
ID
Survival, Co-op Tactical FPS
No
10.1
Ready Or Not
ID ID ID ID DD D ID
No
10.0
Counter-Strike 2
FPS
No
9.9
Core Keeper
Sandbox, Survival Roguelike RPG
No
9.4
Barony
No
9.3
Monster Hunter Wilds
Action RPG
No
8.7
Brotato
Top-Down Shooter FPS, Battle Royale
No
6.1
PUBG: Battlegrounds
No
4.9
Total Used for Train and test
258.7
Ogu and the Secret Forest
OOD
Adventure, Puzzle
No
2.3
Battlefield (Open Beta)
OOD Collection
FPS
2.3
Eternal Return
MOBA, Survival
No
17.3
MapleStory Souths-Southperry (EA)
Collection
Open- Wpend,orld, Sandbox
Overwatch
Collection
FPS, Hero Shooter Survival, RPG
No
10.3
Enshrouded
Collection
No
10.1
Vampire Survivors
Collection
Top-Down Platformer
No
2.8
Skul
Collection
Roguelike Platformer
No
2.0
PEAK
Collection
Casual/Areade
No
1.8
Super Bunny Man
Collection
Platformer, Co-op FPS
No
0.7
VALORANT
Collection
0.3
Total (Collected)
335.6
Table 12: Collected desktop data statistics. The dataset includes internally collected demonstra- tions diverse games and applications.
and resource- maaarrcemmnggmment scenarios; detailed environment statistics are listed in Table 12 All ses- sions were screen-captured FHD or QHD 60 Hz with synchronized mouse and keyboard traces, and ocap's turnkey workflow meant anyone could gather synchronized data with minimal setup; annotators re-ran the calibration sequence whenever their hardware changed.
Game/Application
Category
Genre
External
Hours
Minecraft VPT (Baker et al. 2022
Converted Converted
Open- Woold, Sandbox
Yes
2194
CSGO - CS_DM (Pearee & Zhu 2022
FPS
Yes
100
Total (Converted)
2294.0
Table 13: Converted dataset statistics. Converted data from existing public benchmarks comple- ment the collected corpus.
20

---

Published as a conference paper at ICLR 2026
B.3 CONVERTED DATA
The converted dataset includes Minecraft demonstrations from Baker et al. (Baker et al. 2022) and Counter-Strin Strikk 2 data from Pearce et al. (Pearce Zhu 2022). These external sources were stan- dardized into the OWAMcap format, ensuring consistency and seamless integration across different datasets.
B.4 PREPROCESSED DATASET
Before training, we applied preprocessing to handle temporal offsets. Specifically, after applying temporal offset 7, only the sequences of action labels were shifted, while the observations remained unchanged. We use temporal offset of r = 100 ms to preprocess the training data for both the generalist and specialist IDM models. Additionally, we filtered inactive segments where no actions occurred for extended periods to reduce noise and improve training efficiency.
B.5 PSEUDO-LABELED DATASET
We collect high-quality YouTube gameplay videos through a combination of targeted search and bulk download. For the search phase, we used the query template "GAME_NAME no commentary," where the term no commentary is widely understood to indicate pure gameplay videos without additional overlays, commentary, or editing. After obtaining video links, we downloaded the videos using the open-source tool pollypp To ensure consistency, we restricted the maximum resolution to 480p. addition, frequent cookie renewal and download rate cap of 62.5 Mb/s were necessary to bypass YouTube's automated bot detection mechanisms. Through this pipeline, successfully curated over 1,000 hours of high-quality gameplay footage for mseudo-labeling The total collected video duration per game is summarized in Table 14
Game
Duration (h)
Stardew Valley
69.7
Minecraft
62.8
Monster Hunter Wilds
63.3
Dinkum
60.8
Satisfactory
59.8
Cyberpunk 2077
58.5
Medieval Dynasty
58.4
Raft
58.0
Core Keeper
58.0
Euro Truck Simulator 2
57.3
Grounded
57.2
Rainbow Six
56.3
GTA 5
54.1
Brotato
52.6
PUBG
50.7
Counter-Strike
49.8
Apex Legends
48.7
Slime Rancher
33.3
Ready Not
29.0
Barony
16.7
Total
1054.8
Table 14: Pseudo-labeled Duration by Game (G-IDM), Total effective hours of successfully pro- cessed lseudo-labeled data per game.
21

---

Published as a conference paper at ICLR 2026
C EVENT TOKENIZATION DETAILS
To train the Generalist IDM effectively, raw desktop interaction logs must be converted into struc- tured representation that the model can understand. We represent the entire interaction sequence as a stream of discrete event tokens. Each event corresponds to either an observation or an action. Observation events capture changes in the visual state the environment, such as screen updates (Screen Events), while action events represent user inputs, including Keyboard Events (key presses and releases) and Mouse Events (clicks, movements, and scrolls).
By tokenizing data the event level, unify heterogeneous inputs into consistent, sequential representation that can be modeled effectively using single decoder-only transformer. This repre- sentation accommodates both asynchronous observations and actions while preserving fine-grained temporal alignment between them.
C.1 EVENT TOKEN
We append specialized tokens to the model's vocabulary for desktop interaction modeling. Event structure tokens (<EVENT START> and <EVENT DEVENT_END>) delineate the boundaries of interaction sequences, while event type tokens (<KEYBOARD>, <MOUSE>, <SCREEN>) semantically catego- rize the modality of each event.
Numeric encoding tokens (<0> to <9>) serve multiple purposes:
Mouse movement deltas are encoded using a configurable base system (default: [2. 10, 10, 10]), allowing efficient representation of signed values within 1999 pixel range.
Mouse scroll values similarly quantized using base-10 10 tokens.
Timestamps encoded using temporal bases (default: [10, 10, 10), covering 10-second window with 10ms resolution. Timestamps are eyclic, wrapping from 999 back to 000.
Mouse interaction tokens include:
Sign tokens (<SIGN_PLUS>, <SIGN_MINUS>) for indicating the direction of movement deltas,
Mouse button tokens (<MB_0> to <MB_15>) for encoding mouse button flags in hexadec- imal.
Keyboard interaction tokens consist of:
Virtual key code tokens (<VK_0> <VK__V____0-_ vVK_255>) represent all Windows virtual key inputs,
Action tokens (<press>, <release>) to indicate key state transitions.
This factorized token design creates modular, modality-speciffi token spaces while maintaining a compact vocabulary. Mouse button flag definitions provided in Table 15 and the full virtual key code mapping is shown Table 16
Flag Name
Hex Value
Description
RI HOUSE_NNOS
0x0000
No operation
RI MOUSE LEFT BUTTON, DOWNNUPP
0x0001
Left button press/release
RL RLMOUSE MOUSE.RIGHT RIGHT_BUTION
RIGHT BUTTON, DOWN/UP
0x0004/0x0008
Right button press/release
RI MOMOEE MIDDLE BUTTON DOWN/UP
Ox001 0/0x0020
Middle button press/release
RI MOUSE BUTTON DOWN/UP
0x0040/0x0080
Side button press/release
RI MOUSE BUTTON 5 DOWN/UP
0x0100/0x0200
Side button 5 press/release
RILMOUSE_
0x0400
Vertical scroll wheel
JTLMOUSE_HWHRRY
0x0800
Horizontal scroll wheel
Table 15: Windows Raw Mouse Button Flags
22

---

Published as a conference paper at ICLR 2026
Key Name
VK Code
Description
Key Name
VK Code
Description
LBUTTON
Left mouse button
KEY O-KEY-
48-57
"0"-99 keys
RBUTTON
Right mouse button
KEY A-KEY
6S-90
'A''Z'' keys
CANCEL
Control-breat
LWIN
91
Left Windows key
MBUTTON
Middle mouse button
RWIN
92
Right Windows key
XBUTTONI
5
XI mouse button
APPS
93
Applications key
XBUTTON2
X2 mouse button
NUMPADO-9
96-105
Numpad 0-9
BACK
8
Backspace key
MULTIPLY
106
Numpad
TAB
9
Tab key
ADD
107
Numpad
CLEAR
12
Clear key
SUBTRACT
109
Numpad
RETURN
13
Enter key
DECIMAL
110
Numpad
SHIFT
16
Shift key
DIVIDE
111
Numpad
CONTROL
17
Curl key
F1-F12
112-123
FI-F12 function keys
MENU
Alt key
NUMLOCK
Num Lock
PAUSE
19
Pause key
SCROLL
145 Scroll Lock
CAPITAL
Caps Lock
LSHIFT
160
Left Shift
ESCAPE
27
Esc key
RSHIFT
161
Right Shift
SPACE
32
Spacebar
LCONTROL
162
Left Ctrl
PRIOR
33
Page Up
RCONTROL
163
Right Ctrl
NEXT
34
Page Down
LMENU
164
Left Alt
END
35
End key
RMENU
165
Right Alt
HOME
36
Home key
OEM_
186
key
LEFT
37
Left arrow
OEM PLUS
187
key
UP
38
Up arrow
OEM COMMA
188
key
RIGHT
Right arrow
OEM BEM_MINUS
189
key
DOWN
Down arrow
OEM PERIOD
key
INSERT
Insert key
OEM 2
191
2 key
DELETE
Delete key
OHM_3
192
key
Table 16: Windows Virtual Key Codes
C.2 EVENT TOKEN STRUCTURF
All event tokens follow a consistent structure:
DEVENT_STARTY < event_ type >< timestamp >< event detail >> <EVENT END>
where:
<EVENT_STARE and <EVENT_ENDS special tokens that delimit each event <timestamp> encodes the precise timing the event in nanoseconds
<event
_event_type>> type> specifies the type of event (e.g., <SCREEN>, <KEYBOARD>, <MOUSE>)
contains event-specific information
C.3 SCREEN EVENTS
Screen events capture visual observations from the desktop environment. Each screen event contains an image token sequence:
LEVENLLSTARD timestamp >
image_tokens
<EVENT_END:
For example:
<VENT <ER CONTEXTT 256 <EVENT END
The timestamp <1><8><<> represents 185 time units, and the image is encoded using 256 visual tokens following the Intern VL.3 tokenization scheme.
23

---

Published as a conference paper at ICLR 2026
C.4
KEYBOARD EVENTS
Keyboard events encode key press and release actions using virtual key code tokens:
<EVENT SWVENT_STTRT><<EEBOAAAA timestamp
><
token ><
action
<EVENT_END
For example:
This represents key release event timestamp 200, where <VK_32> corresponds to the spacebar. The action can be either <press> or <release>.
C.5
MOUSE EVENTS
Mouse events are the most complex among input modalities, they encode continuous movement, discrete button states, and scroll actions.
<EVENT <EVENT_START>> ST sinngdds <dy magnitude><<button flags><<croll, data>><EVENT Laa END>
sign><dx magnitude><dy sign>
The optional ascroll_dated token included only when the <button_ field indi- cates the presence of scroll wheel activity.
Mouse Movement Example Consider the following mouse event:
MMNNUS>>>>>>>>>>
<0><0><<><<>><<.BB>> <M 8> <MB DBC.CCEE_.IIIICAPLPPS----------- END>
This token sequence decoded follows:
Timestamp: <2><4><5> represents 2 x 100 4 x 10 + 5 = 245 time units.
Mouse Displacement: The displacement uses separate sign and magnitude encoding:
dx: <STeN__P10>><0>>>xx0>>>>>>>>>>>> +(0 x 1000 x + 10 2) + 2 pixels (4)
dy: <SIGNN_MMNNS2><000000>>>>>>>>><> -(0 x 1000 0 x 100 x 10 9) 19 pixels
(5)
Button Flags: <MB_4><MBB8<<MB encodes button states hexadecimal digits: 0x480
115210-
This corresponds to:
0x400: Vertical scroll wheel event
0x080: Mouse button 4 (side button) released
Scroll Data: <SIGN SEIINN_PUSS>00 indicates scroll delta (magnitude 0).
Final Interpretation: Mouse moved dx = +2, dy -19 pixels at timestamp 245, with scroll wheel activity and side button release.
D MODEL ARCHITECTURE DETAILS
For Generalist-IDM, adopt the Intern IB model (Zhu et al. 2025), which integrates Intern- VIT as the vision encoder and Qwen2.5 YYngg et al.) 2024) as the language backbone. Intern VL3 trained with native multimodal pretraining and demonstrates strong performance video-text interleaved tasks, making it a suitable foundation for our work.
We expand the model's tokenizer by adding additional event tokens represent events in desk- top data. Furthermore. we transfer the semantic initialization from corresponding regular language tokens to the newly added event tokens.
24

---

Published as a conference paper at ICLR 2026
Algorithm Streaming Autoregressive Inference for Generalist-IDM
Require: Screen events O {01 .2....,01 model Po, max context length L Ensure: Predicted action events A
1: C {01} (Initialize context window)
2: A <-D
3: for
4: loop
5: <- Generate (o, C) (Predict next event via NEP-r)
6: if (ā) fails then
7: break (Invalid event)
8: end if
9: timestamp(a) ) > timestamp(ox+1) then
10: break [Temporal stopping criterion)
11: end
12:
C < CU (ā); A+ Au {ā}
13:
end loop
C CU {01+1 (Append next observation]
15: while IC >
do
Remove oldest event from C (Trim context) end while
17:
end for
19:
return A
E TRAINING DETAILS
The Generalist-IDM was trained 8 H100 GPUs (80GB) for approximately 24 hours, totaling 192 H100-hours. The entire training process incurred a cost of only $800 for training on 259 hours of human-collected data, highlighting the efficiency enabled by OWA Toolkit.
All models were trained using a maximum context length of 8192 tokens. For the IDM models, both the generalist and specialist variants, apply temporal offset of r 100 ms when constructing the training dataset.
We used the following training schedules:
Generalist-IDM: epochs
Specialist-IDM: epochs
Generalist-IDM (fine-tuning): epochs
VAPT (w/o pseudo): epochs on the human-collected vision-action dataset
VAPT (w/ pseudo): epoch on the pseudo-labeled dataset, followed by epochs on the
human-collected dataset
All experiments were conducted using identical hyperparameters: global batch size of 128, learning rate of 2 10-3 and the AdamW optimizer.
F EVALUATION DETAILS
F.1 GENERATION METHODS
We implemented an efficient autoregressive inference pipeline for predicting keyboard and mouse actions from desktop screen captures or YouTuue videos. Starting from MCAP files containing syn- chronized, timestamped data streams (screen captures and mouse/keyboard events), we resample the events at fixed intervals (50 ms for screen and mouse events, pass-through for keyboard inputs) and tokenize them as described in Appendix A dynamic context manager maintains a sliding window of recent events with efficient embedding caching, using token context length of 2048. To accelerate inference, we apply several optimization techniques, including PyTorch model com- pilation, FlashAttention and mixed-precision computation with bfloat16. For multi-GPU inference,
25

---

Published as a conference paper at ICLR 2026
leverage NVIDIA MPS The generated token sequences decoded back into structured MCAP events and subsequently evaluated. For aseudo-labeling YouTube videos, we generate MCAP files consisting of two-minute segments of screen events, excluding the first minute and last two minutes to mitigate the influence of introductions and outros.
Throughout this work, evaluate the Generalist-IDM using fully autoregressive action decoding, both for the experiments in Section 5 and for pseudo-labeling YouTube videos. Teacher forcing was used.
F.2 EVALUATION METRICS
We evaluate the Generalist-IDM using fine-grained metrics that capture the correctness of predicted actions across both continuous (mouse) and discrete (keyboard, mouse button) modalities. All met- rics operate on non-overlapping 50 ms temporal bins (corresponding 20 Hz): within each bin, raw events are aggregated before comparison. Given an evaluation episode spanning [fsue fend), we partition the timeline into [(end - stmnt) /50mm] bins. each bin i, all mouse movement deltas (Ar, Ay) are summed to obtain a single ground-truth vector S; (8i,z, 8iy)) and a predicted vector d, (di, d y.
Pearson Correlation (X/Y). Pearson correlation measures the linear correspondence between ground-truth and predicted mouse displacement sequences, computed independently for the and Y axes:
Σ Di=1 (ii (d)(d dz)
(6)
fi. 3²)² V (di, dz)²
where 5, = S1,e and dg 1 E, di (analogously for Tyy) A value of 1.0 indicates perfect directional agreement; values near Ô indicate no linear relationship. This metric captures whether the model predicts movement in the correct direction and with the correct relative timing, independent of magnitude.
72
Scale Ratio (X/Y). Scale ratio quantifies the magnitude mismatch between ground- and pre- dicted movements along each axis:
1i=1 Si,z
ScaleRatio, ΣE
dx
To ensure interpretability, ratios < 1 are inverted so that all reported values satisfy ScaleRatio > 1. A value of 1.0 indicates perfect magnitude match; larger values indicate that predictions are systematically scaled up or down relative to the ground truth. This metric is complementary to Pearson correlation: model may achieve high correlation (correct direction) but poor scale ratio (wrong sensitivity), which is particularly relevant for mouse sensitivity calibration across games.
Keyboard Accuracy. Keyboard accuracy measures whether the model correctly predicts the set of key events within each 50 ms bin. For each bin, extract the multiset of key events, where each event is identified by its type (press/release) and virtual key code (e.g., press_ W,, Let K and preed denote the ground-trute and predicted multisets for bin i. For each unique key i E Ki U K.redd we assign binary accuracy:
[1 count(k, KTT) = count k.podd. otherwise.
acc(k, i)
(8)
The final keyboard accuracy is the mean over all key-bin pairs: KbdAcc = nŘ. ER. acc(k, 1), where IC is the set of all (key, bin) pairs with at least one event in either source or prediction.
Mouse Button/Scroll Accuracy. Mouse button accuracy follows the same per-bin binary match- ing keyboard accuracy, applied to button events (left/right/middlee down/up). Mouse scroll accu- racy compares the total scroll amount within each bin, assigning binary correctness based on exact match.
26

---

Published as a conference paper at ICLR 2026
G ADDITIONAL EVALUATION RESULTS
G.1 ABLATION STUDY ON TEMPORAL OFFSETS
We further conduct comprehensive ablation study to analyze the role of the temporal offset parameter т in Generalist-IDM.
trained models with different temporal offsets, specifically E (0, 50, 100, 150, 200} ms, and evaluated them on six in-distribution video games.
As shown in Tables 17 and removing the temporal offset entirely (T 0) leads to dramatic degradation across all metrics: Pearson correlations collapse to near zero, action-scale errors grow by more than an order of magnitude, and keypress accuracy drops sharply. This confirms that tempo- ral misalignment severely harms multimodal action prediction and that NEP without future context fundamentally insufficient.
Introducing small offset (r 50 ms) improves stability, but performance remains suboptimal, particularly for keypress prediction. This suggests that 50 ms does not provide enough future context for reliable behavior inference. Performance stabilizes once 2 100 ms, with all metrics converging regime and only minor variation between = 100 ms and т = 200 ms.
to a high-performing Notably, no single within this range consistently dominates, indicating that NEP-r robust to the exact offset choice as long as sufficient future context is provided. Based on these results, we adopt 100 ms as the default configuration in all experiments, balancing strong performance with practical responsiveness.
G.2 TRAINING LOSS CURVES
To validate that desktop pretraining provides better initialization for embodied AI tasks, we analyze the training loss curves when fine-tuning on robot manipulation (LIBERO; Figure and navigation (CANVAS; Figure 8) benchmarks, comparing the baseline (InternVLL3-1B VL.31B without desktop pre- training) against VAPT w/o pseudo. All curves are smoothed using exponential moving average (EMA) with a = 0.10 for clarity.
Across both manipulation and navigation settings, VAPT initialization leads to markedly improved optimization behavior:
Stable early-stage learning: LIBERO-Spatin and other benchmarks, the baseline ex- hibits plateau at high loss values for approximately 1,000 steps, indicating the model must learn fundamental representations from scratch. In contrast, VAPT models show smooth, consistent loss reduction from the start.
Throughout training, VAPT maintains lower loss values com- pared to the baseline, suggesting better-aligned representations for embodied control tasks.
Consistently lower loss:
H DOWNSTREAM DETAILS
H.1 ROBOT MANIPULATION
For LIBERO (Lu et 2023) evaluation, we train manipulation policy identical to openvla- oft (Kim et al. 2025). except that the vision- language backbone is replaced with Intern VL33 BB (or its OWA variant). The policy retains the regression head for continuous action prediction, employs bidirectional attention in the policy stack, and uses parallel decoding with action chunking (chunk size = 8).
The inputs consist of third-person image, wrist-camera image, the robot proprioceptive state, and a language instruction, resulting in two images per step (exocentric and egocentric). Training uses a filtered dataset where unsuccessful demonstrations removed.
Optimization follows the openvla-oft recipe: LoRA rank 32, learning rate x 10-4, batch size 8, and image augmentation enabled. Linear decay is applied after 10,000 steps, with total training budget 15,005 steps. Checkpoints are saved every 1,000 steps, keeping both periodic and latest versions.
27

---

Published as a conference paper at ICLR 2026
Pearson
Scale Ratio
Keypress Acc.
Game
Model (ms)
X
Y
X
Y
Kbd
Mouse
0
0.301
0.168
1.71
13.48
0.584
0.997
50
0.879
0.819
1.10
1.29
0.526
0.997
Apex
100
0.839
0.853
1.13
1.23
0.765
0.997
150
0,910
0.865
1.08
1.30
0.769
0.997
200
0.897
0.793
1.28
2.24
0.760
0.998
0.101
-0.098
1.28
24.66
0.636
0.972
50
0.780
0.487
1.08
2.07
0.575
0.972
GTA
100
0.794
0.839
1.09
1.42
0.698
0,941
150
0.906
0.526
1.07
1.19
0.745
0.971
200
0.920
0.782
1.27
1.42
0.734
0.879
0
-0,016
-0.007
13.06
50.40
0.077
0.925
50
0.655
0.607
1.43
1.99
0.353
0.911
Minecraft
100
0,803
0.784
1.24
1.27
0.610
0.917
150
0.854
0.906
1.07
1.07
0.741
0.938
200
0.908
0.895
1.19
1.16
0.753
0.933
0
0.099
0.165
1.50
26.09
0.439
0.977
50
0.962
0.938
1.03
1.03
0.456
0.980
Brotato
100
0.737
0.820
1.37
1.29
0.864
0.985
150
0.963
0.953
1,07
1.04
0.873
0.990
200
0.944
0.910
1.18
1.10
0.795
0.985
0
0.139
0.102
4.18
14.67
0.229
0.960
50
0.748
0.801
1.08
1.10
0.526
0.894
Stardew
100
0,830
0.756
1.13
1.17
0.744
0.964
150
0.823
0.851
1.04
1.05
0.781
0.950
200
0.796
0.768
1.39
1.58
0.704
0.942
0
0.024
-0.027
132.21
4.51
0.354
0.933
50
0.793
0.755
1.29
1.38
0.514
0.935
Core Keeper 100
0.773
0,645
1,43
1.51
0.700
0.940
150
0.888
0.805
1.17
1.19
0.690
0.943
200
0.894
0.872
1.35
1.41
0.696
0.948
Table 17: Ablation temporal offsets for in-domain games (0-200ms)
T (ms)
Avg Pearson
Avg Pearson
Avg Scale
Avg Scale Y
Avg Keypress (Khd)
Avg Keypress (Mouse)
0.108
0.050
25.66
22,30
0.386
0.961
50
0.803
0.734
1.17
1.48
0.492
0.948
100
0.796
0.783
1.23
1.31
0.730
0.957
150
0.818
1.14
0.767
0,965
200
0.893
0.837
1.28
1.49
0.740
0.947
Table 18: Aggregate performance across all games for different temporal offsets r. Per-game break- down is provided in Appendix G.1
Training is conducted on a single node with 8 GPUs via torchrun, with the same launch flags as openvla-oft. except for swapping the backbone to Intern VLteIVLL-II/OWW.
Evaluation is performed the LIBERO benchmark (Liu al. 2023), which includes four suites of manipulation tasks: (1) Spatial, varying scene layouts with fixed objects; (2) Object, varying the set of objects in fixed scene; (3) Goal, testing goal-conditioned behavior; and (4) Long (LIBERO. 10), long-hoo long-horizon compositional tasks involving diverse objects, layouts, and goals. We report the average success rate over 500 episodes for each suite.
In the Meta- World (Yu et al. 2020) evaluation, we use the official LeRobot (Cadene et al. 2024) v codebase train and evaluate the models across various tasks of different difficulty levels:
28

---

Published as a conference paper at ICLR 2026
LIBERO-Spatial
Baseline
VAPT
LIBERO-GoAl
0.6
0.5
Loss
0.4
Loss
0.3
0.2
0.2
0.1
2500
5000
7500
10000 12500 15000
0
2500
5000
7500
10000 12500
15000
Step
Step
LIBERO-Long
LIBERO-Object
0.6
0.5
0.5
0.4
Loss
0.4
Loss
0.3
0.3
0.2
0.2
0.1
0.1
0
2500
5000
10000 Step
7500
12500 15000
0
2500
5000
10000 12500 15000 Step
7500
Figure 7: Training loss curves for all four LIBERO suites (Spatial, Goal, Long, Object). VAPT mod- els consistently show immediate convergence without the initial plateau observed in the baseline.
CANVAS
Baceline
Los5
2000
4000
6000
8000
10000
12000
14000
15020
Figure 8: Training loss curves for CANVAS navigation tasks.
Easy, Medium, Hard, and Very Hard. The training process involves 50,000 steps with learning rate of 1 x 10-4, with all other hyperparameters left at default settings. The InternVL3.1. IB backbone adapted for use with the VAPT model following the same protocol as SmolVLA (Shukor al.) 2025), without modifying any architecture parameters. Each task is evaluated using 10 episodes, and the success rates are reported for each difficulty level.
For the real-world evaluation, we follow the protocol used in SmolVLA (Shukor et al.) 2025) for a pick-and-place task with the SO101 robot arm. The task is defined by the instruction: "Pick the blue cube and place it in the white box." with the cube placed five distinct initial positions. We use the LeRobot (Cadene et al. 2024) framework for data collection, training, and evaluation. The setup includes two RGB cameras (top and side views), green-screen background, and fixed initial pose. total of 208 demonstration episodes are collected, and both bascline and VAPT models are trained using the same downstream learning protocol. Each trained model is evaluated using 30
29

---

Published as a conference paper at ICLR 2026
rollouts (two trials per cube position) with success measured correctly grasping and placing the cube inside the box.
H.2 ROBOT NAVIGATION
We established baseline following CANVAS (Choi al., 2024) by training an InternVL3-basse model architecture on the COMMAND dataset. The baseline model was initialized with the de- fault In3ernVL3 weights, whereas the VAPT w/o pseudo and VAPT w pseudo were trained from pretrained weights. All models were trained with full parameter unfreezing.
For optimization, we employed AdamW with separate learning rates: 2 x 10-5 for the LLM, and 5 x 10-6 for both the projector and vision encoder. Training was conducted with a batch size of 32 over epochs, and each model utilized 128 waypoint tokens. In the main experiments, inference of CANVAS models was performed on single NVIDIA H100 GPU. All evaluations were repeated three times per test dataset with randomized initial orientations.
I ETHICS STATEMENT
We acknowledge and adhere to the ICLR Code of Ethics.
Human Data Collection. Our dataset collected from 14 volunteer annotators who provided informed consent for gameplay recordings. Participants were fully informed about screen capture and input logging procedures and could withdraw any time. All data underwent automated and manual review to remove any personally identifiable information before research use.
Public Data Usage. We processed only publicly available YouTube videos with permissive li- censes for pseudo-labeling. Our focus gaming content inherently minimizes privacy concerns compared to general desktop recording, as gaming interfaces rarely contain sensitive personal infor- mation.
Transparency and Responsible Release. To ensure responsible use, we will publicly release all code, data collection tools, and model weights with comprehensive documentation. We acknowl- edge that vision-action models could have dual-use potential; however, our focus on standardized gaming environments and transparent methodology helps mitigate misuse risks. Our computational approach (requiring only modest GPU resources) democratizes access while reducing environmental impact compared to large-scale training paradigms.
LIMITATIONS
While we validate our approach on both simulation benchmarks (LIBERO, Meta-World, CANVAS) and real-world pick-and-place task with the SO-101 robot arm, the real-world evaluation remains limited single task setting. Broader real-robot validation across diverse manipulation and navi- gation scenarios is important direction for future work. The differential impact of pseudo-labels (improving navigation but degrading manipulation) suggests task-specific transfer mechanisms that require further investigation. Our dataset focuses primarily gaming interactions, which may not capture the full spectrum of desktop activities relevant to general-purpose robotics. Despite these constraints, framework democratizes embodied AI research by reducing storage requirements by 152x x and training costs to under $1000, making large-scale vision-action pretraining accessible resource- limited academic labs.
30


Reviews:
[{"review_id": "lDlrx5YaRw", "reviewer": "Reviewer_5KYr", "summary": "This paper proposes D2E, a framework for pretraining models on desktop data (e.g., game videos paired with human actions). D2E offers a comprehensive toolbox that includes standardized data collection and storage, efficient data reading, and a scalable pseudo-label annotation strategy for desktop data, powered by a specifically designed and trained Inverse Dynamics Model (IDM). Extensive experiments are conducted to validate the framework across multiple aspects, including data collection, processing, and reading efficiency, as well as the effectiveness of using D2E for pretraining embodied models. The results show that adapting the pretrained models to embodied and navigation tasks leads to notable performance improvements, revealing a promising direction for leveraging scalable desktop data in embodied model pretraining.", "questions": "1. Could the authors provide more details about the baseline implementations reported in Table 1? It would be helpful to clarify whether these baselines were reimplemented under the same settings or adopted from existing works.\n\n2. There remains a considerable gap between game data and embodied manipulation data due to the coarse-grained differences in action spaces and the varying requirements for 3D physical understanding. A more comprehensive analysis is needed to quantify how much embodied manipulation tasks actually benefit from desktop data pretraining, and under what conditions such transfer is most effective.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Given the high cost of embodied data collection, exploring new data sources that capture human intention knowledge and can contribute to embodied learning is a highly meaningful research direction. While desktop data has been explored in previous works for training embodied models, those methods primarily focus on developing and validating models within in-domain game environments. This paper extends beyond that scope, aiming to make desktop data more broadly useful for embodied learning.\n\n2. The high-quality datasets collected and the well-designed data collection pipeline represent valuable contributions to the research community. They provide essential infrastructure for future work on large-scale embodied pretraining, facilitating both reproducibility and scalability.\n\n3. The experiments and analyses conducted on each component of the proposed framework are comprehensive. Beyond demonstrating the feasibility of pretraining on desktop data, the paper offers a practical reference and a solid foundation for developing future learning systems capable of utilizing any video-based data for embodied intelligence.\n\n4. The significant performance gains observed on downstream tasks following D2E pretraining clearly demonstrate the framework’s effectiveness in extracting generalizable knowledge from desktop data, highlighting its potential as a scalable and efficient pretraining paradigm.", "weaknesses": "1. The utilization of data with machine-generated pseudo labels leads to inconsistent performance changes across the Manipulation and Navigation benchmarks. This inconsistency suggests that the quality and reliability of pseudo labels may vary significantly depending on the task type, and further analysis is needed to understand their impact on downstream performance.\n\n2. Conducting experiments solely on the Libero benchmark is insufficient to support the claim that pretraining on desktop data contributes to learning generalized embodied knowledge. Validation on real-world manipulation tasks would substantially strengthen the paper’s argument. \n\n## Minor Issue\n\n1. The font style used in the paper is inconsistent with the official ICLR template, and should be adjusted to comply with the formatting standards.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kXU6XMWRuV", "reviewer": "Reviewer_cyb3", "summary": "This paper presents D2E, a framework that scales vision-action pretraining using large-scale desktop interaction data (screen, keyboard, mouse) instead of costly physical robot trajectories. It introduces the OWA Toolkit for efficient multimodal data capture, an IDM for pseudo-labeling internet gameplay videos, and a VAPT that transfers learned representations to robotics. Trained on about 1.3K hours of human and pseudo-labeled data, the model achieves 96.6% success on libero and 83.3% on canvas navigation, showing that desktop interactions can effectively serve as scalable pretraining for embodied AI.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. End-to-end pipeline and scale: The paper provides a complete framework covering data collection, pseudo-labeling, pretraining, and robotic transfer. It builds on 31 desktop games with 335 hours of human demonstrations and expands to over 1,000 hours of pseudo-labeled YouTube gameplay, achieving clear scalability.\n\n2. Solid engineering contribution: The OWA toolkit enables synchronized screen, keyboard, and mouse capture with nanosecond precision and 152× compression, significantly reducing random-access I/O cost and improving data-loading throughput during training.\n\n3. Openness and reproducibility: The authors commit to releasing the toolkit, datasets, and pretrained models with detailed documentation, ensuring transparency and easy reproducibility for future research.", "weaknesses": "The experimental evidence supporting why and when desktop data benefits embodied tasks remains insufficient. While results show positive transfer, the paper lacks a deeper analysis of task suitability—for example, whether desktop-derived sensorimotor patterns genuinely align with the fine-grained control and contact dynamics required in manipulation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9B0tMAyuyZ", "reviewer": "Reviewer_SNK1", "summary": "The paper introduces D2E, a framework that leverages large-scale desktop interactions collected via the OWA toolkit and generalized through a timestamp-based IDM for pretraining embodied AI. Using more than 1.3K hours of human and pseudo-labeled gameplay data, the approach demonstrates strong transfer to robotics tasks, achieving 96.6% on LIBERO manipulation and 83.3% on CANVAS navigation.", "questions": "Did the pretraining stage also include robotics data, or was it purely based on desktop game data?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Using game data for embodied pretraining is an interesting direction.\n\n2. The paper provides a detailed system design for data collection.", "weaknesses": "1. The main contribution of the paper lies in how to collect action data and in the system-level design, while the algorithmic innovation is relatively limited. Currently, there is a large body of work that uses OOD data for pretraining, so my concern is that this paper may not be a very good fit for a learning-focused conference.\n\n2. Using game data for navigation tasks in CANVAS makes sense, but its effectiveness for manipulation tasks remains questionable. In addition, LIBERO can be easily hacked with certain tricks, which makes it difficult to serve as a fair or reliable benchmark for validating algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "78zER026o5", "reviewer": "Reviewer_rEUi", "summary": "This paper proposes D2E (Desktop-to-Embodied AI), a framework that leverages large-scale desktop interaction data (screen, keyboard, mouse) as an alternative to expensive real-world embodied trajectories for pretraining vision-action models.\nThe system consists of three main components: (1) OWA Toolkit – a high-performance desktop data collection and compression pipeline based on an extended MCAP format (OWAMcap), achieving up to 152× compression over prior datasets. (2)Generalist-IDM – a timestamp-aware inverse dynamics model (NEP-τ) that predicts human actions from videos and is used to pseudo-label over 1K hours of YouTube gameplay. (3)VAPT – a vision-action pretrained model that transfers desktop-learned representations to robot manipulation (LIBERO) and navigation (CANVAS), achieving 96.6% and 83.3% success rates respectively.", "questions": "1. Could the authors provide a deeper analysis of how the temporal offset mechanism (NEP-τ) contributes to the observed improvements? In particular, how do results change if τ is removed or varied?\n\n2. To what extent do the downstream results of VAPT originate from the model design (NEP-τ, Generalist-IDM) versus the quality and diversity of the collected OWA data? Some controlled comparisons would help isolate these factors.\n\n3. The paper shows strong zero-shot transfer to unseen games and robotic tasks. What is the hypothesized mechanism behind this generalization? Are certain components (e.g., timestamp-based tokenization) more critical than others?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. **Solid engineering contribution**: The OWA Toolkit is an impressive system-level effort that enables synchronized, multimodal desktop data collection and efficient storage. The compression performance and open-source reproducibility are highly commendable.\n\n2. **Reproducibility and openness**: The authors provide thorough implementation details, training settings, and datasets, which greatly enhance the paper’s credibility and community value.", "weaknesses": "1. **Lack of academic novelty**:\nThe overall contribution is primarily engineering-oriented rather than conceptual or algorithmic. While the proposed OWA Toolkit and data infrastructure are impressive from a systems perspective, the work introduces limited new ideas in terms of representation learning, model design, or theoretical insight.\n\n2. **Generalist-IDM design appears incremental**:\nThe core methodological component—Generalist Inverse Dynamics Model (Generalist-IDM)—is not sufficiently novel or deeply motivated. The NEP-τ formulation is essentially an incremental extension of standard next-event prediction, and its role in improving downstream performance remains unclear. It is uncertain whether the strong results of VAPT stem from the NEP-τ design itself or simply from the high-quality and diverse data collected by OWA Toolkit.\n\n3. **Lack of ablation and sensitivity analysis**:\nThe paper provides limited empirical evidence dissecting the effectiveness of individual components. In particular, the temporal offset parameter τ is introduced as a key idea, yet there is no systematic study on how τ is selected or how sensitive the model performance is to its value. Without such analysis, the robustness and generality of the proposed modeling choice remain uncertain.\n\n4. **Missing causal connection between design and outcome**:\nAlthough the experiments show that a model trained on limited human desktop data and pseudo-labeled gameplay videos can generalize across unseen domains, the paper does not clearly explain why this happens. It lacks causal analysis connecting the proposed components (e.g., NEP-τ, OWA data quality, pseudo-labeling) to the observed generalization behavior. As a result, it is difficult to attribute the performance gains to specific methodological factors rather than data scale or diversity alone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)