D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to
  Embodied AI
Authors: name: Suwhan Choi; name: Jaeyoon Jung; name: Haebin Seong; name: Minchan Kim; name: Minyeong Kim; name: Yongjun Cho; name: Yoonshik Kim; name: Yubeen Park; name: Youngjae Yu; name: Yunsung Lee
Published: 2025-10-07 08:40:33

Abstract: Large language models leverage internet-scale text data, yet embodied AI
remains constrained by the prohibitive costs of physical trajectory collection.
Desktop environments -- particularly gaming -- offer a compelling alternative:
they provide rich sensorimotor interactions at scale while maintaining the
structured observation-action coupling essential for embodied learning. We
present D2E (Desktop to Embodied AI), a framework that demonstrates desktop
interactions can serve as an effective pretraining substrate for robotics
embodied AI tasks. Unlike prior work that remained domain-specific (e.g., VPT
for Minecraft) or kept data proprietary (e.g., SIMA), D2E establishes a
complete pipeline from scalable desktop data collection to verified transfer in
embodied domains. Our framework comprises three components: (1) the OWA Toolkit
that unifies diverse desktop interactions into a standardized format with 152x
compression, (2) the Generalist-IDM that achieves strong zero-shot
generalization across unseen games through timestamp-based event prediction,
enabling internet-scale pseudo-labeling, and (3) VAPT that transfers
desktop-pretrained representations to physical manipulation and navigation.
Using 1.3K+ hours of data (259 hours of human demonstrations, and 1K+ hours of
pseudo-labeled gameplay), we achieve a total of 96.6% success rate on LIBERO
manipulation and 83.3% on CANVAS navigation benchmarks. This validates that
sensorimotor primitives in digital interactions exhibit sufficient invariance
to transfer meaningfully to physical embodied tasks, establishing desktop
pretraining as a practical paradigm for robotics. We will make all our work
public, including the OWA toolkit, datasets of human-collected and
pseudo-labeled, and VAPT-trained models available at
https://worv-ai.github.io/d2e/