From Virtual Games to Real-World Play
Authors: name: Wenqiang Sun; name: Fangyun Wei; name: Jinjing Zhao; name: Xi Chen; name: Zilong Chen; name: Hongyang Zhang; name: Jun Zhang; name: Yan Lu
Published: 2025-06-23 17:59:53

Abstract: We introduce RealPlay, a neural network-based real-world game engine that
enables interactive video generation from user control signals. Unlike prior
works focused on game-style visuals, RealPlay aims to produce photorealistic,
temporally consistent video sequences that resemble real-world footage. It
operates in an interactive loop: users observe a generated scene, issue a
control command, and receive a short video chunk in response. To enable such
realistic and responsive generation, we address key challenges including
iterative chunk-wise prediction for low-latency feedback, temporal consistency
across iterations, and accurate control response. RealPlay is trained on a
combination of labeled game data and unlabeled real-world videos, without
requiring real-world action annotations. Notably, we observe two forms of
generalization: (1) control transfer-RealPlay effectively maps control signals
from virtual to real-world scenarios; and (2) entity transfer-although training
labels originate solely from a car racing game, RealPlay generalizes to control
diverse real-world entities, including bicycles and pedestrians, beyond
vehicles. Project page can be found: https://wenqsun.github.io/RealPlay/