OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling
Authors: name: Yang Zhou; name: Yifan Wang; name: Jianjun Zhou; name: Wenzheng Chang; name: Haoyu Guo; name: Zizun Li; name: Kaijing Ma; name: Xinyue Li; name: Yating Wang; name: Haoyi Zhu; name: Mingyu Liu; name: Dingning Liu; name: Jiange Yang; name: Zhoujie Fu; name: Junyi Chen; name: Chunhua Shen; name: Jiangmiao Pang; name: Kaipeng Zhang; name: Tong He
Published: 2025-09-15 17:59:19

Abstract: The field of 4D world modeling - aiming to jointly capture spatial geometry
and temporal dynamics - has witnessed remarkable progress in recent years,
driven by advances in large-scale generative models and multimodal learning.
However, the development of truly general 4D world models remains fundamentally
constrained by the availability of high-quality data. Existing datasets and
benchmarks often lack the dynamic complexity, multi-domain diversity, and
spatial-temporal annotations required to support key tasks such as 4D geometric
reconstruction, future prediction, and camera-control video generation. To
address this gap, we introduce OmniWorld, a large-scale, multi-domain,
multi-modal dataset specifically designed for 4D world modeling. OmniWorld
consists of a newly collected OmniWorld-Game dataset and several curated public
datasets spanning diverse domains. Compared with existing synthetic datasets,
OmniWorld-Game provides richer modality coverage, larger scale, and more
realistic dynamic interactions. Based on this dataset, we establish a
challenging benchmark that exposes the limitations of current state-of-the-art
(SOTA) approaches in modeling complex 4D environments. Moreover, fine-tuning
existing SOTA methods on OmniWorld leads to significant performance gains
across 4D reconstruction and video generation tasks, strongly validating
OmniWorld as a powerful resource for training and evaluation. We envision
OmniWorld as a catalyst for accelerating the development of general-purpose 4D
world models, ultimately advancing machines' holistic understanding of the
physical world.