Matrix-3D: Omnidirectional Explorable 3D World Generation
Authors: name: Zhongqi Yang; name: Wenhang Ge; name: Yuqi Li; name: Jiaqi Chen; name: Haoyuan Li; name: Mengyin An; name: Fei Kang; name: Hua Xue; name: Baixin Xu; name: Yuyang Yin; name: Eric Li; name: Yang Liu; name: Yikai Wang; name: Hao-Xiang Guo; name: Yahui Zhou
Published: 2025-08-11 15:29:57

Abstract: Explorable 3D world generation from a single image or text prompt forms a
cornerstone of spatial intelligence. Recent works utilize video model to
achieve wide-scope and generalizable 3D world generation. However, existing
approaches often suffer from a limited scope in the generated scenes. In this
work, we propose Matrix-3D, a framework that utilize panoramic representation
for wide-coverage omnidirectional explorable 3D world generation that combines
conditional video generation and panoramic 3D reconstruction. We first train a
trajectory-guided panoramic video diffusion model that employs scene mesh
renders as condition, to enable high-quality and geometrically consistent scene
video generation. To lift the panorama scene video to 3D world, we propose two
separate methods: (1) a feed-forward large panorama reconstruction model for
rapid 3D scene reconstruction and (2) an optimization-based pipeline for
accurate and detailed 3D scene reconstruction. To facilitate effective
training, we also introduce the Matrix-Pano dataset, the first large-scale
synthetic collection comprising 116K high-quality static panoramic video
sequences with depth and trajectory annotations. Extensive experiments
demonstrate that our proposed framework achieves state-of-the-art performance
in panoramic video generation and 3D world generation. See more in
https://matrix-3d.github.io.