4K4DGen: Panoramic 4D Generation at 4K Resolution
Authors: name: Renjie Li; name: Panwang Pan; name: Bangbang Yang; name: Dejia Xu; name: Shijie Zhou; name: Xuanyang Zhang; name: Zeming Li; name: Achuta Kadambi; name: Zhangyang Wang; name: Zhiwen Fan
Published: 2024-06-19 13:11:02

Abstract: The blooming of virtual reality and augmented reality (VR/AR) technologies
has driven an increasing demand for the creation of high-quality, immersive,
and dynamic environments. However, existing generative techniques either focus
solely on dynamic objects or perform outpainting from a single perspective
image, failing to meet the needs of VR/AR applications. In this work, we tackle
the challenging task of elevating a single panorama to an immersive 4D
experience. For the first time, we demonstrate the capability to generate
omnidirectional dynamic scenes with 360-degree views at 4K resolution, thereby
providing an immersive user experience. Our method introduces a pipeline that
facilitates natural scene animations and optimizes a set of 4D Gaussians using
efficient splatting techniques for real-time exploration. To overcome the lack
of scene-scale annotated 4D data and models, especially in panoramic formats,
we propose a novel Panoramic Denoiser that adapts generic 2D diffusion priors
to animate consistently in 360-degree images, transforming them into panoramic
videos with dynamic scenes at targeted regions. Subsequently, we elevate the
panoramic video into a 4D immersive environment while preserving spatial and
temporal consistency. By transferring prior knowledge from 2D models in the
perspective domain to the panoramic domain and the 4D lifting with spatial
appearance and geometry regularization, we achieve high-quality Panorama-to-4D
generation at a resolution of (4096 times 2048) for the first time. See the
project website at https://4k4dgen.github.io.