CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models
Authors: name: Rundi Wu; name: Ruiqi Gao; name: Ben Poole; name: Alex Trevithick; name: Changxi Zheng; name: Jonathan T. Barron; name: Aleksander Holynski
Published: 2024-11-27 18:57:16

Abstract: We present CAT4D, a method for creating 4D (dynamic 3D) scenes from monocular
video. CAT4D leverages a multi-view video diffusion model trained on a diverse
combination of datasets to enable novel view synthesis at any specified camera
poses and timestamps. Combined with a novel sampling approach, this model can
transform a single monocular video into a multi-view video, enabling robust 4D
reconstruction via optimization of a deformable 3D Gaussian representation. We
demonstrate competitive performance on novel view synthesis and dynamic scene
reconstruction benchmarks, and highlight the creative capabilities for 4D scene
generation from real or generated videos. See our project page for results and
interactive demos: cat-4d.github.io.