SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View
  Consistency
Authors: name: Yiming Xie; name: Chun-Han Yao; name: Vikram Voleti; name: Huaizu Jiang; name: Varun Jampani
Published: 2024-07-24 17:59:43

Abstract: We present Stable Video 4D (SV4D), a latent video diffusion model for
multi-frame and multi-view consistent dynamic 3D content generation. Unlike
previous methods that rely on separately trained generative models for video
generation and novel view synthesis, we design a unified diffusion model to
generate novel view videos of dynamic 3D objects. Specifically, given a
monocular reference video, SV4D generates novel views for each video frame that
are temporally consistent. We then use the generated novel view videos to
optimize an implicit 4D representation (dynamic NeRF) efficiently, without the
need for cumbersome SDS-based optimization used in most prior works. To train
our unified novel view video generation model, we curated a dynamic 3D object
dataset from the existing Objaverse dataset. Extensive experimental results on
multiple datasets and user studies demonstrate SV4D's state-of-the-art
performance on novel-view video synthesis as well as 4D generation compared to
prior works.