Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusion
Authors: name: Rishab Parthasarathy; name: Zack Ankner; name: Aaron Gokaslan
Published: 2024-06-17 04:09:04

Abstract: A recent frontier in computer vision has been the task of 3D video
generation, which consists of generating a time-varying 3D representation of a
scene. To generate dynamic 3D scenes, current methods explicitly model 3D
temporal dynamics by jointly optimizing for consistency across both time and
views of the scene. In this paper, we instead investigate whether it is
necessary to explicitly enforce multiview consistency over time, as current
approaches do, or if it is sufficient for a model to generate 3D
representations of each timestep independently. We hence propose a model,
Vid3D, that leverages 2D video diffusion to generate 3D videos by first
generating a 2D "seed" of the video's temporal dynamics and then independently
generating a 3D representation for each timestep in the seed video. We evaluate
Vid3D against two state-of-the-art 3D video generation methods and find that
Vid3D is achieves comparable results despite not explicitly modeling 3D
temporal dynamics. We further ablate how the quality of Vid3D depends on the
number of views generated per frame. While we observe some degradation with
fewer views, performance degradation remains minor. Our results thus suggest
that 3D temporal knowledge may not be necessary to generate high-quality
dynamic 3D scenes, potentially enabling simpler generative algorithms for this
task.