World-consistent Video Diffusion with Explicit 3D Modeling
Authors: name: Qihang Zhang; name: Shuangfei Zhai; name: Miguel Angel Bautista; name: Kevin Miao; name: Alexander Toshev; name: Joshua Susskind; name: Jiatao Gu
Published: 2024-12-02 18:58:23

Abstract: Recent advancements in diffusion models have set new benchmarks in image and
video generation, enabling realistic visual synthesis across single- and
multi-frame contexts. However, these models still struggle with efficiently and
explicitly generating 3D-consistent content. To address this, we propose
World-consistent Video Diffusion (WVD), a novel framework that incorporates
explicit 3D supervision using XYZ images, which encode global 3D coordinates
for each image pixel. More specifically, we train a diffusion transformer to
learn the joint distribution of RGB and XYZ frames. This approach supports
multi-task adaptability via a flexible inpainting strategy. For example, WVD
can estimate XYZ frames from ground-truth RGB or generate novel RGB frames
using XYZ projections along a specified camera trajectory. In doing so, WVD
unifies tasks like single-image-to-3D generation, multi-view stereo, and
camera-controlled video generation. Our approach demonstrates competitive
performance across multiple benchmarks, providing a scalable solution for
3D-consistent video and image generation with a single pretrained model.