VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
Authors: name: Sherwin Bahmani; name: Ivan Skorokhodov; name: Aliaksandr Siarohin; name: Willi Menapace; name: Guocheng Qian; name: Michael Vasilkovsky; name: Hsin-Ying Lee; name: Chaoyang Wang; name: Jiaxu Zou; name: Andrea Tagliasacchi; name: David B. Lindell; name: Sergey Tulyakov
Published: 2024-07-17 17:59:05

Abstract: Modern text-to-video synthesis models demonstrate coherent, photorealistic
generation of complex videos from a text description. However, most existing
models lack fine-grained control over camera movement, which is critical for
downstream applications related to content creation, visual effects, and 3D
vision. Recently, new methods demonstrate the ability to generate videos with
controllable camera poses these techniques leverage pre-trained U-Net-based
diffusion models that explicitly disentangle spatial and temporal generation.
Still, no existing approach enables camera control for new, transformer-based
video diffusion models that process spatial and temporal information jointly.
Here, we propose to tame video transformers for 3D camera control using a
ControlNet-like conditioning mechanism that incorporates spatiotemporal camera
embeddings based on Plucker coordinates. The approach demonstrates
state-of-the-art performance for controllable video generation after
fine-tuning on the RealEstate10K dataset. To the best of our knowledge, our
work is the first to enable camera control for transformer-based video
diffusion models.