Training-free Camera Control for Video Generation
Authors: name: Chen Hou; name: Guoqiang Wei; name: Yan Zeng; name: Zhibo Chen
Published: 2024-06-14 15:33:00

Abstract: We propose a training-free and robust solution to offer camera movement
control for off-the-shelf video diffusion models. Unlike previous work, our
method does not require any supervised finetuning on camera-annotated datasets
or self-supervised training via data augmentation. Instead, it can be plugged
and played with most pretrained video diffusion models and generate camera
controllable videos with a single image or text prompt as input. The
inspiration of our work comes from the layout prior that intermediate latents
hold towards generated results, thus rearranging noisy pixels in them will make
output content reallocated as well. As camera move could also be seen as a kind
of pixel rearrangement caused by perspective change, videos could be
reorganized following specific camera motion if their noisy latents change
accordingly. Established on this, we propose our method CamTrol, which enables
robust camera control for video diffusion models. It is achieved by a two-stage
process. First, we model image layout rearrangement through explicit camera
movement in 3D point cloud space. Second, we generate videos with camera motion
using layout prior of noisy latents formed by a series of rearranged images.
Extensive experiments have demonstrated the robustness our method holds in
controlling camera motion of generated videos. Furthermore, we show that our
method can produce impressive results in generating 3D rotation videos with
dynamic content. Project page at https://lifedecoder.github.io/CamTrol/.