Image Conductor: Precision Control for Interactive Video Synthesis
Authors: name: Yaowei Li; name: Xintao Wang; name: Zhaoyang Zhang; name: Zhouxia Wang; name: Ziyang Yuan; name: Liangbin Xie; name: Yuexian Zou; name: Ying Shan
Published: 2024-06-21 17:55:05

Abstract: Filmmaking and animation production often require sophisticated techniques
for coordinating camera transitions and object movements, typically involving
labor-intensive real-world capturing. Despite advancements in generative AI for
video creation, achieving precise control over motion for interactive video
asset generation remains challenging. To this end, we propose Image Conductor,
a method for precise control of camera transitions and object movements to
generate video assets from a single image. An well-cultivated training strategy
is proposed to separate distinct camera and object motion by camera LoRA
weights and object LoRA weights. To further address cinematographic variations
from ill-posed trajectories, we introduce a camera-free guidance technique
during inference, enhancing object movements while eliminating camera
transitions. Additionally, we develop a trajectory-oriented video motion data
curation pipeline for training. Quantitative and qualitative experiments
demonstrate our method's precision and fine-grained control in generating
motion-controllable videos from images, advancing the practical application of
interactive video synthesis. Project webpage available at
https://liyaowei-stu.github.io/project/ImageConductor/