DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise
  Motion Control
Authors: name: Yujie Wei; name: Shiwei Zhang; name: Hangjie Yuan; name: Xiang Wang; name: Haonan Qiu; name: Rui Zhao; name: Yutong Feng; name: Feng Liu; name: Zhizhong Huang; name: Jiaxin Ye; name: Yingya Zhang; name: Hongming Shan
Published: 2024-10-17 17:52:57

Abstract: Recent advances in customized video generation have enabled users to create
videos tailored to both specific subjects and motion trajectories. However,
existing methods often require complicated test-time fine-tuning and struggle
with balancing subject learning and motion control, limiting their real-world
applications. In this paper, we present DreamVideo-2, a zero-shot video
customization framework capable of generating videos with a specific subject
and motion trajectory, guided by a single image and a bounding box sequence,
respectively, and without the need for test-time fine-tuning. Specifically, we
introduce reference attention, which leverages the model's inherent
capabilities for subject learning, and devise a mask-guided motion module to
achieve precise motion control by fully utilizing the robust motion signal of
box masks derived from bounding boxes. While these two components achieve their
intended functions, we empirically observe that motion control tends to
dominate over subject learning. To address this, we propose two key designs: 1)
the masked reference attention, which integrates a blended latent mask modeling
scheme into reference attention to enhance subject representations at the
desired positions, and 2) a reweighted diffusion loss, which differentiates the
contributions of regions inside and outside the bounding boxes to ensure a
balance between subject and motion control. Extensive experimental results on a
newly curated dataset demonstrate that DreamVideo-2 outperforms
state-of-the-art methods in both subject customization and motion control. The
dataset, code, and models will be made publicly available.