VideoMaker: Zero-shot Customized Video Generation with the Inherent
  Force of Video Diffusion Models
Authors: name: Tao Wu; name: Yong Zhang; name: Xiaodong Cun; name: Zhongang Qi; name: Junfu Pu; name: Huanzhang Dou; name: Guangcong Zheng; name: Ying Shan; name: Xi Li
Published: 2024-12-27 13:49:25

Abstract: Zero-shot customized video generation has gained significant attention due to
its substantial application potential. Existing methods rely on additional
models to extract and inject reference subject features, assuming that the
Video Diffusion Model (VDM) alone is insufficient for zero-shot customized
video generation. However, these methods often struggle to maintain consistent
subject appearance due to suboptimal feature extraction and injection
techniques. In this paper, we reveal that VDM inherently possesses the force to
extract and inject subject features. Departing from previous heuristic
approaches, we introduce a novel framework that leverages VDM's inherent force
to enable high-quality zero-shot customized video generation. Specifically, for
feature extraction, we directly input reference images into VDM and use its
intrinsic feature extraction process, which not only provides fine-grained
features but also significantly aligns with VDM's pre-trained knowledge. For
feature injection, we devise an innovative bidirectional interaction between
subject features and generated content through spatial self-attention within
VDM, ensuring that VDM has better subject fidelity while maintaining the
diversity of the generated video.Experiments on both customized human and
object video generation validate the effectiveness of our framework.