Make-Your-Video: Customized Video Generation Using Textual and
  Structural Guidance
Authors: name: Jinbo Xing; name: Menghan Xia; name: Yuxin Liu; name: Yuechen Zhang; name: Yong Zhang; name: Yingqing He; name: Hanyuan Liu; name: Haoxin Chen; name: Xiaodong Cun; name: Xintao Wang; name: Ying Shan; name: Tien-Tsin Wong
Published: 2023-06-01 17:43:27

Abstract: Creating a vivid video from the event or scenario in our imagination is a
truly fascinating experience. Recent advancements in text-to-video synthesis
have unveiled the potential to achieve this with prompts only. While text is
convenient in conveying the overall scene context, it may be insufficient to
control precisely. In this paper, we explore customized video generation by
utilizing text as context description and motion structure (e.g. frame-wise
depth) as concrete guidance. Our method, dubbed Make-Your-Video, involves
joint-conditional video generation using a Latent Diffusion Model that is
pre-trained for still image synthesis and then promoted for video generation
with the introduction of temporal modules. This two-stage learning scheme not
only reduces the computing resources required, but also improves the
performance by transferring the rich concepts available in image datasets
solely into video generation. Moreover, we use a simple yet effective causal
attention mask strategy to enable longer video synthesis, which mitigates the
potential quality degradation effectively. Experimental results show the
superiority of our method over existing baselines, particularly in terms of
temporal coherence and fidelity to users' guidance. In addition, our model
enables several intriguing applications that demonstrate potential for
practical usage.