DreamVideo: Composing Your Dream Videos with Customized Subject and
  Motion
Authors: name: Yujie Wei; name: Shiwei Zhang; name: Zhiwu Qing; name: Hangjie Yuan; name: Zhiheng Liu; name: Yu Liu; name: Yingya Zhang; name: Jingren Zhou; name: Hongming Shan
Published: 2023-12-07 16:57:26

Abstract: Customized generation using diffusion models has made impressive progress in
image generation, but remains unsatisfactory in the challenging video
generation task, as it requires the controllability of both subjects and
motions. To that end, we present DreamVideo, a novel approach to generating
personalized videos from a few static images of the desired subject and a few
videos of target motion. DreamVideo decouples this task into two stages,
subject learning and motion learning, by leveraging a pre-trained video
diffusion model. The subject learning aims to accurately capture the fine
appearance of the subject from provided images, which is achieved by combining
textual inversion and fine-tuning of our carefully designed identity adapter.
In motion learning, we architect a motion adapter and fine-tune it on the given
videos to effectively model the target motion pattern. Combining these two
lightweight and efficient adapters allows for flexible customization of any
subject with any motion. Extensive experimental results demonstrate the
superior performance of our DreamVideo over the state-of-the-art methods for
customized video generation. Our project page is at
https://dreamvideo-t2v.github.io.