LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion
  Models
Authors: name: Yaohui Wang; name: Xinyuan Chen; name: Xin Ma; name: Shangchen Zhou; name: Ziqi Huang; name: Yi Wang; name: Ceyuan Yang; name: Yinan He; name: Jiashuo Yu; name: Peiqing Yang; name: Yuwei Guo; name: Tianxing Wu; name: Chenyang Si; name: Yuming Jiang; name: Cunjian Chen; name: Chen Change Loy; name: Bo Dai; name: Dahua Lin; name: Yu Qiao; name: Ziwei Liu
Published: 2023-09-26 17:52:03

Abstract: This work aims to learn a high-quality text-to-video (T2V) generative model
by leveraging a pre-trained text-to-image (T2I) model as a basis. It is a
highly desirable yet challenging task to simultaneously a) accomplish the
synthesis of visually realistic and temporally coherent videos while b)
preserving the strong creative generation nature of the pre-trained T2I model.
To this end, we propose LaVie, an integrated video generation framework that
operates on cascaded video latent diffusion models, comprising a base T2V
model, a temporal interpolation model, and a video super-resolution model. Our
key insights are two-fold: 1) We reveal that the incorporation of simple
temporal self-attentions, coupled with rotary positional encoding, adequately
captures the temporal correlations inherent in video data. 2) Additionally, we
validate that the process of joint image-video fine-tuning plays a pivotal role
in producing high-quality and creative outcomes. To enhance the performance of
LaVie, we contribute a comprehensive and diverse video dataset named Vimeo25M,
consisting of 25 million text-video pairs that prioritize quality, diversity,
and aesthetic appeal. Extensive experiments demonstrate that LaVie achieves
state-of-the-art performance both quantitatively and qualitatively.
Furthermore, we showcase the versatility of pre-trained LaVie models in various
long video generation and personalized video synthesis applications.