CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
Authors: name: Zhuoyi Yang; name: Jiayan Teng; name: Wendi Zheng; name: Ming Ding; name: Shiyu Huang; name: Jiazheng Xu; name: Yuanming Yang; name: Wenyi Hong; name: Xiaohan Zhang; name: Guanyu Feng; name: Da Yin; name: Xiaotao Gu; name: Yuxuan Zhang; name: Weihan Wang; name: Yean Cheng; name: Ting Liu; name: Bin Xu; name: Yuxiao Dong; name: Jie Tang
Published: 2024-08-12 11:47:11

Abstract: We introduce CogVideoX, a large-scale diffusion transformer model designed
for generating videos based on text prompts. To efficently model video data, we
propose to levearge a 3D Variational Autoencoder (VAE) to compress videos along
both spatial and temporal dimensions. To improve the text-video alignment, we
propose an expert transformer with the expert adaptive LayerNorm to facilitate
the deep fusion between the two modalities. By employing a progressive training
technique, CogVideoX is adept at producing coherent, long-duration videos
characterized by significant motions. In addition, we develop an effective
text-video data processing pipeline that includes various data preprocessing
strategies and a video captioning method. It significantly helps enhance the
performance of CogVideoX, improving both generation quality and semantic
alignment. Results show that CogVideoX demonstrates state-of-the-art
performance across both multiple machine metrics and human evaluations. The
model weights of both the 3D Causal VAE and CogVideoX are publicly available at
https://github.com/THUDM/CogVideo.