CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
Authors: name: Zhuoyi Yang, name: Jiayan Teng, name: Wendi Zheng, name: Ming Ding, name: Shiyu Huang, name: Jiazheng Xu, name: Yuanming Yang, name: Wenyi Hong, name: Xiaohan Zhang, name: Guanyu Feng, name: Da Yin, name: Xiaotao Gu, name: Yuxuan Zhang, name: Weihan Wang, name: Yean Cheng, name: Ting Liu, name: Bin Xu, name: Yuxiao Dong, name: Jie Tang

We introduce CogVideoX, a large-scale diffusion transformer model designed
for generating videos based on text prompts. To efficently model video data, we
propose to levearge a 3D Variational Autoencoder (VAE) to compress videos along
both spatial and temporal dimensions. To improve the text-video alignment, we
propose an expert transformer with the expert adaptive LayerNorm to facilitate
the deep fusion between the two modalities. By employing a progressive training
technique, CogVideoX is adept at producing coherent, long-duration videos
characterized by significant motions. In addition, we develop an effective
text-video data processing pipeline that includes various data preprocessing
strategies and a video captioning method. It significantly helps enhance the
performance of CogVideoX, improving both generation quality and semantic
alignment. Results show that CogVideoX demonstrates state-of-the-art
performance across both multiple machine metrics and human evaluations. The
model weights of both the 3D Causal VAE and CogVideoX are publicly available at
https://github.com/THUDM/CogVideo.