FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Space
Authors: name: FSVideo Team, name: Qingyu Chen, name: Zhiyuan Fang, name: Haibin Huang, name: Xinwei Huang, name: Tong Jin, name: Minxuan Lin, name: Bo Liu, name: Celong Liu, name: Chongyang Ma, name: Xing Mei, name: Xiaohui Shen, name: Yaojie Shen, name: Fuwen Tan, name: Angtian Wang, name: Xiao Yang, name: Yiding Yang, name: Jiamin Yuan, name: Lingxi Zhang, name: Yuxin Zhang
Organization: ByteDance

We introduce FSVideo, a fast speed transformer-based image-to-video (I2V) diffusion framework. We build our framework on the following key components: 1.) a new video autoencoder with highly-compressed latent space (64times64times4 spatial-temporal downsampling ratio), achieving competitive reconstruction quality; 2.) a diffusion transformer (DIT) architecture with a new layer memory design to enhance inter-layer information flow and context reuse within DIT, and 3.) a multi-resolution generation strategy via a few-step DIT upsampler to increase video fidelity. Our final model, which contains a 14B DIT base model and a 14B DIT upsampler, achieves competitive performance against other popular open-source models, while being an order of magnitude faster. We discuss our model design as well as training strategies in this report.