xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed
  Representations
Authors: name: Can Qin; name: Congying Xia; name: Krithika Ramakrishnan; name: Michael Ryoo; name: Lifu Tu; name: Yihao Feng; name: Manli Shu; name: Honglu Zhou; name: Anas Awadalla; name: Jun Wang; name: Senthil Purushwalkam; name: Le Xue; name: Yingbo Zhou; name: Huan Wang; name: Silvio Savarese; name: Juan Carlos Niebles; name: Zeyuan Chen; name: Ran Xu; name: Caiming Xiong
Published: 2024-08-22 17:55:22

Abstract: We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of
producing realistic scenes from textual descriptions. Building on recent
advancements, such as OpenAI's Sora, we explore the latent diffusion model
(LDM) architecture and introduce a video variational autoencoder (VidVAE).
VidVAE compresses video data both spatially and temporally, significantly
reducing the length of visual tokens and the computational demands associated
with generating long-sequence videos. To further address the computational
costs, we propose a divide-and-merge strategy that maintains temporal
consistency across video segments. Our Diffusion Transformer (DiT) model
incorporates spatial and temporal self-attention layers, enabling robust
generalization across different timeframes and aspect ratios. We have devised a
data processing pipeline from the very beginning and collected over 13M
high-quality video-text pairs. The pipeline includes multiple steps such as
clipping, text detection, motion estimation, aesthetics scoring, and dense
captioning based on our in-house video-LLM model. Training the VidVAE and DiT
models required approximately 40 and 642 H100 days, respectively. Our model
supports over 14-second 720p video generation in an end-to-end way and
demonstrates competitive performance against state-of-the-art T2V models.