xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed
  Representations
Authors: name: Can Qin, name: Congying Xia, name: Krithika Ramakrishnan, name: Michael Ryoo, name: Lifu Tu, name: Yihao Feng, name: Manli Shu, name: Honglu Zhou, name: Anas Awadalla, name: Jun Wang, name: Senthil Purushwalkam, name: Le Xue, name: Yingbo Zhou, name: Huan Wang, name: Silvio Savarese, name: Juan Carlos Niebles, name: Zeyuan Chen, name: Ran Xu, name: Caiming Xiong

We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of
producing realistic scenes from textual descriptions. Building on recent
advancements, such as OpenAI's Sora, we explore the latent diffusion model
(LDM) architecture and introduce a video variational autoencoder (VidVAE).
VidVAE compresses video data both spatially and temporally, significantly
reducing the length of visual tokens and the computational demands associated
with generating long-sequence videos. To further address the computational
costs, we propose a divide-and-merge strategy that maintains temporal
consistency across video segments. Our Diffusion Transformer (DiT) model
incorporates spatial and temporal self-attention layers, enabling robust
generalization across different timeframes and aspect ratios. We have devised a
data processing pipeline from the very beginning and collected over 13M
high-quality video-text pairs. The pipeline includes multiple steps such as
clipping, text detection, motion estimation, aesthetics scoring, and dense
captioning based on our in-house video-LLM model. Training the VidVAE and DiT
models required approximately 40 and 642 H100 days, respectively. Our model
supports over 14-second 720p video generation in an end-to-end way and
demonstrates competitive performance against state-of-the-art T2V models.