Autoregressive Video Generation without Vector Quantization
Authors: name: Haoge Deng; name: Ting Pan; name: Haiwen Diao; name: Zhengxiong Luo; name: Yufeng Cui; name: Huchuan Lu; name: Shiguang Shan; name: Yonggang Qi; name: Xinlong Wang
Published: 2024-12-18 18:59:53

Abstract: This paper presents a novel approach that enables autoregressive video
generation with high efficiency. We propose to reformulate the video generation
problem as a non-quantized autoregressive modeling of temporal frame-by-frame
prediction and spatial set-by-set prediction. Unlike raster-scan prediction in
prior autoregressive models or joint distribution modeling of fixed-length
tokens in diffusion models, our approach maintains the causal property of
GPT-style models for flexible in-context capabilities, while leveraging
bidirectional modeling within individual frames for efficiency. With the
proposed approach, we train a novel video autoregressive model without vector
quantization, termed NOVA. Our results demonstrate that NOVA surpasses prior
autoregressive video models in data efficiency, inference speed, visual
fidelity, and video fluency, even with a much smaller model capacity, i.e.,
0.6B parameters. NOVA also outperforms state-of-the-art image diffusion models
in text-to-image generation tasks, with a significantly lower training cost.
Additionally, NOVA generalizes well across extended video durations and enables
diverse zero-shot applications in one unified model. Code and models are
publicly available at https://github.com/baaivision/NOVA.