Progressive Autoregressive Video Diffusion Models
Authors: name: Desai Xie; name: Zhan Xu; name: Yicong Hong; name: Hao Tan; name: Difan Liu; name: Feng Liu; name: Arie Kaufman; name: Yang Zhou
Published: 2024-10-10 17:36:15

Abstract: Current frontier video diffusion models have demonstrated remarkable results
at generating high-quality videos. However, they can only generate short video
clips, normally around 10 seconds or 240 frames, due to computation limitations
during training. In this work, we show that existing models can be naturally
extended to autoregressive video diffusion models without changing the
architectures. Our key idea is to assign the latent frames with progressively
increasing noise levels rather than a single noise level, which allows for
fine-grained condition among the latents and large overlaps between the
attention windows. Such progressive video denoising allows our models to
autoregressively generate video frames without quality degradation or abrupt
scene changes. We present state-of-the-art results on long video generation at
1 minute (1440 frames at 24 FPS). Videos from this paper are available at
https://desaixie.github.io/pa-vdm/.