Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models
Authors: name: Songwei Ge; name: Seungjun Nah; name: Guilin Liu; name: Tyler Poon; name: Andrew Tao; name: Bryan Catanzaro; name: David Jacobs; name: Jia-Bin Huang; name: Ming-Yu Liu; name: Yogesh Balaji
Published: 2023-05-17 17:59:16

Abstract: Despite tremendous progress in generating high-quality images using diffusion
models, synthesizing a sequence of animated frames that are both photorealistic
and temporally coherent is still in its infancy. While off-the-shelf
billion-scale datasets for image generation are available, collecting similar
video data of the same scale is still challenging. Also, training a video
diffusion model is computationally much more expensive than its image
counterpart. In this work, we explore finetuning a pretrained image diffusion
model with video data as a practical solution for the video synthesis task. We
find that naively extending the image noise prior to video noise prior in video
diffusion leads to sub-optimal performance. Our carefully designed video noise
prior leads to substantially better performance. Extensive experimental
validation shows that our model, Preserve Your Own Correlation (PYoCo), attains
SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks. It
also achieves SOTA video generation quality on the small-scale UCF-101
benchmark with a 10times smaller model using significantly less computation
than the prior art.