Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation
Authors: name: Jiaxi Gu; name: Shicong Wang; name: Haoyu Zhao; name: Tianyi Lu; name: Xing Zhang; name: Zuxuan Wu; name: Songcen Xu; name: Wei Zhang; name: Yu-Gang Jiang; name: Hang Xu
Published: 2023-09-07 08:12:58

Abstract: Inspired by the remarkable success of Latent Diffusion Models (LDMs) for
image synthesis, we study LDM for text-to-video generation, which is a
formidable challenge due to the computational and memory constraints during
both model training and inference. A single LDM is usually only capable of
generating a very limited number of video frames. Some existing works focus on
separate prediction models for generating more video frames, which suffer from
additional training cost and frame-level jittering, however. In this paper, we
propose a framework called "Reuse and Diffuse" dubbed VidRD to
produce more frames following the frames already generated by an LDM.
Conditioned on an initial video clip with a small number of frames, additional
frames are iteratively generated by reusing the original latent features and
following the previous diffusion process. Besides, for the autoencoder used for
translation between pixel space and latent space, we inject temporal layers
into its decoder and fine-tune these layers for higher temporal consistency. We
also propose a set of strategies for composing video-text data that involve
diverse content from multiple existing datasets including video datasets for
action recognition and image-text datasets. Extensive experiments show that our
method achieves good results in both quantitative and qualitative evaluations.
Our project page is available
https://anonymous0x233.github.io/ReuseAndDiffuse/{here}.