Denoising Reuse: Exploiting Inter-frame Motion Consistency for Efficient
  Video Latent Generation
Authors: name: Chenyu Wang; name: Shuo Yan; name: Yixuan Chen; name: Yujiang Wang; name: Mingzhi Dong; name: Xiaochen Yang; name: Dongsheng Li; name: Robert P. Dick; name: Qin Lv; name: Fan Yang; name: Tun Lu; name: Ning Gu; name: Li Shang
Published: 2024-09-19 07:50:34

Abstract: Video generation using diffusion-based models is constrained by high
computational costs due to the frame-wise iterative diffusion process. This
work presents a Diffusion Reuse MOtion (Dr. Mo) network to accelerate latent
video generation. Our key discovery is that coarse-grained noises in earlier
denoising steps have demonstrated high motion consistency across consecutive
video frames. Following this observation, Dr. Mo propagates those
coarse-grained noises onto the next frame by incorporating carefully designed,
lightweight inter-frame motions, eliminating massive computational redundancy
in frame-wise diffusion models. The more sensitive and fine-grained noises are
still acquired via later denoising steps, which can be essential to retain
visual qualities. As such, deciding which intermediate steps should switch from
motion-based propagations to denoising can be a crucial problem and a key
tradeoff between efficiency and quality. Dr. Mo employs a meta-network named
Denoising Step Selector (DSS) to dynamically determine desirable intermediate
steps across video frames. Extensive evaluations on video generation and
editing tasks have shown that Dr. Mo can substantially accelerate diffusion
models in video tasks with improved visual qualities.