Fine-gained Zero-shot Video Sampling
Authors: name: Dengsheng Chen; name: Jie Hu; name: Xiaoming Wei; name: Enhua Wu
Published: 2024-07-31 09:36:58

Abstract: Incorporating a temporal dimension into pretrained image diffusion models for
video generation is a prevalent approach. However, this method is
computationally demanding and necessitates large-scale video datasets. More
critically, the heterogeneity between image and video datasets often results in
catastrophic forgetting of the image expertise. Recent attempts to directly
extract video snippets from image diffusion models have somewhat mitigated
these problems. Nevertheless, these methods can only generate brief video clips
with simple movements and fail to capture fine-grained motion or non-grid
deformation. In this paper, we propose a novel Zero-Shot video Sampling
algorithm, denoted as ZS^2, capable of directly sampling
high-quality video clips from existing image synthesis methods, such as Stable
Diffusion, without any training or optimization. Specifically, ZS^2
utilizes the dependency noise model and temporal momentum attention to ensure
content consistency and animation coherence, respectively. This ability enables
it to excel in related tasks, such as conditional and context-specialized video
generation and instruction-guided video editing. Experimental results
demonstrate that ZS^2 achieves state-of-the-art performance in
zero-shot video generation, occasionally outperforming recent supervised
methods.
  Homepage: https://densechen.github.io/zss/.