Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis
Authors: name: Bichen Wu; name: Ching-Yao Chuang; name: Xiaoyan Wang; name: Yichen Jia; name: Kapil Krishnakumar; name: Tong Xiao; name: Feng Liang; name: Licheng Yu; name: Peter Vajda
Published: 2023-12-20 01:49:47

Abstract: In this paper, we introduce Fairy, a minimalist yet robust adaptation of
image-editing diffusion models, enhancing them for video editing applications.
Our approach centers on the concept of anchor-based cross-frame attention, a
mechanism that implicitly propagates diffusion features across frames, ensuring
superior temporal coherence and high-fidelity synthesis. Fairy not only
addresses limitations of previous models, including memory and processing
speed. It also improves temporal consistency through a unique data augmentation
strategy. This strategy renders the model equivariant to affine transformations
in both source and target images. Remarkably efficient, Fairy generates
120-frame 512x384 videos (4-second duration at 30 FPS) in just 14 seconds,
outpacing prior works by at least 44x. A comprehensive user study, involving
1000 generated samples, confirms that our approach delivers superior quality,
decisively outperforming established methods.