WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent
  Video Diffusion Model
Authors: name: Zongjian Li; name: Bin Lin; name: Yang Ye; name: Liuhan Chen; name: Xinhua Cheng; name: Shenghai Yuan; name: Li Yuan
Published: 2024-11-26 14:23:53

Abstract: Video Variational Autoencoder (VAE) encodes videos into a low-dimensional
latent space, becoming a key component of most Latent Video Diffusion Models
(LVDMs) to reduce model training costs. However, as the resolution and duration
of generated videos increase, the encoding cost of Video VAEs becomes a
limiting bottleneck in training LVDMs. Moreover, the block-wise inference
method adopted by most LVDMs can lead to discontinuities of latent space when
processing long-duration videos. The key to addressing the computational
bottleneck lies in decomposing videos into distinct components and efficiently
encoding the critical information. Wavelet transform can decompose videos into
multiple frequency-domain components and improve the efficiency significantly,
we thus propose Wavelet Flow VAE (WF-VAE), an autoencoder that leverages
multi-level wavelet transform to facilitate low-frequency energy flow into
latent representation. Furthermore, we introduce a method called Causal Cache,
which maintains the integrity of latent space during block-wise inference.
Compared to state-of-the-art video VAEs, WF-VAE demonstrates superior
performance in both PSNR and LPIPS metrics, achieving 2x higher throughput and
4x lower memory consumption while maintaining competitive reconstruction
quality. Our code and models are available at
https://github.com/PKU-YuanGroup/WF-VAE.