OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video
  Diffusion Model
Authors: name: Liuhan Chen, name: Zongjian Li, name: Bin Lin, name: Bin Zhu, name: Qian Wang, name: Shenghai Yuan, name: Xing Zhou, name: Xinghua Cheng, name: Li Yuan

Variational Autoencoder (VAE), compressing videos into latent
representations, is a crucial preceding component of Latent Video Diffusion
Models (LVDMs). With the same reconstruction quality, the more sufficient the
VAE's compression for videos is, the more efficient the LVDMs are. However,
most LVDMs utilize 2D image VAE, whose compression for videos is only in the
spatial dimension and often ignored in the temporal dimension. How to conduct
temporal compression for videos in a VAE to obtain more concise latent
representations while promising accurate reconstruction is seldom explored. To
fill this gap, we propose an omni-dimension compression VAE, named OD-VAE,
which can temporally and spatially compress videos. Although OD-VAE's more
sufficient compression brings a great challenge to video reconstruction, it can
still achieve high reconstructed accuracy by our fine design. To obtain a
better trade-off between video reconstruction quality and compression speed,
four variants of OD-VAE are introduced and analyzed. In addition, a novel tail
initialization is designed to train OD-VAE more efficiently, and a novel
inference strategy is proposed to enable OD-VAE to handle videos of arbitrary
length with limited GPU memory. Comprehensive experiments on video
reconstruction and LVDM-based video generation demonstrate the effectiveness
and efficiency of our proposed methods.