Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation
Authors: name: Yuying Ge; name: Yizhuo Li; name: Yixiao Ge; name: Ying Shan
Published: 2024-12-05 18:53:04

Abstract: In recent years, there has been a significant surge of interest in unifying
image comprehension and generation within Large Language Models (LLMs). This
growing interest has prompted us to explore extending this unification to
videos. The core challenge lies in developing a versatile video tokenizer that
captures both the spatial characteristics and temporal dynamics of videos to
obtain representations for LLMs, and the representations can be further decoded
into realistic video clips to enable video generation. In this work, we
introduce Divot, a Diffusion-Powered Video Tokenizer, which leverages the
diffusion process for self-supervised video representation learning. We posit
that if a video diffusion model can effectively de-noise video clips by taking
the features of a video tokenizer as the condition, then the tokenizer has
successfully captured robust spatial and temporal information. Additionally,
the video diffusion model inherently functions as a de-tokenizer, decoding
videos from their representations. Building upon the Divot tokenizer, we
present Divot-Vicuna through video-to-text autoregression and text-to-video
generation by modeling the distributions of continuous-valued Divot features
with a Gaussian Mixture Model. Experimental results demonstrate that our
diffusion-based video tokenizer, when integrated with a pre-trained LLM,
achieves competitive performance across various video comprehension and
generation benchmarks. The instruction tuned Divot-Vicuna also excels in video
storytelling, generating interleaved narratives and corresponding videos.