VidTok: A Versatile and Open-Source Video Tokenizer
Authors: name: Anni Tang; name: Tianyu He; name: Junliang Guo; name: Xinle Cheng; name: Li Song; name: Jiang Bian
Published: 2024-12-17 16:27:11

Abstract: Encoding video content into compact latent tokens has become a fundamental
step in video generation and understanding, driven by the need to address the
inherent redundancy in pixel-level representations. Consequently, there is a
growing demand for high-performance, open-source video tokenizers as
video-centric research gains prominence. We introduce VidTok, a versatile video
tokenizer that delivers state-of-the-art performance in both continuous and
discrete tokenizations. VidTok incorporates several key advancements over
existing approaches: 1) model architecture such as convolutional layers and
up/downsampling modules; 2) to address the training instability and codebook
collapse commonly associated with conventional Vector Quantization (VQ), we
integrate Finite Scalar Quantization (FSQ) into discrete video tokenization; 3)
improved training strategies, including a two-stage training process and the
use of reduced frame rates. By integrating these advancements, VidTok achieves
substantial improvements over existing methods, demonstrating superior
performance across multiple metrics, including PSNR, SSIM, LPIPS, and FVD,
under standardized evaluation settings.