Scaling Image Tokenizers with Grouped Spherical Quantization
Authors: name: Jiangtao Wang; name: Zhen Qin; name: Yifan Zhang; name: Vincent Tao Hu; name: Björn Ommer; name: Rania Briq; name: Stefan Kesselheim
Published: 2024-12-03 18:01:45

Abstract: Vision tokenizers have gained a lot of attraction due to their scalability
and compactness; previous works depend on old-school GAN-based hyperparameters,
biased comparisons, and a lack of comprehensive analysis of the scaling
behaviours. To tackle those issues, we introduce Grouped Spherical Quantization
(GSQ), featuring spherical codebook initialization and lookup regularization to
constrain codebook latent to a spherical surface. Our empirical analysis of
image tokenizer training strategies demonstrates that GSQ-GAN achieves superior
reconstruction quality over state-of-the-art methods with fewer training
iterations, providing a solid foundation for scaling studies. Building on this,
we systematically examine the scaling behaviours of GSQ, specifically in latent
dimensionality, codebook size, and compression ratios, and their impact on
model performance. Our findings reveal distinct behaviours at high and low
spatial compression levels, underscoring challenges in representing
high-dimensional latent spaces. We show that GSQ can restructure
high-dimensional latent into compact, low-dimensional spaces, thus enabling
efficient scaling with improved quality. As a result, GSQ-GAN achieves a 16x
down-sampling with a reconstruction FID (rFID) of 0.50.