Open-MAGVIT2: An Open-Source Project Toward Democratizing
  Auto-regressive Visual Generation
Authors: name: Zhuoyan Luo; name: Fengyuan Shi; name: Yixiao Ge; name: Yujiu Yang; name: Limin Wang; name: Ying Shan
Published: 2024-09-06 17:14:53

Abstract: We present Open-MAGVIT2, a family of auto-regressive image generation models
ranging from 300M to 1.5B. The Open-MAGVIT2 project produces an open-source
replication of Google's MAGVIT-v2 tokenizer, a tokenizer with a super-large
codebook (i.e., 2^{18} codes), and achieves the state-of-the-art
reconstruction performance (1.17 rFID) on ImageNet 256 times 256.
Furthermore, we explore its application in plain auto-regressive models and
validate scalability properties. To assist auto-regressive models in predicting
with a super-large vocabulary, we factorize it into two sub-vocabulary of
different sizes by asymmetric token factorization, and further introduce "next
sub-token prediction" to enhance sub-token interaction for better generation
quality. We release all models and codes to foster innovation and creativity in
the field of auto-regressive visual generation.