Title: An Image is Worth 32 Tokens for Reconstruction and Generation
Venue: NeurIPS 2024
Accepted: yes
Keywords: image tokenization

image generation
Abstract: Recent advancements in generative models have highlighted the crucial role of image tokenization in the efficient synthesis of high-resolution images. Tokenization, which transforms images into latent representations, reduces computational demands compared to directly processing pixels and enhances the effectiveness and efficiency of the generation process. Prior methods, such as VQGAN, typically utilize 2D latent grids with fixed downsampling factors. However, these 2D tokenizations face challenges in managing the inherent redundancies present in images, where adjacent regions frequently display similarities. To overcome this issue, we introduce **T**ransformer-based 1-D**i**mensional **Tok**enizer (TiTok), an innovative approach that tokenizes images into 1D latent sequences. TiTok provides a more compact latent representation, yielding substantially more efficient and effective representations than conventional techniques. For example, a 256 × 256 × 3 image can be reduced to just **32** discrete tokens, a significant reduction from the 256 or 1024 tokens obtained by prior methods. Despite its compact nature, TiTok achieves competitive performance to state-of-the-art approaches. Specifically, using the same generator framework, TiTok attains **1.97** gFID, outperforming MaskGIT baseline significantly by 4.21 at ImageNet 256 × 256 benchmark. The advantages of TiTok become even more significant when it comes to higher resolution. At ImageNet 512 × 512 benchmark, TiTok not only outperforms state-of-the-art diffusion model DiT-XL/2 (gFID 2.74 vs. 3.04), but also reduces the image tokens by 64×, leading to **410× faster** generation process. Our best-performing variant can significantly surpasses DiT-XL/2 (gFID **2.13** vs. 3.04) while still generating high-quality samples **74× faster**. Codes and models are available at https://github.com/bytedance/1d-tokenizer


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel way to tokenize images to benefit image reconstruction and generation. This paper argues that the convention of compressing images into 2D latent spaces in the VQVAE/VQGAN setting limits the VQ model’s ability to fully exploit the redundancies present in images.\n\nDuring encoding, a fixed number of latent tokens are concatenated with the image patches. A vision transformer (ViT) is used to extract the representation from image patches to the latent representation. During decoding, the latent tokens are concatenated with a set of masked tokens. Another ViT is used to reconstruct image patches from the latent representation. However, instead of training encoder and decoder end2end, this paper first trains the model with the discrete codes generated by an off-the-shelf MaskGIT-VQGAN model, and then only fine-tunes the decoder with RGB pixels. During image generation, MaskGIT is used to generate the latent tokens.\n\nThe model is tested on 256x256 ImageNet for image reconstruction, generation and classification.\n\nSTRENGTHS:\n1. The proposed method is simple yet effective.\n2. Scaling up the tokenizer seems able to achieve even more compact image representations.\n3. The experiments on image classification is very interesting. As the size of the latent representation decreases, the tokenizer increasingly learns semantically rich representations.\n\nWEAKNESSES:\n1. More efforts should be put in writing Two-Stage Training. It is not clear how exactly the \"warmup\" works. Does it train the whole encoder-decoder but with frozen codebook used in VQGAN? Or it means the model is trained by the quantized value of VQGAN instead of by the RGB pixels, i.e., the input and output of the model are the quantized value of VQGAN?\n2. In the ablation test of 2D variant of TiTok-B64, how many tokens are used in this 2D representation? What if we use the same number of tokens for 1D and 2D variants? Will the performance be similar?\n\nQUESTIONS:\nHow important is the off-the-shelf codebook? If the MaskGIT-VQGAN codebook is replaced by some other codebook, let's say FSQ, how would the performance look like? Can we use a continuous autoencoder instead of VQGAN as the outside compressor?\n\nLIMITATIONS:\nThe limitation and potential negative societal impact have been discussed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TiTok (Transformer-based 1-Dimensional Tokenizer) that can tokenize images as compact 1D sequences instead of 2D latent grids. Accompanied with bidirectional non-autoregressive image generator, TiTok achieves SOTA performance on imagenet 256x256 benchmark with much faster generation process. The key contributions are:\n1) A 1D tokenization method that can represent images using significantly fewer tokens compared with traditional 2D approaches without comprising performance (as few as 32 tokens);\n2) Scaling up the size of tokenizer model helps to learn more compact and semantic-rich latent representations;\n3) Significant speed-up for both training and inference with competitive performance compared with 2D tokenizer.\n\nSTRENGTHS:\n1) The idea of using 1D representation instead of the conventional 2D is novel. It shows as few as 32 tokens are able to reconstruct an image well. The significant reduction of token numbers makes the training and inference much more efficient. Furthermore, scaling model sizes enables more compact representation, which provides insights into scaling laws for image tokenization. \n2) The experiments are comprehensive, including different model sizes and token numbers. The ablation of different design choices are well designed. The evaluations on both image reconstruction and generation show superior performance compared with MaskGIT and VQGAN. \n3) The paper is well rewritten, and easy to follow. The codes and models are publicly available.\n\nWEAKNESSES:\nA major concern is the two-stage training method. From Table 3 (c), the gap with and wo proxy codes is big, 1.7 vs 5.1 for rFID, and 195 vs 120 for IS. However, the proxy codes come from MaskGIT. The authors emphasized that due to the missing of a strong training recipe, the sing-stage method is lagging behind. Thus, it seems the training method is the most important one compared with architecture designs. \n\nBTW, recently there is an open-source implementation for MAGVIT2 including training codes: https://github.com/TencentARC/Open-MAGVIT2\nIt will be great if the authors can adopt their training method and make the single-stage training work.\n\nQUESTIONS:\nAnother related work is Seed, which also proposes a 1D tokenizer with semantic meaningful tokens (32 tokens). It's better to cite it:\nGe, Yuying, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. \"Planting a seed of vision in large language model.\" arXiv preprint arXiv:2307.08041 (2023).\n\nLIMITATIONS:\nNo negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a transformer-based image tokenizer (ViT) designed to convert 2D images into a 1D discrete sequence, named TiTok. The authors demonstrate that an image of size 256x256 can be discretized into a compact space of only 32 discrete tokens. This new tokenizer encodes semantic information, in contrast to other local 2D tokenizers. TiTok is trained using a two-stage process: initially, proxy codes are used to avoid the complexity of VQGAN loss, followed by a fine-tuning stage. Finally, they train a MaskGIT model to generate the discrete tokens, achieving an FID score of less than 2.\n\nSTRENGTHS:\n- The proposed method not only demonstrates increased speed compared to previous approaches but also showcases improved quality, as indicated by the FID score and the visualization.\n- Thanks to the compact space, the tokenizer learns semantic features rather than local features, which suggests potential for interesting interpolation and image manipulation applications in the future.\n- The method is original and presents a novel approach to addressing the generative task using discrete tokens in a 1D sequence.\n\nWEAKNESSES:\n- The method relies on proxy training for the tokenizer, but the motivation for this choice is unclear. The VQGAN loss (Reco, LPIPS, and GAN) is not overly complex to train, as evidenced by the availability of many open-source implementations. Utilizing proxy tokens is cumbersome since it requires a pre-trained network with the VQGAN loss.\n\n- The paper may lack sufficient ablation studies on hyperparameters for sampling, such as the number of steps, the role of temperature, or the CFG.\n\n- Providing uncurated visualizations of few samples from one or multiple classes could have helped readers better assess the diversity generated by the model.\n\nQUESTIONS:\n- Figure 4 shows better gFID compared to rFID when using only 32 tokens. How is this possible? How can you achieve a better FID than the actual reconstruction?\n\n- Given that the number of tokens to predict is low (<128) and the structure is 1D, why not use a decoder-only transformer with next-token prediction? This approach would eliminate the burden of the sampling strategy and maintain a manageable number of forward passes thanks to the tokenizers\n\n- I might have missed something, but how do you train TiTok with the proxy codes when the number of tokens output by the VQGAN are more than 32 and when the number of codes in the codebook are not the same?\n\nLIMITATIONS:\n/", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBackground: VQ-GAN with 2D grid of latent tokens and fixed downsampling factors.\nThis paper proposes to use 1D tokens instead of 2D tokens.\n\nKey ideas:\n* Redundancies: adjacent regions are similar. 2D grid explicitly couples the latents and the pixels in the same relative coordinates.\n* 1D tokens are enough for discriminative tasks (classification, etc.). Maybe they are enough for generative tasks too.\n\nArchitecture:\n* Transformer-based encoder (tokenizer) and decoder (de-tokenizer)\n\nTraining:\n* Warm-up stage: train the 1D VQ model with the proxy codes (= discrete codes generated by an off-the-shelf MaskGIT-VQGAN model)\n* Decoder fine-tuning stage: finetune the decoder with frozen encoder and quantizer\n\nSTRENGTHS:\nTechnical advantages:\n* More freedom in designing the architecture of 1D tokenizer\n* More semantic-rich image embedding // -> definition of semantic-rich? grounding experiment?\n* Improvement in FID\n* Faster (70x ~ 410x) generation\n\nThorough analyses:\n* Scaling experiment -> 32 tokens are sufficient for reconstruction.\n* Larger tokenizer enables smaller number of latent tokens.\n* Smaller number of latent tokens have clearer semantics than larger one, regarding linear probing.\n\nWEAKNESSES:\nW1. There is no analysis of 1D tokens. What are the advantages of 1D tokens that match the drawbacks of 2D tokens mentioned in the introduction section? What are the effects of masking a subset of 1D tokens compared to 2D tokens? The paper emphasizes the 1D latent sequence but does not analyze whether the observed characteristics (scaling, compact latent, etc.) are due to this 1D nature. A comparison with methods that further reduce the size of 2D tokens is needed.\n\nW2. For reconstruction results, it is necessary to compare using MSE, PSNR, and SSIM. FID is not sufficient to evaluate “reconstruction.” Why do we need rFID instead of typical reconstruction metrics such as PSNR, SSIM, and LPIPS?\n\nW3. The explanation of decoder fine-tuning is too sparse. Although not a novel contribution of this paper, it plays a significant role in performance improvement as shown in the ablation study, requiring more detailed explanation beyond just the “VQGAN training recipe.”\n\n\nMisc.\n\nSentences could be easier to read.\n- e.g., L60 ... the ViT decoder (is utilized to reconstruct -> reconstructs) the input images ...\n- I recommend reading \"Style, the basics of clarity and grace\".\n\nQUESTIONS:\nQ1. What are the advantages of 1D compared to 2D, and what are the potential disadvantages? Please provide a brief explanation.\n\nQ2. Is TiTok compatible with diffusion models? What is the reason for choosing MaskGIT over diffusion models or VAR (Scalable Image Generation via Next-Scale Prediction)? Appendix mentions computational burden. Is MaskGIT cheaper than the others?\n\nQ3. Could you discuss the relationship between TiTok and \"Vision transformers need registers\"?\nhttps://arxiv.org/abs/2309.16588\n\nLIMITATIONS:\nYes in the appendix sections.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TiTok, a 1D image tokenization method that can represent images using significantly fewer tokens compared to existing 2D approaches. The key contributions are:\n1. A 1D tokenization scheme that breaks the fixed grid constraints of 2D tokenizers, allowing more flexible and compact image representations.\n2. A dual-stage training strategy using proxy codes to improve tokenizer performance.\n3. Extensive experiments demonstrating state-of-the-art performance on ImageNet generation benchmarks while using 8-64x fewer tokens and achieving 74-410x faster inference compared to leading diffusion models.\n\nSTRENGTHS:\n1. The conclusion is shocking. 32 tokens are enough to construct a picture. Although I know that the information in the picture is very redundant, this conclusion still shocks me.\n2. The effect is still good.\n\nWEAKNESSES:\n1. Compared with 2D token, the main advantage of 1D image token (in my opinion) is not the efficiency and training data. Although 2D token is cumbersome, it supports the expansion of 2D dimension and has many other advantages. In applications, for higher resolution images and better effects, 2D token is still the mainstream and is difficult to be shaken by 1D token. The impact of 1D token on image generation may not be its main application. The most important value of 1D image token for me lies in its possibility of combining with multimodal technology. Due to its information compression ability and two-dimensional encoding method, 2D token is difficult to combine with language model to form a multimodal language model. And the encoder of the current multimodal language model has no decoder available. 1D token may be able to support multimodal language model and allow language model to directly output images. This is very potential. But this article does not have corresponding thinking, which is a pity.\n2. Although the image formed by 32 tokens is shocking, there are still many questions to be answered. Compared with the case of more tokens, can the information encoded by each token be estimated? Can we know what specific information is forgotten in the process of token compression? Can 1D tokens encode 2D relationships? How do 1D tokens understand translation and rotation? Is 1D token similar to the concept of high-level dictionary? These important questions are not studied in depth in this article.\n3. There are too few pictures shown in this article to make further judgments.\n\nQUESTIONS:\nI have raised many question in weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-21"}