An Image is Worth 32 Tokens for Reconstruction and Generation
Authors: Qihang Yu
Mark Weber
Xueqing Deng
Xiaohui Shen
Daniel Cremers
Liang-Chieh Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: image tokenization
image generation
Primary area: generative_models

Abstract: Recent advancements in generative models have highlighted the crucial role of image tokenization in the efficient synthesis of high-resolution images. Tokenization, which transforms images into latent representations, reduces computational demands compared to directly processing pixels and enhances the effectiveness and efficiency of the generation process. Prior methods, such as VQGAN, typically utilize 2D latent grids with fixed downsampling factors. However, these 2D tokenizations face challenges in managing the inherent redundancies present in images, where adjacent regions frequently display similarities. To overcome this issue, we introduce **T**ransformer-based 1-D**i**mensional **Tok**enizer (TiTok), an innovative approach that tokenizes images into 1D latent sequences. TiTok provides a more compact latent representation, yielding substantially more efficient and effective representations than conventional techniques. For example, a 256 × 256 × 3 image can be reduced to just **32** discrete tokens, a significant reduction from the 256 or 1024 tokens obtained by prior methods. Despite its compact nature, TiTok achieves competitive performance to state-of-the-art approaches. Specifically, using the same generator framework, TiTok attains **1.97** gFID, outperforming MaskGIT baseline significantly by 4.21 at ImageNet 256 × 256 benchmark. The advantages of TiTok become even more significant when it comes to higher resolution. At ImageNet 512 × 512 benchmark, TiTok not only outperforms state-of-the-art diffusion model DiT-XL/2 (gFID 2.74 vs. 3.04), but also reduces the image tokens by 64×, leading to **410× faster** generation process. Our best-performing variant can significantly surpasses DiT-XL/2 (gFID **2.13** vs. 3.04) while still generating high-quality samples **74× faster**. Codes and models are available at https://github.com/bytedance/1d-tokenizer

An Image is Worth 32 Tokens for Reconstruction and Generation
Qihang Yu' Mark Weber! Xueqing Deng' Xiaohui Shen', Daniel Cremers?, Liang-Chie Chieh Chen' ByteDance
Technical University Munich
"equal contribution
https:/ haap:://yycooeettto github. htt.://o//oettttttttttotto
32 tokens can work well for...
latent size and costs
Image Reconstruction
Image Generation (TiTok 32 tokens)
32 tokens
256 tokens
VOCAN
65536 pixels image
We propose TiTok, compact 1D tokenizer leveraging region redundancy to represent an image with only 32 tokens for image reconstruction and generation.
Figure I:
Abstract
Recent advancements in generative models have highlighted the crucial role of image tokenization in the efficient synthesis of high- images. Tokeniza- tion, which transforms images into latent representations, reduces computational demands compared to directly processing pixels and enhances the effectiveness and efficiency of the generation process. Prior methods, such as VQGAN, typically utilize 2D latent grids with fixed downsampling factors. However, these 2D tok- enizations face challenges in managing the inherent redundancies present images, where adjacent regions frequently display similarities. To overcome this issue, we introduce Transformer-based i-Dimensional Tokenizer (TiTok), an innovative approach that tokenizes images into 1D latent sequences. TiTok provides compact latent representation, yielding substantially more efficient and effective representations than conventional techniques. For example, 256 256 image can be reduced to just 32 discrete tokens, significant reduction from the 256 or 1024 tokens obtained by prior methods. Despite its compact nature, TYTok achieves competitive performance to oate--of-the-ar approaches. Specifically, using the same generator framework, TYTok attains 1.97 gFID, outperforming MaskGIT baseline significantly by 4.21 at ImageNet 256 256 benchmark. The advantages TiTok become even more significant when it comes higher resolution. At ImageNet 512 512 benchmark, TTTok not only outperforms state-of-the-ard dif- fusion model DiT-XI./2 (gFID 2.74 vs. 3.04), but also reduces the image tokens by 64x, leading to 410x faster generation process. Our best-performin variant can significantly surpass DiT-XL/2 (gFID 2.13 vs. 3.04) while still generating high-quality samples 74x faster.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
In recent years, image generation has experienced remarkable progress, driven by the significant advancements in both transformers [19, 65, 70, 10, 71, 72] and diffusion models [16, 58, 29, 52, 21]. Mirroring the trends in generative language models [51, 62], the architecture of many contemporary image generation models incorporate standard image tokenizer and te-tooknizer. This array of models utilizes tokenized image eppresntaaioons-raneee
from continuous [35] to discrete vec- tors [57, 64, 19]-10 perform critical function: translating raw pixels into latent space. The latent space (e.g. 32 x 32) is significantly more compact than the original image space (256 x 256 x 3). It offers compressed yet expressive representation, and thus not only facilitates efficient training and inference of generative models but also paves the way to scale up the model size.
Although image tokenizers achieve great success in image generation workflows, they encounter fundamental limitation tied to their intrinsic design. These tokenizers are based on an assumption that the latent space should retain 2D structure, to maintain direct mapping for locations between the latent tokens and image patches. For example, the top-left latent token directly corresponds to the top-left image patch. This restricts the tokenizer's 's ability to effectively leverage the redundancy inherent in images to cultivate a more compressed latent space.
Taking one step back, we raise the question "is 21) structure necessary for image tokenization?" To answer the question, we draw inspiration from several image understanding tasks where model predictions are based solely on high-level information extracted from input images -such in image classification [17], object detection [8, 81J, segmentation [67, 34, 74, 75], and multi-modal large language models [1, 41, 11]. These tasks do not need de- de-tokenizers, since the outputs typically manifest in specific structures other than images. In other words, they often format a higher-level ID sequence as output that can still capture all task-relevant information. Prior arts, such as object queries [8, 67] or the perceiver resampler [1], encode images into a ID sequence of a predetermined number of tokens (e.g., 64). These tokens facilitate the generation of outputs like bounding boxes or captions. The success of these methods motivates us to investigate a more compact 1D sequence as image latent representation in the context of image reconstruction and generation. is noteworthy that the synthesis of both high-level and low-level information is crucial for the generation of high-quality images, providing a challenge for extremely compact latent representations.
In this work, we introduce transformer-based framework [65, 17] designed to tokenize an image to ID discrete sequence, which can later be decoded back to the image space via de-tokenizer. Specifically, we present Transformer- Transformer (ViT) encoder, a ViT decoder, and a vector quantizer following the typical Vector- Quantized (VQ) model designs [19]. In the tokenization phase, the image is split and flattened into a series of patches, followed by concatenation with a 1D sequence of latent tokens. After the feature encoding process of ViT encoder, these latent tokens build the latent representation of the image. Subsequent the vector quantization step [64, 19], the ViT decoder reconstructs the input images from the masked token sequence [15, 24].
I-Dimensional Tokenizer (TiTok), consisting of a Vision
Building upon TiTok, we conduct extensive experiments to probe the dynamics of ID image tokeniza- tion. Our investigation studies the interplay between latent space size, model size, reconstruction fidelity, and generative quality. From this exploration, several compelling insights emerge:
1. Increasing the number of latent tokens representing an image consistently improves the reconstruction performance, yet the benefit becomes marginal after 128 tokens. Intriguingly. 32 tokens are sufficient for a reasonable image reconstruction.
2. Scaling up the tokenizer model size significantly improves performance of both reconstruc- tion and generation, especially when number of tokens is limited (e.g., 32 or 64), showcasing promising pathway towards compact image representation latent space.
3. ID tokenization breaks the grid constraints in prior 2D image tokenizers, which not only enables each latent token to reconstruct regions beyond a fixed image grid and leads to a more flexible tokenizer design, but also learns more high-level and semantic-rich image information, especially at compact latent space.
4. 1D tokenization exhibits superior performance in generative training, with not only signifi- cant speed-up for both training and inference but also a competitive FID score compared to a typical 2D tokenizer, while using much fewer tokens.
2

---

103
TITok-L-32
TIToK-B-64
TiTok--644
38
150x fnster
416x fanter
TTToK-B-128
23
faster
TiTok-S-120
samplesibes
semplesiios
tattr
1
U-VIT-U/2
DIT-XL/2
F1 Lone
0.5
VIT-VGGAN
LDM-4
U-VIT-H/2
6.7
ADM
MaskGTT
U-VIT-L/4
U-VIT-HI4
DIT-XL/2
6.3
qualtty
1.00
gFID
gRlo
Model comparison on resolution 256
(b) Model comparison on resolution 512
Figure 2: A speed and quality comparison of TiTok and prior arts on ImageNet 256 256 and 512 x 512 generation benchmarks. Speed-up is compared against D/T-XL/2 [52]. The sampling speed (de-tokenization included) measured with an A100 GPU.
In light of these findings, introduce the TiTok family, encompassing models of varying model sizes and latent sizes, capable of achieving highly compact tokenization with as few as 32 tokens. We further confirm the model's efficacy in image generation through the MaskGIT [9] framework. TiTok demonstrated to facilitate state-of-the-ar performance in image generation, while requiring latent spaces that are 8x to 64x smaller, resulting in significant accelerations during both the training and inference phases. It also generates images with similar or higher quality but up to 410x faster than state-of-the-ar diffusion models such as DiT [18] (Fig. 2).
2
Related Work
Image Tokenization. Images have been compressed since the early days of deep learning with autoencoders [27, 66]. The general design of using an encoder that compresses high-dimensional images into a low-dimensional latent representation and then using a decoder reverse the process, has proven to be successful over the years. Variational Autoencoders (VAEs) [35] extend the paradigm by learning map the input to a distribution. Instead of modeling continuous distribution, VQ- VAEs [50, 56] learn a discrete representation forming categorical distribution. VQGAN [19] further improves the training process by using adversarial training [23]. The transformer design of the autoencoder is further explored in ViT-VQGAN [69] and EfficiettVVGGNN [7]. Orthogonal to this,
RQ-VAE [37] and MoVQ [80] study the effect using multiple vector quantization steps per latent embedding, while MAGVIT-v2 [72] and FSQ [48] propose a lookup-free quantization. However, all aforementioned works share the same workflow of an image always being patchwise encoded into 2D grid latent representation. In this work, explore innovative ID sequence latent representation for image reconstruction and generation.
Tokenization for Image Understanding, For image understanding tasks (e.g., image classifica- tion [17], object detection [8. 81, 78], segmentation [67, 74, 76], and Multi-modal Large Language Models (MLLMs) [1, 41, 77D.. it is common to use general feature encoder instead of an autoen- coder to tokenize the image. Specifically, many MLLMS [41, 43, 61, 32, 22, 11] uses CLIP [54] encoder to tokenize the image into highly semantic tokens, which proves effective for image cap- tioning [13] and VQA [2]. Some MILMs also explore discrete tokens [32, 22] or "de-tokenize" the CLIP embeddings back to images through diffusion models [61, 32, 22]. However, due to the nature of CLIP models that focus on high-level information, these methods can only reconstruct an image with high-level semantic similarities (i.e., the layouts and details are not well-reconstruutee due to CLIP features). Therefore, our method is significantly different from theirs, since the proposed TiTok aims to reconstruct both the high-level and low-level details of an image, same typical VQ-VAE tokenizers [35, 57, 19].
Image Generation. Image generation methods range from sampling the VAE [35], using GANs [23] to Diffusion Models [16, 58, 29, 52, 21, 44] and autoregressive models [63, 12, 50J. Prior studies that are most related to this work build on top of a learned VQ-VAE codebook to generate images. Autoregressive transformer [19, 69, 7, 37], similar to decoder-only language models, model each patch in a step-by-step step fashion, thus requiring many steps token number, e.g., 256 1024. Non-
3

---

K
tokens
raconstruction
Titok loncndrr
potches
latent tokens
irput
Exc
Vision Transformer
(a) Image Reconstruction
Quant
Bidirectional
Transformer
Transformer
Iatent
maskedd
predicted
tokens
takens
Image Generation
(e) TiTek Tokenization
Figure 3: Illustration of image reconstruction (a) and generation (b) with the TiTok framework (c). TyTok contains an encoder Enc, a quantizer Quant, and a decoder Dec. Image patches, along with few (e.g., 32) latent tokens, are passed through the Vision Transformer (ViT) encoder. The latent tokens are then vector-quantized. The quantized tokens, along with the mask tokens [15, 24], fed the ViT decoder reconstruct the image.
autoregressive (or bidirectional) transformers [80, 72, 68], such as MaskGIT [9], generally predict more than a single token per step and thus require significantly fewer steps to predict a complete image. Apart from that, further studies looked into improved sampling strategies [39, 40, 38J. As focus on the tokenization stage, we apply the commonly used non-autoregressive sampling scheme of MaskGIT generate a sequence of tokens that is later decoded into image.
3 Method
3.1 Preliminary Background on VQ-VAE
The image tokenizer plays pivotal role facilitating the generative models by providing a compact image representation at latent space. For the scope of our discussion, we primarily focus on the Vector- Quantized (VQ) tokenizer [64, 19]. given its broad applicability across various domains, including but not limited to image and video generation [19, 9, 58, 71], large-scale pretraining [12, 5, 49, 3, 18] and multi-modal models [20, 73].
typical VQ model contains three key components: an encoder Fnnc, a vector quantizer Quant, and decoder Dec. Given an input image I E Hx x3, where H and W denote the image's height and width, the image is initially processed by the encoder Enc and converted to latent embeddings
Zep Fnc(I), where Zep TR. $ + xD which downsamples the spatial dimensions by factor of f- Subsequently, each embedding 2 E R is mapped (via the vector quantizer Quant) to the nearest code c; RDD in learnable codebook C E RND comprising N codes. Formally, have:
Quant(2) C₁. where i argmin ||2 c/ll2-
(1........N
During de- de-tokenization, the reconstructed image i obtained via the decoder Dec as follows:
i
Dee(Quant((Za)))
Despite the numerous improvements over VQ-VAE [64] (e.g., loss function [19], model architec- ture [69], and quantization/codebook strategies [80, 37, 72]), the fundamental workflow (e.g., the 2D grid-based latent representations) has largely remained unchanged.
TiTok: From 2D 1D Tokenization
While existing VQ models have demonstrated significant achievements, notable limitation within the standard workflow exists: latent representation Zen is often envisioned as static 2D grid. Such configuration inherently assumes strict one-to-one mapping between the latent grids and the original image patches. This assumption limits the VQ model's ability to fully exploit the redundancies present in images, such as similarities among adjacent patches. Additionally, this approach constrains

---

the flexibility in selecting the latent size, with the most prevalent configurations being f= 4, 8, or f= 16 [58], resulting in 4096, 1024, or 256 tokens for an image of dimensions 256 x 256 Inspired by the success of 1D sequence representations in addressing broad spectrum of computer vision problems [8, 1, 41], we propose to use a ID sequence, without the fixed correspondence between latent representation and image patches in 2D tokenization, as an efficient and effective latent representation for image reconstruction and generation.
Image Reconstruction with TiTok. To initiate our exploration, we establish a novel frame- work named Transformer-base based I-Dimensional Tokenizer (TīTok), leveraging Vision Transformer (ViT) [17]' to tokenize images into 1D latent tokens and subsequently reconstruct the original images from these ID latents. As depicted in Fig. 3, TiTok employs standard ViT for both the tokenization and de-tokenizztion processes (i.e., both the encoder Enc and decoder Dec are ViTs), During
tokenization, we patchify the image into patches (with patch embedding layer) P E R x 7 x (with patch size equal to the downsampling factor f and embedding dimension D) and concatenate them with K latent tokens L E R They are then fed into the ViT encoder Enc. In the encoder output, we only retain the latent tokens as the image's 's latent representation, thereby enabling more compact latent representation of 1D sequence Z (with length K). This adjustment decouples the latent size from image's 's resolution and allows more flexibility in design choices. That is, we have:
Zn = Enc(P + L);
where @ denotes concatenation, and we only retain the latent tokens from the encoder output.
In the de-tokenization phase, drawing inspiration from [15, 5, 24], we incorporate a sequence of mask tokens M E IR # x 4 xD obbained by replicating a single mask token y + times-to the quantized latent tokens LID. The image is then reconstructed via the VIT decoder Dec follows:
W
i Dec(Quant (Z.D) M).
where the latent tokens Z is first vector-quantized by Quant and then concatenated with the mask tokens M before feeding to the decoder Dec.
Despite its simplicity, we emphasize that the concept of compact ID image tokenization remains underexplored in existing literature. The proposed TyTok thus serves as a foundational platform for exploring the potentials of ID tokenization and de-tokenizatinn that although one may flatten 2D grid latents into ID sequence, it significantly differs from the proposed DD tokenizer, due to the fact that the implicit 2D grid mapping constraints still persist.
for natural images. It is worth noting
Image Generation with TiTok. Besides the image reconstruction task which the tokenizer is trained for, we also evaluate its effectiveness for image generation, following the typical pipeline [19, 9I. Specifically, we adopt MaskGIT [9] as our generation framework due to its simplicity and effectiveness, allowing us to train MaskGIT model by simply replacing its VQGAN tokenizer with our TiTok. We do not make any other specific modifications to MaskGIT, but for completeness, we briefly describe its whole generation process with TYTok.
The image is pre- toeeiokee into ID discrete tokens. At each training step, a random ratio of the latent tokens are replaced with mask tokens. Then, a bidirectional transformer takes the masked token sequence as input, and predicts the corresponding discrete token IDD of those masked tokens. The inference process consists of multiple sampling steps, where at each step the transformer's prediction for masked tokens will be sampled based on the prediction confidence, which are then used to update the masked images. In this way, the image is "progressively generated" from a sequence full of mask tokens to an image with generated tokens, which can later be de-tokenized back into pixel spaces, The MaskGIT framework shows a significant speed-up in the generation process compared to ruto-regressive models. We refer readers to [9] for more details.
3.3 Two-Stage Training of TiTok with Proxy Codes
Existing Training Strategies for VQ Models. Although most VQ models adhere to a straightforward formulation, their training process is notably sensitive, and the model's performance is heavily influenced by the adoption of more effective training paradigms. For instance, VQGAN [19] achieves
Although other Transformer based architectures (e.g., Swin [45]) can also be used to instantiate TiTok, we adopt ViT for its simplicity and effectiveness.
5

---

a significant improvement in reconstruction FID (rFID) on the ImageNet [14] validation set, when compared to d'VEE from DALL-E [55]. This enhancement is attributed advancements in perceptual loss [33, 79] and adversarial loss [23]. Moreover, MaskGIT's modern implementation of VQGAN [9] utilizes refined training techniques without architectural improvements to boost the performance further. Notably, most of these improvements are exclusively applied during the training phase (i.e., through auxiliary losses) and significantly affect the models' efficacy. Given the complexity of the loss functions, extensive tuning of hyper-parameters involved, and, most critically, the missing of a publicly available code-base for reference or reproduction [9, 69, 72], establishing an optimal experimental setup for the proposed TiTok presents substantial challenge, especially when the target is a compact ID tokenization which was rarely studied in literature.
Two-Stage Training Comes to the Rescue. Although training TiTok with the typical Taming- VQGAN [19] setting is feasible, we introduce two-stage training paradigm for an improved performance. The two-stage training strategy contains "warm-up" and "decoder fine-tuning" stages. Specifically, in the first "warm-up" stage, instead of directly regressing the RGB values and employing variety of loss functions (as in existing methods), we propose to train ID VQ models with the discrete codes generated by an off-the-shelf GITGGIN model, which we refer to as prexy codes. This approach allows us to bypass the intricate loss functions and GAN architectures, thereby concentrating our efforts on optimizing the ID tokenization settings. Importantly, this modification does not harm the functionality of the tokenizer and quantizer within TiTok, which still fully function for image tokeniztion and de-tokenization; the main adaptation simply involves the processing of TiTok's s de-tokenizer output. Specifically, this output, comprising set of proxy codes, is subsequently fed into the same off-the-shelf VQGAN decoder to generate the final RGB outputs. It is noteworthy that the introduction of proxy codes differs from simple distillation [26]. As verified in our experiments, TYTok yields significantly better generation performance than MaskGIT-VQGAN.
After the first training stage with proxy codes, we optionally have the second "decoder fine-tuning" stage, inspired by [10, 53], to improve the reconstruction quality. Specifically, we keep the encoder and quantizer frozen, and only train the decoder towards pixel space with the typical VQGAN training recipe [19]. We observe that such a two-stage training strategy significantly improves the training stability and reconstructed image quality, shown in the experiments.
4 Experimental Results
4.1 Preliminary Experiments of 1D Tokenization
Building upon TiTok, we explore a range of configurations, including the model size and the number of tokens, to identify the most efficient and effective setup for a ID image tokenizer. These preliminary experiments serve to provide a thorough evaluation, seeking practical configuration of TiTok.
Preliminary Experimental Setup. Unless specified otherwise, we train all models with images of resolution H = 256 and W = 256, using the open-source MaskGIT- proxy codes for training. The patch size for both tokenizer and de-tokenizer is established with f f= 16, and the codebook C is configured to have N = 1024 entries with each entry a vector with 16 channels. For TiTok variants, we primarily investigate three model tizes-smale base, and large (i.e., TYTok-S, TB TTTok-L)-comprisss comprising 22MM. 86M, and 307M parameters for encoder and decoder, respectively. We also assess the impact of varying the number of latent tokens K from 16 to 256. We perform ablation experiments with an efficient setting (e.g.. shorter training).
[9] to supply
Evaluation Protocol. Evaluation is conducted across multiple metrics thoroughly assess the models, including both reconstruction and generation FID metrics (i.e., rFID and gFID) [25] on the ImageNet dataset. We examine training/inference throughput to offer a direct comparison of generative model's 'ss efficiency relative to different latent sizes. Furthermore, given that the ID VQ model inherently serves form of compact image compression, we further investigate the semantic information retained by the model through linear probing following MAE setting [24]. For the complete details of the training and testing protocols (e.g., parame-sr,,etess, training costs), we refer the reader to the supplementary material Sec. A.
After the setup, we now summarize the preliminary experimental findings below.
An Image Can be Represented by 32 Tokens. The redundancy inherent in image representation is well-acknowledged, as evidenced by the practice of masking significant portions of images (e.g..

---

Trbk-5 Tibke
Taming MaskGIT-VOAAN
talmin throughout Go commpozzossooo
recometruce HD (rfl))
Cemoraae FID (aFiD)
DE number intent tolann (a) IN- Roconstruction
1H.0 to de number latent takens (b) IN- 1K Lintor Probe
The A number litent Lokens (c) IN-1K Generation
12. 31
1o of latent tokens (d) Sampling Speed
Figure 4: Preliminary experimental results with different TiTok variants. We provide prehensive exploration in (a) ImageNet-IK reconstruction. (b) ImageNet-I K linear probing. (c) ImageNet-IK generation. (d) Training and inference throughput of MaskGIT-ViT generator and TiTok as tokenizer (evaluated on A 100 GPUs, inference includes de-tokenization step with TiTok-B). Detailed numbers be found in supplementary material Sec. B.
75% in MAE [24]) to expedite the training process without negatively affecting performance. This strategy has been validated across variety of computer vision tasks that rely on high-level image features [30, 42]. However, the efficacy of such approaches in the context of image reconstruction and generation- where both low-level and high-level details are crucial for creating realistic recon- structed and generated outputs remains underexplored. Consequently, in this experiment, we aim to determine the minimum number of tokens required to reconstruct and generate high-quality images. As depicted in Fig. 4a, although model performance progressively improves with an increase in the number of latent tokens, significant enhancements are predominantly observed when K ranges from 16 to 128. Beyond this point, increasing the latent space size yields only marginal gains. Intriguingly, we find that with merely 32 latent wokns, TiTok-L L achieves performance better than a 2D VQGAN model [19] using 256 tokens. This observation suggests that as few as 32 tokens may suffice as an effective image latent representation, optimizing the utilization image redundancy.
Scaling Up Tokenizer Enables More Compact Latent Size. Another intriguing observation from Fig. 4a is that larger tokenizers facilitate more compact representations. Specifically, TYTok-B with 64 latent tokens achieves performance comparable to TYTok-S with 128 latent tokens, while T with 32 latent tokens matches the performance of -Y with 64 latent tokens. This pattern indicates that with each incremental increase in TiTok size (e.g.. from S to B, or from B to L). it is possible to reduce the size of the latent image representation without compromising performance. This trend underscores the potential benefits of scaling up the tokenizer to achieve even more compact image representations.
Semantics Emerges with Compact Latent Space. To evaluate the learned image representation, we perform linear probing experiments on the image tokenizer, as shown in Fig. 4b. Specifically, we add a batch normalization layer [31] followed by linear layer on top of the frozen features from TYTok encoder, with all the size of the latent representation decreases, the tokenizer increasingly learns semantically rich representations, as indicated by the improved linear probing accuracy. This suggests that the model learns high-level information in scenarios of constrained representation space.
strictly following the MAE protocol [24]. We find that as
Compact Latent Representation Improves Generative Training. In addition reconstruction capabilities, we assess TiTok's s effectiveness and efficiency in generative downstream tasks, as illustrated in Fig. 4c and Fig. 4d. We note that variants of different tokenizer sizes yield comparable outcomes when the number of latent tokens is sufficiently large (i.e., K > 128). However, within the domain of compact latent sizes (i.e., K 64), larger tokenizers notably enhance performance. Furthermore, the adaptability of 1D tokenization in TiTok facilitates more efficient and effective generative model training. For instance, model variants with quality, demonstrate significantly better generative performance, underscoring the advantages of employing a more condensed and semantically rich latent space for generative model training. Additionally, the reduction in latent tokens markedly accelerates training and inference, with 12.8x increase training speed (2815.2 vs. 219.7 samples/s/gpu) and a 4.5x speed up sampling speed (123,1 vs. 27.5 samples/s/gpu), when utilizing K 32 as opposed to K 256.
32, despite inferior reconstruction
7

---

Table 1: ImageNet-1K 256 x 256 generation results evaluated with ADM [16]. 4: Trained on Openlmages [36] $: Trained on Openlmages, SALON-Aestheticee Humbnss [59]. P: generator's
parameters. S: sampling steps. T: throughput as samples per seconds on A100 with float32 precision.
tokenizer
#tokens codebook size IFID
generator
gFID P4.
St
T
diffusion- based generative models
Taming- VQGANF [58]
1024
16384
1.14
LDM-8 [58]
7.76
258M
200
VAET [58]
4096x3
0.27
LDM-4 [58]
3.60
400M
250
0.4
UVIT-L/2 [4]
3.40
287M
50
1.1
VAE [60]
1024x4
0.62
UViT-H/2 [4]
2.29
501M
50
0.6
DiT-XL/2 [52]
2.27
675M
250
0.6
transformer-basee generative models
Taming-VQGAN [19]
256
1024
7.94
Taming- Transformer [19]
15.78
1.4B
256
7.5
8.71
1.4B
16.1
RQ-VAB [37]
256
16384
3.20
RQ- RQ-Trrasforrmer [37]
64
7.55
3.8B
MaskGIT-VQGAN [9]
256
1024
2.28
MaskGIT-ViT
6.18
177M
8
50.5
ViT-VOGAN [69]
1024
8192
1.28
VIM-Large [69]
4.17
1.7B
1024
0.3
Trook-L332
32
4096
2.21
Mask VIT [9]
2.77
177M
101.6
TYTok-B-64
64
4096
1.70
MaskGIT VIT [91
2.48
177M
8 89.8
2.50
8 53.3
TYTo----128
128
4096
1.71
MaskGIT-UViT-L [9.4]
287M
1.97
64
7.8
4.2
Main Experiments
Based on the observations above, the proposed TiTok family effectively trades off a larger model size to more compact latent size. this section, we majorly focus on ImageNet generation benchmarks against prior arts, and evaluate TiTok as tokenizer in the generative MaskGIT framework [9].
Implementation Details. We primarily investigate the following TiTok variants: TiTok-S-128 (i.e., small model with 128 tokens), TiTok-B-6k (i.e., base model with 64 tokens), and TiTok-L-32 (i.e., large model with 32 tokens), where each variant designed to halve the latent space size while scaling up the model size. For resolution 512, we double the latent size to ensure more details are kept at higher resolution, leading to TYTok-L-64 and TYTok-B. Berook---128 128. In the final setting for TYTok training, the codebook is configured to N = 4096, and the training duration is extended to 1 M iterations (200 epochs). We also adopt the "decoder fine-tuning" stage to further enhance model performance, where the encoder and quantizer are kept frozen and the decoder is fine-tuned for 500k iterations. For the training of generative models, we utilize the MaskGIT [9] framework without any specific modifications, except for the adoption of an arccos masking schedule [6]. All other parameters are the same as previous setups, and all design improvements will be verified in the ablation studies.
Main Results. We summarize the results on ImageNet- IK generation benchmark of resolution 256 x 256 and 512 x 512 in Tab. and Tab. 2, respectively.
For ImageNet 256 x 256 results in Tab. 1, TiTok can achieve a similar level of reconstruction FID) (rFID) with much smaller number of latent tokens than other VQ models. Specifically, using merely 32 tokens, TiTok-L-32 achieves a rFID of 2.21, comparable to the well trained VQGAN from MaskGIT [9] (rFID 2.28), while using 8x smaller latent representation size. Furthermore, when using the same generator framework and same sampling steps, TiTok-L-32 improves over MaskGIT by a large margin (from 6818 to 2.77 gFID), showcasing the benefits of a more effective generator training with compact ID tokens. When compared to other diffusion-basee generative models, TYTok can also achieve a competitive performance while enjoying an over 100 speed-up during the sampling process. Specifically, TYTok-L-32 achieves a better gFID than LDM-4 [58] (2.77 vs. 3.60), while generating images dramatically faster by 254 times (101.6 samples/s vs. 0.4 samples/s). Our best-performing variant TYTok-S- 1-28 outperforms state-of-the-an diffusion method DiT-XL/2 [52) (gFID 1.97 vs. 2.27), with 13x speed-up.
For ImageNet 512 x 512 results in Tab. 2, the significantly better accuracy-cost trade-off of TiTok persists. TiTok maintains a reasonably good rFID compared other methods, especially considering that TiTok uses much fewer tokens (i.e., higher compression ratio). For generation, all TiTok variants
For faimess, we mainly consider tokenizers with vanilla VQ modules. More advanced quantization methods [72, 48] may further benefit TTTok but beyond this paper's focus on ID image tokenization. See supplementary material Sec. C for the complete table.
8

---

Table 2: ImageNet-1K 512 x 512 generation results evaluated with ADM [16]. :: Trained on OpenImages, LAION-Aestheties and LAION- [59]. P: generator's parameters. St sampling steps. T: throughput samples per seconds A100 with float32 precision.
tokenizer
#tokens codebnok size rFID
generator
gFTD
P4
St
diffusion tiffesion-bused generative models
UViT-L44 [-]
4.67
287M
1.0
VAB [60]3
4096x4
0.19
UViT- H/4 [4]
4.05
501M
0.6
DIT-XL/2 [52]
3.04
675M
250
0.1
transformer- hased generative models
MaSAGIT-VOGAN [9]
1024
I024
1.97
MaskGIT-ViT [9]
7.32
177M
3.9
TYTok-L-64
4096
1.77
MaskGIT-ViT [9]
2.74
177M
41.0
MaskGIT-ViT [9]
2.49
177M
33.3
TYTok-B-128
128
4096
1.52
MaskGIT-ViI [9]
2.13
177M
64
7.4
Table 3: Ablation study improved final models for main experiments. We ablate the tokenizer designs, and generator designs on ImageNet-Ik lk benchmark. The final settings are labeled in gray. Generation results are based on tokenizers without decoder fine-tuning
(a) TYTok configuration. Results (b) Masking schedules for genera- reported in accumulation manner
(c) Effects proxy codes
tor with TiTok-L-33
FFID
T-Tok-L--32
#IDD
IS1
schedule
gFID
1St
Taming- VQGAN training setting
baseline
6.59
110.3
CoSIDe
5,17
Taming VOGAN (2D)
7.94
larger codchook
5.85
116.6
arccos
4,94
194.0
Wiak B 64 (2D)
200 epachs
5.48
117.3
linear
4.95
193.7
TrTok-
5.15
120.5
decoder linetuning
2.21
195.5
square
5.63
170.9
Two stage training with praxy coder
TrTak-B-64
significantly outperform our baseline MaskGIT [9] by a large margin. When compared with diffusion- based models, TYTok-L-64 shows a superior performance to DiT-XL/2 [52] (2.74 vs. 3.04), while running 410x faster. The best-performing variant TiTok-B-128 can significantly outperform DiT- XL/2 by large margin (2. vs. 3.04) but also generates high-quality samples 74x faster. We also provide visualization results and analysis in supplementary material Sec. D.
4.3 Ablation Studies
We report the ablation studies regarding our final model designs in Tab. 3. Specifically, in Tab. 3a, we ablate the tokenizer designs on image reconstruction. We begin with our baseline TYTok-L-32 which attains 6.59 rFID. Employing a larger codebook size improves the rFID by 0.74, while further increasing the training iterations (from 100 epochs to 200 epochs) yields another 0.37 improvement of rFID. On top of that, the "decoder fine-tuning" (our stage-2 training strategy) can substantially improve the overall reconstruction performance to 2.21 rFID.
In Tab. 3b, we examine the effects of different masking schedules for MaskGIT with TiTok. Inter- estingly, unlike the original MaskGIT setting [9] which empirically found that the cosine masking schedule significantly outperforms the other schedules, we observe that MaskGIT equipped with TiTok changes the preference to the arccos or linear schedules. Additionally, unlike [9] which reported that the root schedule performs much worse than the others, we observe that TYTok is quite robust to different masking schedules. We attribute the observations to TTTok's ability to provide more compact and more semantic meaningful tokens compared to 2D VQGAN, as compared to the cosine masking schedule, linear and arccos schedules have a lower masking ratio in the early steps. This coincides with the observation that masking ratio is usually higher for redundant signals (e.g.. 75% masking ratio in images [24]) while relatively lower for semantic meaningful inputs (e.g.. 15% masking ratio in languages [15]).
We ablate the effects of training paradigm in Tab. 3c. We begin with the training setting of Taming- VQGAN [19], where TiTok- B-66 B-64 obtains 5.15 rFID, outperforming the original 2D Taming- VQGAN's 7.94 rFID under the same training setting. We also show the necessity of 1D tokenization by building a 2D variant of TTTok-B64, where the architecture remains the same except that image patches instead of latent tokens are used image representation. As a result, we observe that the 2D variant suffers from much worse performance (15.58 vs. 5.15 rFID), since the fixed correspondences in 2D tokenization limited reasonable reconstruction under compact latent space. This result

---

demonstrates the effectiveness of the proposed ID tokenization, especially at much more compact latent size. Although TiTok can achieve a reasonably well performance under straightforward single-stage training, there exists performance gap compared to the MaskGIT-VOCE VQGAN [9] due to the missing Therefore, we adopt the two-stage training with proxy codes, which proves to be effective and can (1.70 vs. 2.28 rFID). is noteworthy that the two-stage training
strong training recipe, of which no public reference or access exists.
outperform the MaskGIT-VQGi is not that crucial to obtain reasonable ID tokenizer, and we believe that TiTok, with the simple single-stage Taming-VQGAN s training setting, could also benefit from training on a lagrer-scale dataset [36] as demonstrated in [58] and we leave it for future work due to the limited compute.
Details on the Two-Stage Training. We provide more technical details on the two-stage training. Specifically:
In the first stage (warm-up stage), we use an off-the-shelf preeaine--retrriieed MaskGIT-VO tokenizer to tokenize the input image into 256 tokens, which we refer as proxy codes.
the proxy In the first stage training, instead of regressing the original RGB values, codes as reconstruction targets. Specifically, the workflow is: RGB images are patchified and flattened into sequence and concatenated with 32 latent tokens, then they fed into TyTok-Ene (Encoder of TiTok). Later, the latent tokens are kept as token representation and go through the quantizer. The quantized latent tokens are concatenated with 256 mask tokens and go through the TīTok-Dec (Decoder of TīTok). And the final output mask tokens are supervised by proxy codes using cross-entropy loss.
Afterwards, freeze both the TyTok-Ene and quantizer, and then only fine-tune the TYTok-Der (responsible for reconstructing proxy codes) and MaskGIT-Dec (responsible for reconstructing RGB values from proxy codes) end-to-end towards pixel space, where the training losses include 1.2 loss, perceptual loss, and GAN loss following the common VQGAN paradigm.
Moreover, also note that two-stage training is not necessary for TiTok training, and it works fine with the commonly used and publicly available Saming-VQGAN recipe as is shown Tab. 3c In this case, the whole workflow is pretty straightforward, where the TiTok-Dec will instead directly reconstruct the images at pixel space.
recipe (developed more than 3 years ago) leads to an inferior FID score when compared to state-of-the-an tokenizers, putting TiTok at disadvantage when compared against other methods. Therefore we propose the two-stage training to benefit TiTok from the state- of-art MaskGIT- VQGAN tokenizer, which shares a similar architecture to Suming-VQGAll but has a significantly better score (rFID 2.28 v.s. 7.94).
However, the
We also note that TiTok can work well with single-stage recipe, and it promising incorporate the recent modern VQGAN recipe from [47, 68] in our preliminary experiments.
5 Conclusion
In this paper, we have explored a compact ID tokenization TiTok for reconstructing and generating natural images. Unlike the existing 2D VQ models that consider the image latent space 2D grid, we provide more compact formulation to tokenize an image into a ID latent sequence. The proposed TiTok represent image with 8 to 64 times fewer tokens than the commonly used 2D tokenizers. Moreover, the compact ID tokens not only significantly improve the generation model's training and inference throughput, but also achieve competitive FID on the ImageNet benchmarks. We hope our research can shed some light in the direction towards more efficient image representation and generation models with ID image tokenization.
10

---

References
[1] Jean-Baptiste Alayrae, Jeff Donahue, Pauline Lue, Antoine Micch, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurlPS, 2022.
[2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. ICCV, 2015.
[3] Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei Efros. Sequential modeling enables scalable learning for large vision models. CVPR, 2024.
[4] Fan Bao, Shen Nic, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words; vit backbone for diffusion models. In CVPR, 2023.
[5] Hangho Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2022.
[6] Victor Besnier and Mickael Chen. A pytorch reproduction of masked generative image transformer. arXiv preprint arXiv:2310. 14400, 2023.
[7] Shiyue Cao, Yueqin Yin, Lianghua Huang, Yu Liu, Xin Zhao, Deli Zhao, and Kaigi Huang. Efficient-vggam Towards high-resolution image generation with efficient vision transformers. In ICCV, 2023.
[8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[9] Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022.
[10] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Hing-Hsuann Yang. Kevin Murphy, William T Freeman, Michacl Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. In JCML, 2023.
[11] Jieneng Chen, Qihang Yu, Xiaohui Shen, Alan Yuille, and Liang-Chich Chen. Vitamin: Designing scalable vision models in the vision-language era. In CVPR, 2024.
[12] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020.
[13] Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C LLwrenee Ziticck. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXivv1104. 00325, 2015.
[14] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[15] Jacob Devlin, Ming-Wes Mei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-training of deep bidirec- tional transformers for language understanding. In NAACL, 2018
[16] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 2021.
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR.
2021.
[18] Alaaeldin El-Nouby Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. arXiv preprint arXiv:2401.085 1,05411 2024.
[19] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[20] Oran Gafni, Adam Polyak. Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In ECCV, 2022.
[21] Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is strong image synthesizer. In ICCV, 2023.
11

---

[22] Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge. Chen Li, Xintao Wang, and Ying Shan. Making Ilama see and draw with seed tokenizer. In ICLR, 2024.
[23] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farte Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
[24] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learerrs. In CVPR, 2022.
[25] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. NewrIPS, 2017.
[26] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 550.00231, 2015.
[27] Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786), 2006.
[28] Jonathan Ho and Tim Salimans. Classifier-frce diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
[29] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In /CML., 2023.
[30] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-hert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004. 00849, 2020.
[31] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[32] Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei. An Liu, Chengru Song, Xiaoqiang Lei, et al. Unified language-vision pretraining with dynamic discrete visual tokenization. In ICLR, 2024.
[33] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
[34] Dahun Kim, Jun Xie, Huiyu Wang, Siyuan Qiao, Qihang Yu, Hong-Seok Kim, Hartwig Adam, In So Kweon, and Liang-Chich Chen. debefo:merddeeplab: Video mask transformer. In CVPR, 2022.
[35] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014.
[36] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popoy, Matteo Malloci, Alexander Kolesnikov, al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale. IJCV, 2020.
[37] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, 2022.
[38] Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and WOOK SHIN HAN, Draft-and-revise: Effective image generation with contextual rg-transformer. NewrIPS, 2022.
[39] José Lezama, Huiwen Chang, Lu Jiang, and Irfan Essa. Improved masked image generation with token- critic. In ECCV, 2022.
[40] José Lezama, Tim Salimans, Lu Jiang, Huiwen Chang, Jonathan Ho, and Irfan Essa. Predictor-corrector sampling for discrete diffusion models. JCLR, 2023.
[41] Jumnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-mmae pre-training with frozen image encoders and large language models. In ICML, 2023.
[42] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language- image pre-training via masking. In CVPR. 2023.
[43] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. NewrlPS, 2023.
[44] Qihao Liu, Zhanpeng Zeng. Ju He. Qihang Yu, Xiaohui Shen, and Liang Chich Chen. Alleviating distortion in image generation via multi-resolutie resolution diffusion models. NewrIPS, 2024.
12

---

[45] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[46] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv: 2017.
[47] Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujuu Yang, Limin Wang, and Ying Shan. Open-magyit2: An open- souree project toward democratizing auto-regressive visual generation. arXiv preprint arXiv:2409.0441 2024.
[48] Fabian Mentzer. David Minnen. Eirikur Agustsson, and Michael Tschannen. Finite sealar quantization: VQ- VAE made simple. In ICLR, 2024.
[49] David Mizrahi, Roman Bachmann, Oguzhan Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir, 4m: Massively multimodal masked modeling. NewrlPS, 2023.
[50] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. NewrIPS, 2017.
[51] OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.08771 2023.
[52] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
[53] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:3307.01955 2023.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[55] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
[56] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. NeurlPS, 2019.
[57] Jason Tyler Rolfe. Discrete variational autoencoders. In JCLR, 2017.
[58] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffuusinn models. In CVPR, 2022.
[59] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Muliis, Mitchell Wortsman, et al. Laion-Sb: An open large-scale dataset training next generation image-text models. NeurlPS, 2022.
[60]
stabilityai, 2023. URL http:///ggggggggg..o/////aaaaa// vae- ema.
ct/ www.////tsttttt//eeeeeeeeeeeeeee
[61]
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. In ICLR, 2024.
[62] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307 09288 2023.
[63] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espcholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. NewrIPS, 2016.
[64] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NewrIPS, 2017.
[65] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
[66] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre- AntrreAntoine Manzagol. Extracting and composing robust fcatures with denoising autocncoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103. 110,, 2008
[67] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang Liang-Chieh Chen. dex-deeplab; End-to-end panoptic segmentation with mask transformers. In CVPR, 2021.
13

---

[68] Mark Weber. Lijun Yu, Qihang Yu, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. Maskbit: Embedding-free free image generation via bit tokens. arXiv preprint 2Xiv:2409.162211, 2024.
[69] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. In ICLR. 2022.
[70] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, al. Scaling autoregressive models for content-rich text-to-imape generation. TMLR, 2022.
[71] Lijun Yu, Yong Cheng. Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann. Ming-Hlsuan Yang. Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In CVPR, 2023.
[72] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari. Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats tokeeizeee is key to visual generation. In ICLR, 2024.
[73] Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang. Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint wrXiv:2309.02536 2023.
[74] Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. Cmt-deeplab: Clustering mask transformers for panoptic segmentation. In CVPR, 2022.
[75] Qihang Huiyu Wang. Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means Mask Transformer. In ECCV, 2022.
[76] Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, Liang- Chinh Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip. NeurIPS, 2023.
[77] Qihang Xiaohui Shen, and Liang-Chieh Chieh Chen. Towards open-ended visual recognition with large language model. In ECCV, 2024.
[78] Hao Zhang, Feng Li. Shilong Liu, Lei Zhang, Hang Su, Zhu, Lionel Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint 2xXvv2203.033605, 2022.
[79] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metrie. In CVPR, 2018.
[80] Chuanxia Zheng. Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. NeurlPS, 2022.
[81] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-en and object detection. In ICLR, 2020.
14

---

Appendix
In the supplementary materials, we provide the following additional details:
The comprehensive training and testing hyper- parameters and training costs for TiTok (Sec. A).
The detailed results of the preliminary experiments reported in main paper's Fig. 4(Sec. B).
A more comprehensive comparison with more metrics and baselines (Sec. C).
Qualitative visualizations (Sec. D).
Limitation discussion (Sec. E).
Broader Impacts discussion (Sec. F).
Dataset Licenses (Sec. G).
A Training and Testing Protocols
For image reconstruction (tokenizer) at preliminary experiments, the training augmentation is confined to random cropping and flipping, following [19]. The training regimen spans a short schedule, featuring a batch size of 256 over 500k training iterations, which correlates to roughly 100 epochs the ImageNet dataset. We employ the AdamW optimizer [46] with initial learning rate of x 10-4 (with cosine decay) and weight decay x 10-4 We only adopt stage-1 1 training here (i.e.. only the "warm-up" training stage). For the main experiments, we adopt the improvements as shown in the ablation study (Tab. 3 in main paper), including longer training to 200 epochs and decoder fine-tuning, all other hyper-parameters remain the same. We patch size 16 for all vision transformers at resolution 256 x 256 and increase to 32 for resolution 512 x 512 to ensure a computation efficiency.
TyTok-L refers to using a VIT-L for TiTok encoder and decoder, and TiTok-B, TiTok-S refers to using ViT-B and VIT-S respectively. Moreover, the tokenizer training takes 64 A100-40G for 74 hours (TYTok-L-32), 32 A100-40G for 41 hours (TYTok-B-64 32 A100-40G for 50 hours (TTok-S-128), 32 A100-40G for 70 hours ---1o----128 for resolution 512), and 64 A100-40G for 91 hours (TiTok-L-64 for resolution 512), respectively.
For image generation (generator) at preliminary experiments, we majorly build the training and testing protocols on top of [9]. Specifically, all images are pre-tokenized using center crop and random flipping augmentation, and then processed by MaskGIT [9] to generate images via the masked image modeling procedure. During inference, a cosine masking schedule is utilized with steps. The generative models are trained with a batch size of 2048 and 500k iterations to improve training efficiency. We use AdamW optimizer [46] with learning rate 2 x 10-4 and weight decay 0.03. The learning rate starts from x 10-4 and then decay to 1 10-5 following cosine decaying schedule. We apply a dropout probability of 0.1 on the class condition. The only differences of main experiments paper Tab. 3), all other hyper-parameter remain the same. We follow prior arts [19, 9] to generate 50% samples for generation FID evaluation. We also adopt classifier-free guidance [28] following prior arts [10, 72].
using arccos masking schedule discussed in the ablation study (main
At ImageNet 256 256, we guidance scale 4.5, temperature 9.5 for TiTok-L-32, guidance scale 3.0, temperature 11.0 for TiTok-B-64, guidance scale 2.0, temperature 3.0 for TYTok-S-128, At ImageNet 512 512, we guidance scale 2.0, temperature 7.5 for TiTok-L-64, guidance scale 2.5, temperature 6.5 for TYTok-B-11
The generator training takes 32 A100-40G for 12 hours (TYTok-L-32), 16 hours (TyTok-B-ol B--4), 29 hours (TiTok-S-128) 128), 26 hours (TYTok- B-128 for resolution 512), 18 hours (TYTok-L-64 for resolution 512) respectively.
B Detailed Results of Preliminary Experiments
We summarize the detailed results for Fig. of main paper in Tab. 4.
15

---

Table Detailed results preliminary experiments main paper.
(a) reconstruction FID.
(b) generation FID.
#token
16
32
64
96
128
192
256
#token
16
32
64
96
128
192
256
S
25.3
16.2
9.6
6.9
5.6
4.3
3.7
S
16.1
8.7
4.4
3.6
3.5
3.0
3.2
TTTok-B
16.8
9.7
5.9
4.7
3.8
3.2
2.9
TiTok-B
10.9
7.5
4.1
3,4
3.2
3.1
3.5
TiTok-L
13.0
6.6
4.0
3,4
3.0
2.6
2.5
TiTok-L
9.0
6.3
4.3
3.9
3.7
3.7
3.7
(c) Linear Probing accuracy.
#token
16
32
64
96
128
192
256
TiTok-S
50.46
48.01
48.40
48.12
46.55
47.05
44.88
TTTok-B
57.70
54.79
53,92
53.72
53.59
51.75
52.11
TiTok-L
62.10
60.03
58.85
56.12
54.35
53.95
54.36
Table 5: ImageNet-1K 256 x 256 generation results evaluated with ADM [16]. 4: Trained on Openlmages [36] $: Trained on Openlmages, -AlON-Aestheticc/-HHe Hummans [59]. P: generator's
parameters. S: sampling steps. T. throughput samples per seconds on A100 with float32 precision,
measured with w/ guidance variants available. "guidance" refers to classifier-free guidance.
wlo guidance w/ guidance
tokenizer
IFID
generator
gFID 1St gFIDJ. IS1
P
S4
Tt
diffusion- based generative models
Taming- VQGANT [58]
1.14
LDM-8 [58]
15.82
78.82
7.76
209.5
258M
200
VAE [58]
0.27
LDM-4 [58]
10.56
103.5
3.60
247.7
400M
250
0.4
UVIT-L/2 [4]
9.03
111.5
3.40
219.9
287M
50
VAE [60]#
0.62
UVIT-H/2 [4]
6,60
142.5
2.29
263.9
501M
50
0.6
DiT-XL/2 [52]
9.62
121.5
2.27
278.2
675M
250
0.6
transformer- based generative models
Taming- VQGAN [19]
7.94
Transfommer [19]
15.78
78.3
1.4B
256
7.5
8.71
119.0
1.4B
16.1
RQ-VAE [37]
3.20
RO-arassformer [37]
64
7.55
134.0
3.8B
9.7
MaskGIT-VQGAN [9]
2.28
MaskGIT VIT [9]
6.18
182.1
177M
8
50.5
VIT-VQGAN [69]
1.28
VIM-Large [69]
4.17
175.1
1.7B
1024
0.3
LFQ [72]
0099
MAGVIT-v2 [72]
3.65
200.5
1.78
319.4
307M
1.1
TTTo
2.21
MaskGIT VIT [9]
3.15
173.0
2.77
199.8
177M
8
101.6
TYTok-B-66
1.70
MaskGIT- VIT [9]
3.08
192.5
2.48
214.7
177M
8
89.8
4.61
166.7
2.50
278.7
53.3
TYTok-S-128
1.71
MaskGIT-UViT-L [9. 4]
287M
4.44
168.2
1.97
281.8
64
7.8
C Additional Results
We further report the class generation results comparison with more metrics and baselines in Tab. 5 and Tab. 6 for ImageNet 256 256 and 512 512 generation benchmarks, respectively. Moreover, we report both results without and with classifier-free guidance [28] under column "w/o guidance" and "w/ guidance" respectively.
As shown in Tab. 5, both TiTok-L-32 and TiTok-B-64 set a new state-of-the-art performance for results without classifier-free guidance (i.e., w/o guidance column), while generating images much faster pace. Specifically, TiTok-L-32 achieves 3. 15 gFID, surpassing current state-of-the-at MAGVIT-v2 [72]'s gFID 3.65, while requiring much fewer sampling steps (8 vs. 64) and smaller model size (177M vs. M)), leading to a substantial sampling speed-up (92.4x faster, 101.6 vs. 1.1 samples/sec). Additionally, when compared MaskGIT [9], which uses the exact same generator model (i.e., MaskGIT- ViT) ours and the only difference is the toeknizer, TYTok-L-32 achieves significantly better performance (3.15 ys. 6.18). The improvement demonstrates the efficiency and effectiveness of the learned compact 1D latent space for image representation. When it comes to resolution 512 x 512 in Tab. 6, MaskGIT [9] requires 1024 tokens for image latent representation, while TiTok-L-64 requires 16x fewer. As result, when using the same generator (i.e., MaskGIT- VIT), TiTok-L-64, w/o classifier-free guidance, not only significantly outperforms MaskGIT terms of gFID (3.64 vs. 7.32) but also generates samples much faster. The advantages of TiTok become
16

---

Table 6: ImageNet-IK: 512 x 512 generation results evaluated with ADM [16]. 4: Trained on Openlmages [36] $: Trained on OpenImages, SAlON-Aestheticee Humanns [59]. P: generator's
parameters. sampling steps. throughput as samples per seconds on A100 with float32 precision, measured with w/ guidance variants if available. "guidance" refers to classifier-free guidance.
w/o guidance w/ guidance gFID 1St gFIDl 1St
tokenizer
rFID
generator
P4
St
Tt
diffusion-based generative models
UVIT-L/4 [4]
18.03
76.9
4.67
213.3 287M
50
1.0
VAE [60]#
0.19
UVIT-H/4 [4]
15.71
101.3
4.05
263.8
501M
50
0.6
DIT-XL/2 [52]
12.03 105.3 3,04
240.8
675M
250
0.1
transformer bassd generative models
MaSAGIT-VQGAN []
1.97
MaskGIT-ViT [9]
7.32
156.0
177M
3.9
4.61
192.4
307M
12
3.5
LFQ [72]
1.22
MAGVIT-22 [72]
3.07
213.1
1.91
324.3
307M
64
1.0
TiTok-L-64
1.78
MaskGrT-ViT [9]
3.64
179.8
2.74
221.1
177M
8 41.0
3.91
182.0
2.49
260.4
33.3
TiTok-B-128
1.37
MaskGIT-ViT [9]
177M
4.17
181.0
2.13
261.2
64
7.4
even more significant when compared to the diffusion models such as DiT-XI/22 [52] with guidance: TiTok-L-64 not only shows a superior performance (2.74 vs. 3.04), but also enjoys dramatically higher generation throughput (410x).
An interesting observation is that under w/o guidance case, TiTok-L-32 (for 256 resolution) and TYTok-L-64 (for 512 resolution) can outperform most other methods, including TYTok-S-128 and TiTok-B-122 128, yet they benefit relatively less from the classifier-fres guidance in the w/ guidance column. We note indicates that the great potential TyTok at compact latent size is still fully unleashed yet, and better udaptation of inference time improvements for ID compact tokens, such as classifier-free guidance, which was designed for methods with much more tokens and steps, could be promising future direction.
D Visualizations
We provide visualization of the generated images using TiTok in Fig. 5 and Fig. 6. Moreover, we visualize the reconstruction results under different numbers of tokens and different model sizes in Fig. 7, where we observe that the model tends to keep the high-level layout or salient objects when the latent representation size is limited. Besides, a larger model size reconstructs image with more details under a compact latent space size, demonstrating an effective way towards more compact latent space.
We also provide more uncurated visualization samples in Fig. and Fig. 9.
E
Limitations
This paper proposes a novel tokenization method designed to eliminate the fixed corresponding constraints of existing 2D tokenization methods. The 1D tokenization model is validated using the Vector Quantization (VQ) tokenizer formulation alongside a Masked Transformer generator framework. Despite the promising results, the proposed ID tokenization formulation theoretically has the potential to generalize to other tokenizer formulations (e.g.. ID-VAE), other generation frameworks (e.g., Diffusion Models), and beyond the image modality (e.g.. video). However, exploring these extensions is beyond the scope this paper due to limited computational resources, and we leave these promising directions for future research.
F Broader Impacts
Generative models have numerous applications with diverse potential social impacts. While these models significantly enhance human creativity, they can also misused for misinformation, harass- ment, and perpetuating social and cultural biases. Similar other deep learning methods, generative
17

---

TiTok-L-32
TiTok TiTok-B-64
macaw
lion
jack-o'- lantern
orange
daisy
bubble
valley
Figure 5: Visualization of generated images from TiTok variants with MaskGIT [9]. Correspond- ing ImageNet class names are shown below the images.
Figure 6: Visualization of generated images from TiTok-L-32 with MaskGTT [9] across random ImageNet classes.
models can be heavily influenced by dataset biases, leading to the reinforcement of negative social stereotypes and viewpoints. Developing unbiased models that ensure both robustness and fairness is critical area of research. However, addressing these issues is beyond the scope of this paper.
18

---

Increasing number of latent token /K=16, 32, 64, 128, 256)
Increasing model size (S, B, L)
Figure 7: Visual comparison of reconstruction results. Scaling model size enables a better image quality while using more compact latent space size. is also observed that TiTok tends keep the salient regions when latent space is limited.
Considering the potential risks, this paper is limited to class-conditional generation using a fixed, public, and controlled set of classes.
G Dataset Licenses
The datasets we used for training and/or testing TiTok described as follows.
ImageNet-1K: We train and evaluate TiTok on ImageNet- generation benchmark. This dataset spans 1000 object classes and contains 1,281,167 training images, 50,000 validation images and
19

---

Figure 8: Uncurated 256 256 TiTok-L-32 samples. Class labels (class ids) from left to right and top to down are: "macaw" (88), "volcano" (980), "coral reef" (973), "white wolf" (270).
Figure 9: Uncurated 256 256 TiTok-L-32 samples. Class labels (class ids) from left to right and top to down are: "cheeseburger" (933), "cliff" (972), "fountain" (562), "Siberian husky" (250).
100.000 test images. We use the training set for our tokenizer and generator training. The validation set is used to compute reconstruction FID for evaluating tokenizers. The generation results are evaluated with generation FID using pre-computed statistics and scripts from ADM [16]
License: http:///aaageeeettorrreeeeeeeeee
URL: wwt.ttttt///wwwttteee http://wwwwiaaaeoeeem wwwamaageee.eeeee. org/
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose novel ID tokenization method, achieving significant speed-up while still capable of generating high-quality images, as demonstrated in the abstract and introduction, supported by the Sec. and Sec. 4.2 parts.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Supplementary Material Sec. E.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specfiiaatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [NA]
Justification:
No theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Comprehensive details regarding model training and testing have been provided in Main Paper Sec. 4.1 Preliminary Experimental Setup. Sec. 4.2 Implementation Details, and Supplementary Material Sec. A Training and Testing Protocols
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The code and model available httpp:///gittubb..c////yeeeeddde 1d- tokenizer..
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.
(ogllc/guides/Coeeeee
oeiiionolllloooo icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wctpublic//aeccccccccccccccc /gwpi/gg///////eeee// /Coodeeudmmmiiic wttsSSubbissioo.. ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Comprehensive details regarding model training and testing have been provided in Main Paper Sec. 4.1 Preliminary Experimental Setup. Sec. 4.2 Implementation Details, and Supplementary Material Sec. A Training and Testing Protocols
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We did not report error bars following prior arts [19, 9, 72].
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The training costs are reported in Supplementary Material Sec. A Training and Testing Protocols.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https //teer/is....ee/eeee cctpublic//eiieccccecce wtw.eettihhisiieeennnenacece
didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
The research in the paper conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts have been discussed in Supplementary Material Sec. F Broader Impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper poses no such risks. Although our study includes image generation, we focus on class-conditional image generation with a predefined set of classes, which does not pose the same risks as those text-to-image models.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please refer Supplementary Material Sec. G Dataset Licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "CIPtHJJKQi", "reviewer": "Reviewer_YNgD", "summary": "This paper introduces TiTok, a 1D image tokenization method that can represent images using significantly fewer tokens compared to existing 2D approaches. The key contributions are:\n1. A 1D tokenization scheme that breaks the fixed grid constraints of 2D tokenizers, allowing more flexible and compact image representations.\n2. A dual-stage training strategy using proxy codes to improve tokenizer performance.\n3. Extensive experiments demonstrating state-of-the-art performance on ImageNet generation benchmarks while using 8-64x fewer tokens and achieving 74-410x faster inference compared to leading diffusion models.", "questions": "I have raised many question in weaknesses.", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The conclusion is shocking. 32 tokens are enough to construct a picture. Although I know that the information in the picture is very redundant, this conclusion still shocks me.\n2. The effect is still good.", "weaknesses": "1. Compared with 2D token, the main advantage of 1D image token (in my opinion) is not the efficiency and training data. Although 2D token is cumbersome, it supports the expansion of 2D dimension and has many other advantages. In applications, for higher resolution images and better effects, 2D token is still the mainstream and is difficult to be shaken by 1D token. The impact of 1D token on image generation may not be its main application. The most important value of 1D image token for me lies in its possibility of combining with multimodal technology. Due to its information compression ability and two-dimensional encoding method, 2D token is difficult to combine with language model to form a multimodal language model. And the encoder of the current multimodal language model has no decoder available. 1D token may be able to support multimodal language model and allow language model to directly output images. This is very potential. But this article does not have corresponding thinking, which is a pity.\n2. Although the image formed by 32 tokens is shocking, there are still many questions to be answered. Compared with the case of more tokens, can the information encoded by each token be estimated? Can we know what specific information is forgotten in the process of token compression? Can 1D tokens encode 2D relationships? How do 1D tokens understand translation and rotation? Is 1D token similar to the concept of high-level dictionary? These important questions are not studied in depth in this article.\n3. There are too few pictures shown in this article to make further judgments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1.1: 2D v.s. 1D in image generation**\n\nWe appreciate the valuable insights and comments. We note that the \"2D advantages\" of \"higher resolution images and better effects\" mainly stem from more tokens being used. While the 1D formulation in this paper mainly aims at a compact tokenization, we note that when using the similar number (perhaps slightly fewer) tokens, it is expected to work on par, if not better, to 2D counterparts.\n\n> **W1.2: 1D tokenization for multi-modal large language model**\n\nWe appreciate the suggestion and we strongly agree that TiTok has great potential in the multi-modal large language model. However, due to limitations on computation resources (e.g., The VQ-tokenizer-based multi-modal large language model Chameleon [C] by Meta requires training on 1024 A100-80G for 1 month), we thus mainly verify the tokenizer's effectiveness on the generation tasks, following prior arts [19, 35, 64, 66, 67]. We leave the application to multi-modal models as promising future directions.\n\n> **W2.1: Measure the information encoded by tokens**\n\nWe thank the reviewer for the valuable questions. We tried our best to provide quantitative and qualitative measurements on the information carried by 1D tokens. For example, in Fig.4.c, we show the linear probing accuracy (and thus correlates with the \"semantic\" in the tokens) when the number of tokens varies. We also provide visualization on different numbers of tokens in Fig. 7 where we observe that the tokens tend to carry important/salient regions when the number of tokens is limited. Although we agree that it will be intriguing if we may measure each token individually, it may require further research efforts and we leave it for future work.\n\n> **W2.2: What specific information is forgotten? 2D relationship, translation, rotation? High-level dictionary?**\n\nIt is noteworthy that our work is a very simple and initial attempt at this promising direction. As mentioned in W2.1, it is very challenging to directly \"visualize\" what role each token is responsible for, or how 2D relationship is modeled in the tokenization, although they are interesting topics towards better explainability. Based on experiments from Fig.4 and visualization from Fig. 7, we observe that the model tends to forget the low-level, high-frequency details in the background. For a similar reason, though the tokenizer can handle certain levels of translation/rotation in the input images, it is tricky to figure out which tokens are responsible for this part. High-level dictionary may not be capable to faithfully reconstruct the original images while TiTok tokens aim at a better trade-off between compact and semantic tokenization and faithful reconstruction (Please also see our response to Reviewer mn4Y’s Q1 for detailed examples on reconstructing images from CLIP high-level features, which fail to reconstruct the image layout or details).\n\n> **W3: More figures?**\n\nWe appreciate the suggestions. We provided more visualization in the attached PDF file. As promised in the paper, we will also open-source the code & model for the community to examine.\n\n[C] Team, Chameleon. \"Chameleon: Mixed-modal early-fusion foundation models.\" arXiv preprint arXiv:2405.09818 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u6XsnyQXqx", "reviewer": "Reviewer_Pxnc", "summary": "Background: VQ-GAN with 2D grid of latent tokens and fixed downsampling factors.\nThis paper proposes to use 1D tokens instead of 2D tokens.\n\nKey ideas:\n* Redundancies: adjacent regions are similar. 2D grid explicitly couples the latents and the pixels in the same relative coordinates.\n* 1D tokens are enough for discriminative tasks (classification, etc.). Maybe they are enough for generative tasks too.\n\nArchitecture:\n* Transformer-based encoder (tokenizer) and decoder (de-tokenizer)\n\nTraining:\n* Warm-up stage: train the 1D VQ model with the proxy codes (= discrete codes generated by an off-the-shelf MaskGIT-VQGAN model)\n* Decoder fine-tuning stage: finetune the decoder with frozen encoder and quantizer", "questions": "Q1. What are the advantages of 1D compared to 2D, and what are the potential disadvantages? Please provide a brief explanation.\n\nQ2. Is TiTok compatible with diffusion models? What is the reason for choosing MaskGIT over diffusion models or VAR (Scalable Image Generation via Next-Scale Prediction)? Appendix mentions computational burden. Is MaskGIT cheaper than the others?\n\nQ3. Could you discuss the relationship between TiTok and \"Vision transformers need registers\"?\nhttps://arxiv.org/abs/2309.16588", "limitations": "Yes in the appendix sections.", "rating": 4, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "Technical advantages:\n* More freedom in designing the architecture of 1D tokenizer\n* More semantic-rich image embedding // -> definition of semantic-rich? grounding experiment?\n* Improvement in FID\n* Faster (70x ~ 410x) generation\n\nThorough analyses:\n* Scaling experiment -> 32 tokens are sufficient for reconstruction.\n* Larger tokenizer enables smaller number of latent tokens.\n* Smaller number of latent tokens have clearer semantics than larger one, regarding linear probing.", "weaknesses": "W1. There is no analysis of 1D tokens. What are the advantages of 1D tokens that match the drawbacks of 2D tokens mentioned in the introduction section? What are the effects of masking a subset of 1D tokens compared to 2D tokens? The paper emphasizes the 1D latent sequence but does not analyze whether the observed characteristics (scaling, compact latent, etc.) are due to this 1D nature. A comparison with methods that further reduce the size of 2D tokens is needed.\n\nW2. For reconstruction results, it is necessary to compare using MSE, PSNR, and SSIM. FID is not sufficient to evaluate “reconstruction.” Why do we need rFID instead of typical reconstruction metrics such as PSNR, SSIM, and LPIPS?\n\nW3. The explanation of decoder fine-tuning is too sparse. Although not a novel contribution of this paper, it plays a significant role in performance improvement as shown in the ablation study, requiring more detailed explanation beyond just the “VQGAN training recipe.”\n\n\nMisc.\n\nSentences could be easier to read.\n- e.g., L60 ... the ViT decoder (is utilized to reconstruct -> reconstructs) the input images ...\n- I recommend reading \"Style, the basics of clarity and grace\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1.1: Grounding experiment and analysis of 1D tokens, advantages of 1D tokens against 2D tokens. What if we mask a subset?**\n\nMain advantages of 1D tokens are \"semantic-meaningful\" and \"more compact\". The \"semantic-meaningful\" is grounded by experiments in Fig. 4c, FID score in Tab. 1, 2, and visualization in Fig. 7, where we show that TiTok tends to learn more semantic-meaningful representation with a limited number of tokens. For \"more compact\", as evidenced by Fig.4d and Tab. 1, 2, TiTok uses much fewer tokens and leads to a substantial generation speed-up.\n\nMasking a subset could be challenging. As there is no concept of \"pad token\" or \"placehold\" in the learned codebook (regardless of 1D or 2D), it is not doable to \"mask a subset\" and then reconstruct an image. The best we can do is to randomly replace a subset of tokens with random tokens, yet it is still challenging to provide meaningful insights or comparison between 1D and 2D tokenizers.\n\n> **W1.2: Reducing sizes with 2D tokens**\n\nWe emphasize that the scaling experiments aim at studying the model's behavior under different numbers of tokens, which is not necessarily related to 1D or 2D tokens. However, 1D token representation provides a flexible design to use arbitrary number of tokens, while 2D tokenizers often limit the token number choices among $k^2$ (where $k^2$ is one of [1024, 256, 64]), making it not suitable to study the scaling property with different number latent tokens. Besides, as shown in Tab. 3c, where the 2D variant of TiTok-B using 64 tokens ($8^2$) achieves significantly worse performance than the 1D variant of TiTok-B using the same number of tokens, indicating that 2D tokenizers are not good choices with limited number of tokens.\n\n> **W2: More evaluation metrics for reconstruction**\n\nFID and IS are the main metrics to evaluate the reconstruction in the context of tokenizer and generator [10, 35, 54, 64]. As suggested, we report additional metrics obtained using the same code-base to ensure fairness.\n\n|  | num_tokens | rFID↓ | IS↑    | PSNR↑  | SSIM↑   | MAE↓    | MSE↓    |\n|--------------|------------|------|-------|-------|--------|--------|--------|\n| MaskGIT-VQ      | 256        | 2.28 | 180.4 | 18.14 | 0.4386 | 0.0878 | 0.0188 |\n| TiTok-L-32   | 32         | 2.21 | **195.5** | 15.88 | 0.3635 | 0.1189 | 0.0300 |\n| TiTok-B-64   | 64         | **1.70** | 195.2 | 17.06 | 0.4023 | 0.1011 | 0.0234 |\n| TiTok-B-64 w/ improved training recipe | 64 | 2.43 | 179.3 | **19.01** | **0.4479** | **0.0782** | **0.0154** |\n| TiTok-S-128  | 128        | 1.71 | 177.3 | 17.73 | 0.4255 | 0.0929 | 0.0202 |\n\n\nThe slightly worse PSNR/SSIM scores of TiTok variants match our observation that although TiTok can retain most important/salient information of the image, the high-frequency/details may be ignored/made up, as compensation for using fewer tokens. However, an improved training recipe (see ***“General Questions and Concerns - Single-stage training for TiTok”***) can significantly boost all these metrics.\n\n> **W3: More details on decoder-finetuning:**\n\nSee ***“General Questions and Concerns - Reviewer Pxnc W3: Explanation of decoder fine-tuning?”***\n\n\n> **Q1: Potential advantages and disadvantages of 1D v.s. 2D**\n\n1D tokenizer allows more flexible design and more semantic-meaningful and compact tokens compared to 2D, significantly speeding up the generation process while maintaining competitive scores. It is noteworthy that one may use the same large number of tokens in 1D tokenizer, thus making it a direct alternative to existing 2D tokenizers with similar or better performance.\n\nThe main disadvantages are 1D tokenizers are far more under-explored, compared to 2D tokenizers, demanding for more research efforts to study its applications to different tasks (e.g., image editing, diffusion models, video generation, multi-modal LLM, etc.). Besides, the training paradigm of 1D tokenizer could be further refined, e.g., while we adopt the two-stage training in the paper, we currently observe promising results from an improved single-stage training recipe as suggested by Reviewer Mn4Y, which could be further improved.\n\n\n> **Q2: Other generation models (e.g., diffusion)?**\n\nAs discussed in the limitation section, we use MaskGIT generation framework as it is much more efficient compared to diffusion models or auto-regressive models, e.g., DiT-XL/2 is a 675M model, ViT-VQGAN is a 1.7B model, and the recent VAR [D] requires a 2.0B model for attaining SOTA performance, while our model with MaskGIT only requires a 177M model and achieves better or comparable performance to aforementioned counterparts.\n\nThe TiTok framework is totally compatible with the KL-VAE formulation that is widely used by diffusion models. As this paper mainly focuses on the tokenizer part, we believe the combination with diffusion models can be a promising future direction. \n\n\n> **Q3: Comparison to Vision transformers need registers?**\n\nThanks for the suggestion. We will cite the paper (denoted as ViTreg below) for a discussion in a revision. TiTok uses latent tokens as the image representation, similar to ViTreg and other prior works. (a detailed discussion is available in L102-109). However, the two works have significant differences. ViTreg uses a set of latent tokens to allow a cleaner and more interpretable attention map, as the added latent tokens can help *alleviate the artifacts/outliers in the original self-attention map*. On the other hand, TiTok explicitly uses latent tokens to encode all the information needed to reconstruct the original image, where the latent tokens can be regarded as an information bottleneck between the tokenizer and de-tokenizer. The two works have distinctly different motivations and focuses.\n\n> **W4: Writing improvement**\n\nThanks for the valuable suggestions and we will revise accordingly.\n\n[D] Tian, Keyu, et al. \"Visual autoregressive modeling: Scalable image generation via next-scale prediction.\" arXiv:2404.02905 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IGKRkp2pJy", "reviewer": "Reviewer_8XiP", "summary": "This paper introduces a transformer-based image tokenizer (ViT) designed to convert 2D images into a 1D discrete sequence, named TiTok. The authors demonstrate that an image of size 256x256 can be discretized into a compact space of only 32 discrete tokens. This new tokenizer encodes semantic information, in contrast to other local 2D tokenizers. TiTok is trained using a two-stage process: initially, proxy codes are used to avoid the complexity of VQGAN loss, followed by a fine-tuning stage. Finally, they train a MaskGIT model to generate the discrete tokens, achieving an FID score of less than 2.", "questions": "- Figure 4 shows better gFID compared to rFID when using only 32 tokens. How is this possible? How can you achieve a better FID than the actual reconstruction?\n\n- Given that the number of tokens to predict is low (<128) and the structure is 1D, why not use a decoder-only transformer with next-token prediction? This approach would eliminate the burden of the sampling strategy and maintain a manageable number of forward passes thanks to the tokenizers\n\n- I might have missed something, but how do you train TiTok with the proxy codes when the number of tokens output by the VQGAN are more than 32 and when the number of codes in the codebook are not the same?", "limitations": "/", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method not only demonstrates increased speed compared to previous approaches but also showcases improved quality, as indicated by the FID score and the visualization.\n- Thanks to the compact space, the tokenizer learns semantic features rather than local features, which suggests potential for interesting interpolation and image manipulation applications in the future.\n- The method is original and presents a novel approach to addressing the generative task using discrete tokens in a 1D sequence.", "weaknesses": "- The method relies on proxy training for the tokenizer, but the motivation for this choice is unclear. The VQGAN loss (Reco, LPIPS, and GAN) is not overly complex to train, as evidenced by the availability of many open-source implementations. Utilizing proxy tokens is cumbersome since it requires a pre-trained network with the VQGAN loss.\n\n- The paper may lack sufficient ablation studies on hyperparameters for sampling, such as the number of steps, the role of temperature, or the CFG.\n\n- Providing uncurated visualizations of few samples from one or multiple classes could have helped readers better assess the diversity generated by the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1: Why proxy code (two-stage training) instead of VQGAN loss (recon + LPIPS + GAN)?**\n\nIt is noteworthy that the two-stage training is ***not*** necessary, as is demonstrated in ***\"General Questions and Concerns-Single-stage training for TiTok\"***. We also show that TiTok can work well with the commonly used Taming-VQGAN recipe as shown in Tab.3c. However, as is discussed in L345 - 352, there is a performance gap between Taming-VQGAN recipe and MaskGIT-VQGAN recipe, where the latter one has a significantly better performance but provides no public reference or access. We adopt the two-stage training mainly for bridging the gap between the Taming-VQGAN recipe and other state-of-the-art tokenizers.\n\n\n> **W2: Ablation studies on hyperparameters for sampling**\n\nFollowing prior arts [10, 67], we did a grid search (with step 0.5) on the sampling hyper-parameters and reported the optimal ones. We add a further ablation study based on TiTok-L-32 as suggested below (the final setting is labeled in bold, each grid number is organized as IS/FID). The effects of using CFG or not is already provided in Tab. 6 in the submission (\"w/o guidance\" refers to no CFG and \"w/ guidance\" refers to CFG).\n\n| guidance_scale \\ temperature | 8            | 8.5          | 9            | 9.5           | 10            | 10.5          | 11            |\n|------------------------------|--------------|--------------|--------------|---------------|---------------|---------------|---------------|\n| **3**                            | 197.9/2.78   | 188.8/2.77  | 178.0/2.77   | 169.9/2.84   | 161.6/2.94    | 156.1/3.10    | 150.2/3.27    |\n| **3.5**                          | 207.5/2.92   | 199.4/2.82   | 191.2/2.74   | 182.2/2.76    | 174.3/2.84    | 166.2/2.93    | 159.6/3.04    |\n| **4**                            | 217.9/3.02   | 209.8/2.89   | 200.6/2.80   | 192.7/2.77    | 184.1/2.75    | 176.1/2.80    | 170.5/2.89    |\n| **4.5**                          | 226.2/3.11   | 217.5/3.00   | 209.7/2.87   | **199.8/2.77**    | 193.7/2.77    | 184.7/2.78    | 177.8/2.80    |\n| **5**                            | 234.0/3.28  | 225.0/3.09  | 217.4/2.98   | 208.6/2.87    | 202.0/2.81    | 194.4/2.79    | 187.8/2.77    |\n| **5.5**                          | 241.9/3.42   | 233.0/3.23   | 222.4/3.04   | 215.7/2.96    | 208.7/2.88    | 200.7/2.83    | 194.7/2.79    |\n| **6**                            | 247.7/3.65   | 237.0/3.41   | 230.8/3.19   | 220.9/3.04    | 214.5/2.94    | 208.0/2.88    | 202.0/2.83    |\n\n> **W3: Uncurated visualizations**\n\nThanks for the valuable comments, we provided uncurated visualizations in the attached PDF file.\n\n> **Q1: Why is gFID better than rFID?**\n\nWe appreciate the good question. This is because rFID is computed against the real ImageNet val set, i.e., we compute the FID score between the reconstructed val set and the real ImageNet val set. However, gFID is computed against the virtual ImageNet reference statistics from OpenAI's ADM [D] (see \"Sec 2.2 Sample Quality Metrics\" in their paper). This is also why the gFID of the real val set is 1.78 instead of 0. The rFID and gFID are commonly correlated but they are not directly comparable. It is also noticeable that the evaluation protocols we follow are widely used by most prior works [4, 10, 35, 48, 54, 67].\n\n> **Q2: Why not use auto-regressive transformer for generation?**\n\nAs discussed in the limitation section, we use MaskGIT generation framework as it is much more efficient compared to diffusion models or auto-regressive models, e.g., DiT-XL/2 is a 675M model, ViT-VQGAN is a 1.7B model, and the recent VAR requires training a 2.0B model for SOTA performance, while our model using MaskGIT only trains a 177M model and achieves better or comparable performance to aforementioned counterparts.\n\n> **Q3: \"Token number mismatch\" in the proxy code training?**\n\nPlease see ***\"General Questions and Concerns - Reviewer 8XiP Q3: How is TiTok trained with Proxy codes?\"***\n\n[D] Dhariwal, et.al. \"Diffusion models beat gans on image synthesis.\" NeurIPS 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hAAzK3A59q", "reviewer": "Reviewer_Mn4Y", "summary": "This paper introduces TiTok (Transformer-based 1-Dimensional Tokenizer) that can tokenize images as compact 1D sequences instead of 2D latent grids. Accompanied with bidirectional non-autoregressive image generator, TiTok achieves SOTA performance on imagenet 256x256 benchmark with much faster generation process. The key contributions are:\n1) A 1D tokenization method that can represent images using significantly fewer tokens compared with traditional 2D approaches without comprising performance (as few as 32 tokens);\n2) Scaling up the size of tokenizer model helps to learn more compact and semantic-rich latent representations;\n3) Significant speed-up for both training and inference with competitive performance compared with 2D tokenizer.", "questions": "Another related work is Seed, which also proposes a 1D tokenizer with semantic meaningful tokens (32 tokens). It's better to cite it:\nGe, Yuying, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. \"Planting a seed of vision in large language model.\" arXiv preprint arXiv:2307.08041 (2023).", "limitations": "No negative societal impact", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The idea of using 1D representation instead of the conventional 2D is novel. It shows as few as 32 tokens are able to reconstruct an image well. The significant reduction of token numbers makes the training and inference much more efficient. Furthermore, scaling model sizes enables more compact representation, which provides insights into scaling laws for image tokenization. \n2) The experiments are comprehensive, including different model sizes and token numbers. The ablation of different design choices are well designed. The evaluations on both image reconstruction and generation show superior performance compared with MaskGIT and VQGAN. \n3) The paper is well rewritten, and easy to follow. The codes and models are publicly available.", "weaknesses": "A major concern is the two-stage training method. From Table 3 (c), the gap with and wo proxy codes is big, 1.7 vs 5.1 for rFID, and 195 vs 120 for IS. However, the proxy codes come from MaskGIT. The authors emphasized that due to the missing of a strong training recipe, the sing-stage method is lagging behind. Thus, it seems the training method is the most important one compared with architecture designs. \n\nBTW, recently there is an open-source implementation for MAGVIT2 including training codes: https://github.com/TencentARC/Open-MAGVIT2\nIt will be great if the authors can adopt their training method and make the single-stage training work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1&2: two-stage training method, Improved public available tokenizer training recipe?**\n\nPlease see ***\"General Questions and Concerns - Reviewer Mn4Y W1&2: Two-stage training and better single-stage recipe from Open-MAGVIT2?\"***\n\n> **Q1: Comparison to SEED?**\n\nWe thank the reviewer for the suggestion. SEED is already cited and discussed in the Related Work L102 to L109. Specifically, BLIP, SEED, EMU, and other similar multi-modal LLMs build a tokenizer on top of CLIP encoder, which gives highly-semantic tokens. However, due to the nature of CLIP models that focus on high-level information, they can only feed the tokens into a diffusion model and only reconstruct an image with high-level semantic similarities, while the layouts and details may not be well reconstructed. (see Fig. 3 in the SEED paper as a reference).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OsHc9Lronm", "reviewer": "Reviewer_HiP7", "summary": "This paper proposes a novel way to tokenize images to benefit image reconstruction and generation. This paper argues that the convention of compressing images into 2D latent spaces in the VQVAE/VQGAN setting limits the VQ model’s ability to fully exploit the redundancies present in images.\n\nDuring encoding, a fixed number of latent tokens are concatenated with the image patches. A vision transformer (ViT) is used to extract the representation from image patches to the latent representation. During decoding, the latent tokens are concatenated with a set of masked tokens. Another ViT is used to reconstruct image patches from the latent representation. However, instead of training encoder and decoder end2end, this paper first trains the model with the discrete codes generated by an off-the-shelf MaskGIT-VQGAN model, and then only fine-tunes the decoder with RGB pixels. During image generation, MaskGIT is used to generate the latent tokens.\n\nThe model is tested on 256x256 ImageNet for image reconstruction, generation and classification.", "questions": "How important is the off-the-shelf codebook? If the MaskGIT-VQGAN codebook is replaced by some other codebook, let's say FSQ, how would the performance look like? Can we use a continuous autoencoder instead of VQGAN as the outside compressor?", "limitations": "The limitation and potential negative societal impact have been discussed in the appendix.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed method is simple yet effective.\n2. Scaling up the tokenizer seems able to achieve even more compact image representations.\n3. The experiments on image classification is very interesting. As the size of the latent representation decreases, the tokenizer increasingly learns semantically rich representations.", "weaknesses": "1. More efforts should be put in writing Two-Stage Training. It is not clear how exactly the \"warmup\" works. Does it train the whole encoder-decoder but with frozen codebook used in VQGAN? Or it means the model is trained by the quantized value of VQGAN instead of by the RGB pixels, i.e., the input and output of the model are the quantized value of VQGAN?\n2. In the ablation test of 2D variant of TiTok-B64, how many tokens are used in this 2D representation? What if we use the same number of tokens for 1D and 2D variants? Will the performance be similar?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1: More details on two-stage training?**\n\nPlease see ***\"General Questions and Concerns - Details of two-stage training.\"*** At the warm-up stage, the model's input is still RGB images, and output is the proxy codes, with cross-entropy loss for supervision.\n\n> **W2: Ablation test of 2D variant of TiTok-B64?**\n\nWe thank the reviewer for the question. Both the TiTok-B-64 (2D) and TiTok-B-64 (1D) use the same number (64) of tokens, except that 2D variant kept the patch tokens as latent grid representation and directly reconstructed images from these 2D tokens (similar to a common 2D tokenizer), whereas 1D variant uses the latent tokens to guide the reconstruction from the mask token sequences. This experiment demonstrates that 1D formulation shows superior performance to 2D counterparts, especially when the number of tokens is limited.\n\n> **Q1: How important is the off-the-shelf codebook? Continuous autoencoder instead of VQGAN as the outside compressor?**\n\nPlease see ***\"General Questions and Concerns - Reviewer HiP7 Q1: How important is the off-the-shelf codebook? Can we use a continuous autoencoder?\"***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Questions and Concerns

We thank all reviewers for the initial positive scores and acknowledgements. We address the shared concerns below and upload additional visualization in the PDF attachment.

> **Details of two-stage training:**

To begin with, we describe the two-stage training in detail as follows:

1. In the first stage (warm-up stage), we use an off-the-shelf ImageNet-pretrained MaskGIT-VQ tokenizer to tokenize the input image into 256 tokens, which we refer to as proxy codes.

2. In the first stage training, instead of regressing the original RGB values, we use the proxy codes as reconstruction targets. Specifically, the workflow is: RGB images are patchified and flattened into a sequence and concatenated with 32 latent tokens, then they are fed into TiTok-Enc (Encoder of TiTok). Later, the latent tokens are kept as token representation and go through the quantizer. The quantized latent tokens are concatenated with 256 mask tokens and go through the TiTok-Dec (Decoder of TiTok). And the final output mask tokens are ***supervised by proxy codes using cross-entropy loss***.

3. After 2. is finished, we freeze both the TiTok-Enc and quantizer, and then only fine-tune the TiTok-Dec (responsible for reconstructing proxy codes) and MaskGIT-Dec (responsible for reconstructing RGB values from proxy codes) end-to-end towards pixel space, where the training losses include L2 loss, perceptual loss, and GAN loss following the common VQGAN paradigm.

> **Single-stage training for TiTok:**

As shown in Tab. 3c and discussed in L345-352. Two-stage training is ***not*** necessary for TiTok training, and it works fine with the commonly used and publicly available Taming-VQGAN recipe. In this case, the whole workflow is pretty straightforward, where the TiTok-Dec will instead directly reconstruct the images at pixel space.

However, the Taming-VQGAN recipe (developed more than 3 years ago) leads to an inferior FID score when compared to state-of-the-art tokenizers, putting TiTok at disadvantage when compared against other methods. Therefore we propose the two-stage training to benefit TiTok from the state-of-art MaskGIT-VQGAN tokenizer, which shares a similar architecture to Taming-VQGAN but has a significantly better score (rFID 2.28 v.s. 7.94)

We appreciate the reference to Open-MAGVIT2 per Reviewer Mn4Y's suggestion and some other recently open-sourced tokenizer training code bases such as LlamaGen (note that both are made publicly available after this submission). With these improved training recipes, we obtain much better score under the single-stage training, as shown below:

| Model                                          | rFID | IS    |
|------------------------------------------------|------|-------|
| Taming-VQGAN (256 tokens)                      | 7.94 | -     |
| MaskGIT-VQ (256 tokens)                        | 2.28 | 180.4 |
| TiTok-B-64 (single-stage w/ Taming-VQGAN recipe)| 5.15 | 120.5 |
| TiTok-B-64 (two-stages recipe)                 | 1.70 | 195.2 |
| TiTok-B-64 (***single-stage w/ improved recipe***) | 2.43 | 179.3 |

While there still exists some minor gap to the two-stage recipe, we believe the gap would be bridged further with better hyper-parameter tuning, etc., which we do not have time or resources to fully explore during the short rebuttal period.


> **Reviewer Pxnc W3: Explanation of decoder fine-tuning?**

We appreciate the question and please see the detailed two-stage workflow above in ***Details of two-stage training***.


> **Reviewer 8XiP Q3: How is TiTok trained with Proxy codes?**

We appreciate the question and please see the detailed two-stage workflow above in ***Details of two-stage training***. As proxy codes are used as reconstruction targets and we use the mask token sequence to formulate the de-tokenization process, the number of proxy codes and number of latent tokens do not matter in this training stage.

> **Reviewer Mn4Y W1&2: Two-stage training and better single-stage recipe from Open-MAGVIT2?**

We appreciate the feedback and reference to Open-MAGVIT2. Please see the table above  in ***Single-stage training for TiTok*** for single-stage TiTok with the updated training recipe.

> **Reviewer HiP7 Q1: How important is the off-the-shelf codebook? Can we use a continuous autoencoder?**

We thank the reviewer for the valuable questions. The role of off-the-shelf tokenizer is important for state-of-the-art performance but not for a usable tokenizer as demonstrated in ***Single-stage training for TiTok***. To the best of our knowledge, MaskGIT-VQ offers both a strong performance tokenizer and an open-source checkpoint (no training recipe though), which is why we use MaskGIT-VQ as the off-the-shelf tokenizer for two-stage training. Unfortunately, FSQ has not released their weight, making it challenging for us to experiment with it. In the table above in ***Single-stage training for TiTok***, we demonstrate that the off-the-shelf tokenizer can be removed while maintaining a similar performance, if we have a similar training recipe used by state-of-the-art tokenizers. We hope it can address the concerns regarding the off-the-shelf codebook.

Regarding using an off-the-shelf autoencoder such as KL-VAE, we believe it is totally feasible. However, regressing the continuous embedding requires more careful designs (e.g., GMM prediction head [A] or diffusion head [B]), while using VQ models simplifies the problems into classifying proxy codes. Thus we believe this is an interesting problem to be explored in the near future.

[A] Tschannen, Michael, et al. "Givt: Generative infinite-vocabulary transformers." arXiv preprint arXiv:2312.02116 (2023).

[B] Li, Tianhong, et al. "Autoregressive Image Generation without Vector Quantization." arXiv preprint arXiv:2406.11838 (2024).

Meta-review: The paper introduces TiTok, a 1D image tokenizer that reduces token count while preserving image quality. Reviewer YNgD acknowledges the ability to reconstruct images with just 32 tokens but notes the paper doesn't explore potential in multimodal applications and leaves questions about how 1D tokens capture 2D relationships unanswered. Reviewer Pxnc raises concerns about insufficient analysis of 1D tokens' advantages over 2D tokens and the lack of standard reconstruction metrics like PSNR and SSIM. Reviewer 8XiP likes the method's originality and efficiency but points out the reliance on proxy training and suggests more ablation studies. Reviewer Mn4Y values the novelty but worries that the two-stage training with proxy codes might be the key contributor rather than the architecture itself. Reviewer HiP7 finds the method simple and effective but calls for clearer explanations of the two-stage training process.

The rebuttal clarifies the two-stage training process, explaining how they use an off-the-shelf tokenizer to generate proxy codes in the first stage and fine-tune the decoder in the second. They demonstrate that single-stage training with improved recipes achieves similar performance, addressing concerns about reliance on proxy codes.

All the reviewers vote to accept the paper except Pxnc who complains the paper lacks principles. The AC feels that the novelty of the paper should interest a broad audience of NeurIPS community and warrants publication.
Decision: Accept (poster)