An Image is Worth 32 Tokens for Reconstruction and Generation

Authors: qihang  yu
mark  weber
xueqing  deng
xiaohui  shen
daniel  cremers
liang-chieh  chen
  bytedance
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-30 23:07:57

Abstract: Recent advancements in generative models have highlighted the crucial role of image tokenization in the efficient synthesis of high-resolution images. Tokenization, which transforms images into latent representations, reduces computational demands compared to directly processing pixels and enhances the effectiveness and efficiency of the generation process. Prior methods, such as VQGAN, typically utilize 2D latent grids with fixed downsampling factors. However, these 2D tokenizations face challenges in managing the inherent redundancies present in images, where adjacent regions frequently display similarities. To overcome this issue, we introduce **T**ransformer-based 1-D**i**mensional **Tok**enizer (TiTok), an innovative approach that tokenizes images into 1D latent sequences. TiTok provides a more compact latent representation, yielding substantially more efficient and effective representations than conventional techniques. For example, a 256 × 256 × 3 image can be reduced to just **32** discrete tokens, a significant reduction from the 256 or 1024 tokens obtained by prior methods. Despite its compact nature, TiTok achieves competitive performance to state-of-the-art approaches. Specifically, using the same generator framework, TiTok attains **1.97** gFID, outperforming MaskGIT baseline significantly by 4.21 at ImageNet 256 × 256 benchmark. The advantages of TiTok become even more significant when it comes to higher resolution. At ImageNet 512 × 512 benchmark, TiTok not only outperforms state-of-the-art diffusion model DiT-XL/2 (gFID 2.74 vs. 3.04), but also reduces the image tokens by 64×, leading to **410× faster** generation process. Our best-performing variant can significantly surpasses DiT-XL/2 (gFID **2.13** vs. 3.04) while still generating high-quality samples **74× faster**. Codes and models are available at https://github.com/bytedance/1d-tokenizer

Main paper: 
 introduction: 
 In recent years, image generation has experienced remarkable progress, driven by the significant advancements in both transformers [19, 65, 70, 10, 71, 72] and diffusion models [16, 58, 29, 52, 21] . Mirroring the trends in generative language models [51, 62] , the architecture of many contemporary image generation models incorporate a standard image tokenizer and de-tokenizer. This array of models utilizes tokenized image representations-ranging from continuous [35] to discrete vectors [57, 64, 19] -to perform a critical function: translating raw pixels into a latent space. The latent space (e.g., 32 × 32) is significantly more compact than the original image space (256 × 256 × 3). It offers a compressed yet expressive representation, and thus not only facilitates efficient training and inference of generative models but also paves the way to scale up the model size.Although image tokenizers achieve great success in image generation workflows, they encounter a fundamental limitation tied to their intrinsic design. These tokenizers are based on an assumption that the latent space should retain a 2D structure, to maintain a direct mapping for locations between the latent tokens and image patches. For example, the top-left latent token directly corresponds to the top-left image patch. This restricts the tokenizer's ability to effectively leverage the redundancy inherent in images to cultivate a more compressed latent space.Taking one step back, we raise the question "is 2D structure necessary for image tokenization?" To answer the question, we draw inspiration from several image understanding tasks where model predictions are based solely on high-level information extracted from input images -such as in image classification [17] , object detection [8, 81] , segmentation [67, 34, 74, 75] , and multi-modal large language models [1, 41, 11] . These tasks do not need de-tokenizers, since the outputs typically manifest in specific structures other than images. In other words, they often format a higher-level 1D sequence as output that can still capture all task-relevant information. Prior arts, such as object queries [8, 67] or the perceiver resampler [1] , encode images into a 1D sequence of a predetermined number of tokens (e.g., 64 ). These tokens facilitate the generation of outputs like bounding boxes or captions. The success of these methods motivates us to investigate a more compact 1D sequence as image latent representation in the context of image reconstruction and generation. It is noteworthy that the synthesis of both high-level and low-level information is crucial for the generation of high-quality images, providing a challenge for extremely compact latent representations.In this work, we introduce a transformer-based framework [65, 17] designed to tokenize an image to a 1D discrete sequence, which can later be decoded back to the image space via a de-tokenizer. Specifically, we present Transformer-based 1-Dimensional Tokenizer (TiTok), consisting of a Vision Transformer (ViT) encoder, a ViT decoder, and a vector quantizer following the typical Vector-Quantized (VQ) model designs [19] . In the tokenization phase, the image is split and flattened into a series of patches, followed by concatenation with a 1D sequence of latent tokens. After the feature encoding process of ViT encoder, these latent tokens build the latent representation of the image. Subsequent to the vector quantization step [64, 19] , the ViT decoder reconstructs the input images from the masked token sequence [15, 24] .Building upon TiTok, we conduct extensive experiments to probe the dynamics of 1D image tokenization. Our investigation studies the interplay between latent space size, model size, reconstruction fidelity, and generative quality. From this exploration, several compelling insights emerge:1. Increasing the number of latent tokens representing an image consistently improves the reconstruction performance, yet the benefit becomes marginal after 128 tokens. Intriguingly, 32 tokens are sufficient for a reasonable image reconstruction.2. Scaling up the tokenizer model size significantly improves performance of both reconstruction and generation, especially when number of tokens is limited (e.g., 32 or 64), showcasing a promising pathway towards a compact image representation at latent space.3. 1D tokenization breaks the grid constraints in prior 2D image tokenizers, which not only enables each latent token to reconstruct regions beyond a fixed image grid and leads to a more flexible tokenizer design, but also learns more high-level and semantic-rich image information, especially at a compact latent space.4. 1D tokenization exhibits superior performance in generative training, with not only a significant speed-up for both training and inference but also a competitive FID score compared to a typical 2D tokenizer, while using much fewer tokens. In light of these findings, we introduce the TiTok family, encompassing models of varying model sizes and latent sizes, capable of achieving highly compact tokenization with as few as 32 tokens. We further confirm the model's efficacy in image generation through the MaskGIT [9] framework. TiTok is demonstrated to facilitate state-of-the-art performance in image generation, while requiring latent spaces that are 8× to 64× smaller, resulting in significant accelerations during both the training and inference phases. It also generates images with similar or higher quality but up to 410× faster than state-of-the-art diffusion models such as DiT [18] (Fig. 2 ). 
related work: 
 Image Tokenization. Images have been compressed since the early days of deep learning with autoencoders [27, 66] . The general design of using an encoder that compresses high-dimensional images into a low-dimensional latent representation and then using a decoder to reverse the process, has proven to be successful over the years. Variational Autoencoders (VAEs) [35] extend the paradigm by learning to map the input to a distribution. Instead of modeling a continuous distribution, VQ-VAEs [50, 56] learn a discrete representation forming a categorical distribution. VQGAN [19] further improves the training process by using adversarial training [23] . The transformer design of the autoencoder is further explored in ViT-VQGAN [69] and Efficient-VQGAN [7] . Orthogonal to this, RQ-VAE [37] and MoVQ [80] study the effect of using multiple vector quantization steps per latent embedding, while MAGVIT-v2 [72] and FSQ [48] propose a lookup-free quantization. However, all aforementioned works share the same workflow of an image always being patchwise encoded into a 2D grid latent representation. In this work, we explore an innovative 1D sequence latent representation for image reconstruction and generation.Tokenization for Image Understanding. For image understanding tasks (e.g., image classification [17] , object detection [8, 81, 78] , segmentation [67, 74, 76] , and Multi-modal Large Language Models (MLLMs) [1, 41, 77] ), it is common to use a general feature encoder instead of an autoencoder to tokenize the image. Specifically, many MLLMs [41, 43, 61, 32, 22, 11] uses a CLIP [54] encoder to tokenize the image into highly semantic tokens, which proves effective for image captioning [13] and VQA [2] . Some MLLMs also explore discrete tokens [32, 22] or "de-tokenize" the CLIP embeddings back to images through diffusion models [61, 32, 22] . However, due to the nature of CLIP models that focus on high-level information, these methods can only reconstruct an image with high-level semantic similarities (i.e., the layouts and details are not well-reconstructed due to CLIP features). Therefore, our method is significantly different from theirs, since the proposed TiTok aims to reconstruct both the high-level and low-level details of an image, same as typical VQ-VAE tokenizers [35, 57, 19] .Image Generation. Image generation methods range from sampling the VAE [35] , using GANs [23] to Diffusion Models [16, 58, 29, 52, 21, 44] and autoregressive models [63, 12, 50] . Prior studies that are most related to this work build on top of a learned VQ-VAE codebook to generate images.Autoregressive transformer [19, 69, 7, 37] , similar to decoder-only language models, model each patch in a step-by-step fashion, thus requiring as many steps as token number, e.g., 256 or 1024. Non- autoregressive (or bidirectional) transformers [80, 72, 68] , such as MaskGIT [9] , generally predict more than a single token per step and thus require significantly fewer steps to predict a complete image. Apart from that, further studies looked into improved sampling strategies [39, 40, 38] . As we focus on the tokenization stage, we apply the commonly used non-autoregressive sampling scheme of MaskGIT to generate a sequence of tokens that is later decoded into an image.3 Method 
preliminary background on vq-vae: 
 The image tokenizer plays a pivotal role in facilitating the generative models by providing a compact image representation at latent space. For the scope of our discussion, we primarily focus on the Vector-Quantized (VQ) tokenizer [64, 19] , given its broad applicability across various domains, including but not limited to image and video generation [19, 9, 58, 71] , large-scale pretraining [12, 5, 49, 3, 18] and multi-modal models [20, 73] .A typical VQ model contains three key components: an encoder Enc, a vector quantizer Quant, and a decoder Dec. Given an input image I ∈ R H×W ×3 , where H and W denote the image's height and width, the image is initially processed by the encoder Enc and converted to latent embeddingsZ 2D = Enc(I), where Z 2D ∈ R H f × Wf ×D , which downsamples the spatial dimensions by a factor of f . Subsequently, each embedding z ∈ R D is mapped (via the vector quantizer Quant) to the nearest code c i ∈ R D in a learnable codebook C ∈ R N ×D , comprising N codes. Formally, we have:Quant(z) = c i , where i = argmin j∈{1,2,...,N } ∥z -c j ∥ 2 .(EQUATIONDuring de-tokenization, the reconstructed image Î is obtained via the decoder Dec as follows:EQUATIONDespite the numerous improvements over VQ-VAE [64] (e.g., loss function [19] , model architecture [69] , and quantization/codebook strategies [80, 37, 72] ), the fundamental workflow (e.g., the 2D grid-based latent representations) has largely remained unchanged.While existing VQ models have demonstrated significant achievements, a notable limitation within the standard workflow exists: the latent representation Z 2D is often envisioned as a static 2D grid. Such a configuration inherently assumes a strict one-to-one mapping between the latent grids and the original image patches. This assumption limits the VQ model's ability to fully exploit the redundancies present in images, such as similarities among adjacent patches. Additionally, this approach constrains the flexibility in selecting the latent size, with the most prevalent configurations being f = 4, f = 8, or f = 16 [58] , resulting in 4096, 1024, or 256 tokens for an image of dimensions 256 × 256 × 3.Inspired by the success of 1D sequence representations in addressing a broad spectrum of computer vision problems [8, 1, 41] , we propose to use a 1D sequence, without the fixed correspondence between latent representation and image patches in 2D tokenization, as an efficient and effective latent representation for image reconstruction and generation.Image Reconstruction with TiTok. To initiate our exploration, we establish a novel framework named Transformer-based 1-Dimensional Tokenizer (TiTok), leveraging Vision Transformer (ViT) [17] 1 to tokenize images into 1D latent tokens and subsequently reconstruct the original images from these 1D latents. As depicted in Fig. 3 , TiTok employs a standard ViT for both the tokenization and de-tokenization processes (i.e., both the encoder Enc and decoder Dec are ViTs). During tokenization, we patchify the image into patches (with a patch embedding layer)P ∈ R H f × W f ×D(with patch size equal to the downsampling factor f and embedding dimension D) and concatenate them with K latent tokens L ∈ R K×D . They are then fed into the ViT encoder Enc. In the encoder output, we only retain the latent tokens as the image's latent representation, thereby enabling a more compact latent representation of 1D sequence Z 1D (with length K). This adjustment decouples the latent size from image's resolution and allows more flexibility in design choices. That is, we have:EQUATIONwhere ⊕ denotes concatenation, and we only retain the latent tokens from the encoder output.In the de-tokenization phase, drawing inspiration from [15, 5, 24] , we incorporate a sequence of mask tokensM ∈ R H f × Wf ×D -obtained by replicating a single mask token H f × W f times-to the quantized latent tokens Z 1D . The image is then reconstructed via the ViT decoder Dec as follows:EQUATIONwhere the latent tokens Z 1D is first vector-quantized by Quant and then concatenated with the mask tokens M before feeding to the decoder Dec.Despite its simplicity, we emphasize that the concept of compact 1D image tokenization remains underexplored in existing literature. The proposed TiTok thus serves as a foundational platform for exploring the potentials of 1D tokenization and de-tokenization for natural images. It is worth noting that although one may flatten 2D grid latents into a 1D sequence, it significantly differs from the proposed 1D tokenizer, due to the fact that the implicit 2D grid mapping constraints still persist.Image Generation with TiTok. Besides the image reconstruction task which the tokenizer is trained for, we also evaluate its effectiveness for image generation, following the typical pipeline [19, 9] . Specifically, we adopt MaskGIT [9] as our generation framework due to its simplicity and effectiveness, allowing us to train a MaskGIT model by simply replacing its VQGAN tokenizer with our TiTok. We do not make any other specific modifications to MaskGIT, but for completeness, we briefly describe its whole generation process with TiTok.The image is pre-tokenized into 1D discrete tokens. At each training step, a random ratio of the latent tokens are replaced with mask tokens. Then, a bidirectional transformer takes the masked token sequence as input, and predicts the corresponding discrete token ID of those masked tokens. The inference process consists of multiple sampling steps, where at each step the transformer's prediction for masked tokens will be sampled based on the prediction confidence, which are then used to update the masked images. In this way, the image is "progressively generated" from a sequence full of mask tokens to an image with generated tokens, which can later be de-tokenized back into pixel spaces. The MaskGIT framework shows a significant speed-up in the generation process compared to auto-regressive models. We refer readers to [9] for more details.Existing Training Strategies for VQ Models. Although most VQ models adhere to a straightforward formulation, their training process is notably sensitive, and the model's performance is heavily influenced by the adoption of more effective training paradigms. For instance, VQGAN [19] achieves a significant improvement in reconstruction FID (rFID) on the ImageNet [14] validation set, when compared to dVAE from DALL-E [55] . This enhancement is attributed to advancements in perceptual loss [33, 79] and adversarial loss [23] . Moreover, MaskGIT's modern implementation of VQGAN [9] utilizes refined training techniques without architectural improvements to boost the performance further. Notably, most of these improvements are exclusively applied during the training phase (i.e., through auxiliary losses) and significantly affect the models' efficacy. Given the complexity of the loss functions, extensive tuning of hyper-parameters involved, and, most critically, the missing of a publicly available code-base for reference or reproduction [9, 69, 72] , establishing an optimal experimental setup for the proposed TiTok presents a substantial challenge, especially when the target is a compact 1D tokenization which was rarely studied in literature.Comes to the Rescue. Although training TiTok with the typical Taming-VQGAN [19] setting is feasible, we introduce a two-stage training paradigm for an improved performance. The two-stage training strategy contains "warm-up" and "decoder fine-tuning" stages. Specifically, in the first "warm-up" stage, instead of directly regressing the RGB values and employing a variety of loss functions (as in existing methods), we propose to train 1D VQ models with the discrete codes generated by an off-the-shelf MaskGIT-VQGAN model, which we refer to as proxy codes. This approach allows us to bypass the intricate loss functions and GAN architectures, thereby concentrating our efforts on optimizing the 1D tokenization settings. Importantly, this modification does not harm the functionality of the tokenizer and quantizer within TiTok, which can still fully function for image tokeniztion and de-tokenization; the main adaptation simply involves the processing of TiTok's de-tokenizer output. Specifically, this output, comprising a set of proxy codes, is subsequently fed into the same off-the-shelf VQGAN decoder to generate the final RGB outputs. It is noteworthy that the introduction of proxy codes differs from a simple distillation [26] . As verified in our experiments, TiTok yields significantly better generation performance than MaskGIT-VQGAN.After the first training stage with proxy codes, we optionally have the second "decoder fine-tuning" stage, inspired by [10, 53] , to improve the reconstruction quality. Specifically, we keep the encoder and quantizer frozen, and only train the decoder towards pixel space with the typical VQGAN training recipe [19] . We observe that such a two-stage training strategy significantly improves the training stability and reconstructed image quality, as shown in the experiments.4 Experimental Results 
preliminary experiments of 1d tokenization: 
 Building upon TiTok, we explore a range of configurations, including the model size and the number of tokens, to identify the most efficient and effective setup for a 1D image tokenizer. These preliminary experiments serve to provide a thorough evaluation, seeking a practical configuration of TiTok.Preliminary Experimental Setup. Unless specified otherwise, we train all models with images of resolution H = 256 and W = 256, using the open-source MaskGIT-VQGAN [9] to supply proxy codes for training. The patch size for both tokenizer and de-tokenizer is established with f = 16, and the codebook C is configured to have N = 1024 entries with each entry a vector with 16 channels. For TiTok variants, we primarily investigate three model sizes-small, base, and large (i.e., TiTok-S, TiTok-B, TiTok-L)-comprising 22M , 86M , and 307M parameters for encoder and decoder, respectively. We also assess the impact of varying the number of latent tokens K from 16 to 256. We perform ablation experiments with an efficient setting (e.g., shorter training).Evaluation Protocol. Evaluation is conducted across multiple metrics to thoroughly assess the models, including both reconstruction and generation FID metrics (i.e., rFID and gFID) [25] on the ImageNet dataset. We examine training/inference throughput to offer a direct comparison of generative model's efficiency relative to different latent sizes. Furthermore, given that the 1D VQ model inherently serves as a form of compact image compression, we further investigate the semantic information retained by the model through linear probing following MAE setting [24] . For the complete details of the training and testing protocols (e.g., hyper-parameters, training costs), we refer the reader to the supplementary material Sec. A.After the setup, we now summarize the preliminary experimental findings below.An Image Can be Represented by 32 Tokens. The redundancy inherent in image representation is well-acknowledged, as evidenced by the practice of masking significant portions of images (e.g., 75% in MAE [24] ) to expedite the training process without negatively affecting performance. This strategy has been validated across a variety of computer vision tasks that rely on high-level image features [30, 42] . However, the efficacy of such approaches in the context of image reconstruction and generation-where both low-level and high-level details are crucial for creating realistic reconstructed and generated outputs-remains underexplored. Consequently, in this experiment, we aim to determine the minimum number of tokens required to reconstruct and generate high-quality images.As depicted in Fig. 4a , although model performance progressively improves with an increase in the number of latent tokens, significant enhancements are predominantly observed when K ranges from 16 to 128. Beyond this point, increasing the latent space size yields only marginal gains. Intriguingly, we find that with merely 32 latent tokens, TiTok-L achieves performance better than a 2D VQGAN model [19] using 256 tokens. This observation suggests that as few as 32 tokens may suffice as an effective image latent representation, optimizing the utilization of image redundancy.Scaling Up Tokenizer Enables More Compact Latent Size. Another intriguing observation from Fig. 4a is that larger tokenizers facilitate more compact representations. Specifically, TiTok-B with 64 latent tokens achieves performance comparable to TiTok-S with 128 latent tokens, while TiTok-L with 32 latent tokens matches the performance of TiTok-B with 64 latent tokens. This pattern indicates that with each incremental increase in TiTok size (e.g., from S to B, or from B to L), it is possible to reduce the size of the latent image representation without compromising performance. This trend underscores the potential benefits of scaling up the tokenizer to achieve even more compact image representations.Semantics Emerges with Compact Latent Space. To evaluate the learned image representation, we perform linear probing experiments on the image tokenizer, as shown in Fig. 4b . Specifically, we add a batch normalization layer [31] followed by a linear layer on top of the frozen features from TiTok encoder, with all hyper-parameters strictly following the MAE protocol [24] . We find that as the size of the latent representation decreases, the tokenizer increasingly learns semantically rich representations, as indicated by the improved linear probing accuracy. This suggests that the model learns high-level information in scenarios of constrained representation space.Compact Latent Representation Improves Generative Training. In addition to reconstruction capabilities, we assess TiTok's effectiveness and efficiency in generative downstream tasks, as illustrated in Fig. 4c and Fig. 4d . We note that variants of different tokenizer sizes yield comparable outcomes when the number of latent tokens is sufficiently large (i.e., K ≥ 128). However, within the domain of compact latent sizes (i.e., K ≤ 64), larger tokenizers notably enhance performance. Furthermore, the adaptability of 1D Based on the observations above, the proposed TiTok family effectively trades off a larger model size to a more compact latent size. In this section, we majorly focus on ImageNet generation benchmarks against prior arts, and evaluate TiTok as a tokenizer in the generative MaskGIT framework [9] .Implementation Details. We primarily investigate the following TiTok variants: TiTok-S-128 (i.e., small model with 128 tokens), TiTok-B-64 (i.e., base model with 64 tokens), and TiTok-L-32 (i.e., large model with 32 tokens), where each variant designed to halve the latent space size while scaling up the model size. For resolution 512, we double the latent size to ensure more details are kept at higher resolution, leading to TiTok-L-64 and TiTok-B-128. In the final setting for TiTok training, the codebook is configured to N = 4096, and the training duration is extended to 1M iterations (200 epochs). We also adopt the "decoder fine-tuning" stage to further enhance model performance, where the encoder and quantizer are kept frozen and the decoder is fine-tuned for 500k iterations.For the training of generative models, we utilize the MaskGIT [9] framework without any specific modifications, except for the adoption of an arccos masking schedule [6] . All other parameters are the same as previous setups, and all design improvements will be verified in the ablation studies.Main Results. We summarize the results on ImageNet-1K generation benchmark of resolution 256 × 256 and 512 × 512 in Tab. 1 and Tab. 2, respectively. 2For ImageNet 256 × 256 results in Tab. 1, TiTok can achieve a similar level of reconstruction FID (rFID) with a much smaller number of latent tokens than other VQ models. Specifically, using merely 32 tokens, TiTok-L-32 achieves a rFID of 2.21, comparable to the well trained VQGAN from MaskGIT [9] (rFID 2.28), while using 8× smaller latent representation size. Furthermore, when using the same generator framework and same sampling steps, TiTok-L-32 improves over MaskGIT by a large margin (from 6.18 to 2.77 gFID), showcasing the benefits of a more effective generator training with compact 1D tokens. When compared to other diffusion-based generative models, TiTok can also achieve a competitive performance while enjoying an over 100× speed-up during the sampling process. Specifically, TiTok-L-32 achieves a better gFID than LDM-4 [ We report the ablation studies regarding our final model designs in Tab. 3. Specifically, in Tab. 3a, we ablate the tokenizer designs on image reconstruction. We begin with our baseline TiTok-L-32 which attains 6.59 rFID. Employing a larger codebook size improves the rFID by 0.74, while further increasing the training iterations (from 100 epochs to 200 epochs) yields another 0.37 improvement of rFID. On top of that, the "decoder fine-tuning" (our stage-2 training strategy) can substantially improve the overall reconstruction performance to 2.21 rFID.In Tab. 3b, we examine the effects of different masking schedules for MaskGIT with TiTok. Interestingly, unlike the original MaskGIT setting [9] which empirically found that the cosine masking schedule significantly outperforms the other schedules, we observe that MaskGIT equipped with TiTok changes the preference to the arccos or linear schedules. Additionally, unlike [9] which reported that the root schedule performs much worse than the others, we observe that TiTok is quite robust to different masking schedules. We attribute the observations to TiTok's ability to provide a more compact and more semantic meaningful tokens compared to 2D VQGAN, as compared to the cosine masking schedule, linear and arccos schedules have a lower masking ratio in the early steps. This coincides with the observation that masking ratio is usually higher for redundant signals (e.g., 75% masking ratio in images [24] ) while relatively lower for semantic meaningful inputs (e.g., 15% masking ratio in languages [15] ).We ablate the effects of training paradigm in Tab. 3c. We begin with the training setting of Taming-VQGAN [19] , where TiTok-B-64 obtains 5.15 rFID, outperforming the original 2D Taming-VQGAN's 7.94 rFID under the same training setting. We also show the necessity of 1D tokenization by building a 2D variant of TiTok-B64, where the architecture remains the same except that image patches instead of latent tokens are used as image representation. As a result, we observe that the 2D variant suffers from a much worse performance (15.58 vs. 5.15 rFID), since the fixed correspondences in 2D tokenization limited a reasonable reconstruction under compact latent space. This result demonstrates the effectiveness of the proposed 1D tokenization, especially at a much more compact latent size. Although TiTok can achieve a reasonably well performance under straightforward single-stage training, there exists a performance gap compared to the MaskGIT-VQGAN [9] due to the missing of a strong training recipe, of which no public reference or access exists. Therefore, we adopt the two-stage training with proxy codes, which proves to be effective and can outperform the MaskGIT-VQGAN (1.70 vs. 2.28 rFID). It is noteworthy that the two-stage training is not that crucial to obtain a reasonable 1D tokenizer, and we believe that TiTok, with the simple single-stage Taming-VQGAN's training setting, could also benefit from training on a lagrer-scale dataset [36] as demonstrated in [58] and we leave it for future work due to the limited compute.Details on the Two-Stage Training. We provide more technical details on the two-stage training. Specifically:• In the first stage (warm-up stage), we use an off-the-shelf ImageNet-pretrained MaskGIT-VQ tokenizer to tokenize the input image into 256 tokens, which we refer to as proxy codes. • In the first stage training, instead of regressing the original RGB values, we use the proxy codes as reconstruction targets. Specifically, the workflow is: RGB images are patchified and flattened into a sequence and concatenated with 32 latent tokens, then they are fed into TiTok-Enc (Encoder of TiTok). Later, the latent tokens are kept as token representation and go through the quantizer. The quantized latent tokens are concatenated with 256 mask tokens and go through the TiTok-Dec (Decoder of TiTok). And the final output mask tokens are supervised by proxy codes using cross-entropy loss. • Afterwards, we freeze both the TiTok-Enc and quantizer, and then only fine-tune the TiTok-Dec (responsible for reconstructing proxy codes) and MaskGIT-Dec (responsible for reconstructing RGB values from proxy codes) end-to-end towards pixel space, where the training losses include L2 loss, perceptual loss, and GAN loss following the common VQGAN paradigm.Moreover, we also note that two-stage training is not necessary for TiTok training, and it works fine with the commonly used and publicly available Taming-VQGAN recipe as is shown in Tab. 3c . In this case, the whole workflow is pretty straightforward, where the TiTok-Dec will instead directly reconstruct the images at pixel space.However, the Taming-VQGAN recipe (developed more than 3 years ago) leads to an inferior FID score when compared to state-of-the-art tokenizers, putting TiTok at disadvantage when compared against other methods. Therefore we propose the two-stage training to benefit TiTok from the stateof-art MaskGIT-VQGAN tokenizer, which shares a similar architecture to Taming-VQGAN but has a significantly better score (rFID 2.28 v.s. 7.94).We also note that TiTok can work well with single-stage recipe, and it is promising to incorporate the recent modern VQGAN recipe from [47, 68] in our preliminary experiments. 
conclusion: 
 In this paper, we have explored a compact 1D tokenization TiTok for reconstructing and generating natural images. Unlike the existing 2D VQ models that consider the image latent space as a 2D grid, we provide a more compact formulation to tokenize an image into a 1D latent sequence. The proposed TiTok can represent an image with 8 to 64 times fewer tokens than the commonly used 2D tokenizers. Moreover, the compact 1D tokens not only significantly improve the generation model's training and inference throughput, but also achieve a competitive FID on the ImageNet benchmarks.We hope our research can shed some light in the direction towards more efficient image representation and generation models with 1D image tokenization.  
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We did not report error bars following prior arts [19, 9, 72] .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks. Although our study includes image generation, we focus on class-conditional image generation with a predefined set of classes, which does not pose the same risks as those text-to-image models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Please refer to Supplementary Material Sec. G Dataset Licenses.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We further report the class-conditional generation results comparison with more metrics and baselines in Tab. 5 and Tab. 6 for ImageNet 256 × 256 and 512 × 512 generation benchmarks, respectively. Moreover, we report both results without and with classifier-free guidance [28] under column "w/o guidance" and "w/ guidance" respectively.As shown in Tab. 5, both TiTok-L-32 and TiTok-B-64 set a new state-of-the-art performance for results without classifier-free guidance (i.e., w/o guidance column), while generating images at a much faster pace. Specifically, TiTok-L-32 achieves 3.15 gFID, surpassing current state-of-the-art MAGVIT-v2 [72] 's gFID 3.65, while requiring much fewer sampling steps (8 vs. 64) and smaller model size (177M vs. 307M ), leading to a substantial sampling speed-up (92.4× faster, 101.6 vs.1.1 samples/sec). Additionally, when compared to MaskGIT [9] , which uses the exact same generator model (i.e., MaskGIT-ViT) as ours and the only difference is the toeknizer, TiTok-L-32 achieves significantly a better performance (3.15 vs. 6.18). The improvement demonstrates the efficiency and effectiveness of the learned compact 1D latent space for image representation. When it comes to resolution 512 × 512 in Tab. 6, MaskGIT [9] requires 1024 tokens for image latent representation, while TiTok-L-64 requires 16× fewer. As a result, when using the same generator (i.e., MaskGIT-ViT), TiTok-L-64, w/o classifier-free guidance, not only significantly outperforms MaskGIT in terms of gFID (3.64 vs. 7.32) but also generates samples much faster. The advantages of TiTok become even more significant when compared to the diffusion models such as DiT-XL/2 [52] with guidance: TiTok-L-64 not only shows a superior performance (2.74 vs. 3.04), but also enjoys a dramatically higher generation throughput (410×).An interesting observation is that under w/o guidance case, TiTok-L-32 (for 256 resolution) and TiTok-L-64 (for 512 resolution) can outperform most other methods, including TiTok-S-128 and TiTok-B-128, yet they benefit relatively less from the classifier-free guidance in the w/ guidance column. We note it indicates that the great potential of TiTok at compact latent size is still not fully unleashed yet, and better adaptation of inference time improvements for 1D compact tokens, such as classifier-free guidance, which was designed for methods with much more tokens and steps, could be a promising future direction.We provide visualization of the generated images using TiTok in Fig. 5 and Fig. 6 . Moreover, we visualize the reconstruction results under different numbers of tokens and different model sizes in Fig. 7 , where we observe that the model tends to keep the high-level layout or salient objects when the latent representation size is limited. Besides, a larger model size reconstructs an image with more details under a compact latent space size, demonstrating an effective way towards a more compact latent space.We also provide more uncurated visualization samples in Fig. 8 and Fig. 9 .This paper proposes a novel 1D tokenization method designed to eliminate the fixed corresponding constraints of existing 2D tokenization methods. The 1D tokenization model is validated using the Vector Quantization (VQ) tokenizer formulation alongside a Masked Transformer generator framework. Despite the promising results, the proposed 1D tokenization formulation theoretically has the potential to generalize to other tokenizer formulations (e.g., 1D-VAE), other generation frameworks (e.g., Diffusion Models), and beyond the image modality (e.g., video). However, exploring these extensions is beyond the scope of this paper due to limited computational resources, and we leave these as promising directions for future research.Generative models have numerous applications with diverse potential social impacts. While these models significantly enhance human creativity, they can also be misused for misinformation, harassment, and perpetuating social and cultural biases. Similar to other deep learning methods, generative models can be heavily influenced by dataset biases, leading to the reinforcement of negative social stereotypes and viewpoints. Developing unbiased models that ensure both robustness and fairness is a critical area of research. However, addressing these issues is beyond the scope of this paper. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Although other Transformer-based architectures (e.g., Swin[45]) can also be used to instantiate TiTok, we adopt ViT for its simplicity and effectiveness.For fairness, we mainly consider tokenizers with vanilla VQ modules. More advanced quantization methods[72,48] may further benefit TiTok but beyond this paper's focus on 1D image tokenization. See supplementary material Sec. C for the complete table.https://github.com/openai/guided-diffusion/tree/main/evaluations 


Decision: The paper introduces TiTok, a 1D image tokenizer that reduces token count while preserving image quality. Reviewer YNgD acknowledges the ability to reconstruct images with just 32 tokens but notes the paper doesn't explore potential in multimodal applications and leaves questions about how 1D tokens capture 2D relationships unanswered. Reviewer Pxnc raises concerns about insufficient analysis of 1D tokens' advantages over 2D tokens and the lack of standard reconstruction metrics like PSNR and SSIM. Reviewer 8XiP likes the method's originality and efficiency but points out the reliance on proxy training and suggests more ablation studies. Reviewer Mn4Y values the novelty but worries that the two-stage training with proxy codes might be the key contributor rather than the architecture itself. Reviewer HiP7 finds the method simple and effective but calls for clearer explanations of the two-stage training process.

The rebuttal clarifies the two-stage training process, explaining how they use an off-the-shelf tokenizer to generate proxy codes in the first stage and fine-tune the decoder in the second. They demonstrate that single-stage training with improved recipes achieves similar performance, addressing concerns about reliance on proxy codes.

All the reviewers vote to accept the paper except Pxnc who complains the paper lacks principles. The AC feels that the novelty of the paper should interest a broad audience of NeurIPS community and warrants publication.

Reviews: review: paper_summary: paper_summary: This paper introduces TiTok, a 1D image tokenization method that can represent images using significantly fewer tokens compared to existing 2D approaches. The key contributions are:
1. A 1D tokenization scheme that breaks the fixed grid constraints of 2D tokenizers, allowing more flexible and compact image representations.
2. A dual-stage training strategy using proxy codes to improve tokenizer performance.
3. Extensive experiments demonstrating state-of-the-art performance on ImageNet generation benchmarks while using 8-64x fewer tokens and achieving 74-410x faster inference compared to leading diffusion models.
strength_weakness: strength_weakness: 1. Compared with 2D token, the main advantage of 1D image token (in my opinion) is not the efficiency and training data. Although 2D token is cumbersome, it supports the expansion of 2D dimension and has many other advantages. In applications, for higher resolution images and better effects, 2D token is still the mainstream and is difficult to be shaken by 1D token. The impact of 1D token on image generation may not be its main application. The most important value of 1D image token for me lies in its possibility of combining with multimodal technology. Due to its information compression ability and two-dimensional encoding method, 2D token is difficult to combine with language model to form a multimodal language model. And the encoder of the current multimodal language model has no decoder available. 1D token may be able to support multimodal language model and allow language model to directly output images. This is very potential. But this article does not have corresponding thinking, which is a pity.
2. Although the image formed by 32 tokens is shocking, there are still many questions to be answered. Compared with the case of more tokens, can the information encoded by each token be estimated? Can we know what specific information is forgotten in the process of token compression? Can 1D tokens encode 2D relationships? How do 1D tokens understand translation and rotation? Is 1D token similar to the concept of high-level dictionary? These important questions are not studied in depth in this article.
3. There are too few pictures shown in this article to make further judgments.
questions: questions: I have raised many question in weaknesses.
limitations: limitations: yes
paper_summary: paper_summary: Background: VQ-GAN with 2D grid of latent tokens and fixed downsampling factors.
This paper proposes to use 1D tokens instead of 2D tokens.

Key ideas:
* Redundancies: adjacent regions are similar. 2D grid explicitly couples the latents and the pixels in the same relative coordinates.
* 1D tokens are enough for discriminative tasks (classification, etc.). Maybe they are enough for generative tasks too.

Architecture:
* Transformer-based encoder (tokenizer) and decoder (de-tokenizer)

Training:
* Warm-up stage: train the 1D VQ model with the proxy codes (= discrete codes generated by an off-the-shelf MaskGIT-VQGAN model)
* Decoder fine-tuning stage: finetune the decoder with frozen encoder and quantizer
strength_weakness: strength_weakness: W1. There is no analysis of 1D tokens. What are the advantages of 1D tokens that match the drawbacks of 2D tokens mentioned in the introduction section? What are the effects of masking a subset of 1D tokens compared to 2D tokens? The paper emphasizes the 1D latent sequence but does not analyze whether the observed characteristics (scaling, compact latent, etc.) are due to this 1D nature. A comparison with methods that further reduce the size of 2D tokens is needed.

W2. For reconstruction results, it is necessary to compare using MSE, PSNR, and SSIM. FID is not sufficient to evaluate “reconstruction.” Why do we need rFID instead of typical reconstruction metrics such as PSNR, SSIM, and LPIPS?

W3. The explanation of decoder fine-tuning is too sparse. Although not a novel contribution of this paper, it plays a significant role in performance improvement as shown in the ablation study, requiring more detailed explanation beyond just the “VQGAN training recipe.”


Misc.

Sentences could be easier to read.
- e.g., L60 ... the ViT decoder (is utilized to reconstruct -> reconstructs) the input images ...
- I recommend reading "Style, the basics of clarity and grace".
questions: questions: Q1. What are the advantages of 1D compared to 2D, and what are the potential disadvantages? Please provide a brief explanation.

Q2. Is TiTok compatible with diffusion models? What is the reason for choosing MaskGIT over diffusion models or VAR (Scalable Image Generation via Next-Scale Prediction)? Appendix mentions computational burden. Is MaskGIT cheaper than the others?

Q3. Could you discuss the relationship between TiTok and "Vision transformers need registers"?
https://arxiv.org/abs/2309.16588
limitations: limitations: Yes in the appendix sections.
paper_summary: paper_summary: This paper introduces a transformer-based image tokenizer (ViT) designed to convert 2D images into a 1D discrete sequence, named TiTok. The authors demonstrate that an image of size 256x256 can be discretized into a compact space of only 32 discrete tokens. This new tokenizer encodes semantic information, in contrast to other local 2D tokenizers. TiTok is trained using a two-stage process: initially, proxy codes are used to avoid the complexity of VQGAN loss, followed by a fine-tuning stage. Finally, they train a MaskGIT model to generate the discrete tokens, achieving an FID score of less than 2.
strength_weakness: strength_weakness: - The method relies on proxy training for the tokenizer, but the motivation for this choice is unclear. The VQGAN loss (Reco, LPIPS, and GAN) is not overly complex to train, as evidenced by the availability of many open-source implementations. Utilizing proxy tokens is cumbersome since it requires a pre-trained network with the VQGAN loss.

- The paper may lack sufficient ablation studies on hyperparameters for sampling, such as the number of steps, the role of temperature, or the CFG.

- Providing uncurated visualizations of few samples from one or multiple classes could have helped readers better assess the diversity generated by the model.
questions: questions: - Figure 4 shows better gFID compared to rFID when using only 32 tokens. How is this possible? How can you achieve a better FID than the actual reconstruction?

- Given that the number of tokens to predict is low (<128) and the structure is 1D, why not use a decoder-only transformer with next-token prediction? This approach would eliminate the burden of the sampling strategy and maintain a manageable number of forward passes thanks to the tokenizers

- I might have missed something, but how do you train TiTok with the proxy codes when the number of tokens output by the VQGAN are more than 32 and when the number of codes in the codebook are not the same?
limitations: limitations: /
paper_summary: paper_summary: This paper introduces TiTok (Transformer-based 1-Dimensional Tokenizer) that can tokenize images as compact 1D sequences instead of 2D latent grids. Accompanied with bidirectional non-autoregressive image generator, TiTok achieves SOTA performance on imagenet 256x256 benchmark with much faster generation process. The key contributions are:
1) A 1D tokenization method that can represent images using significantly fewer tokens compared with traditional 2D approaches without comprising performance (as few as 32 tokens);
2) Scaling up the size of tokenizer model helps to learn more compact and semantic-rich latent representations;
3) Significant speed-up for both training and inference with competitive performance compared with 2D tokenizer.
strength_weakness: strength_weakness: A major concern is the two-stage training method. From Table 3 (c), the gap with and wo proxy codes is big, 1.7 vs 5.1 for rFID, and 195 vs 120 for IS. However, the proxy codes come from MaskGIT. The authors emphasized that due to the missing of a strong training recipe, the sing-stage method is lagging behind. Thus, it seems the training method is the most important one compared with architecture designs. 

BTW, recently there is an open-source implementation for MAGVIT2 including training codes: https://github.com/TencentARC/Open-MAGVIT2
It will be great if the authors can adopt their training method and make the single-stage training work.
questions: questions: Another related work is Seed, which also proposes a 1D tokenizer with semantic meaningful tokens (32 tokens). It's better to cite it:
Ge, Yuying, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. "Planting a seed of vision in large language model." arXiv preprint arXiv:2307.08041 (2023).
limitations: limitations: No negative societal impact
paper_summary: paper_summary: This paper proposes a novel way to tokenize images to benefit image reconstruction and generation. This paper argues that the convention of compressing images into 2D latent spaces in the VQVAE/VQGAN setting limits the VQ model’s ability to fully exploit the redundancies present in images.

During encoding, a fixed number of latent tokens are concatenated with the image patches. A vision transformer (ViT) is used to extract the representation from image patches to the latent representation. During decoding, the latent tokens are concatenated with a set of masked tokens. Another ViT is used to reconstruct image patches from the latent representation. However, instead of training encoder and decoder end2end, this paper first trains the model with the discrete codes generated by an off-the-shelf MaskGIT-VQGAN model, and then only fine-tunes the decoder with RGB pixels. During image generation, MaskGIT is used to generate the latent tokens.

The model is tested on 256x256 ImageNet for image reconstruction, generation and classification.
strength_weakness: strength_weakness: 1. More efforts should be put in writing Two-Stage Training. It is not clear how exactly the "warmup" works. Does it train the whole encoder-decoder but with frozen codebook used in VQGAN? Or it means the model is trained by the quantized value of VQGAN instead of by the RGB pixels, i.e., the input and output of the model are the quantized value of VQGAN?
2. In the ablation test of 2D variant of TiTok-B64, how many tokens are used in this 2D representation? What if we use the same number of tokens for 1D and 2D variants? Will the performance be similar?
questions: questions: How important is the off-the-shelf codebook? If the MaskGIT-VQGAN codebook is replaced by some other codebook, let's say FSQ, how would the performance look like? Can we use a continuous autoencoder instead of VQGAN as the outside compressor?
limitations: limitations: The limitation and potential negative societal impact have been discussed in the appendix.

Comments: title: Thanks for Your Review and Support
Thanks for Your Review and Support
Thanks for Your Review and Support
Post-rebuttal review
comment: I thank the authors for the rebuttal. Sorry for coming in late.

I am lowering my rating to BR because this paper is
* We did this and that. It results in this and that.
* *We do not know why. It works.*

The core argument of this paper is using 1D tokens instead of 2D tokens. However, the paper does not provide the principle behind the advantage of 1D tokens over 2D tokens. 2D tokens are capable of representing anything which can be represented by 1D tokens because 2D tokens have more operational freedom. I value the analysis showing various behavior. But, the advantage of 1D tokens is supported by only the behavior, not the principle or theory.

In short, I think using 1D tokens is worth publishing. But the content should be largely revised not to mislead the principle.

(I am open to discussion during reviewer-ac discussion period.)
Thank you so much for the valuable suggestions and for considering our responses! We will revise and refine the confusing parts in the two-stage training as discussed in the rebuttal as suggested. If you need any further information or clarification, please feel free to contact us!
Thank you very much for your insightful feedback and for considering our responses. We're glad we could address all your concerns satisfactorily. Your support in recommending our paper for acceptance is greatly appreciated. If you have any further questions, please feel free to let us know.
Thanks for providing the detailed rebuttal, especially running the single-stage training experiment by borrowing the training recipe from open-magvit2. I will raise my score.
Thanks for the clarification. Since most reviewers have the confusion of the two-stage training, apparently this part is very poorly written. I strongly suggest the authors to revise the manuscript.
Thank you sincerely for your valuable feedback and for taking our responses into account. We're pleased that we were able to address your concerns. We greatly appreciate your support in recommending our paper for acceptance. Should you have any additional questions, please feel free to let us know.
I have read the author response and the review from other reviewers. I will raise my score to "weak accept". Thanks.