ImageFolder: Autoregressive Image Generation with Folded Tokens
Authors: Xiang Li
Kai Qiu
Hao Chen
Jason Kuen
Jiuxiang Gu
Bhiksha Raj
Zhe Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Semantic tokenizer
Autoregressive generation
Product quantization
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Image tokenizers are crucial for visual generative models, \eg, diffusion models (DMs) and autoregressive (AR) models, as they construct the latent representation for modeling. Increasing token length is a common approach to improve image reconstruction quality. However, tokenizers with longer token lengths are not guaranteed to achieve better generation quality. There exists a trade-off between reconstruction and generation quality regarding token length. In this paper, we investigate the impact of token length on both image reconstruction and generation and provide a flexible solution to the tradeoff. We propose \textbf{ImageFolder}, a semantic tokenizer that provides spatially aligned image tokens that can be folded during autoregressive modeling to improve both efficiency and quality. To enhance the representative capability without increasing token length, we leverage dual-branch product quantization to capture different contexts of images. Specifically, semantic regularization is introduced in one branch to encourage compacted semantic information while another branch is designed to capture pixel-level details. Extensive experiments demonstrate the superior quality of image generation and shorter token length with ImageFolder tokenizer.

Published as a conference paper at ICLR 2025
IMAGEFOLDER: AUTOREGRESSIVE IMAGE GENERA- TION WITH FOLDED TOKENS
Xiang L Kai Qiu', Hao Chen', Jason Kuen", Jiuxiang Gu², Bhiksha Raj' Zhe Lin Carnegie Mellon University'. Adobe Research', MBZUAI
Project Page: ImageFolder.gitubb.u
Discrete Tokens
Folded Tokens
90
10
21
Cross Entropy
3
PQ
D
90
Softmax
Softmax
35
77
Logits
AR Model
Input
Reconstruction
(a) Image tokenization with Product Quantization (PQ) (b) Autoregressive (AR) modeling with folded tokens
Figure I: Illustration of ImageFolder tokenizer and its corresponding autoregressive (AR) modeling with parallel prediction. (a) ImageFolder utilizes product quantization to obtain two sets spatially aligned tokens that capture distinct aspects of images. (b) With the tokens from ImageFolder, AR models can predict two tokens from one logit thus significantly shortening the sequence length and benefiting the performance.
ABSTRACT
Image tokenizers are crucial for visual generative models, e.g., diffusion models (DMs) and autoregressive (AR) models, as they construct the latent representation for modeling. Increasing token length is a common approach to improve the im- age reconstruction quality. However, tokenizers with longer token lengths are not guaranteed to achieve better generation quality. There exists trade-off between reconstruction and generation quality regarding token length. In this paper, we in- vestigate the impact of token length both image reconstruction and generation and provide flexible solution to the tradeoff. We propose ImageFolder, a seman- tie tokenizer that provides spatially aligned image tokens that can be folded during autoregressive modeling to improve both generation efficiency and quality. To en- hance the representative capability without increasing token length, we leverage dual-branch product quantization to capture different contexts of images. Specifi- cally, semantic regularization introduced in one branch to encourage compacted semantic information while another branch is designed to capture the remaining pixel-level details. Extensive experiments demonstrate the superior quality of im- age generation and shorter token length with ImageFolder tokenizer.
INTRODUCTION
Image generation (Chang et al. 2022 Dhariwal & Nichol 2021 He et al. 2024 Esser et al. 202
Yu et 2024d Weber et al. (024)) has achieved notable progress empowered by diffusion models (DMs) Dhariwal & Nichol, 2021] Rombach et al. 2022b Peebles & Xie 2023b) and autoregres-
sive (AR) models/large language models (LLMs) (Vaswani et al. 2023). Different from diffusion
models that leverage continuous image representation (Li et al. 2024c Dhariwal & Nichol, 2021
Rombach et al. 20226 Peebles Xie 20236), models and L.MMs typically discretize image
"This work done when Xiang Li was intern at Adohe Research.

---

Published as a conference paper at ICLR 2025
features into tokens Yu et al. 2024b Li al. 2024d Tian al., 2024) and conduct the autore- gressive modeling in the discrete space. Therefore, the performance of AR models/LLMs is largely influenced by the properties of the image tokenizers. This motivates the community to push the edge of image tokenizers for high-quality and efficient visual generation with large language models.
VQGAN (Esser et al. 2021 is pioneer work image tokenization that leverages a vector quanti- zation (Gray] 1984) operation to map the continuous image feature to discrete tokens with a learnable codebook. Recently, several improvements from different perspectives have been made the origi-
nal VOGAN (Lee et al. 2022a; Yu et al.
2023b Mentzer et al.
2023 Zhu et al. (202a]] Iakida et al.,
2023 Huang et al., 2023: Zheng et al. 2022 Yu al. 2023a Weber et al. 2024; Yu et al., 2024a
Luo et al. 2024). For example, VAR (Tian et al.) 2024) proposes a multi-scale vector quantization
(MSVQ) which quantizes images into series of multi-scale tokens and conducts the autoregressive
modeling in scale-based manner, leading to remarkable latency reduction and reconstruction im- provement. However, in trade-off of the speed and reconstruction quality, the sequence length of the quantized tokens is much longer compared to other models which could result in a larger training cost and difficulty of incorporating in LLMs with long contexts (Jin et al. 2024 Ding et al.] 2024). SEED (Ge et al. 2023) and TYTok (Yu et al. 2024d) improve the VQGAN by injecting semantics into the quantized codewords. Since the semantic tokens can convey more compacted information about the image, they can achieve much smaller token number to represent an image compared to the VQGAN. Nevertheless, a smaller token number typically leads to information loss. SEED (Ge
al. 2023) tokenizer discards all pixel-level details and only retains semantics to achieve smaller token number. TiTok ( YYu et al. (024d) conducts concatenated tokenization with multiple quantiza- tion operations achieve higher compression rate while still suffering from detail distortion during reconstruction. Based on previous arts, we can summarize trade-off between reconstruction and generation: (1) longer token number can lead better reconstruction performance while mak- ing the sequence length too long for autoregressive generation (e.g., requiring larger model capacity (Sun et al. 2024), suffering more error propagation during AR sampling (Kaiser et al. 2018) and more training costs (Tian et al.) 20244), and (2) smaller token number can lead to inferior recon- struction performance while benefit generation. Motivated by this, we aim to explore whether is possible to retain sufficient number of tokens for high-quality reconstruction without lengthening the sequence length for generation.
While it is possible to predict multiple tokens parallelly from one logit (Alexey) 22000; |Peebles & Xie (2023)) as shown in Fig. (b). This predicting scheme ignores the dependency between tokens predicted in parallel in the AR modeling. However, as the vector quantization (Esser et al. 2021) is typically conducted on the spatial- preserved image features, each quantized token repre- sents a spatial region of the original image, leading to high spatial dependency among tokens as shown in Fig. [ To address this problem, we aim to maintain spatial de- pendency in AR modeling while constructing distinct tokens within each spatial location to obtaín token pairs with low dependency using product quantization (Jegou et al. 2010).
High Dependency
Low Dependency
Vector Quantization
Product Quantization
Figure 2: Token dependency.
In this paper, we propose ImageFolder, a semaanti image tokenizer that provides spatially aligned image tokens, which can be folded for shorter token length during autoregressive modeling. We demonstrate that even if the overall token number remains the same, the folded tokens demonstrate better generation performance compared to unfolded ones. Specifically, different from previous tokenizers (Lee et al.) 2022a Yu et al., 2023b Mentzer et al. [023] Zhu et al., [2024a] Takida al. 2023 Huang et al. 2023 Zheng al. 2022). we leverage product quantization to separately capture different information of images, i.e., semantics and pixel-level details, with two branches. Within each branch, we use the multi-scale residual quantization as our quantizer. A quantizer dropout is further proposed to force each residual layer to represent an image with different bitrates, thus compensating the unmodeled dependency in the next-scale AR modeling (Tian et al.) 2024). With the above designs, the spatially aligned tokens from ImageFolder can be folded for a parallel prediction during AR modeling as shown in Fig. (b). which leads to half of the modeling token length while achieving a better generation quality. Our contribution can be summarized in three- fold:
2

---

Published as a conference paper at ICLR 2025
We present ImageFolder, a novel image tokenizer that enables shorter token length during autoregressive modeling while not losing reconstructionngeencationn quality.
We explore the product quantization in the image tokenizer. Moreover, we introduce se- mantic regularization and quantizer dropout enhance the representation capability of the quantized image tokens.
We investigate the trade-off between the generation and reconstruction regarding token length with extensive experiments, facilitating the understanding of image generation with AR modeling.
2 RELATED WORKS
Image tokenizers. Image tokenizer has seen significant progress in multiple image-related tasks. Traditionally, autoencoders (Hinton & Salakhutdinoy 2006; Vincent al. 2008) have been used to compress images into latent spaces for downstream work such as (1) generation and (2) under-
standing. In the case of generation, VAEs (Van Den Oord et al., 2017 Razavi et al. 2019a) learn
to map images to probabilistic distributions; VQGAN (Esser et al. 202 Razavi et al. (219b) and
its subsequent variants (Lee et al. 2022a Yu et al. 2023b Mentzer et al. 2023 Zhu et al. 2202aa
Takida et al. 2023 Huang et al. 20.3 Zheng et al. 2022 Yu et al. 2023a Weber et al.] 2024 Yu et al.) 2024a Luo et al. 2024 Zhu et al. 2024b) introduce a discrete latent space for better compres-
sion for generation. On the other hand, understanding tasks, such as CLIP (Radford et al. 2021).
DINO (Oquab et al. 2023 Darcet et al.] 2023] iZhu et al.] (2024)). rely heavily on LLM Vaswani Josoviiiyy et al., (021) to tokenize images into semantic representations (Dong al. Ning ct al.). These representations are effective for tasks like classification (Dosovitskiy et al., object detection (Zhu et al. 2010), segmentation (Wang et al. 2021). and multi-modal appli- cation (Yang et al., (024). For a long time, image tokenizers have been divided between methods tailored for generation and those optimized for understanding. After the appearance of (Yu et al.) 2024c), which proved the feasibility of using LLM as tokenizer for generation, some works (Wu ctt al. (2024) are dedicated to unify the tokenizer for generation and understanding due to the finding in (Gu et al. 2023).
al.
2023
2021
Diffusion models for image generation. Diffusion models, initially introduced by Sohl-Dickstein et al. (Sohl-Dickstein et al.] 2015) as a generative process and later expanded into image genera- tion by progressively infusing fixed Gaussian noise into image
forward process. A model, such as U-Net (Ronneberger et al. 2015), is then employed to learn the reverse process, gradually denoising the noisy image recover the original data distribution. In recent years, this method has witnessed significant advancements driven by various research efforts. Nichol et al. (Nichol & Dhariwal (02T). Dhariwal et al. (Dhariwal & Nichol 202I). and Song et al. (Song et | (022) proposed various techniques to enhance the effectiveness and efficiency of diffusion models, paving the way for improved image generation capabilities. Notably, the paradigm shift towards model- ing the diffusion process in the latent space of a pre-trained image encoder as strong prior (Van
Den Oord et al. 2017; Esser et al.) (02I) rather than in raw pixel spaces (Vahdat et al.) 2021: Rom- bach et al. 2022b) has proven to be a more efficient and instrumental method for high-quality image generation. Moreover, a lot of research on the model architecture replaces or integrates the vanilla U-Net with transformer (Peebles & Xie 2023b) to further improve the capacity and efficiency of multi-model synthesis on diffusion model. Inspired by these promising advancements in diffusion models, numerous foundational models have emerged, driving innovation in both image quality and flexibility. For instance, Glide (Nichol et al. (02T) introduced a diffusion model for text-guided im- age generation, combining diffusion techniques with text encoders to control the generated content. Cogview (Ding et al. (20TT) leveraged transformer architectures alongside diffusion methods to en- hance image generation tasks. Make-A-Scemt (Gafni al.) (002)) and Imagen (Saharia et al. 2022) focused on high-fidelity image synthesis conditioned on textual inputs, showcasing the versatility of diffusion models across modalities. DALL-E (Ramesh et al. 202I) and Stable Diffusion (Rom-) bach et al.) (022a) brought diffusion models to mainstream applications, demonstrating their ability to generate high-resolution photorealistic images. Additionally, recent models such as MidJourney (MidJourney Inc. 2022) and SORA (OpenAl) 2024) have further refined the use of diffusion mod- els in creative and commercial contexts, illustrating the growing influence and adaptability of these models in range of domains. Even though the performance of image generation in diffusion models
3

---

Published as a conference paper at ICLR 2025
Encoder
Token Quantization
Decoder
KxK
Zd
Detail Quantizer
Zd
Concal.
Frozen
Leamable Tokens
21
2
Vision Transformer
90
KxK
Zs
Semantic Quantizer
2's
Tokens
3
77
D
Semantic Regularization
L.X.
DINO
Contrastive Loss
Input
Reconstruction
Figure 3: Overview of ImageFolder. ImageFolder leverages vision transformers Alexey 2020) to
encode and decode images. Given an image, two sets of learnable tokens are used to generate spatially-alignes low-resolution features from the image. After that, product quantization is used to obtain discrete image representation. A semantic regularization applied in one the quantizers to inject semantic constraints. The quantized tokens concatenated to input for the image decoder to reconstruct images.
has shown impressive results, the training cost and inference speed remain bottlenecks, motivating the exploration of more efficient approaches such as leveraging large language models (LLMs) for image generation.
Autoregressive visual generation. Autoregressive models have shown remarkable success in gen- erating high-quality images by modeling the distribution of pixels or latent codes in sequential manner. Early autoregressive models such as PixelCNN (Van den Oord al. (0166) pioneered the approach of predicting pixel values conditioned on previously generated pixels. The transformer architecture VVaswani et al., 2023), first proposed NLP, has spread rapidly to image generation (Shi et al.] 2022 Mizrahi et al. (024) because of its scalability and efficiency. MaskGIT (Chang et al. 2022) accelerated the generation by predicting tokens in parallel, while MAGE (Li et al, 2023 applied MILLM (Bao al. 2022 Peng et al.) 2022) unify the visual understanding and the generation task. Recently autoregressive models continued to show their scalability power in larger datasets and multimodal tasks (He et al., 2024); ; models like LlamaGen (Sun et 2024) adapting Llama (Tourron et al. 2023) architectures for image generation. New directions such as VAR (Tian al. 2024; Li et al.] 2024d). MAR (Li et al. 2024c) and Mamba (Li et al. 2024a) have further improved flexibility and efficiency in image synthesis. Currently, more and more unified multi- modal models like SHOW-O (Xie et al. 2024), Transfusion (Zhou et al. (024)) and Lumina-mGPT (Liu et al. 2024) continue to push the boundaries of autoregressive image generation, demonstrating scalability and efficiency diverse visual tasks.
METHOD
Preliminaries. Product quantization (PQ) (Jegou et al. 2010 Guo et al. 2024) aims to quantize
high-dimension vector to combination of several low-dimension tokens, which has shown
promising capability to capture different contexts across sub-quantizers (Baevski et 2020). Given a continuous feature 2, product quantization performs as:
((z) (z1) @ @ Qp(EP), 2p Cy.
(I)
where denotes channel-wise concatenation (cartesian product), denotes that vector quantizer 2, is associated with the codebook C, = {e,}/=n> i.e., 2, maps a feature to a codeword Ep arg mineyec, || - el12 that minimizes the distance between 2p and e, Cp- P is the product number, i.e., sub-vector number and 2p is sub-vecter from having quantization first divides the target vector into several sub-vectors and then quantizes them sepa- rately. After the quantization, the quantized vectors resemble the original vector by concatenation.
2 Product

---

Published as a conference paper at ICLR 2025
3.1 IMAGEFOLDER
Imagefolder is semantic image tokenizer that tok- enizes an image into spatially aligned semantic and detail tokens with product quantization. The Image- folder designed based on two preliminary obser- vations. (1) As preliminary experiments shown in Tab I increasing the token number generally leads to better reconstruction quality, However, generation behaves differently - longer token sequences can result in inferior outcomes.
# Tokens
10x10
12x
14x 14
IFID
4.15
3.21
2.40
gFID
5.22
5.34
6.24
Table 1:
Performance against #Tokens.
Ge (2) Semantic tokenizers, e.g., SEED ct al. 2023) and Titok (Yu et al. 2024d can encode an image into only few tokens (32 tokens). As shown in Fig. 4 even though losing pixel-level structure/details of the im- age, the reconstructed images are semanti- cally correct spatially. This motivates to leverage semantics to obtain compacted im- age representation while encoding the remaining image details with additional tokens.
Details Preserving
SEED
TiTok
Ours
Input
Figure 4; Image reconstruction.
Architecture. As shown in Fig. 3 given input image I, first patchify it into set of L L tokens where L is the patch size. After that, the image tokens are concatenated with two sets of K K learnable tokens and serve as the imput to the transformer encoder E. The same spatial positional encodings are added to the learnable tokens to inform the spatial alignment. A level embedding is additionally used convey the difference across K x tokens. Let us denote the encoded tokens corresponding to the learnable tokens as εd, To discretize them, we use detail quantizer Qa and a semantic quantizer Q, to conduct product quantization. The quantized tokens 2/2 and z are further concatenated with set of L x L learnable tokens to decode the reconstructed image with a decoder
Token quantization. As shown in Fig. 3 we leverage product quantization to quantize an image with semantics and pixel-level details using two branches separately. Specifically, semantic regu- larization applied to the semantic branch to guide the semantics and distinguish from the detail branch. Moreover, quantizer dropout strategy is used to facilitate the quantizer to learn multi-scale image representation.
(1) Quantizer dropout: We multi-scale residual quantizer (MSRQ) (Tian et al.) 2024) for each branch, leading to multi-resolution to- kens 2/, , where N denotes the num- ber of residual steps. In the i-th residual step, the MSRQ quantizer first downsamples the in- put to a smaller K; K; resolution and then quantizes it by mapping it to its closest vec- tor in the codebook C {e,})=1 as & b]
22
Z3
2
Figure 5:
Illustration quantizer dropout in multi-scale residual quantizer.
arg mine,ec || [a, b) - 11.. where [a. bj is the spatial coordinates. After the nearest-neighhor
look-up, the quantized representation é; is up-sampled back to the original resolution K x K with convolution to handle the blurring as:
2 = x conv(2, + 7) x E.
(2)
where is set to a constant scalar as 0.5. Specifically, to enhance the representation capability, notice that quantizer dropout strategy (Kumar al. 2024 Zeghidour et al., 202T) largely improve generation performance. During training, as shown in Fig. 5 we randomly drop out the last several quantizers. The quantizer dropout happens with a ratio of p. With the quantizer dropout, the final output of the MSVQ can be formulated as:
2/1 Nstart 5 N.
(3)
1m1
5

---

Published as a conference paper at ICLR 2025
Applying quantizer dropout enables residual quantizers to encode images into different bitrates de- pending on the residual steps. This will significantly smooth the next-scale autoregressive predic- tion, we will show in Fig. 7 To ensure training stability, the first Nstart = 3 quantizers with extreme low resolution are never dropped out.
(2) Product quantization: We quantize images with separate MSVQ to capture different informa- tion. Denoting the output of detail and semantic quantizer as z/a and z, respectively, we concatenate them in channel-wise manner 2/z zz to form the final quantized token. it worth noting that both quantizers share the same quantizer dropout setting for each sample. The codebooks are updated separately for each quantizer.
Semantic regularization. To impose seman- ties into the tokenized image representation, we propose semantic regularization term to the quantized token z frozen pre-trained DI- NOv2 model (Oquab et al.) (0033) is utilized to extract the semantic-rie -icc visual feature fa of the input image I. We first pool the quantized token zz to x 1. After that, a CLIP-style contrastive loss (Radford al., 2021 is used to perform visual alignment: maximizing the similarity be- tween the quantized tokens z and their corresponding DINO representation fs. while minimizing the similarity between z/ and other representations Sa from different images within one batch. To facilitate semantic learning, we initialize the image encoder & with the same DINOv2 weight as the frozen one. We notice that the tokens with quantizer dropout will lead to instability during train- ing. In this way, the contrastive loss only applied to the tokens without being applied dropout. As shown in Fig. 6 we visualize reconstructed images with zero 2/ or 2/ to investigate the infor- mation captured by different quantizers. When we zero out the semantic tokens z'g, we notice that the boundary of the reconstructed image remains aligned with the input. In contrast, zeroing out detail tokens reconstructs an image that is more similar to the input while the object identity is not maintained (like what SEED (Ge et al., 2023) and TiTok (Yu al. 2024d) perform).
w.oo semantic w./o. detail Figure 6: Visualization of zero out 2اد or z'a-
Input
We provide a more in-depth discussion here to facil- itate the understanding of adopting semantic regular- ization only in one branch. We notice a concurrent work VILÁ-U (Wu et al. 2024) also aims to inject Table 2: rFID of DINOv2 as tokenizer.
Tuning rFID
Linear
LORA
Full
>5
3.55
1.57
semantics into the quantized tokens while their design results in performance degradation for image generation. As shown in Tab we compare the rFID with pre-trained DINOv2 (Oquab al. 2023) as an image encoder. We can find that the semantic-rich features from DINOv2 are not suitable le for image reconstruction. We consider this can be because the features only contain structural semantics while discards pixel-level details. In this way, directly injecting semantics into all quantized tokens will force the image tokenizer to ignore high-frequency details. To address this issue, we only use semantic regularization in one branch and leave another branch to capture other information required to reconstruct the image.
Loss function.
The ImageFolder is trained with composite losses including reconstruction loss Crecon, vector quantization loss LvQ, adverserial loss Lods Perceptual loss Lp, and CLIP loss Cotsp:
C = Crecon + AveLve + AaaCad XpLp + helsp
(4)
Specifically, the reconstruction loss measures the T.2 distance between the reconstructed image and the ground truth; vector quantization loss encourages the encoded features and its aligned codebook vectors; adversarial loss, applied from a PatchGAN (Isola et al. 2018) discriminator trained simul- taneously, ensures that the generated images are indistinguishable from real ones; perceptual loss compares high-level feature representations from a pre-traina trained LPIPS (Zhang et al., 2018)) to cap- ture structural differences; and CLIP loss performs semantic regularization between semantic tokens and the pre-trained DINOv2 (Oquab et al.) 2023) features. A LeCam regularization (Tseng et al.) 202T) is applied to the adversarial loss to stabilize the training.

---

Published as a conference paper at ICLR 2025
Evaluation Metric
ID
Method
Length
rFID
grID
1
Multi-scale residual quantization Tian et al. 2024
680
1.92
7.52
2
Quantizer dropout
680
1.71
6.03
3
+ Smaller quantized token size K = 11
286
3.24
6,56
Product quantization & Parallel decoding
286
2.06
5.96
5
Semantic regularization on all branches
286
1.97
5.21
6
+ Semantic regularization on one branch
286
1.57
3.53
7
DINO discriminator
286
0.80
2.60
Table 3: Ablation study on the improvement of ImageFolder. We evaluate the rFID and gFID on the ImageNet validation set. DINO discriminator denotes the discriminator used in VAR's (Tian al.] tokenizer. 1 and & denote the larger the better and the smaller the better, respectively.
2024
3.2
AUTOREGRESSIVE MODELING
To demonstrate the effectiveness of the proposed ImageFolder tokenizer, we train an autoregressive (AR) model to perform image generation based on the tokens obtained from ImageFolder. We follow VAR (Tian et al. 2024) to conduct the next-scale AR modeling for a faster inference speed. Recall that, leverage product quantization achieve spatially aligned tokens. Different from regular AR modeling which predicts one token from one logit, shown in Fig. [ we predict two tokens from the same logit with separate softmax operations. During training. the two predicted tokens are separately supervised with the corresponding ground truth. During inference, the tokens are sampled parallel with a top-k-top-p sampling and then fed to the image decoder for image reconstruction. With our parallel predicting scheme, the visual tokens can be folded input the AR model, leading to half reduction of the original token length. We show that shorter token length can benefit the generation quality, though the overall token number remains the same.
Parallel decoding. We discuss the nature and differences between parallel decoding and tradi- tional AR modeling. In vanilla AR models, given two tokens and y (Fig. bution p(x, y) is modeled through the conditional dependency p(æ; y)p()), meaning æ is sampled based on y. contrast, our approach samples the joint distribution parallel, assuming
[ (b)), the joint distri-
p(x, v) p())p(y),
which imposes independence between tokens and y. This independent assumption makes most existing image tokenizers, such as (Esser et al. 2021 Tian et al. 2024), unsuitable for parallel de- coding the spatial dependency is strong. To address this, introduce product quantization, technique commonly used to quantize high-dimensional spaces into several www-dimensional sub- spaces to reduce the dependencies. Specifically, we apply semantic regularization one quantiza- tion branch while allowing the other branch to learn complementary information. This encourages that tokens from different branches capture distinct aspects of the image, resulting weaker depen- dencies compared to the traditional spatial dependency among tokens.
In addition, with the quantizer dropout, each residual step can represent images with different bi- trates. During the next-scale AR modeling, although the dependency within the current scale is not considered, the conditional dependency can still refered from the low-resolution tokens in previous AR steps.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
We test our ImageFolder tokenizer on the ImageNet 256x256 reconstruction and generation tasks.
Metrics. We utilize Fréchet Inception Distance (FID) (Heusel al. 2017), Inception Score (IS)
Salimans et al. 2016), Precision, and Recall metrics for assessing the image generation.
Implementation details. For the ImageFolder tokenizer, there is no other specification, we follow the VQGAN training recipe of LlamaGen (Sun et al. 2024). We initialize the image encoder

---

Published as a conference paper at ICLR 2025
Tokenizer
Generator
Type
Method
IFD L.P.
gF
1st
Pre
Ree
#Para
Leng.
Step
Diff.
ADM
Dhariwal Nichol
2021]
10.94
101.0
0.69
0.63
554M
1000
Diff.
LDM-4
Rombach al.
2022b
3,60
247.7
400M
250
Diff.
DiT-L/2
Peehles & Xie
2025b
0.90
5.02
167.2
0.75
0.57
458M
250
Dift
MAR-B
[r 2024cc
1.22
2.31
281.7
0.82
0.57
208M
64
NAR
MaskGIT [Chang al. (2022
2,28
6.18
182.1
0.80
0.51
227M
256
NAR
RCG (cund.) [La at. 20/44
3.49
215.5
502M
256
250
NAR
TYok-S-128
al.
20244
1.52
50.5
1.94
177M
128
64
NAR
MAGVIT-v2 Yu ct |l. b2024b
0.90
1.78
319.4
307M
256
64
AR.
VQGAN (Esser et all] [202]
7.94
18.65
80.4
0.78
0.26
227M
256
256
AR.
RQ- RO-Transformer
( [Loc al. 2022b
1.83
15.72
86.8
480M
1024
64
AR.
LlamaGen-L. (Sun cf al. 2024
2.19
2.6
3.80
248.3
0.83
0.52
343M
256
256
AR.
VAR (Tian al. 2024
0.90
11.3
3.30
274.4
0.84
0.51
310M
680
10
AR.
ImageFolder (ours)
0.80
58.0
2.60
295.0
0.75
0.63
362M
286
10
Table 4: Performance comparison on class-conditional ImageNet 256x256.
denotes the utilized
image tokenizer is trained on Openlmage (Kuznetsova et al. 2020). and & denote the larger the better and the smaller the better, respectively. L.P. denotes linear probing accuracy on the ImageNet val set. The codebook utilization rate for the Imagefolder tokenizer is 100%.
with the weight of DINOv2-base, We use cosine learning rate scheduler with warmup for I epoch and start learning rate of 3e-5, We set the quantizer drop ratio 0.1. We set Aclip 0.1, Arecon XvQ = 'p = 1 and Xad = 0.5. We set the residual quantizer scales to [1. 1, 3, 3, 4,5, 11] (in total 286 tokens). The codebook size for each tokenizer is set to 4096. The residual quantization of each branch shares the same codebook across scales. For the generator, we adopt VAR's (Tian et al.) 2024) GPT-2-based (Radford et al. 2019) architecture. We double the channel of the output head to predict two tokens in parallel.
4.2 PERFORMANCE ANALYSIS
Roadmap to improve the ImageFolder tokenizer. We first discuss the core designs of Image- Folder. (1) We begin with our baseline using multi-scale residual quantization from VAR (Tian) et al. 2024) with a latent resolution of 16 x 16 and incorporate the proposed modules step by step. (2) As shown in Tab 3 with quantizer dropout, gFID achieves significant improvement of 1.49 FID. We consider this to be due to the smoother next-scale AR modeling as quantizer dropout will force each scale to represent the images with different resolutions. (3) After that, we adjust the latent resolution to 11 x 11 which leads to performance drop for both reconstruction and generation due smaller token number. (4) To reduce the token length without decreasing the token number, we adopt product quantization with corresponding parallel decoding. This modification results in rFID of 2.06 and gFID 5.96 which outperforms the 16 x counterpart's 's gFID of 6.03. (5) We further employ semantic regularization to enrich the semantics in the latent space for a more compact representation. (6) With the analysis of Tab 2 we realize that pure semantics is not enough branch of
for image reconstruction. Therefore, we adjust the semantic regularization to only the product quantization. Significant performance improvement is observed in both reconstruction and generation. Regularization on single branch encourages different branches to capture different information about the image for better reconstruction quality and reduces the feature dependency thus leading to a more robust sampling during generation. With all components, the VAR with ImageFolder achieves 3.53 gFID with only 286 token lengths.
Image generation comparison. As shown in Tab[4] we compare ImageFolder with state-of-the-an generative models. We notice ImageFolder outperforms LlamaGen which shares similar tokenizer training scheme. Compared with VAR (with advanced tokenizer training recipe, e.g., OpenImage, longer training, and stronger discriminator), our method achieves a comparable performance. In ad- dition, the shorter token length of ImageFolder (286 length) also remarkably saves training//nferrnce costs compared to VAR (680 length) which is O(n2) to the length n. Since VAR (Tian et al. 2024) did not release their tokenizer training code and recipe, architecture and training recipe, the vanilla MSVQ (Tian et al. 2024) only leads to a rFID of 1.92 and gFID of 7.52.
not reproduce their result. With our
8

---

Published as a conference paper at ICLR 2025
1x1 1x1
2x2
3x3
3x3
4x4
5x5
6x6
8xB
11x11
w./o. Dropout
Input
w. Dropout
Figure 7: Visualization of reconstructed images from different residual quantization steps. The residual scales used for this visualization is [1, 1, 3.3.3.4.2 from the third scale 2 x 2.
5, 5, 6, 8, 11j. The quantizer dropout starts
Natart
3 5
0 0.1 0.5
2
Adt 0 0.01 0.1
FFID 20 1.57 1.69
FFDD 1.73 1.57 1.94
rFID 2.74 2.74 1.57 0.97
FIDD
1.97 1.67 1.57
(a) Start dropout.
(b) Dropout ratio.
(c) Branch number.
(d) Semantic Loss.
Table 5: Design choices for ImageFolder.
4.3
MORE ANALYSIS
Reconstruction of different residual steps. We demonstrate the visualization of reconstructed images with different residual steps in Fig. 7 Without quantizer dropout, the core information is mainly concentrated on the last several layers. In contrast, with quantizer dropout, the one residual quantizer can perform image tokenization with different bitrates based on the residual step leading progressively improved representation for the next-scale AR modeling.
Conditional image generation. Inspired by the teacher-foreing guidance proposed by Con- troIVAR (Li et al. 2024d), we teacher-force de- tail tokens a reference image to conduct zero- shot conditional image generation with an AR model trained with ImageFolder. Specifically, first encode the reference image to obtain the detail tokens. After that, we teacher-force the detail token during the AR modeling con- duct the conditional image generation et al.]
Ref. Image
Ref. Token
Gen. Image
Figure 8: Conditional image generation.
2024d). As shown in Fig. g we visualize the reference image, the reconstruction of the reference detail token and the generated image. We notice that the generated image follows the spatial shapes of the reference image while containing different visual contents, e.g., rocky mountain frosty mountain. This experiment demonstrates a promising novel usage of ImageFolder on conditional image generation.
Linear probing. We further analyze the performance of the ImageFolder tokenizer by conducting linear probing on the ImageNet val set. As shown Tab we compare the linear probing top-1 accuracy of ImageFolder with LlamaGen and VAR. We notice that ImageFolder achieves signifi- cantly superior linear probing accuracy compared to comparable methods. We notice that VAR also demonstrates decent linear probing accuracy which can be credited its advanced training with the DINO discriminator.
Inference latency. We compare the inference speed of LlamaGen, VAR, and our method, as sum- marized in Tab 6 LlamaGen exhibits slower infer- ence due to its next-token autoregressive (AR) mech- anism. Our approach achieves comparable latency to VAR, as both methods require the same number of AR steps (without CPU optimization). However, it is important to highlight that our method operates with approximately of the FLOPs compared to VAR, indicating significant computational efficiency.
Method Time (s)
LlamaGen
VAR
Ours 0.130
8.851
0.134
Table 6: Inference latency with batchsize=1 on single 100 GPU.
Design choices. We further demonstrate the impact of several design choices for our Imagefolder. As Tab 5 shown, (1) quantizer dropout can significantly improve the capacity of our tokenizer, but

---

Published as a conference paper at ICLR 2025
Figure 9: Visualization of 256 x 256 image generation task within ImageNet classes.
if we drop too many quantizers during training, it will unstabilize the training of our tokenizer and cause degradation. (2) Quantizer dropout will force the model to reconstruct the image with different bitrates while too-large ratio will make the model focus too much on the intermediate outputs thus impacting the reconstruction of the last layer. (3) Increasing the number of branches can further improve the reconstruction quality whereas too many branches in the product quantizer also harm the quality of our generation, e.g., 5.24 gFID for branch quantizer. (4) The use of semantic loss can facilitate the image quality for tokenizer.
Visualization. We demonstrate qualitative visualization of the generative model trained with Im- ageFolder shown in Fig. 9 The classes of generated images among the ImageNet (Deng al.) 2009) dataset with a resolution of 256 x 256.
5 CONCLUSION
Limitations. Though the effectiveness of ImageFolder has been verified with extensive experiments, the tokenizer performance can be further improved with more advanced training scheme which will be our focus in the short future.
In this paper, introduced ImageFolder, novel semantic image tokenizer designed to balance the trade-off between token length and reconstruction quality in autoregressive modeling. By folding spatially aligned tokens, ImageFolder achieves shorter token sequence while maintaining high generation and reconstruction performance. Through our innovative use of product quantization and the introduction of semantic regularization and quantizer dropout, we demonstrated significant improvements in the representation of image tokens. Our extensive experiments have validated the effectiveness of these approaches, shedding light on the relationship between token length and visual generation quality. ImageFolder offers promising direction for more efficient integration of visual generation tasks to AR models/large language models.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Dosovitskiy Alexey, An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010. 1929, 2020.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449-2222240 2020.
Hangbo Bao, Li Dong, Songhao Piao. and Furu Wei. Beit: Bert pre-training of image transformers, 2022. URL
02....106600000 08254
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer, 2022. URL htttttttt///aiitta org. www./rggg..aaeeeeee.../eeeeeeeee htsp:/bb//22000 04200
Huiwen Chang. Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuai Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- eration via masked generative transformers. arXiv preprint arXiv:2301.0077 000004 2023.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021. URL https://arxe wtt./bs//////ee.eeo..eeeeee absaab/ssaaa 05233
Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in neural information processing systems, 34: 19822-19835, 2021.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753 2024.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, Nenghai Yu, and Baining Guo. Peco: Perceptual codebook for bert pre-training of vision transformers. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 552-560, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition scale, 2021. URL https: / arwiv.iiivv...... wwt. /ag/abs /20.:.201...2222
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE conference on computer vision and pattern recogni- fion, pp. 12873-12888 2021.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. In European Conference Computer Vision, pp. 89-106. Springer, 2022.
Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting seed of vision in large www.orrgg/bb////oo 08041
language model, 2023. URL
Robert Gray. Vector quantization. IEEE Assp Magazine, 1(2):4-99, 1984.
Yuchao Gu, Xintao Wang, Yixiao Ge, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Rethinking the objectives of vector-quantized tokenizers for image synthesis, 2023. URL https: : //arxiv.. lorg/abs/2212/ /21122222222 031
11

---

Published as a conference paper at ICLR 2025
Haohan Guo, Fenglong Xie, Dongchao Yang, Hui Lu, Xixin Wu, and Helen Meng. Addressing index collapse of large-codebook speech tokenizer with dual-decoding product-quantized variational auto-encoder. arXiv preprint arXiv:2406.02940, 2024.
Wanggui He, Siming Fu, Mushui Liu, Xierui Wang, Wenyi Xiao, Fangxun Shu, Yi Wang, Lei Zhang, Zhelun Yu, Haoyuan Li, et al. Mars: Mixture of auto-regressive models for fine-grained text-to-image synthesis. arXiv preprint arXiv:2407.07614 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
Geoffrey E Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-077 2006.
Mengqi Huang, Zhendong Mao, Zhuowei Chen, and Yongdong Zhang. Towards accurate image coding: Improved autoregressive image generation with dynamic vector quantization. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22596 22605, 2023.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexci A. Efros. taage-to-imager translation with conditional adversarial networks, 2018.
/aab///llll. 07004
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on patlern analysis and machine intelligence, 33(1):117-228, 2010.
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. Lim maybe longlm: Self-extend 11m context window without tuning. arXiv preprint arXiv:2401.01332 2024.
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In International Con- ference on Machine Learning pp. 2390-2399 PMLR, 2018.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. Advances Neural Information Processing Systems, 36, 2024.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, et al. The open images dataset v5: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128: 1956-1981, 2020.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523-11523 2022a.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive im- age generation using residual quantization, 2022b.
01941
Haopeng Li, Jinyue Yang, Kexin Wang, Xuerui Qiu, Yuhong Chou, Xin Li, and Guoqi Li. Scalable autoregressive image generation with mamba. arXiv preprint arXiv:2408. 12245, 2024a.
Tianhong Li, Huiwen Chang, Shlok Kumar Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder unify representation learning and image synthesis, 2023. URL
09117]
Tianhong Li, Dina Katabi, and Kaiming He. Return of unconditional generation: A self supervied representation generation method, 2024b. htl.:/ttps.::arxxx..orggg
03701
12

---

Published as a conference paper at ICLR 2025
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image gener- ation without vector quantization, 2024c. URL /ttjttts://arxiv.. http://axxvvvooo org/ aaabs//405 100006...880 11838
Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Zhe Lin, Rita Singh, and Bhiksha Raj. Controlvar: Exploring controllable visual autoregressive modeling, arXiv preprint MxXiv244060975555 2024d.
Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal gener- ative pretraining. arXiv preprint arXiv:2408.022: 2024.
Zhuoyan Luo, Fengyuan Shi, Yixiao Ge. Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual generation. arXiv preprint arXiv:2409.044 10, 2024.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: Vq-vae made simple, 2023.
MidJoumey Inc. Midjourney. Software available from MidJourney Inc., 2022. URL https:
77www. www.middourney. com.
David Mizrahi, Roman Bachmann, Oguzhan Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4m: Massively multimodal masked modeling. Advances in Neural Information Processing Systems, 36, 2024.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, 2021. URL htttss///axxs wttpp//bpp./....o.oa.o.oo 09672
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew,
Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with 2021.
text-guided diffusion models. arXiv preprint 1rXiv:2112.1000
J Ning, C Li, Z Zhang, Z Geng, Q Dai, K He, and H Hu. All in tokens: Unifying output space of visual tasks via soft token. arxiv 2023. arXiv preprint arXiv:2301.0220 022.09....
OpenAI. Video generation models as world simulators. OpenAl Blog, 2024. URL https:// world wimulatoors
openai Setenai.com/ae
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
2023a. William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023b. URL
https://arxiv: wwtp:/ogg//aaaaaeee http://bs/222222222 09748
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image model- ing with vector-quantized visual tokenizers, 2022.
06366
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748 8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot cext-to-image generation. In International conference on machine learning, pp. 8831-8831. Pmlr, 2021.
13

---

Published as a conference paper at ICLR 2025
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019a.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2, 2019b. URL
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2022b. URL
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assiser intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings, part III 18, pp. 241. Springer, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36444 -34494 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances Neural Information Processing Systems, 29, 2016.
Jie Shi, Chenfei Wu, Jian Liang, Xiang Liu, and Nan Duan. Divae: Photorealistic images synthesis with denoising diffusion decoder, 2022. URL http:://axxi....orrrrt abs/ 226.. 00386
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics, 2015. URL https://arxiv.org/
3bs/1503.00 03585
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022. URL
www.aaiwww..aaaaaaaooooooooooaea httr//bs/20000 /ttp://xww...r//a////20...0..1.. 02202
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 3oXiv:2406.06525. 2024.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 22268 2016.
Yuhta Takida, Yukara Ikemiya, Takashi Shibuya, Kazuki Shimada. Woosung Choi, Chich-Hsin Lai, Naoki Murata, Toshimitsu Uesaka, Kengo Uchida, Wei-Hsiang Liao, et al. Hq-vae: Hierarchical discrete representation learning with variational bayes. arXiv preprint arXiv: arXiv:2401.00365. 2023.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual meoullaluuuooressiieemdeeiiin Scalable image generation via next-scale prediction, 2024. URL/nttps://axxx..ccrr/aas 24C4.02905
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anee Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL https::/p///......... wttp//iwww...g..a////////e///aee 2302. 13971
Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data, 2021. URL /ttp:v///xiiv../ http///xwwi..oo////////2/.. 03310
14

---

Published as a conference paper at ICLR 2025
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space, 2021. URL
abs/ 22062 05931
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://aaxxi
org/abs 1706. 03762
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoing Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008
Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. Max-deeplab: End- to-end panoptic segmentation with mask transformers, 2021. URL https://arxiv.orr
abs/2012.001 00759
Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang- Chieh Chen. Maskbit: Embedding-fiee free image generation via bit tokens. arXiv preprint
1XXi::4409.16211. 2024.
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04429 2024.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large- scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10371 10371-10381, 2024.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nilion, pp. 10459-10469, 2023a.
Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmamn, Boqing Gong, Ming-Hsual Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation, 2023b.
Lijun Yu. Yong Cheng, Zhiruo Wang. Vivek Kumar, Wolfgang Macherey, Yanping Huang, David Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan Yang, et al. Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms. Advances in Neural Information Processing Systems, 36, 2024a.
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander Hauptmann, Boqing Gong, Ming- Hsuan Yang, Irfan Essa, David A Ross, and Lu Jiang- Language model beats diffusion tokenizer is key to visual generation. The Twelfth International Conference on Learning Representations, 2024b.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chies Chich Chen. An image is worth 32 tokens for reconstruction and generation. arxiv: 2406.07550, 2024c.
15

---

Published as a conference paper at ICLR 2025
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chiek Chen. An image is worth 32 tokens for reconstruction and generation, 2024d. URL httpt:tt/sss:ttt.. erg/abs/ 2406. 07550
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric, 2018,
(1C01.09224
Chuanxia Zheng, Long Tung Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation, 2022. URL
09002
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 10039, 2024.
Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%. arXiv preprint arXiv:2406.11187, 2024a.
Zhu, W Su, L Lu, Li, X Wang, and J Dai. Deformable detr: Deformable transformers for end-to-end object detection. arxiv 2020. arXiv preprint arXiv:2010.041:59 2010.
Yongxin Zhu, Bocheng Li, Yifei Xin, and Linli Xu. Addressing representation collapse in vector quantized models with one linear layer. arXiv preprint arXiv:2411.02038 2024b.
Yongxin Zhu, Bocheng Li, Hang Zhang, Xin Li, Linli Xu, and Lidong Bing. Stabilize the latent space for image autoregressive modeling: unified perspective. arXiv preprint arXi::2410.12490 2024c.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
A
WHY DO WE NEED TOP-K-TOP-P SAMPLING?
Top-k and top-p sampling have proven to be effective methods for controlling randomness and diversity in the output of autoregressive (AR) models. In transformer- based image generation, top-k sampling selects tokens from the top k most likely candidates truncating the probability distribution to avoid low- probability tokens that could degrade image quality. Top- p sampling, the other hand, dynamically adjusts the sampling pool by selecting tokens whose cumulative probability exceeds predefined threshold, ensuring balance between flexibility and diversity in the generation process.
p(x
each step,
p(x)
p()
top-k top-p
top-k top-p
In our approach, we separately sample the semantic and detail tokens using top-k and top-p sampling from the same logit. Since this method assumes the independence
Figure 10: Top-k top-p sampling visual- of the two token types, we aim to prevent situations where ization one token is selected from high-probar probbbbbbyy region while the other comes from a low-probability region. Specifically, as illustrated in Fig. TO top-k and top-p sampling allows us to select tokens from regions where both semantic and detail tokens have high probabilities, ensuring consistent image quality.
B EVALUATION METRICS AND DATASET DETAILS
2017) FID measures the distance between real and generated images the feature space of an ImageNet-IK pre-trained classifier (Szegedy al., 2016), indicating the similarity and fidelity of the generated images to real images.
Fréchet Inception Distance (FID) (Heusel et al.
Inception Score (IS) (Salimans et al.) 2016). IS also measures the fidelity and diversity of generated images. It consists of two parts: the first part measures whether each image belongs confidently to a single class of an ImageNet- K pre-trained image classifier (Szegedy et al. 2016) and the second part measures how well the generated images capture diverse classes.
Precision and Recall (Kynkäänniemi al., 2019). The real and generated images are first converted to non-paraettric representations of the manifolds using k-nearest neighbors, on which the Precision and Recall can be computed. Precision is the probability that a randomly generated image from estimated generated data manifolds falls within the support of the manifolds of estimated real data distribution. Recall is the probability that a random real image falls within the support of generated data manifolds. Thus, precision measures the general quality and fidelity of the generated images, and recall measures the coverage and diversity of the generated images.
ImageNet dataset. The ImageNet dataset (Deng al. 2009 signed for use in visual object recognition research. It contains over 14 million images, which are organized into thousands of categories based the WordNet hierarchy. Each image is annotated with one or more object labels, making it valuable resource for training and evaluating machine learning models in various computer vision tasks, such as image classification, object detection. and image generation. Its training set contains approximately 1.28 million images spanning 1,000 classes. Its validation set contains 50,000 images, with 50 images per class across the same 1,000 classes.
large-scale visual database de-
C VISUALIZATIONS
We provide additional visualizations shown in Fig. 11 and Fig.
12
17

---

Published as a conference paper at ICLR 2025
w./.. Dropout
Input
w. Dropout
w./o. Dropout
Input
Dropout
w./o. Dropout
Input
w. Dropout
Figure 11: More visualization for demonstrating the effectiveness of quantizer dropout.
Ref. Image
Ref. Token
Gen. Image
Ref. Image
Token
Gen. Image
Figure 12: More visualization for zero-shot conditional image generation. During generation sam- pling, we teacher forcing all detail tokens to referenced detail tokens.
D TEACHER FORCING GUIDANCE FOR CONDITIONAL GENERATION
Classifier-free guidance has been proven be effective in AR models (Chang et al. 2023 which take the same form as diffusion models as
(T[CC,,,)) x In Con,rolVAR, model the joint distribution of the controls and images. Therefore, we leverage a different extension of the probabilities as
c(c)1,C,o.) p(I, P(1,Clc)))(c)
p(I, Clc, c))(()
p(1,Clc)
where p(I, Clc, 0e) and p (I,Cle, 0e) can be found from the output of ControlVAR. We follow pre-
vious works (Esser et al. 2021 to ignore the constant probabilities p(c). By rewriting all terms with Baysian rule, we have
log,(T|C,,,co) log
+ logp(T, Cle, (.) - log (T,)c
+ llog([, Cla,) log C) + log C* - p().
18

---

Published as a conference paper at ICLR 2025
This corresponds the image logits
x² xz7 01 0) Yate (x((C)c,) =(CC e.)
Typ =(rC(0,
(6)
Ppa =>r0/0, 0))
where "els: Ptyp. Tpin are guidance scales for controlling the generation.
E ADDITIONAL EXPERIMENTS
ID
Branchl
Branch2
rFID
gFID
2,06
5,96
DINOv2
DINOv2
1.97
5.96
3
DINOv2
1.57
3.53
DINOv2
SAM
1.47
4.05
Table Ablation study on regularization.
Regularization on more branches. We demonstrate the ablation study to investigate the perfor- mance of different encoders conduct distillation. We notice that leveraging different encoders, i.e., DINO & SAM, to conduct regularization can lead to a superior rFID. However, we notice that the gFID does improve as the rFID. Instead, it shows an inferior performance.
Length 265
Scales
rFD
gFlD
1,1.2.333445666..100
1.86
3.98
286
1,1,2,333,445,6688111
1.57
3.53
309
1,1.2.333444..6811222
1.22
3.51
Table 8: Ablation study token length.
Token length. We demonstratee an additional ablation study on token length where the patch size of the last scale increases from 10 to leading to 265 to 309 token length for autoregressive modeling. We notice that the rFID keeps decreasing when the token length increases while the gFID is saturated for token length near 286. Therefore, we utilize 286 token lengths our final setting.
Codebook size
4096
8192
16384
rFID
0.80
0.70
0.67
Table 9: Ablation study on codebook size.
Token length. conduct an ablation study on the codebook size the ImageFolder tokenizer. We noticed that large codebook size of 16384 makes great improvement to the rFID to 0.67 which is comparable to the continuous tokenizers.
Branch setting. We provide additional experiments complimentary Table 3 analyze the branch settings. We notice that in the setting of two branches with one for semantic information and one for detail information, the model achieves the best performance.
F PRODUCT QUANTIZED SPACE
Product quantization divides the original space into subspaces and then conducts quantization the subspaces. As shown in Fig. 13 we demonstrate two scenarios product quantized space with different data sample layouts.
As the product quantization leverages the cartesian product to connect the separately quantized subspaces, will ensure symmetric property of the quantized space (symmetric indicates the
19

---

Published as a conference paper at ICLR 2025
Method
Branch
rFID
gFlD
Single detail branch
3.24
6.56
Single semantic branch
2.79
5.81
Two semantic branches
1.97
5.21
Two detail branches
2.06
5.96
One semantic, one detail
2
1.57
3.53
Table 10: Comparison of different methods with the corresponding branch, rFID, and gFID values.
Codewords
Data sample
Worst case (strictly asymmetric)
case (strictly symmetric)
Figure 13: A visualization of a product quantized space with subspaces.
quantized tokens are symmetric to the axes, showing a grid pattern). In this way, if the original data sample is strictly asymmetric, considering two-subspace case (Fig. 13), will require 2 #ample codewords to perfectly encode all the samples. Differently, if the data samples are arranged symmetric manner, the required number of codewords can be largely reduced to 2 x v #sample.
In this way, potential improvement direction is find subspaces that have desired properties to be quantized. In the ImageFolder framework, this can be achieved by leveraging contrastive loss to guide the latent space in different branches.
20


Reviews:
[{"review_id": "9udrCUTsc5", "reviewer": "Reviewer_GK7c", "summary": "- This paper presents a semantic image tokenizer that provides better token representations for visual autoregressive modeling, with improved generation quality and efficiency. \n\n - The authors leverage product quantization and semantic regularization for a more compact representation. Extensive experiments demonstrate the advantage of the proposed tokenizer over previous ones.", "questions": "- To better understand the method, the authors should add more analysis about the proposed tokenizer, such as the perplexity (codebook usage) with different techniques.\n\n -  Although this work primarily experiments with autoregressive modeling, the tokenizer does not seem to be particularly designed for AR modeling. How would this tokenizer perform on other generative schemes, such as diffusion, and non-autoregressive modeling?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper targets learning more semantically compact and disentangled image token representations, which leads to better autoregressive modeling with improved performance and shorter token length. The issue of information density(token length) and the explanation of previous image tokenizers have been hindering the performance of visual autoregressive modeling.\n\n  - Compared to previous work on image tokenizers, this work investigates product quantization to separate different information about images. With the proposed muti-scale RQ, semantic regularization, and quantizer dropout, the representation of image tokens is enhanced.\n\n - Extensive experiments on autoregressive modeling and visualization demonstrate the superiority of ImageFold.", "weaknesses": "- Several important components in this work have been investigated in previous works, including adopting semantic regularization, multi-scale RQ, and parallel decoding has been explored in earlier works. \n\n - This work lacks either enough theoretical or empirical discussions about product quantization.  Additional details, such as more visualization and discussion as in Fig. 8, should be provided to validate the \"disentangled\" nature of the obtained image tokens.  (see Questions)\n\n - This work missed some important references. Although product quantization has been rarely discussed in image tokenizers. Previous works[1] on tokenizing other modalities such as audio have adopted relevant strategies. \n\n[1] Addressing Index Collapse of Large-Codebook Speech Tokenizer with Dual-Decoding Product-Quantized Variational Auto-Encoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Vr795sBKb", "reviewer": "Reviewer_w2WM", "summary": "This paper proposes a novel method to solve the problem of how to improve image quality under finite length tokens, and proposes a semantic tokenizer, which realizes good image quality under smaller length tokens through Product Quantization.", "questions": "Please solve the problem I mentioned above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The idea of this paper is novel. Through Product Quantization, token processing is carried out in smaller spaces, and good performance is achieved under fewer tokens.\n2. The logic of this paper is rigorous, the argumentation is clear, the method is fully explained, and the experimental setting is rigorous.", "weaknesses": "Although this paper has reached an acceptable level, there are still several small problems that need to be worked on.\n1. The description of performance indicators in the article is not in place. Tables 1, 2 and some later tables do not give a simple indication of superior performance represented by high or low indicators.\n2. In Formula 4, the description of several hyperparameters and their Settings is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mVzKej9uTb", "reviewer": "Reviewer_NEtD", "summary": "This paper introduces a novel image tokenizer called ImageFolder, which addresses the trade-off between image reconstruction quality and generation quality in visual generative models, such as diffusion models and autoregressive (AR) models. The key contributions of the paper can be summarized as follows: 1. The proposed tokenizer provides spatially aligned image tokens that can be folded during autoregressive modeling, resulting in a shorter token length without compromising reconstruction or generation quality. The experiments demonstrate that folded tokens outperform unfolded ones in terms of generation performance, even when the overall token count remains the same. 2. The paper explores the use of product quantization within the image tokenizer. It introduces semantic regularization and quantizer dropout to enhance the representational capability of the quantized image tokens by separately capturing semantic information and pixel-level details. 3. Extensive experiments are conducted to investigate the relationship between token length, generation quality, and reconstruction quality, providing deeper insights into the image generation process in AR modeling.", "questions": "1.The training is based on the tokenizer from LlamaGen. Is it possible to train one from scratch to better demonstrate its effectiveness?\n\n2.How is the token length of 265 derived from the residual scales [1, 1, 2, 3, 3, 4, 5, 6, 8, 11]?\n\n3.Are there other length design comparisons for token length, such as 265, 430, etc., to better validate the results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The ImageFolder tokenizer achieves better generation performance by effectively folding tokens, which enhances the efficiency of the autoregressive modeling process.\n\n2.Despite reducing token length, the proposed method does not sacrifice the quality of image reconstruction, making it a balanced solution for generative tasks.\n\n3.By leveraging product quantization and introducing mechanisms like semantic regularization and quantizer dropout, the tokenizer captures richer semantic and pixel-level information, leading to more accurate image representations.\n\n4.The paper provides extensive experimental results that validate the effectiveness of the proposed method, offering a solid foundation for its claims and demonstrating its superiority over existing tokenizers.\n\n5.The investigation into the trade-off between generation and reconstruction quality with respect to token length contributes to a better understanding of the dynamics involved in image generation, which can inform future research in the field.", "weaknesses": "1.The training is based on the tokenizer from LlamaGen. Is it possible to train one from scratch to better demonstrate its effectiveness?\n\n2.How is the token length of 265 derived from the residual scales [1, 1, 2, 3, 3, 4, 5, 6, 8, 11]?\n\n3.Are there other length design comparisons for token length, such as 265, 430, etc., to better validate the results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BSYmGmevgd", "reviewer": "Reviewer_BcrL", "summary": "The paper is well-structured, comprehensive, and easy to understand, making the proposed concepts accessible. It presents a novel approach to decoupling features using separated quantizers and manages both the separated quantizers and input features through product quantization. Overall, the paper offers valuable insights and design strategies for quantizer development, contributing to advancements in feature token quantization.", "questions": "Q1: In Table 1, does “tokens” refer to the number of input tokens or codebook tokens?\n\nQ2: The symbol “L” in “247 quantizer drop” can be confusing as it is also used for patch size in the previous context. Additionally, in Table 3, patch size is denoted as  K=1, which adds to the confusion. Could you clarify this section further? Also, please consider refining the symbol definitions.\n\nQ3: In Figure 5, should the grid size double after each downsampling? Currently, it shows 1x1 to 2x2 to 3x3—how should this be understood? Why does the grid increase with the step size? Also, why does resolution increase from left to right? Is this sequence reversed? Normally, the input has the highest resolution, which decreases as the step deepens.\n\nQ4: In line 273, is dropout necessary for a simple semantic quantizer? Is it possible to dropout only one branch during training, so that we may know which branch will be impacted much more?\n\nQ5: Has contrastive loss been considered using CLIP’s prompt or image features to create a more robust text-image pre-trained model? The paper mentions that DINO’s semantic-rich feature isn’t suitable for image generation. Would using models like CLIP potentially alleviate this issue, making a separate semantic quantizer branch unnecessary?\n\nQ6: In AR modeling, does dividing one token into two sub-tokens and computing losses for each mean we are increasing the token count without lengthening the token length? Line 408, mentions increasing the token count without enlarging the token length, yet line 340 states that a shorter token length benefits quality while keeping the token count the same. Could you clarify this part?\n\nQ7: A smaller patch size ( k=1 ) impacts performance significantly. Is this necessary, and have you tried other designs or experiments around patch size?\n\nQ8: Could you add a comparison of different tokenizer performances for better illustration, such as comparisons with different VAEs and token quantizers?\n\nQ9: In the 4-branch quantizer, how is the contrastive loss applied? Does this only worsen gFID in your results?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-structured, comprehensive, and easy to understand, making the proposed concepts accessible.\n\n2. It presents a novel approach to decoupling features using separated quantizers and manages both the separated quantizers and input features through product quantization.\n\n3. This approach effectively mitigates the performance drop observed in previous work caused by incorporating semantic features.\n\n4. The model achieves comparable performance to state-of-the-art (SOTA) methods, demonstrating the efficacy of the proposed method.\n\n5. Overall, the paper offers valuable insights and design strategies for quantizer development, contributing to advancements in feature token quantization.", "weaknesses": "1. The results section lacks a comprehensive comparison with a broader range of current state-of-the-art VAE and autoregressive (AR) token quantizers. Including such comparisons would provide stronger validation of the method’s performance.\n\n2. In the main paper, the VAR model training has not reached final results, only intermediate outcomes are presented. Showing the final results would strengthen the evaluation.\n\n3. In terms of ablation studies, the paper would benefit from additional experiments that test the model with only the semantic branch or only the detail branch under the same experimental setting. Additionally, showing results where only a single branch incorporates semantic information would provide a more complete analysis of the method’s effectiveness.\n\n4. Finally, presenting quantizer dropout—a popular regularization technique—as a primary contribution does not seem particularly solid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)