Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation
Authors: Anlin Zheng
Xin Wen
Xuanyang Zhang
Chuofan Ma
Tiancai Wang
Gang YU
Xiangyu Zhang
XIAOJUAN QI
Venue: NeurIPS 2025 poster ()
Keywords: Image tokenizer; Autoregressive Image Generation; Vision Foundation Models;
Primary area: deep_learning
TL;DR: This work utilizes vision foundation models to construct a visual tokenizer, which is trained in an end-to-end manner for AR image generation, achieving state-of-the-art results on the $256\times256$ class-to-image generation task on ImageNet.

Abstract: In this work, we present a novel direction to build an image tokenizer directly on top of a frozen vision foundation model, which is a largely underexplored area. Specifically, we employ a frozen vision foundation model as the encoder of our tokenizer. To enhance its effectiveness, we introduce two key components: (1) a region-adaptive quantization framework that reduces redundancy in the pre-trained features on regular 2D grids, and (2) a semantic reconstruction objective that aligns the tokenizer’s outputs with the foundation model’s representations to preserve semantic fidelity. Based on these designs, our proposed image tokenizer, \textbf{\ours}, achieves substantial improvements in image reconstruction and generation quality, while also enhancing token efficiency. It further boosts autoregressive (AR) generation---achieving a gFID of \textbf{1.36} on ImageNet benchmarks, while accelerating model convergence by \textbf{three times}, and enabling high-fidelity class-conditional synthesis without the need for classifier-free guidance (CFG). The code is available at \href{https://github.com/CVMI-Lab/VFMTok}{https://github.com/CVMI-Lab/VFMTok}.

Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation
Anlin Zheng'
Xin Wen
Xuanyang Zhang
Chuofan Ma
Tiancai Wang
Gang Yu²
Xiangyu Zhang
Xiaojuan Qi
'The University of Hong Kong
StepFuun
Dexmal
MEGVII Technology
Abstract
In this work, we present novel direction to build an image tokenizer directly on top of frozen vision foundation model, which is largely underexplored area. Specifically, we employ a frozen vision foundation model the encoder of our tokenizer. To enhance its effectiveness, we introduce two key components: (1) a region-adaptive quantization framework that reduces redundancy in the pre-trained features regular 2D grids, and (2) semantic reconstruction objective that aligns the tokenizer's outputs with the foundation model's representations preserve semantic fidelity. Based on these designs, our proposed image tokenizer, VFMTok, achieves substantial improvements in image reconstruction and generation qual- ity, while also enhancing token efficiency. It further boosts autoregressive (AR) generation-achieviin a gFID of 1.36 on ImageNet benchmarks, while accelerating model convergence by three times, and enabling high-fidelity conditionn synthesis without the need for classifier free guidance (CFG). The code is available at http:///ihbbbooo///MMMa/a////
1 Introduction
GPT's success in language generation has spurred interest in autoregressive (AR) image genera- tion [42, 43, 49], which relies visual tokenizers like VQGAN [13, 38, 42, 47, 51] to map images into compact, discrete latent spaces. However, these tokenizers, typically trained from scratch and optimized for reconstruction, often yield latent spaces rich in low-level details but poor in high- level semantics and laden with redundancy. Such flawed latent spaces not only prolong AR model training (Fig. 1) but also necessitate techniques like classifier-free guidance (CFG) for high-fidelity, class-conditional image generation, which in turn increases inference time.
In parallel, within the field of computer vision, pre-trained vision foundation models such DINOv2 and CLIP [9, 33, 36, 45, 57] have demonstrated strong capabilities in extracting semantically rich and generalizable visual features. Early explorations in diffusion-based image generation e.g.. REPA [56]- suggest that the semantic representations learned by these models can facilitate the training of generative models. This leads to natural and compelling question: Can the latent features from vision foundation models, originally designed for visual understanding. also serve as robust and structured representations for image reconstruction and generation?
Recent studies [61, 63] have started exploring this direction by leveraging features from vision foundation models to initialize quantizer codebooks [61, 63], augment VQGAN architectures with additional branches [35], or distill these features to guide latent space learning [40]. Although these approaches show promise, they typically treat foundation model features auxiliary components rather than fully capitalizing on their potential as generative priors. As a result, these methods often suffer from inefficiencies and fail to fully utilize the rich semantic information embedded in foundation model features, leaving their generative capabilities largely underexplored.
Corresponding author: xiqi Beee. hk. Project lead.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

VOGAR VPNEAIOROOO NEMTO-CUP
eho-50K
50:
(a) Region-adaptive quantization
(b) Converge speed improvement
Figure I: VFMTok introduces novel features, including: regiionn-daaptive quantization- where it adaptively samples regions of similar patterns and extracts their VFM features for quantization; b).convergence speed improvement compared with vanilla VQGAN [42] for AR image synthesis.
Can VFMs be effective tokenizers? To address this, we initialized the encoder of a VQGAN with different frozen pre-trained foundation models reconstruct images. Once trained, the tokenizer is integrated on top of an AR model for image synthesis (implementation details depicted in Sec. 3.2) As shown in Tab. (middle rows), our results demonstrate that the semantically rich features from these foundation models not only support effective image reconstruction but also achieve generative performance comparable to-or even surpassing- encoder optimized for both reconstruction and generation. These findings highlight the strong potential of pre-trained vision foundation models to serve as efficient and effective tokenizers for image generation tasks, eliminating the need for extensive encoder training while improving qualities.
of a fully trained VQGAN
Can we improve token efficiency for VFMs? Building on this pilot study, we are further motivated by the observation that natural images often consist of irregular regions that exhibit recurring visual patterns. For example, as illustrated Fig. ((a), the upper portion the crystal ball exhibits con- sistent patterns such as texture and transparency; similarly, the moss in the stone possesses similar textural structure. When such images are represented using a regular 2D feature grid extracted from foundation models, this structure-agnen aggnostic representation may introduce significant redundancy. Ex- ploiting redundancy within semantically coherent regions offers a promising direction for improving tokenization efficiency. Motivated by this insight, we propose adgitivve strategy to refine the latent space that aims to enhance both image reconstruction and generation quality while significantly improving token representation efficiency.
Our solution and results. Guided by the preceding experimental anal- ysis and insights, we introduce VFMTok, an image tokenizer that leverages frozen pre-trained vi- sion foundation model for adaptive region-level tokenization. VFM- Tok is designed to achieve high re- construction and generation qual- ity with improved token efficiency. Specifically, VFMTok employs a en-
Table Pilot study of image reconstruction and generation ImageNet [10]. Relative wall-clock inference time for the tokenizer (compared to VFMTok) is reported. L.P. denotes linear probing results the ImageNet validation set, used to estimate the semantic quality of latent tokens.
Image Recon. rFID) rISt
AR Generation
L.P. (%)
Setup
#Tok.
gFID
giSt
Time
VQGAN [42]
576
0.95
197.3
3.71
228.3
4.3
23.1
VQGAN (CLIP)
1.47
182.0
3.45
221.2
4.0
59.5
VQGAN (SigLIP2)
576
0.96
198.4
3.39
267.8
4,0
55.5
frozen pre-trained VFM coder to extract multi-level seman- tie features. A set of learnable an- chor queries performs region-level sampling on these features via de-
VQGAN (DINOv2)
0.99
206.3
3.34
268.6
4.0
56.4
VFMTok (CLIP)
0.99
200.1
3.40
274.7
1.0
63.9
VFMTok (SigLIP2)
256
0.94
218.7
3.01
280.8
1.0
78.5
VFMTok (DINOv2)
0.89
215.4
3.08
274.2
1.0
69.4
formable attention [62], producing region-adaptiv tokens that are subsequently quantized into discrete tokens representing the image's latent representation. These contextual tokens are then processed by lightweight Vision Transformer [12](ViT) in BERT-style framework [11, 15] with two primary reconstruction objectives. First, the original image pixels are reconstructed after de- quantization using a VQGAN [42] decoder. Then, the model reconstructs the features from the frozen foundation model itself, allowing VFMTok to retain the semantic richness and discriminative power of the original representations. Once trained, VFMTok enables standard autoregressive Transformers (e.g., Llama [44]) to generate contextual token sequences, which are decoded back into images via the VQGAN decoder, facilitating high-quality image synthesis with compact and semantically mean-
2

---

ingful representations. As shown in Tab. (bottom rows), VFMTok achieves superior reconstruction and generation performance while using fewer than half the original number of tokens (256 vs. 576).
Extensive experiments validate that VFMTok, by combining the representational power of visual foundation models with novel region-adaptive tokenization strategy based on irregular sampling and learnable anchor queries, enables both high-quality and efficient image reconstruction and autoregressive (AR) generation. First, VFMTok achieves superior reconstruction quality and captures richer semantics using significantly fewer tokens compared to prior methods (e.g., 256 vs. 576 in [42]), resulting in a structured, semantic-awarc and compact latent space. As shown in Tab. 1, VFMTok, with only 256 tokens, outperforms other tokenizers using the same VFM encoder by delivering superior reconstruction quality and stronger semantic representation (as indicated by linear probing). Second, the high-quality latent space produced by VFMTok facilitates effective AR training using simple I.LMM- Abased model, leading to faster convergence (see Fig. ((b) and improved generation performance. Notably, the 1 1.4B AR model surpasses the performance of LlamaGen--3B despite having fewer parameters and requiring fewer training iterations. The 1.5B advanced AR model achieves a new state-of-the-art with gFID of 1.36 on ImageNet [10] 256 x 256, outperforming widely-used diffusion models. Third, due to the compact token space and the reduced number of tokens, VFMTok significantly improves the inference speed of AR models (see Tab. 1). Moreover, the rich semantic content embedded in the latent tokens enables effective cansiconniitonal image synthesis with high fidllity-ithouut without the need for classifier-free guidance further reducing inference time.
Our contributions can be summarized as follows:
We demonstrate that frozen vision foundation models- ranging from self-supervised to language- supervised- are effective for image reconstruction and generation. Leveraging their semantic richness enhances the tokenizer and enables AR generation models to converge faster and perform high-fidelity, CFG-free image synthesis, without bells and whistles.
We propose region-adaptive tokenization framework that effectively leverages inherent redundan- cies in image regions to achieve compact tokenization. This approach reduces the number of visual tokens while enhancing performance, enabling efficient AR generation without sacrificing quality.
Extensive experiments validate the effectiveness of our approach both image reconstruction and AR generation, establishing pre-trained vision foundation models as powerful tokenizers for high-quality and efficient image generation.
2 Related Work
Image Tokenization using Autoencoders. Pixel-space images are highly redundant. Autoencoder- based tokenizers [29, 42, 43, 53] create compact latent tokens to reduce redundancy. VQVAEs [21, 38, 47] and their derivatives evolved using adversarial losses [13], Transformers [51], multistage quantization [24, 59], lookup-free methods [29, 31], and codebook initialization from pre-trained features [61, 63]. These 2D tokenizers map features to a static 2D grid, which limits redundancy exploration. Recent ID tokenizers [1, 32, 48, 55] offer superior compression, reconstruction, and redundancy removal, but often require complex and lengthy training. For example, TYTok [55] requires two-stage process (warming up and fine-tuning! for 200 epochs. Our VFMTok adopts novel region-adaae adaptive tokenization framework to reduce redundancy. With a simpler training strategy for only 50 epochs, VFMTok exhibits discriminative semanties and excellent generation results.
Vision Foundation Models. Vision Foundation Models (VFMs) [3, 6, 14, 16, 17, 20, 33, 36, 45, 57] aim to learn general, transferable visual representations from large-scale, diverse data. The training of these versatile models has shifted from early supervised approaches to more scalable self-supervised learning [3, 6, 9, 11, 14-16, 33]. which leverages inherent data structures. More recently, language-superye supervised pre-trainn pre-training [20, 45, 57] on vast image-text pairs has enabled VFMs to learn rich, semantically grounded representations. Pre-trained VFMs serve as powerful backbones for a wide array of downstream tasks. In this work, we utilize pre-trained VFMs directly as image tokenizers for AR image generation, surpassing other methods [61, 63] with superior performance. Furthermore, using VFMs as tokenizers enables the removal of classifier-fre free guidance.
Autoregressive Image Generation. GPT-style Transformers [5, 24, 37, 42, 43, 46] have spurred interest in autoregressive (AR) image generation, which predicts visual token sequences. While early AR models operated in pixel space 46), current methods [24, 42, 43, 51] generate discrete latent to- kens via next-token prediction, then decode them to pixels using a tokenizer's decoder [13, 38, 47, 51]. To improve the generation quality, recent works [25, 43, 49] add bidirectional attention (e.g., VAR's
3

---

next-scale prediction [43], MAR's BtRT-style framework [25]. Show-o's hybrid attention [49]). These innovations, however, complicate designing universal, multi-modal Transformers adhering to next-token prediction. Instead, our VFMTok enables standard AR transformers to generate contextual token sequences for subsequent decoding, eliminating complex structural modifications.
3 Method
In this section, we first provide preliminary background on quantized image tokenizers. We then present our pilot studies exploring the use of vision foundation models for tokenization. Finally, we introduce VFMTok, novel tokenizer built upon frozen vision foundation models, incorporating region-adaptive strategies to enhance both the efficiency and effectiveness of the tokenization process.
3.1 Preliminary
To apply autoregressive modeling to visual generation, existing methods [42, 43, 51, 52] necessitate image tokenizer to map 2D image into discrete token sequences for AR generation. To achieve this, quantized autoencoders, such as VQVAEs [13, 38, 42, 43, 47, 51, 61], widely used. Typically, image tokenizer consists of an encoder 8(-), quantizer VQ(.), and a decoder D(-). Given an input image E #xxxxx x3 the encoder E(.) first
Quantized Image Tokenizer.
xD with spatial down-sampling factor Then, Zop is fed into the quantizer (-(()) that includes a learnable codebook C e N XD with N
convert image into patch embeddings Zan E R $ x
vectors. Each feature vector 2; E RD is mapped into its nearest vector e; E FD in the codebook C.
7aD =I(,,
VQ(=:) = Ci, where i == arg min || 1
jE(1.2.....N
where and W denote the input image's s height and width, respectively. D depicts the latent feature dimension. Once discrete tokens are acquired, they can be de de-quantized into corresponding code and converted back to image pixels by the decoder D(.), as depicted in Eq. (2).
i= D=D(VQ(ZeD))
To optimize the codebook, the training objective is Lvq Σ |gg(2; (4) -all3 8- ||sg(e,) 2/12 where sg(-) is a stop- gradient function [2, 47]. The second term is a commitment loss weighted by 3 to align extracted features with codebook vectors. For image reconstruction, the loss function is LAE fI + (L, i) + Ar La((), where L2 is pixel-wise reconstruction loss, Lp is perceptual loss from LPIPS [58], and LG is adversarial loss from PatchGAN [19] with weight ÀG.
3.2 Pilot Study: Pre-trained Vision Foundation Models as Tokenizers for AR Generation
To assess whether pre-trained VFM can serve as a tokenizer for image reconstruction and benefit image generation, we performed a pilot study. In our setup. we extract the final 2D grid features from images of size 336 x 336 using a frozen VFM, such as DINOv2, CLIP, and SigLIP2. These features, after quantization, are fed into a VQGAN [42] decoder for image reconstruction. Once trained, the tokenizer is integrated on top of Lama-bbsed AR model for image synthesis. Additionally, the training duration for VQGANs and AR models is 50 and 100 epochs, respectively.
As Tab. illustrates, directly using features from pre-trained VFMs yields decent image reconstruction and generation performance compared to vanilla VQGANs. Notably, these VFM-based tokenizers consistently exhibit stronger semantic representation capabilities (as indicated by the linear probing experiment in Tab. 1). For instance, VQGAN (SigLIP2) achieves reconstruction performance on par with vanilla VQGAN, while exhibiting better semantic representation and superior generation quality, Nevertheless, variations in image reconstruction and generation quality arise when different VFMs are used to initialize the tokenizer's s encoder. Specifically, VQGAN (DINOv2) and VQGAN (SigLIP2) demonstrate similar reconstruction and generation quality, both outperforming vanilla VQGAN, while the reconstruction quality of VQGAN (CLIP) trails that of vanilla VQGAN. One contributing factor that different learning objectives used to train VFMs influence their ability to extract detailed and semantic features from images, thereby affecting downstream image reconstruction and generation quality. As evidence, both DINOv2 [9] and SigL.PP [45] employed a masked prediction objective to optimize their VFMs, whereas CLIP [36] did not.

---

3.3 VFMTok
Building upon the semantically rich features provided by vision foundation caadels-typically structured as regular 2D grids-we introduce VFMTok, a region -adbtivee tokenizer that identifies semantically coherent, irregular local regions produce region- adaative tokens. These tokens are sequentially quantized for decoding. with tailored learning objectives to enhance performance. In the following, we detail the architecture of VFMTok, including its region-adaptive token generation module and dedicated decoder for both image and feature reconstruction. We further describe the training objectives, which combine a pixel-level reconstruction loss for image synthesis with a feature reconstruction loss that preserves the semantic content of the foundation model's representations.
Lyim
Encoding
Quantization
Decoding
multi-level feature
shared VIT
Foundation Model
Deformable Transformer
Quantizer
sharefweights
Decoder
shared ViT
Reshape
O
learnable anchor queries
foundation model features
region-adapin plive tokens
Resture mask embedding
maskc embedding
chared posirional emheddiiggs
reconstructed foundation model features
image Intent features
The framework of VFMTok. VFMTok utilizes frozen VFM to extract multi- image features. A deformable Transformer then processes these features with learnable grid queries to generate region-adaptive tokens. After quantization, these tokens are fed into shared ViT for dual reconstruction: 1) VFM features, targeting similarity with the VFM's s last-layer outputs, and 2) image latent features, which are reshaped to a 2D grid and decoded into pixels.
Figure 2:
Region-adaptive Token Generation. Following our pilot study, we utilize a frozen pre-trained vision foundation model (VFM) to encode an input image / into latent embeddings F. Since features extracted from VFMs contain rich details in shallower layers and high-level semantics in deeper layers [7, 27, 28] bothhhthhooe which are critical for image reconstruction wwe extract multi-level features then projected to uniform embedding dimension using a two-layer MLP. Next, as shown Fig. 2, based on the multi-level features Fm. we introduce region-adaptive sampling mechanism using deformable aites-attention layers [8, 62]. A set of learnable anchor queries, initialized as a 2D grid, are iteratively refined through multiple deformable attention layers. In each layer, an anchor query predicts sampling offsets for each VFM feature level via a linear layer, enabling sampling from irregular, data-dependent positions. These sampled features are then weighted using attention scopes-comppued through another linear layer-and aggregated to update the query. Through this process, the anchor queries are progressively refined to capture semantically coherent, region-specific information. The final refined queries are referred as region-adaptive tokens Z,, which are subsequently quantized into discrete tokens Z,. Compared to fixed 2D feature grid, VFMTok adaptively aggregates features from semantically coherent, irregular regions. This substantially reduces redundancy, enabling the use of fewer tokens while maintaining superior image reconstruction and generation performance. As shown in Tab. just 256 semantically rich tokens from VFMTok are sufficient to achieve high-fidelity reconstruction and generation.
Fm from the VFM. These multi-level features
Vector Quantization. Once the continuous region-adaptive tokens Z, are obtained, a quantizer Qe(+) applied to discretize them into region-adaptive discrete tokens Z,. Given that the design of the codebook plays critical role in the performance of an image tokenizer, follow the practices in [42, 51] by applying fg-normalization to the codebook vectors. Additionally, we adopt a low- dimensional embedding space with large codebook size enhance both reconstruction quality and codebook utilization following [42, 51].
Decoder of VFMTok for Image and VFM Feature Reconstruction. After de-quantization, the region-adaptive tokens Z, are used for image reconstruction. Since these tokens represent irregular,
5

---

region-leved features, decoding them into regular 2D image grid requires alignment. To achieve this, we introduce a set of mask tokens M,, representing a 2D feature map of size Hn, x Wn,, with channel dimension C. The mask tokens are initialized by replicating a single learnable token Hm. x W. times. Position embeddings F,, encoding spatial locations, are then added to form position-awars masked tokens. Next, the de-quantized sequence is processed by a lightweight Transformer Evi(-),, which propagates information from the region-adaptio tokens to the masked image tokens. This Transformer employs causal self-attention, aligning its latent space with the structure of autoregressive models. Following DINOv2 [9], we further enrich the imput sequence by appending a [CLS] token and several register tokens to improve representation learning and capture global context-though these are not used for reconstruction. The output of this Transformer is a refined set of mask tokens FI representing a regular 2D grid structure. These are reshaped into spatial grid and passed into decoder ((-) to reconstruct the image.
tokens Z are concatenated with M, and the combined
To preserve the semantic integrity of the VFMTok tokens, we also reconstruct high-level features (specifically, from the final layer) of the vision foundation model (VFM). This process mirrors image reconstruction: a new set of mask tokens My is initialized and augmented with positional embeddings E, shared with those used in image reconstruction. The concatenation of Z, and M, is then processed by the same shared Transformer (-i(.) to produce Fp, the reconstructed high-level VFM feature map. By sharing (-)(() between image and feature reconstruction, we reduce the model's parameter footprint while ensuring the semantic fidelity of the latent tokens. Note that the VFM feature reconstruction is only applied during tokenizer training.
Training Objective. For tokenizer optimization, we follow the training objectives of VQGAN [13. 42], with one key modification: we replace its original discriminator with a pre-trained DINOv1-S [3] model. This substitution provides adversarial training guidance more semantically meaningful way compared to conventional discriminators such as PatchGAN [19], and we find it consistently improves reconstruction quality. In addition to image reconstruction, we incorporate a feature reconstruction objective by computing the cosine similarity loss between the reconstructed features and the corresponding frozen features from the pre-trained vision foundation model (VFM). The overall training loss is defined as: C = a LAE + X Sim. where LAE denotes the image reconstruction loss and Lsim is the feature reconstruction loss. In our experiments, we set both a and À to 1.
3.4 Autoregressive Image Generation
Once VFM'Tok is trained, the optimized discrete region-adaptive tokens 2, can be integrated into an autoregressive (AR) Transformer, where they are generated sequentially via next-token prediction mechanism, conditioned on class or text embedding c. The generated tokens are then passed through the Transformer encoder εver( (-) to produce latent image features Ji, which are subsequently decoded into images using the decoder D(-). In the AR model, the region-adaptive tokens Z, are augmented with positional -speddiiggssseefiiillll 2D Rotary Position Embeddings (RoPE) [41] to better capture their spatial locality and structure.
4 Experiment
4.1 Sctup
Image Tokenizer. In the main experiment, we initialize the encoder of VFMTok with frozen pre-trained DINOv2-L [9]. Considering its composition of 24 Transformer layers, we extract features from the 6th, 12th, 18th, and 24th layers to create multi-level features. Consistent with [42, 55], we set the codebook vector dimension of the quantizer to 12 with a codebook size of 16384, to achieve a better reconstruction quality and efficient codebook utilization. Mcanwhile, VFMTok utilizes 256 tokens to represent an image. Besides, the depth of the Transformer is set to 6 (following [62]). The model is trained on the ImageNet [10] training set and evaluated on its validation set.
Given that the resolution of vision foundation models (VFMs) [9, 20, 36, 45, 57] is typically x 3366, while VFMTok represents images with fixed 256 tokens by default, it's comparable to vanilla tokenizers [13, 42, 61]. Thus, we train the tokenizer on 336 x 336 images. Except this, we keep the training settings unchanged as LlamaGen [42]. During evaluation, the reconstructed images of 336 x 336 are resized to 256 x 256 for evaluation, which consistent with the evaluation procedure in LlamaGen [42].

---

Classicondi Autoregressive Image Generation. Following the generation procedure in LlamaGen [42], the AR models first generate images of 336 x 336 and then resize them to 256 x 256 for evaluation. Considering computational costs, we set the training duration based on the number of models' parameters. Models with fewer than parameters are trained for 300 epochs, while the remaining models are trained for 200 epochs. Beyond the resolution and training duration, all models are trained with the same settings as LlamaGen [42]. Furthermore, we also incorporated the same VFMTok into the RAR [54] autorerressive generation framework, with all training settings remaining consistent with RAR [54]. Additionally, in our experiments, AR generation is conducted with both classifier-free guidance (CFG) and CFG-free protocol.
Evaluation metries. To evaluate image generation performance, we use Fréchet inception distance (FID) [18] and Inception Score (IS) [39] as the main metries to measure the generation quality of different models. In addition, sFID, Precision, and Recall [23] are also reported following [42].
4.2 Main Results
Table 2:
Comparison with other image tokenizers. oim. indicates trained on OpenImages [22]. 2 denotes the codebook usage in contextual and patch-level quantizers, respectively.
Image Reconstruction. We compare VFMTok against rep- resentative 2D image tokeniz- ers, VQGAN [13], MaskGiT [4],
Tokenizer Setup
Image Recon.
Usage (%
Method
ViT-VQGAN [51], and 1D tok- enizer, TYTok [55]. As shown in Tab. 2, our tokenizer represents an image with just 256 tokens, considerably fewer than some counterparts, For instance, the VQGAN variant LlamaGen [42]
f
Size
Dim. #Tok.
rFID
rISt
Qc
Qp
TYTok [55]
8192
64
256
1.05
191.5
100
ImageFolder [26]
32768
32
286
0.69
201.5
100
VQGAN viim. [13]
256
1.44
VQGAN [13]
8192
256
1.49
8
1024
VTTVQGAN [51]
8192
32
1.28
192.3
95.0
uses 576 tokens, while VQ- GAN [13] and ViT- [51]
VQGAN"im! [13]
16384
4
1.19
VQGAN [13]
7.94
1024
256
256
even utilize up to 1024 tokens. Despite this efficiency, VFMTok achieves a strong rFID of 0.89, and further demonstrates 100% utilization of the codebook.
MaskGiT [4]
16
2.28
VAR [43]
4096
32
680
0,92
196.0
100
RQ-VAE [24]
32|
16384
256
1024
1.83
VQGAN [13]
256
256
4.98
VQGAN [42]
16
16384
441
1.21
189.1
99.2
The rIS score of 215.4 achieved by VFMTok significantly out- performs other methods, e.g., TiTok [55] and the VQGAN series. The rIS metric quantifies the KL-divergence between the original label distribution and the logit distribution of reconstructed images after softmax normalization, thereby measuring the semantic consistency between reconstructed and original images. The higher rIS confirms VFMTok is more effective at preserving semantic consistency during reconstruction.
VQGAN [42]
8
576
0.95
197.3
99.7
VFMTook (Ours)
16384
12
256
0.89
215.4
100
class-conditional Image Generation. We evaluate VFMTok on vanilla autoregressive models- LlamaGen [42], and advanced generative model - RAR [54] with different scales by conducting 256 x 256 class-conditional image generation task on ImageNet [10], where comparing them with the mainstream generation models, including diffusion models (Diff.) [30, 34, 50, 60], BERT-style masked-predictto models (Mask.) [4], and AR generation models (AR) [13, 24, 38, 42, 51, 55].
As shown in Tab. 3, our models exhibit competitive performance across all metrics compared to mainstream image generation models. Notably, VFMTok beats BERT-style models [4] in terms of gFID without the requirement of complicated sampling tuning. With comparable or even fewer parameters, our method surpasses most AR generative models [13, 21, 38, 51, 55] in both gFID and gIS metrics. Under the same training setting, VFMTok surpasses LlamaGen [42] by significant gFID gains and notable gIS improvements. Specifically, VFMTok-B outperforms LlamaGen-B [42] with gains of 2.56 in gFID and 69.7 in glS. Besides, our VFMTok-L model achieves gFID of 2.75 at 300 epochs, also obtaining gain of 22.7 gIS. Notably, when compared with LlamaGen-3B with 3B parameters, our VFMTook- XXL achieves even better generation performance with less than half the number of parameters and fewer training iterations. Futhermore, when VFMTok is incorporated into RAR [54], it achieves generative performance with gFID of 1.36, which is the of-te-of-theart generation performance at present. Additionally, class-conditional image generation results are visualized in the Appendix.
7

---

Table 3: Class-conditional image generation quality estimated on ImageNet [10] validation bench- mark. indicates it is implemented by us, and '-re' indicates using rejection sampling.
Generation w/ CFG
Generation w/o CFG Pre. Rec.
Type
Method
#Epoch #arra. #Tok.
gFID sFID gIS Pre. Rec.
sFID gIS
MaskDiT [60]
1600
675M
2.28 5.67 276.6 0.80 0.61
5.69
10.34 177.9 0.74 0.60
DiT [34] SiT [30]
1600
675M
2.27 4.60 278.2 0.57
9.62
6.85
121.5 0.67 0.67
Diff.
1600
675M
2.06 4,50 270.3 0.59
8.61
6.32
131.7 0.68 0,67
FasterDIT [50]
400
675M
2.03 4.63 264.00.81
0.60
7.91
0.67 0.69 5.45
MaskGiT
6.18
182.1
0.80 0.51
Mask.
555
227M
256
MaskGiT-re
4.02
355.6
VAR [43]
350
310M
680
3.30
2.8.400.4
0.51|
TiTok-B [55]
111M
6.76
7.82
175.3 0,43
19.6
57.9
7.54
0.64 0.60
300
256
TYTok-I. [55]
343M
4.03
6.93 219.50.84 0.52
11.4
88.8
7.14
0.68 0.64
LlamaGen-B
111M
6.09 7.24 082.5 0.42
32.2
11.84
39.9 0.57 0.61
LlamaGen-L
343M
3.07
6.09 256.1 0.83 0.52
19.1
8.67
64.3 0,61 0.67
300
576
-lamaGen-XXI.
1.4B
2.34
6.00
253.9 0.60
14.6
8.69
86.3
0.63 0.68
LlamaGGn-3B
3.1B
2.19
5.97 263.3 0.58
9.38
8.24
112.9 0.69 0.67
RAR-L [54]
461M
1.70
299.5 0.58
6.72
5,56
129.2
0,74 0.61
AR
RAR-XI. [54]
400
955M
256
1.50
306.9 0.62
4.62
5.27
158.3 0.77 0.62
RAR-XXL [54]
1,5B
1.48
326.0
3.85
5.18
176.2 0.61
VFMTok-B
111M
3.43 5.88 252.2 0.85 0.53
3.09
5.67
173.6 0.80 0.58
300
VFMTok-T.
343M
2.75
5.58
278.8
0.84 0.57
2.11
5.46
230.1 0.82 0.60
256
VFMTok-XXL
200
1.4B
2.19
5.53 278.00.83 0.60
1.95
5.65
0.82 0.62 0.82
VFMTok-3B
200
3.1B
2.07
6.23
280.4 0.81 0.62
2.04
5.43
267.6 0.82 0.61
RAR-L.VFMTok)
461M
1.44
6.03
312.80.78 0.66)
2.02
5.51
210.4 0.79 0.63
RAR-XL (VFMXL)
400
955M
256
1.38
5,86
3.7. 0.65
1.74
5.33
233.0 0,80 0.63
RVV-XI_VVMMTok)
1.5B
1.36
5.85
301.3
0.78 0.66
1.65
5.55
253.7 0.80 0.63
VFMTok-I4SigLIPP)
300
343M
2.69 5,31
2.8.4 0.56
2.11
5.39
225.6 0.60
VFMTok- (SL(igI)
200
1.4B
256
2.16
5,45 272.00.83 0.60
1.98
5.53
265.3 0,82 0.82 0.62
VFMT))-22(SigI.IP2)
200
2.2B
2.17 5.43 281.40.83 0.60
1.98 5.41 269.7 0.82 0.62
Furthermore, we conducted experiments by removing classifier-free free guidance (CFG). Remarkably, the generation results without CFG show that most evaluation metrics- such as sFID, Precision, and Recall-remain comparable to those obtained with CFG. While gIS experiences slight decline, gFID improves compared to its CFG-enabled counterpart. Similar trends are observed when VFM- Tok's encoder is replaced with other frozen pre-trained vision foundation models like SigLIP2 [45]. These results demonstrate that our method supports high-fidelity autoregressive image generation without CFG, which significantly accelerates inference. In contrast, baseline methods suffer substantial performance degradation without CFG- for example, LlamaGen-3 model sees gFID worsen to 9.38, whereas our 1.4B model VFMTok- XXL achieves a gFID of 1.95 without CFG.
4.3 Ablation Study and More Analysis
Component study. To assess the contribution of each proposed component to image reconstruction and synthesis, we conduct a step-by-step component analysis using baseline tokenizer built on vanilla VQGAN [42]. We incrementally add the following components: (I) replace the VQGAN encoder with frozen pre-trained foundation model (DINOv2-L [9]): (2) introduce learnable queries and a deformable attention for region-adaptive tokenization, using only single-level features from the final layer; (3) incorporate multi-level features to enrich representations with both low-level detail and high-level semantics; and (4) add a feature reconstruction objective based on pre-trained VFM outputs. After training each tokenizer, for autoregressive image synthesis. Both the tokenizer and AR model are trained for 50 epochs. Additionally, we perform linear probing on the [CLS] token, following the MAE [15] protocol.
integrate it with our AR generation model, VFMLok-LL,
As shown in Tab. 4, replacing VQGAN's encoder with a frozen pre-trained vision foundation model yields reconstruction and generation performance par with a VQGAN trained specifically for
8

---

visual reconstruction using 576 tokens. This substitution also significantly enhances the semantic quality of the tokenizer's ' representations. To further improve token efficiency, we introduce region- adaptive tokenization using deformable attention to exploit the spatial redundancy inherent in regular 2D grid features. This reduces the number of visual tokens 256. However, this performance gain comes at a cost: reconstruction and generation quality degrade slightly due to two factors: (I) fewer visual tokens limit representational capacity, and (2) the absence of explicit supervision hinders the effective optimization of the region-adaptive tokens. To address this, we incorporate multi-level feature extraction, which improves the reconstruction capability by leveraging both low- and high-level information. However, without additional guidance, the semantie consistency of the learned tokens may still degrade. Finally, we introduce a pre-trained feature reconstruction objective, which significantly boosts both image reconstruction and generation quality. This objective encourages alignment with semantic features from the frozen VFM and effectively halances the contri- butions of low- and high-level features to the contextual tokens therebyy preserving semantic fidelity.
With these three key components- (I) deformable attention for region- adaptive tokenization to reduce re- dundancy, (2) multi-level features for enhanced reconstruction, and (3)
Table 4: Ablation study on VFMTok's 's components.
Image Recon. #Tok. rFID) rSSt
Usage
AR Gen.
L.P.
Setup
Qc t
gFID
glSt
(%)
VQGAN
0.95
197.3
99.7%
3.71
228.3
23.1
576
feature reconstruction loss for se-
Frozen VFM
0.99
206.3
100%
3.69
267.5
56.4
mantic alignment-VFMII
pro-
Region Adapt.
1,20
199.0
3.98
241.6
41.5
duces compact, semantically rich,
Multi-level Feat.
256
0.92
199.5
100%
3.71
251.1
22.7
and efficient tokens. Using only
Reconstruct Feat.
0.89
215.4
3.42
277.3
69.4
256 tokens, VFMTok outperforms Frozen VFM
256
0.95
196.3
100%
3.73
248.7
248.7|59.1
59911
its VQGAN baseline with 576
kens in reconstruction quality, generative performance, and semantic representation. Supplemental ablations are discussed in the Appendix.
Convergence and efficiency analysis. Beyond above analysis. we experiment VFMTok with randomly initialized encoder instead of pre-trained VFM with other components remaining unchanged. As shown in Tab. (last row), its reconstruction quality dropped to the level of VQGAN, Meanwhile, both its semantic representation capability and generation performance also decreased. This indicates a frozen VFM benefits tokenizer training as it provides a latent space advantageous for image reconstruction and generation. Besides, those semantic-rich, structured latent tokens accelerate AR model training convergence. As evidenced in Fig. l(b), VFMTok enables AR models achieve 3 speedup in convergence compared VQGAN. Moreover, an AR model's generation time is quadratically proportional to the number of tokens. At the same resolution, VFMTok uses approximately half the tokens for image representation compared counterparts like DINOv2- VQGAN and CLIP-VQGAN Consequently, VFMTok achieves 4x generation speedup over these counterparts depicted in Tab. This acceleration can be further enhanced with CFG-free generation.
Conclusion
In this work, we have demonstrated that frozen pre-trained vision foundation models (VFMs) ranging from self-supervised language-supervised- are sufficient for high-quality image recon- struction and generation. To fully exploit their potential while addressing the redundancy inherent in 2D feature grids, we introduce VFMTok, a novel image tokenizer that incorporates region-adaptin tokenization to enhance token efficiency. By reducing feature redundancy, integrating multi-level feature representations, and introducing a pemantic-preserving feature reconstruction objective, VFMTok yields a compact and semantically rich latent space. This facilitates high-quality image reconstruction and generation, accelerates convergence in autoregressive (AR) models, and enables efficient, high-fidelity, classifier-frre (CFG-free) image synthesis without the need for additional heuristics. Furthermore, the reduced number of tokens significantly lowers the computational cost of AR inference, making the approach both scalable and effective. Looking forward, the rich semantic structure of the learned latent space offers exciting potential for extending this work toward unified visual generation and understanding.

---

6
Acknowledgments
This work has been supported by the National Key R&D Program of China (Grant No. F022YB33608300) Hong Kong Research Grant Council Early Career Scheme (Grant No. 27209621), General Research Fund Scheme (Grant No. 17202422, 17212923, 17215025) Theme- based Research (Grant No. T45-701/22-R) and Shenzhen Science and Technology Innovation Commission (SGDX202205301 1 1405040), Part of the described research work is conducted the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust. We are deeply grateful to Lufan Ma for the contribution in polishing up this paper. We would also appreciate Tong Yang for providing the DINO discriminator script.
7 Author Contribution Statement
X.Q. proposed the initial concept of region-adaptive quantization. Based on this, A.Z. built the VFMTok, conducted the overall experiments, and led the writing of the initial draft. X.W., X.Q.. and C.M. were deeply involved in the project progress and manuscript writing. X(iangyu).Z. ZZ., X(uanyang).ZZ ZZ., G.Y., Y., and T. W.W provided sufficient computational resources. X(uanyang) joint discussion where suggested ablation studies along with T.W., and discussed the writing of the draft. All authors contributed critical feedback, shaping the research, analysis, and the final manuscript.
References
[1] Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini, Oğuzhan Fatih Kar, Elmira Amirloo, Alaaeldin El-Nouby, Amir Zamir, and Afshin Dehghan. Flextok: Resampling images into 1d token sequences flexible length. arXiv preprint arXiw:2502.13965 139677 2025.
[2] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. stochastic neurons for conditional computation. arXiv preprint
Estimating or propagating gradients through
2013.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[4] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[5] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference machine learning, pages 1691-1703. PMIRR, 2020.
[6] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming Hc. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.00 24227,,00.04297, 2020.
[7] Xuangeng Chu, Anlin Zheng, Xiangyu Zhang, and Jian Sun. Detection in crowded scenes: One proposal, multiple predictions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12214-12223, 2020.
[8] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 764-773, 2017.
[9] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309, 16588, 2023.
[10] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255. 2009.
[11] Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 4171 4186, 2019.
10

---

[12] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprini arXiv:2010.11929, 2020.
[13] Patrick Esser, Rohin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12873-12883, 2021.
[14] Bran-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised lcarning. Advances in neural information processing systems, 33:21271-21284 2020.
leaningg
[15] Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.0572 2019.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
[19] Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 1125 1134, 2017.
[20] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[21] Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6144 2013.
[22] Alina Kuznetsova, Mohamad Hassan Mohamad Rom, Neil Alldrin, Jasper Uijlings. Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Ducrig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128(7):1956 1981, 2020.
[23] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
[24] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11523- 11532, 2022.
[25] Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming Hc. Autorogressive image generation without vector quantization. arXiv preprint arXiv:2406. 1838, 2024.
[26] Xiang Li, Kai Qiu, Hao Chen. Jason Kuen, Jiuxiang Gu, Bhiksha Raj. and Zhe Lin. Imagefolder: Autoregressive image generation with folded tokens. arXiv preprint 4rXiv:2410.01756, 2024.
[27] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21 17-2125, 2017.
[28] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2980-2988, 2017.
[29] Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang. and Ying Shan. Open meen-magit2:: An open- source project toward democratizing auto regressive visual generation. arXiv preprint MXiv:2409.04411 2024.
11

---

[30] Nanye Ma. Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pages 23-40. Springer, 2024.
[31] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309. 2023.
[32] Keita Miwa, Kento Sasaki, Hidchisa Arai, Tsubasa Takahashi, and Yu Yamaguchi. One-d-piece: Image tokenizer meets quality contoly-o compression. arXiv preprint Sil::2501.10064, 2025.
[33] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Pao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assan, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199 2023.
[34] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4195-4205, 2023.
[35] Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang. Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069, 2024.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabricl Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[37] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.
[38] Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advancey in Neural Information Processing Systems, 32, 2019.
[39] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Neural Information Processing Systems, 29, 2016.
[40] Wei Song, Yuran Wang. Zijia Song, Yadong Li, Haoze Sun, Weipeng Chen, Zenan Zhou, Jianhua Xu, Jiaqi Wang, and Kaicheng Yu. Dualtoken: Towards unifying visual understanding and generation with dual visual vocabularies. arXiv preprint arXvv:2033. 14324, 2025.
[41] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 068:12700 127063, 2024.
[42] Peize Sun. Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autore- gressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.0652. 2024.
[43] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404. 02905, 2024.
[44] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ann Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302. 13971, 2023.
[45] Michael Tschannen, Alexey Gritsenko, Xiao Wang. Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vasion-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint 250: 14786, 2025.
[46] Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelcnn decoders. Advances in Neural Information Processing Systems, 29, 2016.
[47] Aäron van den Oord, Oriol Vinyals. and Koray Kavukcuoglu. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
12

---

[48] Xin Wen, Bingchen Zhao, Ismail Elezi, Jiankang Deng. and Xiaojuan Qi. "Principal components" enable new language of images. arXiv preprint arXiv:2503.0866. 08688, 2025.
[49] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen. Zhenheng Yang. and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
[50] Jingfeng Yao, Cheng Wang, Wenyu Liu, and Xinggang Wang. Fasterdit: Towards faster diffussion transformers training without architecture modification. Advances in Neural Information Processing Systems, 37:56166-56189 2024.
[51] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vegan. arXiv preprint arXiv: S.xi:2111004400 10.04627, 2021.
[52] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan. Ben Hutchinson, Wei Han. Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022.
[53] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang. Alexander G. Hauptmann, Ming-Hsuan Yang. Yuan Hao, Irfan Essa, and Jiang. Magvit: Masked generative video transformer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10459- 10469, 2023.
[54] Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chich Chen. Randomized autoregressive visual generation. arxiv, 2024.
[55] Qihang Yu, Mark Weher, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chich Chen, An image is worth 32 tokens for reconstruction and generation. arXiv preprint 2rXiv:2406.07550 2024.
[56] Sihyun Yu, Sangkyung Kwak, Huiwon Jang. Jongheon Jeong, Jonathan Huang. Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint 241.:24100009400 2024.
[57] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference computer vision, pages 11975-11986, 2023.
[58] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features a perceptual metric. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018.
[59] Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors high-fidelity image generation. Advances in Neural Information Processing Systems, 35:23412-23425, 2022.
[60] Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training diffussoo models with masked transformers. arXiv preprint 23X.v:2306.09305, 2023.
[61] Lei Zhu, Fangyun Wei, Yanye La,, and Dong Chen. Scaling the codebook size vegan to 100,000 with a utilization rate of 99%. arXiv preprint arXiv:2406. 1837, 2024.
[62] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04155 2020.
[63] Yongxin Zhu, Bocheng Li, Hang Zhang, Xin Li, Linli Xu, and Lidong Bing. Stabilize the latent space for image autoregressive modeling: A unified perspective. arXiv preprint arXiv:2410. 124,99 2024.
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and the introduction clearly state that we aim to improve the effectiveness of AR generation method and we summarize the main contributions of our paper in the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
The limitation of this work will be discussed in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: There is no theory assumptions in paper. Therefore, there is need to provide proof in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We describle the main experiment setting in Sec.4.1 1 and more implementation details be found in the Appendix.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use open-source datasets, e.g. ImageNet- The code will be released to reproduce the main reported results in this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
cc/
publ / gui des CodeSubmi ss on ioolice eyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht (https: //nips... cctpssccc/ooo
cc/ ctce/public/ poublic//uidee iugiee/////eee Cooe/ooeuubbn assonnollls. MipPooico icy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the experimental setting and details in Sec. 4.1 and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The experiments do not include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details on compute resources are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /nnuriirsss ps cwtppccpppcccccccccccccccccccccc wealll///tttiie caGu de nise?
Answer: [Yes]
Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Potential social impacts are discussed in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
18

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original papers of assets are properly cited and we follow the original license of these assets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package popular datasets, papers rawi thoode. com dat sets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
should be provided.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
19

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not non involve LLMs as any important, original, non- non-standard components.
Please refer to our LLM policy (https: /tt////erriipe https/ccecoccceceacaaeaaaaaaaaee LLM) for what should should not be described.
20


Reviews:
[{"review_id": "kStmouYWj2", "reviewer": "Reviewer_KT8u", "summary": "This paper proposes using pre-trained vision foundation models (VFMs) as image tokenizers. To improve the tokenizer's performance, the authors introduce a region-adaptive quantization framework and a semantic reconstruction objective. Experimental results show that the proposed tokenizer, VFMTok, achieves competitive reconstruction performance and can improve the generation quality of LlamaGen on the ImageNet dataset.", "questions": "- What is the performance of VFMTok on higher-resolution images (e.g., 512x512 or 1024x1024)? The current experiments seem limited to lower resolutions.\n\n- The proposed region-adaptive quantization framework appears to be a general method. Is it independent of using a VFM-based tokenizer? How would this framework perform if applied to other existing tokenizers?\n\n- Could you provide some direct qualitative comparisons? For example, showing the same image reconstructed or generated using VFMTok versus other baseline tokenizers would make it easier to see the practical differences.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The motivation to use pre-trained vision foundation models as the basis for an image tokenizer is an interesting idea.\n\n- Experimental results on ImageNet demonstrate that the proposed VFMTok can improve the generation performance of the LlamaGen model.\n\n- The paper includes adequate ablation studies that help verify the effectiveness of each component within the proposed VFMTok.\n\nWeaknesses\n\n- Missing Baselines: The paper is missing comparisons to many important and relevant baseline tokenizers. For a thorough evaluation, VFMTok should be compared against methods like VQ-VAE, MAGVIT-v2, and MAETok, among others.\n\n- Limited Compatibility Testing: The tokenizer is only tested within the LlamaGen framework. It is unclear how it would perform with other modern autoregressive visual generation models, such as EMU, Janus Pro, or MAR. This limited scope makes it difficult to assess the general applicability of the tokenizer.\n\n- Lack of Efficiency Analysis: The paper does not include any comparison of latency or inference speed against baseline tokenizers. This is an important aspect for a component like a tokenizer.\n\n- The tokenizer's ability to scale and generalize to more realistic, complex tasks is not demonstrated. Key evaluations, such as performance in text-to-image generation scenarios, are missing.", "was_revised": true, "final_justification": "Thanks the author for the detailed rebuttal.  I'm in general satisfied with the rebuttal with the comparison with more baselines and the efficiency comparisons and willing to increase the score. However I still feel the missing comparison with the SOTA models makes the paper weaker.", "rebuttal": "Dear reviewer `KT8u`,\n\nThank you for your thorough review of our paper and detailed suggestions for empirical results. We have made every effort to add the evaluations, which provided a more complete validation of our method, and will be incorporated to improve the paper. Your concerns and questions are answered as follows:\n\n> **W1:** Missing comparisons to tokenizers like VQ-VAE, MAGVIT-v2, and MAETok.\n\nThanks for your reminder. We are sorry that a fair comparison with the discrete tokenizers VQ-VAE and MAGVIT-v2 is not possible due to their absence of officially released checkpoints. Nevertheless, we are able to compare with the strongest suggested baseline MAETok. As MAETok is a continuous VAE, we also convert VFMTok into a continuous one (denoted as VFMVAE), train and measure its image reconstruction quality on ImageNet. A comparison is listed below. We’ll incorporate this into the manuscript for a more detailed comparison.\n\n|Tokenizer| rFID($\\downarrow$)|rIS($\\uparrow$)|\n| :--- |:---:|:---:|\n|MAETok|0.48|–|\n|VFMVAE |**0.46**|**224.4**|\n\n> **W2:** The tokenizer is only tested within the LlamaGen framework, and its performance with other AR frameworks like EMU, Janus Pro, or MAR is unclear.\n\nPlease note that 1) both EMU and MAR utilize continuous VAEs, while VFMTok is a discrete tokenizer. Besides, 2) EMU and Janus Pro are only evaluated on text-to-image generation tasks, while related works [23, 26, 44, 52, 55], including ours, mainly verify their effectiveness through class-to-image generation tasks. Thus, it is hard to perform an apple-to-apple comparison with other tokenizers under these AR frameworks.\n\nDuring the limited rebuttal period, we managed to integrate VFMTok into a recent AR framework, RandAR. As listed below, this integration allows VFMTok to achieve better image reconstruction and generation performance.\n\n|Tokenizer|\\#code|\\#tok|rFID($\\downarrow$)|rIS($\\uparrow$)|gFID($\\downarrow$)|gIS($\\uparrow$)|\n| :---| :---: |:---:|:---:|:---:|:---:|:---:|\n|RandAR|16384|256|2.19|172.4|2.55|288.8|\n|RandAR+VFMTok|16384|256|**0.89**|**215.4**|**2.25**|**300.6**|\n\nTo better verify the compatibility of VFMTok, we also integrated it into the MaskGiT[4] framework. For fairness, the images are generated without Classifier-Free Guidance (CFG). VFMTok presents consistent improvement, further supporting its compatibility with AR and non-AR generation frameworks.\n\n|Generator  | gFID($\\downarrow$) | gIS($\\uparrow$) |\n|:---|:---:|:---:|\n|MaskGiT[4] | 6.18 | 182.1|\n|MaskGiT[4]+VQGAN[43]| 7.25 |139.8|\n|MaskGiT[4]+VFMTok|**4.93**|**190.4**|\n\n> **W3:** Lack of comparison of latency or inference speed against baseline tokenizers.\n\nBelow, we show the decoding (inference) computation and throughput of tokenizers under different resolutions, where VFMTok is faster at higher resolutions thanks to the fixed-length tokens. Note that the comparison does not consider the time for AR generation of tokens. When considered, VFMTok requires a fixed length of 256 tokens, while VQGAN requires #token quadratic to the resolution, the inference efficiency would be more profound.\n|Tokenizer|Resolution|Decoding GMac|Inference Throughput($\\uparrow$,imgs/s)|\n| :--| :---:|:---:|:---:|\n|VQGAN[43]|256|126.63|370.38|\n| VFMTok|256|104.24|343.87|\n| VQGAN[43]|384|285.67|162.24|\n| VFMTok|336|176.08|205.72|\n\n> **W4:** Performance in text-to-image generation scenarios is missing.\n\nThanks for your suggestion. Due to the limited rebuttal period, we could only supplement with class-to-image generation experiments. Besides, highly related works such as ImageFolder [26], VAR [44], ViTVQGAN [52], TiTok [55], and RQ-VAE [23] also primarily validate their effectiveness through class-to-image generation tasks. We will continue to work towards the goal of text-to-image generation in the future, which, however, might be out of the scope of this paper.\n\n> **Q1:** What is the performance of VFMTok on higher-resolution images?\n\nPer your request, we added an experiment on image generation at $512\\times512$ resolution. All AR models (VFMTok-L) are trained for 100 epochs and tested without CFG. As shown below, our method is consistently stronger on higher resolution settings.\n|Generator| rFID($\\downarrow$) | rIS($\\uparrow$)|gFID($\\downarrow$,w/o cfg)|gIS($\\uparrow$, w/o cfg)| gFID($\\downarrow$, w/ cfg)| gIS($\\uparrow$, w/ cfg)|\n| :-- | :---:|:---:|:---:|:---:|:---:|:---:|\n| BigGAN|–|–|8.43|232.5|--|--|\n|  ADM|–|–|23.24|58.06 |--|--|\n|MaskGiT[4]|–|–|7.32|156.0 |--|--|\n|VQGAN[43]|**0.49**|211.1|14.50|77.1|4.03|224.7|\n|VFMTok|0.65|**233.7**|**2.60**|**191.4**|**3.08**|**285.0**|\n\n> **Q2:** The region-adaptive quantization framework appears to be general. How would it perform if applied to other tokenizers?\n\nFor simplicity, we adapted VQGAN[43] by replacing its PatchGAN with DinoGAN and inserting a region-adaptive module, constructing VQGAN-D and VQGAN-D-RA. The training setup is: tokenizer (50 epochs), VFMTok-L (100 epochs). As shown in the table below, it also improves image generation quality.\n|Tokenizer| rFID($\\downarrow$) | rIS($\\uparrow$)   | gFID($\\downarrow$)  |  gIS($\\uparrow$)   | \n| :--|:---:|:---:|:---:|:---:|\n| VQGAN-D|**0.80**|193.2|3.46|**230.9**|\n| VQGAN-D-RA|0.88|**198.3**|**3.39**|230.1|\n\n> **Q3:** Could you provide some direct qualitative comparisons?\n\nWe are sorry that due to the rebuttal policy, we cannot include new figures at present. We will incorporate them into the paper in a future revision following your suggestions."}, {"review_id": "ugboDwOOci", "reviewer": "Reviewer_APhE", "summary": "This paper proposes VFMTok, an image tokenizer built on frozen vision foundation models like DINOv2 and CLIP. It introduces region-adaptive quantization to reduce feature redundancy and a semantic reconstruction objective to preserve high-level semantics. VFMTok enables efficient autoregressive image generation, achieving 3× faster convergence over baselines.", "questions": "Please refer to the weakness.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. This paper is easy to follow.\n2. The authors introduce deformable sampling to improve token representation efficiency by enabling flexible, content-aware feature selection. This approach reduces redundancy in dense encoder features and naturally incorporates multi-scale information, enhancing the expressiveness of the resulting tokens.\n3. The authors conduct ablation studies on various vision foundation models and core components of their framework, validating the effectiveness of each design choice.\n\nWeakness: \n1. The use of vision foundation model (e.g., DINO v2, CLIP, SigLIP) pretraining for tokenizers is not novel, as several recent works—particularly in the VAE/VQVAE domain (e.g., LightningDiT, VILA-U, VQGAN-LC, SoftVQ-VAE)—have explored similar directions. \n2. The generation results are not particularly impressive, showing only limited improvements over the baseline LlamaGen. Moreover, the paper lacks comparisons with more recent methods, such as PAR and RandAR from CVPR 2025. Notably, the proposed tokenizer does not outperform non-VFM-based tokenizers and in some cases even underperforms them. Additionally, linear probing on ImageNet-1K using tokens produced by the VFM-based tokenizer yields significantly lower accuracy compared to directly using the original VFM features. These observations raise concerns about the overall effectiveness of the approach.\n3. Although deformable sampling reduces FLOPs theoretically, its dynamic offset and weight calculations can cause slower inference in practice due to complex computations, limited parallelization, and inefficient memory access. The paper lacks an experimental comparison of inference speed to assess this issue.\n4. There is a concern regarding the evaluation setting: generating images at 384×384 resolution and then resizing to 256×256 for evaluation may lead to unfair comparisons, potentially inflating performance metrics.", "was_revised": true, "final_justification": "After reading the authors' rebuttal and the responses from other reviewers, I have decided to lower my score to borderline reject. Using VFM as a tokenizer is not uncommon, and while the method introduces high-level semantics, it provides limited evaluation on semantic understanding tasks, with only moderate results on image classification. On the generative side, under a fair setting with 256 tokens, the performance lags significantly behind state-of-the-art autoregressive models (as shown in Appendix Table A1).", "rebuttal": "Dear reviewer `APhE`,\n\nThank you for your time and efforts in helping us improve our paper! We hereby answer your questions and resolve your concerns as follows:\n\n> **W1:** The use of VFM for tokenizers is not novel. LightningDiT, VILA-U, VQGAN-LC, SoftVQ-VAE have explored similar directions.\n\nThank you for bringing these works to our attention. Despite using VFM for VAE/VQVAEs, we note clear distinctions from these works. We would like to clarify that our motivation is to show that **frozen** pre-trained VFM features **alone** are sufficient to build an image tokenizer. To achieve this, we leverage multi-level features from the VFM, and reduce redundancy via the region-adaptive quantization strategy. In contrast:\n\n- Approaches like LightningDiT and SoftVQ-VAE use a pre-trained VFM only as a feature distillation target for optimizing its tokenizer.\n- VQGAN-LC investigates initializing its codebook by heuristically clustering CLIP patch features, and its reconstruction and generation quality is substantially lower than VFMTok's.\n- VILA-U initializes its tokenizer's encoder with a pre-trained VFM, whose parameters are explicitly updated during training.\n\nIn conclusion, all of them cannot build a tokenizer with frozen VFMs solely. We are sorry for the confusion will incorporate the discussion into the manuscript for a clearer relation with related works. Besides, we provide a more detailed discussion on the primary purpose of the VFM in our response to Reviewer vMYn's first question (`vMYn`-**W1**). We kindly ask you to refer to that for more information.\n\n> **W2.1:** Lack of comparison with PAR and RandAR.\n\nPer your request, we integrated VFMTok into RandAR and also compared it with that of PAR. As shown in the table below, VFMTok achieves better image reconstruction and generation performance. We will incorporate the comparison to the manuscript for a more complete comparison with recent works.\n\n|Approach | #code| #tok |rFID($\\downarrow$) | rIS($\\uparrow$)|gFID($\\downarrow$)|gIS($\\uparrow$)|\n|      :---      |    :---:     |      :---:     |     :---:   |      :---:    |    :---:   |   :---:  |\n|RandAR+VQGAN[43]| 16384 |  256 | 2.19    | 172.4| 2.55 | 288.8 |\n|RandAR + MaskGiT[4] |1024 |256 |2.28 |171.4 |2.47 | 271.0|\n|PAR + VQGAN[43]|16384| 256 |2.19 | 172.4| 3.76| 218.9|\n|RandAR + VFMTok|16384|256 |**0.89**|**215.4**|**2.25**|**300.6**|\n\n> **W2.2:** VFMTok does not outperform non-VFM-based tokenizers; linear probing accuracy is lower than original VFM features.\n\nIn Table 2 of the manuscript, the image reconstruction quality of VFMTok is inferior to that of ImageFolder [26]. This is because the experiment setting is aligned with LlamaGen [43] baseline for a fair comparison. Once the codebook size and the number of tokens are aligned with those of ImageFolder [26], VFMToks's image reconstruction and linear probing accuracy surpass ImageFolder's, while its generation quality is comparable. The results aligned with ImageFolder are shown below.\n\n|Tokenizer |    #code.     |       #tok.     |    rFID($\\downarrow$)    |       rIS($\\uparrow$)  |  #epochs  |    gFID($\\downarrow$)    |    gIS($\\uparrow$)   | top-1($\\uparrow$)|\n|    :---:     |    :---:     |      :---:     |    :---:   |      :---: |   :---:   |    :---:   |   :---:  |:---:  |\n|ImageFolder[26]| $2\\times16384\\times32$  |     $286\\times2$     | 0.69 |  201.5 | 300 | 2.60 |295.0 |58.1|\n|VFMTok|   $32768\\times12$   | \t  400\t    | **0.60**  | **221.6** |    200    | **2.58** | 273.7 |**74.9** |\n\nBesides, there is a concern that VFMToks's linear probing accuracy is lower than that of the original VFM. This occurs because the quantization introduces information loss, consequently compromising the semantic fidelity of the tokens. \n\n> **W3:** Deformable sampling can cause slower inference; lacking experimental comparison of inference speed.\n\n- Please kindly note that deformable sampling is adopted only in the tokenizer's training stage. During inference (image generation), only the decoder of the tokenizer is used.\n- The question might be whether deformable sampling makes VFMTok slower to train. Thanks to the use of frozen VFM features, these can be cached, and the trainable module in the encoder is only a lightweight deformable sampler. Together, the training of our tokenizer is also substantially faster.\n- Below, we provide a complete comparison of the compute and throughput of different stages of the tokenizer across multiple resolutions. Both training and inference are efficient, especially as resolution increases.\n\n| Tokenizer     | Resolution | Encoding GMac | Decoding GMac | Training throughput ($\\uparrow$, img/sec) | Inference throughput ($\\uparrow$, img/sec)|\n| :-- | :---:|    :---:      |     :---:     |      :---:        |        :---:     |\n|VQGAN[43]|  256 |    69.36      |     126.63    |      205.46       |        370.38    |\n| VFMTok  |  256 |    9.39       |     104.24    |      286.43       |        343.87    |\n| VQGAN[43]|  384 |    156.62     |     285.67    |      89.84        |        162.24    |\n| VFMTok  |  336 |    9.39       |     176.08    |      183.67       |        205.72    |\n\n> **W4:** Generating images at 384×384 resolution and then resizing to 256×256 may lead to unfair comparisons.\n\n- This follows a common setting proposed in LlamaGen[43] and PAR, where comparisons are often conducted by generating images at $384\\times384$ resolution and subsequently resizing them to $256\\times256$ for evaluation. This paradigm requires 576 image tokens to represent an image.\n- Instead, VFMTok generates at a $336\\times336$ resolution and resizes to $256\\times256$ for evaluation, representing each image with just 256 tokens and thereby accelerating the AR generation speed.\n- For fairness, we also report the $256\\times256$ image generation performance in Appendix Table A1. Please kindly take a look for reference."}, {"review_id": "EVbnzAWKqC", "reviewer": "Reviewer_SUoJ", "summary": "This paper proposes VFMTok, an image tokenizer built on frozen vision foundation models. It introduces region-adaptive quantization and a semantic reconstruction objective to reduce redundancy and preserve semantics. The method improves image generation quality, achieves a gFID of 2.07 on ImageNet, and enables class-conditional synthesis without classifier-free guidance.", "questions": "See weaknesses.", "limitations": "The authors discussed both the limitations and potential negative social impacts.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper is well-written and easy to follow\n- The method achieves competitive reconstruction quality and generation quality.\n- The paper achieves strong generative performance without relying on classifier-free guidance.\n\n\n\n**Weaknesses:**\n\n- The paper lacks novelty, as using vision foundation models as image tokenizers to improve VQGAN has already been explored in prior work[1][2].\n- Could you provide a detailed explanation of why the model achieves better performance without classifier-free guidance than those that incorporate it?\n- The paper lacks a thorough analysis of the *Region-adaptive Token Generation* module. While the authors claim that it “adaptively samples regions of similar patterns and extracts their VFM features for quantization,” it remains unclear whether the model truly behaves in this way. A deeper investigation into which aspects of the multi-level features are most beneficial for reconstruction would be important to support this claim and better understand the effectiveness of the approach.\n- According to TiTok, MaskGIT-style generation offers higher efficiency compared to autoregressive methods. Has the author considered or experimented with such a decoding strategy for this work?\n\n\n\n----\n\n[1] Chen Z, Wang C, Chen X, et al. Semhitok: A unified image tokenizer via semantic-guided hierarchical codebook for multimodal understanding and generation[J]. arXiv preprint arXiv:2503.06764, 2025.\n\n[2] Qu L, Zhang H, Liu Y, et al. Unified image tokenizer for multimodal understanding and generation[J]. arXiv preprint arXiv:2412.03069, 2024.", "was_revised": true, "final_justification": "I have carefully reviewed the authors’ rebuttal and found that it successfully addresses my concerns. While I find the use of VFM to improve quantization not particularly novel, the strong performance achieved without classifier-free guidance is quite surprising. I believe there may be an underlying reason behind this phenomenon that is worth deeper investigation. Therefore, I am currently inclined to raise my score to ``Borderline Accept''. I encourage the authors to further analyze and explore this aspect in future work.", "rebuttal": "Dear reviewer `SUoJ`,\n\nThank you for reviewing our paper. Your concerns are answered as follows.\n\n> **W1:** The paper lacks novelty, as using VFMs as image tokenizers to improve VQGAN has already been explored in prior work (SemHiTok and TokenFlow).\n\n- Our motivation is to show that **frozen** VFM features **alone** are sufficient to build an image tokenizer.\n- Instead, both SemHiTok and TokenFlow adopt a dual-encoder approach, having one VQGAN branch for pixel-level features and a VFM branch for high-level semantics. VFMTok leverages the multi-level features from a single-branch VFM to capture both low-level and high-level features that benefit image reconstruction and generation.\n- Moreover, we also designed a light-weight region-adaptive quantization strategy to reduce redundancy in irregular image regions.\n- Besides, we provide a more detailed discussion on the primary purpose of the VFM in our response to Reviewer vMYn's first question (`vMYn`-**W1**). We kindly ask you to refer to that for more information.\n- Together, VFMTok sets apart prior works in both motivation and model architecture.\n- Thank you for bringing these works to our attention, and we will add a citation for discussion.\n\n> **W2:** Why does the model achieve better performance without classifier-free guidance than those that incorporate it?\n\n- Our intuition is that introducing semantic information renders the latent tokens semantically aware and helps stabilize the sampling process of the autoregressive (AR) model, thereby reducing the reliance on classifier-free guidance (CFG).\n- As shown in our results, the performance in both settings is comparable, with the CFG-free setup achieving slightly better gFID. However, we also observe that incorporating CFG slightly improves the gIS, indicating enhanced diversity. This reflects the typical trade-off between fidelity (gFID) and diversity (gIS) commonly seen in CFG-based approaches.\n- We will clarify these observations and the underlying trade-offs in the paper.\n\n> **W3.1:** Whether the region-adaptive token generation module adaptively sample regions of similar patterns?\n\n- Within the region-adaptive generation module, each region contains multiple sampling points. We utilize these points to sample from the last spatial features of the original DINOv2, creating a feature representation for each point.\n- These features are then aggregated via average pooling to form the representation for the entire region. The average similarity between each intra-region sampling point and the region feature is then computed. Additionally, we randomly sample points from outside the region, extract their features, and compute their similarity to the same region feature.\n- Below, $S_{in}$ and $S_{in}$ denote the sampling points within and outside a sampling region, respectively.\n\n| |  similarity($\\uparrow$)|\n|:---:|:--:|\n| $S_{in}$| 0.860 |\n|$S_{out}$| 0.337 |\n\nAs presented in the table above, it indicates a significant similarity among points *within* the region, whereas points from *outside* the region exhibit comparatively low similarity, which consolidates our hypothesis.\n\n> **W3.2:** Which aspects of the multi-level features are most beneficial for reconstruction?\n\nAccording to the suggested setup, we begin by conducting an ablation study to evaluate the impact of each layer on image reconstruction and generation. Subsequently, we cumulatively add each feature layer to observe the combined effect.\n\nThe results, as presented in the table, show that while a single feature layer offers no significant advantage, the quality of both image reconstruction and generation markedly improves as more layers are incorporated.\n\nAll tokenizers and AR generation models (VFMTok-L) are trained for 50 and 100 epochs, respectively. As shown in the tables, our results show that while individual layers provide little benefit, cumulatively adding more layers significantly improves both image reconstruction and generation quality. $F_i$ represents the indexed feature level.\n\n(a) Single-level feature investigation\n| $F_i$| rFID($\\downarrow$)|rIS($\\uparrow$)|gFID($\\downarrow$)|gIS($\\uparrow$)|\n| :---:|:---:|:---:|:---:|:---:|\n|   1   | 1.04 | 186.4 | 3.84 | 257.9 |\n| 2|  0.95 | 200.6 | 3.79 | 272.7 |\n|3 | 1.03 |  208.5 | 3.69 | 274.9 |\n| 4|  1.23 | 214.8 | 3.64 |  277.7 |\n\n(b) Analysis of cumulatively adding multi-level features\n| $F_i$| rFID($\\downarrow$) |rIS($\\uparrow$)|gFID($\\downarrow$)|gIS($\\uparrow$)|\n| :---:|:---:|:---:|:---:|:---:|\n|   1   | 1.04 | 186.4 | 3.84 | 257.9 |\n|   +2  | 0.94 | 205.0 | 3.69 | 274.4 |\n|   +3  | 0.94 | 210.8 | 3.27 | 272.5 |\n|   +4  | 0.89 | 215.4 | 3.09 | 274.2 |\n\n> **W4:** Has the author considered or experimented with MaskGIT decoding strategy for this work?\n\nWe added an experiment that integrates VFMTok into MaskGiT [4]. For fairness, the images are generated without Classifier-Free Guidance (CFG).\n| Approach  | gFID($\\downarrow$)| gIS($\\uparrow$)|\n| :---|:---:|:---:|\n|MaskGiT [4] | 6.18 | 182.1|\n|MaskGiT [4] + VQGAN[43]| 7.25 |139.8|\n|MaskGiT [4] + VFMTok|**4.87**|**194.0**|"}, {"review_id": "WIRC0zx3K4", "reviewer": "Reviewer_vMYn", "summary": "This paper proposes using vision foundation models (VFMs) such as CLIP and DINOv2 as part of an image tokenizer to be used for training visual autoregressive generative models. In particular, they leverage frozen VFMs as the encoding part of a VQ-VAE style discrete tokenization framework. Given an image, VFM extracts features from it at several levels (or layers). Afterwards, these features are resampled by a Transformer with deformable cross-attention layers to reduce redundancy and then quantized. The quantized features are then fed as input to another lightweight Transformer whose goal is to both predict the original VFM features (to preserve the semantics) as well as image latent features. The latter is fed to the decoder of VQ-VAE to reconstruct the input image. After the tokenizer is trained, the resulting discrete tokens can be used to train an autoregressive Transformer for class-conditioned image generation, which they did on ImageNet to show the effectiveness of the method.", "questions": "Please see the strengs & weaknesses for details.\n\n- How do you compare against recent work leveraging VFMs as part of a visual generation pipeline? Both conceptually and empirically.\n\n- What is the performance of different VFMs within the proposed framework? What are their characteristics leading to varying performance?\n\n- Could you clarify the questions regarding evaluation setup? First of all, I think you should also report total training FLOPs, as right now it's a bit unclear as you are operating at a different resolution. Second, some reported numbers are confusing. For example, TiTok reimplementation results are much worse than the original numbers reported, could you explain why? \n\n- What are the contributions of different aspects of the method? For example, I would be curious to see 1) whether other VFMs show similar trends and 2) what if you simply extract multi-level features from frozen VFMs without region-adaptive tokenization and with & without reconstructing the features. I also found it interesting that linear proping accuracy of the randomly initialized encoder is much better than vanilla VQGAN (15.0 vs 59.1). Could it be because of multi-level features and the reconstructing the features as an objective? More clarifications and analysis would be nice. Could you also clarify if running the ablations with 50 epoch training leading to stable results?\n\n- What would be the performance on other generative benchmarks such as COCO and GenEval? \n\n- What are the limitations of the method? What are possible future directions to investigate?", "limitations": "No limitation is provided. I provided some pointers on the strengths and weaknesses section on several aspects that can be improved.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Improving efficiency and effectiveness of autoregressive visual generation models is a timely topic. The paper observes a critical point about the convergence issues of VQGAN-style tokenizers and proposes using semantically rich features from VFMs. This is not a new idea, as several papers previously investigated it [A] [B] [C] [D], yet the paper further boosts performance with a region-adaptive tokenization strategy, reducing the total number of tokens required. While I view this to be an interesting direction, there are several issues that must be addressed first:\n\n- Missing related work. As I mentioned, using VFMs as part of tokenization pipeline is not a new idea. I'd expect to see a dedicated related work section on this and contrasting the paper to those approaches.\n\n- Importantly, recent approaches like REPA showed promising results in terms of improving convergence speed by aligning representations to a VFM, namely DINOv2. FlexTok used this idea to improve the quality and efficiency of discrete image tokenizers. Since this paper claims similar gains, a direct analysis and comparison would be useful to those REPA-based approaches.\n\n- I found the evidence provided in L181 weak. Why do we think VFMs trained with masked prediction objective do better than CLIP? What about MAE then? I think, overall, a more clear and focused analysis on the performance of different VFMs is needed to make the papers claims stronger.\n\n- Evaluation setting and some evaluation results require further clarifications. First of all, I think you should also report total training FLOPs, as right now it's a bit unclear as you are operating at a different resolution. Second, some reported numbers are confusing. For example, TiTok reimplementation results are much worse than the original numbers reported, could you explain why? \n\n- Ablations are interesting, I would be curious to see 1) whether other VFMs show similar trends and 2) what if you simply extract multi-level features from frozen VFMs without region-adaptive tokenization and with & without reconstructing the features. I also found it interesting that linear probing accuracy of the randomly initialized encoder is much better than vanilla VQGAN (15.0 vs 59.1). Could it be because of multi-level features and the reconstructing the features as an objective? More clarifications and analysis would be nice. Could you also clarify if running the ablations with 50 epoch training leading to stable results?\n\n- While I acknowledge ImageNet is a typical benchmark to show generative capabilities, I would be curious to see the performance on other benchmarks such as COCO and GenEval. In particular, as the paper claims, I would be curious if the semantic features from those VFMs are helpful in more challenging settings.\n\n- A dedicated limitation section is also needed.\n\n[A] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think\n\n[B] Making LLaMA SEE and Draw with SEED Tokenizer\n\n[C] Emu: Generative Pretraining in Multimodality\n\n[D] FlexTok: Resampling Images into 1D Token Sequences of Flexible Length", "was_revised": true, "final_justification": "Thanks for the further clarifications. Overall, rebuttal sufficiently addressed my concerns, thus I am happy to increase my rating to borderline accept.\n\nI agree with fellow reviewers SUoJ and APhE that the novelty of the submission is limited which is a factor for me to not increase the rating further. Still, some of the introduced components like region adaptive quantization and the analysis made in the paper might be relevant to the community.\n\nRebuttal wasn't enough at addressing my question on the VFMs trained with masked prediction objective doing better than CLIP. I guess we can pose this question more generally and say: What makes a VFM a good visual tokenizer? Having a solid answer to this would have improved the significance (and rating) of this paper a lot more.", "rebuttal": "Dear reviewer `vMYn`,\n\nThank you for your thorough review of our paper and detailed suggestions for improvement. Your questions and concerns are answered below:\n\n> **W1:** Using VFMs as part of tokenization pipeline is not a new idea. A dedicated related work section is needed.\n\nThanks for your suggestion. We note that our motivation is to show that **frozen** VFM features **alone** are sufficient to build an image tokenizer, while all previous works use VFM for other purposes or in other ways. To make a clear comparison, we provide a table on the purpose of using VFMs below: \n\n|Approach|Encoder initialization| Feature regularization|Conditional feature generation|Codebook initialization|\n|:---|:---:|:---:|:---:|:---:|\n|REPA[A]| | $\\checkmark$| | |\n|SEED[B]| |\t| $\\checkmark$| |\n|EMU[C] | | | $\\checkmark$|\t|\n|FlexTok[D]| |$\\checkmark$ || |\n|Lightning-DiT| | $\\checkmark$| | |\n|SoftVQ-VAE| | $\\checkmark$| | |\n|VILA-U| $\\checkmark$| | | |\n|SemHiTok| $\\checkmark$|$\\checkmark$|||\n|TokenFlow| $\\checkmark$|$\\checkmark$|||\n|VQGAN-LC| | | | $\\checkmark$|\n\n Table (a). The primary purpose of VFM.\n\n|Tokenizer|\\#code.|\\#tok.|rFID$\\downarrow$|rIS$\\uparrow$|gFID$\\downarrow$|gIS$\\uparrow$|\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|\n|VQGAN-LC[58]| 100000|256|2.62|--|15.4|--|\n|VFMTok-L|16384| 256|**0.89**|**215.4**|**2.11**|**230.1**|\n\nTable  (b). Performance comparsion between VQGAN-LC[58] and VFMTok.\n\n- As shown in the Table.(a), concerning the purpose of VFMs, prior methods have primarily used VFM for: 1) encoder initialization (e.g., EMU[C], VILA-U, SemHiTok, TokenFlow); 2) feature regularization (e.g., REPA[A], Lighting-DiT, SoftVQ-VAE, Flextok[D]); 3) conditional feature generation for diffusion models (e.g., EMU, SEED[B]); and 4) codebook initialization (e.g., VQGAN-LC).\n- Moreover, approaches that utilize a VFM to initialize the tokenizer's encoder (such as EMU, VILA-U, and TokenFlow) all explicitly update the VFM's parameters during training.\n- Empirically, the most related work is VQGAN-LC[58], which investigates initializing its codebook by heuristically clustering CLIP patch features. Similarly, its parameters in VFM also remain frozen during the training phase. Nevertheless, its performance in image reconstruction and generation is substantially lower than that of VFMTok. As indicated in the Table.(b) above.\n- In conclusion, all of them cannot build a tokenizer with frozen VFMs solely. We will incorporate the discussion into the manuscript for a clearer relation with related works.\n\n> **W2:** REPA showed improved convergence by aligning to DINOv2. FlexTok used REPA to improve discrete image tokenizers. A comparison is needed.\n\n**Comparison with REPA.** Here we convert VFMTok into a continuous VAE (VFMVAE) and integrate it into SiT/XL. Image reconstruction and generation performance are presented below. For simplicity and fairness, images are generated without classifier-free guidance(CFG).\n\n|| rFID($\\downarrow$)| rIS($\\uparrow$)| gFID($\\downarrow$)| gIS($\\uparrow$)|\n|:---| :---: | :---: | :---: | :---: |\n|SiT-XL|0.61|212.1|8.30|131.7|\n|SiT-XL (REPA)|0.61|212.1|5.90|157.8|\n|SiT-XL (VFMVAE)|**0.46**|**224.4**|**3.08**|**175.0**| \n\n**Comparison with FlexTok.** VFMTok achieves better reconstruction and generation quality with approximately half of the parameters in the AR generation model.\n\n||\\#code.|\\#tok.|rFID($\\downarrow$)| rIS($\\uparrow$)|AR params.|gFID($\\downarrow$)| gIS($\\uparrow$)|\n|:---|:---|:---|:---|:---|:---|:---|:---|\n|FlexTok|64000|256|1.08|--|1.3B|2.45|258.3|\n|VFMTok (XXL)|16384|256|**0.89**|**215.4**|**775M**|**2.41**|**276.8**|\n\n> **W3:** Why do we think VFMs trained with masked prediction objective do better than CLIP? What about MAE then?\n\n- In this setup, the encoder of VFMTok is substituted with MAE-L. Training setup: VFMTok (50 epochs), AR model, VFMTok-L (100 epochs). As shown in the table below, while the generation and reconstruction quality of VFMTok (MAE) does not match that of VFMTok (CLIP), it demonstrates a significant improvement over the baseline. Moreover, its reconstruction performance is better than that of VQGAN (CLIP).\n- This suggests that VFMTok, integrating pre-trained models with a masked prediction objective, is advantageous for image reconstruction. Nevertheless, due to factors such as the VFM's model size, the volume of training data, and time constraints, VFMTok (MAE) is not yet comparable to VFMTok (CLIP).\n- We will investigate this further and provide a more complete discussion in the paper.\n\n||\\#code.|\\#tok.|rFID($\\downarrow$)| rIS($\\uparrow$)|gFID($\\downarrow$)| gIS($\\uparrow$)|\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|\n|VQGAN[43]|16384|576|0.95|197.3|3.71|228.3|\n|VQGAN (CLIP)|16384|576|1.47|182.0|3.45|221.2|\n|VFMTok (CLIP)|16384|256|0.99|**200.1**|3.40|**274.7**|\n|VFMTok (MAE)|16384|256|**0.98**|191.8|**3.34**|258.9|\n\n> **W4.1:** Total training FLOPs should be reported, as currently you operate at a different resolution.\n\n- We presented the required training computation and throughput for VFMTok below.\n- Thanks to the use of frozen VFM features, these can be cached, and the trainable module in the encoder is only a lightweight deformable sampler. Together, the training of our tokenizer is also substantially faster.\n- Besides, we have also included 256x256 generation experiments in Appendix Table A1 for reference.\n\n| Tokenizer     | Resolution | Encoding GMac | Decoding GMac | Training throughput ($\\uparrow$,img/sec) |\n| :-- | :---:|    :---:      |     :---:     |      :---:        |\n|VQGAN[43]|  256 |    69.36      |     126.63    |      205.46       |\n| VFMTok  |  256 |    9.39       |     104.24    |      286.43       |\n| VQGAN[43]|  384 |    156.62     |     285.67    |      89.84       |\n| VFMTok|  336 |    9.39       |     176.08    |      183.67       |\n\n> **W4.2:** TiTok reimplementation results require further clarification.\n\nWe used the official TiTok weights, but integrated with an AR model (LlamaGen) for fair comparison with other methods instead of the original MaskGiT.  We’ll revise the text for clarity.\n\n> **W5.1:** Whether other VFMs show similar trends?\n\nOther VFM, such as VFMTok (SigLIP), also show a similar trend, which is presented in Table A6 in the Appendix.\n\n> **W5.2:** What about w/o region-adaptive tokenization, and w/ & w/o reconstructing the features?\n\nBelow, we show the image reconstruction performance under the suggested settings, where the first two rows do not use region-adaptive tokenization. We train VFMTok for 50 epochs, while the AR models are trained for 100 epochs.\n\n|Tokenizer|\\#code.|\\#tok.|rFID($\\downarrow$)| rIS($\\uparrow$)|\n|:---|:---:|:---:|:---:|:---:|\n|w/o VFM recon.|16384|576|0.63|207.3|\n|w/ VFM recon.|16384|576|0.71|212.8|\n|VFMTok|16384|576|0.89|215.4|\n\n> **W5.3:** If running the ablations with 50 epoch training lead to stable results?\n\nWe list the AR generation quality at both 50 and 100 epochs in the table below. The trend is consistent.\n\n|AR model|\\#epochs|gFID($\\downarrow$)| gIS($\\uparrow$)|\n|:---|:---:|:---:|:---:|\n|w/o VFM recon|50|3.56|252.0|\n|w/ VFM recon|50|3.58|274.8|\n|VFMTok|50|**3.42**|**277.3**|\n|w/o VFM recon.|100|3.43|273.5|\n|w/ VFM recon.|100|3.20|274.0|\n|VFMTok|100|**3.09**|**274.2**|\n\n> **W5.4:** Why linear probing accuracy of the randomly initialized encoder is much better than vanilla VQGAN?\n\n- To analyze the cause of VQGAN [43]'s low linear probing accuracy, we begin by substituting the PatchGAN in VQGAN with DinoGAN. Subsequently, we incorporate the objective of feature reconstruction.\n- The table below shows that adding DinoGAN, with its semantic discriminability, remarkably improves VQGAN's accuracy. VFM feature reconstruction further boosts this accuracy.\n- We can therefore conclude that a semantically-informed loss function enhances the linear probing accuracy of VQGAN.\n|Tokenizer|top-1($\\uparrow$,\\%)|\n|:---|:---:|\n|VQGAN[43]|15.0|\n|VQGAN[43]+DinoGAN|40.5|\n|+Semantic recon.|58.3|\n\n> **W6:** What would be the performance on other generative benchmarks such as COCO and GenEval?\n\nTime constraints prevented us from completing the text-to-image generation task for evaluation on COCO and GenEval. We will continue to work on this and integrate VFMTok with a text-to-image AR generation model to conduct these evaluations in the future. Thanks for your suggestion!\n\n> **W7:** What are the limitations of the method? What are possible future directions to investigate?\n\nThe limitations section is presented in the Appendix (page 4), and we will elaborate more on this part in the next revision."}]

Author rebuttal:

Meta-review: All reviewers found the paper was well written and its core idea of using pre-trained vision foundation models (VFMs) as tokenizers to be solid, well-motivated, and interesting. 
The reviewers agreed that the experimental setup provided a strong validation of the method and its components, and were impressed by the generation performance without relying on classifier-free guidance.

There were concerns about the novelty of the method compared to the large body of work on pretrained tokenizers. There were also concerns about the scope of the experimental setup that focuses only on image generation rather then image understanding - a natural thing to test when relying on tokens that come from a pretrained VFM model. In addition, the reviewers requested more information and experiments.

In the discussion period, the authors provided the main additional experiments that were requested by the reviewers, and further emphasized the novelty of the approach as the first to demonstrate that features from a frozen VFM alone are sufficient for high-quality generation. They also emphasized that the scope of the paper was to test image generation only (without testing any semantic understanding tasks).

While most of the reviewers acknowledged that the main concerns were addressed and raised their score to recommend acceptance, one reviewer is still of the opinion that the novelty compared to prior work is limited, in particular because the semantic capacity of the tokens are not tested.

After reading all the reviews and discussions with the authors and among reviewers, I think this paper should be accepted, provided that the authors update the final version with all the results and clarifications discussed.
Decision: Accept (poster)