Title: Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Image tokenizer; Autoregressive Image Generation; Vision Foundation Models;
Abstract: In this work, we present a novel direction to build an image tokenizer directly on top of a frozen vision foundation model, which is a largely underexplored area. Specifically, we employ a frozen vision foundation model as the encoder of our tokenizer. To enhance its effectiveness, we introduce two key components: (1) a region-adaptive quantization framework that reduces redundancy in the pre-trained features on regular 2D grids, and (2) a semantic reconstruction objective that aligns the tokenizer’s outputs with the foundation model’s representations to preserve semantic fidelity. Based on these designs, our proposed image tokenizer, \textbf{\ours}, achieves substantial improvements in image reconstruction and generation quality, while also enhancing token efficiency. It further boosts autoregressive (AR) generation---achieving a gFID of \textbf{1.36} on ImageNet benchmarks, while accelerating model convergence by \textbf{three times}, and enabling high-fidelity class-conditional synthesis without the need for classifier-free guidance (CFG). The code is available at \href{https://github.com/CVMI-Lab/VFMTok}{https://github.com/CVMI-Lab/VFMTok}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes using vision foundation models (VFMs) such as CLIP and DINOv2 as part of an image tokenizer to be used for training visual autoregressive generative models. In particular, they leverage frozen VFMs as the encoding part of a VQ-VAE style discrete tokenization framework. Given an image, VFM extracts features from it at several levels (or layers). Afterwards, these features are resampled by a Transformer with deformable cross-attention layers to reduce redundancy and then quantized. The quantized features are then fed as input to another lightweight Transformer whose goal is to both predict the original VFM features (to preserve the semantics) as well as image latent features. The latter is fed to the decoder of VQ-VAE to reconstruct the input image. After the tokenizer is trained, the resulting discrete tokens can be used to train an autoregressive Transformer for class-conditioned image generation, which they did on ImageNet to show the effectiveness of the method.\n\nSTRENGTHS AND WEAKNESSES:\nImproving efficiency and effectiveness of autoregressive visual generation models is a timely topic. The paper observes a critical point about the convergence issues of VQGAN-style tokenizers and proposes using semantically rich features from VFMs. This is not a new idea, as several papers previously investigated it [A] [B] [C] [D], yet the paper further boosts performance with a region-adaptive tokenization strategy, reducing the total number of tokens required. While I view this to be an interesting direction, there are several issues that must be addressed first:\n\n- Missing related work. As I mentioned, using VFMs as part of tokenization pipeline is not a new idea. I'd expect to see a dedicated related work section on this and contrasting the paper to those approaches.\n\n- Importantly, recent approaches like REPA showed promising results in terms of improving convergence speed by aligning representations to a VFM, namely DINOv2. FlexTok used this idea to improve the quality and efficiency of discrete image tokenizers. Since this paper claims similar gains, a direct analysis and comparison would be useful to those REPA-based approaches.\n\n- I found the evidence provided in L181 weak. Why do we think VFMs trained with masked prediction objective do better than CLIP? What about MAE then? I think, overall, a more clear and focused analysis on the performance of different VFMs is needed to make the papers claims stronger.\n\n- Evaluation setting and some evaluation results require further clarifications. First of all, I think you should also report total training FLOPs, as right now it's a bit unclear as you are operating at a different resolution. Second, some reported numbers are confusing. For example, TiTok reimplementation results are much worse than the original numbers reported, could you explain why? \n\n- Ablations are interesting, I would be curious to see 1) whether other VFMs show similar trends and 2) what if you simply extract multi-level features from frozen VFMs without region-adaptive tokenization and with & without reconstructing the features. I also found it interesting that linear probing accuracy of the randomly initialized encoder is much better than vanilla VQGAN (15.0 vs 59.1). Could it be because of multi-level features and the reconstructing the features as an objective? More clarifications and analysis would be nice. Could you also clarify if running the ablations with 50 epoch training leading to stable results?\n\n- While I acknowledge ImageNet is a typical benchmark to show generative capabilities, I would be curious to see the performance on other benchmarks such as COCO and GenEval. In particular, as the paper claims, I would be curious if the semantic features from those VFMs are helpful in more challenging settings.\n\n- A dedicated limitation section is also needed.\n\n[A] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think\n\n[B] Making LLaMA SEE and Draw with SEED Tokenizer\n\n[C] Emu: Generative Pretraining in Multimodality\n\n[D] FlexTok: Resampling Images into 1D Token Sequences of Flexible Length\n\nQUESTIONS:\nPlease see the strengs & weaknesses for details.\n\n- How do you compare against recent work leveraging VFMs as part of a visual generation pipeline? Both conceptually and empirically.\n\n- What is the performance of different VFMs within the proposed framework? What are their characteristics leading to varying performance?\n\n- Could you clarify the questions regarding evaluation setup? First of all, I think you should also report total training FLOPs, as right now it's a bit unclear as you are operating at a different resolution. Second, some reported numbers are confusing. For example, TiTok reimplementation results are much worse than the original numbers reported, could you explain why? \n\n- What are the contributions of different aspects of the method? For example, I would be curious to see 1) whether other VFMs show similar trends and 2) what if you simply extract multi-level features from frozen VFMs without region-adaptive tokenization and with & without reconstructing the features. I also found it interesting that linear proping accuracy of the randomly initialized encoder is much better than vanilla VQGAN (15.0 vs 59.1). Could it be because of multi-level features and the reconstructing the features as an objective? More clarifications and analysis would be nice. Could you also clarify if running the ablations with 50 epoch training leading to stable results?\n\n- What would be the performance on other generative benchmarks such as COCO and GenEval? \n\n- What are the limitations of the method? What are possible future directions to investigate?\n\nLIMITATIONS:\nNo limitation is provided. I provided some pointers on the strengths and weaknesses section on several aspects that can be improved.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes VFMTok, an image tokenizer built on frozen vision foundation models. It introduces region-adaptive quantization and a semantic reconstruction objective to reduce redundancy and preserve semantics. The method improves image generation quality, achieves a gFID of 2.07 on ImageNet, and enables class-conditional synthesis without classifier-free guidance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The paper is well-written and easy to follow\n- The method achieves competitive reconstruction quality and generation quality.\n- The paper achieves strong generative performance without relying on classifier-free guidance.\n\n\n\n**Weaknesses:**\n\n- The paper lacks novelty, as using vision foundation models as image tokenizers to improve VQGAN has already been explored in prior work[1][2].\n- Could you provide a detailed explanation of why the model achieves better performance without classifier-free guidance than those that incorporate it?\n- The paper lacks a thorough analysis of the *Region-adaptive Token Generation* module. While the authors claim that it “adaptively samples regions of similar patterns and extracts their VFM features for quantization,” it remains unclear whether the model truly behaves in this way. A deeper investigation into which aspects of the multi-level features are most beneficial for reconstruction would be important to support this claim and better understand the effectiveness of the approach.\n- According to TiTok, MaskGIT-style generation offers higher efficiency compared to autoregressive methods. Has the author considered or experimented with such a decoding strategy for this work?\n\n\n\n----\n\n[1] Chen Z, Wang C, Chen X, et al. Semhitok: A unified image tokenizer via semantic-guided hierarchical codebook for multimodal understanding and generation[J]. arXiv preprint arXiv:2503.06764, 2025.\n\n[2] Qu L, Zhang H, Liu Y, et al. Unified image tokenizer for multimodal understanding and generation[J]. arXiv preprint arXiv:2412.03069, 2024.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors discussed both the limitations and potential negative social impacts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes VFMTok, an image tokenizer built on frozen vision foundation models like DINOv2 and CLIP. It introduces region-adaptive quantization to reduce feature redundancy and a semantic reconstruction objective to preserve high-level semantics. VFMTok enables efficient autoregressive image generation, achieving 3× faster convergence over baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. This paper is easy to follow.\n2. The authors introduce deformable sampling to improve token representation efficiency by enabling flexible, content-aware feature selection. This approach reduces redundancy in dense encoder features and naturally incorporates multi-scale information, enhancing the expressiveness of the resulting tokens.\n3. The authors conduct ablation studies on various vision foundation models and core components of their framework, validating the effectiveness of each design choice.\n\nWeakness: \n1. The use of vision foundation model (e.g., DINO v2, CLIP, SigLIP) pretraining for tokenizers is not novel, as several recent works—particularly in the VAE/VQVAE domain (e.g., LightningDiT, VILA-U, VQGAN-LC, SoftVQ-VAE)—have explored similar directions. \n2. The generation results are not particularly impressive, showing only limited improvements over the baseline LlamaGen. Moreover, the paper lacks comparisons with more recent methods, such as PAR and RandAR from CVPR 2025. Notably, the proposed tokenizer does not outperform non-VFM-based tokenizers and in some cases even underperforms them. Additionally, linear probing on ImageNet-1K using tokens produced by the VFM-based tokenizer yields significantly lower accuracy compared to directly using the original VFM features. These observations raise concerns about the overall effectiveness of the approach.\n3. Although deformable sampling reduces FLOPs theoretically, its dynamic offset and weight calculations can cause slower inference in practice due to complex computations, limited parallelization, and inefficient memory access. The paper lacks an experimental comparison of inference speed to assess this issue.\n4. There is a concern regarding the evaluation setting: generating images at 384×384 resolution and then resizing to 256×256 for evaluation may lead to unfair comparisons, potentially inflating performance metrics.\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes using pre-trained vision foundation models (VFMs) as image tokenizers. To improve the tokenizer's performance, the authors introduce a region-adaptive quantization framework and a semantic reconstruction objective. Experimental results show that the proposed tokenizer, VFMTok, achieves competitive reconstruction performance and can improve the generation quality of LlamaGen on the ImageNet dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The motivation to use pre-trained vision foundation models as the basis for an image tokenizer is an interesting idea.\n\n- Experimental results on ImageNet demonstrate that the proposed VFMTok can improve the generation performance of the LlamaGen model.\n\n- The paper includes adequate ablation studies that help verify the effectiveness of each component within the proposed VFMTok.\n\nWeaknesses\n\n- Missing Baselines: The paper is missing comparisons to many important and relevant baseline tokenizers. For a thorough evaluation, VFMTok should be compared against methods like VQ-VAE, MAGVIT-v2, and MAETok, among others.\n\n- Limited Compatibility Testing: The tokenizer is only tested within the LlamaGen framework. It is unclear how it would perform with other modern autoregressive visual generation models, such as EMU, Janus Pro, or MAR. This limited scope makes it difficult to assess the general applicability of the tokenizer.\n\n- Lack of Efficiency Analysis: The paper does not include any comparison of latency or inference speed against baseline tokenizers. This is an important aspect for a component like a tokenizer.\n\n- The tokenizer's ability to scale and generalize to more realistic, complex tasks is not demonstrated. Key evaluations, such as performance in text-to-image generation scenarios, are missing.\n\nQUESTIONS:\n- What is the performance of VFMTok on higher-resolution images (e.g., 512x512 or 1024x1024)? The current experiments seem limited to lower resolutions.\n\n- The proposed region-adaptive quantization framework appears to be a general method. Is it independent of using a VFM-based tokenizer? How would this framework perform if applied to other existing tokenizers?\n\n- Could you provide some direct qualitative comparisons? For example, showing the same image reconstructed or generated using VFMTok versus other baseline tokenizers would make it easier to see the practical differences.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}