Title: End-to-End Vision Tokenizer Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Discrete Vision Tokenizer

Autoregressive Model

Multimodal Generation and Understanding
Abstract: Existing vision tokenization isolates the optimization of vision tokenizers from downstream training, implicitly assuming the visual tokens can generalize well across various tasks, e.g., image generation and visual question answering. The vision tokenizer optimized for low-level reconstruction is agnostic to downstream tasks requiring varied representations and semantics. This decoupled paradigm introduces a critical misalignment: The loss of the vision tokenization can be the representation bottleneck for target tasks. For example, errors in tokenizing text in a given image lead to poor results when recognizing or generating them. To address this, we propose ETT, an end-to-end vision tokenizer tuning approach that enables joint optimization between vision tokenization and target autoregressive tasks. Unlike prior autoregressive models that use only discrete indices from a frozen vision tokenizer, ETT leverages the visual embeddings of the tokenizer codebook, and optimizes the vision tokenizers end-to-end with both reconstruction and caption objectives. ETT can be seamlessly integrated into existing training pipelines with minimal architecture modifications. Our ETT is simple to implement and integrate, without the need to adjust the original codebooks or architectures of the employed large language models. Extensive experiments demonstrate that our proposed end-to-end vision tokenizer tuning unlocks significant performance gains, i.e., 2-6% for multimodal understanding and visual generation tasks compared to frozen tokenizer baselines, while preserving the original reconstruction capability. We hope this very simple and strong method can empower multimodal foundation models besides image generation and understanding.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes end-to-end tuning of the vision tokenizer to mitigate the misalignment errors steaming from decoupled training of the vision tokenizer and the target Vision-Language (VL) tasks. It proposes to jointly optimize the vision tokenizer and the LLM with the reconstruction and captioning objectives. This is achieved by using the continuous visual embeddings of the tokenizer codebook instead of the discrete indices from the frozen vision tokenizer.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Well written paper, easy to follow.\n* Strong results (though not SOTA) using less data and fewer LLM parameters compared to the evaluated methods.\n\nWeaknesses:\n* It is not clear to me how much of the performance boost reported in the experiments is obtained from the introduced curated SOL-recap data and how much from the proposed method. In other words, the experiments do not decouple data and fine-tuning contributions.\n* The novelty is somewhat limited. In my opinion, it is expected that fine tuning the models on downstream tasks improves the results.\n\nQUESTIONS:\n* As I wrote above, it is hard to judge from where the improvements in Table 2 and 3 come from. Is it because your curated data is of higher quality or is it because of the ETT method or both? I think it would be very valuable to decouple these, for example with an experiment where you train one of the benchmark methods on your data and re-evalute.\n* How does your approach affect the latency of the forward pass during inference?\n* Why decouple the training in Stage 1 and 2? What happens when Stage 1 training isn’t performed?\n* Maybe a naive question, but why do you need quantization at all? Couldn’t one work with continuous embeddings directly? Is this only because you are using pre-trained VQ tokenizers and not doing the training from scratch?\n* Minor:\n   - Table 3 shows strong results on all categories of the GenEval except for counting? Why is this one so degraded?\n   - In L 206, you wrote “ETT achieves superior performance compared to Show-o on wide range of benchmarks”. What I read from Table 2 is that  this only seems to be the case for 2 out of 8 reported benchmarks, and I find this statement too exaggerated.\n\nLIMITATIONS:\n/", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ETT, an end-to-end vision tokenizer that jointly optimizes both image reconstruction and downstream auto-regressive tasks. While prior works typically rely on pre-trained vision tokenizers that are kept frozen during downstream training, this paper demonstrates that integrating the tokenizer into the end-to-end training loop is straightforward and beneficial. By doing so, ETT helps unlock the tokenizer's potential for downstream understanding tasks, which are often hindered when the tokenizer is only trained for reconstruction objectives. Experiments show that ETT achieves competitive performance in both image understanding and generation.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper is well-organized and easy to follow, with a clean methodology, straightforward solutions, and clear experimental setups and comparisons.\n \n## Weaknesses\n\n- The technical novelty is somewhat limited. While the paper demonstrates that end-to-end training is feasible and beneficial, the core idea is relatively incremental.\n- In Table 4, the performance drop from row 1 to row 2—when replacing visual tokens from indices to embeddings—is surprisingly large. The paper should better explain why this change has such a significant impact on generation performance.\n- In the ablation study exploring the trade-off between understanding and generation, only rFID is reported for generation quality. It would be more informative to also include sample image outputs or additional quantitative metrics (e.g., FID, IS) to illustrate how generation quality degrades.\n- The paper should discuss how ETT could integrate with recent trends in jointly trained tokenizers for both understanding and generation, such as VILA-U or UniTok. Would the proposed method still provide gains in such setups?\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nPlease see weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an innovative methodology, termed End-to-End Vision Tokenizer Tuning (ETT), designed to address the inherent limitations of conventional vision tokenization techniques. The authors contend that the prevalent practice of optimizing tokenizers independently of downstream applications can result in substantial misalignments. Such discrepancies are particularly detrimental in tasks requiring sophisticated visual representations, including image generation and visual question answering. ETT facilitates the joint optimization of both the vision tokenizer and downstream autoregressive tasks, thereby leveraging visual embeddings to improve multimodal understanding and generative capabilities. The authors provide empirical evidence demonstrating a 2-6% performance enhancement over traditional frozen tokenizer pipelines, without compromising the original reconstruction quality.\n\nSTRENGTHS AND WEAKNESSES:\nPROS:\n\n1. I like the concept of end-to-end optimization of vision tokenizers, which is a significant advancement in the field. By integrating tokenizer training with downstream tasks, the authors address an important gap in existing methodologies.\n\n2. The extensive experiments show substantial improvements in multimodal understanding and generation tasks, with metrics indicating a clear advantage over existing models. This suggests that ETT can effectively enhance model performance with relatively simple modifications.\n\n3. The authors provide extensive experimental results across multiple benchmark datasets, reinforcing the robustness of their findings and demonstrating the versatility of their approach.\n\n4. ETT maintains the original reconstruction performance of the vision tokenizer, which is essential for tasks that rely on high-quality visual outputs.\n\n5. The paper is well-written, which makes it easy to follow.\n\n- CONS:\n\n1. The experiments are conducted using a specific vision tokenizer (IBQ-based) and a specific LLM (Qwen2.5-1.5B). While the results are strong, the paper does not explore whether the ETT method is equally effective with other popular vision tokenizers (e.g., VQGAN, MaskGIT) or different LLM families. While this is a reasonable scope limitation for a single paper, it leaves the question of the method's generalizability open. The authors rightly point to this as a direction for future work.\n\n2. The paper provides good qualitative examples of generated images and reconstruction quality. However, the analysis could be more impactful. For instance, to support the claim that ETT improves high-level semantics like text rendering, a direct side-by-side comparison showing a failure case of the baseline model (e.g., garbled text in an image) that is corrected by the ETT-tuned model would be highly compelling.\n\n3. The paper presents a promising proof of concept for ETT, but questions persist regarding its scalability and generalizability. Key concerns include how ETT’s benefits evolve with larger LLMs (e.g., 70B+ parameters) and bigger datasets, as a powerful LLM might mitigate a frozen tokenizer’s limitations. Additionally, its applicability to non-autoregressive architectures (like MaskGIT) and other modalities (video, audio) remains unclear, with challenges such as capturing temporal consistency in video. At least, they should be discussed.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new approach to visual tokenization that can be applied to both visual generation and understanding. Instead of using discrete indices from a VQ tokenizer, the method proposes using codebook embeddings along with a projector layer. The approach shows improvements on various visual understanding and generation benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper addresses a relevant gap in autoregressive multimodal models by proposing to align vision tokenization with downstream tasks, rather than relying on independently trained tokenizers. This is a practical consideration for models performing both visual understanding and generation.\nThe method is straightforward, involving the use of codebook embeddings and a projector layer to interface with a decoder-only language model. It does not require architectural changes to the language model.\nThe approach shows consistent improvements over frozen tokenizer baselines across multiple benchmarks. These gains suggest that end-to-end tuning of the tokenizer can improve performance without sacrificing too much understanding quality.\n\n\nWeaknesses:\nLimited Generalizability: The method is only evaluated with a single LLM (Qwen2.5-1.5B) within a large-scale, tightly controlled training pipeline. It remains unclear whether ETT is truly model-agnostic or if its effectiveness is contingent on the specific architecture and training setup used. The lack of validation across different language models raises concerns about the robustness and transferability of the approach.\nNo Comparison with Continuous Visual Encoders: The paper does not include comparisons with CLIP-style vision encoders or other continuous visual representation methods, which are widely adopted in models such as LLaVA, or Qwen VL. Without such baselines, it's difficult to assess the true benefit of discrete tokenization and end-to-end tuning proposed in ETT, especially when simpler or more modular alternatives exist.\nQuestionable Need for Joint Objectives: Table 5 suggests limited synergy between visual reconstruction and multimodal understanding. In fact, optimizing for one often comes at the expense of the other. This raises the question of whether it is necessary, or even desirable, to jointly optimize for both tasks within a single model. The paper does not justify why a unified approach is preferable over specialized models optimized independently for understanding or generation.\n\nQUESTIONS:\n1. Why isn’t there any comparison with other types of visual tokenization methods, such as continuous embeddings used in popular models like Qwen, LLaVA, or Gemma? I understand that there are limited works that handle both generation and understanding, but there should at least be a comparison with methods that address one of these tasks.\n2. Have you observed any synergy from jointly tuning for both reconstruction and understanding? What benefits does this type of joint tuning offer compared to tuning for a single task?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-08"}