End-to-End Vision Tokenizer Tuning
Authors: Wenxuan Wang
Fan Zhang
Yufeng Cui
Haiwen Diao
Zhuoyan Luo
Huchuan Lu
Jing Liu
Xinlong Wang
Venue: NeurIPS 2025 poster ()
Keywords: Discrete Vision Tokenizer
Autoregressive Model
Multimodal Generation and Understanding
Primary area: deep_learning
TL;DR: We propose ETT, an end-to-end tokenizer tuning approach that enables joint optimization between vision tokenization and target autoregressive tasks.

Abstract: Existing vision tokenization isolates the optimization of vision tokenizers from downstream training, implicitly assuming the visual tokens can generalize well across various tasks, e.g., image generation and visual question answering. The vision tokenizer optimized for low-level reconstruction is agnostic to downstream tasks requiring varied representations and semantics. This decoupled paradigm introduces a critical misalignment: The loss of the vision tokenization can be the representation bottleneck for target tasks. For example, errors in tokenizing text in a given image lead to poor results when recognizing or generating them. To address this, we propose ETT, an end-to-end vision tokenizer tuning approach that enables joint optimization between vision tokenization and target autoregressive tasks. Unlike prior autoregressive models that use only discrete indices from a frozen vision tokenizer, ETT leverages the visual embeddings of the tokenizer codebook, and optimizes the vision tokenizers end-to-end with both reconstruction and caption objectives. ETT can be seamlessly integrated into existing training pipelines with minimal architecture modifications. Our ETT is simple to implement and integrate, without the need to adjust the original codebooks or architectures of the employed large language models. Extensive experiments demonstrate that our proposed end-to-end vision tokenizer tuning unlocks significant performance gains, i.e., 2-6% for multimodal understanding and visual generation tasks compared to frozen tokenizer baselines, while preserving the original reconstruction capability. We hope this very simple and strong method can empower multimodal foundation models besides image generation and understanding.

End-to-End Vision Tokenizer Tuning
Wenxuan Wang 1,2.3,, 12..3,,Fam Fan Zhang Yufeng Cui Haiwen 3
Zhuoyan Luo Huchuan Lu Jing Liu Xinlong Wang
Institute Automation, Chinese Academy of Sciences
School of Artificial Intelligence, University of Chinese Academy of Sciences Beijing Academy of Artificial Intelligence
DDalian University of Technology Tsinghua University
(vangkenxuann222330i..cmo ac. en, zhangfan@baas ac wwnggviinooggaaaiicccco cn.
Abstract
Existing vision tokenization isolates the optimization of vision tokenizers from downstream training, implicitly assuming the visual tokens can generalize across various tasks, e.g., image generation and visual question answering. The vision tokenizer optimized for low-level reconstruction agnostic to downstream tasks re- quiring varied representations and semanties. This decoupled paradigm introduces a critical misalignment: The loss of the vision tokenization can be the representa- tion bottleneck for target tasks. For example, errors in tokenizing text in image lead to poor results when recognizing or generating them. To address this, we propose ETT, an end-to-end vision tokenizer tuning approach that enables joint optimization between vision tokenization and target autoregressive tasks. Unlike prior autoregressive models that only discrete indices from frozen vision tokenizer, ETT leverages the visual embeddings of the tokenizer codebook, and optimizes the vision tokenizers end-to-end with both reconstruction and caption ob- jectives. Our ETT is simple to implement and integrate, without the need to adjust the original codebooks or architectures of large language models. Extensive experi- ments demonstrate that our end-to-end vision tokenizer tuning unlocks sigmificant performance gains, i.e., 2-6% for multimodal understanding and visual generation tasks compared to frozen tokenizer baselines, while preserving the original recon- struction capability. We hope this very simple and strong method can empower multimodal foundation models besides image generation and understanding.
Introduction
Recently, the rapid advancement of large language models (LLMs) and multimodal pre-training has propelled autoregressive (AR) modeling beyond its dominance in natural language processing, extending its influence into the vision and multimodal tasks. Under the next-token prediction (NTP) paradigm of autoregressive models, multimodal learning typically encodes multimodal data such as images and text into compact discrete tokens for unified sequence modeling. For example, a recent work Emu3 [54] tokenizes text, images, and video into discrete tokens and performs next-token prediction in a unified token space. Numerous subsequent works [37, 61, 69, 57], have further advanced this direction, achieving improved performance in visual generation and perception. This unified next-token prediction paradigm enables flexible multimodal learning framework for both training and inference at scale.
Tokenization plays key role in an autoregressive framework. For modalities such as image and video, training efficient and general-purpose tokenizer raises significant challenges as the tokenization can
Equal contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Ensiig
Understanding & Generation
VQ Decoder
LosScu
VQ Decoder
LoSicE
T
5.9
5,2
Large Language Model
Large Language Model
Projector
Discrete Index
Codebooki Encedding
SEEDBench
T
VQ Tokenizer
Text Tokenizer
VQ Tokenizer
Text Takenizer
3.0 I
GonEvel
TextiQ
O Siberion Husky puppy lies on gray cushioned surface
Siberion Husky puppy lies on gray cushioned surface
Freezing
Baseline
Training
w/ ETT (Ours)
(o) Previous Pipeline
(b) Our ETT
(c) Improvements
Figure Left:
Existing autoregressive pipeline uses the discrete indices from a frozen vision tokenizer optimized with low-level reconstruction. Middle: We present ETT, an end-to-end tokenizer tuning approach which takes advantage of the visual codebook embeddings and optimizes the vision tokenizer and downstream training jointly. Right: Our proposed ETT unlocks significant performance gains on multimodal understanding and generation benchmarks.
hardly be compact and lossless at the same time. The loss of the tokenization can be the bottleneck for target tasks. For example, erros in tokenizing text in image lead to poor results when recognizing or generating them. However, existing tokenization approaches neglect this misalignment, and train vision tokenizer separately and directly integrate it in the downstream training, assuming that the obtained visual tokens can generalize well across tasks. The vision tokenizer optimized for autoencoding agnostic to the downstream tasks that require various representations and semantics. For instance, most tokenizers focus on low-level pixel-wise reconstruction. The quality of the learned representations inherently limited by the information loss caused by vector quantization (VQ), leading to inferior performance in visual understanding tasks compared to models using continuous high-level representations like CLIP [38]. Besides, existing autoregressive pipelines typically use only the discrete indices from the vision tokenizer, associated with random initialization of visual embeddings in large language models for realizing downstream tasks. This poses challenges to learn vision representations and vision-language alignment, which are crucial for multimodal learning.
In this work, we present end-to-end vision tokenizer tuning approach (ETT), which enables joint optimization of vision tokenization and target autoregressive downstream tasks. Inspired by recent language models [47, 46, 27, 26] that update their continuous vision encoders during training to optimize the visual representations and vision- lisiun-language alignment for better visual understanding, we propose to tune the discrete vision tokenizer in end-to-end manner and leverage large language models as the vision tokenizer's s visual assistant. Specifically, we introduce the vision tokenizer's is codebook embeddings instead of solely using discrete indices, and integrate token-level caption loss to optimize the representations of the vision tokenizer. Our results highlight that ETT significantly boosts the downstream performance of multimodal understanding and visual generation tasks, demonstrating improved discriminative and generative representations in the vision tokenizer. To be noticed, ETT can preserve the vision tokenizer's original image reconstruction performance. Our ETT is also simple implement and integrate, without the need to adjust the LLM's original text codebook or expand its embedding layers and classification heads to learn visual embeddings and dision-languae alignment from scratch.
Our main contributions can be summarized as follows:
We present new vision tokenizer training paradigm to unlock the vision tokenizer for downstream autoregressive tasks. The vision tokenizer is aware of and optimized for downstream training. We introduce a simple yet effective approach ETT for end-to-end vision tokenizer tuning. ETT leverages the tokenizer's codebook embeddings instead of only using discrete indices, and apply token-level caption loss to optimize the representation of the vision tokenizer.

---

ETT greatly improves the downstream results in next-token prediction paradigm, for both multi- modal understanding and generation, while maintaining the tokenizer's 's reconstruction performance.
2 Related Work
Vision Tokenizer. A vision tokenizer quantizes an image video into discrete tokens while preserving high reconstruction quality. Specifically, VQ-VAE [53] incorporates quantizer within an auto-encoder framework, where the quantizer learns map continuous features into discrete representations. VQGAN [10] improves the reconstruction quality via integrating perceptual loss and adversarial loss. MoVQ [70] proposes to incorporate the spatially conditional normalization to modulate the quantized vectors, generating images with high fidelity. Some other studies focus on improving codebook utilization ratio [71, 63], or incorporating more advanced quantization techniques [68, 35, 65, 25]. Recent works [61, 69, 37, 58] make efforts to integrate semantic information into the tokenizer to improve visual representations. Quantized indices are typically used in downstream tasks, while the tokenizer itself remains frozen during downstream training. In our work, we adopt IBQ [43] as the strong baseline and leverage the codebook embeddings instead of the conventional discrete indices for end-to-end tuning.
Tokenization for Visual Generation & Understanding. Discrete visual generation has made great progress in recent years. Some works [40, 64, 45, 62, 21] employ autoregressive approaches to generate images or videos by predicting tokens sequentially. While some others, such as MaskGIT [5] and Muse [4], adopt masked vision token modeling for image generation. Recently, a few studies [54, 48, 59, 58, 55] have explored unifying visual understanding and generation within a single model with the frozen discrete vision tokenizers. For example, Emu3 [54] unifies video, image, and text in token space and achieves strong understanding and generation abilities across modalities via next- token prediction. Show-o [59] proposes to incorporate mask image modeling for image generation within an autoregressive model. Janus [55] introduces two visual encoders for understanding and generation tasks separately to alleviate intrinsic conflicts between two tasks. In this work, we focus on the next-token prediction paradigm and propose to optimize both the vision tokenizer and the autoregressive models jointly for improved visual representation, promoting downstream multimodal generation and perception tasks.
3 Methodology
3.1 Vision Tokenizer
Preliminary. VQ-based vision tokenizer includes an encoder, a quantizer and a decoder. The encoder projects input image / E # x3 to feature map R in which D is the feature dimension and h = H/s w = W/s with & as the downsampling factor. The quantizer maps each feature in f to the nearest code from codebook B R to get quantized embeddings E R where K is the codebook size. Finally, the decoder reconstructs the quantized embeddings back into the original image. The traditional VQ models often struggle in scaling both codebook size and code dimension. As one of the pioncering works, IBQ [43] presents the first attempt to expand the code dimension to 256 while maintaining extreme large codebook size (i.e., 262,144) by updating the entire codebook simultaneously at each training step.
Vision Tokenizer in ETT. We primarily adopt the framework of IBQ [43] for image tokenization, using a downsampling factor
= 16. Each discrete token codebook has the dimension of D 256. Building upon the original IBQ, we adjust the codebook size to 131,072. The loss function Lug for tokenizer training is:
Lrg = Lree + Lquaat + Lipips + AG LGAN + Xg -eettrpp
where Cree is the pixel reconstruction loss, Lquant is the quantization loss between quantized embeddings and encoded features, Lipips is the perceptual loss from LPIPS [67]. LGAN is the adversarial loss from a PatchGAN [18] and Lentropy is the entropy loss [65]. Ag and Ag are the weight of adversarial loss and entropy loss separately.
3

---

3.2 End-to-End Vision Tokenizer Tuning
Discrete Indices Codebook Embeddings. Methods like Emu3 [54] and similar approaches such as [48], which use only the discrete indices of the vision tokenizer in downstream tasks, discard the rich representational capacity of vision tokenizer embeddings. By depending only on discrete codebook indices, these methods prevent gradient propagation, making end-to-end training infeasible. To address this limitation, we present ETT, which directly connects codebook embeddings from vision tokenizer to the LLM, effectively leveraging the richer feature representations encoded within the vision tokenizer while enabling end-to-end training.
LLM Bridges End-to-End Tuning. Specifically, as illustrated in Figure 1, given an input image I #x x3, we first obtain its quantized embeddings E R from the tokenizer's codebook. To ensure compatibility with the pretrained LLM, we employ a multilayer perceptron with the GeLU activation as lightweight projector. This projector layer maps the quantized visual embeddings to x' E Rxxxxxx where C' denotes the hidden dimension size of the large language model. Since the entire computation graph, including both the pretrained I.I.M and the vision tokenizer, remains differentiable, the whole structure can be trained end-to-end using gradient- optimization. For text input T, we utilize the tokenizer and text embedding layer from the pretrained LLM to convert into text token embeddings x² E RNC
Preservation of Reconstructive Capability. While end-to-end training enhances the representations of the vision tokenizer, it is essential to maintain its reconstructive capability to ensure high-fidelity image synthesis. To achieve this, we set the overall training objective the combination of caption loss Leap and VQ loss Leg- Specifically, we feed both the image token embeddings x' and text token embeddings x" into the LLM. For text tokens, we apply the cross-entropy (CE) loss:
Leap Σ log P(z x22)
(2)
Besides, directly reuse the loss function Luy for visual reconstruction. Thus, our end-to-end vision tokenizer tuning objective becomes:
L= Loap + o Leg
where a is the loss weight that controls the trade-offs between multimodal perception and visual reconstruction. By jointly training the tokenizer encoder and decoder with LLM, our approach maintains the model's reconstructive capability while ensuring that learned visual tokens remain semantically meaningful and effective for multimodal understanding and generation.
3.3 Training Recipe for Multimodal Generation and Understanding
Following previous works [7, 8]. the whole training process for downstream multimodal perception and generation follows three sequential training stages. The employed training data comprises publicly available image datasets supplemented with diverse instruction data for understanding and generation, as shown in Table 1.
Table 1: Details of training data across all stages for multimodal generation and perception.
Stage
Dataset
#Num
Total
Stage I: Alignment Learning
SOL-recap
12.0M
12.0M
Stage 2: Semantic Learning
SOL-recap
12.0M
12.0M
SOL-recap
32.0M
Stage 3: Post Post-Training Training Chat
LLaVA-One Vision [27]
3.5M
67.3M
InfinityMM [15]
31.8M
Af-generated Data" 14.0M
Stage 3: Post-Trainne Training Gen
30.0M
High-Acstheties Web Data
16.0M
Stage 1: Alignment learning. The first training stage is to effectively establish the vision-language alignment. With the pretrained large language model and vision tokenizer, we keep them frozen and train only the visual projector layer with image-to-text caption loss Leap- This setup enables the

---

LLM to acquire visual concepts and entities directly from the tokenizer, effectively bridging vision and language modalities. Specifically, we curate a 12M 12M-image subset from our constructed dataset SOL--eeap, which comprises 32M image-text pairs sourced from publicly available datasets, i.., SA-IB [20], OpenImages [22], and LAION [42]. To be noticed, all images are recaptioned using an improved captioning engine following [8]. The high-quality data at this stage can enhance training stability and cross-modality alignment.
Stage 2: Semantic Learning. The second stage acts as the most critical part in the entire training
pipeline, realizing end-to-end vision tokenizer tuning. At this stage, we unfreeze the weights of the LLM, projector, and vision tokenizer, optimizing them using the caption loss Leap and the reconstruction loss Lug jointly, as defined in Equation The high-quality subset, i.e., 12M image- text pairs from SOL-recap. is utilized for multimodal understanding and reconstruction learning. This stage enables efficient learning of the vision tokenizer's perceptual capability, supporting both visual reconstruction and understanding. This well designed stage 2 can enhance alignment between vision tokenizer and downstream tasks while preserving the original reconstructive ability.
Stage 3: Post-Training, After acquiring the enhanced vision tokenizer via the proposed end-to-end vision tokenizer tuning, we follow the standard post-training pipeline to realize multimodal under- standing and generation. During this training stage, we further post-train two specialist models, Ee.ETT-Chat and ETT-Gen, by freezing the vision tokenizer part, and tuning the visual projector, as well as the large language model layers to enhance cestruction-oollwiiin capabilities for multi- modal understanding and text-to-image generation, respectively. For multimodal understanding, we collect a diverse set of high-quality. multi-source instruction data, including SOL-reeap, VLaVA- One Vision [27]. and Infinity-MM [15]. For visual generation, we construct 14M Al-generated samples using the Flux model [23] and additionally curate 16M image-text pairs from open-source web data [12, 3], filtering them based on image resolution and LAION aesthetic score [24].
4 Experimental Results
4.1 Training Settings
Data Preparation. (1) Lasione-anguage Pre-training & Vision Tokenizer Datasets. We adopt the
rre-proeessing pipeline [8] to refine SA-1B [20]. Openlmages [22], and LAION [42], resulting in 1 11M, 7M, and 14M images respectively. We utilize the caption engine following [8] to produce 32M high-quality captions. (2) Supervised Fine-tuning Datasets. For understanding datasets, we extract 31.8M multi-task samples from Infinity-Mi [15] and 3.5M instruction data from LLaVA- One [27], prioritizing complex conversational structures. For generation datasets, we generate 14M Al-created samples with the Flux model [23] and further select 16M image-text pairs from open-source web data [12,3] 3]. applying filters based on image resolution and aesthetic scores [24].
Implementation Details. We train ETT on 8-A100 nodes using the Adam optimizer [19]. The batch sizes for Stages 1, 2. and 3 are set to 1024, 1024, and 1024, respectively, with maximum learning 10-5, and
rates of 4 x 10 use cosine decay scheduler across all stages. Unless otherwise specified, images are processed resolution of 512?, and ablation studies are reported using LLaVA (AA-mix-665k [31] at Stages 3. For all the experiments in our work, we adopt Qwen2.5-1.55 [50] as the large language model for multimodal sequence modeling.
10-5 We apply a warm-up strategy with a 0.03 ratio and
For the vision tokenizer in ETT, we employ Adam optimizer [19] with a fixed learning rate of 1 x 10-4, 31 0.5 and 32 0.9. The tokenizer trained for 500,000 steps with global batch size of 256 and an input resolution of 256 x 256. The adversarial loss weight Ag is set to 0. and the entropy loss weight Aø is set 0.05. We also adopt LeCAM regularization [52] for discriminator training to improve training stability.
4.2 Multimodal Understanding Evaluation
We validate ETT on various widely known vision-language perception benchmarks, covering task-specific evaluations (GQA [17] and TextVQA [44]), hallucination detection (POPE [29]). open-domain multimodal understanding (MME [11], MMBench [33], SEED-Bench [28], and MM- Vet [66]), and scientific reasoning (ScienceQA- IMG [34]).
5

---

Table 2: Comparison with existing state-of-the-art vision-language models various multi-
modal understanding benchmarks, including MMBEN; MMBencchEEN [33]; SEED!: SEE.Denchh- Img [28]; MMV: MM [66]; MME [11]; POPE [29]; GQA [17]: SQA ScienceQA-Ing [34]; TQA: VeX [44]. Note that #LLM--Parae denotes the number LLM parameters, #Data repre-
sents the pre-training fine-tuning data volume, denotes that the images of related training datasets
are observed during training, and the best results are marked in bold.
Method
#LLM-Paraa
#Data
MMB SEED' MMV MME POPE GQA SQA' TQA
Continuous VLMs:
QwenVL-Chat
7B
7.2B 50M
60.6
58.2
1848.0
57.5
68.2
61.5
EVE [7]
33M 1.8M
52.3
64.6
25.7
1628.0
85.0
62.6
64.9
56.8
Cambrian [51]
7B
10B+ / 7M
75.9
74.7
64.6
80.4
71.7
LLa [31]
7B
0,4B+/ 665K
64.3
64.3
30.5
1859.0
85,9
62.0
66,8
46.1
LLaVA-1.6 [32]
7B
0.4B+ /60B+
67.4
64.7
43.9
1842.0
86.4
64.2
70.2
64.9
Janus [55]
1.3B
-/-
69.4
63.7
34.3
1338.0
87.0
Discrete VLMs:
Chamelcon [48]
7B
1.4B+ 1.8M
31.1
30.6
8.3
170.0
47.2
4.8
LWM [30]
IB+/-
9.6
75.2
44.8
VILA-U [58]
7B
700M/7M
33.5
85.8
60.8
60.8
Emu3 [54]
8B
-/-
58.5
68.2
37.2
85.2
60.3
89.2
64.7
Show-o [59]
1.3B
2.0B+/665K
80.0
58.0
Liquid [57]
40M 3,5M
1119.3
58.4
42.4
ETT
1.5
32M 35.3M
58.8
66.6
29.3
1532.6
82.4
91.7*
56.8
ETT
3B
32M / 35.3M
61.3
67.8
32.2
1513.9
83.2
60.5
93.3
61.1
As shown in Table 2, our ETT consistently outperforms discrete counterparts, such as Chameleon [49], LWM [30]. and Liquid [57], even with smaller model and data scales. This highlights the efficacy of ETT's end-to-end tuning strategy, which enables to achieve strong results with fewer parameters and less data. Additionally, ETT achieves superior performance compared to Show-o o [60] across a wide range benchmarks, despite being trained on significantly less data. This underscores the effectiveness of ETT's data utilization strategies and its ability generalize well even with limited training resources. Furthermore, ETT demonstrates competitive performance against state-of-the-e (SOTA) continuous encoder-based VLMs, including QwenVL-Chat [1], EVE [7], and Janus [56], without relying on additional visual encoders for complex image encoding tasks. This not only simplifies the model architecture but also reduces computational overhead, making ETT more practical and scalable solution for multimodal tasks. The success of ETT lies its end-to-end training approach for visual tokenization, which seamlessly integrates multimodal understanding and generation while resolving internal conflicts within the tokenizer.
Table 3: Comparison with existing state-of-the-art vision-language models on various text- to-image generation benchmarks, including GenEval [14] and T21-CompBench [16]. Note that #LLM-Param denotes the number of LLM parameters, #Data represents the training data volume, and the best results marked in bold.
GenEval
22l-CompRench
M--.M-Param Overall Single Two Counting
Colors t Position t ColorAttr 1 Color Shape 1 Texture
on Continuous
SEED X [13]
17B
0.51
0.96
0.65
0.31
0.80
0.18
0.14
PixArt-
82
70.44
SD v1.5 [41]
1B
2B
0.43
0.97
0.38
0.35
0.76
0.04
0.06
37.50
37.24
42.19
[41]
49.82
SDXL [36]
2.6B
0.55
0.98
0.44
0.39
0.85
0.15
0.23
63.69
$4.08
56.37
DALL-E2
6.5B
650M
0.52
0.66
0.49
0.77
0.10
0.19
57.50
54
63.74
DALL-E3 [2]
0.67
0.96
0.87
0.47
0.83
0.43
0.45
81.10
67.50
80.70
SD3 [9]
0.62
0.74
0.63
0.67
0.34
0.36
Genevation on Discrete Foatunes:
LlamaCien [45]
Show-o [59]
1,38
35M
0.53
0.95
0.52
0.49
0.82
0.11
0.28
[30]
Chameleon [48]
34B
0.39
Emu3 [54]
8B
0.81
0.49
79.13
58.46
74.22
Janus [S5]
1.38
0.61
0.97
0.68
0.30
0.84
0.46
0.42
0.45
81.03
58.19
72.14

---

4.3 Visual Generation Evaluation
We comprehensively evaluate the text-to-image generation capabilities of our model against previous diffusion- and autoregressive-batt SOTA methods, including both multimodal specialists and generalists, on widely adopted benchmark datasets GenEval [14] and 121-CompBench [16]. As demonstrated in Table 3, our method achieves competitive performance while utilizing much fewer LLM parameters and smaller-scale training dataset. Specifically, under the inference configuration of top-k=131,072 (i.e., visual vocabulary size) and top-p=1.0, our model attains an overall score of 0.63 on the GenEval dataset, outperforming advanced diffusion models such SDXL [36]. Furthermore, our approach surpasses autoregressive--ng
methods, including both specialists (e.g., LlamaGen [45]) and generalists (e.g., Chameleon [48]), requiring either less training data fewer model parameters. When enhanced with prompt rewriting, the achieved model accuracy by our method closely approaches the performance of current leading models such DALL-E [2] and EMU3 [54]. On 121-CompBench dataset [16], our model achieves promising performance with scores of 81.03, 58.19, and 72. 14 color, shape, and texture pattern, respectively, demonstrating competitive performance compared to SOTA diffusion-based counterparts. These results fully prove the efficacy of our end-to-end vision tokenizer tuning approach, highlighting its ability achieve superior multimodal understanding and generation performance across diverse benchmarks.
A snowy mountain landscape under storry sky, with joggedt peaks, reflective river, and cratered moon, balanced by and cool
A lorge, textured green mushroom with white spots grows from a massy forest fleor, fromed by misty trees and soft, glowing light.
winding pebble path cuts through colorful flower field toward rustic hut. backed by forest and bathed worm autumn light, richly detoiled lidld shorp
surreal oil painting Sri Lonkan sunset, with in the foreground and a glowing ocean horizon, nich volumetric light and shodow.
A lone figure walks towards mist shrouded, dilapidated castle, surrounded by barren landscape. The dark, muted tones create a sense of isolation and foreboding.
mountain landscape reveals snow-capped peaks, rugged rocks, and a clear lake reflecting the sky, evoking peace wonder.
A realistic portrait of with soft eyes, with yellow flowers, wearing green and cream dress, with shear vell background.
As the sets, sky, streaked with pink and purple clouds. Below the city bustles, traffic lights flickering life.
light
surreal portnait of womon platinum blonde hair and green eyes, fromed pink wreath soft pastel background.
Wide- angle
vivid
close-up of lotus flower formed from crystol-clear water droplets, set against lily pods on tranquil pond, with surlight highlighting shimmering petals.
fantasy castle, with torch-lit pathway through an abstract portol, moody, sci-fi 8K detail.
Grand Canyon pink and orange light, silhouetting towering rocks the Colorado winds
fontasy
Figure 2: Visual generation results with our ETT. We present 512 x 512 results spanning different styles, subjects, and scenarios. Note that the presented prompts are simplified versions, which convey the general meaning.
7

---

Fig. 2 presents qualitative results generated by ETT, demonstrating its ability to follow prompts accurately while generating diverse visual content. The model exhibits proficiency in producing images across various artistic styles, subjects, and backgrounds, adapting to different compositional structures and aesthetic preferences.
4.4 Ablation Study
To verify the effectiveness of our ETT for downstream multimodal generation and understanding tasks, we conduct comprehensive ablation studies on several prevalent understanding benchmarks (e.g., SEEDBench-Img [28], GQA [17], TextVQA [44] and MME-Perception [1I] that assess diverse capabilities, as well as the GenEval dataset [14] for the evaluation of text-to-image generation.
Table 4: Ablation study on the benefits of ETT for multimodal understanding and generation.
ETT
Und. TQA
ETT for Gen.
Visual
Tuning
SEED!
GQA
MME
GenEval-Of
Index
54.8
50.9
40.1
1028.7
0.40
Embed
X
54.8
52.6
40.9
1034.0
0.34
Embed
60.0
54.8
46.9
1124.7
0.43
End-to-End Tuning Benefits. We first probe into the effectiveness of our ETT for promoting multimodal downstream tasks. To ensure a fair comparison in validating the potential of ETT in optimizing vision tokenizer's feature representations, we train all models for understanding and gen- eration tasks with SOL-recap. Additionally, we apply LLaVA-mix-6652 [31] for an extra supervised fine-tuning stage for understanding tasks. As presented in Table 4, introducing ETT consistently yields significant performance improvements across both understanding and generation tasks, com- pared with the traditional tokenizer exploitation manner. Specifically, without end-to-end tuning, replacing the discrete indices with codebook embeddings partially alleviates the issue of information loss and brings notable performance gains on multimodal understanding benchmarks. Although this replacement degrades the visual generation performance, it establishes a fully differentiable model architecture, allowing for end-to-end optimization. Building on this foundation, incorporating end-to-end tuning the visual tokenizer further enhances performance for both understanding and generation tasks compared with the conventional setting (i.e., first row), particularly on tasks that heavily rely on visual features (e.g., . optical character recognition [44]).
1 5% for general visual question answering [28] and 1 6% for
Table 5: Ablation study on the impact of ETT for the trade-offs between multimodal perception and visual reconstruction.
Tuning Tasks for VQ
Understanding GQA
Reconstruction ImageNet-PSNR/
a
SEED
TQA
MME
ImageNet-rFID.
ImageNet-SSIMf
54.8
52.6
40,9
1034.0
1.03
21.73
0.60
Und. only
61.2
55,2
48.0
1164.3
45.70
13.22
0.19
&:Re.
0.25
60.0
54.8
46.9
1124.7
1.65
20,89
0.58
&:Rec
0.5
59.9
54.7
46.9
1118.1
1.66
21.14
0.59
Und. 1.0
59.3
53.9
45.7
1088.2
1.50
21.48
0.60
Trade-offs between I-to-T & Reconstruction. Then, we investigate the inherent task trade-off between visual reconstruction and multimodal understanding of ETT. The commonly used reconstruc- tion metrics including rFID, PSNR and SSIM are all adopted for the comprehensive evaluations. As shown in Table 5, compared to untuned baseline (i.e., first row), tuning vision tokenizer consistently delivers substantial gains for the understanding task, albeit at the cost of reconstruction performance, which deteriorates to varying extents. Specifically, tuning the vision tokenizer with only image- to-text understanding task (i.e., second row) yields the best performance on various understanding benchmarks but greatly deteriorate in reconstruction, e.g., the rFID on ImageNet 256 x 256 setting dramatically drops from 1.033 to 45.701. Introducing auxiliary reconstruction target with a small weight 0.25 slightly reduces understanding accuracy while significantly improving the reconstruction, indicating the importance of joint training on both understanding and reconstruction tasks. Increasing the reconstruction weight to 1.0 achieves the best reconstruction accuracy but results in the weakest

---

Plic
NC1 averiayly ohener
MUN
NAILE
(o) Input
(b) before Turing
(c) ofter Turing
(o) Input
(b) before Turing
(c) ofter Tuning
Comparison of visual reconstruction results of the input image before and after end-to-end tuning by our ETT. The vision tokenizer tuned by our ETT benefits from retaining the original rich low-level detail representations while being effectively injected with high-level semantics, producing visual details comparable to the pre-tuned counterpart and even performs better in some detail reconstruction, e.g., text rendering.
Figure 3:
perception capability. Therefore, to balance both understanding and reconstruction tasks, we choose 0.25 as the default reconstruction loss weight a.
Besides, we also visualize the reconstruction results before and after introducing ETT Figure The vision tokenizer, when tuned with ETT, generates visual details comparable its untuned counterpart while enhancing specific aspects, such as text rendering. This suggests that ETT can not only preserve the original rich low-level details but also improve high-level semantic representations.
Conclusion
In this work, we focus on addressing the representation bottleneck of the vision tokenizer for multimodal learning. We introduce simple yet effective approach of end-to-end vision tokenizer tuning, termed ETT. ETT involves codebook embeddings instead of solely discrete indices and applies token-level caption loss for end-to-end optimization of both the tokenizer and downstream training. ETT significantly enhances the multimodal understanding and generation with a decoder- only architecture, while almost preserving the tokenizer's reconstruction capability and even boosting reconstruction performance on specific aspects such as text rendering.
6 Limitations and Future Topics
One potential limitation of this work is that both the data scale for end-to-end fine-tuning and the model capacity could be further expanded to enhance visual representations and downstream task performance. Moreover, our current approach primarily focuses designing simple yet effective framework that optimizes the visual features of existing vision tokenizers, leveraging the semantic capabilities of LLMs, rather than constructing vision tokenizer inherently designed for

---

both understanding and generation through unified end-to-end training. While ETT shows the potential of using LLM-driven semantic feedback to enhance vision tokenization, still relies on fine-tuning pre-existing tokenizers rather than developing one from the ground up. Therefore, for future research we will explore end-to-end training of vision tokenizer from scratch, aiming to create more comprehensive and adaptable representation for multimodal tasks. Besides, extending beyond image and text modalities, such as incorporating video and audio, presents an exciting avenue for further advancements. We hope that our simple yet effective method can contribute to the broader development of multimodal foundation models beyond visual generation and understanding.
Acknowledgment
We thank all the insightful reviewers for the helpful suggestions, and the colleagues Beijing Academy of Artificial Intelligence for support throughout this project. This research is supported by Artificial Intelligence-National Science and Technology Major Project 30033D0121200 and the National Natural Science Foundation of China (62437001, 62436001, U21B2043), and the Natural Science Foundation of Jiangsu Province under Grant BK20243051.
10

---

References
[1] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding. localization, text reading, and beyond. arXiv preprint ArXiv:2308. /2966, 1()):, 2023. 6
[2] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang. Linjie Li, Long Quyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. comfpapers/dall--33 pdf. 2(3):8, 2023. 6,7
[3] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. httpa:/ // ///ggttuub www/kaaaobrarii caterraiinnoooooaaaeeeee cattaaassooodddettt...aooaa 2022. 5
[4] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint
2023. 3
[5] Huiwen Chang, Han Zhang, Lu Jiang. Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11315-11325, 2022. 3
[6] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie. Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, al. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023.
[7] Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang. Huchuan Lu, and Xinlong Wang. Unveiling encoder-free vision-language models. arXiv preprint arXiv:2406. /832, 2024, 4,
[8] Haiwen Diao, Xiaotong Li. Yufeng Cui, Yueze Wang, Haoge Deng, Ting Pan, Wenxuan Wang. Huchuan La, and Xinlong Wang. Evev2: Improved baselines for eneoder-frce vision-language models. arXiv preprint arXii:2502.06788 2025. 4,5 5
[9] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024. 6
[10] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021. 3
[11] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv: 2306. 13394, 2023. 5, 6. 8
[12] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, al. Datacomp: In search of the next generation of multimodal datasets. NeurIPS, 36, 2024. 5
[13] Yuying Ge, Sijic Zhao, Jinguo Zhu, Yixiao Gc, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi- munti-granularity comprehension and generation. arXiv preprint arXiv:2404. 14396. 2024. 6
[14] Dhruba Ghosh, Hannanch Hajishirzi, and Ludwig Schmidt. Geneval: An oojucc-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024. 6, 7,
[15] Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data. arXiv preprint arXiv:2410. 18558, 2024. 5.5
[16] Kaiyi Huang, Kaiyue Sun, Enze Xie. Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723 78747, 2023. 6,
[17] Drew A. Hudson and Christopher D. Manning, GQA: A new dataset for world visual reasoning and compositional question answering. In CVPR, pages 6700-6709, 2019. 5, 6. 8
[18] Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to to to-ima translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 1125 1134, 2017. 3
11

---

[19] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In JCLR, 2015. 5
[20] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. arXiv: 2304.02643, 2023.
[21] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung. Vighnesh Birodkar, Jimmy Yan, King-Chang Chiu, et al. Vidcopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:
2023. 3
[22] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings. Ivan Krasin, Jordi Pont-Tuset. Shahab Kamali, Stefan Popov, Mattco Malloci, Tom Ducrig, and Vittorio Ferrari. The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. arXiv: 181 1800022,
2018. 5
[23] Black Forest Labs. Flux. httpp:///gtaab..ccoobbaaaeffrrre
2024. 5
[24] LAION. Laihn-ass attps::/laaion.a//bbo0t www.aaaionnntttttssessseeeoeeeee 2022.
[25] Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Partern Recognition, pages 11523-11532, 2022.
[26] Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal capabilities in the wild, 2024. 2
[27] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint 4rXiv:2408.033226 2024.
2, 5
[28] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal ilms with generative comprehension. arXiv: 2307.16125, 2023. 5, 6, 8
[29] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen, Evaluating object ballucination in large vision- languaee models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, EMNLP, pages
2023. 5,
[30] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. arXiv preprint arXiv:2402.08266 2024, 6
[31] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv: 2310.03744, 2023. 5, 6,
[32] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. 6
[33] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? arXiv: 067.06681, 2023. 5, 6
[34] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurlPS, 2022. 5,
[35] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309. 15505, 2023.
[36] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattannn Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusson models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 6,7
[37] Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang. Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, 7ehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069, 2024. 1.3
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR,
2021. 2
12

---

[39] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint vrXiv:2204.06155, 1(2):3, 2022.
[40] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821-8831. Pmlr, 2021. 3
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[42] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5h: An open large-scale dataset for training next generation image-text models. NeurlPS, 35:25278-25294, 2022. 5
[43] Fengyuan Shi, Zhuoyan Luo, Yixiao Ge, Yujiu Yang, Ying Shan, and Limin Wang. Taming scalable visual tokenizer for autoregressive image generation. arXiv preprint 2rXiv:2412.02692, 2024. 3
[44] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. CVPR, 2019. 5, 6,
[45] Peize Sun, Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng. Ping Luo, and Zehuan Yuan. Autore- gressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525 2024. 3, 6.
[46] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang. Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14398-14444 2024.
[47] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang. Xiaosong Zhang, Yueze Wang. Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. arXiv preprint wrXiv:2307.05222. 2023. 2
[48] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv:2405.098/8, 2024. 3, 4, 6,7
arXiv preprint
[49] Chameleon Team. Chamelcon:
Mixed-modal carly-fusion foundation models. arXiv preprint
arXiv:2405.098/5 2024. 6
[50] Qwen Team. Qwen2.5: A party of foundation models, September 2024. 5
[51] Shenghang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-L 1: A fully open, vision-centric
exploration of multimodal Ilms. arXiv preprint arXiv:2406. 1406.6660006 2024. 6
[52] Hung- Yu Tseng, Lu Jiang. Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7921-7931, 2021. 5
[53] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information proessing systems, 30, 2017. 3
[54] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, al. Emu3: Next-token prediction all you need. arXiv preprint arXiv:2409.18m0 18869, 2024. 1, 3, 6,7
[55] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan. Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 13848, 2024. 3, 6
[56] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint Xix::4100.13846 2024. 6
[57] Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable multi-modal generators. arXiv preprint ar arXiv:2412.04333 2024. 1,6 6
13

---

[58] Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, al. Vila-u: unified foundation model integrating visual understanding and generation. arXiv preprint (xXi:24409004229
2024. 3,6 6
[59] Jinheng Xie, Weijia Mao. Zechen Bai, David Junhao Zhang. Weihao Wang, Kevin Qinghong Lin, Yuchao Gu. Zhijie Chen, Zhenheng Yang. and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint
225224408.1228, 2024. 3,6
[60] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang. and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 125288, 2024. 6
[61] Rongchang Xie, Chen Du, Ping Song, and Chang Liu. Muse-vl: Modeling unified vim through semantic discrete encoding. arXiv preprint 2r11v2241..17762 2024. 3,
[62] Wilson Yan, Yunzhi Zhang. Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint WwXiv:2104.000 2021. 3
[63] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantizer image modeling with improved vggan. arXiv preprint arXiv:211.004627, 2021. 3
[64] Jiahui Yu, Yuanzhong Xu. Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022. 3
[65] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar. Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion tokenizer key to visual generation. arXiv preprint arXiv:2310.057:: 2023.
[66] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv: 02490, 2023. 6,
[67] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep fcatures as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[68] Yue Zhao, Yuanjun Xiong, and Philipp Kräähnhüühl. Image and video tokenization with binary spherical quantization. arXiv preprint 2456::200000544444 2024. 3
[69] Yue Zhao, Fuzhao Xue, Scott Reed, Linxi Fan, Yuke Zhu, Jan Kautz, Zhiding Yu, Philipp Krähenbühl, and An-An Huang. Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation. arXiv preprint 17X 2025. 1,3
[70] Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems, 35:23412-23425, 2022. 3
[71] Lei 7hu, Fangyun Wei, Yanye Lui, and Dong Chen. Scaling the codebook size of vggan 100,000 with utilization rate of 99%. arXiv preprint arXiv:2406. 1837, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes]
Justification: The abstract and introduction accurately summarize the paper's 's key contribu- tions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? [Yes]
Answer:
Justification:
We discuss the limitations of the work in Sec. 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
15

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully discloses the necessary information for reproducing the main experimental results. Detailed experimental settings are provided in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use publicly accessible datasets for the main experiments in this paper. Once the blind review period is finished. we'll open-source the codes, instructions, and model checkpoints
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips cc/ Sawaddde///ooSSuuiissseee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not
16

---

reproducible, they including code, unless this is central to the contribution (e.g., for new open-souree benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips. wc/ppcpppieecccccc c/guidgsssssseee SCooeSubmmmmiicceee sssiissionn pessoooniccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the necessary training and test details in the experimental section, ensuring clear understanding of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper presents the results from single run for each experiment, which is consistent with previous works on the same task.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
17

---

Justification:
The paper provides detailed information on the computer resources used for each experiment within Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / nwwrips.....ee..eeee. cctpublic//tieeeecccccc ceGudeeieee
Answer:
[Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [ÑA]
Justification: This work proposes method for end-to-end tuning of a vision tokenizer with large language models. As the contribution is methodological and does not directly target any downstream application, believe does not raise any immediate or foreseeable societal impacts requiring discussion. Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No]
18

---

Justification:
We do not foresee any high risk for misuse of this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credit datasets that use in experiments and ensure that they properly licensed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thhoode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- ee-packaedd, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release assets. Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
19

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable). such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required. Answer: [Yes]
Justification: The core method in our paper incorporates the powerful LLM central component for vision tokenizer tuning and multimodal reasoning, which significantly contributes the originality and performance of the approach.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy (https: pestpa://neuriae httm///ipsscc/ooofeeeeeeeeeeeeee for what should or should not be described.
20


Reviews:
[{"review_id": "becPmec5p0", "reviewer": "Reviewer_Wh8j", "summary": "This paper presents a new approach to visual tokenization that can be applied to both visual generation and understanding. Instead of using discrete indices from a VQ tokenizer, the method proposes using codebook embeddings along with a projector layer. The approach shows improvements on various visual understanding and generation benchmarks.", "questions": "1. Why isn’t there any comparison with other types of visual tokenization methods, such as continuous embeddings used in popular models like Qwen, LLaVA, or Gemma? I understand that there are limited works that handle both generation and understanding, but there should at least be a comparison with methods that address one of these tasks.\n2. Have you observed any synergy from jointly tuning for both reconstruction and understanding? What benefits does this type of joint tuning offer compared to tuning for a single task?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe paper addresses a relevant gap in autoregressive multimodal models by proposing to align vision tokenization with downstream tasks, rather than relying on independently trained tokenizers. This is a practical consideration for models performing both visual understanding and generation.\nThe method is straightforward, involving the use of codebook embeddings and a projector layer to interface with a decoder-only language model. It does not require architectural changes to the language model.\nThe approach shows consistent improvements over frozen tokenizer baselines across multiple benchmarks. These gains suggest that end-to-end tuning of the tokenizer can improve performance without sacrificing too much understanding quality.\n\n\nWeaknesses:\nLimited Generalizability: The method is only evaluated with a single LLM (Qwen2.5-1.5B) within a large-scale, tightly controlled training pipeline. It remains unclear whether ETT is truly model-agnostic or if its effectiveness is contingent on the specific architecture and training setup used. The lack of validation across different language models raises concerns about the robustness and transferability of the approach.\nNo Comparison with Continuous Visual Encoders: The paper does not include comparisons with CLIP-style vision encoders or other continuous visual representation methods, which are widely adopted in models such as LLaVA, or Qwen VL. Without such baselines, it's difficult to assess the true benefit of discrete tokenization and end-to-end tuning proposed in ETT, especially when simpler or more modular alternatives exist.\nQuestionable Need for Joint Objectives: Table 5 suggests limited synergy between visual reconstruction and multimodal understanding. In fact, optimizing for one often comes at the expense of the other. This raises the question of whether it is necessary, or even desirable, to jointly optimize for both tasks within a single model. The paper does not justify why a unified approach is preferable over specialized models optimized independently for understanding or generation.", "was_revised": true, "final_justification": "I thank the authors for their thorough response. While I remain unconvinced by the rationale for merging visual understanding and generation into a single framework, I acknowledge that it is an established problem in the field, and the authors' work represents a meaningful advancement in this area. I strongly recommend that the revised paper include results using a jointly trained tokenizer for both understanding and generation as the discrete visual tokenizer candidate, as this would further strengthen the contribution. I am raising my score to 5 and recommend acceptance.", "rebuttal": "We sincerely appreciate your friendliness and recognition of our meaningful research perspective, great potential of our ETT and the comprehensive experiments. We have addressed all of your questions in detail in the following Response section and have incorporated all feedback in the revision. We genuinely hope that our detailed explanations and the additionally supplemented results with our best efforts will give us the precious opportunity to raise the evaluation score of our work.\n\n**Our Responses to Paper Weaknesses:**\n\n> [**Q1**]. Limited Generalizability: The method is only evaluated with a single LLM within a large-scale, tightly controlled training pipeline. It remains unclear whether ETT is truly model-agnostic or if its effectiveness is contingent on the specific architecture and training setup used. The lack of validation across different language models raises concerns about the robustness and transferability of the approach. \n\n[**A1**]. First, we would like to clarify that describing our setup as a “large-scale, tightly controlled training pipeline” is not fair. Our training pipeline follows a standard and conventional three-stage setting widely adopted in both continuous and discrete MLLM works, including [1]–[3]: stage 1 focuses on vision-language alignment, stage 2 on high-level semantic learning, and stage 3 on supervised fine-tuning. **Therefore, the core insight of our proposed end-to-end tuning is not tightly constrained by the design of the training pipeline.** Second, our ETT method does not impose any requirements on the choice of LLM. In response to your valuable suggestion, **we have added ablation studies using both Qwen2.5-3B and Vicuna-7B-v1.5 as LLM candidates on 12M vision-language understanding and generation data (Table 1), as well as SOTA results using Qwen2.5-3B on the full dataset (Table 2), to demonstrate that ETT is scalable and model-agnostic**. The results for Qwen2.5-3B have been presented as follows, and those for Vicuna-7B-v1.5 are currently in progress and will be updated during discussion phase. Additionally, **to further verify the generalizability of our ETT framework, we are also conducting experiments using one of the jointly trained tokenizers for both understanding and generation as the discrete visual tokenizer candidate. These results are also on the way and will be made available during discussion.** All the results will be included in the revised version.\n\n\nTable 1: Ablation study on the benefits of ETT for multimodal understanding and generation with different LLMs.\n| ETT | LLM | SEED$^I$ | GQA | TQA | MME$^P$ | GenEval-Overall |\n|:------|:-------:|:----:|:-----------:|:----:|:-----------:|:----:|\n|   | Qwen2.5-3B |  53.9  |  52.4  |  40.6  |  986.9  |  0.29  |\n| √ | Qwen2.5-3B |  64.0  |  56.7  |  49.4  |  1188.9  |  0.32  |\n\n\nTable 2: Comparison with existing state-of-the-art VLMs on various multimodal benchmarks.\n| Method | Category | #LLM-Param | #Und. Data | MMB$^{en}$| SEED$^I$ | MMV | MME | POPE | GQA | SQA | TQA | #Gen. Data | GenEval-Overall |\n|:------|:-------:|:----:|:-----------:|:----:|:-----------:|:----:|:-------:|:----:|:-----------:|:----:|:-----------:|:----:|:----:|\n| EVE  | Continuous VLMs |  7B  |  33M / 1.8M  |  52.3  |  64.6  |  25.7  |  1628.0  |  85.0  |  62.6  |  64.9  |  56.8  |  -  |  -  |\n| Janus | Continuous VLMs |  1.3B  |  - / -  |  69.4  |  63.7  |  34.3  |  1338.0  |  87.0  |  59.1  |  -  |  -  |  -  |  0.61  |\n| EMU3 | Discrete VLMs |  8B  |  - / -  |  58.5  |  68.2  |  37.2  |  -  |  85.2  |  60.3  |  89.2  |  64.7  |  -  |  0.66  |  \n| ETT (ours) | Discrete VLMs |  1.5B |  32M / 35.3M  |  58.8  |  66.6  |  29.3  |  1532.6  |  82.4  |  59.4  |  91.7  |  56.8  |  30M  |  0.63  |\n| ETT (ours) | Discrete VLMs |  3B  |  32M / 35.3M  |  61.3  |  67.8  |  32.2  |  1513.9  |  83.2  |  60.5  |  93.3  |  61.1  |  30M  |  0.68  |\n\n\n[1] Wu C, Chen X, Wu Z, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 12966-12977.\n\n[2] Diao H, Cui Y, Li X, et al. Unveiling encoder-free vision-language models[J]. Advances in Neural Information Processing Systems, 2024, 37: 52545-52567.\n\n[3] Bai J, Bai S, Chu Y, et al. Qwen technical report[J]. arXiv preprint arXiv:2309.16609, 2023.\n\n\n> [**Q2**]. No Comparison with Continuous Visual Encoders: The paper does not include comparisons with CLIP-style vision encoders or other continuous visual representation methods, which are widely adopted in models such as LLaVA, or Qwen VL. Without such baselines, it's difficult to assess the true benefit of discrete tokenization and end-to-end tuning proposed in ETT, especially when simpler or more modular alternatives exist. \n\n[**A2**]. Thanks for this comment. Indeed, continuous embeddings preserve more fine-grained visual features, making them an intuitive and ideal choice for multimodal understanding tasks. However, in the context of mainstream autoregressive generation tasks, continuous embeddings suffer from a critical issue of error accumulation. Prior work such as [Nexus-gen: A unified model for image understanding, generation, and editing] has extensively discussed this problem: during training, models typically use ground-truth continuous embeddings as context inputs via teacher forcing, whereas during inference, the model must rely on its own predicted embeddings for the next-step input. Small numerical discrepancies in floating-point vectors at each generation step can recursively propagate through the context, and such errors gradually accumulate, ultimately leading to severe degradation in image generation quality. In contrast, the next-token prediction (NTP) paradigm using discrete token indices is much more resilient to this issue. The process of retrieving the nearest neighbor token index at each step effectively absorbs and corrects small prediction errors, preventing them from cascading into subsequent steps. This greatly enhances the stability and consistency of generation over long sequences. **Nevertheless, traditional discrete visual tokenizers are typically pre-trained and decoupled from downstream task training. While they benefit from the error isolation properties of discrete tokens, their representational capacity is often suboptimal for complex tasks such as multimodal understanding and generation. This creates a visual bottleneck that limits the overall performance of the system.**\n\nTo address this, we propose End-to-End Vision Tokenizer Tuning, a method that jointly optimizes the visual tokenizer and the large language model in an end-to-end fashion. **ETT bridges the gap between the tokenizer and downstream training, allowing discrete tokens to retain the stability benefits of the discrete paradigm while gaining richer semantic capacity better suited for multimodal downstream tasks. By resolving the non-differentiability problem of traditional VQ index outputs, we enable effective gradient backpropagation through the tokenizer to substantially enhance the representational power of discrete tokens.** As we emphasize in the main text, our ETT method is specifically designed to address the visual bottleneck issue in MLLMs that use discrete visual tokenizers. In contrast, continuous visual encoders, such as those used in Qwen-VL or LLaVA, have already seen relatively mature development. **Therefore, optimizing continuous visual representations is beyond the scope of our current work.** As shown in Table 2 of the main paper, our method achieves comparable performance to existing MLLMs based on continuous vision encoders, despite using significantly smaller model sizes and less training data. This further demonstrates the effectiveness of ETT in closing the performance gap between discrete and continuous MLLMs, and highlights its potential to unlock further improvements for discrete-token-based models under the NTP paradigm.\n\n\n> [**Q3**]. Questionable Need for Joint Objectives: Table 5 suggests limited synergy between visual reconstruction and multimodal understanding. In fact, optimizing for one often comes at the expense of the other. This raises the question of whether it is necessary, or even desirable, to jointly optimize for both tasks within a single model. The paper does not justify why a unified approach is preferable over specialized models optimized independently for understanding or generation.\n\n[**A3**]. We would first like to clarify that the core motivation behind ETT is to enable the proposed end-to-end tuning to enhance the capability of discrete vision tokenizers in serving downstream multimodal understanding and generation tasks. Therefore, our design intentionally integrates both visual reconstruction and multimodal understanding objectives during training. This is essential: **the multimodal understanding objective allows the discrete VQ tokenizer to acquire high-level vision-language alignment semantics required by understanding tasks, while the visual reconstruction objective ensures the preservation of low-level visual details crucial for visual generation tasks.** We appreciate that Reviewer Snt5 has also recognized this motivation and highlighted it positively in their comment: \"ETT maintains the original reconstruction performance of the vision tokenizer, which is essential for tasks that rely on high-quality visual outputs\". As both your comment and our analysis in the ablation study on Table 5 (lines 257–269) indicate, there exists an inherent trade-off between multimodal perception and visual reconstruction, we have not yet observed any potential synergy between the two objectives. Nonetheless, in order to support a unified representation that is useful for both multimodal understanding and generation, we deliberately select a joint tuning configuration that balances this trade-off and identifies a practical sweet spot."}, {"review_id": "BJxp7lf2Y3", "reviewer": "Reviewer_Snt5", "summary": "This paper presents an innovative methodology, termed End-to-End Vision Tokenizer Tuning (ETT), designed to address the inherent limitations of conventional vision tokenization techniques. The authors contend that the prevalent practice of optimizing tokenizers independently of downstream applications can result in substantial misalignments. Such discrepancies are particularly detrimental in tasks requiring sophisticated visual representations, including image generation and visual question answering. ETT facilitates the joint optimization of both the vision tokenizer and downstream autoregressive tasks, thereby leveraging visual embeddings to improve multimodal understanding and generative capabilities. The authors provide empirical evidence demonstrating a 2-6% performance enhancement over traditional frozen tokenizer pipelines, without compromising the original reconstruction quality.", "questions": "None", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "PROS:\n\n1. I like the concept of end-to-end optimization of vision tokenizers, which is a significant advancement in the field. By integrating tokenizer training with downstream tasks, the authors address an important gap in existing methodologies.\n\n2. The extensive experiments show substantial improvements in multimodal understanding and generation tasks, with metrics indicating a clear advantage over existing models. This suggests that ETT can effectively enhance model performance with relatively simple modifications.\n\n3. The authors provide extensive experimental results across multiple benchmark datasets, reinforcing the robustness of their findings and demonstrating the versatility of their approach.\n\n4. ETT maintains the original reconstruction performance of the vision tokenizer, which is essential for tasks that rely on high-quality visual outputs.\n\n5. The paper is well-written, which makes it easy to follow.\n\n- CONS:\n\n1. The experiments are conducted using a specific vision tokenizer (IBQ-based) and a specific LLM (Qwen2.5-1.5B). While the results are strong, the paper does not explore whether the ETT method is equally effective with other popular vision tokenizers (e.g., VQGAN, MaskGIT) or different LLM families. While this is a reasonable scope limitation for a single paper, it leaves the question of the method's generalizability open. The authors rightly point to this as a direction for future work.\n\n2. The paper provides good qualitative examples of generated images and reconstruction quality. However, the analysis could be more impactful. For instance, to support the claim that ETT improves high-level semantics like text rendering, a direct side-by-side comparison showing a failure case of the baseline model (e.g., garbled text in an image) that is corrected by the ETT-tuned model would be highly compelling.\n\n3. The paper presents a promising proof of concept for ETT, but questions persist regarding its scalability and generalizability. Key concerns include how ETT’s benefits evolve with larger LLMs (e.g., 70B+ parameters) and bigger datasets, as a powerful LLM might mitigate a frozen tokenizer’s limitations. Additionally, its applicability to non-autoregressive architectures (like MaskGIT) and other modalities (video, audio) remains unclear, with challenges such as capturing temporal consistency in video. At least, they should be discussed.", "was_revised": true, "final_justification": "Authors made a good rebuttal, where all my concerns have been solved properly. I thus give ACCEPT as my final recommendation.", "rebuttal": "We sincerely appreciate your friendliness and recognition of our meaningful research perspective, good writing, great potential of our ETT and great efforts for comprehensive evaluation experiments. We have addressed all of your questions in detail in the following Response section and have incorporated all feedback in the revision. We hope that our detailed explanations will give us the precious opportunity to raise the evaluation score of our work in your perspective.\n\n**Our Responses to Paper Weaknesses:**\n\n> [**Q1**]. The experiments are conducted using a specific vision tokenizer (IBQ-based) and a specific LLM (Qwen2.5-1.5B). While the results are strong, the paper does not explore whether the ETT method is equally effective with other popular vision tokenizers (e.g., VQGAN, MaskGIT) or different LLM families. While this is a reasonable scope limitation for a single paper, it leaves the question of the method's generalizability open. The authors rightly point to this as a direction for future work.\n\n[**A1**]. Many thanks for raising the important question of generalizability, and for your kindness toward this exploratory work. As we claimed in the main paper, since we innovatively propose to leverage the original VQ's visual codebook embeddings, rather than the discrete indices that make end-to-end tuning impossible, our ETT requires appropriate selection of discrete vision tokenizers. For instance, **a larger codebook embedding dimension allows ETT to better unleash its potential**. However, current VQ models that meet this criterion are extremely limited, as most previous efforts have focused on designing better VQs under the framework of using discrete indices, without paying attention to scaling up the vision codebook embedding dimension. Since our work mainly aims to enhance performance on downstream multimodal understanding and generation tasks under the next-token prediction paradigm by optimizing both the training paradigm and the way VQ features are utilized, we adopt IBQ, one of the more suitable candidates, as our discrete vision tokenizer. However, **we would like to emphasize that our ETT is also capable of bringing downstream performance gains to other VQ designs**. To further demonstrate that our ETT architecture is generalizable and promising, we have additionally conducted experiments using one of the jointly trained tokenizers for both understanding and generation as the discrete vision tokenizer candidate. These results are currently on the way and will be updated during the discussion phase for your reference. Moreover, our ETT architecture imposes no specific constraints on the choice of LLM. Based on your valuable suggestion, we have included ablation results using Qwen2.5-3B and Vicuna-7B-v1.5 as LLM candidates on 12M-scale multimodal generation and understanding data (Table 1), as well as SOTA performance using Qwen2.5-3B on the full dataset (Table 2) to further demonstrate that our ETT method is both scalable and model-agnostic. The results for Qwen2.5-3B are shown below, while those for Vicuna-7B-v1.5 are also on the way and will be updated during the discussion phase. All corresponding results will be added to the final revised version. **As an exploratory effort to optimize the training paradigm of visual tokenizers under the discrete next-token prediction framework, there remain many opportunities for future exploration, and we are committed to further advancing the capabilities of ETT.**\n\n\nTable 1: Ablation study on the benefits of ETT for multimodal understanding and generation with different large language models.\n| ETT | LLM | SEED$^I$ | GQA | TQA | MME$^P$ | GenEval-Overall |\n|:------|:-------:|:----:|:-----------:|:----:|:-----------:|:----:|\n|   | Qwen2.5-3B |  53.9  |  52.4  |  40.6  |  986.9  |  0.29  |\n| √ | Qwen2.5-3B |  64.0  |  56.7  |  49.4  |  1188.9  |  0.32  |\n\n\nTable 2: Comparison with existing state-of-the-art vision-language models on various multimodal benchmarks.\n| Method | Category | #LLM-Param | #Und. Data | MMB$^{en}$| SEED$^I$ | MMV | MME | POPE | GQA | SQA | TQA | #Gen. Data | GenEval-Overall |\n|:------|:-------:|:----:|:-----------:|:----:|:-----------:|:----:|:-------:|:----:|:-----------:|:----:|:-----------:|:----:|:----:|\n| EVE  | Continuous VLMs |  7B  |  33M / 1.8M  |  52.3  |  64.6  |  25.7  |  1628.0  |  85.0  |  62.6  |  64.9  |  56.8  |  -  |  -  |\n| Janus | Continuous VLMs |  1.3B  |  - / -  |  69.4  |  63.7  |  34.3  |  1338.0  |  87.0  |  59.1  |  -  |  -  |  -  |  0.61  |\n| EMU3 | Discrete VLMs |  8B  |  - / -  |  58.5  |  68.2  |  37.2  |  -  |  85.2  |  60.3  |  89.2  |  64.7  |  -  |  0.66  |  \n| ETT (ours) | Discrete VLMs |  1.5B |  32M / 35.3M  |  58.8  |  66.6  |  29.3  |  1532.6  |  82.4  |  59.4  |  91.7  |  56.8  |  30M  |  0.63  |\n| ETT (ours) | Discrete VLMs |  3B  |  32M / 35.3M  |  61.3  |  67.8  |  32.2  |  1513.9  |  83.2  |  60.5  |  93.3  |  61.1  |  30M  |  0.68  |\n\n\n> [**Q2**]. The paper provides good qualitative examples of generated images and reconstruction quality. However, the analysis could be more impactful. For instance, to support the claim that ETT improves high-level semantics like text rendering, a direct side-by-side comparison showing a failure case of the baseline model (e.g., garbled text in an image) that is corrected by the ETT-tuned model would be highly compelling.\n\n[**A2**]. Many thanks for your valuable suggestion. Per your advice, we have further included a side-by-side visual comparison showing a typical failure case of the baseline model in rendering text, which is effectively corrected by the ETT-tuned model. This visual evidence supports our claim that ETT enhances high-level semantic fidelity such as accurate text rendering. As we are unable to include external links during the rebuttal phase, we have incorporated the corresponding visualizations into the revised version, hoping that this analysis could be more impactful.\n\n\n> [**Q3**]. The paper presents a promising proof of concept for ETT, but questions persist regarding its scalability and generalizability. Key concerns include how ETT’s benefits evolve with larger LLMs (e.g., 70B+ parameters) and bigger datasets, as a powerful LLM might mitigate a frozen tokenizer’s limitations. Additionally, its applicability to non-autoregressive architectures (like MaskGIT) and other modalities (video, audio) remains unclear, with challenges such as capturing temporal consistency in video. At least, they should be discussed.\n\n[**A3**]. Many thanks for your valuable comment. First, regarding how ETT’s benefits evolve with larger LLMs (e.g., 70B+ parameters) and bigger datasets, we believe that with larger LLM sizes, our ETT can still bring improvements to downstream multimodal understanding and generation by optimizing the visual representation of discrete vision tokenizers themselves. Although, as you pointed out, a powerful LLM might mitigate the limitations of a frozen tokenizer, our focus is to explore how to maximize the potential of discrete vision tokenizers under the next-token prediction paradigm within relatively limited resource settings. While the community offers many LLMs of various sizes, even the largest current LLMs still heavily rely on strong visual capabilities to perform downstream multimodal tasks effectively. Therefore, **the benefits of ETT may be somewhat diminished with extremely large LLMs, but they remain significant and cannot be ignored.** Experimental results with 1.5B LLMs in the main text and additional experiments with 3B and 7B models added during the rebuttal phase sufficiently demonstrate the strong scalability and generalizability of our ETT framework. Notably, **optimizing the vision tokenizer representation based on a smaller LLM (e.g., 3B or 7B) to boost the visual capabilities of VQ for larger LLMs (e.g., 14B, 32B) is a promising future direction we plan to explore, which could greatly improve training efficiency and reduce costs.** Moreover, **with increased data scale, the adopted discrete vision tokenizer continuously learns more generalized vision-language alignment semantics and a better balance of high- and low-level representations through the LLM-provided caption loss, further unleashing ETT’s potential**. We have indeed observed such data scaling properties where ETT’s gains improve as data size grows. Finally, regarding ETT’s applicability to non-autoregressive architectures and other modalities, we believe our ETT framework can generalize to other architectures and modalities. Although our current focus is on enhancing the semantic awareness of discrete vision tokenizers via end-to-end tuning with image-to-text caption loss, **the supervision form and included modalities are not restricted.** For example, ETT can inject necessary representations for downstream tasks into non-autoregressive architectures (like MaskGIT), and it can also be extended to discrete vision tokenizers for video or audio modalities to provide the high-level semantic representations required for multimodal understanding and generation. We will incorporate these analyses into the revised version following your suggestions."}, {"review_id": "bfz4GdDDQe", "reviewer": "Reviewer_xzNc", "summary": "This paper introduces ETT, an end-to-end vision tokenizer that jointly optimizes both image reconstruction and downstream auto-regressive tasks. While prior works typically rely on pre-trained vision tokenizers that are kept frozen during downstream training, this paper demonstrates that integrating the tokenizer into the end-to-end training loop is straightforward and beneficial. By doing so, ETT helps unlock the tokenizer's potential for downstream understanding tasks, which are often hindered when the tokenizer is only trained for reconstruction objectives. Experiments show that ETT achieves competitive performance in both image understanding and generation.", "questions": "Please see weaknesses.", "limitations": "Please see weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n- The paper is well-organized and easy to follow, with a clean methodology, straightforward solutions, and clear experimental setups and comparisons.\n \n## Weaknesses\n\n- The technical novelty is somewhat limited. While the paper demonstrates that end-to-end training is feasible and beneficial, the core idea is relatively incremental.\n- In Table 4, the performance drop from row 1 to row 2—when replacing visual tokens from indices to embeddings—is surprisingly large. The paper should better explain why this change has such a significant impact on generation performance.\n- In the ablation study exploring the trade-off between understanding and generation, only rFID is reported for generation quality. It would be more informative to also include sample image outputs or additional quantitative metrics (e.g., FID, IS) to illustrate how generation quality degrades.\n- The paper should discuss how ETT could integrate with recent trends in jointly trained tokenizers for both understanding and generation, such as VILA-U or UniTok. Would the proposed method still provide gains in such setups?", "was_revised": true, "final_justification": "This paper presents an intuitive approach that addresses an important area currently lacking sufficient research. Thorough experiments are conducted to validate both the effectiveness and extensibility of the proposed method. Therefore, I recommend Accept.", "rebuttal": "Thanks very much for acknowledging our nice research perspective, good writing and great efforts for comprehensive evaluation experiments. We have addressed all your questions in detail in the Response section below and have incorporated all feedback in the revision. It is our sincere hope that our thorough explanations to the best of our abilities will contribute to an improved evaluation score of our work.\n\n\n**Our Responses to Paper Weaknesses:**\n\n> [**Q1**]. The technical novelty is somewhat limited. While the paper demonstrates that end-to-end training is feasible and beneficial, the core idea is relatively incremental.\n\n[**A1**]. We respectfully disagree. In fact, this work originates from the key observation that under the next-token prediction (NTP) paradigm using discrete vision tokenizers for visual encoding, the main bottleneck in MLLMs lies in the discrete vision tokenizer itself, caused by the disjoint training between upstream and downstream stages as well as the current insufficient training paradigm of VQ to learn representations required by downstream tasks. The insight of improving downstream task performance in the NTP paradigm via end-to-end optimization from the training paradigm perspective is well worth exploring. This has also been recognized by Reviewer Snt5's positive comment: \"**I like the concept of end-to-end optimization of vision tokenizers, which is a significant advancement in the field. By integrating tokenizer training with downstream tasks, the authors address an important gap in existing methodologies.**\" As we emphasized in the Introduction, the conventional disjoint training paradigm under NTP is not fundamental and needs to be reformed. **This is a fundamental problem that has been overlooked, under-explored, and is non-trivial in the current research field. Our approach of end-to-end tuning to optimize discrete vision tokenizer representations and making the entire scheme practically work is sufficiently innovative both in insight and implementation details. This includes overcoming the challenge of the originally non-differentiable discrete VQ index output and enabling gradient backpropagation to VQ itself, making end-to-end tuning feasible; and maximizing the utilization and optimization of the original visual features of VQ to boost downstream tasks as much as possible. These represent substantial challenges in the current NTP field, all of which are fully addressed in our work. We believe that identifying pain points in the research domain and proposing feasible insights and solutions constitute true innovation and importance. The simplicity and effectiveness of a solution are bonuses, and a straightforward design should not be mistaken for limited technical novelty.**\n\n\n> [**Q2**]. In Table 4, the performance drop from row 1 to row 2—when replacing visual tokens from indices to embeddings—is surprisingly large. The paper should better explain why this change has such a significant impact on generation performance.\n\n[**A2**]. We appreciate the reviewer’s insightful comment. The sharp performance drop from row 1 to row 2 in Table 4 is primarily due to the fact that autoregressive generation requires precise modeling of low-level visual details, which are better preserved through discrete token indices. **In IBQ, the indices are directly optimized via gradient-based index supervision, which enforces strong token separability and facilitates the LLM’s ability to model fine-grained token transitions during generation.** In contrast, when switching to raw embeddings from a non-tuned IBQ tokenizer, **such embeddings lack task-specific discriminative structure and are not explicitly optimized for generation, making it harder for the model to capture visual token dynamics effectively**. However, **once we apply ETT tuning, the embeddings are adapted jointly with the generation objective, allowing generation-aware signals to be injected back into the tokenizer and improving the quality of the embeddings accordingly**. This alignment enables the embeddings to better approximate the role of discrete indices, thus recovering generation performance despite the shift in representation form.\n\n\n> [**Q3**]. In the ablation study exploring the trade-off between understanding and generation, only rFID is reported for generation quality. It would be more informative to also include sample image outputs or additional quantitative metrics (e.g., FID, IS) to illustrate how generation quality degrades.\n\n[**A3**]. We believe the phrase \"ablation study exploring the trade-off between understanding and generation\" should be referring to \"ablation study exploring the trade-off between multimodal understanding and visual reconstruction\". Regarding the ablation experiment in Table 5, we have already presented sample image outputs corresponding to Table 5's first and third rows of the numerical results in Figure 3. **Based on your valuable suggestion, we further supplement Table 5 with additional commonly used reconstruction metrics, including SSIM and PSNR, and add visual samples corresponding to each row of Table 5’s results in Figure 3**, making the ablation study more informative and convincing. Since we cannot upload images or external links during this rebuttal phase, we have included the quantitative results in the table below. The revised version will incorporate the supplemented quantitative metrics and more informative visualizations.\n\nTable 1: Ablation study on the impact of ETT for the trade-offs between multimodal perception and visual reconstruction.\n| Tuning Tasks for VQ | α | SEED$^I$ | GQA | TQA | MME$^P$ | ImageNet-rFID↓ | ImageNet-PSNR↑ | ImageNet-SSIM↑ |\n|:------|:-------:|:----:|:-----------:|:----:|:-----------:|:----:|:----:|:----:|\n|  -  | - |  54.8  |  52.6  |  40.9  |  1034.0  |  1.03  |  21.73 |  0.60  |\n| Und. only | - |  61.2  |  55.2  |  48.0  |  1164.3  |  45.70  |  13.22  |  0.19  |\n| Und.&Rec. | 0.25 |  60.0  |  54.8  |  46.9  |  1124.7  |  1.65  |  20.89  |  0.58  |\n| Und.&Rec. | 0.5 |  59.9  |  54.7  |  46.9  |  1118.1  |  1.66  |  21.14  |  0.59  |\n| Und.&Rec. | 1.0 |  59.3  |  53.9  |  45.7  |  1088.2  |  1.50  |  21.48  |  0.60  |\n\n\n> [**Q4**]. The paper should discuss how ETT could integrate with recent trends in jointly trained tokenizers for both understanding and generation, such as VILA-U or UniTok. Would the proposed method still provide gains in such setups?\n\n[**A4**]. Thank you for your valuable suggestion. Our ETT framework can be combined with recent trends of jointly trained tokenizers for both understanding and generation and bring performance improvements. **Actually, these two approaches are orthogonal and do not interfere with each other.** The recent jointly trained tokenizers learn high-level semantics by distilling CLIP vision-language alignment during the VQ pretraining stage, which is separated from downstream training. Although this enables VQ to learn beyond mere reconstruction, the disjoint upstream-downstream training paradigm still prevents VQ from fully acquiring the comprehensive representations required downstream, thus somewhat creating a visual bottleneck for downstream tasks. Therefore, our proposed end-to-end tuning that directly optimizes the vision tokenizer’s representations is highly meaningful. Since architectures like VILA-U and UniTok introduce additional uncertainties when combined with ETT (e.g., requiring extra transformer blocks), to quickly validate during the rebuttal phase that our ETT framework is general and promising, we have additionally conducted experiments using another one of the jointly trained tokenizers for both understanding and generation as the discrete vision tokenizer candidate. These results are currently on the way and will be updated during the discussion phase for your reference. **Although the jointly trained tokenizers can learn more comprehensive and holistic visual representations during pretraining, our ETT can still bring non-negligible downstream task performance gains.** We will include the corresponding analyses and quantitative results in the revised version."}, {"review_id": "fM4483GUOV", "reviewer": "Reviewer_t4q5", "summary": "The paper proposes end-to-end tuning of the vision tokenizer to mitigate the misalignment errors steaming from decoupled training of the vision tokenizer and the target Vision-Language (VL) tasks. It proposes to jointly optimize the vision tokenizer and the LLM with the reconstruction and captioning objectives. This is achieved by using the continuous visual embeddings of the tokenizer codebook instead of the discrete indices from the frozen vision tokenizer.", "questions": "* As I wrote above, it is hard to judge from where the improvements in Table 2 and 3 come from. Is it because your curated data is of higher quality or is it because of the ETT method or both? I think it would be very valuable to decouple these, for example with an experiment where you train one of the benchmark methods on your data and re-evalute.\n* How does your approach affect the latency of the forward pass during inference?\n* Why decouple the training in Stage 1 and 2? What happens when Stage 1 training isn’t performed?\n* Maybe a naive question, but why do you need quantization at all? Couldn’t one work with continuous embeddings directly? Is this only because you are using pre-trained VQ tokenizers and not doing the training from scratch?\n* Minor:\n   - Table 3 shows strong results on all categories of the GenEval except for counting? Why is this one so degraded?\n   - In L 206, you wrote “ETT achieves superior performance compared to Show-o on wide range of benchmarks”. What I read from Table 2 is that  this only seems to be the case for 2 out of 8 reported benchmarks, and I find this statement too exaggerated.", "limitations": "/", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* Well written paper, easy to follow.\n* Strong results (though not SOTA) using less data and fewer LLM parameters compared to the evaluated methods.\n\nWeaknesses:\n* It is not clear to me how much of the performance boost reported in the experiments is obtained from the introduced curated SOL-recap data and how much from the proposed method. In other words, the experiments do not decouple data and fine-tuning contributions.\n* The novelty is somewhat limited. In my opinion, it is expected that fine tuning the models on downstream tasks improves the results.", "was_revised": true, "final_justification": "I thank the author for the thorough response. I raised my rating accordingly.", "rebuttal": "Thanks for acknowledging our meaningful research perspective, good writing and great efforts for comprehensive evaluation experiments. We have addressed all your questions in detail in the Response section below and have incorporated all feedback in the revision. It is our sincere hope that our thorough explanations to the best of our abilities will contribute to an improved evaluation score.\n\n**Our Responses:**\n\n> [**Q1**]. It is...re-evalute.\n\n[**A1**]. The experiments reported in our paper have been deliberately designed to decouple the benefits brought by data and those by the method. In Tables 4 and 5, we use exactly the same data for fair comparison to demonstrate that our ETT approach indeed brings considerable gains to downstream multimodal understanding and generation tasks by optimizing the VQ’s representations. These gains arise purely from the method rather than the data. Regarding the performance improvements in Tables 2 and 3, it is due to both training data and the ETT method. **It is important to note that many existing large-scale multimodal models use in-house closed-source datasets, and the quality and scale of data vary significantly across methods. From the Data and LLM-Param columns in Tables 2 and 3, it is clear that previous methods use much larger datasets and model sizes compared to ours.** Under these conditions, a strong method must be supported by high-quality data to achieve comparable or better results against prior methods that employ larger LLMs and more data. **The SOL-recap dataset we use is collected from publicly available community datasets, offering no advantage in scale or quality compared to previous SOTA datasets. Moreover, to ensure reproducibility and promote community development, we detail the SOL-recap data collection process in the main paper, which reflects our contribution to the community.** Further, following your suggestion, we present direct performance comparisons with EVE and EMU3 in Table 1 below. **Since we follow their data construction process and collect a comparable amount of data, the comparison is fair and intuitive.** Compared to EMU3, our method achieves better or comparable performance with a smaller LLM size. Compared to EVE, our discrete next-token prediction paradigm attains better multimodal understanding performance with a smaller LLM. We also provide SOTA performance results at Qwen-2.5-3B to demonstrate the strong scaling property of our ETT method.\n\nTable 1: Comparison with SOTA VLMs on various multimodal benchmarks.\n| Method | Category | #LLM-Param | #Und. Data | MMB$^{en}$| SEED$^I$ | MMV | MME | POPE | GQA | SQA | TQA | #Gen. Data | GenEval-Overall |\n|:------|:-------:|:----:|:-----------:|:----:|:-----------:|:----:|:-------:|:----:|:-----------:|:----:|:-----------:|:----:|:----:|\n| EVE  | Continuous |  7B  |  33M / 1.8M  |  52.3  |  64.6  |  25.7  |  1628.0  |  85.0  |  62.6  |  64.9  |  56.8  |  -  |  -  |\n| EMU3 | Discrete |  8B  |  - / -  |  58.5  |  68.2  |  37.2  |  -  |  85.2  |  60.3  |  89.2  |  64.7  |  -  |  0.66  |  \n| ETT | Discrete |  1.5B |  32M / 35.3M  |  58.8  |  66.6  |  29.3  |  1532.6  |  82.4  |  59.4  |  91.7  |  56.8  |  30M  |  0.63  |\n| ETT | Discrete |  3B  |  32M / 35.3M  |  61.3  |  67.8  |  32.2  |  1513.9  |  83.2  |  60.5  |  93.3  |  61.1  |  30M  |  0.68  |\n\n\n> [**Q2**]. The novelty...results.\n\n[**A2**]. We respectfully disagree. In fact, this work originates from the key observation that under the next-token prediction (NTP) paradigm using VQ for visual encoding, the main bottleneck in MLLMs lies in the VQ itself, caused by the disjoint training between upstream and downstream stages as well as the current insufficient training paradigm of VQ to learn representations required by downstream tasks. **The insight of improving downstream task performance in the NTP paradigm via end-to-end optimization from the training paradigm perspective is well worth exploring.** This has also been recognized by Reviewer Snt5's positive comment: \"I like the concept of end-to-end...the authors address an important gap...\" The conventional disjoint training paradigm under NTP is not fundamental and needs to be reformed. **This is a fundamental problem that has been overlooked, under-explored, and is non-trivial in the current research field. Our approach of end-to-end tuning to optimize VQ representations and making the entire scheme practically work is sufficiently innovative both in insight and implementation details. This includes overcoming the challenge of the originally non-differentiable discrete VQ index output and enabling gradient backpropagation to VQ itself, making end-to-end tuning feasible; and maximizing the utilization and optimization of the original visual features of VQ to boost downstream tasks as much as possible. These represent substantial challenges in the current NTP field, all of which are fully addressed in our work. We believe that identifying pain points in the research domain and proposing feasible insights and solutions constitute true innovation and importance. The simplicity and effectiveness of a solution are bonuses, and a straightforward design should not be mistaken for limited technical novelty.**\n\n\n> [**Q3**]. How...inference?\n\n[**A3**]. Our ETT introduces virtually no impact on inference latency for downstream tasks. As shown in Figure 1, the conventional pipeline for MLLMs using VQ involves feeding the indices output by the VQ tokenizer into the LLM, which then performs joint embedding of visual and textual indices followed by sequence modeling. In comparison, the only difference introduced by ETT lies in the embedding strategy: instead of feeding discrete indices, we utilize the original VQ visual codebook embeddings, apply two visual projection layers, and then pass the projected visual embeddings into the LLM for subsequent sequence modeling. **This modification has negligible impact on inference efficiency. Notably, dislike conventional pipeline, our ETT design does not require any modification to the architecture of the original LLM or expanding the visual vocabulary into the textual codebook, which makes ETT more scalable and generalizable.**\n\n\n> [**Q4**]. Why...performed?\n\n[**A4**]. Our current training pipeline follows a standard conventional setting consisting of three stages: Stage 1 for V-L alignment, Stage 2 for learning high-level semantics, and Stage 3 for SFT. This training paradigm is widely adopted in many MLLM works, such as Janus, EVE, Qwen-VL, which further supports that the core insight of our ETT is not tightly constrained by training pipeline. **Removing Stage 1 places a greater burden on subsequent stages to implicitly learn alignment from scratch, resulting in a noticeable degradation in downstream performance especially on multimodal understanding tasks**, where we observe an average drop of 1–2 points.\n\n\n> [**Q5**]. Why...scratch?\n\n[**A5**]. First, continuous embeddings preserve more fine-grained visual features and are intuitively an ideal choice for multimodal understanding tasks. However, in the context of mainstream autoregressive generation, continuous embeddings suffer from a critical issue, error accumulation. As highlighted in works such as [Nexus-gen: A unified model for image understanding, generation, and editing], this problem stems from the training–inference discrepancy. During training, models use ground-truth continuous embeddings as contextual inputs (i.e., teacher forcing), while during inference, they rely on previously generated embeddings. **Small numerical errors in floating-point vectors are recursively propagated and amplified at each generation step, leading to severe degradation in output quality as generation progresses.** In contrast, discrete token index-based next-token prediction effectively mitigates this issue. The discretization step, typically involving a nearest-neighbor lookup, naturally absorbs small prediction errors, thereby preventing their recursive amplification and ensuring stable generation quality. **However, traditional VQ are usually pretrained separately and decoupled from downstream tasks. This means that while discrete token-based models enjoy greater generation stability, their visual token representations may not be well aligned with the semantic needs of complex downstream tasks, resulting in a visual bottleneck that limits performance.**\n\nTo address this, we introduce ETT to jointly optimize the vision tokenizer and LLMs in an end-to-end manner. **This breaks the separation between pretrained tokenizers and downstream tasks, allowing discrete tokens to not only retain their robustness against error accumulation but also gain task-adaptive semantic expressiveness. We overcome the non-differentiability of traditional VQ outputs, enabling effective gradient backpropagation and better utilization of VQ's visual features, which significantly enhances the representational capacity of discrete tokens.** In summary, our choice of VQ is not merely due to reliance on a pretrained VQ, but is a deliberate design to circumvent the inevitable error accumulation of continuous embeddings in autoregressive generation and to ensure stable generation quality. **The innovation of our work lies in enhancing and adapting the discrete token paradigm to better serve both multimodal understanding and generation, an aspect that is both necessary and valuable in current research community.**\n\n\n> [**Q6**]. Table 3...exaggerated.\n\n[**A6**]. 1. The current suboptimal performance on the counting category is primarily due to the limited availability of relevant training data. We believe that incorporating more text-image data specifically designed for counting patterns would lead to significant performance improvements in this category. 2. To ensure the rigor of our claims, we have removed the phrase \"across a wide range of benchmarks\" from the main text."}]

Author rebuttal:

Meta-review: Reviewers initially raised concerns regarding generalizability (reviewer `Wh8j`), experimental comparisons (reviewer `Wh8j`, `Snt5`, `t4q5`), deeper ablation studies (reviewer `xzNc`), and technical contribution (reviewer `xzNc`, `t4q5`). After the rebuttal, the authors successfully addressed these issues by clarifying their methodology and adding comparisons with relevant baselines. As a result, all reviewers updated their recommendations to clear accept. The authors are suggested to add the essential ablation studies and experiments in the revision.

The paper makes contribution to the visual tokenizer for unified generation and understanding model. The proposed method demonstrates strong empirical performance across diverse benchmarks and highlights a promising direction toward bridging generative and interpretive capabilities. 

Given the significance of the contribution, the satisfactory resolution of reviewer concerns, and the consensus among reviewers after the rebuttal, I recommend acceptance of this paper.
Decision: Accept (poster)