Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations
Authors: Jiaming Han
Hao Chen
Yang Zhao
Hanyu Wang
Qi Zhao
Ziyan Yang
Hao He
Xiangyu Yue
Lu Jiang
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Model
Unified MLLM
Image Generation
Primary area: applications
TL;DR: We propose a text-aligned visual representation to unify both visual understanding and generation within a single MLLM.

Abstract: This paper presents a multimodal framework that attempts to unify visual understanding and generation within a shared discrete semantic representation. At its core is the Text-Aligned Tokenizer (TA-Tok), which converts images into discrete tokens using a text-aligned codebook projected from a large language model's (LLM) vocabulary. By integrating vision and text into a unified space with an expanded vocabulary, our multimodal LLM, **Tar**, enables cross-modal input and output through a shared interface, without the need for modality-specific designs. Additionally, we propose scale-adaptive encoding and decoding to balance efficiency and visual detail, along with a 
generative de-tokenizer to produce high-fidelity visual outputs. To address diverse decoding needs, we utilize two complementary de-tokenizers: a fast autoregressive model and a diffusion-based model. To enhance modality fusion, we investigate advanced pre-training tasks, demonstrating improvements in both visual understanding and generation. Experiments across benchmarks show that **Tar** matches or surpasses existing multimodal LLM methods, achieving faster convergence and greater training efficiency. All code, models, and data will be made publicly available.

Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations
Jiaming Han 12 Hao Chen Yang Zhao?, Hanyu Wang?, Qi Zhao?,
Ziyan Yang?, Hao Hel? Xiangyu Yue"!, Lu Jiang?!
CUHK MMLab "ByteDance Seed
Abstract
This paper presents a multimodal framework that attempts to unify visual under- standing and generation within a shared discrete semantic representation. At its core is the Text-Aligned Tokenizer (TA-Tok),, which converts images into discrete tokens using a text-aligned codebook projected from large language model's (LLM) vocabulary. By integrating vision and text into unified space with an expanded vo- cabulary, our multimodal LLM, Tar, enables cross-modal input and output through a shared interface, without the need for modality-specife designs. Additionally, we propose scale-adaptive encoding and decoding to balance efficiency and visual detail, along with generative de-tokenizer to produce high-fidelit, visual outputs. To address diverse decoding needs, we utilize two complementary de-tokenizers: fast autoregressive model and diffusion-based model. To enhance modality fusion, we investigate advanced pre-training tasks, demonstrating improvements both visual understanding and generation. Experiments across benchmarks show that Tar matches or surpasses existing multimodal LLM methods, achieving faster convergence and greater training efficiency. Code, models, and data available at https:://// //tt:t///rrrrr csuhan. com
Introduction
Multimodal large language models (MLLMs) [2, 12, 23, 40, 82] have demonstrated the ability of LLMs to handle visual understanding tasks within an autoregressive framework. A true MLLM expected not only to understand images but also generate them, laying the foundation for perception, reasoning, and interaction with the world.
For instance, MLLMs for visual understanding typically have three components: semantic visual encoder (e.g.. CLIP [50]), an LLM, and a Assion-tollanguse adapter [40, 82]. With a pre-aligned visual representation, LLaVA VA [40] efficiently aligns CL.IP features LLM'' latent space using just 0.6M image-text pairs and a simple linear adapter. However, visual generation representation remains an open research problem, with several key design choices outlined below.
Separate vs. Shared. Visual understanding and generation often rely on features at different levels of abstraction, leading some methods to adopt separate representations. For example, CLIP for understanding and VQVAE for generation [10, 70]. However, this separation limits unified reasoning and complicates tasks like interleaved generation multi-turn editing. We therefore representation for both tasks, which ensure understanding and generation do not conflict but instead complement each other, both modalities are learned within a single latent space.
shared
Continuous ys. Discrete. Continuous visual features preserve rich information and work well for the understanding tasks but require objectives such as regression [57] or diffusion [84] for generation,
Project lead. #Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Figure 1: Text-to- Text-eo-Image Generation Results, using Tar-7B and 1024 pixel de-tokenizer.
diverging from the autoregressive paradigm essential for scaling LLMs. Discrete tokens [17] align naturally with LLMs but often face quantization errors [58, 73]. We propose unifying vision and language with shared discrete representation, simplifying the modeling paradigm, improving scalability, and reducing complexity by operating in a more efficient space. To address quantization emoos, we propose scale-adaptive representation that uses longer sequences to minimize errors and a generative de-tokenizer to enhance generation capabilities.
Pixel vs. Semantic. Pixel-level tokens (e.g., VAE [17]) provide fine-grained details but are difficult to align with LLMs [58, 76, 84]. Semantic representations like CLIP [50] efficiently capture high-level semantics for interaction with LLMs but struggle to recover image details. Hybrid methods [43, 73, 83] attempt combine both, but balancing them remains challenging. Building on the success of semantic representations in visual understanding, we extend their use to visual generation, enabling faster convergence and simplifying the unification of understanding and generation.
This paper studies Text-aligned representation (Tar), a fully discrete and semantic representation that attempts unify visual understanding and generation within shared space. Central to our method is the Text-Aligned Tokenizer (TA-Took which converts images into discrete tokens using a text-aligned codebook initialized from an LLM's vocabulary and adapted vision through Icarnable projection layers. This approach enables seamless cross cross-modal input and output without relying on modality-speciif designs, and supports advanced multimodal reasoning within a unified framework.
To balance efficiency and detail, we introduce Scale-Adaptive Pooling and Decoding, which lets the model adjust token length needed: coarse-grained tokens for efficient generation and fine- grained tokens for detailed understanding. For decoding, we use two complementary Generative De-Tokenizers: fast autoregressive (AR) model for discrete VAE latents, and diffusion-based model for continuous VAE latents. The AR de-tokenizer is fast and works well with discrete LLM tokens, while the diffusion de-tokenizer leverages powerful pretrained image generators [75] for high-quality outputs. Together, they provide flexible balance between speed, compatibility, and visual fidelity. Besides the common understanding (image-to-text) and generation (text-to-image) tasks, we further improve modality fusion with new pre-training tasks like image-to-imagee text- boosting both visual understanding and generation.
We summarize our key features as follows:
We propose a Text-Aligned Tokenizer that unifies visual understanding and generation in a shared semantic, discrete space. This multimodal framework eliminates the need for modality-specific designs and allows seamless input and output across modalities through common interface.
Our Adale-A Pooling and Decoding provide flexible control over visual detail for different tasks. Additionally, introduce Generative De- Tokenizers that generate images from discrete semantic tokens using either autoregressive or diffusion-basee models.
We explore advanced pre-training schemes to enable both visual understanding and genera- tion within single model, achieving strong performance on various benchmarks.

---

TA-Tok
Cute kilten with tongue
generated image
Autoregressive Large Language Model
input image
A golden retriever sits on grass.
De-Tokenizer
visual understanding
visual generation
visual token
text token
Figure 2: Architecture of Tar, multimodal LLM that unifies visual understanding and generation in an autoregressive paradigm. Refer to Sec. 3.3 for training and inference detail.
2 Related Work
Unified Multimodal Large Language Models. With the development of LLMs [1, 4, 46, 63],
MLLMs have attracted a lot of research interest due their strong multimodal understanding and reasoning capabilities [2, 12, 30, 40, 82]. Beyond visual understanding, several recent works [14, 20, 57, 67, 70, 76, 84] attempt integrate both visual understanding and generation within unified MLLM. Emu2 [57] enables LLMS to generate CLIP embeddings, which are decoded into images using a diffusion model. Emu3 [69] and Chameleon [58] use VQVAE [17] as both the visual encoder and decoder, allowing unified next token prediction across images and text. However, VQ s focus on pixel dependency limits MLLM's ' ability to handle both low-level image details and high-level semantics. Show-o [76] and Transfusion [84] integrate diffusion objectives into LLMs for image generation, but this design breaks the autoregressive paradigm and complicates the unification of the two tasks. Janus [10, 70] takes modular approach with separate encoders for understanding and generation, but results in distinct modalities for image understanding and generation, which can hinder tasks like multi-turn image editing and interleaved generation. VILA-U [73] and UniTok [43] train a fused tokenizer using both pixel reconstruction and image-text alignment losses, but the model struggles to converge optimally for both tasks. ILLUME [67] applies vector quantization to a semantic visual encoder, using discrete tokens for image generation. However, ILLUME still relies continuous visual features for visual understanding, resulting in separate encoders for the two tasks. In contrast, we propose a fully discrete, semantic and shared representation that unifies understanding and generation within single MILMM.
Visual Tokenization. For visual generation, both continuous and and discrete VAES [26, 65] are popular visual tokenizers. The continuous VAE [26] used the default tokenizer for diffusion models [27, 52], while discrete VAE [65] is used for masked generative models [6] and autoregressive models [17, 61, 68]. For visual understanding, the best practice is employing a continuous, semantic tokenizer like CLIP [50]. SigLIP [81] and DINO [5]. However, it is still unclear what type of tokenizer good for both visual understanding and generation. As discussed in Sec. 1, some works [58, 69] leverage continuous tokenizers [57, 84], while others use discrete tokenizers [58, 69, 76]. Some works use pixel-level tokenizers [58, 69], other works leverage semantic tokenizers [57, 67] or hybrid tokenizers [43, 73]. In contrast, we leverage discrete and LLM-aligned tokenizer for visual understanding and generation. The generated visual tokens are then decoded with autoregressive or diffusion de-tokenizer for high quality image generation.
3 Method
In this section, we first propose Text-Aligned Tokenizer (TA-Tok), which converts images into text-aligned, scale-adaptive, discrete tokens in Sec. 3.1. In Sec. 3.2, we introduce Generative De-Tokenizers leveraging powerful generative models (e.g., autoregressive [56] and diffusion models [74)) for decoding high-quality images from our text-aligned visual conditions. Built on TA-Tok and de- de-tokenizers, design unified MLLM with the simple next token prediction approach.
3

---

Text-Aligned Codebook
codebook init.
LLM embedding
SigLIP2
Text-pretrained LLM
(teacher)
vocab expand
projection
Lrac
lookup
1
6
2
image
SigLIP2
SA Pooling
8
SA Decoding
4
7
5
train only
trainable
frozen
vector quantization
Figure 3: Text-Aligned Tokenizer. An input image is first encoded into continuous tokens using SigLIP2 vision model [64], followed by Adaptivde (SA) Pooling to adjust spatial resolution. These tokens are then discretized via Text-Aligned Codebook. which is initialized from LLM embeddings. To guide training, SA Decoding reconstructs the pre-quantized tokens and is supervised by SigLIP2 teacher model using reconstruction loss Lree- The decoder and teacher only used during training and discarded at inference. Once trained, the codebook can serve as a visual vocabulary of the LLM.
The overall architecture is shown in Fig. 2. Finally, we illustrate in Sec. 3.4 the training recipe, especially the proposed unified pretraining tasks.
3.1 Text-Aligned Tokenizer
TA-Tok is designed to align visual representations to LLM's latent space. Below we will introduce the basic concept and architecture of TA-Tok. The overall architecture TA-Tok is shown Fig.
Vector Quantization (VQ) a technique to discretize continuous representations into finite set of tokens, thereby transforming nigh-dimensional vectors into set of quantized representations. Given a continuous input vector N = E()), which is obtained from an image encoded by a visual encoder ε, the goal is to map the closest vector in codebook C. The quantization process is formulated as: 2, = argmingec ||z, cll², where C = {c1, C2 c cK) and K is the number of codebook entries. The aim is map the input to the most representative vector C₁ in C.
Text-Aligned Codebook. Traditional VQ codebooks are usually random initialized. To align visual and textual tokens in the latent space of LLM, initialize the VQ codebook using the token embeddings of a pretrained LLM E E IR KxD {ee, (e,, ......, ek and a projection matrix W E DDD {W1, W2 .... WD}. The codebook C E IRXD is defined as:
C= EW (Ew1, Ew2,......... Ewp).
Ew, = [ei e.w,....,ee wa)-
(2)
Note the LLM embeddings E are always frozen and we only train the projection matrix W, which ensures each codebook entry is projected version of corresponding LLM token embedding. By grounding the visual codebook in the LLM's latent space, this design ensures that visual tokens are semantically aligned with textual tokens, facilitating unified representation across modalities.
Since LLM vocabularies are often large (e.g., 150K for Qwen [78]), using the entire embedding set codebook is computational impractical. We select the top-k most representative embeddings based on their average distance to others, ensuring broad semantic coverage with minimal redundancy:
E
argsort
e;
ej |||||
[: k),
N
c;
where N is the vocabulary size of the LLM and i, (1,2....,NN)
Scale-Adaptive Pooling and Decoding. Different tasks vary their need visual details, e.g., understanding and editing often rely on fine- fine-grained features, while generation may benefit from

---

coarser representations [18]. To accommodate these differences, we introduce Scale-Adaptive Pooling (SAP) and Decoding (SAD) to extract mulli-granullann features. Given image features 2,, we apply SAP with scale factor E 21, 3) to obtain zj = SAP(zi, 8), allowing control over visual details based on task needs or compute budget. During decoding, we follow SigLIP2 [64] by resizing 2D) positional embedding to match the input scale, enabling the VIT decoder [64] to process multi-scale latent features effectively.
Architecture and Training Objective. As shown in Fig. 3, TA-Tok consists of a SigLIP2 encoder, SAP, a Text-Aligned Codebook, a SAD and a SigLIP2 teacher model. SAP is implemented via adaptive pooling, while SAD uses lightweight VIT decoder with three VIT blocks. TA-Tok is trained with combination of feature reconstruction loss Lree and codebook losses Codde. The reconstruction loss encourages semantic alignment between the decoded features and the SigLIP2 teacher output,
defined as: Lree 1
2, 2, ||,,|||,||1 where Zy and by features from SAD and SigLIP2 teacher,
respectively. The codebook losses Leade ensures the quantized features are close the codebook entries. We freeze the L.M token embeddings E and only train the projection matrix W. The loss is defined as:
(4) where sg(-) denotes the stop-gradient operation. During training, the SigLIP2 encoder, SAP and SAD are jointly optimized, while the SigLIP2 teacher remains frozen.
Leode ||sg(C) 1 sg (z, 112
W) 2,112 EW sg(zz)ll.
3.2 Generative De-Tokenizer
Since TA-Tok produces only semantic tokens without im- age generation capability, we introduce a Generative De- Tokenizer to decode high-quality images from its quan- tized outputs. As shown in Fig. 4, we propose two vari- ants: autoregressive de-tokenizer (AR-DTok) and diffu- sion de-tokenizer (Dif-DToo),, corresponding to two dom- inant paradigms in image generation.
image
image
TA-Tok
TA-Tok
N
Autoregressive Model
Diffusion Model
Autoregressive De- Tokenzeie In Fig. 4 (a). we for- mulate image decoding as an autoregressive generation task. Let AAR be the parameters of AR-DTTok, and y [/,, .2...-..177 yr) be the image tokens (in yellow) from VQVAE encoder. The AR-DTok predicts each token Yt conditioned on the semantic visual tokensz, (in blue) from TA-Tok and previous generated tokens y<z:
VOVAE
image
image
(a) AR De-Tokenizer
(b) Dif. De-Tokenizer
((OAR)= Σ
Logp(yilzg. Bees Bet ARR)
(5)
Figure 4: Architecture of Generative De- De-Tokenizer Variants.
Diffusion De-Tokenizer. Fig. 4 (b) illustrates the second variant, where the quantized tokens Z serve as conditioning input to a diffusion model via cross attention, similar to text conditioning in traditional diffusion models [52, 54]. Let F be the diffusion model parameterized by Bay, which denoised noised latent to predict the original clean latent yo- The training objective is:
[(Oas)) = E, F(y,, Zy; (asf))1
(6)
In practice, we reuse most parameters of a pretrained diffusion model, simply replacing the original text condition with TA-Tok's visual tokens, enabling high-fidelity image synthesis with minimal adaption.
Discussion. AR-DTook and Dif-DTok offer complementary strengths. AR-DTok aligns naturally with our discrete condition zy, enabling unified autoregressive modeling framework. It also benefits from faster inference due to its sequential decoding. In contrast, Dif-DTok leverages powerful pretrained diffusion models, allowing quick adaptation to new conditions like zg. While it is more computational intensive at inference time, it requires less training data and remains competitive with AR-DTO in overall performance. Its strong generation prior also makes it particularly useful in tasks involving complex scenes or where high visual fidelity is desired.
5

---

Table 1: Results on Visual Understanding Benchmarks, including POPE [33], MME [19],
MMB [41], SEED [28], GQA [25] and MMMU [79]. Token: Token type, including Continu-
ous (C), Discrete (D), Semantic (S), Pixel (P) and Hybrid (H).
Model
LLMM
Token
POPE
MME-P1
MME-CT
MMB1
SEED1
GQAt
MMMU1
Understanding Only Model
LLaVA-Phi [76]
1128
56.5
30.7
MobileVLM-V2 [1]]
1.48
C.S
84.3
1303
57.7
59.3
DzepSeckVL. [42]
1.3B
C.S
88.3
1307
64.6
59.3
33.8
MiniGemini [32]
C.S
83.9
1341
59.8
59.9
LLaVA-v1.5 [38]
85.9
1511
62.0
35.4
Qwen Qwen-VI. Vha-Chat
7B
C.S
1488
60.6
58.2
57.5
Emu3-Chat [69]
8B
D.P
85.2
1244
58.5
68.2
60.3
31.6
Unified Model
Show-o [76]
1.3B
D.P
80.0
1097
248
58.0
26.7
Harmon [72]
87.6
1155
321
58.9
Janus [70]
1.5
$7.0
1338
222
69.4
63.7
59.1
30.5
Ponus-Pro/ii [10]
1.5B
C.S
86.2
1444
268
75.5
68.3
59.3
36.3
D-Dit [35]
2.0B
C,P
84.0
1125
59.2
Tar (Ours)
88.4
1390
61.1
36.0
ILLUME [67]
88.5
1445
65.1
72.9
38.2
Chameleon [58]
7B
D,P
22.4
LWM [37]
7B
D.P
75.2
44.8
Liquid[71]
81.1
1119
58.4
UniTok [43]
D.H.
83.2
1448
61.1
VILA-U [73]
7B
D.H
85.8
1402
59.0
60.8
Janus-Pro [10]
7B
C.S
87.4
1567
260
79.2
72.1
62.0
41.0
MetaMorph [62]
71.8
41.8
Tar (Ours)
D,S
87.8
1571
355
74.4
73.0
61.3
39.0
3.3
Unified Multimodal Modeling
Built on TA-Tok and Generative De-Tokenizers, we propose a unified MILM. Tar (Text-aligned representation), with simple autoregressive objective and eliminating the need for modality-specif designs. The architecture is shown in Fig.
Visual Embedding Initialization. We represent both text and images as discrete tokens in shared vocabulary by expanding the LLM's embedding matrix E MxD with a visual token C E IR K Rather than randomly initializing C, we use our Text-Aligned Codebook C = EW (with W wD as the visual embeddings. Since C and E share the same dimensionality, we can simply set: E C) {E, EW). This operation eliminates any extra embedding alignment stage. The unified embedding enables the LLM to natively process and generate both modalities without additional connectors or decoding heads.
Training. We train Tar with the standard Cross-Entropy loss mixed sequence of text and visual [U1, U2, uN] (uı may be text or visual token) and model -i=1 log(u:(u.<;0)
tokens, Given a target sequence 11
parameter 0, the loss is: LCE
Inference. At inference time, Tar can either take tokens from both modalities as input or generate them. (a) Visual understanding: Feed TA-Tok's visual tokens Zg and any text prompt into LLM, and generate text tokens for image captioning, visual question answering, etc. (b) Visual generation: Provide text prompt and autoregressively sample sequence of visual tokens, then pass these tokens de-tokenizer to decode the final image.
3.4 Training Recipe
Data Curation. Our training data consists of image, text and multimodal datasets. Since open- sourced datasets for image. text and image-to-tex tasks are widely available [13, 53, 60], our focus on curating high-quality data for image generation. The pipeline includes: (1) Image caption. We use Qwen2.5- VL [3] to generate rich, detailed captions for general image datasets [13, 51]. (2) Synthetic image generation. We adopt FL.UX [27] to generate high quality images based on real user prompts [55, 66] and image captions from Step which yield diverse, prompt-aligned content.
In total, we curate dataset of 23M high-quality text-image pairs for training.

---

Table 2: Results on Visual Generation Benchmarks. * We use an AR-DTok with 256px resolution. Due to space limit, we omit some metries and put full results to Appendix Sec. F.
GenEval [22]
DPG Bench [24]
Method
Two Obj.
Counting
Color Attri.
Overall
Entity
Attribute
Relation
Overall
Generation Only Model
Emu3-Gen [69]
0.71 0.74
0.34
0.21 0.23
0.54 0.55
86,66 06.68
86 84
90.22
80.60
SDXL [48]
0.39
82.43
80.91
86.76
74.65
Playgrund v2.5 [29]
82.59
81.20
84.08
75.47
Hunyuan DIT [34]
80.59
88
74.36
78.87
PixArt-)2 [8]
82.89
88.94
86.59
80.54
DALLE3 [36]
0.47
89.61
90.58
SD3-Medium [16]
0,94
0.72
0.60
0.74
91.01
88.83
80.70
84.08
SA5A-155
0.93
0.86
0.65
0.81
84.70
Unified Model
Chameleon-7B [58]
0.39
LWM-7B [37]
0.41
0.46
0.15
0.47
SEED-X-13B [21]
0.58
0.26
0.14
0.49
Show-o-1.3B [76]
0.52
0.49
0.28
0.53
Transfusion-7B [84]
0.63
D-DiT-2B [35]
0.80
0.54
0.50
0.65
ILLUME-7B [67]
0.86
0.45
0.28
0.61
Janus-1.3B [70]
0.68
0.30
0.42
0.61
87.38
87.70
85,46
79.68
Janus-Pro-IB [10]
0.82
0.51
0.56
0.73
88.63
88.17
88.98
82.63
Hammon-1.5B [72]
0.86
0.57
0.48
0.76
Janus-Pro-7B [10]
0.89
0.59
0.66
0.80
88.90
89.40
89.32
84.19
Tar-1.5B (Ours)
0,91
0,76
0.51
0.76
89.35
86.91
93,50
82.96
w/ Self Reflect
0,92
0.77
0.55
0.78
88.48
87.83
93.38
84.10
Tar-7B (Ours)
0.92
0.83
0.84
88.62
88.05
93.98
84.19
w/ Self Reflect
0.93
0.86
0.70
0.85
88.60
88.78
93.59
84.65
Tokenizer and De-Tokenizer Training. TA-Too is trained 100M raw and 100M aesthetic-filtered images from LAION [53], balancing capability visual understanding and generation. For AR-DTok, due the lack of pretrained models, train from scratch at 256px resolution and finetune to 512px and 1024px using 50M aesthetic images from LAION [53] and 23M synthetic images. For Dif-DTok.
initialize from pretrained SANA-0.6B [74]. allowing direct finetuning 512px resolution on the 23M synthetic dataset.
Unified MILM Pretraining. Our LLM is trained on a diverse mix of data types, including standard image-to-text (I-TT), text-to-image (T-I) and text-only (T-T) tasks. To further bridge the gap between visual understanding and generation, we introduce two additional task type: text-image-to- image (TI+I) and image-to-image (I-I). For I-+I, we use FLUX to generate two images from the same prompt. For TI-1, we let Qwen [78] to split a prompt into text with an image placeholder and a reference caption. For example, "A dog rumning on the grass" becomes: "A dog running on <image>" and "the grass". The goal is to generate an image based the input image (I-I) image and text (TI-+I), which encourage deeper multimodal integration and empirically, we observe that they accelerate convergence and improve alignment between understanding and generation.
Supervised MILM Finetuning. For visual understanding, we use open-source instruction tuning datasets from LLaVA-v1.5 11.5 [38] and LLaVA-Nexx [39]. For visual generation, we filter high-quality subset from our pretraining dlttasets with CLIP score >0.25. Additional, we collect few human- preferenced and task-aligned examples using advanced generation models [49, 75), For more details on training datasets, please refer to Appendix Sec. C.
Experiments
4.1 Experiment Details
For TA-Tok, we use
[64] as the visual encoder and three-layers
ViT [64] as the decoder. We select 65536 tokens from Qwen2.5 [78] as LLM embeddings in TA Codebook. An 384x384 image encoded into (729, 169, 81} tokens at scale (1, 3). For AR- DTok, we adopt the LLaMA architecture [63] implemented in I Llamagen [56]. The autoregressive model is trained from scratch. The discrete VAE for image decoding is pretrained by Llamagen. For
Dif-DTok. we pretrained SANA-0.6B [74] and only finetune the cross attention and condition
7

---

VOVAE
Janus
Ours
VOVAE
Janus
Hybrid
Curs
100
59
61
70
58
90
94
52
63
65
56
45
48
48
Acouracy (%)
60
Acourny
39
72
70
64
21
32
60
60
10M
Data Scale
Data Scale
Figure 5: Comparisons of Visual Representations on Generation and Understanding Tasks. Left: Generation performance evaluated by DPG Score [24]. Right: Understanding performance measured by the harmonic mean over benchmarks [19, 25, 28, 333.
VOVAE
Janus
Hybrid
Ours
AR22M6226px
ART7aM8258px AR77SMB55220 2ox
DiIBOO 2px
bib Fear-
PULIN!
Let's
Let's PAINT
LA PIRATES
Figure 6: Qualitative Comparison of Different Representations (Left) and De-Tokenizers (Right),
embedding layers. For MLLM, we adopt Qwen2.5-Instrwee [78] as the backbone LLM. The LLM is fully finetuned at both pretraining and finetuning stages, For training, we randomly select scale from 3,.. For inference, we set it to if not specified. See Appendix Sec. D for more detail.
4.2 Main Results
Visual Understanding, As shown in Tab. 1, Tar model demonstrates strong visual understanding performance across broad range of benchmarks. Our 1.5B model surpasses most understanding only models and unified models at 1.5B/7B scale. Our 7B model matches the performance of Janus-Pro-7B, a
model with continuous visual tokens. These results confirm that unified modeling framework using fully discrete tokens, when coupled with strong text-aligned representation, can match and even surpass specialized continuous-token models in visual understanding.
Visual Generation. In Tab. 2, Tar achieves strong performance on both GenEval [22] and DPG Bench [24]. On GenEval, it reaches 0.76/0.84 overall score, surpassing all unified models. On DPG Bench, 1ar-1.5B achieves 82.96 score, outperforming Janus-Pro- IB and even approaching Janus- Pro- 7B. To fully leverage Tar's multimodal reasoning ability, we propose a Self Reflect strategy, which allows the model to assess image-promp alignment using its own visual understanding capabilities, leading to further performance improvement. The image generation results are visualized in Fig. and Appendix Fig. 8. For more details about the Self Reflect strategy, please refer to Appendix Sec. E.
4.3 Comparisons with Other Visual Tokenization Methods
In this section, we compare our text-aligned representation (TA-Tok) with other visual tokenization methods for unified visual understanding and generation. We considering the following recent methods [43, 69, 70]: (a) VQVAE: A full pixel-level representation. We use pretrained VQVAE from Llamagen [56], which transforms images into discrete tokens for multimodal modeling. (b) Janus: Use separate encoders for understanding (SigLIP2) and generation (VQVAE). Note our implementation follows this approach but differs from the original one [70]. (c) Hybrid: A hybrid model that maintains both pixel and semantic representations. We follow UniTok [43] to train

---

Table 3: MLLM Embedding Initialization.
Table 4: Ablation Different De- Tokenizers.
Harmonic mean of understanding bench- marks [19, 25, 28, 33]. Gen: DPG score [24].
Und:
Type
Size
Res
GenEval
DPG
775M
256
0.76
83.0
emb. init
Data
Und
Gen
775M
512
0.79
82.7
AR
775M
1024
0.79
82.6
random
25M
91.1
69.6
111M
256
0.75
82.2
pre-align
50M+25M
89.8
70.3
22M
256
0.74
81.7
TA-Tok
25M
92.9
70.6
pre-align
100M+25M
91.5
72.5
Dif.
600M
512
0.77
82.4
Table 5: The Effect of Scale-Adaptive Pooling. Results on understanding tasks with 10M data model. Right: Different data scale on DPG
Table Separate Train vs. Joint Train. We evaluate the performance of different visual rep- resentations under joint understanding genera- tion training or separate training.
Und:
Bench. : Model after supervised finetuning.
Und
Data Scale on Generation
#Token
Und
Gen
Model
Avg
10M
20M
40M"
Avg
A (sep. -> joint)
(sep. -> joint)
729
70.6
73.8
82.3
75.6
Janus
-0.6 08.0
-0.1 33.4)
169
89.1
72.7
74.0
82.2
76.3
VQVAE
+0.3 (73.9 - 74.2
+8.1 (10.1 48.2)
81
86.6
73.0
73.5
80.1
75.5
Ours
+0.1 (92.8 92.99
+5.3 (65.3 70.6)
hybrid tokenizer on 100M image-text pairs [53], using both pixel reconstruction loss and image-text alignment loss. For training MILMMs with these representations, we sample subset of our training data for controlled experiments: 10M T2I data, 10M 12T data and 5M text-only data. All models are trained with the same configuration and tested visual understanding [19, 25, 28, 33] and generation tasks [24].
TA-Tok Outperforms in Visual Generation. As shown in the left of Fig. 5, TA-Tok excels in visual generation. It achieves the highest performance across all data scales, with VQVAE, Janus and TA-Tok showing similar convergence curves. Although Hybrid starts with higher performance, it does not scale effectively with increasing data. Notably, Janus underperforms VQVAE, likely due to conflicts between the understanding and generation tasks. Besides, we found TA-Tok generates high-fidelity images (see appendix), while models using pixel representations are struggle with image details, suggesting semantic representation is more suitable for L-M--hssed image generation.
TA-Tok Matches Continuous-Semantee
Representations in Visual Understanding. In visual understanding tasks (right of Fig. 5), Janus performs slightly better due to its continuous semantic encoder. However, TA-Tok is close behind with scores of 93 vs. 94 10M data. Hybrid, despite joint training with pixel and semantic losses, performs similar to VQVAE, not achieving the expected performance due its bias to pixel representation. Overall, TA-Tok achieves the best balance between visual generation and understanding tasks, outperforms other methods in both domains.
4.4 Ablation Experiments
In this section, conduct ablation experiments our key designs and demonstrate the effectiveness of the proposed method. We use the same subset of training data in Sec. 4.3 if not specified.
Text-Aligned Codebook for Better MILM Initialization. One benefit of our TA-Tok is the learned codebook can be directly transferred to LLM for image embedding initialization, as discussed in Sec. 3.3. In Tab. 3, we show that initializing the LLM embeddings with TA Codebook leads to better performance on both understanding and generation tasks compared to random initialization. Another common approach pre-align, used in previous works [10, 70]. involves separate alignment stage with additional training data. While pre-align with 50M extra data can match TA-Tok's 's performance on generation, which further highlights the efficiency and effectiveness of our approach.
Generative De-Tokenizer. Tab. shows that AR-DTok performs well across resolution and scales, with 512px offering the best trade-off between quality and efficiency. Larger models (775M) perform slightly better, though smaller variants remain competitive. Dif-DTook achieves similar scores at 512px, but adapts quickly to generating high-fidelij images thanks to its pretrained diffusion backbone.

---

However, Fig. 6 (right) shows that the visual quality of generated images varies between de- de-tookensers, even when their scores are similar.
Saale-Adaptive Pooling for GranlaGrry Visual Tasks. The design of SAP makes our model produce multi-granularity visual tokens. In Tab. 5, show that visual understanding tasks require more tokens to capture image details. However, adding more visual tokens does not significantly improve image generation performance. On the contrary, longer visual sequences make it harder for LLMs to learn (e.g., 81 tokens are optimal at 10M training data). While longer sequences can enhance T2I performance as the training data increase, we found 169 tokens are already sufficient, which is similar to conventional T2I models [16, 75].
Shared Representation Unifies Visual Understanding and Generation. Previous works [62. 72] studied the mutual effect of both tasks. In Tab. 6, we further show that the two tasks can benefit each other as long as we shared visual representation. For Janus-style representation, jointly training does not decrease the performance of each task under separate training. But the two tasks can benefit each other with a shared representation such as VQVAE and our text-aligned representation, around 8.1% and 5.3% improvement on the generation task, respectively. We also notice an imbalance in the improvements between the understanding and generation tasks, which may be attributed two factors: (a) Task and supervision mismatch. Generation tasks benefit more from joint train because they requires understanding before generating coherent images. Understanding tasks, on the other hand, are more discriminative and do not directly benefit from generation supervision. (b) Task difficulty and data requirements. Understanding tasks require less data to achieve strong performance, whereas generation tasks are more complex and require more data. Thus, joint training benefits generation tasks more significantly.
Unified Multimodal Pretraining with Advanced Tasks. In the right table, we demonstrate that the proposed T12I and I2I tasks can further improve the generation performance, thus nar- rowing the gap between understanding and generation. Ratio:
Task
Ratio
Und
Gen
baseline
(3:0:0)
89.2
66.4
121
(2:1:0)
89.4
68.5
TI2I
(2:0:1)
89.8
70.1
the data ratio of T2I vs. 12I vs. TI2I.
121+TT2
(4:1:1)
89.8
70.7
5 Conclusion
We introduced Tar, unified model that bridges visual understanding and generation using a shared, discrete, text-aligned representation. By aligning image tokens with I.I embeddings via TA- Tok, and supporting scale-adaptive tokenization and generative de-tokenizers. Tar achieves strong performance across both tasks without modality-specifff designs. Our results show that fully discrete, semantic representation enables efficient and effective multimodal learning, moving toward true unification of vision and language.
Acknowledgments
This work is partially supported by the National Natural Science Foundation of China (Grant No. 62306261), and the SHIAE Grant (No. 8115074). This study was supported in part by the Centre for Perceptual and Interactive Intelligence, a CUHK-led InnoCentre under the InnoHK initiative of the Innovation and Technology Commission of the Hong Kong Special Administrative Region Government. This work is also partially supported by Hong Kong RGC Strategic Topics Grant STG1/E-403/24-N and CUHK SHHK--UUK(SZ2-GDD>> Joint Collaboration Fund YSP26-4760949.
References
[1] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. Technical report. Alibaba Group, 2023.
[2] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vi vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-v1 technical report. arXiv preprint 139223 2025.
10

---

[4] Tom Brown, Benjamin Mann. Nick Ryder. Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan. Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berne,, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in super-ud vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[6] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman, Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 11315-11325, 2022.
[7] Jiuhai Chen, Rifaa Qadri, Yuxin Wen, Neel Jain, John Kirchenbauer, Tianyi 7hou, and Tom Goldstein. Genqa: Generating millions of instructions from handful of prompts. arXiv preprint arXiv:2406. 10323, 2024.
[8] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- (sigma: Weak stroon training of diffusion transformer for 4k -ext-t--mmage generation. arXiv preprint arXiv:2403 046622 2024.
[9] Junsong Chen, Jincheng Yu, Chongjian Ge. Lewei Yao, Eaze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis, arXiv preprint arXiv:2310.00266, 2023.
[10] Xiaokang Chen, Zhiyu Wu, Xingehao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 2025.
[11] Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang. Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint 2rXiv:2402.033777
2024.
[12] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
general-purpose purpose lision-language models with
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pages 248-255. Ieee, 2009.
[14] Rumpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamlim: Synergistic multimodal comprehension and creation. arXiv preprint X30:::230..1 1399. 2023.
[15] Draw TLingAAL Megalith- 10m dataset. httpps:/huugiiggaace wttp://oo//acc./eeeeeeee megalith- 100m 2024. Accessed: 2025-05-09.
http:////nggggggcc.oooooo///t/ts co/daatasets/dravtiiggal
[16] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[17] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for nigh-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873 12883, 2021.
[18] Rongyao Fang, Chengqi Duan, Kun Wang, Linjiang Huang, Hao Li. Shilin Yan, Hao Tian, Xingyu Zeng. Rui Zhao, Jifeng Dai, Xihui Liu, and Hongsheng Li. Got: Unleashing reasoning capability of multimodal large language model for visual generation and editing. arXiv preprint ar0iv:2503. 10639, 2025.
[19] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulci Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benehmark for multimodal large language models, 2024.
[20] Yuying Ge. Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan, Making Ilama see and draw with seed tokenizer. arXiv preprint arXiv:2310.012/8.200 2023.
11

---

[21] Yuying Ge, Sijie Zhao, Jinguo Zhu. Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified gulti-granularty comprehension and generation. arXiv preprint arXiv:2404. 14396, 2024.
[22] Dhruba Ghosh, Hannaneh Hajishirzi. and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132--211 2023.
[23] Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang. Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[24] Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with Ilm for enhanced semantic alignment, 2024.
[25] Drew A Hudson and Christopher D Manning. Gqa: A dataset for real-world visual reasoning and compositional question answering. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700 00096679990000 2019.
[26] Diederik Kingma, Max Welling, et al. Auto-cncoding variational bayes, 2013.
[27] Black Forest Labs. Flux. tettpa://githinn httpstttu..coobbbeeee wwwm///ac/-o///eoee/e/eeeaeaeaee foncck--frrettlabbbee flux, 2024.
[28] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal ilms with generative comprehension. arXiv preprint arXivv2307.16125, 2023.
[29] Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.17245. 2024.
[30] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference machine learning. pages 19730-19742. PMLR, 2023.
[31] Xianhang Li. Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng. Yuyin Zhou, and Cihang Xie. What if we recaption billions of web images with llama-3? arXiv preprint 2rXin:2406.08478 2024.
[32] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 188144 2024.
[33] Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji- Ri-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[34] Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng. Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint 2oXiv:2405008444 2024.
[35] Zijie Li, Heary Li, Yichun Shi, Amir Barati Farimani. Yuval Kluger. Linjie Yang. and Peng Wang. Dual difffusion for unified image generation and understanding, arXiv preprint arXiv:2501. 002889 2024.
[36] Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:2404.0129) 2024.
[37] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model million-length video and language with ringattention. arXiv preprint arXiv:2402.08266 2024.
[38] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction uningg In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[39] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[40] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
12

---

[41] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, al. Mmbench: Is your multi-modal model all-around player? arXiv preprint arXiv:2307.022.
2023.
[42] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang. et al. Deepseck-vl: towards real-world vision-language understanding. arXiv preprint XrXivv2403.05525, 2024.
[43] Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang. Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. Unitok: A unified tokenizer for visual generation and understanding. arXiv preprint arXiv:2502. 20321, 2025.
[44] Xu Ma. Peize Sun, Haoyu Ma. Hao Tang, Chih-Ya Ma, Jialiang Wang, Kunpeng Li, Xiaoliang Dai, Yujun Shi, Xuan Ju, et al. Token-shuffle: Towards high-resolution image generation with autoregressive models. arXiv preprint arXiv:2504. 17789, 2025.
[45] Beijing Academy of Artificial Intelligence (BAAI). Infinity-instruct. Baaeeettts/AAAIInnnnina Instruct.
httpe///uugiingfacecoo
htnt:a//nttnttttttttttttteeteet 2024. Accessed: 2025-05-19.
[46] OpenAI. Gpt-4 technical report. arXiv:2303.08774, 2023.
[47] Xichen Pan. Satya Narayan Shukla, Aashu Singh. Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang. Zhiyang Xu, Jiuhai Chen, Kunpeng Li. Felix Juefei-Xu, Hou. and Saining Xie. Transfer between modalities with metaqueries. arXiv preprint arXiv:2504.06256 2025.
[48] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach, Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXi::2307.0195 2023.
[49] Qi Qin, Le Zhuo, Yi Xin, Ruoyi Du, Zhen Li, Bin Fu, Yiting Lu, Xinyue Li, Dongyang Liu, Xiangyang Zhu, Will Beddow, Erwann Millon, Wenhai Wang Vietor Perez, Yu Qiao, Bo Zhang, Xiaohong Liu, Hongsheng Li, Chang Xu, and Peng Gao. Lumina-image 2.0: A unified and efficient image generative framework, 2025.
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMI.R, 2021.
[51] Tal Ridnik, Emanuel Ben- Ben-Baruch, Asaf Noy, and Lihi Zelnik-Mand Imagenet-21k pretraining for the masses, arXiv preprint arXiv:2104.1097 2021.
[52] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF conference on computer vision and puttern recognition, pages 10684-10695, 2022.
[53] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Muliis, Mitchell Wortsman, et al. Laion-5h: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 01225278-52294 2022.
[54] Team Scawead, Ceyuan Yang, Zhijic Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, et al. Scaweed-7b: Cost-cffective training of video generation foundation model. arXiv preprint (=Xiv:2504008633 2025.
[55] Keqiang Sun, Junting Pan, Yuying Ge. Hao Li. Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang. et al. Journeydb: A benchmark for generative image understanding. Advances in neural information processing systems, 66:49659-49677 2023.
[56] Peize Sun, Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng. Ping Luo, and Zehuan Yuan. Autore- gressive model beats diffusion: Llama for scalable image generation. arXiv preprint XXXi:22400006225 2024.
[57] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.132666 2023.
[58] Chameleon Team. Chameleon: Mixed-modal early- fusion foundation models, 2024.
13

---

[59] Teknium. Openhermes. cessed: 2025-05-19.
http://awwaa.oaaaattee/eeseess/s http:///ww....aaaae///////aaaaaa 2023. Ac-
[60] Teknium. Openhermes 2.5: An open datasct of synthetic data for generalist Ilm assistants, 2023.
[61] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances neural information processing systems, 37:84839-84865, 2024.
[62] Shengbang Tong. David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412. 14164, 2024.
[63] Hugo Touvron. Thibaut Lavril, Gautier Izacard, Xavier Martinet, Manie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv:2302.13977 2023.
[64] Michael Tschannen, Alexey Gritsenko, Xiao Wang. Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin. Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual encoders with improved semantic understanding, localization, and dense features. arXiv preprint ArXi::5002. 24771 2025.
[65] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[66] Vivym.
Midjourney prompts dataset. 2023. Accessed: 2025-05-09.
[67]
Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu, Illume: Illuminating your Ilms see, draw, and self-enhance. arXiv preprint arXiv:2412.00663, 2024.
[68] Hanyu Wang. Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava, Larp: Tokenizing videos with learned autoregressive generative prior. arXiv preprint arXiv:2410.21264, 2024.
[69] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang. Fan Zhang, Yueze Wang, Zhen Li. Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.1888 18869, 2024.
[70] Chengyue WU, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.138: 13848, 2024.
[71] Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable multi-moda, generators. arXiv preprint arXiv:2412.0433 043.2. 2024.
[72] Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Zhonghua Wu, Qingyi Tao, Wentao Liu, Wei Li, and Chen Change Loy. Harmonizing visual representations for unified multimodal understanding and genera- tion, 2025.
[73] Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang. Ligeng Zhu, Enze Xie. Hongxu Yin, Li Yi, et al. Vila-u: unified foundation model integrating visual understanding and generation. arXiv preprint
(tt.:2409.044299 2024.
[74] Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han. Sana: Efficient high-resolution image synthesis with linear diffusion transformer, 2024.
[75] Euze Xie. Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Chengyue Wu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen. Han Cai, et al. Sana 1.5: Efficient scaling of training-ting -tme and inference-tine compute in lincar diffusion transformer, 2025.
[76] Jinheng Xie, Weijia Mao. Zechen Bai, David Junhao Zhang, Weihao Wang. Kevin Qinghong Lin, Yuchao Gu. Zhijie Chen. Zhenheng Yang. and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint 2408. /2528, 2024.
[77] Zhangchen Xu, Fengqing Jiang. Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Aligmment data synthesis from scratch by prompting aligned lmss with nothing. arXiv preprint 240: 08464, 2024.
14

---

[78] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, al. Qwen2. technical report. arXiv preprini arXiv:2412.15115, 2024.
[79] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang. Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9567, 2024.
[80] Xiang Yue, Tuney Zheng. Ge Zhang. and Wenhu Chen, Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems, 2024.
[81] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1195 11986, 2023.
[82] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao, Llama- adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16/99, 2023.
[83] Yue Zhao, Fuzhao Xue, Scott Reed, Linxi Fan, Yuke Zhu, Jan Kautz, Zhiding Yu, Philipp Krähenbühl, and De-An Huang. Qlip: Text- Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation. arXiv preprint XrXiv:2502.yyyyx 2025.
[84] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model, arXiv preprint arXiv:2408. 11099, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification:
The abstract and introduction (Sec. clearly state the paper's core contribu- tions. These claims are well-supported by the methodology and experimental results (Sec. 4) presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification:
The paper has discussed the limitations in the appendix (Sec. I).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [NA].
Justification: The paper does not contain formal theoretical results or proofs. It focuses on model design, representation learning, and empirical evaluation across understanding and generation tasks.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes].
Justification: The paper describes the full architecture of T'T--ook (Sec. 3.1) and the generative de-tok te-tokenizers (Sec. 3.2), specifies the datasets used, training objectives, model sizes, and evaluation benchmarks (Sec. 4). Additional implementation and training details are provided in the appendix to facilitate reprodueibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
17

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No].
Justification: We are not releasing code and data at submission time. However, we plan to make both publicly available with full instructions in the arXiv and camera-ready versions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. ccc/pbblcc/eeee wwtcc//guiee//eee Cilees//ooSSuubisee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: The paper provides full details on training and evaluation settings, including dataset sources, model configurations, training resolutions, and evaluation benchmarks (Sec. 4). Additional hyperparameters and training strategies are described in the appendix to ensure clarity and reproducibility. Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification: We performed multiple runs for key experiments confirm the consistency results across random seeds. While error bars are not shown in the paper due to space constraints, observed low variance and consistent trends across runs.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification: We provide sufficient information on the computational cost in the appendix (Sec. E).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https: /ttp:////euusspeeeeee cc htpppblcc//hiiccsuuueeeeeeeeeeee
Answer: [Yes].
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
19

---

Justification:
We have discussed the societal impacts of the work in appendix (Sec. J).
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimesss considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: The model is trained with open- open-sourced datasets and models, without obvious risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
The creators or original owners of the open-source assets (models and data) are properly credited in the paper. The specific version of each asset mentioned (Sec. 4), along with the relevant license (e.g.. CC-BY 4,0 or other applicable licenses). The terms of use are respected, and where available, URLs for the original assets are provided.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
20

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. . com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
We not plan to release new assets at the submission time.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
We do use crowdsoureing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA].
Justification: We do not crowdsourcing and research with human subjects.
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA].
Justification: The core methods developed by the authors.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy (https: htt.ppps...coooeeeeeeee//2/525 for what should or should not be described.
22

---

Appendix
A
Overview
Sec. B: Additional method
Sec. C: Datasets
Sec. D: Additional Implement Details
Sec. E: Additional Ablation Experiments
Sec. Additional Main Results
Sec. G: More Visualization
Sec. H: Limitation
Sec. I: Social Impact
B Additional Method
B.1
Training Recipe
TA-Tok Training TA-Tok is trained on both 100M raw web images and 100M festhetic-filtered images from LAION-5B [53] balance its ability encoding general images for understanding and high-quality images for generation. Note we only use the images (without text) in LAION-5B, The aesthetic images are with resolution above 512px and resized 384px. To enable Scale- Scale-Adaptive Pooling and Decoding, we random select scale from {1, 2, 3), resulting in {729, 169, 81) tokens. Since learning longer sequence is usually harder,
the sampling ratio of scale {1, 2, 3) to (2 1::::0 1). The training of De- re-Toeenizers and MLLM also follows the same sampling ratio.
De-Tokenizer Training The training of De-Toeenizers is to align low-level image representations (e.g., VAE [26] and VQVAE [65]) with TA-Tok's semantic representation. For AR-DTok, train a series of models with different resolutions, ranging from 256px to 1024px. (a) 256px. Since AR-DTok is only developed from high-quality image generation, train 256px AR-DToo 50M aesthetic-filtered images from LAION. (b) 512px. We finetune 256px AR-DTok to 512px using 23M synthetic images (described in Sec. 3.4 Data Curation). 1024px. To demonstrate the resolution decoupling between our TA-Tok and De- Te-Tokenizer, we further finetune AR-DTok from 512px to 1024px using 3M synthetic images of 1024px resolution from [15]. For Dif-DTok,
directly leverage pretrained text-to-image model, SANA-0.6B [74] as the starting model. We train Dif-DTok with 23M synthetic images 512px. Notably, we observe that Dif-DTok achieves good convergence with just 5M training samples. However, we continue training on the full 23M dataset to ensure broader image converge and diversity.
MILMM Prompt Format Since TA-Tok and MLLM connected with discrete tokens, expand the vocabulary of MLLM and convert TA-Tok tokens into text that MLLM can understand. The newly added vocabularies are:
<im_start> <im_enda <SO>, <S1>, <S2>, <IO>, <I1>, .... <I65535>,
where
and <im_end> are the starting and end tags, <S <S[0-2]> are the scale tokens and <I[065553)> image tokens. Following Qwen2.5-Instruct [78], the prompt of one-turn conversation is formatted as:
<| sii__starte / >>user (n)) im__eddlll | < im_start Dasssistant /
where < < iimmsstrrr//// > and <
the starting and end tags of an instruction (Q) response (A). To distinguish image understanding and generation, we use different prompt format for them. For image understanding.
g=(textt)<I1xxI1>I11111 ... <I520>(text), A=(tx) response),
23

---

where we do not add image start and end tags. Note the length of image tokens can be /729, 169, 81), depending on the selected scale. For image generation, the prompt is:
(= prompt), A=<dm_starrt<(1111111111111 in
where we add image tags to the prompt. During inference, the generation prompt is:
<I im_ start >user Duser\n(text prompt>< im_ ><| _ellimmstarrr /tttart/aassstaaat >assistant lnniinnnniiaaaaaaaa start>,
The image start tag <im_ start> at the end of prompt encourages the model to generate image tokens
instead of text tokens.
B.2 Discussion with Related Methods
Our approach utilizes discrete visual tokens, which differs significantly from traditional continuous visual representations, such as MetaMorph [62] and MetaQuery [47]. MetaMorph constructs visual representation by regressing SigLIP latents through an additional vision head, followed by training diffusion decoder with an image autoencoding objective. However, this regression and reconstruction pipeline limits the ability to generate diverse images from prompt, lacking the natural generative process seen autoregressive language models (LLMs). contrast, our discrete visual tokens allow direct integration with autoregressive sampling strategies, enabling diverse and controllable generation within unified token space.
MetaQuery, on the other hand, uses frozen multimodal LLM for condition extraction, but the image understanding and generation modules are decoupled. The understanding is handled by the frozen LLM, and the generation is handled by diffusion decoder. This approach is closer to traditional diffusion-basse based methods (e-g.. DiT) rather than unified model with shared representations. In contrast, our method explicitly unifies both image understanding and generation within a single discrete semantic space (i.e., TA-Tol tokens).
The role of our De- Te-enizii also differs from that of traditional diffusion decoders [20]. Rather than generating new images conditioned tokens, the Dif-ITTok reconstructs the input image from discrete visual tokens produced by TA-Tok. These tokens already preserve essential 2D structure and visual details (e.g., a 27x27 grid of 729 tokens), and Dif-DTok's 's purpose map these structured tokens into pixel space. Therefore, we view the De-Tokenizer as a modular component within our pipeline, rather than full generative decoder.
Furthermore, the generated visual tokens are not merely conditioning signals but 2D visual tokens that encode all necessary information for image reconstruction. As demonstrated in Appendix Tab. 11, images encoded by TA-Tok can be faithfully reconstructed by the De-Tokenizer. This is a key distinction from MetaQuery, where image generation heavily relies diffusion model handle both multimodal understanding and image generation. In our approach, a simple De- Tokenizer is sufficient to convert TA-Tok's tokens into VQ-VAE VAE tokens, eliminating the need for complex diffusion model.
C Datasets
C.1 Dataset Curation Details
The common text-to- text-to-image data collection process include complex pipeline, e.g., aesthetic filtering, watermark/logo detection and image-text alignment filtering. Instead, training with synthetic datasets is a more efficient solution [10]. We only need to focus on the quality of text prompts. The state-of- the-art generation models can provide high-quality images of given prompt. Therefore, we develop simple and efficient data curation pipeline:
Image Caption To encourage the diversity prompts, we use an open-source MILM, Qwen2.5- VL-7B[3] to generate long and detailed captions for multiple datasets, including image classification datasets ImageNet- [13] nad ImageNet-21 [51], and photo dataset Megalith- 10M [15]. The caption prompt to Qwen2.5-VL

---

Here some text-to-image prompts:
Example I: (long and detailed prompt 1)
Example 2: [prompt 2)
Example 3: (prompt 3)
Now generate prompt for this image. Do copy the above content. Just follow their prompt style. Do not output anything else. The image is: [image]
This step is similar to re-caption, but differently, we only need the text prompts generated from these images, no matter the quality of these images.
Synthetic Image Generation In the above step, we have obtained millions of text prompts real world images. However, users may not input such long prompts of real world objects to the model, they often provides short, simple, and creative prompts made up few words, e,g., "boy, dog. the grass, happy play, 4K" Therefore, Midjourney-Promph [66]. Although JourneyDB a text-to-image dataset, we argue that its images generated by very early generation models like MidJourney before May. 2023. Here adopt state-of- of-the-art and fast model, FLUX. --schnell [27] as the image generator. With only 4 sampling steps and 512px resolution, and user prompts. Finally, we collect 23M high-quality synthetic dataset.
also collect user prompts datasets: JourneyDB [55] and
quickly generate lot of images using prompts from Qwen2.5-V2
C.2 Unified MILM Pretraining Data
Except the traditional pretraining tasks, i.e, image-to-text (I-+T), text-to-image (T-+), text-only (I-+),, we also propose two new tasks: anage-to-image (I-+I) and (ext-mmage-to-imal (TI++I).
Image-to-Image Given text prompt, we FLUX to generate two images with different seed. This task requires the MILM to understand the input image first and generate similar one with the same semantic. The prompt is:
Q=Generate image similar {image 1), A=(image 2}
Image-Text-to-Imagg Given a text prompt, we first split into two parts using an open-source LLM Qwen2.5-7B [78]:
For given prompt, you need to replace part of its content with a placeholder <image>.
For example:
INPUT:
In a cinematic and realistic portrayal of 1920s girls at college, we see them studying in university with dark academic atmosphere, where haunting and creepy ghost story unfolds.
OUTPUT EXAMPLE
I I"prompt": "In a cinematic and realistic portrayal of <image>, we see them studying in university with a dark academic atmosphere, where a haunting and creepy ghost story unfolds.". "<image>": "1920s girls at college.")
OUTPUT EXAMPLE 2: (omitted)
Now will give you another prompt, you need output one sample for each prompt. Just output the result in json format, and do not output anything else.
"https://www wwwdmmiddddee.e. com
25

---

Table 7: Dataset Summary.
Model
Stage
Data
Type
Size
TA-Tok
LAION (100M) [53]. LAION Aes (100M) [53]
Image
200M
256px
LAION Aes (50M) [53]
Image
50M
AR-DTok
512px
Gen23M [13, 15, 51, 66]
Image
23M
1024px
Gen3M [15]
Image
3M
Dif-DTok
512px
Gen23M [13, 15, 51, 55, 66]
Image
23M
[39]. DataComp (20M) [31]
12T
32M
Gen23M [13, 15, 55, 66]
T21
23M
Pretrain
Gen15M [13, 15, 51, 55)
121.
TI21
15M
MLLM
Magpie (4M) [77]. WebInstruct (12M) [80]
Text
28.5M
OpenHermes [59]. GenQA [7]. Infinity-Instruct Instruct [45]
LLaVA-v1.5 (665K) [38], LLaVA- ALaVA-Recap-5555 Recap-558K [39),
12T
1.6M
LLLVA-Recap-I1888 [39], Self-Reflect-30KK
SFT
Gen-SFT-1.6M
T21
1.6M
Magpie (IM) [77]
Text
1M
Table 8: Training Parameters.
TA-Tok
AR-DTok
Dif-DTok 512px
MILM
config
256px
512px
1024px
Prertain
SFT
learning rate Ir schedule optimizer
2c-4
4e-4
lc-4 cosine
lc44
lc-4 constant CAME
5c-5
cosine
consine
AdamW
AdamW
Adam
8,=0.9,8=00.999
optimizer params
R1=0.9,32=0.99
81 91=0.9,2200.95
88=0.99999
weight decay
le-4
0.05
0.0
0.0
input resolution
384
256
512
1024
512
384
warmup epochs
0.04
0.04 1
0.01
0.03
epochs
total samples total batch size
200M
50M
23M
3M
23M
100M
4M
512
768
96
48
48
1024
256
codebook loss reconstruction loss gradient clip token drop prob
1.0
1.0
1.0
1.0
0.1
1.0
0.1
0.1
data ratio
((en)
The task is to generate image based on a multimodal prompt. Therefore, to generate the target image, the model must understand both the text part and the image part of the multimodal prompt. At the training time, organize the data
Q=(text part Mimagge 1)teet part 2}, A=(image 2}.
Using the same data source as Sec. C.I, we finally curate 15M dataset for I~I and TI-+I tasks.
Relations to Other Tasks Our proposed tasks have similar format of tasks like image editing and subject-driven image generation. However, their goals are different. For image editing, the goal to modify part of an input image, following a text instruction. This task focuses on lowelevel image detail. For subject-driven image generation, its goal is to compose the objects of the input image into a scene, emphasizing visual consistency and preserving pixel-level details. contrast, our task is to mitigating the modality gap between visual understanding and generation. Rather than persevering exact pixel information, we focus on capturing and conveying the semantic content the input image.
26

---

C.3 Dataset Summary
We summary all the training data in Tab.
D Additional Implement Details
We list the training hyper-parameters in Tab. 8
E Additional Ablation Experiments
Ablation of Text-Aligned Codebook As shown in Tab. we compared different settings of TA Codebook. Tab. 9 (a) is a conventional setting, with random initialized, low dimension codebook: Tab. (b) increases the codebook dimension to 1536, but fails converge during training; Tab. (c) is our LLM embedding aligned codebook with large vocabulary and high dimension. The results indicate that our TA Codebook (c) outperforms the conventional codebook (a) by a large margin in understanding tasks and is comparable on the generation task.
Table 9: Ablation Different Codebook Settings.
setting
init
size
dim
GQA
MME
POPE
DPG
(a)
random
32768
16
59.4
1261
86.7
70.9
(b)
random
32768
1536
not converge
(c)
LLM
65536
1536
61.0
1428
87.4
70.6
Self Reflect Since unified MILM can handle both image understanding and generation tasks, we leverage its understanding ability to assess the generation quality, i.e., image-prompt alignment, which is relative simple task for modern MILMMs. Therefore, we build Seff-Reflect3340K a dataset to judge image-prompp alignment. We collect set of prompts in the style of GenEval and DPG Bench, and query Qwen2.5-VL-
generate judgments. In Tab. 10, the model trained with Self Reflect show consistent improvements on the visual generation task. Notably, weaker models benefit more: the relative improvements 0.04 for 10K-step model, but 0.016 for 60K-step model.
Table 10: Ablation of Self Reflect. The evaluation metric is GenEval overall score.
Train step
10K
20K
30K
40K
50K
60K
baseline
0.717
0.747
0.753
0.757
0.759
0.764
w/ Self Reflect
0.757
0.779
0.788
0.784
0.785
0.780
+4.0%
+3.2%
+3.5%
+2.7%
+2.6%
+1.6%
Table 11: Performance on ImageNet 256 Benchmark. FID: Frechet inception distance. IS:
inception score. cfg: classifier-free guidance.
Model
#Param
cfg
FID
IS
Precision
Recall
Llamagen- Tok
2.19 (rFID)
Llamagen XL
775M
1.75
2.62
244.08
0.80
0.57
Llamagen-XXe XXL
1.4B
1.75
2.34
253.90
0.80
0.59
TR-Too-AR-
1.2B
10.0
2.60
208.46
0.76
0.64
Image AutoEncoding Performance of TA-Took Our TA-Tok and de-tokenizers can be viewed as an image autoencoding pipeline, where an input an image is encoded and then reconstructed. Since our de-tokenizers are based on generative models (AR or diffusion), one concern is whether they can faithfully recover the original image. In Tab. 1, we compare our method against LLamagen's 's tokenizer (Llamagen-Te -ook), Llamagen-Xie XL//genn-XXXXXI... on the ImageNet 256x 256 benchmark. Note that we take TA-Tok's 's output as condition and I Llamagen-XI/XXXX XXXXXL. are conditioned on class labels, so the results not directly aligned. Anyway, this serves comparison assess the autoencoding
27

---

capability of our method. Our method achieves comparable FID compared with Llamagen-XX XL, demonstrating effective reconstruction quality. We also observe that our model requires higher classifier-free guidance (10.0) compared with Llamagen's 1.5, indicating stronger reliance of condition.
Compositional Generation Emerges from Unified Pretraining Tasks In Sec. 3.4, we propose unified pretraining with new tasks: image-to-image (I~+I) and text-image-to-mmage (TI+1I, which encourages multimodal prompt conditioned generation. As shown in Fig. 7, Tar demonstrates emergent aubject-driven generation and reference-basee based style transfer.
<image> the sidewalk crowcedd
<imege> green grass sunflowers
<image?
A photo boof the styla of dimage>
photo a bird the style of <image>
photo of potted plant the styla of <images
A photo of zebra the style of <image>
pphoto of suitcase the style of <imege>
A photo of bench in the style of <image>
OPEN
Figure 7: Emergent Compositional Generation Ability. The first row denotes subject-driven generation. The middle and bottom rows are reference-ber based style transfer. For each example, the left image is <image>, the right image is the generated image.
Additional Main Results
We list the full visual generation results in Tab. 12 and Tab. 13.
G More Visualization
We give more visualization of our model on the text-to-image generation task, shown in Fig. Notably, our model can follow Chinese prompts even it is not trained with Chinese prompt dataset, by leveraging the multilingual ability the base LLM, Qwen2.5 [78].
H Limitation and Future Work
Our method has several limitations. First, the vector quantization in TA-Tok introduces quantization errors. Despite mitigating this with a larger codebook and extended training, some information loss remains particularly in tasks requiring fine-grat fine-grained understanding, such as OCR. This could be improved by adopting longer visual sequences and incorporating techniques like Token-Shuffe [44]. Second, while our method excels at generating diverse images. it underperforms in accurately reconstructing the input image compared to traditional tokenizers [65]. This limitation stems from using generative models as de-tokenizers. A potential solution is to train the de-tokenizer as an image super-resolution model better preserve local consistency between the input and reconstructed images.
28

---

I
Societal Impact
This work advances unified MILMs by introducing shared, discrete semantic representation that enhances both visual understanding and generation, enabling efficient language-vision integration for applications like assistive tools, creative content, and education.
However, high-quality image generation also poses risks, including potential misuse for misinfor- mation manipulation. While model is not identity-ppecifie, downstream use should include safeguards such as watermarking and prompt filtering. We advocate for ethical use, emphasizing fairees,, robustness, and transparency.
Table 12:
Visual Generation Results on GenEval [22].
Method
Single Obj.
Two Obj.
Counting
Colors
Position
Color Attri
Overall
Generation Only Model
SDv1.5 [52]
0.97
0.38
0.35
0.76
0.04
0.06
0.43
PixArt-a [9]
0.98
0,50
0.44
0.80
0..8
0.07
0.48
SDv2.1 [52]
0.98
0.51
0.44
0.85
0.07
0.17
0.50
Emu3-Ges Menn--Genna [69]
0.98
0.71
0.34
0.81
0.17
0.21
0,54
SDXL [48]
0.98
0.74
0.39
0.85
0.15
0,23
0,55
DALLE3 [36]
0.96
0,87
0.47
0.83
0.43
0.45
0.67
SD3-Medium [16]
0.99
0,94
0.72
0.89
0.33
0,60
0.74
SANA-1.5 [75]
0.99
0,93
0.86
0.84
0.59
0.65
0.81
Unified Model
Chameleon-P2 [58]
0,39
LWM-7B [37]
0.93
0.41
0.46
0.79
0.09
0.15
0.47
SEED-X- 13B [21]
0.97
0.58
0.26
0.80
0.19
0.14
0.49
Show-o-1.38 [76]
0.95
0.52
0.49
0.82
0.11
0.28
0.53
Transfusion-2B [84]
0.63
D-DiT-2B [35]
0.97
0,80
0.54
0.76
0.32
0.50
0.65
IILUMME7B [67]
0.99
0.86
0.45
0.71
0.39
0.28
0.61
Janus-1.33 [70]
0.97
0.68
0.30
0.84
0.46
0.42
0.61
Janus-Pro-1BB [10]
0.98
0.82
0.51
0.89
0.65
0.56
0.73
Hlammon-1.5B [72]
0.99
0,86
0.66
0.85
0.74
0.48
0.76
Janus-Pro-7B [10]
0.99
0.89
0.59
0.90
0.79
0.66
0.80
Tar-1.5B (Ours)
0.99
0.91
0.76
0.81
0.57
0.51
0.76
w/ Self Reflect
0.99
0.92
0.77
0.81
0.62
0.55
0.78
Tar-7B (Ours)
0.98
0.92
0.83
0.85
0.80
0.65
0.84
w/ Self Reflect
0.98
0.93
0.86
0.85
0.80
0.70
0.85
Table 13: Visual Generation Results on DPG Bench [24].
Method
Global
Entity
Attribute
Relation
Other
Overall
Generation Only Model
SDv1.5 [52]
74.63
74.23
75.39
73.49
67.81
63.18
PixArt-o [9]
74.97
79.32
78.60
82.57
76.96
71.11
Emu3-Gen [69]
85.21
86.68
86.84
90.22
83.15
80.60
SDXL [48]
83.27
82.43
80.91
86.76
80.41
74.65
Playground v2.5 [29]
83.06
82.59
81.20
84.08
83.50
75.47
Hunyuan DiT [34]
84.59
80.59
88.01
74.36
86.41
78.87
PixAnt-E
86.89
82.89
88.94
86.59
87.68
80.54
DALLE3 [36]
90.97
89.61
88.39
90.58
89.83
83,50
SD3-Mcdium [16]
87.90
91.01
88.83
80.70
88.68
84,08
SANA-1.5 [75]
84.70
Unified Model
Janus-1.3 1.3B [70]
82.33
87.38
87.70
85,46
86.41
79.68
Janus-Pro- [10]
87.58
88.63
88.17
88,98
88,30
82.63
Janus-Pro-75 Pro-7B [10]
86,90
88.90
89.40
89.32
89.48
84,19
Tar-1.5B (Ours)
83.59
89.35
86.91
93.50
80.80
82.96
w/ Self Reflect
84.17
88.48
87.83
93.38
84.07
84.10
Tar-7B' (Ours)
83,98
88.62
88.05
93.98
84.86
84.19
w/ Self Reflect
84.09
88.60
88.78
93.59
85.15
84.65
29

---

An strdion
worting soitted, coesiied urics worke rolenting The ctunti mosueey cartciial spea rith
comeert biikk badggroung ahonca untiiet cinernctinn ata acailudinn, scata (ffondd banroe, vocddm plar abount tuctuaa Euroooluerr ASArStalos
palde--oo
sntuaaion with moveaai angie detaled caat
D-00 Food
farul background
Livet dlat cando wee osker antia weod pravaable cark
right
acrttd puice Whth green an grean planns very calatul lower persrre bolke tachground fstash arrffll serer parto puldi lovenbs dby contsirsd.
pa
Posterr noverrs, Clild Cook youu it Plarncet oteer
sprikes
Crouke of Sarto Vogre scaet
Depoedt Journyy End Done Daller frows chardder reporbdin Pricens shoutd have Priseet's calee mos exparssion, put with vustl coleiau Per concrnt tip. Sumoard ofternd glowe synterlcnng of viador magiiaal
paar
ald
"Wmrrr's wards
rudies with Soder blat coltate gron and anchhessir
Wiroom Sepeoee claki coler autine come
chring
organn forma, grwagy wevembbed lowed whi Horen, 3D, storuct ligh budkgssud coeelio flowets
Doppmin Ballory The bomade of am of and Bovara peccr
A bile detal ad ciyy on
handa
sheel daplayed
Eowers dristing btylin
Baaulls wo an Forn pratoeal
Paintlyy Shin, iat of palit sople paint wotercoiie, del sulennse cn foood yoar in dryln Whoid VD's dogh cleer Cluud brrve Stynn of Roon fin vart Siylt of Canille
anchindt
veacin poae
crean mor Ergirs
Aistnne phatogguh of par and white
goons Selt The ont porr dighlly whth podingige rnvontt wil caln my reriiiie in tiri woth with Rudin wilii potcone an cain and
my for and car gaite variie The image fats degii Soli phoiiingg the
soriile vally ttaned Wee denoaaalloo mannal,
diniing wore the pain to, pays
MAAA SE AAE i Ci- aa do in ou or BB3 R SPIEW sort 9C300
volil ai offandeccio roppinn into callt baae
Burning leine that incentine chdle pointed eare rolling hile shope the tate cunne fovees and resciowe bersow parses
griing
reea litre smaliryy vell muke polor pelidss ott arell aito
ltatraiin tole or Quality
beskgrann
arveball atrziphers delabldd Feal for books glviing coult Seeding movernet cinerae apatt Phoogass To apra virarccee CoTo amentary dibrs
The made shooosses metioukosn lounly and moodonatttia Fiburs The ubot ini icutddy desig red wethwwhhaaiis componennts includingg dinggn heodddddi with two long pestrud rng tntemnas with verous compantnennts omno and logo Tho figar loyllyyd on wouztmn surfaco, and chobbccke sund Siterod www.hissiiiiiiiiiioo..o cmigrisicing sickiingnn tho robut the mene sulecot, The ammbar suggectt a colmn and serens encienment, possiily a roem with ambiene Lighiin.
yellow vest with white ahtit undermathh and boerse. Above character, thane's spesch tutrol with symoot instieit, bas ligoundd there ane twens cheriee, and in uma liftls mboie lhe cremus The overll ant wnde the imagg a joyhul aniin
cheericl expressten. The character westing
& inver chowioses surral soene ahore kis rnd poie ots weseen hodding for cark. Reskde bin celescal cruni couture with a Mar bodly legtuingg swiriin galawy like patterr numerosa tartacio. The bucictopia a sar- right dhy creating an otherwerldy anc thought provoking scrooshere.
sycs. Rowers purppirr. The charecter sumourced by col, sutumnal ng. There pumplira moon shinnig siightn sumo inded by mrst www.sardeee... ondideeceeedd with colornl light..
magh
aarge
pumplón fecm, and abreematic,. The buclaground featurea serene lake with full
5. image ettercsses 3D-renteed shornctw poodiitty a femraia with hair nyne. She adterned wii comon en ber haad and amors black top whis parr prnncient print
deelgns
the bacogroordd Argaggrann
Jeart, straed bodo and bacc claoo Alown momeomm nomby th (t
Figure 8: Visual Generation Results. We use Tar- 7B and 1024px AR-DTok to generate these images. Most prompts are from the web, and a few prompts are from [49, 75J. Zoom in for better view.
30


Reviews:
[{"review_id": "w4PsPtaXeX", "reviewer": "Reviewer_fcrb", "summary": "This paper tries to unify the visual understanding and generation within a shared discrete semantic representation. The core design is the Text-Aligned Tokenizer, which builds a discrete semantic visual representation initialized from LLM embeddings. This paper also proposes the Scale-Adaptive Pooling to adjust the number of tokens for visual understanding and generation. Finally, autoregressive and diffusion Generative De-Tokenizers are introduced to enable visual generation.", "questions": "Please refer to the Weakness I have outlined and provide clarifications for each point individually. I would love to raise the rating is my concerns are well-addressed.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "[Strength] \n\n1. The thinking of the visual representation outlined in the introduction is clear and insightful.\n\n2. The paper is well-written and easy to understand. The experiments and ablation studies are detailed and comprehensive.\n\n[Weakness]\n1. **My biggest concern is the motivation and rationale behind the proposed TA tokenizer**: (1) The continuous SigLIP2 features already provide rich semantic information. As shown in Section 4.3 and Figure 5, the proposed visual representations actually lead to worse visual understanding performance. (2) If the authors’ goal is to use a single visual representation, it is unclear why this representation must be discrete. Methods like MetaMorph [1] and MetaQuery [2] use continuous single visual representations effectively. This paper lacks detailed comparison with them in Section 4.3. Methods like VILA-U [3] and UniTok [4] employ discrete representations because they leverage unified tokenizers, where the generated tokens can be directly de-tokenized into images. However, in this paper, the generated tokens are only used as conditions, and the so-called de-tokenizer still needs to synthesize the image based on these conditions. This undermines the justification for requiring discrete tokens.\n\n2. The authors emphasize in line 32 that \"both modalities are learned within a single latent space.\" However, I question the validity of this statement. In this paper, the visual generation process is not end-to-end: one model is used to generate the condition, and a separate model is used to project that condition into images.\n\n3. The *Understanding Only Model* listed in Table 9 are outdated, and comparisons with them are not meaningful. The authors should include state-of-the-art models for a more relevant and fair evaluation.\n\n4. In lines 163–169, the authors compare the training and inference efficiency of autoregressive and diffusion models, but provide no supporting evidence. I am not convinced by this conclusion. To strengthen the argument, the authors should include empirical data such as training time and inference latency.\n\n[1] Tong, Shengbang, et al. \"Metamorph: Multimodal understanding and generation via instruction tuning.\" arXiv preprint arXiv:2412.14164 (2024).\n\n[2] Pan, Xichen, et al. \"Transfer between modalities with metaqueries.\" arXiv preprint arXiv:2504.06256 (2025).\n\n[3] Wu, Yecheng, et al. \"Vila-u: a unified foundation model integrating visual understanding and generation.\" arXiv preprint arXiv:2409.04429 (2024).\n\n[4] Ma, Chuofan, et al. \"Unitok: A unified tokenizer for visual generation and understanding.\" arXiv preprint arXiv:2502.20321 (2025).", "was_revised": true, "final_justification": "Although I’m not fully convinced, I think it is a meaningful exploration backed by solid experimental support. I choose to raise the score.", "rebuttal": "Dear Reviewer fcrb,\n\nThank you very much for your thoughtful and constructive review.\n\n**Q1: Comparisons to continuous visual representations (MetaMorph and MetaQuery)**\n\nAns: We appreciate your suggestion to clarify the motivation behind using discrete representations, and we will expand in the revision to include a more detailed  comparison with MetaMorph and MetaQuery.\n\n**(1) MetaMorph** builds a visual representation by regressing SigLIP latents through an additional vision head, followed by training a diffusion decoder with an image autoencoding objective. Due to this **regression and reconstruction pipeline**, MetaMorph is not able to **sample diverse images** from a prompt. It lacks a natural generative process like that in autoregressive LLMs. In contrast, our discrete visual tokens allow **direct integration with autoregressive sampling strategies**, enabling diverse and controllable generation in a unified token space.\n\n**(2) MetaQuery** leverages a frozen MLLM for multimodal condition extraction. However, the image understanding and generation modules remain **decoupled**. The former is handled by the frozen MLLM, the latter by a diffusion decoder. This is **conceptually closer to traditional diffusion-based methods** (e.g., DiT) than to a unified model with shared representations. In contrast, our method explicitly aims to unify understanding and generation through a single discrete semantic space (i.e., TA-Tok).\n\n**(3)** Why do we refer to Dif-DTok as a “De-Tokenizer” instead of a diffusion decoder (generator)?\nOur Dif-DTok is trained to **reconstruct** the input image from discrete visual tokens produced by TA-Tok, rather than to generate new images conditioned on TA-Tok’s tokens. Since TA-Tok tokens already **preserve 2D structure and rich visual details** (e.g., 27*27=729 tokens 2D grid), Dif-DTok’s role is to **faithfully** transform these structured tokens into the pixel space. We believe “de-tokenizer” more accurately captures its function as part of a modular pipeline, **rather than a full generative decoder**.\n\n**(4) The generated visual tokens are not merely conditioning signals, but 2D visual tokens that encode the full information necessary for image reconstruction—including structure and detail.**\nThis is demonstrated in Appendix Table 5, where images encoded by TA-Tok can be **faithfully reconstructed by a De-Tokenizer**. We believe it is a **key difference** from MetaQuery. Our Tar model can **directly compose and organize visual content** in discrete token space, similar to how LLMs generate coherent text sequences. Importantly, our system **does not rely on a strong diffusion model** to handle both multimodal prompt understanding and image generation like MetaQuery. Instead, a simple De-Tokenizer that reliably converts TA-Tok’s tokens into VQ-VAE or VAE tokens is sufficient.\n\nTherefore, we consider our method to be **in line with prior work such as VILA-U and UniTok**, but with a more expressive and **LLM-aligned tokenizer design**. While we acknowledge that discrete tokens currently underperform continuous features in some understanding tasks. However, as noted in Appendix I, we see this as a **technical rather than conceptual limitation**. We expect the gap to narrow with longer token sequences, higher image resolutions, and improved vector quantization methods.\n\n**Q2: Motivation to use discrete tokens?**\n\nAns: The community recognizes the value of developing a unified model for both visual understanding (VLM) and generation. Among potential architectures, **LLMs provide the most natural and scalable foundation** for unifying these tasks-offering a flexible framework centered on **next-token prediction**.\n\nThe central challenge is how to **represent visual information within the LLM paradigm**. While approaches like Transfusion adopt separate diffusion objectives for generation, they break the unified modeling pipeline for visual understanding and generation. In contrast, **discretizing images into tokens enables the use of a single autoregressive modeling strategy, next-token prediction, for both text and image modalities.**\n\nHowever, existing discrete visual tokenizers (e.g., VQVAE) tend to operate at the pixel level, which **limits their semantic alignment with LLMs**. Unified tokenizers like VILA-U and UniTok attempt to achieve two **fundamentally conflicting objectives simultaneously**: pixel-level reconstruction and semantic-level alignment, making it difficult to strike a good balance between the two. To address this, we propose TA-Tok, a tokenizer designed to produce **fully semantic, discrete** visual tokens, initialized from LLM embedding. This alignment allows visual tokens to **seamlessly interact** with language tokens during autoregressive modeling.\n\nWhile TA-Tok does not directly decode images, it preserves **rich 2D spatial and structural information**. This makes the output tokens **more than just abstract conditions**: they encode image layout and visual detail in a structured form. Then a lightweight de-tokenizer suffices to reconstruct the image, unlike previous works that require powerful diffusion decoders to synthesize images from scratch.\n\nIn summary, **discrete tokens are essential** for unifying visual understanding and generation within an LLM framework. Our design of TA-Tok bridges the semantic gap between vision and language, enabling a more integrated and efficient modeling pipeline.\n\n**Q3: Whether both modalities are learned within a single latent space?**\n\nAns: In Line 32, we emphasize the advantages of a unified visual representation for both image understanding and generation, when compared to approaches that rely on **two types of visual tokens** (e.g., Janus). Our approach aims to unify modalities at the **token level**, where both vision and language operate in a compatible discrete semantic space.\n\nAs clarified in our response to Q1 and Q2, the de-tokenizer in our pipeline **does not serve as a standard generative model**, but rather transforms the semantic tokens from TA-Tok into VQVAE/VAE latent space for image reconstruction. The **planning of image structure and visual details**-traditionally done by the image generator-is instead **handled by the LLM** through discrete token generation.\n\nMethods like VILA-U and UniTok also claim to unify modalities with simple CNN decoders. In contrast, we introduce autoregressive and diffusion-based de-tokenizers, which are more expressive and better suited to decoding semantic visual token from TA-Tok. Thus, despite architectural differences, our method **belongs to the same line** of unified modeling works like VILA-U and UniTok, but pushes further in terms of semantic alignment and generation fidelity.\n\n**Q4: The Understanding Only Models in Tab.1 are outdated.**\n\nAns: We followed prior works [6, 55] in selecting these baselines and will include more recent models in the revision. However, Table 1 mainly aims to **compare unified models**, with understanding-only models serving as **reference points**. Direct comparison is also challenging, as recent understanding models often use more data and complex training pipelines. Despite this, our Tar model shows **state-of-the-art performance among unified approaches**.\n\n**Q5: Empirical results of training and inference latency of two De-Tokenizers.**\n\nAns: Thank you for pointing this out. We have conducted a comparison between AR-DTok and Dif-DTok in both training and inference phases, as summarized in the table below. While the two models differ in architecture, parameters, and codebases, we carefully controlled for factors such as batch size, token count, and hardware setup to ensure a fair comparison. Some key observations:\n- Training speed is comparable across both models.\n- Inference: AR-DTok uses less GPU memory and runs faster at 256 tokens, but becomes slower at 1024 tokens due to its autoregressive nature.\n- Dif-DTok scales better at high token counts but consumes more memory.\n\nWe will include this analysis and clarify the differences in the revision.\n\n| Method | Param | #Token | Train Speed (b/s) | Inference Speed (s/b) | Inference Memory |\n|---|---|---|---|---|---|\n| AR-DTok | 775M | 256 | 2.85 | 3.45 | 26.1GB |\n| Dif-DTok | 600M | 256 | 2.92| 3.63 | 34.6GB |\n| AR-DTok | 775M | 1024 | 1.84 | 14.14 | 31.2GB |\n| Dif-DTok | 600M | 1024 | 1.90 | 9.77 | 44.9GB |\n\nb/s: batch per second. s/b: second per batch. The batch size is 6 for training and 4 for inference."}, {"review_id": "LrU1CgwfZm", "reviewer": "Reviewer_xYB2", "summary": "This paper introduces a novel multimodal framework designed to unify visual understanding and generation through discrete representations. Specifically, the authors leverage a pre-trained LLM to generate an initial codebook, selecting representative codes based on their average distance from others. Subsequently, the framework aligns visual and textual feature spaces by training a SigLIP2 model alongside a projector. To capture multi-scale visual features, a scaling-adaptive strategy is adopted, and the frozen SigLIP2 model serves as a teacher to provide reconstruction targets, jointly guiding the model’s optimization. For generation tasks, the framework integrates two mainstream generative models—autoregressive and diffusion models—externally to enable image generation.", "questions": "1. Figure 5 shows that Janus Pro performs considerably worse in generation quality under the same data scale. What factors contribute to this discrepancy?\n\n2. Given that visual information is generally considered more complex than textual information, why is the visual codebook smaller than its textual counterpart? Furthermore, considering the semantic granularity differences between modalities, is it appropriate to assume a shared visual-textual feature space?\n\n3. Based on the generation results, Tar seems to exhibit weak identity consistency. What might be causing this issue?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strong:\n\n1. The paper proposes an innovative multimodal framework that unifies visual understanding and generation based on discrete tokens. Its approach of constructing a visual codebook from the LLM's feature space may provide meaningful insights for future work in this domain.\n\n2. The proposed method, Tar, demonstrates performance close to SOTA levels across several benchmarks and significantly outperforms Janus Pro on generative metrics.\n\nWeakness:\n\n1. Lines 120–121 mention that representative embeddings are selected based on their average distance to others. Is this achieved through clustering?\n\n2. The paper states that 150k vocabularies are excessively large, yet later in the appendix, it is mentioned that the model uses 65k vocabularies, an amount not substantially smaller. This appears contradictory.\n\n3. Additionally, the ablation study does not report results for the configuration in which random initialization is expanded to 65,536 tokens. Why is this omitted?\n\n4. A further concern arises from the comparison between the first and third configurations: performance from random initialization and LLM-based initialization appears similar. How should this be interpreted?", "was_revised": true, "final_justification": "After carefully reviewing the comments from fellow reviewers and the authors' response, I believe this paper is above the acceptance threshold for NeurIPS. Therefore, I will maintain my initial rating.", "rebuttal": "Dear Reviewer xYB2,\n\nThank you very much for your helpful comment!\n\n**Q1: Are representative LLM embeddings selected through clustering?**\n\nAns: We do not use clustering algorithms (e.g., K-Means) for selection. Instead, we adopt a simple and efficient **distance-based ranking** strategy. Specifically, we compute the average cosine similarity between each token embedding and all others, and select the tokens with the lowest average similarity (i.e., most representative and diverse). The selection process is shown below and will be included in the revision:\n\n```\n# embeddings.shape=[150K, D]\nnorms = torch.norm(embeddings, dim=1, keepdim=True)\ncosine_sim = torch.matmul(embeddings, embeddings.T) / (norms * norms.T)\naverage_similarity = torch.mean(cosine_sim, dim=1)\ntopk=65536\nrepresentative_tokens = torch.argsort(average_similarity)[:topk]\n```\n\nThis method allows us to efficiently obtain a **diverse and informative subset** of the LLM's embedding space, forming the initial visual vocabulary.\n\n**Q2: The 65K vocabulary is still large.**\n\nAns: We agree that 65K may still seem large. However, in vector quantization, **a larger vocabulary is generally beneficial**, which helps **preserve more visual detail** and **reduce quantization error**. Ideally, we would use the full 150K vocabulary from the LLM.\n\nIn practice, though, using all 150K high-dimensional embeddings (≥1536) results in **excessive GPU memory usage** during TA-Tok training. To address this, we reduce the vocabulary size to 65K, so that a 40GB A100 GPU can hold it and keep training stable.\n\n**Q3: Where is the result of randomly initialized codebook expanded to 65536 tokens?**\n\nAns: Traditional VQ codebooks typically use smaller vocabularies (8K–64K) with low-dimensional embeddings (e.g., 4–32). In contrast, to preserve the semantic richness of SigLIP features (1152-D), we increase the codebook dimension to 1536.\nHowever, this combination of **large vocabulary size and high dimension** is **extremely difficult to train** when initialized randomly. As shown in Appendix Table 3, even with 32,768 tokens and 1536-D (setting (b)), the model **struggles to converge** due to codebook collapse. Expanding the random-initialized codebook to 65,536 tokens only **worsens this issue**, making training **infeasible** in our experiments.\nThis challenge further motivates our use of LLM-initialized embeddings, which offer much more stable and meaningful training dynamics.\n\n\n**Q4: Performance from random initialization and LLM-based initialization appears similar.**\n\nAns: First, the improvement of LLM-based initialization compared to random initialization is still significant after sufficient training, about 2.7%, 13% improvement on the challenging GQA and MME respectively, which are **non-trivial** for these tasks.\n\nMore importantly, under **data-limited settings**, the benefit becomes more significant. As shown in the table below, when we reduce the training data from 25M to 12.5M and 6.25M samples, **LLM-initialized codebook consistently outperforms random initialization**, especially on tasks that rely more heavily on semantic alignment, such as MME and GQA. This highlights that LLM-based initialization helps **align TA-Tok with the LLM’s embedding space more efficiently**, particularly in low-resources scenarios.\n\n| Data | setting | GQA | MME | POPE | DPG |\n|---|---|---|---|---|---|\n| 12.5M | random | 53.5 | 965 | 85.8 | 66.0 |\n|  | LLM | 57.2(+6.9%) | 1258(+30%) | 86.8(+1.2%) | 67.7(+2.6%) |\n| 6.25M | random | 49.1 | 868 | 84.5 | 60.6 |\n|  | LLM | 55.5(+13%) | 1235(+42%) | 86.3(+2.1%) | 62.0(+2.3%) |\n\n**Q5: Why Janus performs worse in generation quality in Figure 5?**\n\nAns: As noted in Line 246-247, the ‘Janus’ baseline used here is **different from the original Janus Pro** [6], it denotes a baseline method which adopts SigLIP for image understanding and VQVAE for image generation. \n\nAs illustrated in Line 256-257, we believe the poor generation performance stems from a **conflict** between the two visual representations: SigLIP (semantic) and VQVAE (pixel-level). Since these representations are **not aligned**, the training signals from image understanding and generation tasks **do not reinforce each other**. \n\nThis hypothesis is further supported in **Table 6**, where **joint training fails to improve** Janus on either task, while it brings **significant gains** for both VQVAE baseline and our proposed method. This highlights the benefit of using a unified visual representation, as enabled by our TA-Tok design.\n\n**Q6: Why is the visual codebook smaller than the text codebook?**\n\nAns: As mentioned in Q2, the visual codebook size reflects a **trade-off between encoding quality and computational cost**. While a large codebook can better capture visual details, it significantly increases the **computational burden** during both tokenizer and MLLM training.\nMoreover, increasing the codebook size will also encounter the low codebook usage issue.\n\n**Q7: Is it appropriate to assume a shared visual-textual feature space?**\n\nAns: Thanks for your thoughtful question! We agree that visual and textual modalities differ in semantic granularity. Ideally, a unified model should capture both pixel-level detail and semantic-level features.\n\nHowever, in practice, integrating pixel-level information directly into fully semantic LLMs is **extremely resource-intensive and technically challenging**. As shown in Figure 5, methods like VQVAE, which focus on pixel-level reconstruction, fall behind Hybrid approaches and our method in both understanding and generation tasks.\n\nTo **achieve better cross-modal alignment**, we use a **shared semantic feature space**, where visual and textual tokens are **compatible**. We then use a lightweight de-tokenizer to project these semantic visual tokens back to the pixel space for image generation. This setup allows us to **maintain semantic consistency** while still supporting high-quality image synthesis.\n\n**Q8: Tar’s generation results are weak in identity consistency?**\n\nAns: Thank you for the observation. As discussed in our Limitations (Appendix Sec. I), autoregressive and diffusion models are **not original designed for image reconstruction**, but in our framework, we repurpose them as de-tokenizers for this task. Since they are not explicitly optimized for preserving fine-grained identity features, some loss in identity consistency may occur.\n\nTo address this, future work can explore **adding local consistency constraints**, such as token-to-token or patch-to-path supervision,which may better preserve identity than the current image-level objectives."}, {"review_id": "bWaNSZQPF0", "reviewer": "Reviewer_GJa8", "summary": "This work introduces a unified multimodal framework that bridges visual understanding and generation through a shared discrete semantic representation. The core innovation is from TA-ToK, which converts images into discrete tokens using an aligned codebook projected from an LLM's vocabulary embedding space. The system employs scale-adaptive encoding/decoding for flexible visual detail control and complementary generative AR-DTok/Diff-DTok for high-quality image synthesis. Experimental results demonstrate that Tar achieves competitive performance on both visual understanding benchmarks and generation tasks, matching or surpassing specialized models while maintaining a unified architecture.", "questions": "1. In Figure 6 (right), AR-DeTok shows better capability in generating complete and correct text within images. The authors are encouraged to provide insights into this phenomenon.\n2. Line 282: DPG is designed for evaluating T2I performance. It would be beneficial to demonstrate the impact of SAP design for I→I and TI→I scenarios.\n\nPlease also check the \"weakness\" section. \nI am willing to change my score if my concerns are addressed.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The model has an elegant unified architecture through text-aligned discrete representations.\n2. The comprehensive experimental design demonstrates strong empirical performance across both understanding and generation tasks.\n\n\nWeaknesses:\n1. In Table 6, joint training provides much higher gains on generation tasks than understanding tasks. The authors are encouraged to provide intuition for this imbalance. This observation suggests that image generation contributes little to visual understanding.\n2. Text + multiple images → image is a practical scenario (e.g., generating the next story frame based on description and prior images). While the authors mention related tasks in Appendix lines 102-109, the boundary between such text+images-to-image and the proposed TI→I remains vague. It would be beneficial if the authors could provide guidance for this application, such as prompt format design.\n3. Several images in Figure 7 in the appendix do not align well with the prompts. For example, row 3-4, 3-6, 5-4.", "was_revised": true, "final_justification": "I thank the authors for their response to the review comments and for conducting complementary experiments on the SAP design for TI2I tasks. I have also read the authors' responses to feedback from other reviewers. Most of my concerns have been adequately addressed.\n\nRegarding the TI2I task, while the authors prioritize semantic alignment over finer-grained consistency, this focus could potentially limit the model's applicability in light of recent advancements in unified MLLMs. Nevertheless, the concept of an aligned codebook across modalities remains compelling and shows some promise.\n\nTaking these factors into account, my final assessment falls between 'borderline accept' and 'accept'. I raise my score to 'accept' on the condition that the authors adequately incorporate the discussed points and complementary experiments into the final version of the manuscript.", "rebuttal": "Dear Reviewer Gja8,\n\nThank you very much for your insightful comment!\n\n\n**Q1: Add intuition for the imbalance gain of understanding and generation tasks in Tab.6.**\n\nAns: Thanks for your valuable suggestion. We will clarify this point in the revision. We believe the observed imbalance arises from two main factors:\n\n**(a) Task and supervision mismatch.** Generation tasks directly benefit from joint training because they require the model to **understand the concept (prompt)**-whether text or image-before generating coherent images. In this sense, improvements in visual understanding directly enhance generation quality.\n\nIn contrast, understanding tasks are often **discriminative** in nature (e.g., classification or multiple choice), which **do not directly benefit** from generation supervision. For example, predicting visual token sequences may not significantly help with answering a Yes/No question.\n\n\n**(b) Asymmetry in task difficulty and data requirements**: The model’s **understanding ability is already relatively strong**, often requiring less data to achieve good performance. For example, models like LLaVA reach strong image understanding with only 0.6M training samples. In contrast, image generation is a **more complex task** that requires substantially more data (often 10M+ samples) and benefits more from improved multimodal representations.\nAs a result, joint training helps generation more noticeably—since it benefits from enhanced visual understanding and alignment. Meanwhile, understanding tasks gain less from generation supervision, as their performance tends to **saturate earlier**, and the benefits from joint training **diminish** once the understanding capability reaches a certain level.\n\n\n**Q2: The relation between the proposed TI2I task to traditional TI2I task.**\n\n\nAns: Traditional TI2I tasks, such as image editing, subject-driven generation, or next-frame prediction, primarily emphasize **pixel-level consistency**. For example, image editing requires unedited regions unchanged, while subject-driven generation ensures the identity or appearance of a subject is preserved between input and output images.\n\nIn contrast, our proposed TI2I task aims to **bridge the gap between visual understanding and generation**, rather than enforce pixel-level similarity. Given a sample triplet (T, $I_1$, $I_2$), the model is trained to **first understand both the text T and the input image $I_1$**, and then generate a new image $I_2$ conditioned on both modalities. Pixel-level consistency is **not required**-only **semantic alignment**.\n\nFor example, if $I_1$ is “a picture of grass,” $I_2$ is “a dog running on grass,” and the text is “A dog running on \\<image\\>,” the model is expected to understand that “\\<image\\>” refers to “grass” and generate an appropriate scene. The grass in $I_1$ and $I_2$ may differ visually, as long as they share the same **semantic concept**. This formulation allows the model to **learn multimodal alignment and reasoning** within a single training sample, and gradually unifies image understanding and generation in a **semantically coherent** way.\n\nWe also give some **prompt format design and application in Appendix Figure 1**, where the model trained on the TI2I task can be adapted to subject-driven generation and zero-shot style transfer.\n\n**Q3: Several images in Figure 7 are not well aligned with the prompt.**\n\nAns: Thanks for pointing this out. In Figure 7, we used a set of prompts collected from the web and prior works, and generated the images without cherry-picking. While many results are reasonable, some generations are not well aligned with the prompts. This is primarily due to limited training data and compute budget. Our current model is not strong enough to handle some unseen prompts, and may produce incorrect results. We believe it can be mitigated by scaling up training data and incorporating more diverse scenarios in the future work.\n\n**Q4: Why AR-DTok shows better capability in generating complete and correct text within images in Figure 6?**\n\nAns: We believe this phenomenon is due to two key factors:\n**(a) Training data.** As shown in Appendix Table 1, AR-DTok is trained with more data than Dif-DTok (50+23M vs 23M). This provides AR-DTok with **better alignment** to TA-Tok tokens, particularly for fine-grained content such as text within images.\n\n**(b) Architecture difference.** As shown in Figure 4, the two de-tokenizers differ in how they integrate TA-Tok tokens:\n- In AR-DTok, TA-Tok tokens z and VQVAE tokens y are concatenated as [y, z] and processed jointly via **self-attention** in an autoregressive manner.\n- In Dif-DTok, TA-Tok tokens are injected  through **cross-attention** with the noise input during denoising.\n\nBecause TA-Tok tokens carry **explicit** 2D structure and visual details, the **self-attention fusion** in AR-DTok is more effective in learning fine-grained **token correspondences** between TA-Tok and VQVAE. This contributes to AR-DTok’s superior ability in generating coherent and correct text within images.\n\n**Q5: Evaluation of SAP design for I2I and TI2I tasks.**\n\nAns: As discussed in Q2, the proposed I2I and TI2I tasks are designed as **auxiliary tasks** to mitigate the gap between visual understanding and generation, and are different from traditional TI2I tasks. Therefore, we evaluate the design of SAP on image editing, a traditional TI2I task. Specifically, we simply finetune Tar 1.5B on OmniEdit [1] and evaluate it on Emu-Edit [2]. The results are shown below. \nWe find that different SAP settings perform **similarly on CLIP-I and CLIP-T**, which measures the semantic similarity between the input image, prompt and edited image. However, for the L1 metric, which reflects pixel-level consistency, using **more tokens** (e.g., 729) provides better reconstructions. We will include the evaluation of TI2I tasks in the revision.\n\n| #Token | CLIP-I↑ | CLIP-T↑ | L1↓ |\n|---|---|---|---|\n| 729 | 0.839 | 0.279 | 0.131 |\n| 169 | 0.841 | 0.279 | 0.160 |\n| 81 | 0.845 | 0.278 | 0.224 |\n\n\n[1] Wei, Cong, et al. Omniedit: Building image editing generalist models through specialist supervision. ICLR.\n\n[2] Sheynin, Shelly, et al. Emu edit: Precise image editing via recognition and generation tasks. CVPR."}, {"review_id": "VPXVhutZPp", "reviewer": "Reviewer_7n9v", "summary": "The paper introduces Tar, a framework to integrate visual understanding and generation within a unified architecture, operating on discrete tokens. The main ingredient is Text-Aligned Tokenizer (TA-Tok), which projects images into a discrete codebook of text-aligned tokens through a trainable SigLIP2 encoder + quantization. By pairing this tokenization strategy with either autoregressive or diffusion-based de-tokenizers, Tar demonstrates strong performance across a variety of benchmarks for both image understanding and generation tasks.", "questions": "- The paper claims that the codebook is semantically aligned. However, since the projection matrix W is trainable, is it possible that this alignment is compromised? How is semantic alignment preserved?\n- What is the criterion used to select the top-k most representative embeddings when constructing the visual codebook?\n- In Table 2, individual best-performing scores are not highlighted. Would the authors consider emphasizing these for easier comparison?", "limitations": "Limitations and societal impact are discussed in the supplementary material.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- TA-Tok provides a novel approach to image tokenization for multimodal LLMs. The idea of pairing the semantic richness of continuous tokens produced by a fine-tuned encoder (SigLIP2) and the advantages of discrete tokens for image generation is interesting and promising for future architectures. \n\n- Tar exhibits competitive or state-of-the-art performance across various benchmarks, both in the image generation and image understanding domains.\n\n**Weaknesses:**\n\nI believe this work is solid on the technical side, but it would greatly benefit from improvements in clarity and presentation. Several aspects are currently difficult to follow or underexplained:\n\n- **LLM training strategy.** It is unclear whether the backbone LLM is trained from scratch or fine-tuned. Section 3.4 suggests that it is pre-trained and then fine-tuned, while line 225 mentions initialization from Qwen2.5-Instruct. This is further confused by the subsequent statement that the LLM is 'fully finetuned at both the pretraining and finetuning stage'. Clarification on the training pipeline would be helpful.\n\n- **Scale Adaptive Pooling and Decoding.** These components are listed as key contributions of the paper, yet their mechanisms are not described in sufficient detail. A more thorough explanation of how these modules work and of their specific role within the overall architecture would strengthen the paper.\n\n- **Comparison with other visual representations (Section 4.3).** The setup of this experiment is somewhat opaque. It would be useful to clarify: i) whether all methods are trained on the same data ii) how the authors isolate the impact of visual tokenization from the influence of different decoding strategies, which also vary across configurations.\n\n- **Pre-training strategies.** The I->I and TI->I strategies introduced in Section 3.3 are rather unclear, and a more detailed explanation would be valuable. Specifically, for TI->I the example does not really seem to help: what is the role of the second chunk of sentence ('the grass')?\n\n- **Figure 4.** The figure illustrating the two decoding strategies is somewhat confusing. It is not immediately clear how the de-tokenizers relate to the autoregressive LLM at inference time. The direct connection from TA-Tok to the decoding modules, and the presence of an autoregressive model in only one of the two pathways, makes the architectures difficult to interpret.\n\n- **Self Reflect.**  The authors briefly discuss the introduction of a 'Self Reflect strategy', which improves the generation results. I believe some hints on its internals should be provided in the main text (or, at least, a pointer to the relevant appendix section).\n\n**Typos:**\n\nThe manuscript contains some minor typos, including:\n\n- line 144 and 188 ('to decoding' -> 'to decode')\n- line 157 ('vis' -> 'via')\n- line 168 ('particular' -> 'particularly')\n- lines 205-206 ('text-image to text' should probably be 'text-image to image')\n- line 226 ('random' -> 'randomly')\n- line 233 ('a fully discrete' -> 'fully discrete')\n- lines 274-275 (meaning is clear but sentence structure needs revision)\n- line 277 ('performs' -> 'perform')\n- lines 282-283 ('makes our module can produce' needs revision)", "was_revised": true, "final_justification": "I believe this paper is technically sound and presents a promising approach to unified text-image understanding and generation, demonstrating strong results across multiple benchmarks. My main concerns lie in the clarity of the manuscript; however, the authors have addressed most of these issues in their response and have committed to improving the presentation in the final version. For these reasons, my overall assessment is borderline, but I am leaning toward acceptance.", "rebuttal": "Dear Reviewer 7n9v,\n\nThank you for your detailed comments!\n\n**Q1: LLM training strategy. It is unclear whether the backbone LLM is trained from scratch or fine-tuned.**\n\nAns: In Sec.3.4, we follow previous works on multimodal LLMs [2,31,35] and Unified MLLMs [6,55], which typically divide the training pipeline into two stages: Pretraining and Supervised Finetuning.\n\nIn this context, Pretraining refers to training the multimodal LLM, initialized from a **pretrained LLM backbone**. Specifically, we initialize our model using Qwen2.5-Instruct, a pretrained LLM, and then conduct MLLM Pretraining followed by Supervised Finetuning. We will clarify the difference between LLM and MLLM pretraining in the revision.\n\n\n**Q2: More explanation of Scale Adaptive Pooling and Decoding**\n\nAns: **(a) Mechanism:** As described in **Figure 3 (caption), Lines 122–129, 130–140 (especially Lines 131–132), and Lines 226–227**, Scale-Adaptive Pooling (SAP) is implemented using 2D adaptive pooling with a predefined scale set {1, 2, 3}. During training, a scale is randomly sampled; at inference, a fixed scale is used. For a 384×384 input image, these scales produce 729 (27×27), 169 (13×13), and 81 (9×9) tokens, respectively.\nTo support variable token lengths, we introduce Scale-Adaptive Decoding (SAD): a lightweight ViT decoder (identical to SigLIP2’s ViT, see **Lines 127–129**) that can process variable-length sequences by dynamically resizing its 2D positional embeddings.\n\n**(b) Role:** SAP and SAD allow TA-Tok to extract multi-granularity features, as noted in Lines 122–125. This flexibility enables the MLLM to better adapt to different tasks:\n- Coarse-grained features (e.g., 81 tokens) are suitable for text-to-image generation, especially under limited training data.\n- Fine-grained features (e.g., 729 tokens) are crucial for detailed image understanding tasks.\nThis design is empirically validated in **Table 5**: visual understanding tasks consistently benefit from more tokens, while generation tasks prefer fewer tokens in low-data regimes, but improve with more tokens when training data is scaled up.\n\n**Q3: Comparisons to other visual representations.**\n\n**(1) Whether all methods are trained on the same data?**\n\nAns: Yes. Line 250-252: “For training MLLMs with these representations, we sample **a subset of our training data for controlled experiments**: 10M T2I data, 10M I2T data and 5M text-only data. All models are **trained with the same configuration** and tested on visual understanding [15, 20, 22, 26] and generation tasks [19].”\n\n**(2) How the authors isolate the impact of visual tokenization from the influence of different decoding strategies.**\n\nIn Section 4.3, we compare TA-Tok with other discrete visual tokenization methods, including VQ-VAE, Janus (which also uses VQ-VAE for image generation), and a hybrid tokenizer (i.e., UniTok [36]). All these methods adopt an **encoder–decoder architecture**, where the encoder (tokenizer) converts input images into discrete tokens, and the decoder (de-tokenizer) reconstructs the image from those tokens.\nWe understand that the reviewer may question whether our decoding setup is fair compared to the baselines. However, we would like to emphasize that the proposed **de-tokenizer (AR-DTok and Dif-DTok) is one of the key contributions** of our paper. They are specifically designed to decode **fully semantic discrete tokens** (produced by TA-Tok) back to pixel-level images, using autoregressive and diffusion-based models, respectively. Therefore, while decoding strategies do differ across methods, these differences are **an integral part of the overall design and contribution** of our approach, not confounding factors in the comparison.\n\n**Q4: Pre-training strategies**\n\n**(1) More detailed explanation of the proposed I2I and TI2I tasks.**\n\nAns: As described in **Sec. 3.4 and Appendix Sec. D.2**, the proposed I2I and TI2I tasks encourage the MLLM to first understand the input image and then generate another image, which means the model needs to improve its understanding and generation task in one training sample. We will add more details for the two tasks.\n\n**(2) \"A dog running on the grass\" becomes: \"A dog running on \\<image\\>\" and \"the grass\". Explain this example.**\n\nAns: As described in **Line 206-208 and Appendix Sec. D.2**, we use FLUX to generate **two** images based on the prompts “A dog running on the grass” and “the grass”. The image with prompt “the grass” is used as an input image. The task is to generate a final image based on a **multimodal prompt**: “A dog running on \\<image\\>”.\n\n**Q5: The two decoding strategies in Figure 4**\n\n**(1) How the two-de-tokenizers relate to the autoregressive LLM at inference time?**\n\nAns: As described in **Figure 2 and Sec. 3.3 Inference (Line 186-188)**, the LLM first generates a sequence of visual tokens during inference. These tokens are then passed to either AR-DTok or Dif-DTok to reconstruct the final image. In other words, the de-tokenizers operate **after the LLM** and are responsible for converting the predicted tokens back to the pixel space.\n\n**(2) Why does the autoregressive model only appear in one of the two decoding pathways?**\n\nAns: The “Autoregressive Model” shown in Figure 4(a) refers **specifically** to the **image generation model used within AR-DTok**, which is a separate autoregressive model trained to reconstruct images from visual tokens. It is not the same as the autoregressive LLM depicted in Figure 2. To clarify: (a) Both AR-Dtok and Dif-DTok receive the LLM-generated tokens at inference time. (b) During training, they both consume tokens from TA-Tok.\n\n**Q6: Self Reflect requires more details in the main text**\n\nAns: Due to the space limit, we mainly introduce the Self Reflect strategy in Appendix Sec. F and its ablation experiments. We will add more detail of Self Reflect in the main text and give a link to the appendix for more details.\n\n\n**Q7: Since the projection matrix W in Text-Aligned Codebook is trainable, is it possible that this alignment is compromised?**\n\nAns: As shown in Figure 3, we initialize the codebook using **frozen LLM embeddings** from a text-pretrained LLM. This provides a **strong semantic prior**: each token in the codebook corresponds to a position in the LLM’s text space. So even though the projection is learned, it must learn to **align with a fixed semantic space**. This constraint naturally preserves semantic consistency during training.\n\n\n**Q8: What is the criterion used to select the top-k most representative embeddings when constructing the visual codebook?**\n\nAns: As described in Line 120-121 and the following code, we select the top-k most representative embeddings based on their average distance to others.\n```\n# embeddings.shape=[150K, D]\nnorms = torch.norm(embeddings, dim=1, keepdim=True)\ncosine_sim = torch.matmul(embeddings, embeddings.T) / (norms * norms.T)\naverage_similarity = torch.mean(cosine_sim, dim=1)\ntopk=65536\nrepresentative_tokens = torch.argsort(average_similarity)[:topk]\n```\n\n**Q9: In Table 2, individual best-performing scores are not highlighted.**\n\nAns: We will highlight them in the revision."}]

Author rebuttal:

Meta-review: This work aims to unify visual understanding and generation via discrete vision tokens. To align with the vision and text space, vision tokens are initialized by projecting from the discrete text tokens. Meanwhile, the vision tokens are employed to generate images by AR-DTok/Dif-DTok. Although the initial review scores are mixed, the rebuttal addressed many concerns. All reviewers find that the exploration in this work is novel and the empirical results are solid. Therefore, all reviewers have positive scores after rebuttal. Please incorporate the comments and suggestions from reviewers into the final version and improve the presentation accordingly.
Decision: Accept (poster)