DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies
Authors: Wei Song
Yuran Wang
Zijia Song
Yadong Li
Zenan Zhou
Long Chen
Xu Jhua
Jiaqi Wang
Kaicheng Yu
Venue: ICLR 2026 Poster (2026)
Keywords: multimodal large language model
visual tokenizer
Primary area: generative models

Abstract: The differing representation spaces required for visual understanding and generation pose a challenge in unifying them within the autoregressive paradigm of large language models. A vision tokenizer trained for reconstruction excels at capturing low-level visual appearance, making it well-suited for visual generation but lacking high-level semantic representations for understanding tasks. Conversely, a vision encoder trained via contrastive learning aligns well with language but struggles to decode back into the pixel space for generation tasks. To bridge this gap, we propose DualToken, a method that unifies representations for both understanding and generation within a single tokenizer. However, directly integrating reconstruction and semantic objectives creates conflicts, leading to degraded performance in both reconstruction fidelity and semantic accuracy. Instead of forcing a single codebook to capture both visual appearance and semantics, DualToken disentangles them by introducing separate codebooks for high-level semantics and low-level visual details. As a result, DualToken achieves 0.25 rFID and 82.0% zero-shot accuracy on ImageNet, and demonstrates strong effectiveness in downstream MLLM tasks for both understanding and generation. Specifically, our method surpasses VILA-U by 5.8 points on average across ten visual understanding benchmarks and delivers a 13% improvement on GenAI-Bench. Notably, incorporating dual visual tokens outperforms using a single token type on both understanding and generation tasks. We hope our research offers a new perspective on leveraging dual visual vocabularies for building unified vision–language models. Project page is available [here](https://songweii.github.io/dualtoken-project-page/).

Published as a conference paper at ICLR 2026
DUALTOKEN: TOWARDS UNIFYING VISUAL UNDERSTANDING AND GENERATION WITH
DUAL VISUAL VOCABULARIES
Wei Song Yuran Wang Zijia Song Yadong Li* Zenan Zhou* Long Chen Jianhua Xu Jiaqi Wang"* Kaicheng Yu" 12.3+ 'Zhejiang University *Shanghai Innovation Institute "Westlake University *Baichuan Inc. 'Peking University "iiaomi EV thxu_orge1n com, wjgdevegmail cco [songwei, kyu) laestlake, edu.cn
ABSTRACT
The differing representation spaces required for visual understanding and genera- tion pose a challenge in unifying them within the autoregressive paradigm of large language models. A vision tokenizer trained for reconstruction excels at capturing low-level visual appearance, making well-suited for visual generation but lacking high-level semantic representations for understanding tasks. Conversely, vision encoder trained via contrastive learning aligns well with language but struggles to decode back into the pixel space for generation tasks. To bridge this gap, we pro- pose Dual'Token, a method that unifies representations for both understanding and generation within a single tokenizer. However, directly integrating reconstruction and semantic objectives creates conflicts, leading to degraded performance in both reconstruction fidelity and semantic accuracy. Instead of forcing a single codebook to capture both visual appearance and semantics, DualToken disentangles them by introducing separate codebooks for high-level semantics and low-level visual details. As result, DualToken achieves 0.25 rFID and 82.0% zero-shot accuracy on ImageNet, and demonstrates strong effectiveness in downstream MLLM tasks for both understanding and generation. Specifically, method surpasses VILA-U by 5.8 points on average across ten visual understanding benchmarks and delivers a 13% improvement GenAl-Bench Notably, incorporating dual visual tokens out- performs using a single token type on both understanding and generation tasks. We hope research offers new perspective leveraging dual visual vocabularies for building unified vaniun-e models. Project page is available here.
INTRODUCTION
Unifying visual understanding and generation within the pure autoregressive (AR) paradigm of Large Language Models (LLMs) offers simple, end-to-end alternative to the increasingly common yet structurally complex approach of coupling LLMs with external diffusion modules (Dong et al., 2024; Huang et al., 2025; Pan al., 2025; Chen et al., 2025b). To enable fully unified AR modeling of vision and language, a model requires a visual tokenizer to map images into discrete tokens and corresponding detokenizer that can faithfully reconstruct them back into pixel space.
Wang et al., 2024b) directly adopt the Early methods in this direction (Yu et al., 2023a; Team, 2024;
encoder and decoder of VQ-VAH, as the visual tokenizer and detokenizer. While these approaches demonstrated the feasibility of unifying visual understanding and generation within the AR paradigm, their understanding capabilities are typically lacking compared to multimodal large language models (MLLMs) specialized for understanding tasks (Liu et al., 2023; Yue et al., 2023; Fu et al., 2024; Song et al., 2024). We argue that this performance gap stems from inadequate visual representations: traditional VQ-VAEs optimized solely for reconstruction, producing image tokens that preserve
"Conresoondigg authors. I Projecc Leader.

---

Published as a conference paper at ICLR 2026
tokens
Semantic takens
Dufffoce (oang
VOVAE
CLIP-based
Poor understanding
Hard decode
Decopling duad chicatve and aensace dbseeve
performance
back to phee-space
Reconstruction- -haseed
X
CLIP-hraed emaoders
Semantic tokions
ViAU
Unitied takans
Recon, Loss Semi loss
DualToken
ACVAZ
Objecttve Confilet
CLIU
Semwrsconyy
X Direct combination
Unified architeccure with V
the ubjectivves
choil visural vocahullarics
Zaro-shat Accuscy (5)
Iepuc
ViLA-4
Dusifiben
Figure 1: (Left) Challenges faced by existing visual tokenizers, (Middle) We compare zero- shot classification accuracy and reconstruction FID on ImageNet-1K((al)) across baseline methods and Dual'oken. Dual'ooken achieves results comparable to or surpassing both semantic-only and reconstruction-on -oly methods in both tasks. (Right) Reconstruction results of VILA-U and Dual'Token our DualToken significantly outperforms VILA-U, which suffers from severe distortion and blurriness.
low-level visual details but fail capture high-level semantics aligned with language. By contrast, MLLMS designed for understanding tasks (Liu et al., 2024c: Chen et al., 2024b; Li et al., 2024c; 2025b; Bai et al., 2025) typically rely on CLIP--amill encoders (Radford et al., 2021; Zhai et al., 2023), which are pretrained with text alignment and thus inherently encode high-level semantics, making them more suitable for downstream visual understanding tasks in MLLMs.
To fully leverage the language-aligned se- mantic representations of CLIP, a natural approach is to quantize the features of a CLIP encoder and train a decoder for im- age reconstruction (Wu et al., 2025). This involves learning to reconstruct images for downstream generation tasks while preserv- ing its semantic capabilities as much as pos- sible (Wu et al., 2025). However, as shown in Fig.1 1 and 1a directly combining re- construction and semantic objectives often leads to severe distortions and blurriness in reconstruction tasks, along with a noticeable decline in semantic metrics such as zero-shot classification and image-text retricval, com- pared to its original pretrained model (Zhai et al., 2023). This degradation, as discussed in Wu et al. (2025), reflects the inherently conflict between the two training objectives, ultimately limiting both the quality of down- stream image generation tasks and the perfor- mance of multimodal understanding tasks.
Table 1:
Comparison to state-of-the-an visual to- kenizers. DualToken achieves the best performance among existing unified visual tokenizers semantic metrics. It also mitigates the distortion and blurriness faced by VILA-U during reconstruction, and surpasses dedicated models in reconstruction metrics.
METHODS
ScrcRIk
Zam-szal TTHROST
Recomtraction Only
MuNDDFAN (2hing in 1003)
1.33
37442
DATS
RQ-AAE flee stal, 2002)
2.55
SEEVOGAN
2111)
1.51
Upen MADIVEE thas 2004)
LID
Uidermedig daly
CLIP L114 (ualled d. 30213
76.6
21.2
21.5
X
ligh PPL 250 (2t0) et 2003)
3155
21.0
21.4
X
X
X
Sigt Sag-iP--aatt-354 (2ba) 2013)
35.2
21.7
21.5
X
Sil Sull (Tuchasmm 3005)
83.4
31.5
23.0
Vilanc--Lit (Cher 20240
81.1
20.5
Underttoniin
ULI usque GCen al. 31251
74.3
15.5
5,21
21.16
DA2S
ULI (oiqu)) (2an ct. Us Thk (M at 2015)
70,3
2044
21.0
1.46
23.36
(AMA
78.
0.38
15.5
lekeefoow 1250pe: (Qu etsl. 2004
1.33
21.41
Trkeatoow 108 (Qs 21.1. 7000
1161
0791
Miae-LL (200be))
30256
2.26
20.14
0.646
VILA-U Sil 1/to 250
73.3
100
VILA-l (seUu S0014-18 (-33)) atal. 20125)
7550
1.25
Duilikks (SggUP--11 2510
70
20.8
31.4
1.36
27.12
6/93
Doulloken (Rid Sort4 394;
X2.0
21.5
21.5
0823
25,00
0.44
Duliikka (Rigl Swrl 6 6.35)
80.3
21.8
0.57
74.00
0.73
To disentangle the two conflicting objectives, we propose interpreting visual appearance and visual cennantics-rquiiree for visual generation and understanding- as distinct visual vocabularies: a pixel codebook that captures low-level appearance features for generation, and a semantic codebook that encodes high-level semantic features essential for understanding. Specifically, inspired by the hierarchical structure of the human visual system (Groen et al., 2017), we partition the Vision Transformer (ViT) (Dosovitskiy al., 2020) into shallow, middle, and deep stages based on the cosine similarity (Chen et al., 2025a) across layers and observe that shallow layers of ViT predominantly capture low-level perceptual information-suu reconstruction tasks, whereas high-level semantic representations emerge in the deeper layers (Chen et al., 2023b; 2025a). To fully exploit this inherent property of ViT, we utilize shallow-layer features for reconstruction and deep-layer features for semantie learning, thereby enabling the simultaneous derivation of both a pixel codebook and a semantic codebook within a unified tokenizer.
as texture and color-making them suitable for
This hierarchical decoupling effectively resolves the conflict between the two objectives. As result, our DualToken achieves the best semantic performance among established unified tokenizers (Wu et al., 2025: Zhao et al., 2025; Qu et al., 2024; Ma et al., 2025) while also attaining state-of-the-
2

---

Published as a conference paper at ICLR 2026
art performance in reconstruction. Building upon this, we further demonstrate how MLLM can effectively utilize the dual visual vocabularies to achieve unified vision understanding and generation.
Our analysis reveals three key findings: i) Dual visual vocabularies resolve conflicts: Decoupling visual appearance and visual semantics with separate visual vocabularies mitigates the conflict between reconstruction and semantic objectives. Our tokenizer achieves state-of-the-arr performance in both sides, using only 10% of the pretraining data required by VILA-U; ii) DualToken is better than combining dual encoders: We observe that DualToken, as a unified architecture, outperforms the direct combination of two heterogeneous visual encoders, demonstrating both simplicity and effectiveness; iii) Dual-token promote each other: On one hand, visual appearance tokens (pixel tokens) are not only used for generation but also contribute fine- ane-grained low-level features that enhance visual understanding. On the other hand, visual semantic tokens tokens-beyond their role in understanding tasks-ack as positive supervision during autoregressive generation, leading to more semantically aligned image outputs compared generating pixel tokens alone.
2 RELATED WORKS
Unified Multimodal Models A classic strategy for integrating visual understanding and generation within single MLLM is to externally connect an LLM with a Diffusion Model (Sun et al., 2024; Dong et al., 2024; Pan et al., 2025; Chen et al., 2025b). However, pure AR architectures offer a more elegant, fully end-to- and-to-end solution by unifying both tasks within the same autoregressive framework. Representative works like Chameleon (Yu et al., 2023a; Team, 2024) and Emu3 (Wang et al., 2024b), have demonstrated the feasibility of jointly modeling vision and language through unified next-token prediction objective. Specifically, visual inputs are first tokenized into visual tokens. These visual tokens are then interleaved with text tokens to construct multimodal sequence. However, these pure AR architectures introduce generative capabilities at the cost of considerably weaker visual understanding. An empirical explanation for this (Wu al., 2025; Xie et al., 2025b) is that their vision tokenizers are trained solely for reconstruction and thus primarily captures low-level visual details for generation rather than the high-level semantics required for dision-languee
understanding.
A straightforward way to bypass such conflict is to employ two heterogeneous vision encoders (Wu et al., 2024a; Chen et al., 2025e; Deng et al., 2025b): semantic tokenizer (e.g. CLIP) for understand- ing and a reconstruction--aasee adds extra modules and structural complexity. making understanding and generation two loosely coupled systems with distinct pathways rather than truly unified model. In contrast, the text modality relies on a single tokenizer (e.g., BPE) (Sennrich et al., 2015) that discretizes text into a unified token space. This ensures consistent input-output space: the input tokens that provide signals for understanding and the output tokens produced during generation share the same vocabulary. This unified design allows L.MSs to seamlessly integrate text understanding and generation within the next-token prediction paradigm, thereby supporting broad generalization across diverse linguistic tasks. Therefore, the visual modality urgently requires a tokenizer that, like text tokenizers, can support both understanding and generation within a unified, coherent token space.
tokenizer (e.g. VQ-VAE) for generation. Yet this design inevitably
Unified Visual Tokenizers Recent research has actively explored solutions in this direction. VILA- U (Wu et al., 2025) and MUSE-VL (Xie et al., 2025b) strive to build unified tokenizer by jointly training on both reconstruction and semantic objectives. However, due to the inherent disparity between semantic and texture features. they struggle to strike an optimal balance between the two objectives, resulting in subpar performance both tasks. As discussed in FQGAN (Bai et al., 2024), decomposing the codebook in to this conflict. TokenFlow (Qu et al., 2024) employs separate codebooks with a shared-mapping mechanism. However, key differences set our approach apart: (i) TokenFlow relies on distinct vision towers to extract semantic and low-level features, rather than leveraging a unified architecture; (ii) the shared IDs obtained through the shared-mappinn mechanism may not be the optimal matches for either semantics or texture, potentially introducing additional losses in both domains.
manner may offer more fundamental solution
3 METHOD
This section introduces the design of our unified tokenizer and explains how its dual visual codebooks utilized within the next-token prediction paradigm for multimodal understanding and generation.
3

---

Published as a conference paper at ICLR 2026
(I) Naive approach
[M]] Decoupling reconstruction semantic obyectives (ours)
Grund
Initial State (pretrained)
Semuntic lop
our method
Inttaa State [pretrained]]
Semante lea
Sing Youal
Vision Encoder
Decader
Vision Encoder
Oecona Scor
Decoder
Figure 2: Comparing the design of a naive (Left) and our decoupled approach (Right), Naively combining reconstruction and semantic loss with single visual vocabulary leads to distorted reconstruction and degraded semantic performance. We decouple the two objectives through a hierarchy approach, where reconstruction loss is applied to supervise the shallow layers, while semantic supervision is applied to the deep layers. This enhances both reconstruction fidelity and semantic quality. Consequently, derive two complementary visual vocabularies: pixel codebook for low-level visual appearance, and semantic codebook for high-level visual semantics.
3.1 MOTIVATION AND VERIFICATION
As discussed in Qu et al. (2024), CLIP Table 2: Downstream visual understanding performance encoders cluster images by semantic with different vision encoders within the LLaVA-1.5 similarity, whereas VQVAE-based en- framework. The CLIP-based encoder corresponds to the coders group images by low-level at- siglip-so400m-14-333 model (Alabdulmohsin al., 2023), tributes such as color and texture. This whereas CLIP- -aseed (recon.) denotes an encoder with the same architecture but trained solely for reconstruction from scratch, controlling for factors like model size and archi- tecture. For the VQVAE-based encoder, we adopt SBER-
suggests that encoders trained for recon- struction primarily capture low-level vi- sual appearance, while those trained with text alignment excel at capturing high-level semantics. We argue that this difference in representation space is a key factor underlying downstream MLLM performance. Yet such claim has not been formally validated before.
MoVQGAN-270M, well-established reconstruction model.
Vision Encoder Type MMB"
MME*
SEED"
VQAv2"
Zero-Shot' rFID
CLIP-based
61.8
1492.9
58.4
78.5
83.2
CLIP based (recon.)
36.2
822.4
30.6
47.5
X
0.96
VQVAF-based
35.8
792.0
34.1
45.2
X
0.68
To validate this viewpoint, we started by a preliminary experiment following the LLaVA-1.5 pipeline (Liu et al., 2024b). Table.2, compared to the original SigLIP model, encoders trained with reconstruction objective exhibit significant drop in downstream MILM vision-language understand- ing performance, validating that high-level semantic features are more critical for visual reasoning in MLLMs than low-level perceptual features. However, to achieve both visual understanding and generation within a single MLLM, is essential to decode the visual tokens back into pixel space as accurately possible. However, since the SigLIP encoder focuses on high-level semantic information rather than texture details, simply discretizing its features and training a decoder without tuning the encoder results in poor image reconstruction quality. Therefore, proposing unified tokenizer is crucial to enable high-quality visual understanding and generation within a singe MI..M.
3.2 UNIFIED VISION TOKENIZER WITH DUAL CODEBOOKS
To build unified tokenizer, we started with the simplest approach, where we directly combine the reconstruction loss and semantic loss to optimize the entire vision tower and use a single visual vocabulary tokenize feature, similar to VILA-U (Wu al., 2025). Specifically, as illustrated in Fig.2 (left), we initialize the vision encoder with pretrained weights from SigLIP (Zhai et al., 2023) to ensure strong text-image alignment. Then the semantic loss is computed between the leeper-layer features of the model and its initial state to constrain the model from losing its semantic capability.
However, as shown in Table.3 (a), this straightforward approach leads to a clear conflict between the two objectives. On one hand, although the semantic loss is applied to preserve the model's original semantic representation capabilities, achieving this objective proves difficult, as semantic performance metrics show significant decline compared the original model, reflecting the disruption caused by
4

---

Published as a conference paper at ICLR 2026
Table 3: DualTooen resolves the conffict between reconstruction and semantic objectives. Directly combining the two objectives leads to a drastic decline in reconstruction performance (a vs. b), while incorporating reconstruction and semantic losses hierarchically results in better reconstruction performance (a vs. d). We highlight our method in the last row. We adopt the pretrained weights from the hglii--s4400m-patch 14-344 in this experiment.
Exp.
Learaing Objective (layer) Feature Type
Zero-Shot Ace.
Recunstructon
PSNR SSTM
Znitial State
Conimnous
832
X
Initil State (quonfized)
Discrete
82.4
x
X
X
Koron, (26) Sem. (26)
Discrote
72.3
3.86
12.44
0.574
(b)
Recon. (26)
0.722
(c)
Recon. (6)
Doscrtte
x
0.29
28.12
0.745
(4) Recon. (6) + Sem (26)
Dusrette
82.0
0.25
28.89
0.748
the reconstruction training objective on semantic capabilities. On the other hand, as shown in the cropped region of Fig.2, the model also struggles to achieve satisfactory reconstruction quality, often producing distorted and blurry images.
To resolve this conflict, we begin by ana- lyzing the intrinsic properties of the SigLIP encoder. Specifically, we divide the ViT into shallow, middle, and deep layers based on the cosine similarity of features across layers, as shown in Fig.3 (left). Guided by this partition, we extract features from the shallow and deep layer to perform clus- et al., 2023) based on the cosine similarity of features tering on the image representations. As across layers. Distinct bright square regions are observed shown in Fig.3 (right), we observe that fea- in the ranges of layers 1-7 and 8 8-17, indicating strong intra-group similarity within each interval; the remaining layers are treated as deep layers. (Right) Visualization of image clusters derived from features of (a) the 6th layer and (b) the 26th layer of SigLIP. Features from deep lay- ers cluster images based on semantic content, whereas
Cuacer
Caner
(0)
Cazeer
Claner 300
Qlater 423
Figure 3: (Left) Partitioning of the SigLIP encoder (Zhai
tures from the shallow layer tend cluster images based on low-level attributes such as color and texture, whereas features from the deep layer form clusters according to se- mantics. This suggests that shallow SigLIP features capture fine-grai tine-grained perceptual de- features from shallow layers form clusters based on low- tails, while deeper layers encode high-level level cues such as color and texture. For example, images semantics, aligning naturally with the in cluster exhibit similar grid-like textures (c.g., win- spective requirements of downstream vi- dow screens or monitor meshes). Implementation details sual generation and understanding tasks. of the clustering process are provided in Appendix.F F.
Motivated by this, we introduce a hierarchical approach to decouple the learning of the reconstruction and semantic objectives. Specifically, as shown in Fig.2 (right), reconstruction loss is applied to supervise the shallow layers (1-6) of the vision tower, while semantic loss is applied to the deep 26-th layer (Please refer to Appendix.B for the selection of the reconstruction layer). Features from the shallow and deep layers are discretized separately via residual vector quantization (Lee et al., 2022), resulting in low-level and high-level visual vocabularies, referred to as the pixel codebook and the semantic codebook, respectively. To ensure the encoder outputs align closely with the codebook entries, we utilize a Vector Quantization (VQ) commitment loss, which is defined as
L. = ||= - quantize(=)]1
Consequently, the total loss is formulated as weighted sum of reconstruction loss, semantic loss, and VQ commitment loss
Ciotal = d1 Crecon + x2 Leem + As (L.1 + La)
(2)
where the reconstruction loss is the combination of pixel-wise L2 loss (Dosovitskiy Brox, 2016), LPIPS loss (Zhang et al., 2018) and adversarial loss (Isola et al., 2017) for reconstructing an input image
Crecon || x + C (#. x)) La(tt
while the semantic loss is computed as the distance between the model's final-layer feature F and its initial value Fo
Leem -cos(FF F, Fo) + || Foll
5

---

Published as a conference paper at ICLR 2026
(i) Direct contiination discinct cokenteers
unified anchitecture with deal visual codabooiks
Visual Decoder
SigLiP
VOGAN
Text tokket
(1) Unified tokenizer with dual risund socebalorics (ours)
LLM
Faccan Briverte
Dua QualToken (ours)
Transformer
Fleel tokens
Semandic
Figure 4: (a) Direct combination of two heterogeneous tokenizer. Bascline method (Huang et al., 2025) that directly uses VQGAN and CLIP-based encoder to separately acquire high-level (semantic) and low-level (pixel) visual codebooks. (b) Our unified tokenizer with dual codebook. We decoupling high-level and low-level visual codebooks within a unified vision tokenizer. The image is converted into low-level visual appearance tokens (green) and text-aligned semantic tokens (red). (c) Architecture for unifying generation and understanding task. In image generation task, the generated low-level tokens decoded by the visual decoder reconstruct the visual content.
where cos(.) denotes cosine similarity. Interestingly, as shown in Table.3 (d), even without adding additional contrastive learning phase, but solely by applying simple constraint the semantic representation, incorporating reconstruction objective within our hierarchical learning strategy causes minimal damage to the model's semantic capability, Moreover, as shown in Table.3 (d), compared to training solely for reconstruction, incorporating semantic supervision in the deeper layers does not degrade reconstruction performance in the shallow layers, effectively resolving the conflict between semantic and reconstruction objectives.
3.3 UNIFYING UNDERSTANDING AND GENERATION
In this section, we demonstrate how to integrate the dual visual codebooks of DualToken within unified MILM. As illustrated in Fig.4 (c), model both textual and visual content within the autoregressive paradigm of LLMs, the pixel and semantic visual tokens are first passed through a 2-layer MLP projector align their dimensions with the LLM backbone. These tokens then concatenated along the embedding dimension (which does not increase the sequence length) to form unified visual tokens. Next, the unified visual tokens are concatenated with text tokens construct multimodal token sequence. The model is then trained in an autoregressive manner to predict the next token across both visual and textual content.
For simplicity, we define the language vocabulary of our MLLM as finite set X' = {II, Tni J. while the low-level and high-level visual vocabulary as y {y1, y2. Vea and Z {21,22, 22. Eng }. where n1, n2, and 73 represent the vocabulary sizes for language tokens, low- level visual tokens, and high-level visual tokens, respectively.
For visual tokens, since residual quantization introduces a depth-stacked structure of codes at each visual position p, we implement our visual heads based on the depth transformer from RQ- VAE (Lee et al., 2022). As shown in Fig.4, the semantic tokens and pixel tokens processed by independent visual heads-the pixel head and the semantic head. Both heads share the same structure, comprising three layers of depth transformers and corresponding classification head for each depth.
Given the L.M hidden state hp for visual tokens at position p. our depth transformer autoregressively predicts D residual tokens (rp1. Ty2- ... TpD). For d > 1, the input the depth transformer at depth d, denoted as pds is defined as the sum of the token embeddings up to depth d
Ipd Σ e(ppdr)
(5)
where y for the pixel head and Z for the semantic head. The initial input depth 1 is given by In1 = hp. This formulation ensures that the depth transformer incrementally refines the predicted feature representation by leveraging previous estimations up to depth d 1. Consequently, the overall

---

Published as a conference paper at ICLR 2026
whit
cel Prodecclo
puipll
VILAU
Shiry
Burry
Bibrryy
Slaryy
Burr
Destr tan
Good
Good
Good
V Good
Good
Good
Good
gx onlyl
"Cruy"
Only
"resbuk"
poar ls leava"
Figure 5: Qualitative results visual generation.
negative log-likelihood loss for the entire multimodal sequence length N is defined, if text token appears at position i,
NTT
Σ
Pi., where P. log P (x:(
(6)
and if visual tokens appears at position i, as
P. = Σ
log P (VialVi,ca) + log P (sialzi,ca))
(7)
d=1
4 EXPERIMENTS
4.1 VISION TOKENIZER
Experimental Setup We trained two versions of our vision tokenizers at 256 x 256 and 384 x 384 resolutions. fair comparison with VILA-U, we adopted same quantization strategies and pretrained weights (SigLIP-L/16-255 and SigLIP- Lo..IP--o//--33- so/14-),4), yielding 256 / 7229 tokens with residual depths 4 D = (whereas VILA-U uses
4 = 16). To test stronger backbones, further trained on S with D = 8, and show that method generalizes to other backbones in Appendix. B. All models were trained on ImageNet- (Deng et al., 2009), CC12M (Changpinyo al., 2021), and 50M images from LAION-400M (Schuhmann al., 2021).
Reconstruction We measured reconstruction FID (rFID), PSNR, and SSIM on the ImageNet- K (val). As shown in 1able. Dual'Token achieves the highest structural similarity and the lowest rFID among various state-of-the-arr dedicated methods, including and SBER-Mo VQGAN (SberBank, 2023). This demonstrates that our method effectively mitigates the structural distortion and blurriness issues encountered by VILA-U during reconstruction.
(Luo et al., 2024)
Semantie Metries For semantic metries, report the Top-1 accuracy for zero-shot classification on ImageNet-IK (val), along with text-to-image and image-to-text retrieval performance (R @ 1) on Flickr8K. As shown 1able. our DualToken significantly outperforms VILA-U and the latest concurrent work, UniTok, while also surpassing dedicated models like CLIP-L-11-336 in zero-shot image classification and achieves performance on par with the state-of-the SigL.P models.
Downstream Performance within LL- Before formally introducing the performance of our unified model, we first conducted a controlled experiment to validate the effectiveness of our vision tokenizer in downstream MILM understanding tasks within the LLaVA-1.5 (Liu et al., 2024b) framework. Specifically, we replace the vision encoder of LLaVA-1. 1.5 with DualToken. while strictly adhering to its training data and using LLaMA-2-7B (Touvron al., 2023) the foundational LLM. As shown Table.4 (a), our DualToken, as a discrete unified vision tokenizer, outperforms VILA-U and surpasses the original continuous IPI model.

---

Published as a conference paper at ICLR 2026
Table 4: Controlled comparison across ten visual understanding benchmarks. We evaluate differ- ent vision encoders/tokenizers, including siglip-large-162266 16-2566, VILA-U, and DualToken within the LLaVA-1.5 framework. MMB refers to MMBench-dev (Liu et al., 2023), OCRB to OCRBench (Liu et al., 2024d), and TVQA to TextVQA (Singh et al., 2019). The MME (Fu et al., 2024) score is normalized based on its total score. Sem. +Pxx is the original setting DualTToken where semantic and pixel tokens are concated along embedding dimension to serve as visual input. Sem. only means only the semantic tokens are fed as visual input.
Vision Encedlrr
Ros
MMB
MME
SEED
VQA+2
MMVer
AE20
MMMU
POPE
OCRB
TVQA
AVG.
(8)
suglip-larg--10 16-25-
60.9
62.9
78.2
34.5
30.3
26.3
(b)
VILA-U
256
55.3:54)
53.8-4.
51.2-56
73.166.1)
24.9.99.)
49.4-4.0
28.403.4)
78.2
23.8
42.8:1.)
48. 18.16477)
DualToken only)
(9.8:3.)
46.3.33
77.6.4.6
63.7.033)
30.3.0.5
2..6-1.
43.2
52.2.46
(d)
DualToken (sem,+ pis.)
256
61.3
64.6
572 0.30
77.0..12)
34.6
55.9 2.4
30,2 0.6
83.0
(27)
29.
46.2
1.9
53.9 1.3)
Table 5: Quantitative results on visual understanding and generation benchmarks.
Type
Method
Params
MMBBnchh
SEED
MMMU
MM
MathVista
MME
IPstracrBIJPP (Dai et at., 2023)
7B
36.0
59.88
30..5
26.2
34.4
1137.1
PLi (Zan at. 2024)
2.78
85.0
59.8
25.9
1335.1
LLaVA-15 (Liu et 2024b)
7B
85.9
64.3
58.6
35.4
31.1
27.4
1510.7
Und.
LLaVA-NoNT (Liu 2024c)
70.2
43.9
1519.0
IIaVA-NeXT (I at. 2024c)
348
87.7
79.1
75.9
$1.1
57.4
46.5
1631.0
ShareGPT4Y (Chen al., 2024b)
7B
68.8
69.7
37.2
37.6
26.5
1567.4
VILA (Lin et al., 2024a)
7B
85.5
68.9
61.1
34,9
1533.0
BAGAL (Deng
20250)
148
85.0
55.3
07.2
1687.0
Chameloon (Team, 2024)
7B
31.1
22.4
8.3
Ema3 (Wang et 20244b
8B
85.2
58.5
68.2
31.6
Show-o al. 2024)
1.58
73.8
25.1
948
lanes (Wa et ol. 2024g)
1,58
87.0
69.4
63.7
30.5
34.3
1338.0
Liquid al., 2024b)
1448.0
MUSE-VL
SINEEVVL225ppp (256px) OKie et al., 2025b)
7B
72.1
69.1
39.7
51.3
1-180.9
TokenFlow (384px)
al., 2024)
13B
86.8
68.9
40.7
UnfTok (256p)) (Mo et al., 2025)
78
83.2
61.1
33.9
1448.0
UniToken (384p)) (Jiao et 2025)
7B
71.1
69.9
32.8
38.5
Show o2 (132ps) (Xie al., 2025a)
1.58
67.4
65.6
37.1
1450.9
Show-o2 (432px) (Xie al.. 20255a
7B
74.3
69.8
48.9
1620.5
VII.A-U (Wu 2025)
7B
85.8
59.0
33.5
1401.8
Doal Takena 3B (256px)
3B
86.0
70.9
70.2
38.6
32.5
46.5
1489.2
(38-px)
88.1
76.2
72.2
40.2
1588.4
DoalToken- 7B (256ps)
7B
88.6
74.9
71.8
45.8
40.5
55.8
1502.7
DiulToken-78 (384ps)
89.4
44.3
1625.0
(a) Evaluation on multimodal understanding benchmarks.
Logical
Type
Method
Architecture
Count
Differt
Comparet
Overall
Negatic
Universal
SD-XL (Podell et al., 2023)
Diffusion
0.71
0.73
0.50
0.63
Gen.
Midjourney vo (Midjoumey, 2024)
Diffusion
0.78
0.78
0.79
0.50
0.76
0.69
DALL-E (Betker et al., 2023)
Diffusion
0.82
0.78
0.82
0.448
0.80
0.70
Show (Xie al., 2024)
Discrete Diff.
0.70
0.62
0.71
0.51
0.65
0.60
I.LUME (Wang et al., 2024g)
AR+Difff
0.66
0.68
0.67
0..49
0.63
0.60
LWM (Lia 2024g)
Autonegressive
0.49
Liquid (Wu et al., 202
Autoregressive
0.76
0.73
0.74
0.46
0.74
0.65
UniTok al., 2025)
Autoregressive
VII.A-U (Wa et al., 2025)
Autoregressive
0.70
0.71
0.74
0.53
0.66
0.64
VILA-U (256)
Autonggressive
0.68
0.66
0.70
0.49
0.64
0.60
DuaTTkknn- 3B (256)
Autorsgressive
0.76
0,76
0.78
0.50
0.72
0.68
Duaffoken-BB (pix. only)
Autoregressive
0.59
0.59
0.59
0.47
0.59
0.55
(b) VQAScores on advanced prompts of GenAl- -ench (Lin et al., 2024b)
4.2 UNIFIED MODEL FOR GENERATION AND UNDERSTANDING
Building on the unified tokenizers, we further verified its potential within unified AR framework based on Qwen- 2.5--B 3B (Yang et al., 2024). Our training process consists of four stages: (1) Freeze the
LLM and pretrain on image-caption data, training only the visual projector for multimodal alignment. (2) Unfreeze the I.I. and fine-tune on visual understanding data to enhance comprehension. (3) Freeze the LLM and train only the visual heads text-to-image data. (4) Unfreeze all components and perform joint training on a mixture of understanding, generation, and interleaved datasets.
To ensure fair comparison with VILA-U (Wu et al., 2025), we additionally provide reproduced version of VILA-U using Qwen-2.5-3B as the language backbone, trained with the same dataset and procedure method. We evaluate our model against widely used vision-language understanding benchmarks, including VQAv2 (Goyal et al., 2017), POPE (Li et al., 2023b), MME (Fu et al., 2024), SEED-IMG (Li et al., 2023a), MMBench (Liu et al., 2023), and MM-Vet (Yu et al., 2023b).
8

---

Published as a conference paper at ICLR 2026
As shown in Table.5, our DualToken (3B) demonstrates strong understanding performance compared to other unified models and surpasses dedicated understanding models like LLaVA-NeXT and ShareGPT4V (Chen et al., 2024b). Mcanwhile, as illustrated in Fig. 5, thanks to the significantly improved reconstruction quality of Dual'Token, the generated images are rich in detail and structurally realistic, accurately capturing fine textures such animal fur and other intricate patterns sinterns-ffeectveeeee resolving the blurriness and distortions observed in VILA-U. What's more, the generated images exhibit remarkable alignment with the text, even for long and complex prompts. This especially evident when compared with the pix. only method (which only predicts pixel tokens during image generation), as often ignores important semantic content during generation highlighing the crucial role that semantic tokens play in helping the model grasp the semantic structure of images throughout the generation process. Results on more generation benchmarks are presented in Appendix.E.
Beyond its impressive performance, we observed two interesting findings:
Pixel tokens enhance understanding. As shown in Table.4 (d)( we compared using only the semantic tokens (sem.), and a combination of semantic and pixel tokens (sem.+pept), concatenated along the embedding dimension to serve as visual input. Surprisingly, compared to using semantic tokens alone, jointly leveraging both semantic and pixel tokens leads to consistent improvements across various aspects, including general VQA (Liu et al., 2023; Fu et al., 2024), hallucination detection (Li et al., 2023b), and OCR-reelated benchmarks (Singh et al., 2019; Liu et al., 2024d). Suggesting that the supplementation of high-frequency details by pixel tokens can compensate for the subtle semantic loss introduced by vector quantization.
Semantic tokens also helps to generate. As shown in Fig.5 and Table.5 (b), incorporating semantic tokens into the model's autoregressive generation process leads to more semantically aligned image generation compared to using visual appearance tokens alone. This indicates that visual semantic tokens- bokens-beyond their role in understanding tasks iasks-can can also assist the model in grasping the semantic composition of images, thereby producing outputs that better align with the intended semantics. This is also clearly reflected in the model's performance on GenAl-Bens Bench.
DualToken versus dual-encoder. Recently, some studies have adopted dual-encoder designs to obtain visual representations (Huang et al., 2025). Specifically, VQVAE-based pixel encoder and CLIP-based semantic encoder. To address a fundamental question- whet is it necessary to obtain dual visual vocabularies within unified tokenizer rather than simply combining existing specialized encoders? we conducted an experiment using the codebook from ABER-MoVQGAL as the lew-level vocabulary and VQ-processed SigLIP as the high-level vocabulary, as illustrated in Fig.4 (a).
As shown in Table.6, this straight- forward approach leads to signifi- cantly inferior image generation performance (See Appendix. F.4 for implementation details). To
Table 6: Results on the MJHQ-30K dataset (Li et al., 2024a).
Image
Mesheel
Roe F111
Maygnoond 3036)
explain this discrepancy, we vi-
Ligad (Wi 20240)
Astorgreaiie
sualize the feature spaces of Du-
LWM et 20146)
20344)
Astoegrewiie
Asterareanc
Dur Sarr--- Dean-124
alToken'ss 's 6th and 26th layers, as
Staneo Diee al, 2024)
Dascets ball.
12
VELA-1 (Wo al, 2019)
Assingmett
well as those of MoVQGAN and SigLIP with UMAP (Fig.6).
Astorgreaire
1515
Figure 6: Visualized feature spaces on Imagenet-1K 1 (val).
Duad Sts Daal Plerddr
Assonarrnn
256
Assingresibe
256
17:55
As shown, while DualToken's 6th and 26th layers yield features specialized for different purposes, they still share a largely overlapping representational space. In contrast, features from the two separate encoders (MoVQGAN and SigLIP) show significant divergence, forming clearly disjoint clusters. Therefore, we attribute the performance gap to the incompatibility of representational spaces between heterogeneous encoders. This mismatch imposes a burden on the downstream language model, which is forced to learn two entirely disjoint visual representation systems. This observation further highlights the simplicity and effectiveness of Dual'Token as unified architectural solution.
CONCLUSION
This paper presents DualToken, which, to the best of our knowledge, is the first to demonstrate that from shallow layers while learning semantics from deep
dual-codeboof
9

---

Published as a conference paper at ICLR 2026
layers- effectively resolve the long-standing conflict between reconstruction and semantic objec- tives within a single visual tokenizer. Building upon DualToken, we develop a pure autoregressive (AR) unified model that achieves state-of-the-at performance in both understanding and generation among all existing discrete AR approaches. Orthogonal to concurrent works that focus on improving the VQ mechanism itself (Ma et al., 2025), our method emphasizes hierarchical architectural design. Consequently, as more advanced VQ techniques emerge, our framework can naturally benefit from these improvements. We hope that DualToken offers a new perspective for designing unified visual tokenizers and sheds light on building truly unified architecture for vision-languae models.
ACKNOWLEDGMENTS
This work is partially supported by the National Natural Science Foundation of China under grant No. 62403389, the Provincial Natural Science Foundation of Zhejiang under grant No. QKWL25F0301, and the Zhejiang Key Laboratory of Low-Carbon Intelligent Synthetic Biology (2024ZY01025).
REFERENCES
Ibrahim M Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. Advances in Neural Information Processing Systems, 16416400-16425,, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923: 2025.
Zechen Bai, Jianxiong Gao, Ziteng Gao, Pichao Wang, Zheng Zhang, Tong He, and Mike Zheng Shou. Factorized visual tokenization and generation. arXiv preprint arXiv:2411.1668/ 2024.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall--3. pdf. 2(3):8, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800 2022.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Haoran Chen, Junyan Lin, Xinhao Chen, Yue Fan, Xin Jin, Hui Su, Jianfeng Dong, Jinlan Fu, and Xi- aoyu Shen. Rethinking visual layer selection in multimodal llms. arXiv preprint arXiv:2504.21447, 2025a.
Jieneng Chen, Qihang Yu, Xiaohui Shen, Alan Yuille, and Liang-Chieh Chen. Vitamin: Designing scalable vision models in the vision-language era. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024a.
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: family fully open unified multimodal moches-architecture,, training and dataset. arXiv preprint arXiv:2505.0955 09568, 2025b.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi- modti- models with better captions. In European Conference on Computer Vision, pp. 370-387. Springer, 2024b.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, arXiv preprint arXiv:2501. /78/ 2025c.
10

---

Published as a conference paper at ICLR 2026
Yongjie Chen, Hongmin Liu, Haoran Yin, and Bin Fan. Building vision transformers with hierarchy aware feature aggregation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp.
2023b.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, al. How far multimodal models with open-source suites. arXiv preprint arXiv:2404. 16821, 2024c.
gpt-4v? closing the gap to commercial
Wenliang Dai, Junnan Li. Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose purpose vision-language models with instruction tuning, 2023.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. / /4683, 2025a.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint 2rxi::2505.4466 2025b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. leee, 2009.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Yi. DreamLLM: Synergistic multimodal comprehension and creation. In The Twelfth International Conference on Learning Representations, 2024.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. Advances in neural information processing systems, 29, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 1929, 2020.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309. /7425, 2023.
Yuxin Fang. Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, pp. 105171, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 52132-52152, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Ster Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Patlern Recognition (CVPR), 2017.
Iris IA Groen, Edward H Silson, and Chris Baker. Contributions of low-and high-level properties to neural processing of visual scenes in the human brain. Philosophical Transactions of the Royal Society B: Biological Sciences, (77(111))220160100 2017.
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li. Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docow docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint XrXiv:2403. /2895, 2024.
11

---

Published as a conference paper at ICLR 2026
Runhui Huang, Chunwei Wang, Junwei Yang, Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Lanqing Hong, Hengshuang Zhao, et al. Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement. arXiv preprint arXiv:2504.01934 2025.
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. -o-iget translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1125-1134, 2017.
Yang Jiao, Haibo Qiu, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 3600-3610, 2025.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, 2024a.
Hugo Laurençon, Léo Tronchon, and Victor Sanh. Unlocking the conversion of web screenshots into html code with the websight dataset, 2024b.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523-11532, 2022.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench; Bench- marking multimodal Iims with generative comprehension. arXiv preprint 2xX1v:2307.16125. 2023a.
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi, Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint M.Nv:22002.12245 2024a.
Tianpeng Li, Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang. Zheng Liang, Zehuan Li, Mingan Lin, Guosheng Dong. et al. Baichuan-audio: A unified framework for end-to-end speech interaction. arXiv preprint arXiv:2502.17233 2025a.
Xiaotong Li, Fan Zhang, Haiwen Diao, Yueze Wang, Xinlong Wang, and Lingyu Duan. Densefusion- Im: Merging vision experts for comprehensive multimodal perception. Advances in Neural Information Processing Systems, 37:18535- 2024b.
Yadong Li, Haoze Sun, Mingan Lin, Tianpeng Li, Guosheng Dong, Tao Zhang, Bowen Ding, Wei Song. Zhenglin Cheng. Yuçi Huo, Song Chen, Xu Li, Da Pan. Shusen Zhang, Xin Wu, Zheng Liang Jun Liu, Tao Zhang, Keer Lu, Yaqi Zhao, Yanjun Shen, Fan Yang, Kaicheng Yu, Tao Lin, Jianhua Xu, Zenan Zhou, and Weipeng Chen. Baichuan-ommi technical report. arXiv preprint arXiv:2410.08565, 2024c.
Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Gu- osheng Dong, Da Pan, et al. Baichuan-omni-I technical report. arXiv preprint arXiv:2501. 15366. 2025b.
Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination large vision-language models. arXiv preprint arXiv:2305. 10355, 2023b.
Haokun Lin, Teng Wang, Yixiao Ge, Yuying Ge, Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun, and Ying Shan. Toklip: Marry visual tokens to clip for multimodal comprehension and generation. arXiv preprint arXiv:2505.0542 2025.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi. and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEÉ/CVF conference on computer vision and pattern recognition, pp. 26689-266999 2024a.
12

---

Published as a conference paper at ICLR 2026
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprins arXiv:2404.01291, 2024b.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint urXiv:2402.08228 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lec. Llava-next: Improved reasoning. ocr, and world knowledge, January 2024c. URL https: //llava-v// github..h..... htt.o//ll////2-0-3000000
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et Mmbench: your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv preprint Xxix:2305.0789) 2024d.
Zhuoyan Luo, Fengyuan Shi, Yixiao Ge. Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual generation. arXiv preprint arXiv:2409.044 10, 2024.
Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiao- juan Qi. Unitok: A unified tokenizer for visual generation and understanding. arXiv preprint arXiv:2502.20321, 2025.
Midjourney. Midjourney version 6.1, 2024. URL wwwph:tp/s/wwwt http:/:///wwmmmmuureee.... com.
Yuwei Niu, Munan Ning. Mengren Zheng, Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Chaoran Feng, Kunpeng Ning, Bin Zhu, et al. Wise: A world snowledge-ifformed semantic evaluation for text-to-image generation. arXiv preprint arXiv:2503.0722 2025.
Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Xu, et al. Transfer between modalities with metaqueries. arXiv preprint S24i::25044062566 2025.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprini arXiv:2307.00152 2023.
Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03066 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8748-8763, 2021.
Robin Rombach, Andreas Blattmann, Domínik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
SberBank. Sber-movggan, 2023. URL https: //habr....///. www.co////////.. http////cccmaaaeeeeeeaaaaaa
articles/74062/
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2 11.02/11110111 2021.
13

---

Published as a conference paper at ICLR 2026
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv: VrXiv:1508.07909, 2015.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Wei Song, Yadong Li, Jianhua Xu, Guowei Wu, Lingfeng Ming, Kexin Yi, Weihua Luo, Houyi Li, Yi Du, Fangda Guo, et al. M3gia: A cognition inspired multilingual and multimodal general intelligence ability benchmark. arXiv preprint arXiv:2406.0532 053443 2024.
Kegiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding. Advances in neural information processing systems, 36:49659 49678, 2023.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14398-14409, 2024.
Chameleon Team. Chameleon: Mixed-modas early-fusion foundation models. arXiv preprint arXiv:2405.098/8 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint 2rXiv:2502.147788 2025.
Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu. Illume: Illmiinaiin your Ilms to see, draw, and self-enhance. arXiv preprint arXiv:2412.0661 066773, 2024a.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang. Zhen Li. Qiying Yu, al. Emu3: Next-token prediction is all you need. arXiv preprint (0t1::2409.188666 2024b.
Cong Wei, Zheyang Xiong, Weiming Ren, Xinrun Du, Ge Zhang, and Wenhu Chen. Om- niedit: Building image editing generalist models through specialist supervision. arXiv preprint arXiv:241..07199, 2024.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint 24i::2410.13844 2024a.
Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable and unified multi-modi generators. arXiv preprint arXiv:24/2.043322 2024b.
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: unified foundation model integrating visual under- standing and generation. In The Thirteenth International Conference on Learning Representations, 2025.
Jinheng Xie, Weijia Mao, Zechen Bai. David Junhao Zhang. Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528, 2024.
Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2: Improved native unified multimodal models. arXiv preprint 2rXiv:2506./555 /5564, 2025a.
14

---

Published as a conference paper at ICLR 2026
Rongchang Xie, Chen Du, Ping Song, and Chang Liu. Muse-vl: Modeling unified vlm through se- mantic discrete encoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 24135-2411 24146, 2025b.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2024.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vogan. arXiv preprint 0rXiv2210000622. 2021.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, al. Scaling autoregressive mudti-modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.0233 02591, 2023a.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308 02490, 2023b.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-disscipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint 2xxiv:2311.166110 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986, 2023.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. NeurlPS, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.
Yue Zhao, Fuzhao Xue, Scott Reed, Linxi Fan, Yuke Zhu, Jan Kautz, Zhiding Yu, Philipp Krähenbüühl, and De-An Huang. Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation. arXiv preprint arXiv:2502.05178, 2025.
Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems, 35:23412-23425, 2022.
Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language-image pre-training with long captions. In ECCV, 2024.
Yichen Zhu, Minjie Zhu, Ning Liu, Zhiyuan Xu, and Yaxin Peng. Llava-phi: Efficient multi-modal assistant with small language model. In Proceedings of the Ist International Workshop Efficient Multimedia Computing under Limited, pp. 18 22, 2024.
15

---

Published as a conference paper at ICLR 2026
A LARGE LANGUAGE MODEL USAGE
In this paper, Large Language Models (LLMs) are used exclusively for grammatical error correction.
B LAYER SELECTION AND GENERALIZABILITY
As discussed in Sec. 3.2, we can partition the vision encoder into shallow and deep regions based on the cosine similarity between layers (see Fig. and Appendix F.1). Empirically, selecting the last layer within the shallow region, typically corresponding to the first quarter to third of layers, yields the best results. As validated in the main text, our method successfully generalizes across multiple backbones, including SigLIP-L/16-256, SigLIP-S0/14-384, and SigLIP2-S0/16-556
SQL-P-U16--56
SigLIP2-8/16-256
5igUF2-50015-256
30-
10
-88
-025
022
20
25
Lagers
Lage's
BRNZB-CUP-VIT-L
ViTamin-Xi-22
VITamin-Xi-30
20
020
20
Figure 7: Additional visualizations of inter-layer cosine similarity across different backbones, revealing consistent hierarchical pattern across architectures, model scales, and resolutions. A cohesive high-similarity region corresponding to the shallow layers can be clearly observed.
To further strengthen our claim, we include a comprehensive validation of the method's gener- alizability, covering six mainstream
CLIP (Radford al., 2021), Apple's DFN (Fang et al., 2023), BAAI's EVA (Fang et al., 2024), Google's ' SigLIP (Zhai et al., 2023), architecture ViTamin (Chen
SigLIP2 (Tschannen al., 2025), and the hybrid CNN-Transfon et al., 2024a) under consistent settings (RVQ = 8, codebook size = 16,384, 2M training samples). Our findings are summarized as follows.
(1) Ablations on model backbones and model size. As shown in Table. and Table.8, across all tested backbones with varying types and total layer counts, selecting the first quarter of layers for reconstruction consistently yields the best reconstruction quality and semantic performance.
Table 7: Model backbone.
Table 8: Model size and total layer.
Table Robustness on layers.
Baschin
Lagernonee.rrt Rotsl Zeno-shor e1113
Rackhone
Layer reone Rotal
Zaro-chot (FI
Backbone
Layer recon.tutal Zero sbox #FDD
DEN-UR4-036
NC1
79.2
0,54
DFN-R/16-234
3/12
71.4
74.1
0.98
1.58
DEN B/16 224
4/12
74.1
73.9
0.95
0.97
1524
11.2
03 224
1224
1424
75.2
77.8
0.50
1.36
9/2
66.1
3.21
DFN-1/14-224
6/24
79.2
0.84
1324
69.9
3.22
DEN-L/14-224
12/24
77.1
1.35
624
18/24
72.2
3.25
6/32
$1.1
0.75
70.5
DEN 1014 224
1974
1224
N74
96.1
05.
76.
6.7%
1.27
3.16
DFN-H/14-224
34/52
10/2
74.3
79.5
2.78
1.28
SigL:1234/16-266
10/32
5/24
6/24
30.4
$0.4
80.7
0.74
6.72
6.72
1924
72.9
$0.4
0.72
7/2
80.2
0.75
Sipl 91116-116--56
13/4
77.5
SigL11244/16-256
12/24
77.6
1.09
$1.00
1504
624
$8.0
72.9
6.39
2.55
7/27
81.2
0.699
SigLIP2
6/27
$1.2
0.66
ViTuni
12/44
SigLIP2-80016-256
14/27
78.4
1.08
8/27
$1.3
0.67
21/27
73.7
3.02
16

---

Published as a conference paper at ICLR 2026
(2) Robustness to specific layer choice. As shown in Table. 9, our method remains stable long as the reconstruction layers lie roughly within the first quarter the network. Minor shifts (#1-2 layers) cause negligible changes, confirming its robustness and broad applicabilityy
These findings indicate that for a new VIT architecture, one can confidently select the first quarter of layers for reconstruction achieve optimal results. Moreover, As shown in Fig. 7, we also provide more visualizations of inter-layer cosine similarity across backbones, revealing a consistent hierarchical pattern divisible into shallow and deep layers (also supported by prior study (Chen et al., 2025a)), further supporting the universality of our method.
C DISCUSSION AND COMPARISON WITH UNITOK
Since UniTok adopts a more advanced visual backbone, decoder, and discriminator architecture, we conduct a fair comparison by re-training our DualToken under the same encoder and decoder settings used in UniTok, i.e., choosing ViTamin-L/16. hybrid architecture of CNN and transformer, to instantiate DualToken. Under this setup, Dual'Token achieves stronger semantic performance and competitive reconstruction quality, as evidenced by the comparison between (a) and (b) in Table 10.
Table 10: Comparison with UniTok.
Tokenizer
rFID +
Zero-Shot Acc
(a) UniTok
0.38
78.6
(b) DualToken (RVQ)
0.39
80.3
(c) DualToken (MCQ)
0.25
82.2
Furthermore, DualToken and UniTok are complementary. Specifically, replacing original RVQ quantizer with UniTok's proposed MCQ, we observe consistent improvements in both reconstruction fidelity and zero-shot classification, as evidenced by the comparison between (b) and (c) in Table 10.
These results suggest that future work may benefit from integrating our dual visual vocabulary formulation with more advanced quantizers such as MCQ.
D COMPUTATIONAL ANALYSIS
Introducing two codebooks DOES NOT significantly increase the computational overhead, demon- strated by two aspects: parameter count and memory usage with inference latency.
D. PARAMETER COUNT
The ONLY additional parameters arise from 3 components:
The MLP projector's dimension changes from (1024~20048 -2048) to +248)) which adds 2.1M parameters.
An additional visual head: 258M parameters.
An additional VQEmbedding layer: 16M parameters.
Together, these account for only 8.93% % of the total parameters compared to the LLM backbone (3B). When scaling to larger backbones (e.g., 7B), the relative impact becomes even more negligible.
D.2 MEMORY USAGE AND INFERENCE LATENCY
Since dual tokens concatenated along feature dimension rather than sequence dimension. and the input dimension to the LLM remains unchanged, no new pathway is introduced to the LLM, and the computational cost of the LLM backbone remains strictly the same. The only increase stems from the components listed above.
Memory usage is measured under the same local batch size and device. FLOPs and inference time are averaged on T2I task (256px) over the MJHQ-30K dataset. Statistics for the VQA task have also been added to the paper.
17

---

Published as a conference paper at ICLR 2026
Table 11: Memory Usage and Inference Latency
Training Memory Usage Inference Time Cost
Single Forward GFLOPs
single token dual token
73.8G
11.42s
328.98
78.2G
12.97s
337.20
E RESULTS ON MORE GENERATION BENCHMARKS
Following VII.A-U, report results GenAl-Bench and MJHQ-306 in the main text. We now extend our evaluation to include GenEval (Ghosh et al., 2023) and WISE (Niu et al., 2025). The results demonstrate that DualToken achieves competitive performance across both benchmarks.
Table 12: Evaluation results on GenEval and WISE benchmarks.
Model
GenEval (Overall) WISE (Overall) 1
SDvI.5 (Rombach et ul., 2022)
0,43
0.32
SDXL (Podell et al., 2023)
0.55
0.43
Chamelcon-7B (Team, 2024)
0.39
EMU3-8B (Wang et al., 2024b)
0.66
0.39
Jamus (Wu ct al., 2024a)
0.61
0.23
Janus-Pro-7B (Chen al., 2025c)
0.80
0.35
ILL.MME-78 (Wang et al., 2024a)
0.61
TokenHlowXXI--14B (Qu et al., 2024)
0.63
Muse-V1.-77B (Xie al., 2025h)
0.57
UniToken-7B (Jiao et al., 2025)
0.63
Show-o2-1.5B (Xie et al., 2025a)
0.73
0.35
Show-o2-7B (Xie et al., 2025a)
0.76
0.39
VILA-U-7B (Wu al., 2025)
0.31
DualTokken3BB
0.72
0.35
DualToken-7B
0.75
0.39
F IMPLEMENTATION DETAILS
F.1 PARTITIONING OF THE SIGLIP ENCODER
We feed the ImageNet-1 K (Deng et al., 2009) validation set into SigLIP-S0400M-Pacchh445 et al., 2023). For each image, we extract the representations from all layers of the model, each with a shape of 729 x 1152. Then, we apply average pooling along the spatial dimension (the first axis) of each layer's representation, resulting in 1152-dimensional vector per layer.
(Zhai
Specifically, for each image, we obtain feature vectors from 26 layers, and compute the pairwise cosine similarity between these layer-wise representations to construct a 26 x 26 cosine similarity matrix. To capture the overall similarity structure across layers the model, we average the cosine similarity matrices across all images. The final similarity matrix S* is computed as:
1 n
S
(8)
1=1
where S; denotes the cosine similarity matrix for the th image, and is the total number of images. S* thus represents the average inter-layer similarity across the dataset.
F.2 IMAGE CLUSTERING
We extract intermediate representations from the 6th and 26th layers of 384 (Zhai et al., 2023) for each image in the ImageNet- IK validation set (Deng et al., 2009). The original representation shape is 729 x 1152, and we apply average pooling along the spatial dimension to obtain single 1152-dimensional feature vector per image. For both the 6th-layer and 26th-layer
14-
18

---

Published as a conference paper at ICLR 2026
features, perform k-means clustering with 1000 cluster centers (Cluster 0 to Cluster 999). The cluster analysis reveals that shallow- features (from the 6th layer) tend to capture low-level visual attributes such texture and color, while deep-layer features (from the 26th layer) predominantly encode high-level semantic content. The implementation code is provided in the supplementary material, and additional visualizations presented in Fig. 8.
Cluster 17
Cluster 60
Cluster 110
(b)
Cluster 70
Cluster 33
Cluster 41
Figure 8: More visualizations of image clusters derived from features of (a) the 6th layer and (b) the 26th layer of SigLIP. Features from deep layers primarily cluster images based high-level semantic content, whereas shallow-laye features tend to group images according to appearance-level cues such as color and texture. For instance, Cluster 17 contains images with similar scaly textures, while Clusters 60 and 110 predominantly group images by dominant colors (c.g., red or blue).
F.3 UMAP FEATURE SPACE VISUALIZATION
We perform dimensionality reduction using UMAP to visualize the feature spaces from DualToken's 6th and 26th layers, as well as those from MoVQGAN and SigLIP. Specifically, we sample 1,000 images from the ImageNet-1K 1K validation set and visualize the UMAP projections of their encoded features from each model. To ensure a fair comparison among the different visual models, all extracted features are first flattened and then uniformly processed via adaptive average pooling to maintain consistent dimensionality.
F.4 MODEL IMPLEMENTATION DETAILS
Our backbone model is built upon a decoder-only transformer architecture, and adopt Qwen2.5 (Yang et al., 2024) our initialization due its strong performance and public availability. The model uses R.M.NormZaang & Sennrich (2019) for normalization. For visual inputs to the LLM, we apply projector map the visual tokens into the same embedding space as the LLM. When predicting image tokens, the output hidden states of the LLM are passed through two separate projectors to align with the dimension of the semantic visual head and the pixel visual head. Each projector consists of two linear layers with GeLU activation in between. We use special tokens lokens-<image, Lei__geeettrr_eeee start> and <image_gees gee___en______ gen_endd to indicate the boundaries of the image to be generated.
For visual heads, since residual quantization introduces depth-stacked structure of codes at each visual position p. we implement our visual heads based on the depth transformer from RQ- VAE (Lee et al., 2022). Unlike the original depth transformer, which employs single head to predict logits across all depths, we introduce separate classification heads to compute the logits for residuals at each corresponding depth (Li et al., 2025a). As shown in Fig.4, the semantic tokens and pixel tokens are processed by independent visual heads the pixel head and the semantic head. Both heads share the same structure, comprising three layers of depth transformers and corresponding classification head for each depth. Detailed training hyper-parameters are provided in Table 13.
Implementation of the Dual Encoder Baseline As described in Sec. 4.2 of the main paper, some concurrent works adopt dual- -ecoddr designs to obtain visual representations (Huang et al., 2025), specifically combining VQVAB-baaeed pixel encoder with Cl.se-based semantic encoder.
19

---

Published as a conference paper at ICLR 2026
Table 13: Training hyper-parameteen
Visual Tokenizer
MLLM
Setlings
Stage
Stage 2
Stage
Stage 4-1
Stage 4-2
Projector 2e-5
Projector (Gen) le-4
All Projectors le-5
Learning Rate
7.2e-5
Projector Ie-3
LLM 2e-5
Visual Heads le-4
Visual Heads 1e-5; LLM le:5
Batch Size
64
64
256
128
512
256
Optimizer
A
AdamW
AdamW
AdamW
AdamW
AdamW
This raises natural question: Beyond architectural elegance and simplicity, does learning dual visual codebooks within a unifted visual tokenizer (ours) lead to better downstream performance in unified MLLMs compared to directly combining two heterogeneous encoders?
Since these concurrent works adopt different training datasets and downstream architectures (e.g.. external diffusion decoders (Rombach et al., 2022)), it is difficult to conduct fair comparison in the context of downstream unified models. To isolate the effectiveness of the tokenization strategy itself that is, dual tokens within single unified tokenizer vs. dual visual tokenizers from separate encoders-we implemented both designs under the same unified architecture proposed in our work.
Specifically,
SigL.P LP-L--aatt 6-256 (Zhai et al., 2023) and OBER-MOVQGAN (SherBank, 2023) to build the semantic tokenizer and pixel tokenizer, respectively:
The semantic tokenizer applies an RVQ quantizer (depth=4) to the penultimate layer of the frozen SgLLI---PPatch6--555 encoder. The encoder is fully frozen, and only the codebook is updated using commitment loss, aiming to reconstruct the input semantic features as faithfully as possible.
The pixel tokenizer is derived from a modified version of we added a downsampling and upsampling modules to its
To match the
token length of LegLiP---Patchh6 encoder and decoder, adjusting the downsampling and upsampling rate from 8 to 16. Additionally, we replaced the original quantizer with residual vector quantizer (RVQ) of depth 4 to ensure compatibility with our unified model architecture.
Apart from the different tokenizers used to provide pixel and semantic tokens, the rest of the architecture remains fully consistent with DualToken. Specifically, we concatenate pixel and semantic tokens along the embedding dimension form the visual input, map them into the LLM embedding space via projector, and use separate visual heads (pixel head & semantic head) for predictions. To ensure fairness, we standardized all other components except for the source of dual visual tokens:
All components kept identical, including image resolution, token length (16x16), RVQ depth
( = 4), embedding dimension, model architecture, and training data.
Both tokenizers are trained on the same datasets as DualTokenn as described in the main text.
G DATASETS
Our MLLM training process consists four stages: (1) Freeze the LLM and pretrain on image-caption data, training only the visual projector for multimodal alignment. (2) Un- freeze the L.LM and fine-tune on visual un- derstanding data enhance comprehension. (3) Freeze the LLM and train only the visual heads on text-to-image data. (4) Unfreeze all components and perform joint training on mixture of understanding, generation, and interleaved datasets, enabling the model to acquire generative capabilities while main- taining strong understanding performance. We listed the data in Table. 14.
Table 14: Training data list.
Stuge
Dataset
Visual Tokenizer
CC12M (Changpinyo et al., 2021), ImageNet- IK, 50M
images from LAION-400M (Schubmann al., 2021)
DenscFusion-11 LM (Li et al., 2024b), DrcamLIP (Zheng
MLLM Stagel
etal. 2024), InternVL-SA
Caption (Chen 2024c)
DoeStrutsM (Hu atal. 2024), WebSight (Laurengon al.
MLLM Stage2
2024b), WuKong, 2M house VQA data, pure text data
lillered subsct ImageNet-21K laloo-acsilittcc-
MLLM Stage3
JoumeyDB (Sun et 2023)
lo-house aesthetics data, OmniEdit (Wei et al,, 2024),
teat2face Cauldren (Laurençon et al., 2024a), Instract- Pix2Pix (Brooks et al., 2022), Inhouse IFT data (Und.), OBELICS (Laurençon al., 2023), text data
MLLLM Stage4
"The text and image are reversed and used for image generation training.
20


Reviews:
[{"review_id": "Jqeukhy4m6", "reviewer": "Reviewer_GZrq", "summary": "This paper proposes DualToken, a novel approach aimed at unifying visual understanding and generation within autoregressive large language models (LLMs). The core idea is to introduce dual visual vocabularies—one for low-level visual details (used in generation) and another for high-level semantics (used in understanding). By decoupling these two objectives, the authors resolve the conflict between reconstruction and semantic tasks, leading to improved performance in both areas. The approach also shows strong effectiveness in multimodal understanding benchmarks, making it a promising solution for unified vision-language models. The method is efficient, with minimal additional computational overhead, and the results highlight the advantages of unifying visual understanding and generation in a single framework.", "questions": "Please refer to the above weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The writing in this paper is clear and well-structured, effectively explaining the novel concept of dual visual vocabularies and its benefits in unifying visual understanding and generation. The experimental design is robust, with thorough evaluations across multiple benchmarks, demonstrating the superiority of DualToken over existing methods in both reconstruction and semantic tasks. The authors also provide comprehensive comparisons and detailed ablation studies, which strengthen the validity of their claims and offer valuable insights into the practical impact of their approach.", "weaknesses": "1.Heuristic layer selection limits generalization and reusability. The method applies reconstruction supervision to early layers and semantic supervision to a fixed deep layer, yet this design choice lacks thorough ablation studies and relies heavily on heuristics, raising concerns about its generalizability across architectures and tasks.\n\n2.Overly simplified semantic supervision may undermine representation quality. The use of L2 regression to pretrained SigLIP’s final-layer features, without contrastive objectives or cosine similarity constraints, raises questions about whether this feature-level alignment is sufficient to preserve rich semantic capacity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jJasLLJ25g", "reviewer": "Reviewer_AqP7", "summary": "The paper proposes DualToken, a unified visual tokenizer that decouples the tokenization process by applying quantization to shallow and deep layers of a single SigLIP backbone for reconstruction and semantic learning, respectively. The resulting tokenizer demonstrates strong reconstruction performance and achieves consistent improvements over VILA-U across both visual understanding and generation tasks.", "questions": "1. Table 1 would benefit from a clearer organization, for example by grouping models according to factors such as the number of model parameters, downsampling ratio or number of tokens, codebook size.\n2. The experiment of comparing single-encoder versus dual-encoder designs would be more convincing if compared against recent dual-branch architectures such as TokenFlow[1] or MuseVL[2], which also aim to reconcile semantic and reconstruction requirements rather than merely stacking two unrelated encoders.\n\n[1] Qu, Liao, et al. \"Tokenflow: Unified image tokenizer for multimodal understanding and generation.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[2] Xie, Rongchang, et al. \"Muse-vl: Modeling unified vlm through semantic discrete encoding.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method demonstrates solid results across both understanding and generation tasks.\n- The paper is well written and easy to follow.", "weaknesses": "1. Insufficient Experiments: The paper would benefit from more comprehensive ablations on model design, such as analyzing how the choice of reconstruction layer affects the final performance.\n2. Limited Baseline Comparison: The single-encoder vs. dual-encoder experiment lacks comparison with stronger dual-branch architectures, which undermines the validity of the method’s claimed effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "12QBoX0nHo", "reviewer": "Reviewer_PrxC", "summary": "This paper proposes DualToken, a unified tokenization pipeline that decomposes visual inputs into both content tokens and discrepancy tokens, aiming to improve visual-language and vision-only tasks under the same framework. The method is evaluated on several benchmarks to show improved representation consistency and task performance.", "questions": "1. Why were stronger modern tokenizers not included in comparisons? Without these, the results are incomplete.\n\n2. Can the authors provide concrete justification that the content/discrepancy decomposition corresponds to meaningful semantic factors, rather than just acting as an ad-hoc feature split?\n\n3. What is the computational overhead introduced by the two-branch tokenization versus standard single-stream tokenizers?\n\n4. How does the method perform when scaling to higher-resolution or longer input sequences?\n\n5. Could the authors release qualitative examples to demonstrate that discrepancy tokens capture non-trivial representation variation?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation to unify visual tokenization across tasks is reasonable.\n\n2. The writing is generally clear and the architecture is presented in an organized manner.\n\n3. Experiments cover multiple downstream tasks.", "weaknesses": "1. The core idea of decomposing visual features into semantic and pixel parts has been explored extensively in prior tokenizer and unified representation works (e.g., TokenFlow, UniTok, UniToken, Bagel, Show-o2, etc). The paper does not clearly articulate what is conceptually new here.\n\n2. The experimental comparisons omit recent strong tokenizer-based or unification SOTA models. Without these, the claimed performance improvement is not convincing.\n\n3. The paper frames itself as “unifying tokenizers,” but the design appears largely heuristic, and no strong evidence is provided that this formulation generalizes broadly or scales to high-performing unification models.\n\n4. There is limited explanation of how or why the proposed discrepancy tokens help. The model may simply benefit from an increased token budget rather than a meaningful representational factorization.\n\n5. Some baselines are outdated or under-optimized. The improvement margins are small, and significance is unclear.\n\n6. The conclusion reads brief and generic. It does not help reinforce the contributions or position the work in the broader literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0f1zo36ui9", "reviewer": "Reviewer_kK8G", "summary": "This paper tackles the challenge of unifying visual understanding and generation within the autoregressive framework of large language models. The authors propose DualToken, which disentangles high-level semantic and low-level visual representations through separate codebooks to reconcile their conflicting objectives. This design enables effective integration of reconstruction and semantic learning, achieving strong results on both understanding and generation tasks. Overall, the work presents a well-motivated direction toward unified multimodal representation learning.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper presents a clear and well-articulated motivation, with a logical flow from problem identification to solution design.\n\n2. The evaluation covers a broad range of downstream tasks across both visual understanding and generation, demonstrating comprehensive empirical validation.\n\n3. The proposed disentangled dual-codebook design is conceptually elegant and effectively resolves the conflict between reconstruction and semantic alignment.", "weaknesses": "1. Although the proposed disentangled dual-codebook design is effective, it lacks substantial architectural novelty. Moreover, the insights presented in this work have already appeared in recent vision tokenizer studies, such as UniTok.\n\n2. The claim that “dual tokens promote each other” seems somewhat overstated. As shown in Table 4 (rows (c) and (d)), the pixel tokens do not consistently enhance performance on understanding tasks.\n\n3. The reported downstream results are primarily based on experiments with a 3B model. Additional studies across different LLM scales are needed to validate the method’s generality and scalability.\n\n4. The paper lacks a detailed ablation study on how the boundary between low-level and high-level layers in the vision encoder is determined, which would help clarify the effectiveness of the proposed disentanglement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)