SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation
Authors: Zisheng Chen
Chunwei Wang
Runhui Huang
Hongbin Xu
Xiuwei Chen
Jun Zhou
Jianhua Han
Hang Xu
Xiaodan Liang
Venue: ICLR 2026 Poster (2026)
Keywords: unified image tokenizer
multimodal learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This paper introduces a unified image tokenizer, named SemHiTok, which achieve a better trade-off between semantic information and texture information,  and obtains competitive performance on the unified MLLM based on this training.

Abstract: In this paper, we introduce SemHiTok, a unified image Tokenizer via Semantic Guided Hierarchical codebook that provides consistent discrete representations for multimodal understanding and generation. Recently, unified image tokenizers have sparked exploration within the research community, which is designed to capture high-level semantic features for understanding and retaining low-level pixel features for generation. Previous works attempt to train a unified image tokenizer by combining loss for semantic distillation and pixel reconstruction. However, due to the differing levels of features prioritized by multimodal understanding and generation, joint training methods face significant challenges in achieving a good trade-off. SemHiTok addresses this challenge through a novel semantic-guided
hierarchical codebook, which builds pixel sub-codebooks on a pretrained semantic codebook. This design decouples the semantic and pixel in terms of structure and training strategy, enabling the tokenizer to capture pixel features while retaining its ability to comprehend high-level semantic information. Our experiments demonstrate that SemHiTok achieves leading performance in image reconstruction and multimodal understanding under the LLaVA-v1.5 setting. Further, we develop a unified MLLM with SemHiTok, which exhibits superior performance across multimodal understanding and generation tasks. Extensive experiments confirm our analysis, showing that our unified image tokenizer architecture achieves a better trade-off.

Published as a conference paper at ICLR 2026
SEMHITOK: A UNIFIED IMAGE TOKENIZER VIA SAMANTIC-GUIDER HIERARCHICAL CODEBOOK FOR MULTIMODAL UNDERSTANDING AND GENERATION
Zisheng Chen Chunwei Wang", Runhui Huang", Hongbin Xu", Xiuwei Chen', Jun Zhou', Jianhua Han", Hang Xu², Xiaodan Liang' Iff
1 Sun Yat-sen University 2 Huawei Noah's 's Ark Lab
3 University of Hong Kong
4 South China University of Technology
Work done as an intern at Huawei Noah's Ark Lab. Corresponding Author
ABSTRACT
In this paper, we introduce SemHiTok, a unified image Tokenizer via Semantic- Guided Hierarchical codebook that provides consistent discrete representations for multimodal understanding and generation. Recently, unified image tokenizers have sparked exploration within the research community, which designed to capture high-level semantic features for understanding and retaining low-level pixel features for generation. Previous works attempt to train a unified image tokenizer by combining loss for semantic distillation and pixel reconstruction. However, due to the differing levels of features prioritized by multimodal understanding and generation, joint training methods face significant challenges in achieving good trade-off. SemHiTok addresses this challenge through a novel semantic-guided hierarchical codebook, which builds pixel sub-codebooks on pretrained semantic codebook. This design decouples the semantic and pixel in terms of structure and training strategy, enabling the tokenizer to capture pixel features while retaining its ability to comprehend high-level semantic information. Our experiments demon- strate that SemHiTok achieves leading performance image reconstruction and multimodal understanding under the I.L.VA-v1.5 setting. Further, we develop a unified MI.LM with SemHiTok, which exhibits superior performance across multimodal understanding and generation tasks. Extensive experiments confirm our analysis, showing that our unified image tokenizer architecture achieves a better trade-off.
INTRODUCTION
In recent years, autoregressive models have achieved great success in natural language processing and have been extended to the multimodal understanding domain, demonstrating immense potential. This triggers researchers' interest in unified multimodal understanding and generation by employing single autoregressive framework. To achieve unified multimodal large model, the key challenge designing tokenizer suitable for both multimodal generation and understanding tasks.
However, there vast gap in the visual information required for these two tasks. For instance, models from the CLIP (Radford et al. 2221a: Sun et al. 20233] Zhai et al. 2023) family, commonly the contrary, the
used in multimodal understanding tasks, tend lose visual pixel information.
VQGAN (Yu et al.] [221aa Zhu et al.) (024a) family models, often used in autoregressive generation tasks, lack the ability to extract semantic features for multimodal understanding tasks. This leads to poor performance when single tokenizer is applied to a unified MILM (Wu et al. 2024bb Jin et al.) 2023 Li et al., 2024b). In light of the aforementioned issues, some recent work has attempted to incorporate semantic learning branch into the original VQGAN training pipeline, aiming obtain a unified tokenizer via joint optimization. VILA-U (Wu et al. 2024c) employs a straightforward combination of semantic alignment loss and pixel reconstruction loss, which allows the model to

---

Published as a conference paper at ICLR 2026
Hybrid Codebook
Sem Codebook
Sem Codebook
Semantic Encoder
Semantic Encoder
n >>>s
Semantic Encoder
Sem Loss
Joint Training
$
Joint Training
Pixel Encoder
Pixel
Pixel Encoder
Pix Loss
Encoder
3333
Pixel Codebook
Pixel Codebook
(a) HyStruct & J'Tran
(b) DStruct & J'Train
(c) DStruct & DTrain e-g. Ours(Sem(HiTok)
egg, SDE
e.g. TokenFlow
Figure 1: Illustration of other tokenizers and SemHiTok. HyStruct: Using single model to extract information at different levels; DStruct: Using different models to extract information at various levels; JTrain: Using joint optimization training strategy; DTrain: Adopt phased optimization training strategy.
capture both low-level and high-level information. Nevertheless, the reliance on hybrid structure and joint optimization often drives the tokenizer toward suboptimal solution. Furthermore, some recent works (Xie et al., 2025 Qu et al. 2024) have built upon it with further improvements. As shown Fig [a), [a), while SDE (Xie et al. 2025) further decouples the encoders, the remaining hybrid codebook continues to impede the model's optimization. TokenFlow (Qu et al. [024] uses shared mapping to decouple the semantic branch and pixel branch while maintaining the consistency of the codebook index, but joint training still affects the final performance.
straightforward approach is to use CLIP and VQGAN to extract semantic and pixel information, re- spectively, and the concatenation of these token sequences is then used unified representation. Janus (Wu et al.] (2044) introduces a dual-encoder method that separates encoders for understanding and generation tasks to address this conflict, but this increases the complexity of handling mixed tasks and does not fundamentally resolve the feature conflict challenge. However, this leads to doubling of the token sequence count or multiplicative expansion vocabulary size, depending on whether the concatenation is applied along the length or dimension. These limitations underscore a fundamental challenge the field: How to balance semantic-level and pixel-level information effectively, without compromising the ease of integration into MILLM frameworks?
To address this challenge, we propose SemHiTok, unified image tokenizer that provides consis- tent feature representations for multimodal understanding and generation tasks through a unique hierarchical codebook design. Inspired by the observation that image patches with the same seman- tic code tend to have similar pixel features, we introduce novelty hierarchical codebook which uses a sub-codebook to model the pixel-level space associated with each semantic code, named Semantic-Guiee Hierarchical Codebook(SGHC) Unlike existing approaches, SemHiTok supports stage-wise training paradigm where each stage exclusively optimizes specific hierarchy level features, allowing us achieve better trade-off between semantic and pixel feature extraction. In addition, SemHiTok can be seamlessly integrated into existing MLLMs following the next-token paradigm through a simple codebook flattening operation.
Our contributions can be summarized as follows: (1): A novel unified tokenizer that achieves trade-off between semantic and pixel information, demonstrating outstanding performance in both image reconstruction and multimodal understanding tasks. (2): We develop a unified MLLM architecture that demonstrates superior performance across both multimodal understanding and generation tasks, validating its versatility. (3): Our approach further pushes the performance boundary of unified discrete MILMMs, enabling improved scalability and representation capacity within next- token prediction frameworks.
2
METHOD
The main objective of SemHiTok is to establish simple and unified image tokenizer for multimodal understanding and generation. this model, the image is transformed into discrete tokens that contain semantic information and pixel information. We begin with semantic codebook training recipe that reconstructs semantic features extracted from a language image pre-training model (Zhai et al.) 2023. Wei al, 2022), and point out the semantic codebook's poor texture feature representation in
2

---

Published as a conference paper at ICLR 2026
continue
SBHC
discrete
Sementic Encoder
Semantic Decoder
Sem Lees
Pix Less
Encoder
Decoder
Input Image
(a) Overview of SomtliTok
dog playing grass
BPE Tokenizer
allocate
flatton
SemHiTok
pxm--1 *** pxm++m1
(b) Codebook
(c) Uni-MLLM Architecture
(a) SemHiTok is structurally composed of two branches: semantic branch and pixel branch. The semantic branch is trained following the VQKD (Wei al., (022). where the semantic codebook is learned through semantic loss. We propose semantic-guided hierarchical (SGG))) composed of multiple pixel sub-codebooks, in which each pixel sub-codebook in one-to-one correspondence with semantic code. The selection of pixel sub-codebook is indexed by the semantic code from semantic quantization. To enable unified discrete representation, we concatenate the quantized semantic and pixel features along the channel dimension and feed the result into the pixel decoder for reconstruction. (b) Each semantic code is allocated to the corresponding pixel codeoooo,,,, and their features are concatenated along the dimension. (c) An illustration of the unified MLLM framework.
Figure 2:
section 2.1 In section 2.2 we conduct preliminary discussion and observation. Building on this observation, we introduce Comaantcc-Guidad Hierarchical Codebook(SGHC), incorporating texture information while perfectly inheriting the semantic information of the semantic codebook, to enable pixel reconstruction enablement. Finally, introduce the application of SemHiTok unified MLLM in section 2.3
The overview framework is shown Fig
2.
SEMANTIC CODEBOOK TRAINING
For multimodal understanding, using text-aligned visual encoder (Zhai al. 2023 Radford et al., 2021a, Sun et al.) 2023 Wang et al. 2024b) as an image tokenizer can accelerate convergence and improve performance. However, these text-aligned visual encoders typically output continuous semantic features. In this work, to achieve unified visual tokenizer, we first train a semantic
codebook to quantize the continuous semantic feature following VQKD (Wei et al. 2022).
Given image X xxxx3 the semantic encoder Exem extract continuous semantic features:
Zsem Esem (X) R hhxww xxxxx
Where Eyem is frozen text-aligned image encoder, e.g., CLIP (Radford et al. 2021a) or SigLIP ZZhii 2023). Then 7sem are transformed into discrete feature space Csem {CI, C2 ..... CK E
R A through quantization function Qxem (+). The quantization process Qsem is as follows:
Zi qrem arg min [|Zsem - -sem KE
Where T E [Csem)] 1 quantized index, 7q is discrete feature indexed from Csem- Finally, semantic decoder D.emm maps Zazem to raw semantic feature space Zeem. The D Deem are end-to-end trainable by minimizing semantic distill loss:
Leem 1 cos( Zy.co. 2 + 12 Z
For Csem. we adopt EMA (Hunter 1986 VQ as the semantic codebook. Unlike traditional quantiza- tion methods, the EMA VQ is not updated via gradient descent, but instead through Exponential
3

---

Published as a conference paper at ICLR 2026
Semantic Code
V
Frank
Rec
V1048
Geldfish
GT
V2071
eyes
V3108
Rec
combó
V14312
GT
1163B4
Figure 3:
Visualization of reconstruction by quantized semantic features. Semantic codebook tends to produce inaccuracies in the reconstruc- tion of image textures and color information.
Figure 4: Visualization of semantic code. Each code corresponds of image patches that share similar pixel-level fea- tures.
Moving Average (EMA) algorithm:
N
(t-1) + m))
m Cj m
Zi:
N
where cf denotes the k-th codebook vector at update step L, m is the momentum term, and the update based the average of all input vectors z, assigned code k in the current batch. However, we further conduct experiment that reconstructs the original pixel from the quantized semantic features extracted by Csem- The reconstructed images exhibited noticeable blurriness and a significant loss of high-frequency details, as shown in Fig. This indicates that the semantic codebook lacks pixel information.
2.2 PIXEL RECONSTRUCTION ENABLEMENT
we demonstrate that semantic code lacks the ability to model pixel information. In order to pixel reconstruction enablement and avoid reduction of understand ability, straightforward approach to add an extra VQGAN (Yu et al.] 2021a) la) model. Semantic codebook extracts discrete semantic tokens for multimodal understanding, and VQGAN extracts discrete texture tokens for generation. The two token sets are concatenated either dimensionally or sequentially, and passed to the LLM. However, the resulting token inflation oversized codebook introduces significant computational burden, limiting its feasibility for MLLMs.
Discussion. In section 2.1
Furthermore, we present the visualization results of the semantic code, shown in Fig It can be observed that image patches corresponding to the same code exhibit similar pixel features. For example, the code F14312 is more likely to be assigned to the rooster comb element in the image. At the same time, the image patches corresponding to these combs exhibit similar pixel features, such as color, patterns, and shapes. Based this observation, propose Semantic-Guided Hierarchical Codebook to model the pixel feature space corresponding to each semantic code using a sub-codebook.
Gemantic-Guider Hierarchical Codebook (SGHC). The SGHC consists of pretrained semantic codebook and several sub-codebooks, where each sub-codebook corresponds to a semantic code of the semantic codebook, shown in figure (a). Specially, given the pre-trained semantic codebook Ceem {c1, ..... CK} E I Xxdeem the pixel codebook Cpiz = [C Cz Cpi...... CK e I K xmxi, where C.. E mxdgix is ken semantic code's pixel sub-codebook. is sub-codebook size. At first, the semantic codebook quantizes to a discrete semantic token Zsem and semantic codebook index T In parallel, pixel encoder Epiz extract continuous pixel features 7piz
4

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of reconstruction quality on the ImageNet-50k validation set. c: quantizer uses residual quantization (RQ), where the total Code Shape is multiplied by RQ depth. 1 quantizer uses multiple codebooks and product quantization.
Sem
PRE
Methall
Res
Code Shape
Codebook Sire
(ID)
ED
Chiy Resouutraction
LiamaGen Sun a M0
256
16 16
16,394
2.19
ROYAE [L 0 Br pizza
256
16
16 x
16,354
3.20
VOGANTrC 7he 1e 2004a
256
16 16
100,000
2.62
180
Shi 1P 2014
256
16 16
16, 354
1.37
18Q
256
16 16
262,144
1.00
RQ0 UNTT 20224
256
16
10 x
16, 384 x 2'
0.94
Uighel
VILA U (Wa al 70248
SDEMMUSEVLI 17 AIt
2025
156
256
16 x 16
32,758
16,354
2.25
1.80
Figure 5: Visualized reconstruction results from the ablation of key modules. PRE brings
Takeni yeu (o et
2021
256
650
32,768
1.37
TokLIP
256
16 x 16
16,354
2.19
Q1II-RR
256
16 16
285
3.21
about a significant improvement in recon- struction quality. Moreover, the Enhance De- coder(ED) further improves reconstruction on hard samples.
UniTok
Ma
256
16,384
0.39
256
16
196,008
1.16
SemfiiTokreurs)
384
196,08
0.66
Epiir (X). For the quantization process of the pixel codebook, the corresponding pixel sub-codebook selected based on the quantization result of the semantic codebook. For instance, given image patch i, its semantic quantization codebook index & and continuous pixel feature 2 Zpiz. SGHC selects
pixel sub-codebook Cpia. to quantize Zjai. The process is follows:
Z
1 Open Apir
arg min je(I....mm)
1pipe -Crin Dill
(5)
where j is the selected sub-codebook internal index. Finally, the semantic quantized tokens and pixel quantized tokens are concatenated to Zy concatedim ( (Zerem+ Zppce as the input of pixel decoder Dpix to reconstruct the raw pixel image:
X D Dpiz
(6)
where X reconstrueted pixel image. The Epize Cpiz and Picc are end-to-end trainable by minimiz- ing reconstruction loss Limg (X X). codebook loss Le, perceptual loss Ipper and represents adversarial loss Lyan YYu et al. 202 The reconstruction loss is formulated as:
Lrec = Limg + Ac + A2 + AzTagan
where \1, X2 and A loss weight of each item.
Our SGHC can be regarded as the refinement of a semantic discrete space to enable pixel recon- struction. We place a specific emphasis two key advantages of SGHC: (1) Non-Confticting Extension: Our method leverages a pre-trained semantic codebook as a foundation, with pixel reconstruction losses exclusively employed optimize pixel branch modules during the PRE. This strategic approach effectively circunvents the suboptimal solutions that arise from joint optimization processes. Furthermore. SGHC's final output is generated by concatenating remantic-quantizel features with pixel- mixel-quantized features, preserving the full expressive capacity of the original semantic features while integrating complementary texture information through this unified feature fusion paradigm. Subsequent tasks, such as reconstruction, multimodal understanding. and generation, all share the same discrete token representation; (2) Efficient Downstream Applications: SGHC effectively avoids two critical predicaments: token quantity inflation and codebook overexpansion. As defined before, the semantic codebook size is K, and each pixel sub-codebook size is m. Due to dimensional concatenation, the complete codebook flattens to K x m, where m is much smaller than K. In our experimental default settings, we extend the complete codebook to a size comparable to existing I.LMs' text vocabulary size, e.g., the size of Qwen2 vocabulary is 150k.
5

---

Published as a conference paper at ICLR 2026
Table 2: Comparative analysis of tokenizers on multimodal comprehension tasks. *; Both the tokenizer and LLM are reproduced in our setting. Our method achieves SOTA performance compared with other discrete tokenizers.
Model
LLM
Data
Res.
POPE
MME-P
SEED
GQA
SigL.P Zhai et al.
2023
Vicuna-7B
LLaVA-vI.5
256
83.76
1481.0
65.28
61.9
LlamaGen (Sun al.
2024a
Vicuna-7B
LLaVA-v1.5
256
65.6
716.8
35.0
39.8
VILA-U (Wu et al.
20240
Vicuna-7B
LLaVA-v1.5
256
81.6
1311.6
56.9
55.3
SDE(MUSE VL.
Xie et al.
2025
Vicuna-78 7B
LLaVA-vi.5
256
77.3
1240.0
56.7
58.0
TokLIP (Lin al.
2025
Qwen2.5.77 S-n-2.S-7B-Ins
LLaVA-v1.5
256
81.2
1346.8
59.8
57.4
SemlliTon((Urs)
Vicuna-7B
LLaVA-v1.5
82.5
1355.8
62.9
60.3
TokenFlow-3Sk et al.
2024
Vicuna-7B
LLaVA-v1.5
384
84,9
1416.4
62.7
61.2
TokLIP (Lin 2025
Owen2.5-7B-Imm
LLVVA-v1.5
384
82.7
1410.2
65.2
59.3
Semllittos-384(0urss
Vicuna-7B
LLaVA-v1.5
384
86.3
1465.6
64.1
62.3
2.3
UNIFIED MLLM EQUIPPED WITH SEMHITOK
The framework diagram for unified MLLM shown in Cc) We use SemHiTok develop unified multimodal model, which models discrete vision and text token sequences with universal next-token prediction loss. Particularly,
image processing, SemHiTok is utilized discretize images into token sequences. On the model side, merely expand the text vocabulary and adjust the head layer to incorporate visual token IDs. To enable a unified head layer, we flatten SGHC by merging all sub-codebooks into single flat representation as shown in Fig 2 Specifically, for the jeh semantic code the 4sh pixel sub-codebook, the discrete code index in the completed codebook m + j. where m the sub-codebook size. It is also worth noting that the vocabulary expansion is merely for implementation convenience. We still use the features extracted from SGHC as imput and align with LLM through a lightweight adapter layer. After training is completed, we replace the visual component in the embedding layer in order to achieve consistency between training and inference. To enable LLM to better handle features at two different levels, we introduce
Dual-MLP adapter layer, which projects semantic features and pixel features separately, and then concatenates them along the dimension before feeding them into the L.LM. To enable classifier- guidance (Ho& & Salimans| 2022), randomly replace the text condition with probability of 0.1 to the unconditioned text during training. More deployment details are provided in the Supplementary
Material 7.2
3 EXPERIMENTS
3.1 EXPERIMENTAL. SETUP
Tokenizer. For the semantic branch, we employ SigLIP Zhai et al., 2023) as the semantic encoder and three self-attention layers as the semantic decoder reconstruct semantic features. For the pixel branch, we employ ViT as both the pixel encoder and decoder, assigning 8 pixel sub-codes to each semantic code. More tokenizer detail, please refer to Supplementary Material 7.3
Unified MILM. We use Brren2.5-7B-Intrrune (Yang al., 2024) as the base LLM, and expand its vocabulary and output head layer. We evaluate visual understanding on standard VQA benchmarks including SEEDB (Li et al. 2023a), POPE (Li et al. (023c). GQA (Hudson & Manning (019). MMMU (Yue al.) 2024). MMB (Liu et al.) (024b) and MME (Fu et al.) (023).. For visual generation evaluation, report results MJHQ-30K 3-00KK33K (Li et al.) (202c)), GenAl-Bench (Li et al.) 2024a), GenEval (Ghosh et al., 2023) and DPG (Hu et al. 2024). More unified MLLM detail, please
reter Supplementary Material 7.4
3.2 UNIFIED IMAGE TOKENIZER
Image Reconstruction. We present the reconstruction performance on the ImageNet-50k validation set in Tab[I] Notably. SemHiTok excels in reconstruction quality compared the unified tokenizer, recording an impressive 1.16 rFID with 16x downsampling ratio. While SemHiTok's 's vocabulary size(approx 1 196k) appears larger than baselines, it is crucial consider the effective representational

---

Published as a conference paper at ICLR 2026
Table Quantitative results on multimodal understanding benchmarks. SemHiTok achieves SOTA performance on most benchmarks among Gnd&Gen Discrete MLLMs, and is comparable to or even surpasses some Only Und and Und&Gen. Continuous models. The performance on Und& Discrete with top-1 and top-2 values is denoted in bold and underline, respectively.
Method
# Params
Res.
SEED
POPE
GQA
MMMU
MMB
MME
MME-P
MMV
Ouly Und.
ALiNA-Phi Zhu
2024b
2,7B
256
85.0
59.8
1335,1
28,9
LaVA-vl
1510.7
Owen-VI-Chil ea 20236
7B
448
57.7
57.5
30.5
1848.3
1487.5
ShareGPT4V
Undd&Gen. Continuous
LaVIT Din (2023)
224
58.0
Janus (Wir er 20244
1.58
384
63.7
87.0
59.
30.5
69.4
1338.0
34.3
Janus. Pro-
2025
1,5B
384
68.3
86.2
58.9
38.9
65.5
1444.
2025
1.58
67.1
58.9
38.9
65.5
1155.0
Gen, Discrete
LWM
et al. 2024a
7B
256
75.2
44.8
9.6
SEED-1
20244b
13B
256
53.7
1097.22
Liquid
Wu a 20246
7B
512
81.1
71.3
1119.3
EMU3
68.2
85.2
60.3
31,6
58.5
1509.9
1243.8
37.2
VILA-U
Wa sh 202400
7B
256
56.3
83.9
58.3
1336.2
27.7
VILA-U
Wo
7024g
78
384
59.0
85.8
60.8
1401.8
33,5
Dian
TokLIP
Li 2025
7B
384
76.9
84.1
59.5
43.1
67.6
1488.4
29.8
TokenFlow--
2024
13B
224
84.0
59.3
34.2
58.3
1660.4
1353.6
22.4
TokenFlow.1
Cu al
2024
13B
256
62.6
85.0
60.3
34.4
60.3
1622.9
1365.4
27.7
SynerGen-V VL er 2025
2,48
512
62.0
85.3
59.7
34.2
53.7
1837.0
1381,0
34,5
SemHHiTok(OOss/
69.7
22.3
1775.9
SemlliTlok(Ours)
7B
384
79,8
85.5
61.7
41.0
75.2
1993.8
1512.8
36.6
Table 4: Comparison of generation quality GenAl and SemHiTok achieves comparable results with specialist models and unified MLLMs.
#Training Images
GenAl-Bench Basic
MUHQ30K
Model
Params
Type
Res.
Advanced
gFID
Only Gen.
v2.1
RRomaac al.
2022
Diff
2000M
1024
0.78
0.62
DALL-E
[Btkeer
Diff
1024
0.90
0.70
PixArt-n
Chen
2025
0.6B
Diff
1024
6.14
SDXI.
Pocieli et an
2.6B
Diff
2000M
1024
0.83
0.63
9.55
PlaygroundvZ.>
2024c
Diff
1024
4.48
Gen.
LWM (Liu et al.
2024a
7B
AR
256
0.63
0.53
17.77
Show-o Xre
2024
1,5B
Diff
36M
256
0.70
0.60
15.18
Janus
Wu ct a 2024a
1.3B
AR
65M
384
10.10
VILA-U Wi al.
2024e
7B
AR
ISM
256
0.76
0.64
12.81
VILA-U WW al.
20240
ISM
0.73
0.61
7.69
VL
2
2025
2.4B
25M
256
6.10
SDF(MUSF-VIJ) Xie
2025
7B
AR
10M
256
7.73
ILLUME Wang el PAAA
7B
AR
ISM
512
0.75
0.60
7.76
ILLUME+ Huang et al. 2125a
7B
AR
15M
512
0.75
0.60
7.76
TokenFlow Qu mll 20244
7B
AR
60M
0.65
0.65
7.78
Liquid (Wu al. 202410
7B
30M
0.65
5,47
UniTok
2025
7B
30M
256
0.85
0.67
7.46
SemHrTok(ours)
7B
AR
I5M
256
0.83
0.64
5.40
SemHrTokkouus)
7B
AR
I5M
384
0.83
0.66
5,70
capacity defined by the code shape. VILA-U(using RQ) and FQGAN(using Product Quantization) operate within combinatorial search D or Ni x N2), resulting in significantly larger effective capacity than SemHiTok's 's linearly constrained structure( x m). Furthermore, these approaches typically employ denser code shape, whereas SemHiTok relies hierarchical index. This indicates that our superior performance derives from the structured efficiency of SGHC rather than brute-force capacity expansion. Increasing the training 384 resolution led to
single unified
7

---

Published as a conference paper at ICLR 2026
Table 5: Impact of key design choices on reconstruction and multimodal understanding.
Semantic Codebook
Dual
Enhance Decoder
SGHC
MME-P1
MMB1
SEED1
MMU1
rFID
MLP
1387.5
61.3
62.3
35.6
3.17
1355.8
60.7
62.9
35.8
1.42
1393.0
61.6
63.2
36.1
1.42
1393.0
61.6
63.2
36.1
1.16
significant improvement in the rFID score, reaching 0.66. The results validate the effectiveness of SGHC design in modeling pixel feature space of the semantic code.
Va-11.5 Multimodal Understanding. To ensure fair comparison, we conduct experiments to evaluate the multimodal understanding performance of existing open-source tokenizers and SemHiTok under the standard I.La LLaVA-v1.5 setting. The results are as shown in Tab LlamaGen's performance is notably inferior, consequence of its deficient pre-alignment with text. In contrast, unified image tokenizers show considerable advancements understanding. However, due to their inherent hybrid structures joint training strategies, substantial disparity in understanding performance remains between prior discrete tokenizers and continuous representations. Notably, even though TokLIP uses stronger base model (Qwen2.5-7B-In its performance remains inferior to ours. SemHiTok achieves state-of-the-art results for discrete tokenizers, nearing the performance of continuous inputs such SigLIP.
3.3 UNIFIED MLLM
Multimodal Understanding. We evaluate the understanding performance of SemHiTok on diverse benchmarks in Table 3 Und means support multimodal understanding, Gen means support text-to- image generation. Among Und&Gen. Discrete, SemHiTok achieves state-of-the performance on most metrics, such as SEED, MMB, MME, and MME-P. Compared to other unified tokenizers models demonstrate significant advantages. Notably,
(e-g., VILA-U, TokenFlow, UniToken), our model surpasses expert-level models on key benchmarks, achieving 3.8 and 6.4 points higher than ShareGPT4V on MMMU and MMB, respectively. This bridges the gap between discrete visual tokens and continuous visual tokens in multimodal understanding tasks, strongly demonstrating the superiority and potential of our approach. Visualizations on understanding tasks are available in Supplementary Material 7.5
Text-to-Image Generation To evaluate the text-to-image generation, we use GenAl-bench (Li et al. 2024a) and MJHQ30K (Li et al.) 2024c) benchmark, and the results are shown in Tat4 For GenAl-bench, we use clip-flant5-xxl as the VQA score model to reflect the consistency between text descriptions and generated images. On this challenging benchmark, our model achieves competitive performance, closely matching Liquid, even though Liquid employs generation-focuse Furthermore, our model even outperformed some diffusion-based expert models, such SDXL and SD v2. The strong results underscore the superior capability of our unified MLLM in complex tat-tto-imae generation tasks. For MJHQ30K, images and high-quality images. On this benchmark, SemHiTok-256 attains 5.40 gFID, setting a new state-of-the-art in autoregressive image generation. In addition, conduct further quantitative comparisons on GenEval and DPG, which are available in Supplementary Material [7.6] More quantitative analysis results and visualizations on generation tasks are available in Supplementary Material 7.7
tokenizer.
the generation FID metric generated
4 ABLATION
4.1 IMPACT OF KEY DESIGN.
In Ta we validate the impact of our key design choices in SemHiTok: semantic codebook. semantic-guided hierarchical codebook(SGHHC, Dual MLP, and Enhance Deocoder. For efficiency, we only test the understanding performance under LLaVA-v1.5. 11.5, and train 40 epochs on ImageNet- for reconstruction tasks. We begin with the semantic codebook, which suffices for multimodal
8

---

Published as a conference paper at ICLR 2026
Table Impact of training strategy and structure. DStruct: Using different learnable codebooks to model semantic and pixel information; DTTrain: Adopt phased optimization training strategy. Exp used the same architecture as SDE and adopted joint training strategy: Exp using the same architecture as SemHiTok but with joint training; Exp 3: using two separately pretrained tokenizers (Llamagen and SigLIP) to independently extract semantic and pixel information.
Multimodal understanding MME-P1
Reconstruction
Exp
DStruct
DTrain
GQA
SEEDB
POPE1
rFID
Usaget
x
58.0
1240.0
56.7
77.3
3.78
92.9%
2
/
x
57.8
1357.4
55.3
80.4
3.22
45.9%
3
/
58.7
1210.9
56.1
80.1
2.19
97.0%
4
SemHiTok
60.3
1355.8
62.9
82.6
1.42
93.7%
understanding but suffers from poor reconstruction. Incorporating SGHC enables pixel reconstruction and reduces rFID by 1.75, without noticeably affecting understanding performance. Introducing Dual-MLP further enhances multimodal understanding, even surpassing the semantic codebook alone, highlighting the effectiveness of multi-level feature modeling. Finally, stronger pixel decoder brings additional improvements in reconstruction quality. figure demonstrates the effects of various modules on reconstruction results. Compared the semantic codebook, SGHC delivers more detailed reconstructions. Additionally, the enhanced decoder boosts performance on difficult samples. More reconstruction comparisons are available in Supplementary Material 7.8
4.2 IMPACT OF TRAINING STRATEGY AND STRUCTURE.
We conduct comparative experiments under the same hyperparameters to investigate the impact of the training strategy and structure, shown in Table It can be observed that Exp I's performance the lowest across both multimodal understanding and reconstruction tasks. After decoupling the architecture, Exp achieves superior performance compared to Exp 1, yet its codebook usage remains at a low level. Furthermore, Exp achieves marked gains in both efficacy and utilization over Exp 2, which underscores the importance of phased training strategy for the proposed SemHiTok hierarchical architecture. Compared with Exp 3, Exp 4 achieves superior performance on both multimodal understanding and reconstruction tasks. This suggests that naively incorporating pixel features may negatively affect the alignment between image features and the L.M.. In contrast, Exp 4 leverages pixel features as a complementary refinement semantic features, thereby effectively bridging the gap between semantic and pixel representations.
4.3 MORE COMPARISONS AND ABLATION STUDIES
In Supplementary Material 7.9
attempt to provide quantitative evidence to further demonstrate pixel similarity within the semantic code. Meanwhile, to further illustrate how different hyperpa- rameters in method affect model performance, also conduct ablation studies Concat Type and Sub-Codebook Size in Supplementary Material 7.10 Semantic VQ Type, Codebook dim and Codebook size in Supplementary Material 77.1 and different K and more comparison with other large codebook reconstruction methods in Supplementary Material 7.12
RELATED WORK
5.1 SPECIALIZED IMAGE TOKENIZER
Tokenization for Autoregressive Generation. VQVAE Van Den Oord et al. 2017) learns a discrete representation using a learnable codebook auto-encoder architectures. VQGAN further improves the perceptual quality by using adversarial GooddnloGood et al. 2014). improved the architec- ture with perceptual and adversarial based losses for better reconstruction. This approach yields more precise and detailed image representations, significantly improving upon previous methodologies in image generation and processing. In recent literature, researchers are turning to efficient codebook structures Shi et al. 2024 Zhang et al.
2023 eu 2021b Bai at. 2024 and hetter quantization

---

Published as a conference paper at ICLR 2026
methods (Zha et al. 2024 Yu et al.
2025) improve generation performance and compression rates.
IBQ(Shi et al. 2024) proposes the Index Backpropagation Quantization codebook update method, achieving stable training of large-scale codebooks. FQGAN(Bai et al. 2024) uses multiple codebooks and product quantization, where each codebook encodes a different type of feature. However, this work focuses only image reconstruction and generation tasks, performing poorly on multimodal understanding tasks WW et al., 2024b Liu et al.
2024a).
Tokenization for Understanding. In multimodal large language models (MLLMs) et al. 2023b
2022 Radford et al. 2021b Liu et al., 2023a Bai et al. 2023a Chen et al. 2024c), researchers
leverage CadPoRddiord et al., 2021b) and BLIP et al.) 2022) to extract visual characteristics that align with the language during its pre-training phase. Building upon many works (Liu et al.) 2023a]
Bai et al. 2023a Chen et al., 2024c) have been collected and trained on high-quality datasets to achieve remarkable performance. LLaVA(Liv et al., 2023a) utilizes vision encoder to align the vision inputs before L.Mss. Qwen VI V et al. 2023a) and InterVL.(Chen et al. 2024c) achieve better results through increased resolution, higher-quality datasets, etc. However, these text-aligned image encoders tend to focus on semantic information and ignore texture information, which is important for the generation task.
5.2 UNIFIED IMAGE TOKENIZER
Numerous efforts have emerged to develop unified visual generation and understanding within one
Mang et al. 2024a e Xie et al. 2024 Dong et al., 2023 Ge et al., 2024 Sun et al., 2024b)
Team
2024:
Wu et al. 2024d). There are two main lines to bridge the gap. Many workers (Dong
et al.
2024 Sun et al. 2024b) combine diffusion models with LLMs for image 2023 Ge et al.
generation. DREAMI M et al. 2023) presents a unified framework that not only provides multimodal understanding but also creates multimodal content via diffusion models. Emmu ct al. 2024bb trains unified generative model using diffusion- based decoder. These approaches inevitably increase model complexity and are not simple enough. Other workers (Team) 2024
Wu et al. 2024d Xie et al.) 2024 Wang al., 2024c) adopt VOVAE-based encoders convert images into discrete tokens. Chameleon Team 2024) and EMU3 (Wang et al.] 2024c) directly use VQGAN al. 2021a), which is optimized by pixel reconstruction as the image tokenizer, while this method increases resource consumption during the pre-training stage and degrades multimodal understanding performance. VILA- (Wu al. (024d) introduces a unified image tokenizer that incorporates a text-aligned branch within the VQGAN training paradigm. However, due to the gap between the semantic feature and the pixel feature, the joint optimization approach may lead to suboptimal solutions In contrast, our proposed SemHiTok can add the ability of extracting texture features without changing the discrete semantic features, and avoids the challenges brought by joint optimization.
6 CONCLUSION
We present SemHiTok, unified image tokenizer featuring a Semantic-Guided Hi- erarchical Codebook (SGHC) to balance semantic depth with reconstruction fidelity. SemHiTok achieves SOTA results LLaVA-v1.5 understanding and ImageNet-50k reconstruction benchmarks. When integrated into a unified MLLM, it delivers competitive performance in both understanding and generation, offering a powerful, discrete, and fully compatible alternative to existing tokenizers.
Conclusion:
We present two limitations: (1). Low generation efficiency: Owing to the use of standard image quantization methods and settings, each 256-resolution image is represented by 256 tokens. This results in relatively low efficiency and high computational cost. (2). Unified large model potential: In tasks involving natural language understanding and multimodal reasoning. advanced post-training techniques such as Chain-of- -houuhh (CoT) are not explored. This remains a promising direction for future research in the community.
Limitation:
Future Work: In the future, we can explore the potential of unified image tokenizers and test their performance on more difficult tasks, such as image editing and multiple rounds of conversations. In addition, improving the compression ratio of the model and designing a tokenizer that serializes the image in one dimension are also expected.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work is supported by National Key Research and Development Program of YFE022200 Scientific Research Innovation Capability Support Project for National Natural Science Foundation of
Young Faculty (No. Dww.2XXXNNKKYXXNZZXMMMM China (NSFC) under Grants No.62476293 and No.62372482, and General Embodied AI Center of Sun Yan-sen University.
ETHICS STATEMENT
This research does not involve potentially harmful insights, methodologies, applications, and raises no concerns regarding conflicts of interest, sponsorship. discrimination, bias, fairnsss, privacy, security, legal compliance, or research integrity.
REPRODUCIBILITY STATEMENT
Data. The training datasets employed in this study are detailed in Appendix 7.3 and Appendix. 7.4 and all are publicly accessible open-source resources.
Method. To support reproducibility, we elaborate on the methodological details in Section 2 and the implementation will be open- open-sourced after acceptance.
Performance. All evaluations are carried out on open benchmarks, thereby ensuring the reproducibil- ity of our results.
Code And Model Weight. We will open-source the code and model weight files.

---

Published as a conference paper at ICLR 2026
REFERENCES
Inclusion AI, Biao Gong, Cheng Zou, Dandan Zheng, Hu Yu, Jingdong Chen, Jianxin Sun, Junbo Zhao, Jun Zhou, Kaixiang Ji, et al. Ming-lite-uni: Advancements in unified architecture for natural multimodal interaction. arXiv preprint arXiv:2505.02 0,000, 2025.
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision- Assion-anguuag model with versatile abilities. ArXiv, abs/2308. 12966, 2023a. URL/tttss:/ api. semanticscholat org/CorpusID? 263875678
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 1(2):3, 2023b.
Zechen Bai, Jianxiong Gao, Ziteng Gao, Pichao Wang, Zheng Zhang, Tong He, and Mike Zheng Shou. Factorized visual tokenization and generation. arXiv preprint arXiv:241 1. 16681, 2024.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dallle-3 pdf. 2(3):8, 2023.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Saehoon Kim. Coyo-700m: Image-text pair dataset, 2022.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023.
Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan. Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng. Kuo Yang, et al. Emova: Empowering language models to see, hear and speak with vivid emotions. arXiv preprint arXiv:2409. 020.4099.8844 18042, 2024a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pp. 370-387. Springer, 2024b.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501. 2501.17811, 2025.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generie lingul-li tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185 221988, 2024c.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference computer vision and pattern recognition, pp. 248-255,,L00 Icve, 2009.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309. //49,, 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang. Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. / 13394, 2023.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi muatu-granularity comprehension and generation. arXiv preprint (tt::2004.14966 2024.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 52132-52152, 2023.
12

---

Published as a conference paper at ICLR 2026
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Jiaming Han, Hao Chen, Yang Zhao, Hanyu Wang, Qi Zhao, Ziyan Yang. Hao He, Xiangyu Yue, and Lu Jiang. Vision as a dialect: Unifying visual understanding and generation via text-aligned representations. arXiv preprint arXiv:2506. 18898, 2025.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.159%. 2022.
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with I1m for enhanced semantic alignment. arXiv preprint arXiv:2403.0511 2024.
Runhui Huang, Chunwei Wang, Junwei Yang. Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Lanqing Hong, Hengshuang Zhao, et al. Illume+: Illuminating unified milm with dual visual tokenization and diffusion refinement. arXiv preprint arXiv:2504.0199 2025a.
Ziyuan Huang, DanDan Zheng, Cheng Zou, Rui Liu, Xiaolong Wang, Kaixiang Ji, Weilong Chai, Jianxin Sun, Libin Wang. Yongjie Lv, et al. Ming-univision: Joint image understanding and 2025b.
generation with a unified continuous tokenizer. arXiv preprint
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
J Stuart Hunter. The exponentially weighted moving average. Journal of quality technology, 18(4): 203-210, 1986.
Yang Jiao, Haibo Qiu, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 3600-3610, 2025.
Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin Chen, Chenyi Lei, An Liu, Chengru Song, et al. Unified language-vision pretraining in Ilm with dynamic discrete visual tokenization. arXiv preprint arXiv:2309.04660 239996669999, 2023.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 1 11523-11532 2022.
Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, et al. Genai-bench: Evaluating and improving compositional text-to-visual generation. arXiv preprint arXiv:2406. 13743, 2024a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Beed-bench: Bench- marking multimodal Iims with generative comprehension. arXiv preprint 23X1:23377.1115. 2023a.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed- bench Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12299-13308 2024b.
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi, Playground v2. 5: Three insights towards enhancing aesthetic quality in eox-tt--mmge generation. arXiv preprint arXiv:2402.17244 2024c.
Hao Li, Changyao Tian, Jie Shao, Xizhou Zhu, Zhaokai Wang, Jinguo Zhu, Wenhan Dou, Xiaogang Wang, Hongsheng Li, Lewei Lu, et al. Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29767-29779, 2025.
13

---

Published as a conference paper at ICLR 2026
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference on PMLR, 2022.
machine learning, pp.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. B Bootstrapping language -inuage-image pre- pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination large vision-language models. arXiv preprint arXiv:2305. 10355, 2023c.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces, 2023.
Haokun Lin, Teng Wang, Yixiao Ge, Yuying Ge, Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun, and Ying Shan. Toklip: Marry visual tokens to clip for multimodal comprehension and generation. arXiv preprint wrXiv:2505.05422 2025.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv e-prints. pp. arXiv-2402, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangho Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233, Springer, 2024b.
Zhuoyan Luo, Fengyuan Shi, Yixiao Ge. Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual generation. arXiv preprint arXiv:2409. 044 10, 2024.
Chuofan Ma. Yi Jiang, Junfeng Wu, Jihan Yang. Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiao- juan Qi. Unitok: unified tokenizer for visual generation and understanding. arXiv preprint arXiv:2502.20331, 2025.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. PmLR, 2021a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
PmLR, 2021b.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:204.006125, 1(2):3, 2022.
K Ramamohan Rao and Ping Yip. Discrete cosine transform: algorithms, advantages, applications. Academic press, 2014.
14

---

Published as a conference paper at ICLR 2026
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the JEEE/CVF Confer- on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Fengyuan Shi, Zhuoyan Luo, Yixiao Ge, Yujiu Yang. Ying Shan, and Limin Wang. Taming scalable visual tokenizer for autoregressive image generation. arXiv preprimt arXiv:2412.02692 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024a.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303. 15389, 2023.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang. Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. Proceedings of the IEEE/CVÉ Conference on Computer Vision and Puttern Recognition, pp. 14398-14409, 2024b.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818. 2024.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu. Illume: Illuminating your lims to see, draw, and self-enhance. arXiv preprint arXiv:2412.06673, 2024a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2/1,, 2024b.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint 2rX9 18869, 2024c.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervise visual pre-training. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 14668-14678, 2022.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint 24X0 13848, 2024a.
Junfeng WU, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable multi-modal generators. arXiv preprint arXiv:2412.04332 2024b.
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Zhonghua Wu, Qingyi Tao, Wentao Liu, Wei Li, and Chen Change Loy. Harmonizing visual representations for unified multimodal understanding and generation. arXiv preprint arXivv2503..1975 2025.
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04422 2024c.
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xic, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint ar arXiv:2409.0444 2024d.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijic Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint a408v:2408, /2528, 2024.
15

---

Published as a conference paper at ICLR 2026
Rongchang Xie, Chen Du, Ping Song, and Chang Liu. Muse-vl: Modeling unified vlm through se- mantic discrete encoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 141155244466 2025.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024a.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned Ilms with nothing, arXiv preprint arXiv:2406.08465 2024b.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1:X11:2412.15115, 2024.
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505. 2025.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:2110.04622 2021a.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXhv:2110.04627 2021b.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737 2023.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chiee Chigh Chen. An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 37: 189400118886, 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Kaiwen Zha, Lijun Yu, Alireza Fathi, David Ross, Cordelia Schmid, Dina Katabi, and Xiuye Gu. Language- Language-guided guided image tokenization for generation. arXiv preprint arXhv:2412.05776, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986, 2023.
Jiahui Zhang, Fangneng Zhan, Christian Theobalt, and Shijian Lu. Regularized vector quantization for tokenized image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18467-18476, 2023.
Yue Zhao, Fuzhao Xue, Scott Reed, Linxi Fan, Yuke Zhu, Jan Kautz, Zhiding Yu, Philipp Krähenbühl, and De-An Huang. Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation. arXiv preprint arXiv:2502.05177 2025.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.1 1039, 2024.
Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vggan to 100,000 with utilization rate of 99%. arXiv preprint arXiv:2406. /137, 2024a.
16

---

Published as a conference paper at ICLR 2026
Yichen Zhu, Minjie Zhu, Ning Liu, Zhiyuan Xu, and Yaxin Peng. Llava-phi: Efficient multi-modal assistant with small language model. Proceedings of the 1st International Workshop on Efficient Multimedia Computing under Limited. pp. 18 22, 2024b.
Yongxin Zhu, Bocheng Li, Yifei Xin, Zhihua Xia, and Linli Xu. Addressing representation collapse in vector quantized models with one linear layer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2025.
17

---

Published as a conference paper at ICLR 2026
7 TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL
7.1 LLM USAGE STATEMENT
We clarify that the use of LLMs in this study is restricted to writing assistance, specifically for grammar correction and enhancing readability. No LLM was involved in the research design, experimental execution, or data analysis.
7.2 DEPLOYMENT DETAILS OF UNIFIED MLLM.
Training Data Form. For multimodal understanding, we use <lim_starth and <lim_endi: to delimit the image segment within the input sequence. To distinguish between modalities and enable visual content generation,
insert special tokens: <IMG XXXXX>, which represent image codes in LLMs vocabulary. Specifically, the i-th code in the unified tokenizer codebook corresponds to <IMG i>> In addition, we add <start of image> and <end of image> to indicate the start and end of image generation.
Vocabulary Embedding Processing Flow. For understanding samples, follow the LLaVA setting: discrete features are first extracted by the tokenizer and then fed into the LLM through an adapter layer. To ensure consistency between understanding and generation, we employ the same features for generation samples instead of using the embeddings of <IMG_X <IMG_XXXXX: XXXX>> After training, we replace the visual code embedding the vocabulary align training with inference. The detailed procedure is illustrated in Algorithm
Algorithm 1: Procedure: Training and Generation Pipeline
Before Training::
Add special tokens <IMG XXXXX> into LLM vocabulary 2. Preprocess training images, c.g. "generate dog <Image>" + "generate dog <start_of _e____maaa<<__ii>>> ix. .IGG [oock><<edd_o of image>"
During Training::
1. Feed preprocessed samples into LLM tokenizer > text id;
2. Lookup embedding V from text_id + Etenti
Get image IDs from text_id img_id;
Lookup unified codebook embedding from img id + Esngis Apply adapter layer: Esng E'iini'
Replace image embeddings in Etezt with Ei Elezti
Feed Flezt into LLM backbone for training;
Generation::
1. Apply adapter layer codebook embeddings Cong - Oaanii
2. Replace image part of V with Cl'g V Start autoregressive generation;
7.3 TOKENIZER EXPERIMENTAL DETAILS
The training of SemHiTok is conducted in two stages. During semantic codebook training, train the semantic tokenizer for one epoch on 50M subset of COYO-700M (Byoon et al. (022). For the PRE stage, we first train the model 50M COYO to improve its generalization, following Lla et al. 2024a). To further improve the reconstruction and generation performance, we enlarge the size of the pixel decoder( Recoder(VVTT-Lrgge LVig-Large) and fine-tune on the 20M COYO data and 20M MidJourney-styy synthetic data. The full training takes about days on v100 GPUs.
on ImageNet(Deng et al. 2209), and then fine-tune on
7.4 UNIFIED MLLM EXPERIMENTAL DETAILS
Following existing work Wu et al. 2024c Ma et al. 2025 we first pretrain the model and adapter layer mix of multimodal data, which is composed of 3.5M language data from Magpie et al.)
18

---

Published as a conference paper at ICLR 2026
2024b) and OpenorcaLLian al. 2023). 10M caption image-text pairs data, and 15M MidJoumey- style synthetic data. Subsequently, we finetune the model on 1M language dataset from Magpie(Xu et al. 2024b) and Evol-Instruct
al. 2024a) 4M generation data and 4M understanding data from emova Chen et al., 2024a) and MA-FVA-SFFLLI et al. 2023b). The full training takes about days 32 A800 GPUs. During image generation inference, apply bassifier-free guidance(Ho & Salimans 2022) with scale factor of 2.5.
7.5
VISUALIZATIONS UNDERSTANDING TASKS
We present more visualization of multimodal understanding samples in figure 6
there a red brick building in the image?
Does this artwork belong the type of landscape?
Is overcast?
Yes
Yes
No, is clear
What lanmark this? And where is
What is the relationship between two animals images?
Which of following captions best describes this image?
woman applying makeup mirror
Taj Agro, India
Symbiotic relationships
Mochu Picchu Peru
B. Predatory relationships
A person taking photo with camero
Windmills at Kinderdijk, Holland
Competitive relationships
group of people watching movie
D. The Great Chiness Woll in China
Parasitic relationships
A person reading book in library
C
Which animal in the picture is larger, the the dog?
To determine which animal is langer, can compare the size of the cat and the dog based their relative positions in the image. The cat appears to be standing upright with body fully visible, while the dog is crouched down, making it difficult to height, However, by comparing the proportions of their bodies they stand next to each other, it's clear that cat is taller than the dog. Therefore, the cat is larger than the dog this picture.
full
Figure 6: Visualizations understanding tasks.
7.6 MORE COMPARISON OF GENERATION ON GENEVAL AND DPG
To more fully demonstrate the superiority of SemHiTok, we conduct further comparisons with other generative models and unified models on GenEval Ghosh al. (2023) and DPG Hu et al. (2024). As shown in Tab 7 our method still demonstrates competitive performance.
7.7 VISUALIZATIONS GENERATION TASKS
We present more visualization of generated images in figure
19

---

Published as a conference paper at ICLR 2026
Table 7: Comparison with other methods on GenEval and DPG Bench. SemHiTok still achieves
competitive performance even with smaller amount of data.
WTraining Images
Model
Params
Type
Res.
GenEval
DPG
Only Gen.
SD v2.1
Rombach al. 2022
Diff
2000M
1024
0.50
DALL-E2
Kamesh al
2022
Diff
650M
1024
0.52
DALL-E
Betker et al.
2025
Diff
1024
0.67
83.50
PixArt-o (Chen et 2025
0.6B
Diff
0.48
SDXL (Püdell
2.6B
Diff
2000M
1024
0.55
74.65
Playgroundvz.5
2024c
Difl
1024
75.47
Gnn&Gen
Transfusion Zhou
3.5B
Ming-Lite- Lite-Uni
2025
8B
AR-Scale
5M
512
0.62
Show-o (Xee
al.
2024
1.5B
Diff
36M
256
0.53
67.27
Emu33 Wang el
2024c
8B
AR
256
0.66
80.60
Janus Wu ct
2024g
1.3B
AR
65M
384
0.61
SynerGen- VL
(L
202.5
2.4B
AR
25M
256
0.61
UniFork (Li
2025
0.76B
384
0.46
ILLUME Wang al.
2024a
7B
AR
15M
512
0.61
I.I.UMF+ Truaan el ain. 40A55
3B
AR
46M
512
0.72
MUSE-VL Xxe el al 2025
7B
AR
10M
256
0.57
QLIP-B ZZhao et al. 2025
1,5
AR
18M
78.17
TokenFlow (Qu
7B
AR
60M
256
0.63
73.38
MMMDA Yang ct 2025
8B
Diff
512
0.63
69.97
Liquid (Wi a1.
A024h
7B
AR
30M
512
0.55
83.45
UniTok Ma
202A
AR
30M
256
0.59
83.45
Tar-1.5B [Tan el al. 2025
Tar-1.5B W/Sell-Releec HHan al.
2025
AR
46M
256
0.78
84.10
Tar-7B (Han et al., 2025
7B
AR
46M
256
0.84
84.19
Tar-7B W/SetF-Keheer (Han al.
2025
7B
AR
46M
256
0.85
84.65
Ming-UniVissinn (Huang cral. 20256
16B-A3B
AR-Continuous
512
0.85
82.12
(emHrTTk((urr)
15M
0.71
83.59
Table 8: Quantitative evaluation of pixel consistency using Variance Reduction Ratio (VRR). SemHiToksen significantly outperforms the random baseline, verifying the intrinsic semantic-pixe correlation. Furthermore, our hierarchical refinement (SemHiTokpix) achieves the best consistency, exceeding the standard VQGAN baseline.
Experiment
Random Baseline
SemHiToksem
VQGAN
SemHiTokpix
VRR
0.185%#0.005%
5.245%:40.0854 9.535%:2000959
13.705%+4..75%%
7.8
MORE VISUALIZED RECONSTRUCTION RESULTS FROM THE ABLATION OF KEY MODULE
We show more reconstruction effects on the ablation of key modules in figure
7.9
QUANTITATIVE ANALYSIS OF PIXEL FEATURES IN THE SEMANTIC CODEBOOK
In Fig we provide visualization results showing that image patches corresponding the same semantic code share similar pixel features. To further support this analysis, we attempt to provide quantitative evidence using the following method:
1. Using the tokenizer to extract the code indices along with their corresponding image patches.
2. Applying the (DT(rrre Cosine Transform) |Roo & Yip 2014) to extract patch's 's pixel features.
3. For each code, we compute the variance the DCT features extracted from its associated image patches, followed by averaging these variances over all codes, denoted Vnean- The more similar the pixels of the tokenized results are, the smaller the V/ Viean becomes.
4. Computing the global variance of all image patches, denoted V Vglobal.
20

---

Published as a conference paper at ICLR 2026
M
Figure 7: More generated images presentation.
5. Computing Variance Reduction Ratio (VRR), denoted as V RR 1 Vmnen /aaon///gtbbd larger VRR indicates that the patches corresponding to the given code have more similar pixel features.
Vgiobal- The
We randomly sampled 10K images from the ImageNet- Val (50K) dataset across independent runs. The Random Baseline refers to randomly tokenized results, serving the lower bound for the VRR metric. VQGAN (Yu et al.) 2021a), as an expert model for pixel reconstruction, serves as the standard reference for the VRR metric.
21

---

Published as a conference paper at ICLR 2026
Sem
+ PRE
+ +ED
GT
Figure 8: More visualized reconstruction results from the ablation of key module.
As illustrated in Tab. the results provide strong quantitative evidence for the effectiveness of our Semantic-Guided Hierarchical Codebook (SGHC) design. Validation of Semantic- Pixel Correlation. The SemHiTok,em achieves VRR of 5. 5.245%, which is significantly higher than the Random Bascline (8.1..)) This quantitatively verifies our observation in Subsed2.2 that image patches assigned to the same semantic code naturally exhibit intrinsic pixel-level similarities. This correlation forms the theoretical foundation for using semantic codes to guide pixel quantization. 2hy Necessity of Pixel Branch. However, compared VQGAN (9.535%) the lower VRR of SemHiTokeen confirms that semantic codes alone are insufficient to capture fine-grained high- frequency details. This justifies the necessity of pixel branch design to bridge the gap between semantic abstraction and pixel fidelity. 3.) S.JSuporyy of Hierarchical Design. Most notably, SemHiTokpis achieves the highest VRR of 13.705%. surpassing the expert pixel reconstruction model VQGAN by a substantial margin (+4.17%). This result demonstrates that decomposing the complex global pixel space into semantic-conditioned sub-spaces allows for significantly tighter feature clustering and lower variance than standard global quantization methods. This explains why SemHiTok achieves superior reconstruction performance despite utilizing a decoupled training strategy.
7.10 IMPACT OF CONCAT TYPE AND SUB-CODEBOOK SIZE.
For efficiency, only conduct training and evaluation ImageNet-IK. IK. We investigate the impact of the concat type semantics between pixel and the sub-codebook size, on reconstruction performance
Table 9: Impact of Concat type and sub- codebook size. w/o sem: use the semantic discrete token. The gray bar represents the default setting in our ex- periments.
Table 10: Impact of VQ Type and dim of the semantic codebook. We evaluate multimodal understanding per- formance under the LLa setting. The gray bar represents the default setting our experiments.
VQ Type
Codebok Dim
Concat
Sube Size
rFID 1
Usaget
MME-P
MMB1
SEED
MMU1
w/o
12
1.99
95.4%
Norm
48
1249.6
52.2
$2.8
34,8
Len
12
1.45
94.1%
Vanilla
48
1319.9
56.3
57.2
33.1
Dim
1.42
96.4%
EMA
32
1385.9
58.3
61.2
35.1
Dim
12
1.26
93.7%
EMA
48
1387.5
61.3
62.3
35.6
Dim
16
1.19
79.3%
EMA
64
1428.7
60.9
62.5
35.5
22

---

Published as a conference paper at ICLR 2026
Table 12: Comparative exper- iments with Different K x m on reconstruction task.
Table 11: Ablation of Semantic Codebook K. We evaluate on multimodal understanding tasks.
POPE
MME-P
SEED
GQA
Usage
K
m
rFID
Usage 1
12
1.58
95.5
4096
79.7
1297.7
60.4
56.6
100
16
1.45
92.7
8192
8192
81.9
1343.0
61.7
59.7
99.7
24
1.31
89.8
1.22
83.7
16384
82.5
1355.8
62.9
60.3
99.0
32768
82.8
1364.2
62.5
60.6
93.0
1.42
96.4
16384
12
1.26
93.7
1.19
79.3
Table 13: Quantitative comparison of reconstruction quality under consistent total codebook sizes. SemHi'Tok outperforms existing quantization methods at the 65k scale and achieves comparable rFIl) to expert models at the 262k scale.
Method
Codebook Size
rFID.
Usaget
VQGAN-LC ZZu et al.
2024a
65546
2.63
100.0
VQGAN-LCCCLIP) (Zhu al. 2024a)
65546
2,40
100.0
FSQ Bai et al. 2024)
64000
2.80
100.0
LFQ el al.
2023
65536
2.88
100.0
Sim V
Zhu al. 2025)
65536
2.24
100.0
8192 x ((65536)
1.93
98.1
SemHiTok
16384x4(65536)
1.84
99.0
Sim VQ (Zhu et al. Open-MAGVIIZ Luo et al. 2024
2025)
262144
1.99
100.0
2621 144
1.17
100.0
IBQ Shi et al. 2024)
262144
1.00
79.3
8192x32(262144)
1,21
83.7
SemHiTok
16384x8(262144)
1.19
79.3
as shown Tab. For the Concat type, w/o semantic tokens leads to a significant drop in reconstruction quality, with score increase of 0.73 compared to the default setting, Furthermore, concatenation along the sequence length performs worse than along the dimension, as the two token sets are spatially aligned, making dimensional concatenation more appropriate. For sub-codebook size, increasing the size can improve the model's reconstruction performance, but it exhibits marginal utility. In addition, the codebook usage significantly decreases when the sub-codebook size set 16, which indicates that too large sub-codebook size is not cost-effective.
7.11 IMPACT OF SEMANITC VQ TYPE, CODEBOOK DIM AND CODEBOOK SIZE.
In Tab T0 we ablate VQ type and codebook dimension multimodal understanding. Specifically, Norm VQ is not suitable for semantic discretization and shows the worst performance in understanding tasks. This indicates that difficult discretely model complex and rich semantic information by Norm VQ. Replacing it with vanilla VQ brings a clear improvement. To further stabilize the semantic codebook, we use EMA VQ, which achieves the best results. For the semantic codebook dimension, higher values offer better representation but with marginal effect. We empirically set the dimension to 48 as the default.
Furthermore, investigate the impact of semantic codebook size K multimodal understanding in Table II The results indicate that while larger codebooks generally enhance performance, increasing K beyond 16,384 yields only marginal gains and leads noticeable decline in codebook usage. Therefore, we select K = 16, 384 as the optimal balance between representational capacity and utilization efficiency.
23

---

Published as a conference paper at ICLR 2026
7.12 IMPACT OF DIFFERENT K X AND MORE COMPARISON WITH OTHER LARGE CODEBOOK RECONSTRUCTION METHODS.
We investigate the impact of semantic codebook size K and sub-codebook size m in Table 12 The results indicate that increasing either K or m generally enhances reconstruction fidelity (lower rFID). However, excessive expansion leads to a significant decline in codebook usage. Furthermore, when controlling for the total codebook size (K X m., we observe that prioritizing a larger semantic codebook K yields slightly better performance than increasing the sub-codebook size m.
To further demonstrate the effectiveness of our method, we conduct more detailed comparison with other approaches that use large codebooks. As show in Tab[3]] we conduct comparisons on the reconstruction task under two codebook sizes (65K and 262K). Under the 65K setting, our model surpasses the expert reconstruction model while maintaining good usage. Under the 262K setting, method still achieves competitive performance compared to the expert model.
24


Reviews:
[{"review_id": "xSwWglMJwK", "reviewer": "Reviewer_YG9D", "summary": "This paper introduces SemHiTok, a unified image tokenizer designed to effectively capture both high-level semantic features for understanding and low-level pixel details for generation. The authors identify a key challenge in joint training: the inherent conflict between the feature priorities of multimodal understanding and generation tasks. To bridge this gap, they propose a novel Semantic-Guided Hierarchical Codebook (SGHC), which employs a set of sub-codebooks to model the pixel-level space under the guidance of each semantic code. A notable advantage of SemHiTok is its compatibility with existing next-token-prediction-based MLLMs, achieved through a straightforward codebook flattening operation.", "questions": "* Does the authors mention the specific value of K and m for the SemHiTok's codebook? Have you ever carried out ablation of different value sets of (K, m) and their effectiveness? It could bring more insights if the authors could provide more details about how they decide the scale of codebook.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is well-structured and clearly written, making it a pleasure to read. The core idea of SemHiTok is both novel and compelling, presenting a fresh perspective on the problem. Consequently, I have no major questions regarding the technical content presented. Should I have overlooked any aspect of the work, I welcome the authors to clarify it in their rebuttal.", "weaknesses": "Although this paper proposes an alternative method to bridge the gap between low-level visual cues and high-level semantic features, I still have several concerns.\n\nFirst, the hierarchical structure significantly increases the codebook size. As shown in Table 1, SemHiTok has a total size of K × M = 196,000. It appears impractical to expand this codebook further. However, this already large size limits the value of K. I wonder whether a small K is sufficient to represent the full diversity of semantics in visual content.\n\nSecond, the comparisons regarding codebook capacity do not seem entirely fair. As shown in the table, methods like FQGAN and IBQ achieve better reconstruction quality with the same resolution and codebook dimension, yet have a smaller codebook size. This raises a question: given abundant data (far beyond ImageNet-50K), would simply using a large enough standard codebook be sufficient, thereby diminishing the necessity of the proposed SemHiTok? In other words, can SemHiTok maintain its competitiveness under such conditions? For instance, the EMU series (e.g., the recently released EMU-3.5) employs a very large visual codebook and massive pre-training data, which leads to strong performance.\n\nI would appreciate it if the authors could address these concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "esJLASVS52", "reviewer": "Reviewer_JSAk", "summary": "This paper proposes SemHiTok, a unified image tokenizer enhanced by semantic information guidance. It innovatively introduces a hierarchical codebook structure, which builds a pixel sub-codebook based on a pre-trained semantic codebook. The semantic part and the pixel part are trained separately to decouple the structure and training strategy. This enables the tokenizer to capture pixel features while retaining its ability to comprehend high-level semantic information. Additionally, SemHiTok is applied to the MLLM structure, and the experimental results demonstrate the performance of this method.", "questions": "- Typo: In the header of Table 5, \"Ehance\" should be corrected to \"Enhance\".\n- How is the size m of the sub-codebook determined, and does it impact the model's performance?\n- Is it necessary to retrain the entire LLM to verify SemHiTok's capabilities? Perhaps one can only replace the Tokenizer in the existing MLLM (Multimodal Large Language Model) structure. Is this new MLLM architecture necessary?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The experiments are comprehensive, comparing with multiple state-of-the-art (SOTA) models and demonstrating the superiority of the SemHiTok method.\n- The writing language is accessible, and the diagrams are clear.\n- As a Tokenizer, it underwent complete training on the MLLM architecture, validating its effectiveness.", "weaknesses": "- The paper only conducted ablation experiments on the joint training vs. phased training of the SemHiTok architecture, but did not compare joint training and phased training across other methods. Thus, the conclusion that \"Joint training degrades performance\" lacks sufficient support.\n- The roles of the sub-codebook and phased training have not been ablated individually in the experiments, making it insufficient to demonstrate the degree of validity of each component on its own.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DBYUlwoWjK", "reviewer": "Reviewer_HDbp", "summary": "The paper proposes SemHiTok, an image tokenizer for unified MLLMs, which provides both semantic features and pixel-level features for multimodal understanding and generation tasks. The tokenizer features a hierarchical architecture, where an image is first quantized into semantic codes and then the pixel-level tokens are selected based on the corresponding pixel sub-codebook. Experiments demonstrate the effectiveness of SemHiTok on both image understanding & generation tasks.", "questions": "About the observation that image patches with the same semantic code tend to have similar pixel feature. Are there any quantitative verification (e.g., specific metrics or analysis) for this observation? This would better solidify the motivation for using the semantic code to index pixel sub-codebooks", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The concept of the hierarchical codebook for high-level semantic features and low-level pixel features is novel, and well-motivated. The method decouples the optimization of the two conflicting objectives for semantics and pixel-level details.\n* There are comprehensive experiments which effectively demonstrate SemHiTok's strong performance both at the tokenizer level and in its application on a unified MLLM.", "weaknesses": "* While the selection of pixel sub-codebooks is guided by the semantic codes, the training of the pixel-branch is decoupled, so the link between the two hierarchical levels is weak. The authors argue that SemHiTok is superior to naively combining two separately trained tokenizers (in Table 6 Exp 3 & 4), but the conceptual difference feels incremental. Perhaps the authors should provide a more explicit discussion on the specific advantages of this hierarchical design over a simpler, two-stage concatenation approach.\n\n* Though the authors state that SemHiTok avoids codebook overexpansion, the codebook size of SemHiTok is still large (196608), much larger than baseline methods. So the comparison of performances may not be fair, since a much larger codebook inherently allows for higher-fidelity reconstruction.\n* There are several grammatical errors, and maybe some mistakes in the formulations (Eq 2, 5).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GoLlidpVcq", "reviewer": "Reviewer_bysp", "summary": "This paper introduces a unified tokenzier for both image understanding and generation. By linking a set of pixel code with a semantic code, the proposed SemHiTok makes a good trade-off between semantic image understanding and pixel image generation. Experiments on image reconstruction, understanding and generation confirm its effectiveness.", "questions": "no.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation and illustration of the proposed method are clear and easy to follow.\n- Experiment results are good on multiple tasks, including image reconstruction, understanding and generation.\n- The ablation experiments are clear to demonstrate the effect of each component.", "weaknesses": "- Image reconstruction. The rFID is not enough to prove the real performance of a tokenizer for image reconstruction. Other metrics like PSNR are encouraged. Besides, unified tokenizers like UniTok, MUSE-VL should be compared in Tab.1.\n- Training setting. Most unified models are jointly trained on a mixture of multimodal understanding and text-to-image data, the proposed method are only trained on LLaVA-1.5 and text-to-image settings seperately, which may not reflect the relation between image understanding and generation under a unified model.\n- Generation benchmarks. Recent popular benchmarks such as Gen-Eval and DPG are missing. Besides, it's better to include more recent methods on Tab.4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)