Towards Semantic Equivalence of Tokenization in Multimodal LLM
Authors: Shengqiong Wu
Hao Fei
Xiangtai Li
Jiayi Ji
Hanwang Zhang
Tat-Seng Chua
Shuicheng YAN
Venue: ICLR 2025 Poster (2025)
Keywords: MLLM
Vision-Language Understaning
Vision Generation
Vision Editing
Vision Tokenization
Primary area: generative models
TL;DR: We propose a Semantic-Equivalent Vision Vision Tokenizer to achieve fine-grained semantic alignment between visual and text tokens, facilitating to improve various vision-language tasks.

Abstract: Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in processing vision-language tasks. One of the crux of MLLMs lies in vision tokenization, which involves efficiently transforming input visual signals into feature representations that are most beneficial for LLMs. However, existing vision tokenizers, essential for semantic alignment between vision and language, remain problematic. Existing methods aggressively fragment visual input, corrupting the visual semantic integrity. To address this, this paper proposes a novel dynamic Semantic-Equivalent Vision Tokenizer (SeTok), which groups visual features into semantic units via a dynamic clustering algorithm, flexibly determining the number of tokens based on image complexity. The resulting vision tokens effectively preserve semantic integrity and capture both low-frequency and high-frequency visual features. The proposed MLLM (Setokim) equipped with SeTok significantly demonstrates superior performance across various tasks, as evidenced by our experimental results.

Published as a conference paper at ICLR 2025
TOWARDS SEMANTIC EQUIVALENCE OF TOKENIZA- TION IN MULTIMODAL LLM
Shengqiong Wu'n, Hao Fei't, Xiangtai Li", Jiayi Ji', Hanwang Zhang', Tat-Seng Chua', Shuicheng YAN4.1
National University of Singapore, "ByteDance Seed,
Nanyang Technological University, *Skywork AI
swulu. nus edu,, (haofei37, yansc, dcscts edu. sg, {i (jjyxmu, xiangtai 94 8 com, hanwangzhanggntu edu..g
ABSTRACT
Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in processing vision-language tasks. One of the crux of MLLMs lies in vision tokenization, which involves efficiently transforming input visual signals into feature representations that most beneficial for LLMs. However, existing vision tokenizers, essential for semantic alignment between vision and language, remain problematic. Existing methods aggressively fragment visual input, corrupting the visual semantic integrity. To address this, this work presents a novel dynamic Sonantic-Equivalen Vision Tokenizer (SeTook,, which groups visual features into semantic units via dynamic clustering algorithm, flexibly determining the number of tokens based on image complexity. The resulting vision tokens effectively preserve semantic integrity and capture both low-frequency and high-frequency visual features. The proposed MILM (SETOKIM) equipped with ScTok significantly demonstrates superior performance across various tasks, evidenced by our experimental results. The project page is Intpp::////// //sqwu ttop/ C.op/SeTok-web/
INTRODUCTION
Recently, the research on MLLMs has garnered intense interest (Zhang et al., 2024a Lin et al. 2023 Dong al., 2024 Wu et al. 2024a). By building upon the unprecedented intelligence of language-based LLMs (Chiang et al. 2023 Touvron et al. (023a), and integrating multimodal encoders (Radford al. 2021 at the input side and decoders (Rombach et al. 2022) the output side, current MLLMs have developed powerful multimodal capabilities. Particularly. in the visual modality, the state-of-the-arr (SoTA) MLLMs have now achieved a grand slam across the four major visual- visual-language task groups, i.e., understanding (Liu et al.) 2024b Wu et al. (222cc Team 2024). generating (Ge et al. 2023 Dong et 2024 Jin et al.) 2024b Pan
(024), segmenting (Ren
et al. [2244) You et al. 2023), and editing (Huang et al. [024 Un et al. [202bb] Fu et al. 2024).
Central to this capability the design of vision tokenization (Dosovitskiy et al.]|0021] Esser et al..
2021: Yu et al. 2024), which focuses on effectively converting input visual signals into visual tokens that can be seamlessly understood by LLMs. Existing vision tokenizers primarily produce three types of visual tokens: 1) patch-level continuous tokens (cf. Figure [a), 2) patch-level discrete tokens (cf.
Figure I [b)), and 3) learnable query tokens (cf. Figure (c)..
While existing MLLMs have achieved promising performances across various tasks, significant bottleneck remains with current visual tokenization methods, i.e., resulting in insufficient semantic alignments between language and vision tokens. On the language side, linguistic tokens (or words) are naturally discrete, representing well-encapsulated semantic units, whereas, on the vision side, visual pixels are inherently continuous data with no physical boundaries. Intuitively, language tokens should correspond to semantically encapsulated objects (or compositional regions) within an image. For example, when "a dog" mentioned, the "dog" token should correspond to the direct pixel
Work done when Shenggiong Wu is an intern at Skywork AI.
1 CCorrsponning Author.

---

Published as a conference paper at ICLR 2025
Visual Ouiput
Visual Output
Visual Dutput
Visual Outpur
Image
Decoder
Decoder
Image
Decoder
Text Image
Décoder
Visual
Visual Encoder
Visual Encoder
Visual Encoder
Encoder
LLM
LLM
LLM
LLM
MLPiLinear
Quantization
Resımplor
Text
Leamalie
(a) Patch-level contimacus token (e.g. Emu-2, DeammLLM) )
(b) Patch-leve discrete taken (e.g.. Unified-10 I0, CM3Leon)
(c) Leamablee query token (e.g.. MiniGPT-5)
(d) Semantic semaanic-quiivllntt contimuuu token (Ours)
QdWbbat are letler appearing inage??
brown field single large
Understanding
Gonenation
Itag*
lelles appeess several Ligee,, and word slenty apgents image.
Longuage-driven Vision Tosks
Graassad z0age
Tepet
The boclemmand hill
Remove the beot.
Visian Segmentation
Vision Editing
LOUEE
Lazz smage
Vodon mek
Bttndingg
(e) Four language driven vision lask clusters
Figure 1: Comparison between existing MLLMs in tokenized visual inputs: (a) patch-level continuous token, (b) patch-level discrete token, (c) learnable query token, and (d) semantic-equivalent continuous token (ours). In (e), we show four language ddiienn--venn vision tasks enhanced with semantic- semantic-equivalen equivalent vision tokens, with token masks showing regions of the same color representing single vision token.
region of the dog the image. However, as illustrated in Figure Ta&b), both existing tokenization methods divide the image into fixed patch squares, fragmenting objects across multiple patches. This disrupts the integrity of visual semantic units, resulting significant loss high-frequency visual information (Zhang et al. 2023 e.g., the object's edges and contours. Moreover, methods employing a fixed number of query tokens, as depicted in (igure[[[c) struggle to align with actual visual semantic units and meanwhile offer limited interpretability (Yang al.) 2022)) Wu et al. (022b)) Ultimately, this misalignment between vision and language within MLLMs undermines the effective understanding of visual signals, significantly hindering progress in range of vision- vision-language tasks that require precise, fine-grained semantic alignment between vision and language elements.
To this end, this work proposes a Semantic- Equivale- Tokenizer (SeTok) for enhancing MILMs. where we encourage the vision and language tokens to be semantically congruent. The core idea involves automatically grouping visual features from input images by applying clustering algorithm (Engelckee et al., 2021), such that each unique cluster represents an encapsulated semantic unit within the vision. As illustrated in Figure [)), the red visual area aggregated by ScTok corresponds to complete semantic amccptt"persort while the yellow area corresponds the "surface board" concept. Furthermore, we recognize that tokenizing images into a fixed number patches is impractical. From semantic perspective, different images should contain varying numbers of semantically encapsulated objects, and the granularity of compositional regions also needs to be flexibly determined. For example, only need identify person in the image, while other times, we may need to delineate the person's head precisely. This implies that is more reasonable dynamically determine the division of visual tokenization. To address this, we propose a dynamic clustering mechanism (Engelcke et al. (02T)) that iteratively determines cluster centers based on density peaks, assigning visual features these centers until all features are allocated. The design of this mechanism allows for the dynamic determination of the number of concept visual tokens, rather than fixing the ratio (Jin et al., 2024a) or merely merging the top-k visual tokens (Bolya et al. 2023). After clustering, devise token merger aggregate the visual features within each cluster, that dedicated to learning a complete visual semantic unit feature. including both high frebffrreunccc and low-frequency information. To enable the effective learning of the mmmantic-equivaleee token, propose reconstructing the raw image based on these tokens, and further introducing the concept-level image-text contrastive loss explicitly align the language and vision the concept level.
2

---

Published as a conference paper at ICLR 2025
photo ef man
Contrastive
Text Encoder
photo of beach
Loss C...
photo of sea
POS
Inter Cluster
Inner-Cluster
Traindoomee Layer
Croes Attertion
Fied FasdFaare Forward
Reed Porwandd
Self-
Vision Encoder
Mask
Decodia
Plora
Token
Reconstructions
Semantic-
Ingut Images
Token Cluster
Token Merger Equivalent Token
Semantic-Equivalent Tokenizer (Sk)
Detokenizer
Overview of SeTok. SeTok tokenizes visual features extracted from image by vision encoder into semantically equivalent vision tokens, which then are fed into detokenizer to reconstruct the image and meanwhile employed to perform the concept-level image-text alignment.
Figure 2:
Built pre-trained LLM (Touvron al. 2023b), SETOKIM performs reasoning multimodal sequence concatenating text token with visual tokens generated by Setok. During inference, SETOKIM yields the text and visual tokens autoregressively, which then processed by a visual detokenizer and mask decoder to produce images and corresponding masks. Inspired by [i] al. (2024a), we introduce unified autoregressive training objective for optimization through pre- training and instruction-tuning on the massive multimodal data. We evaluate SETOKIM on various common visual language tasks, including visual question-answering image generation & editing, and referring segmentation. Our results reveal that comaaticc-qqivallee tokenization significantly enhances vision-language learning compared to standard patch-level tokenization learnable queries, achieving higher performance various tasks. Meanwhile, in-depth analyses and visualizations intuitively show SeTok enjoys the superiority of tokenizing vision input into more interpretable and semantic-complete vision tokens, achieving fine-grained vision-language alignment.
unified
METHODOLOGY
In this work, we aim to generate semantically complete vision tokens aligned with text tokens to facilitate fine-grained semantic interactions between vision and language, thereby enhancing the performance of MILMs in various multimodal tasks. In pursuit of this goal, we propose constructing semantic-equivalent tokenizer, called SeTok, which tokenizes the given input image into sequence of semantically complete visual tokens, as illustrated in Figure 2 Integrated with the SeTok, we further design multimodal large language model, i.c., SETOKIM shown in Figure where the semantic-equivalee vision tokens concatenated with text tokens are fed into LLMs for interleaved image-text understanding and generation.
sequue
2.1
SEMANNIC-RQUIVALEENT VISION TOKENIZER
Given an input image Hx we first employ vision encoder to extract a sequence of visual patch embeddings X { pxxxxd where d is the embedding dimension IT Then, to obtain semantically complete visual tokens, we propose to amalgamate the visual embeddings into concept-like scene components by Token Cluster.
Token Cluster. We take the visual patch embeddings X as input and then assign individual patches into a semantic complete cluster, which can be formulated as obtaining a variable number concept masks M E [0, 1|1xwxC with E. Maj.c where C' is the number of semantic- eqqiivic- tokens. Inspired by Engelcke et al. (202T). this intuitively achieved by (1) selecting the location (i, of the visual patch feature that has not yet been assigned to a cluster, (2) creating a cluster assignment according to the distance of the embeddings the selected location all other embeddings according distance kernel yp-) and (3) repeating the first two steps until all visual embeddings are accounted for or a stopping criterion is met. Different from (Du al. 2016) employing uniformed seed scores performing the stochastic selection of visual embeddings, we propose to choose the visual embeddings based on their density
for all patch coordinate tuples (i.j) j) in image,
where p is the patch size. Similarly, p denotes the downsampling factor when using CNN-based encoder. In this work, we employ ypu,, v) = exp( v1 -C Ch12
'When using ViT-based vision encoder,
3

---

Published as a conference paper at ICLR 2025
peaks, as higher density shows higher potential to be the cluster center. Specifically, first calculate the local density Pi,j of the token zi E X by referring its neighbors:
PL,
exp(
1
Σ
(1)
K
m (KNN
where KNN( X) denotes the -nnearest neighbors of æij in X. We then measure the minimal distance 6r, between the feature æi and other features with higher density:
minm,,spm... mipm.n. (f Big); if m,n n Pm.n > Pi.j 8 maxm., "p(mman Ri.j);
(2)
otherwise
Finally, we summarize the score 8i,j of the feature by combining the local density Pt,j and minimal
distance di,j Pijj X ôi.j. Based on the score, we select the location (i. j) of the visual feature that has the highest score ši,j and has not yet been assigned to a cluster and iteratively assign the visual feature into certain cluster until stopping condition is satisfied, at which point the additional mask added for any remaining visual embeddings. The detailed algorithm is described in Appendix :
Token Merger. After clustering, the visual embeddings grouped based on the attention masks M. To optimally retain information within each cluster, we adopt a token merger that aggregates visual embeddings beyond merely using cluster centers as definitive vision tokens. In addition, considering the significance of positional information for representing a semantic concept in an image, we integrate 2D position embeddings (PE, Heo al. (2024)) into the merger, calculated as X, = PE(X) @ M, ① X ④ Mc- Then, we apply I,ineers Transformer layers on all the visual embeddings within a cluster, followed by an average pooling to obtain the final token feature ur Avg( Trransformee(A... LimBer=) E Rdd To facilitate the representation of coherent scenes with semantic equivalent tokens, we add inter-cluster Transformer layers model relationships between vision tokens, i.c., V = [V11 vc} Transformer((us, (u uc) inter 5) RCxx
SeTok Training. To facilitate diverse visual understanding and generation tasks when building MLLMs, argue that effective semantic-equivales tokens should embody two key attributes: complete and enriched high-level semantic information, and undistorted pixel-level details. Therefore, we propose include concept-level image-text contrastive loss and image reconstruction loss, as shown in Figure 2 During the training phase, to ensure each token's semantic independence and completeness, we adopt concept-level image-text conservative loss, inspired by Xu al. (2022). This loss aligns visual tokens with corresponding textual concepts semantically, thereby enhancing their suitability for integration in LLMss. Additionally, to ensure tokens retain adequate pixel-level details, we feed these tokens into detokenizer (Yu et al. (2024) to reconstruct the original image and calculate the reconstruction loss. Finally, we employ a weighted sum combine the contrastive loss and reconstruction loss, optimizing both semantic fidelity and visual detail retention:
Leetak aLrre + B
In practice, and 3 are set to 1. We use ImageNet-1K: (Deng et al. 2009) for reconstruction learning and Openlmages (Kuznetsova et 2020 for both reconstruction and alignment learning.
2.2 SETOKIM
Upon acquiring SeTok, we propose to integrate with the pre-trained LLM to construct MLLM, i.e., SE- TOKIM. The overall framework is depicted in Figure The input im- age will be tokenized into a sequence of semantic-equivalent visual tokens by SeTok, which are then concate- nated with text tokens to form a uni- fied multimodal sequence. To effec- tively distinguish between modalities and facilitate visual content genera- tion, two special tokens, [[mq]] and "[/ Img]" are introduced to signify the beginning and the end of the vi- sual sequence, respectively. The backbone LLM subsequently processes this multimodal sequence
The man is standing the beach hold- ing a sortb
Mask Decoder
Visual Detokenizer
Text Decoder
LLM
ling
Mmmpl
Text Encoder
SeTok
Input
Input Image
Figure 3:
The overview of SETOKIM.
4

---

Published as a conference paper at ICLR 2025
Method
Size
Vis. Tok.
Plickr30K
VQA
OK-VQA
GQA
POPE
MME
Me-VVe
InstructBI.IP
2023a
13B
49.5
78.9
1212.8
Owen-VL-Chrn
(3m
7B
QCC
78.2
$7.5*
1487.5
Emu
Sun ct al.
2024b
7B
Q.C
77.4
57.2
43.4
DreamLLM Dong ct
2024
7B
P.C
72.9
41.8
36.6
LLaVA-1.5 Ju el al.
2024bb
7B
P.C
78.5*
62.0
85,9
1510.7
33.1
NEXT-GPT Wu al
20246
7B
P.C
84.5
66.7
$2.1
SEED-X (Ge aL 2024
17B
PC
52.3
47.9
84.2
1435.7
LaVIT [Jin
2024h
7B
P.C
83.0
66.0
54.6
46.8
Unified-10-2
(2024)
6.8B
P.D
79.4*
87.7
CM3Leon Yu al. 2023a
7B
23.8
Chamelcon Team
34B
74.7
SETOKIM
7B
SEC
86.9
78.5*
60.2
65.6
89.1
1537.8
45.2
Table 1: Comparison of MLLMS on image understanding benchmarks. indicates the training sets observed during training. "C" and "D" represent continuous and discrete visual tokens, respectively. "P" refers to patch-level features, "Q" denotes learnable queries, and "SE" is semantic-equivalent.
perform multimodal understanding and generation. The output vision tokens are then fed into the visual detokenizer to restore the images. Meanwhile, we observe that the generated concept-centric tokens inherently embed approximate locations of each concept within the original image, as illus- trated in 6 To exploit this spatial and semantic encoding,
incorporate lightweight mask decoder (Kirillov et al. 2023) utilizing the generated vision tokens as input to yield the referring mask. The detailed implementations are provided in the Appendix !
Training Objectives. To facilitate autoregressive modeling across both text and visual generation, we unified adopt next-token prediction:
p(y yn.) p(un 11. 91, .3))
(4)
Specifically, in terms of text generation, we adopt the Cross-entropp loss Lterts i.e., the standard language modeling objective, to maximize the likelihood of text tokens. For image generation, drawing inspiration from Li et al. (2024a), we utilize the LLM, to produce a conditioning vector based on previous tokens: 2; LLM
-/1-1) ). We then model the probability of the next token by p(y (z)). and employ the Diffusion loss (Li ct al. 2024a), denoted Luie. to optimize the parameters of LLM. Morcover, we follow Li ct al. (2024c) to use binary cross-entropy loss Lice and
dice loss Ldice for optimizing the parameters in mask decoder.
Training Receipts Given that we try to perform generation and understanding with one single generative model, large-scale pretraining is required to achieve effective alignment between textual and visual content. To this end, we propose a two-stage training procedure. Stage-I: Multimodal Pretraining. In this stage, we focus on enhancing the aligument between text and image. We employ massive multimodal data, including ImageNet- 1K and 28M text-image pair dataset, to train model for conditional image generation and image captioning. Furthermore, we utilize the English text corpus from the SlimPajama (Soboleva et al., (023) dataset to reduce catastrophic forgetting of the reasoning capacity during LLM training. Toward the end of the first phase of training, once the trainable modules of SETOKIM have converged, we freeze these modules and exclusively train the mask decoder on the segmentation datasets, like MSCOCO (Lin et al. 2014)) to promote the learning of fine-gratd tne-grained object boundaries. Stage-II: Instruction Tuning. Building upon the pre- trained weights, we further perform multimodal instruction tuning with both public datasets covering multimodal instruction datasets (e-g.. ALLaVA (Chen et al. 2024) and LLL (Liu al.
2023ch ). fine-grained visual QA (e.g.. VQA 22 (Goyal et al. 2019). GQA (Hudson & Manning 2019). OK-VQA (Marino et al. 2019), A-OKVQA (Schwenk et al.) 2022)), image generation (c.g.. LAION- (Schuhmann al. 2022 ), and editing (c.g., InstructPix2Pix (Brooks al. 2023
and Magicbrush (Zhang et al. 2024c More details can be found the Appendix DD3
SETTINGS
Our experiments employ the LLaMA-2-7B Touvron et al. 2023b) to initialize our LLM backbone. For the SeTok, we apply pre-trained SigLIP- 104000-ppater 1aatch11-33 384 (Zhai et al.) (023) as our vision 12, and 8,
encoder, and the numbers of inner-cluster and inter-cluster transformer layers respectively. The dimension of the cemantic-equivalers token 512. For the detokenizer, we adopt
T, = 12 blocks for mask tokens query visual information contained in the semantic-equivalent
5

---

Published as a conference paper at ICLR 2025
MS-COCO
MagicBrush
MASK
EVR
Method
Type
Decoder
FID
CLIPP 1. 4 LPIPS) 1, +. CL.P m
Diffusion- Method
Muke-A-Scene (Gafni ct al.
2022
Autoregressive
11.8
Ins. Ppp.
[Brooks et al.
2023
Diffusion
83,4
12.1
35.9
17.6
81.4
18.9
SD v2.1
Rombach
2022
Diffusion
9.0
MhsMM-based Method
DreamLLM Dong el al
2024
LLM (Cont.)
SDv2.1
8.7
SEED-X Go ct al.
2023
LLM (Cont.)
SDXL
14.9
CM3Lcon Yuu
2023a
LLM (Disc.)
VOGAN
10.3
LaVIT Jin
2024b
LLM (Disc.)
SDv1.5
7.4
81.1
25.3
36.9
25.1
73.8
26.8
LWM et
2024al
LLM (Disc.)
VQGAN
12.6
MGIE ef al
2024
LLM (Cont.)
SDv1.5
91.1
8.2
29.8
13.3
81.7
16.3
Emu-2-gen" (Sun et al.
2024a
LLM (Cont.)
SDXL
85.7
19.9
28.4
20.5
80.3
22.8
Morph-Token" (Pan
2024
LLM (Cont.)
VQGAN
87.9
7.6
27.9
14.6
82.6
15.3
SKTOKIM
LLM (Cont.)
SeTok
8.5
89.6
6.9
26.4
83.5
14.1
Table 2: Performance of various models in zero-shot text-to-image generation and editing on bench- marks. #: denotes editing performances sourced from (Pan et al. 2024). "LLM (Cont.)" means LLM outputs continuous representation utilized in the decoder generate images, while "LLM (Disc.)" stands for discrete representation generated for image generation.
tokens, and then an upsampler inspired by the architect of Yu et al. (2024) is employed as the pixel decoder. More implementation details are provided in Appendix !D.1
For examining visual understanding ability, we evaluate our model on Flicker30K Young et al. 2014), VQA Gooal al. 2019), GQA (Hudson & Manning, 2019), OK-VQA (Marino al. 2019), as well as three MLLM benchmarks, c.g., POPE (Li et al. 2023). MME (Fu et al. 2023) and Vet (Yu al. 2023b). Besides, evaluate the visual generation fidelity the MSCOCO Lin et al. 2014) dataset. Following Pan et al. (2024). we evaluate the image editing capabilities of the SEKTOIM on Magicbrush (Zhang al. [202cc). EVR (Tan al.) 2019) and MA5K (Shi et al. 2021). Furthermore, refCOCOg (Mao et al., 2016), refCOCO+ Yu et al. (016), and Reaseg (Lai et al. 2024) are utilized to examine the potential referring segmentation capabilities of the proposed model.
EXPERIMENTAL RESULTS
4.1
MAIN RESULTS
The Quality of SeTok We employ reconstruction FID (rFID) and Top-1
Model
#Tokens
Latent size
Top-1
accuracy for image classification on
VQ-GAN (Esse
2021
Fixed
x
7.94
ImageNet to measure the reconstruc- tion and text alignment capabilities of the Se in Table SeTok can achieve comparable reconstruction quality to well-trained VQ models. Unlike prior methods that typically utilize 2D latent grids preserving spatial mappings between latent to- kens and image patches, which al- lows for the retention of precise low-level information but limits high-level semantic acquisition and development of more compressed latent space, SeTok integrates both high- and low-level information that is crucial for producing high-quality images and creating semantic compact and complete latent representations. In comparison, the latest models like TiTok utilize a fixed number of ID latent representations that suffer from a lack of semantic interpretability and poor textual alignment, i.c., obtaining inferior image classification performance (72.6 75.4 top-1 accuracy). We visualize the visual token in Section 4.2 and more reconstruction examples can be found in Appendix E
VAE (Rombach
Fixed
32 x 32
2,63
RQ- VAF. (Le
Fixed
16 x 16
3.20
VIT-VQGAN Yu al
2022
Fixed
32 x 32
1.28
MQ-VAE (Huangg el al
2025
Fixed
32 x 32
5.29
TiTok Yu ctal. 2024
Fixed
32 x
2,21
72.6
SeTokk
Dynamic
2.07
75.4
Table Reconstruction results (rFID) and image classification
performance (Top-1 Accuracy) on 256 x 256 ImageNet(val.) dataset. #Tokens refers to the number of tokens.
Visual Understanding. We evaluate the visual understanding capabilities of our model and other leading MLLMs across wide range of benchmarks, as detailed in Table Different from the prevalent use patch-level continuous visual tokens by foundational models like CLIP, the discrete tokens utilized in VQGAN models show weaker semantic alignment with text, which detracts from

---

Published as a conference paper at ICLR 2025
refCOCOg
refCOCO+
Reaseg
Method
Mechanism Hard-clustering
#Tokens
TELOPs
Flickr30K OK-VQA
val(U) les((U) val lestA tesiB gloU cloU
25"
8.3
86,9
60.2
ReLA
65.0
66.0 66.0
71.0 57.7
Soft-custering
23*
8.2
86.7
58.9
SEEM
65.7
24.3 18.7
256
15.7
85.1
51.7
PixeILM
69.3
70.5
66.3
71.7
58.3
64
13.9
84.1
53.6
Fixed
NExT-Chat
67.0
67.0
65.1
71.9
56.7
32
10.1
83.4
51.1
LISA
67.9
70.6
65.1
70.8
58.1 47.3
48.4
8
8.0
82.1
50.1
SETOKIM
71.3
71.3 68.0 72.4 61.2
50.7
52.7
Table 5:
The effect of different clustering strate- gies. The first three rows consist of dynamic strate- gies. #Tokens is the number of tokens, and * de- notes the average token number.
Table 4: Results on referring expression segmentation benchmarks. We report cloU for RefCOCO+/g.
ImageNet ((FIDL)
Flickr30K (CIDErt)
VQA-2
GQA
MSCOCO (FIDL)
Method
(Accuracyf)
(Accuracy1)
SeTok
2.07
86.9
78.5
65.6
8,5
w/o Leite
4.15
78.1
65.8
49.7
9.6
w/o PF
3,56
86.1
76.2
61.44
12.8
w/o inter-cluster Transformer
7.91
82.7
71.4
54.2
13.9
w/o inner-cluster Transformer
6.25
85.4
73.7
53.4
11.0
w/o Token Merger
8.64
80.3
66.1
50.5
14.7
Table 6: Ablation Study on SeTok to image reconstruction, visual understanding, and generation.
their performance in various understanding tasks. Besides, learnable continuous queries transformed via Q-former or cross-attention framework are introduced to alleviate the efficiency issues. However, these methods still struggle with fine-grained semantic alignment with text, potentially limiting the depth of interaction between textual and visual content. By incorporating semantic- semantic-equivalent tokens via SeTok,, our model secures competitive performances in various vision-understanding tasks. Moreover, our model demonstrates performance improvement on GQA by 3.6%, highlighting our method's superior capability in complex relationships and object quantities reasoning.
Visual Generation and Editing. Table 2 demonstrates comparative analysis of SETOKIM and other iiffusion-based and LLM-based methods in vision generation and editing. Notably, compared to other MILMs integrated with advanced vision decoders such as SD v2.1 (Rombach et al. (002) and SD-XI. (Podell et al. 2024), our method achieves comparable performance on complex prompts. This highlights the effectiveness and efficiency of ScTok in learning the correlations between visual and textual modalities within our unified framework. Further evaluations on instruction-bart editing are conducted. Standard pixel difference (L1),, LPIPS (Zhang et al., 2018)) and visual feature similarity (CLIP,m) im) are employed as metrics. Our model exhibits marked superiority in L1 and CLIP scores compared existing MLLMs. This enhanced performance can be attributed SeTok's ability to capture semantically equivalent visual tokens, thereby enhancing the semantic interaction between text and images. Moreover, editing tasks typically involve conceptual replacements within images, and the concept-level token representations learned by our model are inherently well-suited to such tasks involving straightforward replacements or modifications.
image
Referring Expression Segmentation. Table presents MLLMs' performances referring expres- sion segmentation tasks. Our model consistently outperforms the current SoTA on the RefCOCO+/g and ReaSeg dataset, demonstrating the proficiency of our vision tokens derived from SeTok in capturing not only object-centric semantic details but also the high-frequency boundary information.
4.2 IN-DEPTH ANALYSIS AND QUALITATIVE EVALUATION
Ablation Study. Table summarizes the results of an ablation study evaluating the design benefits of SeTok and the influence of SETOKIM across various vision-languag tasks. Firstly, observe that while the model can achieve commendable reconstruction quality without using contrastive loss, its performance markedly decreases in downstream vision understanding tasks. This suggests that exclusive reliance on reconstruction learning may cause the model to prioritize low-level information the expense of high-level semantic insights. Furthermore, replacing the token merger with simple average visual representation for each cluster also results in significant decline in fine-grained visual understanding and generation performance, possibly due the averaging process potentially leading to information loss. Lastly, the removal of positional encoding (PE) and both the inner-cluster and inter-cluster transformers degrade the model's performance across various tasks to some extent.
7

---

Published as a conference paper at ICLR 2025
W here can I see this sign? What does it mean?
Do you see any person who playing on the beach?
want open my own coffee shop, which should have comfortable and warm end troement. Can you help me design it? Show me picture it.
OOS
Sum,, can show you a picture of cozy coffee shop. Here is picture:
there playing beach in this image. The beach mostly empty except for few like striped towel, blue surfboant, bog, and couple chairs. are also few people in the distance, they are not actively playing.
It's traffic sign which usually seen beside the strest, instructing drivers stup completaty.
Describe the image detail.
please expl anatinn your design
show me a scene of a blue squirel holding flower perched Fence beside cluster of purple flowers!
In this image, shop. The shop is lilled with wooden furnitum, incloding tables and chairs, which adds warm and inviting atmosphere. There also some pottad plants placed around the shop, which beings touch of tables, there are various cups and saucers,
cozy codfee
The image captures un outdoor seating ama resturant cafe.
Here is the some
The seating arsa is uranged uruudd a central table, with multiple chairs pluced avound it In the foreground, there
indlors On the
black umbrella
bottles glasses,
providing shade for the sesting area. To
seggesting that the shop offers a variety of beverages, The walls udorned with pl ctures and sbelves, which add character to space. lighting in the shop is dim, creating relaxed and intimate amblance. Overall, coffee shop is designed to be comfortable welcoming place customers to enjoy their beverages and socialize.
sooting sign that read s *A NEW PERIENCE
Can you change cartoun style?
YOUR
WAY', The sign is white with blakk and placed on In the bockgrond, there building with a glass facade. appears to be commercial space, possibly related the new expertenc ence mentioned in the sign...
Sure, here are the curtoun imagu.
boilding retall
Figure 4: Qualitative results image understanding and generation. The words marked in green key elements in questions and answers. Best view it on screen.
The Impact of the Clustering Mechanism. Here, we compare the impact of different clustering mechanisms on model performance. As shown Table we can observe that tokenizers constructed using dynamic clustering mechanisms achieve superior overall performance compared to those with fixed setup while simultaneously accelerating training time and reducing computational costs during inference. In contrast to soft-clustering, which yields soft attention masks, findings suggest that hard-clustering produces better results, as it may be because hard clustering leads to higher consistency of cluster outcomes (Haurum et al.) 2023), leading more stable visual tokens and enhancing both the stability and performance of the model. When employing fixed number of clusters, the critical challenge T different datasets achieve optimal performance at varying numbers of clusters, with a uniform count across all datasets, resulting suboptimal outcomes.
determine the optimal number of clusters. As demonstrated in
Qualitative Analysis of Visual Understanding and Generation. As illustrated in Figure 4 our model exhibits proficiency in intricate image understanding tasks, such as deciphering reversed text, exemplified by the word "stop", and accurately identifying text "A NEW EXPERIENCE COMING YOUR WAY" that is partially covered. tasks involving detailed image descriptions, our approach prioritizes object-level information within images, which substantially mitigates the incidence of hallucinatory responses commonly observed in MLLMs. Moreover, in text-to-image generation, our model demonstrates remarkable capabilities in synthesizing coherent images, which maintain high fidelity and relevance to the textual context, such as the "flower", "fence" and "squirrel".
Qualitative Analysis of Visual Editing. Here, we evaluate the efficacy of image manipulation using model compared to the previous diffusion-based method MagicBrush (Zhang al. 2024c and various MLLMS including Emu-2-Gen (Sun et al. 2024a). MGIE (Fu et | 2024). and Mini- Gemini (Li al.) 2024d). As depieted in Figure SETOKIM displays superior performance by closely adhering to the provided instructions and preserving intricate image details. For instance, model seamlessly adds "tomato slices" image without altering other elements on the pizza,
8

---

Published as a conference paper at ICLR 2025
Original
MagicBruch
Emu-2-Gen
MGIE
Mini-Gemini
Our
Ad some tomato pizza
Change the color of umbrella to red.
Remove unasual part of this image
Change it daa--sty
Figure 5: Qualitative comparison between MLLMs for the image editing. SETOKIM excels in adhering to instructions and preserving low-level image details.
(1)
(3)
(4)
(6)
Figure 6: Token mask M visualization visual tokens generated by SeTok.
while Emu-2-Gien and MGIE fall short. Furthermore, our model exhibits remarkable precision in changing the color of an umbrella, while visual objects intended for alteration retain high level of consistency before and after editing. Additionally, SETOKIM demonstrates to precisely follow implicit user instructions to remove unusual elements from an image, i.e., the banana, preserving the surrounding context, whereas Emu-2--Ger mistakenly removes a telephone cord and MGIE fails to remove the banana properly, altering the cord's texture. These examples underscore the effectiveness of SETOKIM for high-precision image manipulation, leveraging semantically equivalent visual tokens to achieve nuanced and context-aware results.
Qualitative Analysis of Visual Tokens. In Figure 6 we demonstrate how input visual features are assigned to visual tokens after tokenization. First, we observe that our tokenization process resembles partial segmentations, producing semantically complete units. For example, in the second image, visual tokens correspond distinct elements such as the giraffe, grass, tree, and background, aligning with semantic intuition. Second, the number of tokens obtained from Setok is dynamic and not fixed. Third, SeTok capable of adapting to different levels of semantic granularity for the same concept, seen in images (4) and (5), where the person is represented in the image (1), the person is divided into tokens for the head, body, and legs. Lastly, in complex scenes, such the image (7), SeTok can still tokenize elements like traffic lights and billboards into semantically complete tokens. Overall, our approach ensures that similar visual features are consistently recognized and processed, improving both coherence and efficiency in tokenization.
single token. In contrast,

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
Currently, benefiting from the emergent phenomenon, LLMs have demonstrated near-human-level intelligence in language processing (Chiang et al. Simultaneously, researchers have been attempting to develop MLLMs by integrating mu Itmodal
2023 Touvron et al.
2023a Taori al. 2023).
encoders and decoders into LLMs (Dong et al. 2024: Koh al.
2023 Lu et al., 2024 Li al.
2024d Sun et al. 2024a 2023 Fei et al. (224). From the initial MLLMs that could only understand
multimodal input signals (Liu et al.
2024b (023c) to later versions supporting the generation of
multimodal contents (Sun et al. 2023 2024a| Koh al. [023] Wu et al.) (022c)) MLLMS have shown
powerful capabilities and a broader range of applications. Among all modalities, the integration of vision, known visual MLLM, has received the most extensive research and application (Gao et al. 2023] Sccwennk et al. 2022 Liu et al. 2023b Lu et al. 2021). The latest MILM research has not only achieved both understanding and generation of visual content, but also developed more refined, pixel-level visual modeling, including segmentation and editing functions (Yuan et al. 2024 Rasheed
al.
2024 Zhang al. 2024b You et al. 2023 Lai al. 2024).
On the other hand, an increasing body of research indicates that visual tokenization (Dosovitskiy et al. 2021] Ge et al., [023] Pin et al.; (2024b) significantly impacts MILM capabilities in vision tasks.
The fundamental approach involves encoding the input visual content into feature representations via
visual encoder (e.g.. Clip-VIT Radford al. (202T) and mapping these to LLM, thus enabling
a language- based LLM to understand vision. The corresponding method involves patchifying the
original visual images of various sizes into smaller dize-size patches (Dosovitskiy et al., [20T]]
Bavishi et al.) 2023 Liu et al. [023c] [Sun et al. (202)), treating these as tokens, and encoding each patch/token to obtain corresponding embeddings, which are then fed into the L.M.M Subsequent research (Jin et al. 2024b Ge et al. 2023), aiming further to unify the training objectives of language and visual modalities by introducing codebook techniques, where visual elements are represented as discrete tokens. This allows visual training to be treated similarly to language training, i.e., conducting next token prediction (Ge et al.) 2023). Unfortunately, whether in the above visual encoding tokenization techniques, there is significant bottleneck of MILM performance: the integrity of visual semantic units, either visual objects or compositional regions, is compromised during the patchifying process. This results in a less effective semantic alignment between vision and language within the LLM. This paper is the first to propose a solution to this problem, introducing a novel Semantic Equivalent Tokenization for MILM.
In addition, this work is also related to scene decomposition (Yang et al. 2022 Niu et al. 2024
Locatello et al. 2020 Li et al., 2020 2024b). which involves segmenting scene into objects. Typically, these methods use a fixed number of query tokens (Kirillov et al. 2023 Suzuki 2022)
and apply cross-attention (Yang et al. 2022] Qi et al., 2023] Li et al.] (024c) to aggregate visual features implicitly. However, this fixed-token approach may not only correspond to the actual visual content but also requires complex network architectures (Caron et al. 2018| Gansbeke et al. 202T) and extensive data for optimization. When combined with LLMs, such complexity significantly increases computational resource demands. Conversely, we learn a dynamic number of semantic objects and do not require complex model structures for optimization, thereby enhancing resource efficiency and providing a more adaptable solution for integrating visual and language modalities.
6 CONCLUSION
In this paper, we introduce SeTok,, a viable semantic-equivalent tokenizer, that enables to tokenize automatically patch-level visual features into a variable number of semantic-complete concept vi- sual tokens. Then, we integrate SeTTok with a pre-trained LLM to build an MLLM, SETOKIM, optimized using a unified autoregressive objective and woo-stage training strategy. Extensive experi- ments demonstrate that our model performs better on broad range of comprehension, generation, segmentation, and editing tasks, highlighting the effectiveness of Setok.
ACKNOWLEDGMENTS
This work is partially supported by NUS Start-up Grant A-0010106-00-00.
REFERENCES
Jean- Jean-Baptin Baptiste Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford,
10

---

Published as a conference paper at ICLR 2025
Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadch, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In Proceedings of the NeurIPS, 2022.
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. abs/1600.064500 2016.
CoRR,
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-vl: frontier large vision-language model with versatile abilities. CoRR, abs/2308. 12966, 2023.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023.
a
blog/ Btrhe/uyuuuBB 8b
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In Pruceedings of the ICLR, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning follow image editing instructions. In Proceedings of the CVPR, pp. 18392-18402, 2023.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the ECCV, pp. 139-156, 2018.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In Proceedingy of the CVPR, pp. 3558-3568, 2021.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 902023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Proceedings of the CVPR, pp. 248-255, 2009.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Dreamllm: Synergistic multimodal comprehension and creation. Proceedings of the ICLR, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the ICLR, 2021.
Mingjing Du, Shifei Ding, and Hongjie Jia. Study on density peaks clustering based on k-nearest neighbors and principal component analysis. Knowledge Based System, 99: 19:135-,45, 2016.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. GENESIS-V2. inferring unordered object representations without iterative refinement. In Proceedings of the NeurlPS, pp. 8085-8090 2021.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the CVPR, pp. 12873-12883 2021.
Hao Fei, Shengqiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision Ilm for understanding, generating, segmenting, editing. In Proceedings of the NeurIPS, 2024.
11

---

Published as a conference paper at ICLR 2025
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin. Jinrui Yang, Xiawu Zheng. Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. CoRR, abs/2306. 13394, 2023.
Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guiding instruction-ben based image editing via multimodal large language models. In Proceedings of the ICLR, 2024.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a- scene: Scene-based text-to-image generation with human priors. In Proceedings of the ECCV, pp. 89-106, 2022.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Unsupervised semantic segmentation by contrasting object mask proposals. In Proceedings of the ICCV, pp. 10032-10042, 2021.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efffcient visual instruction model. arXiv preprint arXiv:2304. 15010, 2023.
Yuying Ge, Yixiao Ge, Ziyun Zeng. Xintao Wang, and Ying Shan. Planting a SEED of vision in large language model. CoRR, abs/2307.08041 2023.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. SEED-X: multimodal models with unified multi-granularity comprehension and generation. CoRR, 242.24444.14396, 2024.
Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the in VQA matter: Elevating the role of image understanding in visual question answering. IJC,, 127(4):398-41) 414, 2019.
Joakim Bruslund Haurum, Sergio Escalera, Graham W. Taylor, and Thomas B. Moeslund. Which tokens to use? investigating token reduction in vision transformers. In Proceedings of the ICCV, pp. 773-783, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the CVPR, pp. 770-778, 2016.
Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. CoRR, abs/2403.13298, 2024.
Menggi Huang, Zhendong Mao, Quan Wang, and Yongdong Zhang. Not all image regions matter: Masked vector quantization for autoregressive image generation. In Proceedings of the CVPR, pp. 2002-2011, 2023.
Yuzhou Huang. Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, and Ying Shan. Smartedit: Exploring complex instruction- based image editing with multimodal large language models. In Proceedings of the CVPR, pp- 8362 8362-8371, 2
Drew A Hudson and Christopher D Manning. Gqa: A dataset for real-world visual reasoning and compositional question answering. In Proceedings of the CVPR, pp. 6700-6709, 2019.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the CVPR, pp. 13700-13710, 2024a.
Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan. Quzhe Huang. Bin Chen, Chengru Song, Dai Meng, Di Zhang, Wenwu Ou, Kun Gai, and Yadong Mu. Unified language-vision pretraining LLM with dynamic discrete visual tokenization. In Proceedings of the ICLR, 2024b.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the EMNLP, pp. 787-798, 2014.
12

---

Published as a conference paper at ICLR 2025
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In Proceedings of the ICCV, pp. 3992-4003, 2023.
Jing Yu Koh, Daniel Fried, and Russ Salakhutdinov. Generating images with multimodal language models. In Proceedings of the NeurlPS, 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123(1):32-73, 2017.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale. IJCV, 2020.
Xin Lai. Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. LISA: reasoning segmentation via large language model. In Proceedings of the CVPR, pp. 9579-9589, 2024.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, 2024.
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the CVPR, pp. 11513-11522, 2022.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. In Proceedings of the NeurlPS, 2024a.
Xiangtai Li, Ansheng You, Zhen Zhu, Houlong Zhao, Maoke Yang, Kuiyuan Yang, Shaohua Tan, and Yunhai Tong. Semantic flow for fast and accurate scene parsing. In Proceedings of the ECCV, pp. 775-793. Springer, 2020.
Xiangtai Li, Henghui Ding, Haobo Yuan, Wenwei Zhang, Jiangmiao Pang, Guangliang Cheng, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer-basedd visual segmentation: A survey. IEEE Trans. Puttern Anal. Mach. Intell., 46(12):10138-00163 2024b.
Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. Omg-seg: Is one model good enough for all segmentation? In Proceedings of the CVPR, pp. 27948-27959, 2024c.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. CoRR, abs/2403. 1222403.181444 18814, 2024d.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- lssounlanguage models. In Proceedings of the EMNLP, pp. 292-305, 2023.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning. Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by aligmment before projection. CoRR, abs/2311.101122, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Proceedings of the ECCV, pp. 740-755, 2014.
Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings of the CVPR, pp. 23592-23601, 2020a,
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions the the Association for Computational Linguistics, 1:635-651,, 2023b.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. CoRR, abs/2402.08268, 2024a.
13

---

Published as a conference paper at ICLR 2025
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proceedings of the NewrIPS, 2023c.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction 2024b.
tuning. In Proceedings of the CVPR, pp.
Francesco Locatello, Dirk Weissenborn Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Proceedings of the NeurIPS, 2020.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. In Proceedings of the CVPR, pp. 26429-26445 2024.
Pan Lu, Liang Qiu, Jiaqi Chen, Tanglin Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chur Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In Proceedings of the NeurIPS Datasets and Benchmarks, 2021.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In Proceedings of the CVPR, pp. 11-20, 2016.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In Proceedings of the CVPR, pp. 3195-3204, 2019.
Dantong Niu, Xudong Wang, Xinyang Han, Long Lian, Roci Herzig, and Trevor Darrell. Unsu- pervised universal image segmentation. In Proceedings of the CVPR, pp. 22744-22754. IEEE, 2024.
Kaihang Pan, Siliang Tang, Juncheng Li, Zhaoyu Fan, Wei Chow, Shuicheng Yan, Tat-Seng Chua, Yueting Zhuang, and Hanwang Zhang. Auto-encoding morph-tokens for multimodal LLM. In Proceedings of the ICMI., 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. In Proceedings of the ICLR, 2024.
Lu Qi, Jason Kuen, Tiancheng Shen, Jiuxiang Gu, Wenbo Li, Weidong Guo, Jiaya Jia, Zhe Lin, and Ming-Hsuan Yang. High quality entity segmentation. In Proceedings of the ICCV, pp. 4024-4033, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the ICML, pp. 8748-8763, 2021.
Hanoona Abdul Rasheed, Muhammad Maaz, Sahal Shaji Mullappilly, Abdelrahman M. Shaker, Salman H. Khan, Hisham Cholakkal, Rao Muhammad Anwer, Eric P. Xing, Ming-Hsuan Yang. and Fahad Shahbaz Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the CVPR, pp. 13009-13018, 2024.
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the CVPR, pp. 26364-26373, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the CVPR, pp. 10674- 10685, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: an open large-scale dataset for training next generation image-text models. In Proceedings of the NeurIPS, 2022.
14

---

Published as a conference paper at ICLR 2025
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In Proceedings of the ECCV, pp. 146-162, 2022.
Jing Shi, Ning Xu, Yihang Xu, Trung Bui, Franck Dernoncourt, and Chenliang Xu. Learning by planning: Language-gui ggggge global image editing. In Proceedings of the CVPR, pp. 13590-13599, 2021.
Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu
Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu, Eagle: Exploring the design space for multimodal Ilms with mixture of encoders. CoRR, abs/2408. 15998, 2024.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers. Jacob R Steeves, Joel Hes- A 627B token cleaned and dedu-
tness, and Nolan Dey. SlimPajama:
plicated version of RedPajama. sl slimpa jama---aaaaa----aaa
https: http://ww.cerebbasneett/////
June 2023.
URL
https: htt.D:///mgggggga......///ttttts
S1 6lim2 1mimPaap jamaaajam--- jamjjjmaaa... 627B
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. CoRR, abs/2307.05222 0,2222,, 2023.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang. Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the CVPR, pp. 14398-14409, 2024a.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In Proceedings of the ICLR, 2024b.
Teppei Suzuki. Clustering as attention: Unified image segmentation with hierarchical clustering. CoRR, hbs//205500999, 2022.
Hao Tan, Franck Dernoncourt, Zhe Lin, Trung Bui, and Mohit Bansal. Expressing visual relationships via language. In Proceedings of the ACL, pp. 1873-1883, 2019.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An followwng llama model. 2023. URL htRps:/:s:/// /ttttps://itthub.com htt.a//w..ccotttttttaaaaaaataaaa
attp://aaaaooddaapaaaaaaaaaa alpaca]
Chameleon Team. Chamelcon: Mixed-modal carly-fusion foundation models. CoRR, abs/2405.09818. 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ana. Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jereamy Fu, Wenyin Fu, Brian Fuller. Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and Line-tuned chat models. CoRR, abs/2307.09288, 2023b.
15

---

Published as a conference paper at ICLR 2025
Jianzong Wu, Xiangtai Li, Chenyang Si, Shangchen Zhou, Jingkang Yang, Jiangning Zhang, Yining Li, Kai Chen, Yunhai Tong, Ziwei Liu, and Chen Change Loy. Towards language-drivee video inpainting via multimodal large language models. In Proceedings the CVPR, pp. 12501-12511, 2024a.
Junyi Wu, Bin Duan, Weitai Kang, Hao Tang, and Yan Yan. Token transformation matters: Towards faithful post-hoc explanation for vision transformer. In Proceedings of the CVPR, pp. 10926-10935, 2024b.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal LLM. In Proceedings of the ICML, 2024c.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas M. Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the CVPR, pp. 18113 18113-18123, 2022.
Tao Yang. Yuwang Wang. Yan Lu, and Nanning Zheng. Visual concepts tokenization. In Proceedings of the NeurIPS, 2022.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang. Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. CoRR, abs/2310.0771 2023.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Proceedings of the TACL, 2:67-78, 2014.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In Proceedings of the ICLR, 2022.
Licheng Yu, Patrick Poirson, Shan Yang. Alexander C Berg, and Tamara Berg. Modeling context in referring expressions. In Proceedings of the ECCV, pp. 69-85, 2016.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang-Wer Wen Li, Susan Zhang, Richard James, Gargi Ghosh, Yaniv Taigman, Maryam Fazel-Zarandi. Asli Celikyilmaz, Luke Zettlemoyer, and Armen Aghajanyan. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. CoRR, abs/2309.02591, 2023a.
Qihang Yu, Mark Weher, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang- -Chihh Chen. An image is worth 32 tokens for reconstruction and generation. CORR, abs/2406. htss/2406007550, 2024.
Weihao Yu, Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. CoRR, abs/2308. 024900. 2023b.
Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang. Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In Proceedings of the CVPR, pp. 28201, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the ICCV, pp. 11941-11952, 2023.
Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua. Vpgtrans: Transfer visual prompt generator across 1ms. In Proceedings of the NeurIPS, 2024a.
Ao Zhang, Yuan Yao, Wei Ji, Zhiyuan Liu, and Tat-Seng Chua. Next-chat: An LMM for chat, detection and segmentation. In Proceedings of the ICML, 2024b.
Guiwei Zhang, Yongfei Zhang, Tianyu Zhang, Bo Li, and Shiliang Pu. PHA: patch-wise high- frequency augmentation for transformer- based person re-identification. In Proceedings of the CVPR, pp. 14133-14142, 2023.
16

---

Published as a conference paper at ICLR 2025
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guider image editing. In Proceedings of the NeurIPS, 2024c.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the CVPR, pp. 586-595, 2018.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing lainulla understanding with advanced large language models. CoRR, abs/2304. 105522.
2023.
17

---

Published as a conference paper at ICLR 2025
A ETHIC STATEMENT
This work aims to build semantic equivalence tokenization to segment input images into semantic complete tokens enhance the MLLMs in vision understanding, generation, segmentation, and editing capabilities. Here we discuss all the possible potential impacts of SETOKIM.
Use of Generative Content The SETOKIM, limited by the quantity of fine-tuning data and the quality the base models, may generate some low-quality content. Also, generative model, the LLM will produce hallucinated content in multimodal formats that may be harmful to society. We have reminded users to interpret the results with caution. Anyone who uses this LLM should obey the rules in license. And also commercial use of our system is not allowed.
Data Privacy and Security Our research utilizes datasets that are either publicly available or collected with explicit consent. We adhere to strict data privacy and security protocols to protect the information and ensure it is used solely for this research.
Bias Mitigation Recognizing the potential for bias in AI models, particularly in vision-language tasks, rigorously test tokenizer across diverse datasets. This approach is designed identify and mitigate biases that may affect the model's performance or lead to unfair outcomes in its applications.
B LIMITATION
While SETOKIM has achieved further improvements across various language-driven vision tasks, becoming zero-shot general specialist, still faces several limitations.
Model Scale. The evaluation of our model is currently constrained to configurations with 7B parameters. As shown in (Laurençon et al. 2024), the performance of MILMM is limited by the scale of the core backbone LLM. Despite the impressive results achieved, the potential benefits of employing significantly larger models, such as 65B or 130B, are worth exploring in future studies.
The Resolution of Image. Our model supports images with resolutions up to 384 384, enabling the understanding of visually fine-grained content. While there have been improvements in under- standing visually fine-grained content, challenges remain when processing higher-resolutie particularly for tasks requiring detailed visual reasoning. Recent advancements have explored various strategies to address these challenges. For instance, (Sh et al. (2024) highlights that straightforward channel concatenation between low- and high-resolution features serves as an efficient and effective fusion strategy, achieving a balance between performance and computational efficiency. Moreover, the use of mixture-of -ffture-of-eexpers (MoE) structures has shown significant improvements when combining different vision encoders. Despite these advances, there is still need to enhance the understanding of low-resolution inputs and the ability to generalize across diverse modalities, particularly for tasks where fine-grained details embedded in low-resolution visual data.
images,
Hallucination. Although our model has made some progress in mitigating hallucination through fine-grained vision- liniunnl alignment, as demonstrated in experiments the POPE dataset, halluci- nations remain inevitable. This area continues to pose challenges and is crucial for future exploration and enhancement.
C DETAILED METHOD
C.1 TOKEN CLUSTER
The formal token clustering algorithm described in Algorithm 1. Specifically, a scope 2 = [0, km is initialized to matrix of ones 1k to track the degree which visual embeddings have been assigned to clusters. In addition, the seed scores are initialized by combining the local density in (1 and distance in Eq. to perform the selection of visual embeddings. At each iteration,
18

---

Published as a conference paper at ICLR 2025
single embedding vector Z.j is selected at the spatial location (i.j) which corresponds to the argmax of the element-wise wise multiplication
the seed scores and the current scope. This ensures that cluster seeds sampled from pixel embeddings that have not yet been assigned to clusters. An alpha mask oa E [o, 1JAxu is computed as the distance between the cluster seed embedding æi, and all individual pixel embeddings according distance kernel yp. The output of the kernel yp is one if two embeddings are identical and decreases to zero as the distance between pair of embeddings increases. Additionally, negative penalty Bs applied the alpha mask by misusing the seed scores, where 3 is a hyper-parameter. This encourages the selection of elements similar to the current feature with lower information density. The associated concept mask Mc obtained by the element-ve wise multiplication of the alpha masks by the current scope. An clement-wiss multiplication with the complement of the alpha masks then updates the scope. This process is repeated until stopping condition is satisfied, at which point the final scope is added an additional mask to explain any remaining embeddings.
Algorithm Token Clustering Algorithm
Require: visual embeddings X E Rhxwxd
Ensure: masks M E [o, 1] XxwxC with E, Maj.e =1
1: Initialize: masks M = 0, scope = w seed scores & E R
2: while not MM) do
3: (i,j) arg max < @)
4: a = sigmoid(y((X, (i.j)) 3s)
5: M.append( ②
6: == = @ (1 a)
7: end while
8: M.appen(zz)
C.2 CONCEPT-LEVEL IMAGE-TEXT CONTRASTIVE LOSS
To enable effective visual concept token learning, propose concept-level image-text contrastive loss. Specifically, we randomly select K objects the image, and acquire the corresponding object labels, and then prompt each of them with set of handcrafted sentence templates, e.g., 'A photo of jobject label]'. The motivation for selecting objects is that they are the smallest units of image representation with complete semantics and have corresponding relationship with the semantic units in the text. Next, we employ contrastive losses between the new sets of image- 'prompted text" pairs {(I, Ti), (I, T2),
(I, (K)] where \ are all prompted sentences generated from the objects sampled from the image I. Among the batch B, each image has K positive text pairs and B(K - 1) negative pairs. Similarly to the standard image-text contrastive loss (Radford et al. 2021 define the concept-level image-text contrastive loss sum of two two-way contrastive losses.
1
B
Et=rexp(V) exp(VV V /T) E ΣS exp(V/ V /7) K
log
(5)
B
K b= expl Th V//T)
C(TU) = B
log
(6)
Σ exp( V V//T) K
1=1
LITTTOX = (I +
where the concept representation ViTh is extracted by the pre-trained CLIP-based text encoder, which frozen during training.
D DETAILED EXPERIMENTS SETTINGS
D.1 IMPLEMENTATION DETAILS
For the Soko we apply pre-trained 00LL---0000M-pacc4--300000 (Zhai et al.) (023) as our vision encoder, and the numbers of inner-cluster and inter-cluster transformer layers set 12, and 8,
19

---

Published as a conference paper at ICLR 2025
Name
Size
ImageNet-II IK
(Dng al.
2009
1.2M
SeTok
OpenImages
Kuznetsova et al.
2020
9M
CC12M (Changpinyo al.
2021
12M
MuioNNacsthetiss1222 12M Schuhmann et
2022
12M
ALLaVA-Caption-Ay ( Chen
2024
715K
Stage-I
InstructPixPPix (Brooks el 2023
313K
LLaVA-595K (L.u 20256
595K
MSCOCO (Lin ct 2014)
313K
Visual Genome (Krishna
2017
108K
OpenImages Kuznetsova et al.
2020
9M
SlimPajama Soholeva al.
2023
ALLaWA-Instruct--V (Chen al.
2024
661K
ShareGPT4V (Krishna al.
2017
80K
Alpaca Taori et al 2023
LLaVA 1..Vvvll.-mmxx6666
(Liu et
2023c
665K
Stage-II
VQA Goyal et al
2019
83K
GQA Hudson Manning
2019
72K
OKVQA Murino et al
2019
9K
AOKVQA
Schwenk
2022
50K
RefCOCO/+s
(Kazemzadeh et
2014
Mao ct al. 2016
65K
InstructPix2Pix
Brooks at.
2023
313K
MagicBrush Zhang et al. 2024c)
10K
Table 7: The training data used in our experiments.
respectively. The dimension of the semantic-cquivalentt token 512. For the detokenizer, we adopt L 12 transformer-ben based layers with cross-attention, where the keys and values are derived from fixed number of masked tokens. This process converts the dynamic number tokens into a fixed-size representation. Also, inspired by Yu et al. (2024), we employ CNN-based pixel decoder with an upsampler to reconstruct the original images.
In the SETOKIM framework, we employ the LLaMA-2-7B (Touvron et al. 2023b to initialize LLM backbone. Following Kirillov et al. (023). we take the image embedding extracted in the vision encoder in SeTok and the visual tokens generated by LLM as inputs, which are both fed into the mask decoder. This decoder uses prompt self-attention and cross-attention in two directions (prompt-to-image embedding and vice-versa) update all embeddings. After running two blocks, we upsample the image embedding and an MLP maps the output token to dynamic linear classifier, which then computes the mask foreground probability at each image location. Following [i al.) (2024a), we employ small MIPP consisting of three residual blocks (He et al., 2016) for computing diffusion loss. Each block sequentially applies LayerNorm (LNN (Ba et al. 2016). linear layer, SiLU [llfwigg al. 2018), and another linear layer, merging with residual connection.
D.2 TRAINING DATA
Here, we detail the training data utilized for training SeTok and SETOKIM in Table 7 In the training phase SeTok, ImageNet-1K (Deng et al. 2009) is employed for reconstruction tasks, while OpenImages (Kuznetsova et al.) 2020) supports both reconstruction and alignment learning. Additionally, some overlap exists between datasets used in Stage-I and Stage-II training. For instance, datasets like VQA (Goyal et al. 2019) ShareGPT4V (Krishna al. 2017] and GQA (Hudson & Manning, 2019) have been included in LLaVA-v1.S 1.55-m..-mmixxx.... and comprehensive view of the training data sources and their usage, explicitly enumerate all datasets included the training pipeline.
(Liu et al. (023c). To provide a clear
D.3 TRAINING RECEIPT
In Table we list the detailed Syper-parameters setting at three stages, ie.. Setok training and two-stage SETOKIM training. All training is conducted 64x H100 (80G) GPUs.
20

---

Published as a conference paper at ICLR 2025
Data Size
Model
LLM
Vision Encoder
Image Resolution
Pretrain
Finetune
InstructBLIP
Vicuna- 13B
ViT-g/14
224
129M
1.2M
Qwen-VL-Chat VL-Chat
Qwen-7B
ViT-bigG (Fine-tuned)
14B
50M
Emu
LLaMA-7B
EVA-01-CLIP
224
>600M
312K
DreamLLM
Vicuna-7B
CLIP
32M
120K
LLaVA-1.5
Vicuna-1.5 1,5 7B
CLIP VIT-L/336px
336
558K
665K
SEED-X
Llama2-chat-13B
Qwen-VL VL
158M
250M
LaVIT
LLaMA-7B
ViT-G/14 EVA-CLIP EVA-CLIP
224
100M
193M
Unified-1O-2
ViT-B
384
1.127B
559M
CM3Leon
VQVAE
256
2.4T tokens
114M
Chameleon
VQVAE
512
>1.4
1.8M
SETOKIM
Llama2-7B
35M
1.2M
Table 8: Configuration comparison between baselines and SETOKIM. indicates training the LLM from scratch.
D.4 BASELINES.
Here, we explicitly demonstrate configuration comparison in terms of the LLM version, vision encoder, and data size used in the baselines and SETOKIM in Table g
Configuration
Tok
Stage-I
Stage-II
Optimizer
A blloat16
AdamW
Adam
Precision
b6loat
blooat
Peak leamingg of LLM
5e-5
5e-5
Pcak learming rate of Visual Part
5c-4
lc-4
2c-4
Weight Decay
0.05
0.1
0.01
Learning Rate Scheduler
Cosine
Cosine
Cosine
Warmup Steps
10K
2K
5K
Input image resolution
384 x384
384x384
384x
Batch Size GPU
16
16
Gradient Accumulation Steps
8
8
Maximum Token Length
2048
2048
Table 9: Training recipes for SeTok, SETOKIM of Stage-I: Multimodal Pretraining and Stage-II: End-to-end Instruction Tuning.
E EXTENDED EXPERIMENTAL ANALYSIS
Setting
Tr-v Ir-t Text Multi-modal Humanities STEM Social Sciences Other Average
LLaMA-2-7B
3c-4 100%
0%
42,9
36.4
51.2
52.2
45.3
SETOKIM
le-4 Se-5 70%
30%
41.7
34.8
49.4
51.0
43,9
SETOKIM
le-4 Se-S 50%
50%
37.5
31.4
46.3
45.9
40.1
SETOKIM
le-4 Se-5 30%
70%
30.3
31.7
44.7
41.1
Table 10: LLM comparison by varying the language-vision dataset ratio.
The Impact of Language Volume. Before performing Stage-2 instruction training, we conduct experiments with mixing text and image data in various proportions to identify the optimal balance of additional text data. The experimental results on the MMLU dataset are summarized in Table TO] Our findings suggest that ratio of 7:3 (Language: Vinguun is optimal, as it minimally impacts the LLM's language performance (-1.4 on MMLU) while achieving the best results on both multimodal understanding and generation tasks.
21

---

Published as a conference paper at ICLR 2025
Method
Flickr30K (CIDErt) VQAv2 (Accuracy)) GQA (Accuracy1)
SeTok
86.9
78.5
65.6
w/ Cree 78.1
49.7 65.8
w/ Ceite
83.6
76.3
63.4
Table 11: The effect of unlocking vision encoder in training Setok and SETOKIM.
The Loss Impact for Setok. We argue that reasonable tokenizer must possess two essential attributes: 1) Complete and enriched high-level semantic information and 2) Undistorted pixel-level details. Therefore, we design to optimize the Setok by minimizing the reconstruction loss and concept-level image-text contrastive loss. Here. we conduct further experiments to explore the effect of each loss on tokenizer performance. As the results shown in Table [1 we observe that the performance with only Leite is superior to that with only Cree. We attribute this to the fact that relying solely on free causes the tokenizer to focus primarily on pixel-level information, often the neglect of high-level semantic information. This imbalance may introduce challenges for the LLM when interpreting image semantic content with limited training data.
Setting
ImageNet (rFID)) Flickr30(IIDE:))) VQA (Acc.f)
Frozen
123.6
85.4
77.5
UnFrizen 2.07
86.9
78.7
Table 12: The effect of unlocking vision encoder in training Setok and SETOKIM.
The Impact of Unfreeze Vision Encoder. To evaluate the impact of unfreezing the vision encoder, we conduct an ablation experiment where the vision encoder is kept frozen, and only the token merger optimized. We observe that SeTok fails to reconstruct the image as freezing the
and detokenizer vision encoder hinders its ability to learn the low-level features required for accurate reconstruction. In this scenario, the vision decoder alone is tasked with reconstruction, but it is unable to do so effectively using only high-level semantic features, Interestingly, freezing the vision encoder did not noticeably impact SeTok's performance in vision-language semantic understanding.
Mechanism
#Tokens
TFLOPs
Flickr30K
VQA*2
OK-VQA
SigLIP MLP Liu
2024b)
256(Fixed)
15.8
80.6
72.4
56.1
SigLIP Q-former (huu 2023
32(Fixed)
12.4
81.3
71.0
$4.6
SigLIP Resampler Alayrac et al. 2022
64(Fixed)
13.4
83.4
72.5
54.9
ScTook"
Dynamic
8.2"
86.9
78.7
60.2
Table 13: Comparison between Setok and other vision tokenization approaches, all of which generate continuous visual tokens that are subsequently fed into the LLM.
The Comparison of Vision tokenizer. To evaluate whether our proposed SeTTok effectively inte- grates with LLMS to enhance model performance, we experimented with different connector strategies. such as MLP (Liu et al. 2024b), Q-former (Zhu et al., (023) and Resampler (Alayrac et al. (002)). Using the same vision encoder (i.e., SigLIP-SO40OM 10000000000--aachhh-0-00 patch14-384), we construct various MLLM architectures. We follow two-stage training process the same dataset. Finally, we assessed the models' performance on vision-languages tasks, and the results are presented in Table [3] As observed, Tok demonstrates higher efficiency, achieving lower TFLOPS while delivering superior vision understanding capabilities. These findings validate that SeTok is capable of learning more aligned and compact visual tokens, leading to better semantic integration and improved performance.
Furthermore, we retrained SETOKIM using the same dataset as VL- focusing solely on performance in visual understanding tasks. As shown in Table our model consistently outperforms LLaVA across benchmarks, highlighting SETOK's 's ability to achieve more effective vision-language alignment and enhance overall performance.
Qualitative Analysis of Visual Segmentation.
We present the segmentation examples in Figure It is easy to note that the attention mask closely aligns with the object mask, and our model
22

---

Published as a conference paper at ICLR 2025
Method
VQA
GQA
VisWiz
POPE
MME
MM-Vet
LLaVA-1.5
78.5
62.0"
50.0
85.9
1510.7
33.1
SETOKIM
78.6*
63.8*
$2.7
87.6
1521.4
Table 14: Comparison between SETOKIM and LLaVA using the same dataset for training. indicate the training datasets are observed during training.
Resconmuction
Original
Rescontrction
Original
Figure 7: The image reconstruction results from the visual detokenizer in Setok.
Imput
Token Mask
SETOKIM
Ospery
GLaMM
Figure 8: The visualizations for segmentation results compared with GLaMM (Rasheed al. 2024 and Osprey (Yuan et al. 2024).
shows superiority in achieving more accurate and detailed segmentation results than other LLM- based segmentation methods. Notably, as depicted the second row of this figure, the visual token generated by method encompasses all depicted fish, effectively achieving complete segmentation of the fish in the scene. In contrast, other models produce only partial segmentation. This effectiveness of the segmentation highlights the precise content representation and improved interpretability of the visual tokens. Such visual tokens can eventually enhance the vision- -anguge understanding incorporated with the text tokens.
23

---

Published as a conference paper at ICLR 2025
A cozy library filled with books, with annchaie the conner and flickening nearby.
bool of romen steeming broth, sices of pork, soft-boiled eggs, green onions, presented a traditional Japanese setting.
futuristic city skyscnopers and flying cars zooming through the sky of night, with neon lights illuminating the streets.
glowing
majestic mountain range with snow oopped peoks under a clean blue sky, aurrounded by lush
fireplace
green forests and crystal-clean lake.
selfie
An ostronout flooting gozing the Earth with stors twenking the bockground.
dag sitting on esting waherne
ghest.
mysterious cloak, glowing crystal hand while standing
superhere wind, watching over city night.
building, cape
brove knight riding white harze through enchanted forest.
humon body tuxedo sitting chair, reading book
transporent sculpture of a duck made out of gless. The soulppure in front of pointing of lardsoope.
Figure 9: The visualization of generation images from SETOKIM.
The Quantitative Reconstruction of SeTok. In Figure 7 we visualize some reconstructed exam- ples by Setok. can be seen that, given the tokenized visual tokens, the original input images can be successfully recovered. The reconstructed examples exhibit high degree of the construction of the method.
Visual Generations.
Figure 9 visualizes the images generated by SETOKIM.
Visual Understanding. Figure 10 presents additional examples of vision- understanding and reasoning tasks. Notably, as shown Figure IT SETOKIM exhibits strong in-context learning and multi-image reasoning capabilities.
24

---

Published as a conference paper at ICLR 2025
sleeping bed with cell phone car
Two dogs cat laying bed
parked front of a trolley
An older and wearing hat are playing gome with a gamepad
head
(a) Image Captioning Task
Q: How many opples the picture? A: Two
Q: What transportation does the man take?
Q: What sign mean in the picture?
What color the hat on mon' head? A: Green.
What the unusual of the picture?
A: Train.
A: Stop.
The frog has toil.
Quesion-Answeerigg
Figure 10: The SETOKIM'S performance visualization of image captioning (a) and VQA (b) task.
Prompt: Can aninaal <ingts?
Four cows
The gimaffs, which is hand trammel typically foundl in seventes and opca woodlands connot live in the ocean, somn second image.
ŠETOKIM:
dog
Two giraffes
Pronpt: What the common thing and the differnce between <im and diegs??
Two people skiing on snewy nountain
SēToro::
Beth images capture cats bathrom, but the fiss tocuses on rat neir dink, swcrnd shows
playing
A person surfing on a kirge wave
(a) In-conlext Learning Capabilily
(b) Reasoning Capabilily Over Mulliple Images
Figure 11: Illustration of SETOKIM performing in-context learning in (a) with two image-text pairs and third image as context to prompt the model, and reasoning across multiple images in (b) with two images with the question as context to guide the model.
25


Reviews:
[{"review_id": "SKorpbD1mX", "reviewer": "Reviewer_zAwn", "summary": "This work purpose a novel dynamic Semantic-Equivalent Vision Tokenizer(SeTok)，using adynamic cluster method to group visual features into semantic units, , flexibly determining the number of tokens based on image complexity. The proposed SETOK-based MLLM (SETOKIM) shows superior performance on various text-image tasks.", "questions": "1.In Table 2 that SETOKIM and some similar methods still have a gap in performance with SOTA under some data sets, please give some explanations.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper points out that the main bottleneck of current MLLMs is visual tokenization method, which cannot effectively align semantic and image features. Current visual tokenization methods only divide the image into a fixed number of patches, which not only destroys the visual context information, but also makes it difficult to align the visual semantic units.\n\n2.During SeTok training, this work introduces concept-level text-image contrastive loss and image reconstruction loss. The former ensures the semantic independence and integrity of each patch, and semantically alignes visual tokens with the corresponding text concepts. The latter ensures that the token retains enough pixel-level detail by calculating the loss when feeding the token into the detokenizer to reconstruct the original image. Finally, the sum of the two losses is calculated using weighting.\n\n3.The ablation experiments of key modules are sufficient, and the performance of the method on various language image tasks is fully demonstrated, demonstrating the superiority of the method from multiple perspectives.", "weaknesses": "1.Table 2 compares the performance of SETOKIM method and some similar methods under the task of zero-shot text-to-image generation and editing under several data sets. The results show that the indicators of some data sets are not optimal, and there is still some gap with SOTA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sRF3AinPe3", "reviewer": "Reviewer_sWMT", "summary": "This paper proposes a new visual tokenization method SeTok, which groups visual features into semantic units via a dynamic clustering algorithm. This design is aimed to address the information fragmenting problem and the semantic alignment problem in previous visual tokenization methods. A MLLM called SETOKIM is trained based on the proposed tokenization method.", "questions": "- How is the visual detokenizer trained? As the tokenized visual tokens have dynamic numbers, it is very confused for me on how the visual tokenizer is trained based on the dynamic visual tokens. And I found no description on this in the paper.\n\n- I also have an open question for the author. Any answers should be ok. The proposed tokenization method groups the vision inputs without any semantic instruction. But as we know, vision has infinite divisibility, which means that all objects can always be decomposed into smaller parts. Based on this, maybe a more reasonable way is to perform clustering during the reasoning process inner the MLLM according to corresponding contexts, instead of the input part. So, I would like to hear the authors’ thoughts on this.\n\nI intend to raise my score if the authors can properly address my concerns in the weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The motivation of dynamic SeTok makes a lot of sense. As the paper mentioned, fragment visual input can corrupt the visual semantic integrity, which is a long-lasting problem in visual tokenization. This paper provides a simple solution though clustering but exhibit high performance, which could inspire future works.\n- Sufficient experimental evaluation. Various tasks are included, like referring expression segmentation, visual understanding, and text-to-image generation. This could take much efforts. Also, this paper present visualization of the token masking results of SeTok in Fig. 7, which I think is a strong support for the motivation.", "weaknesses": "- Unfair comparison. 1) The model is finetuned on a lot of multimodal instruction datasets, whereas the compared works, like the ones in Tab. 1, use smaller data sizes for finetuning. Performing comparison on different training data hardly demonstrates the priority. 2) The visual encoder introduces more parameters for clustering (totally 20 transformer layers). 3) A more advancing pretrained encoder SigLIP-384 is used. In conclusion, many settings, including the data, the model parameters, and the pretrained encoder, exist unfair comparison. These settings conceal the true effectiveness of the proposed tokenization method. I understand that it is hard for an MLLM to perfectly align with previous settings, and it is ok. But my intension is to make sure the authors can explicitly present these differences in the tables, which helps prevent misleading and ensure fair comparisons in future research.\n\n- Potential data leak. The benchmarks used in Tab. 1 all have released answers, which may be potentially seen during training. It is suggested to perform additional evaluation on some addition benchmarks that do not have released answers, like MMB.\n\n- Potentially poor performance on fine-grained visual perception. The clustering-based visual tokenization may be unfriendly to some fine-grained, like OCR, because clustering always fuses information for coarse representation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BR5bOkIpls", "reviewer": "Reviewer_QXWj", "summary": "This paper focuses on the design of visual tokenizer for MLLMs. Current MLLMs do not consider the visual semantic integrity when processing images into patches. To address this, the authors propose SeTok that employs a dynamic clustering strategy to groups visual features into semantic units. Furthermore, they integrate SeTok into MLLM, creating SeTokIM, and demonstrate promising results on both generation and understanding tasks.", "questions": "1. Is SeTok trainable during multimodal pre-training stage?\n2. How much additional text data needs to be added to avoid degrading the performance of LLMs?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed SeTok is interesting and valuable. \n2. The experimental details in the paper are thorough, ensuring reproducibility.", "weaknesses": "1. The performance of MLLMs is greatly influenced by SFT data and LLMs. Therefore, when comparing image understanding capabilities, it is important to make sure under the same settings, such as using the same settings as LLaVA, to better reflect the effectiveness of the proposed methods. However, there is lack of this comparison in the experiments.\n2. In the comparison of image understanding capabilities, there is a lack of evaluation on more fine-grained benchmarks such as TextVQA, OCRBench, SEED-Bench-2-Plus, as well as more general benchmarks like MMBench and SeedBench. A potential concern is that SeTok's dynamic clustering strategy might lead to the loss of fine-grained information, making it challenging to complete more difficult tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DIcVPbqs1t", "reviewer": "Reviewer_JdMJ", "summary": "In this paper, the authors propose a dynamic Semantic-Equivalent Vision Tokenizer (SeTok) to preserve semantic integrity in visual features. SeTok can produce different numbers of visual tokens for various images. The authors introduce concept-level image-text contrastive loss and image reconstruction loss to balance the high-level understanding and fine-grained detail preservation abilities of SeTok. SeTokIM, an MLLM equipped with SeTok, shows superior performances on image-text understanding and generation tasks.", "questions": "1. Why do we choose ImageNet-1K and OpenImages for reconstruction and alignment learning separately?\n2. How to do fixed clustering? k-means?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The experiments are well-conducted and quite comprehensive.\n3. The ablation studies of SeTok is quite clear and valuable, highlighting the effectiveness of the proposed components.\n4. I think that the use of dynamic clustering to generate semantic-equivalent visual tokens in MLLM is quite interesting, compared to standard operations like average pooling and query-based compression. The comparison of dynamic and fixed clustering demonstrates the superiority of this clustering mechanism.", "weaknesses": "I have some concerns regarding the comparisons in Table 1. The authors claim that SeTok is superior to Q-former or cross-attention framework. However, SeTokim and the other compared MLLMs do not share the same training scheduler, base ViT, LLM, or training datasets. A fair comparison is to evaluate various kinds of visual tokenizers in an environment with tightly controlled variables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5UEEd0mig4", "reviewer": "Reviewer_zdxE", "summary": "This paper introduces one viable semantic-equivalent tokenizer for MLLMs, namely, SeTok. It tokenizes automatically patch-level visual features into a variable number of semantic-complete concept visual tokens, further achieving accurate alignment between vision and text. \nAnd then use a unified autoregressive objective to perform two-stage optimization on SETOKIM (one MLLM). Experiments covering comprehension, generation, segmentation, and editing tasks, demonstrate the effectiveness.", "questions": "[-] Training/inference efficiency. Due to the clustering of visual tokens, the number of tokens obtained from each image is dynamic. So, how to perform parallel training on a batch, without affecting the efficiency of training and testing due to clustering?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "[+] For MLLMs, the pursuit of a more reasonable tokenizer to unify visual tasks and generate visual tokens that are accurately aligned with text semantics has always been a key challenge that deserves further research. \n\n[+] The paper is easy to follow and understand, having clear formulation.\n\n[+] Many experiments are conducted to demonstrate the idea’s performance, across comprehension, generation, segmentation, and editing tasks, showing good generality.", "weaknesses": "[-] Complex images. SeTok seems to be effective in handling simple images, but when faced with complex images, the quality of clustering seems difficult to guarantee (for example, there are dense people, objects, or significant repetition between people and objects in the image). As a result, clustering errors are injected into visual tokens, which corrupt the final output results.\n\n[-] Source of gains. For MLLMs, there are usually two types of visual bottlenecks: A. maintaining detail resolution to form hierarchical information [1]; B. one reasonable tokenizer. When demonstrating the value of the tokenizer, this paper still uses the detokenizer (in Fig. 2) to reconstruct pixel level images, i.e., keeping visual details. Can you decouple these two to demonstrate the effectiveness of each component separately? For example, report the performance with only \\mathcal{L}_{citc}, or make more comparisons with [1].\n\n[1] Eagle: Exploring the design space for multimodal llms with mixture of encoders", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)