Towards Semantic Equivalence of Tokenization in Multimodal LLM

## Abstract

Abstract Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in processing vision-language tasks.
One of the crux of MLLMs lies in vision tokenization, which involves efficiently transforming input visual signals into feature representations that are most beneficial for LLMs.
However, existing vision tokenizers, essential for semantic alignment between vision and language, remain problematic.
Existing methods aggressively fragment visual input, corrupting the visual semantic integrity.
To address this, this work presents a novel dynamic Semantic-Equivalent Vision Tokenizer ( SeTok ), which groups visual features into semantic units via a dynamic clustering algorithm, flexibly determining the number of tokens based on image complexity.
The resulting vision tokens effectively preserve semantic integrity and capture both low-frequency and high-frequency visual features.
The proposed MLLM ( Setokim ) equipped with SeTok significantly demonstrates superior performance across various tasks, as evidenced by our experimental results.
The project page is at https://chocowu.github.io/SeTok-web/ .

## 1 Introduction

Recently, the research on MLLMs has garnered intense interest (Zhang et al., ; Lin et al., ; Dong et al., ; Wu et al., ).
By building upon the unprecedented intelligence of language-based LLMs (Chiang et al., ; Touvron et al., ), and integrating multimodal encoders (Radford et al., ) at the input side and decoders (Rombach et al., ) at the output side, current MLLMs have developed powerful multimodal capabilities.
Particularly, in the visual modality, the state-of-the-art (SoTA) MLLMs have now achieved a grand slam across the four major visual-language task groups, i.e., understanding (Liu et al., ; Wu et al., ; Team, ), generating (Ge et al., ; Dong et al., ; Jin et al., ; Pan et al., ), segmenting (Ren et al., ; You et al., ; Zhang et al., ), and editing (Huang et al., ; Jin et al., ; Fu et al., ).
Central to this capability is the design of vision tokenization (Dosovitskiy et al., ; Esser et al., ; Yu et al., ), which focuses on effectively converting input visual signals into visual tokens that can be seamlessly understood by LLMs.
Existing vision tokenizers primarily produce three types of visual tokens: 1) patch-level continuous tokens (cf. Figure 1(a)), 2) patch-level discrete tokens (cf. Figure 1(b)), and 3) learnable query tokens (cf. Figure 1(c)).

Figure: Figure 1: Comparison between existing MLLMs in tokenized visual inputs: (a) patch-level continuous token, (b) patch-level discrete token, (c) learnable query token, and (d) semantic-equivalent continuous token (ours). In (e), we show four language-driven vision tasks enhanced with semantic-equivalent vision tokens, with token masks showing regions of the same color representing a single vision token.
Refer to caption: x1.png

While existing MLLMs have achieved promising performances across various tasks, a significant bottleneck remains with current visual tokenization methods, i.e., resulting in insufficient semantic alignments between language and vision tokens.
On the language side, linguistic tokens (or words) are naturally discrete, representing well-encapsulated semantic units, whereas, on the vision side, visual pixels are inherently continuous data with no physical boundaries.
Intuitively, language tokens should correspond to semantically encapsulated objects (or compositional regions) within an image.
For example, when “*a dog*” is mentioned, the “*dog*” token should correspond to the direct pixel region of the dog in the image.
However, as illustrated in Figure 1(a&b), both existing tokenization methods divide the image into fixed patch squares, fragmenting objects across multiple patches.
This disrupts the integrity of visual semantic units, resulting in a significant loss of high-frequency visual information (Zhang et al., ), e.g., the object’s edges and contours.
Moreover, methods employing a fixed number of query tokens, as depicted in Figure 1(c), struggle to align with actual visual semantic units and meanwhile offer limited interpretability (Yang et al., ; Wu et al., ).
Ultimately, this misalignment between vision and language within MLLMs undermines the effective understanding of visual signals, significantly hindering progress in a range of vision-language tasks that require precise, fine-grained semantic alignment between vision and language elements.

To this end, this work proposes a Semantic-Equivalent Tokenizer (SeTok) for enhancing MLLMs, where we encourage the vision and language tokens to be semantically congruent.
The core idea involves automatically grouping visual features from input images by applying a clustering algorithm (Engelcke et al., ), such that each unique cluster represents an encapsulated semantic unit within the vision.
As illustrated in Figure 1(d), the red visual area aggregated by SeTok corresponds to a complete semantic concept—“*person*”, while the yellow area corresponds to the “*surface board*” concept.
Furthermore, we recognize that tokenizing images into a fixed number of patches is impractical.
From a semantic perspective, different images should contain varying numbers of semantically encapsulated objects, and the granularity of compositional regions also needs to be flexibly determined.
For example, we only need to identify a person in the image, while at other times, we may need to delineate the person’s head precisely.
This implies that it is more reasonable to dynamically determine the division of visual tokenization.
To address this, we propose a dynamic clustering mechanism (Engelcke et al., ) that iteratively determines cluster centers based on density peaks, assigning visual features to these centers until all features are allocated.
The design of this mechanism allows for the dynamic determination of the number of concept visual tokens, rather than fixing the ratio (Jin et al., ) or merely merging the top-$k$ visual tokens (Bolya et al., ).
After clustering, we devise a token merger to aggregate the visual features within each cluster, that is dedicated to learning a complete visual semantic unit feature, including both high-frequency and low-frequency information.
To enable the effective learning of the semantic-equivalent token, we propose reconstructing the raw image based on these tokens, and further introducing the concept-level image-text contrastive loss to explicitly align the language and vision at the concept level.

We further build an MLLM equipped with our SeTok, named Setokim, capable of addressing four types of language-driven vision tasks simultaneously, as demonstrated in Figure 1(e).
Built on a pre-trained LLM (Touvron et al., ), Setokim performs reasoning on a unified multimodal sequence concatenating text token with visual tokens generated by Setok.
During inference, Setokim yields the text and visual tokens autoregressively, which are then processed by a visual detokenizer and mask decoder to produce images and corresponding masks.
Inspired by Li et al. (), we introduce a unified autoregressive training objective for optimization through pre-training and instruction-tuning on the massive multimodal data.
We evaluate Setokim on various common visual language tasks, including visual question-answering, image generation & editing, and referring segmentation.
Our results reveal that semantic-equivalent tokenization significantly enhances vision-language learning compared to standard patch-level tokenization or learnable queries, achieving higher performance on various tasks.
Meanwhile, in-depth analyses and visualizations intuitively show SeTok enjoys the superiority of tokenizing vision input into more interpretable and semantic-complete vision tokens, achieving fine-grained vision-language alignment.

## 2 Methodology

In this work, we aim to generate semantically complete vision tokens aligned with text tokens to facilitate fine-grained semantic interactions between vision and language, thereby enhancing the performance of MLLMs in various multimodal tasks.
In pursuit of this goal, we propose constructing a semantic-equivalent tokenizer, called SeTok, which tokenizes the given input image into a sequence of semantically complete visual tokens, as illustrated in Figure 2.
Integrated with the SeTok, we further design a multimodal large language model, i.e., Setokim shown in Figure 3, where the semantic-equivalent vision tokens concatenated with text tokens are fed into LLMs for interleaved image-text understanding and generation.

Figure: Figure 2: Overview of SeTok. SeTok tokenizes visual features extracted from an image by a vision encoder into semantically equivalent vision tokens, which then are fed into a detokenizer to reconstruct the image and meanwhile employed to perform the concept-level image-text alignment.
Refer to caption: x2.png

### 2.1 Semantic-equivalent Vision Tokenizer

Given an input image $I\in\mathbb{R}^{H\times W\times 3}$, we first employ a vision encoder to extract a sequence of visual patch embeddings $\bm{X}=\{\bm{x}_{i,j}\}\in\mathbb{R}^{h\times w\times d}$, where $d$ is the embedding dimension (^1^11When using ViT-based vision encoder, $h=\frac{H}{p},w=\frac{W}{p}$, where $p$ is the patch size. Similarly, $p$ denotes the downsampling factor when using a CNN-based encoder.).
Then, to obtain semantically complete visual tokens, we propose to amalgamate the visual embeddings into concept-like scene components by a Token Cluster.

#### Token Cluster.

We take the visual patch embeddings $\bm{X}$ as input and then assign individual patches into a semantic complete cluster, which can be formulated as obtaining a variable number of concept masks $\bm{M}\in[0,1]^{h\times w\times C}$, with $\sum_{c}\bm{M}_{i,j,c}=1$ for all patch coordinate
tuples $(i,j)$ in an image, where $C$ is the number of semantic-equivalent tokens.
Inspired by Engelcke et al. (), this is intuitively achieved by (1) selecting the location $(i,j)$ of the visual patch feature that has not yet been assigned to a cluster, (2) creating a cluster assignment according to the distance of the embeddings at the selected location to all other embeddings according to a distance kernel $\varphi(\cdot)$ (^2^22In this work, we employ $\varphi(\bm{u},\bm{v})=\text{exp}(-\|\bm{u}-\bm{v}\|^{2}\cdot C\ln{2})$), and (3) repeating the first two steps until all visual embeddings are accounted for or a stopping criterion is met.
Different from (Du et al., ) employing uniformed seed scores performing the stochastic selection of visual embeddings, we propose to choose the visual embeddings based on their density peaks, as a higher density shows a higher potential to be the cluster center.
Specifically, we first calculate the local density $\rho_{i,j}$ of the token $\bm{x}_{i,j}\in\bm{X}$ by referring its neighbors:

$$ $\rho_{i,j}=\text{exp}(-\frac{1}{K}\sum_{\bm{x}_{m,n}\in\text{KNN}(\bm{x}_{i,j} ,\bm{X})}\varphi(\bm{x}_{m,n},\bm{x}_{i,j})),$ (1) $$

where $\text{KNN}(\bm{x}_{i,j},\bm{X})$ denotes the $K$-nearest neighbors of $\bm{x}_{i,j}$ in $\bm{X}$.
We then measure the minimal distance $\delta_{i,j}$ between the feature $\bm{x}_{i,j}$ and other features with higher density:

$$ $\delta_{i,j}=\begin{cases}\min_{m,n:\rho_{m,n}>\rho_{i,j}}\;\varphi(\bm{x}_{m, n},\bm{x}_{i,j}),&\text{if }\exists\;m,n:\rho_{m,n}>\rho_{i,j}\\ \max_{m,n}\;\varphi(\bm{x}_{m,n},\bm{x}_{i,j}),&\text{otherwise}\end{cases}$ (2) $$

Finally, we summarize the score $s_{i,j}$ of the feature by combining the local density $\rho_{i,j}$ and minimal distance $\delta_{i,j}$ as $\rho_{i,j}\times\delta_{i,j}$.
Based on the score, we select the location $(i,j)$ of the visual feature that has the highest score $s_{i,j}$ and has not yet been assigned to a cluster and iteratively assign the visual feature into a certain cluster until a stopping condition is satisfied, at which point the additional mask is added for any remaining visual embeddings.
The detailed algorithm is described in Appendix $\S$C.1.

#### Token Merger.

After clustering, the visual embeddings are grouped based on the attention masks $\bm{M}$.
To optimally retain information within each cluster, we adopt a token merger that aggregates visual embeddings beyond merely using cluster centers as definitive vision tokens.
In addition, considering the significance of positional information for representing a semantic concept in an image, we integrate 2D position embeddings (PE, Heo et al. ()) into the merger, calculated as $\hat{\bm{X}}_{c}=\text{PE}(\bm{X})\odot\bm{M}_{c}\oplus\bm{X}\odot\bm{M}_{c}$.
Then, we apply $L^{\text{inner_c}}$ Transformer layers on all the visual embeddings within a cluster, followed by an average pooling to obtain the final token feature $\bm{u_{c}}=\text{Avg}(\text{Transformer}(\bm{\hat{X}}_{c}),L^{\text{inner_c}}
)\in\mathbb{R}^{d}$.
To facilitate the representation of coherent scenes with semantic equivalent tokens, we add inter-cluster Transformer layers to model relationships between vision tokens, i.e., $\bm{V}=\{\bm{v}_{1},\cdots,\bm{v}_{C}\}=\text{Transformer}(\{\bm{u}_{1},\cdots
,\bm{u}_{C}\},L^{\text{inter_c}})\in\mathbb{R}^{C\times d}$.

#### SeTok Training.

To facilitate diverse visual understanding and generation tasks when building MLLMs, we argue that effective semantic-equivalent tokens should embody two key attributes: complete and enriched high-level semantic information, and undistorted pixel-level details.
Therefore, we propose to include concept-level image-text contrastive loss and image reconstruction loss, as shown in Figure 2.
During the training phase, to ensure each token’s semantic independence and completeness, we adopt a concept-level image-text conservative loss, inspired by Xu et al. ().
This loss aligns visual tokens with corresponding textual concepts semantically, thereby enhancing their suitability for integration in LLMs.
Additionally, to ensure the tokens retain adequate pixel-level details, we feed these tokens into a detokenizer (Yu et al., ) to reconstruct the original image and calculate the reconstruction loss.
Finally, we employ a weighted sum to combine the conservative loss and reconstruction loss, optimizing both semantic fidelity and visual detail retention:

$$ $\mathcal{L}_{setok}=\alpha\mathcal{L}_{rec}+\beta\mathcal{L}_{citc}.$ (3) $$

In the experiments, we set the hyper-parameter $\alpha$ and $\beta$ to 1.
We employ ImageNet-1K (Deng et al., ) and OpenImages (Kuznetsova et al., ) for reconstruction and alignment learning.

### 2.2 Setokim

Figure: Figure 3: The overview of SeTokim.
Refer to caption: x3.png

Upon acquiring SeTok, we propose to integrate it with the pre-trained LLM to construct an MLLM, i.e., Setokim.
The overall framework is depicted in Figure 3.
The input image will be tokenized into a sequence of semantic-equivalent visual tokens by SeTok, which are then concatenated with text tokens to form a unified multimodal sequence.
To effectively distinguish between modalities and facilitate visual content generation, two special tokens, ‘[Img]’ and ‘[/Img]’ are introduced to signify the beginning and the end of the visual sequence, respectively.
The backbone LLM subsequently processes this multimodal sequence to perform multimodal understanding and generation.
The output vision tokens are then fed into the visual detokenizer to restore the images.
Meanwhile, we observe that the generated concept-centric tokens inherently embed approximate locations of each concept within the original image, as illustrated in Figure 7.
To exploit this spatial and semantic encoding, we incorporate a lightweight mask decoder (Kirillov et al., ) utilizing the generated vision tokens as input to yield the referring mask.
The detailed implementations are provided in the Appendix $\S$D.

#### Training Objectives.

To facilitate autoregressive modeling across both text and visual generation, we unified adopt a next-token prediction:

$$ $p(y_{1},\cdots,y_{n})=\prod_{i=1}^{n}p(y_{i}|y_{1},\cdots,y_{i}).$ (4) $$

Specifically, in terms of text generation, we adopt the Cross-entropy loss $\mathcal{L}_{text}$, i.e., the standard language modeling objective, to maximize the likelihood of text tokens.
For image generation, drawing inspiration from Li et al. (), we utilize the LLM, to produce a conditioning vector $z_{i}$ based on previous tokens: $z_{i}=\text{LLM}(y_{1},\cdots,y_{i-1})$.
We then model the probability of the next token by $p(y_{i}|z_{i})$, and employ the Diffusion loss (Li et al., ), denoted $\mathcal{L}_{vis}$, to optimize the parameters of LLM.
Moreover, we follow Li et al. () to use binary cross-entropy loss $\mathcal{L}_{bce}$ and dice loss $\mathcal{L}_{dice}$ for optimizing the parameters in mask decoder.

#### Training Receipts.

Given that we try to perform generation and understanding with one single generative model, large-scale pretraining is required to achieve effective alignment between textual and visual content.
To this end, we propose a two-stage training procedure.
Stage-I: Multimodal Pretraining.
In this stage, we focus on enhancing the alignment between text and image.
We employ massive multimodal data, including ImageNet-1K and 28M text-image pair dataset, to train our model for conditional image generation and image captioning.
Furthermore, we utilize the English text corpus from the SlimPajama (Soboleva et al., ) dataset to reduce catastrophic forgetting of the reasoning capacity during LLM training.
Toward the end of the first phase of training, once the trainable modules of Setokim have converged, we freeze these modules and exclusively train the mask decoder on the segmentation datasets, like MSCOCO (Lin et al., ), to promote the learning of fine-grained object boundaries.
Stage-II: Instruction Tuning.
Building upon the pre-trained weights, we further perform multimodal instruction tuning with both public datasets covering multimodal instruction datasets (e.g., ALLaVA (Chen et al., ) and LLaVA-665K (Liu et al., ) ), fine-grained visual QA (e.g., VQA^v2 (Goyal et al., ), GQA (Hudson & Manning, ), OK-VQA (Marino et al., ), A-OKVQA (Schwenk et al., )), image generation (e.g., LAION-aesthetics (Schuhmann et al., )), and editing (e.g., InstructPix2Pix (Brooks et al., ) and Magicbrush (Zhang et al., )).
More details can be found in the Appendix $\S$D.3.

**Table 1: Comparison of MLLMs on image understanding benchmarks. ^∗ indicates the training sets observed during training. “C” and “D” represent continuous and discrete visual tokens, respectively. “P” refers to patch-level features, “Q” denotes learnable queries, and “SE” is semantic-equivalent.**
| Method | Size | Vis. Tok. | Flickr30K | VQA^v2 | OK-VQA | GQA | POPE | MME | MM-Vet |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| InstructBLIP (Liu et al., ) | 13B | Q.C | - | - | - | 49.5 | 78.9 | 1212.8 | - |
| Qwen-VL-Chat (Bai et al., ) | 7B | Q.C | - | 78.2* | - | 57.5* | - | 1487.5 | - |
| Emu (Zhang et al., ) | 7B | Q.C | 77.4 | 57.2 | 43.4 | - | - | - | - |
| DreamLLM (Dong et al., ) | 7B | P.C | - | 72.9 | - | 41.8 | - | - | 36.6 |
| LLaVA-1.5 (Liu et al., ) | 7B | P.C | - | 78.5* | - | 62.0* | 85.9 | 1510.7 | 33.1 |
| NExT-GPT (Wu et al., ) | 7B | P.C | 84.5 | 66.7 | 52.1 | - | - | - | - |
| SEED-X (Ge et al., ) | 17B | P.C | 52.3 | - | - | 47.9 | 84.2 | 1435.7 | - |
| LaVIT (Jin et al., ) | 7B | P.C | 83.0 | 66.0 | 54.6 | 46.8 | - | - | - |
| Unified-IO-2 (Lu et al., ) | 6.8B | P.D | - | 79.4* | - | - | 87.7 | - | - |
| CM3Leon (Yu et al., ) | 7B | P.D | - | 47.6 | 23.8 | - | - | - | - |
| Chameleon (Team, ) | 34B | P.D | 74.7 | 66.0 | - | - | - | - | - |
| \cdashline1-10<br>Setokim | 7B | SE.C | 86.9 | 78.7* | 60.2* | 65.6* | 89.1 | 1537.8 | 45.2 |

## 3 Settings

Our experiments employ the LLaMA-2-7B (Touvron et al., ) to initialize our LLM backbone.
For the SeTok, we apply pre-trained SigLIP-SO400M-patch14-384 (Zhai et al., ) as our vision encoder, and the numbers of inner-cluster and inter-cluster transformer layers are set as 12, and 8, respectively.
The dimension of the semantic-equivalent token is 512.
For the detokenizer, we adopt $L=12$ blocks for mask tokens to query visual information contained in
the semantic-equivalent tokens, and then an upsampler inspired by the architect of Yu et al. () is employed as the pixel decoder.
More implementation details are provided in Appendix $\S$D.1.

For examining visual understanding ability, we evaluate our model on Flicker30K (Young et al., ), VQA^v2(Goyal et al., ), GQA (Hudson & Manning, ), OK-VQA (Marino et al., ), as well as three MLLM benchmarks, e.g., POPE (Li et al., ), MME (Fu et al., ) and MM-Vet (Yu et al., ).
Besides, we evaluate the visual generation fidelity on the MSCOCO (Lin et al., ) dataset.
Following Pan et al. (), we evaluate the image editing capabilities of the Sektoim on Magicbrush (Zhang et al., ), EVR (Tan et al., ) and MA5K (Shi et al., ).
Furthermore, refCOCOg (Mao et al., ), refCOCO+ (Yu et al., ), and Reaseg (Lai et al., ) are utilized to examine the potential referring segmentation capabilities of the Sektoim.

**Table 2: Performance of various models in zero-shot text-to-image generation and editing on benchmarks. *: denotes editing performances sourced from (Pan et al., ). “LLM (Cont.)” means LLM outputs continuous representation utilized in the decoder to generate images, while “LLM (Disc.)” stands for discrete representation generated for image generation.**
| Method | Type | Decoder | MS-COCO | MagicBrush | MA5K | EVR |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FID $\downarrow$ | CLIP${}_{\text{im}}\uparrow$ | L${}_{1}\downarrow$ | LPIPS$\downarrow$ | L${}_{1}\downarrow$ | CLIP${}_{\text{im}}\uparrow$ | L${}_{1}\downarrow$ |  |  |  |
| $\bullet$ Diffusion-based Method |  |  |  |  |  |  |  |  |  |
| Make-A-Scene (Gafni et al., ) | Autoregressive | - | 11.8 | - | - | - | - | - | - |
| Ins.P2P* (Brooks et al., ) | Diffusion | - | - | 83.4 | 12.1 | 35.9 | 17.6 | 81.4 | 18.9 |
| SD v2.1 (Rombach et al., ) | Diffusion | - | 9.0 | - | - | - | - | - | - |
| \cdashline1-9   $\bullet$ MLLM-based Method |  |  |  |  |  |  |  |  |  |
| DreamLLM (Dong et al., ) | LLM (Cont.) | SDv2.1 | 8.7 | - | - | - | - | - | - |
| SEED-X (Ge et al., ) | LLM (Cont.) | SDXL | 14.9 | - | - | - | - | - | - |
| CM3Leon (Yu et al., ) | LLM (Disc.) | VQGAN | 10.3 | - | - | - | - | - | - |
| LaVIT* (Jin et al., ) | LLM (Disc.) | SDv1.5 | 7.4 | 81.1 | 25.3 | 36.9 | 25.1 | 73.8 | 26.8 |
| LWM (Liu et al., ) | LLM (Disc.) | VQGAN | 12.6 | - | - | - | - | - | - |
| MGIE* (Fu et al., ) | LLM (Cont.) | SDv1.5 | - | 91.1 | 8.2 | 29.8 | 13.3 | 81.7 | 16.3 |
| Emu-2-gen* (Sun et al., ) | LLM (Cont.) | SDXL | - | 85.7 | 19.9 | 28.4 | 20.5 | 80.3 | 22.8 |
| Morph-Token* (Pan et al., ) | LLM (Cont.) | VQGAN | - | 87.9 | 7.6 | 27.9 | 14.6 | 82.6 | 15.3 |
| \cdashline1-9<br>Setokim | LLM (Cont.) | SeTok | 8.3 | 89.6 | 6.3 | 26.4 | 15.7 | 83.5 | 14.1 |

## 4 Experimental Results

### 4.1 Main Results

**Table 3: Reconstruction results (rFID) and image classification performance (Top-1 Accuracy) on $256\times 256$ ImageNet(val.) dataset. #Tokens refers to the number of tokens.**
| Model | #Tokens | Latent size | rFID $\downarrow$ | Top-1 $\uparrow$ |
| --- | --- | --- | --- | --- |
| VQ-GAN (Esser et al., ) | Fixed | 16 $\times$ 16 | 7.94 | - |
| VAE (Rombach et al., ) | Fixed | 32 $\times$ 32 | 2.63 | - |
| RQ-VAE (Lee et al., ) | Fixed | 16 $\times$ 16 | 3.20 | - |
| ViT-VQGAN (Yu et al., ) | Fixed | 32 $\times$ 32 | 1.28 | - |
| MQ-VAE (Huang et al., ) | Fixed | 32 $\times$ 32 | 5.29 | - |
| TiTok (Yu et al., ) | Fixed | 32 $\times$ 1 | 2.21 | 72.6 |
| \cdashline1-5<br>SeTok | Dynamic | - | 2.07 | 76.4 |

#### The Quality of SeTok.

We employ reconstruction FID (rFID) and Top-1 accuracy for image classification on ImageNet to measure the reconstruction and text alignment capabilities of the SeTok in Table 3.
SeTok can achieve a comparable reconstruction quality to well-trained VQ models.
Unlike prior methods that typically utilize 2D latent grids preserving spatial mappings between latent tokens and image patches, which allows for the retention of precise low-level information but limits high-level semantic acquisition and development of more compressed latent space, SeTok integrates both high- and low-level information that is crucial for producing high-quality images and creating semantic compact and complete latent representations.
In comparison, the latest models like TiTok utilize a fixed number of 1D latent representations that suffer from a lack of semantic interpretability and poor textual alignment, i.e., obtaining inferior image classification performance (72.6 vs 76.4 top-1 accuracy).
We visualize the visual token in Section 4.2, and more reconstruction examples can be found in Appendix $\S$E.

#### Visual Understanding.

We evaluate the visual understanding capabilities of our model and other leading MLLMs across a wide range of benchmarks, as detailed in Table 1.
Different from the prevalent use of patch-level continuous visual tokens by foundational models like CLIP, the discrete tokens utilized in VQGAN models show weaker semantic alignment with text, which detracts from their performance in various understanding tasks.
Besides, learnable continuous queries transformed via Q-former or cross-attention framework are introduced to alleviate the efficiency issues.
However, these methods still struggle with fine-grained semantic alignment with text, potentially limiting the depth of interaction between textual and visual content.
By incorporating semantic-equivalent tokens via SeTok, our model secures competitive performances in various vision-understanding tasks.
Moreover, our model demonstrates performance improvement on GQA by 3.6%, highlighting our method’s superior capability in complex relationships and object quantities reasoning.

**Table 4: Results on 3 referring expression segmentation benchmarks. We report cIoU for RefCOCO+/g.**
| Method | refCOCOg | refCOCO+ | Reaseg |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| val(U) | test(U) | val | testA | testB | gIoU | cIoU |  |
| ReLA | 65.0 | 66.0 | 66.0 | 71.0 | 57.7 | - | - |
| SEEM | 65.7 | - | - | - | - | 24.3 | 18.7 |
| \cdashline1-8<br>PixelLM | 69.3 | 70.5 | 66.3 | 71.7 | 58.3 | - | - |
| NExT-Chat | 67.0 | 67.0 | 65.1 | 71.9 | 56.7 | - | - |
| LISA | 67.9 | 70.6 | 65.1 | 70.8 | 58.1 | 47.3 | 48.4 |
| \cdashline1-8<br>Setokim | 71.3 | 71.3 | 68.0 | 72.4 | 61.2 | 50.7 | 52.7 |

**Table 6: Ablation Study on SeTok to image reconstruction, visual understanding, and generation.**
| Method | ImageNet | Flickr30K | VQA^v2 | GQA | MSCOCO |
| --- | --- | --- | --- | --- | --- |
| (rFID$\downarrow$) | (CIDEr$\uparrow$) | (Accuracy$\uparrow$) | (Accuracy$\uparrow$) | (FID$\downarrow$) |  |
| SeTok | 2.07 | 86.9 | 78.5 | 65.6 | 8.3 |
| \cdashline1-6<br>w/o $\mathcal{L}_{citc}$ | 4.15 | 78.1 | 65.8 | 49.7 | 9.6 |
| w/o PE | 3.56 | 86.1 | 76.2 | 61.4 | 12.8 |
| w/o inter-cluster Transformer | 7.91 | 82.7 | 71.4 | 54.2 | 13.9 |
| w/o inner-cluster Transformer | 6.25 | 85.4 | 73.7 | 53.4 | 11.0 |
| w/o Token Merger | 8.64 | 80.3 | 66.1 | 50.5 | 14.7 |

#### Visual Generation and Editing.

Table 3 demonstrates a comparative analysis of Setokim and other diffusion-based and LLM-based methods in vision generation and editing.
Notably, compared to other MLLMs integrated with advanced vision decoders such as SD v2.1 (Rombach et al., ) and SD-XL (Podell et al., ), our method achieves comparable performance on complex prompts.
This highlights the effectiveness and efficiency of SeTok in learning the correlations between visual and textual modalities within our unified framework.
Further evaluations on instruction-based image editing are conducted.
Standard pixel difference (L1), LPIPS (Zhang et al., ), and visual feature similarity (CLIP${}_{\text{im}}$) are employed as metrics.
Our model exhibits marked superiority in L1 and CLIP scores compared to existing MLLMs.
This enhanced performance can be attributed to SeTok’s ability to capture semantically equivalent visual tokens, thereby enhancing the semantic interaction between text and images.
Moreover, editing tasks typically involve conceptual replacements within images, and the concept-level token representations learned by our model are inherently well-suited to such tasks involving straightforward replacements or modifications.

#### Referring Expression Segmentation.

Table 4 presents non-LLM-based methods (i.e., ReLA (Liu et al., ), and SEEM (Zou et al., )) and LLM-based methods (i.e., PixelLM (Ren et al., ), NExT-Chat Zhang et al. () and LISA Lai et al. ()) performances on referring expression segmentation tasks.
Our model consistently outperforms the current SoTA on the RefCOCO+/g and ReaSeg dataset, demonstrating the proficiency of our vision tokens derived from SeTok in capturing not only object-centric semantic details but also high-frequency boundary information.

### 4.2 In-depth Analysis and Qualitative Evaluation

#### Ablation Study.

Table 6 summarizes the results of an ablation study evaluating the design benefits of SeTok and the influence of Setokim across various vision-language tasks.
Firstly, we observe that while the model can achieve commendable reconstruction quality without using contrastive loss, its performance markedly decreases in downstream vision understanding tasks. This suggests that exclusive reliance on reconstruction learning may cause the model to prioritize low-level information at the expense of high-level semantic insights.
Furthermore, replacing the token merger with a simple average visual representation for each cluster also results in a significant decline in fine-grained visual understanding and generation performance, possibly due to the averaging process potentially leading to information loss.
Lastly, the removal of positional encoding (PE) and both the inner-cluster and inter-cluster transformers degrade the model’s performance across various tasks to some extent.

#### The Impact of the Clustering Mechanism.

Here, we compare the impact of different clustering mechanisms on model performance.
As shown in Table 5, we can observe that tokenizers constructed using dynamic clustering mechanisms achieve superior overall performance compared to those with a fixed setup while simultaneously accelerating training time and reducing computational costs during inference.
In contrast to soft-clustering, which yields soft attention masks, our findings suggest that hard-clustering produces better results, as it may be because hard clustering leads to higher consistency of cluster outcomes (Haurum et al., ), leading to more stable visual tokens and enhancing both the stability and performance of the model.
When employing a fixed number of clusters, the critical challenge is to determine the optimal number of clusters.
As demonstrated in Table 5, different datasets achieve optimal performance at varying numbers of clusters, with a uniform count across all datasets, resulting in suboptimal outcomes.

Figure: Figure 4: Qualitative results on image understanding and generation. The words marked in green are key elements in questions and answers. Best view it on screen.
Refer to caption: x4.png

#### Qualitative Analysis of Visual Understanding and Generation.

As illustrated in Figure 4, our model exhibits proficiency in intricate image understanding tasks, such as deciphering reversed text, exemplified by the word “stop”, and accurately identifying text “A NEW EXPERIENCE COMING YOUR WAY” that is partially covered.
In tasks involving detailed image descriptions, our approach prioritizes object-level information within images, which substantially mitigates the incidence of hallucinatory responses commonly observed in MLLMs.
Moreover, in text-to-image generation, our model demonstrates remarkable capabilities in synthesizing coherent images, which maintain high fidelity and relevance to the textual context, such as the “flower”, “fence” and “squirrel”.

#### Qualitative Analysis of Visual Segmentation.

We present the segmentation examples in Figure 5.
It is easy to note that the attention mask closely aligns with the object mask, and our model shows superiority in achieving more accurate and detailed segmentation results than other LLM-based segmentation methods.
Notably, as depicted in the second row of this figure, the visual token generated by our method encompasses all depicted fish, effectively achieving a complete segmentation of the fish in the scene.
In contrast, other models produce only partial segmentation.
This effectiveness of the segmentation highlights the precise content representation and improved interpretability of the visual tokens.
Such visual tokens can eventually enhance the vision-language understanding incorporated with the text tokens.

Figure: Figure 5: The visualizations for segmentation results compared with GLaMM (Rasheed et al., ) and Osprey (Yuan et al., ).
Refer to caption: x5.png

Figure: Figure 6: Qualitative comparison between MLLMs for the image editing. Setokim excels in adhering to instructions and preserving low-level image details.
Refer to caption: x6.png

#### Qualitative Analysis of Visual Editing.

Here, we evaluate the efficacy of image manipulation using our model compared to the previous diffusion-based method MagicBrush (Zhang et al., ), and various MLLMs including Emu-2-Gen (Sun et al., ), MGIE (Fu et al., ), and Mini-Gemini (Li et al., ).
As depicted in Figure 6, Setokim displays superior performance by closely adhering to the provided instructions and preserving intricate image details.
For instance, our model seamlessly adds “tomato slices” to an image without altering other elements on the pizza, while Emu-2-Gen and MGIE fall short.
Furthermore, our model exhibits remarkable precision in changing the color of an umbrella, while visual objects not intended for alteration retain a high level of consistency before and after editing.
Additionally, Setokim demonstrates to precisely follow implicit user instructions to remove unusual elements from an image, i.e., the banana, preserving the surrounding context, whereas Emu-2-Gen mistakenly removes a telephone cord and MGIE fails to remove the banana properly, altering the cord’s texture.
These examples underscore the effectiveness of Setokim for high-precision image manipulation, leveraging semantically equivalent visual tokens to achieve nuanced and context-aware results.

Figure: Figure 7: Token mask $\bm{M}$ visualization of visual tokens generated by SeTok.
Refer to caption: x7.png

#### Qualitative Analysis of Visual Tokens.

In Figure 7, we demonstrate how input visual features are assigned to visual tokens after tokenization.
First, we observe that our tokenization process resembles partial segmentations, producing semantically complete units.
For example, in the second image, visual tokens correspond to distinct elements such as the giraffe, grass, tree, and background, aligning with semantic intuition.
Second, the number of tokens obtained from Setok is dynamic and not fixed.
Third, SeTok is capable of adapting to different levels of semantic granularity for the same concept, as seen in images (4) and (5), where the person is represented as a single token.
In contrast, in the image (1), the person is divided into tokens for the head, body, and legs.
Lastly, in complex scenes, such as the image (7), SeTok can still tokenize elements like traffic lights and billboards into semantically complete tokens. Overall, our approach ensures that similar visual features are consistently recognized and processed, improving both coherence and efficiency in tokenization.

## 5 Related Work

Currently, benefiting from the emergent phenomenon, LLMs have demonstrated near-human-level intelligence in language processing (Chiang et al., ; Touvron et al., ; Taori et al., ).
Simultaneously, researchers have been attempting to develop MLLMs by integrating multimodal encoders and decoders into LLMs (Dong et al., ; Koh et al., ; Lu et al., ; Li et al., ; Sun et al., ; ; Fei et al., ; Zhang et al., ).
From the initial MLLMs that could only understand multimodal input signals (Liu et al., ; ) to later versions supporting the generation of multimodal contents (Sun et al., ; ; Koh et al., ; Wu et al., ), MLLMs have shown powerful capabilities and a broader range of applications.
Among all modalities, the integration of vision, known as visual MLLM, has received the most extensive research and application (Gao et al., ; Schwenk et al., ; Liu et al., ; Lu et al., ).
The latest MLLM research has not only achieved both understanding and generation of visual content but also developed more refined, pixel-level visual modeling, including segmentation and editing functions (Yuan et al., ; Rasheed et al., ; Zhang et al., ; You et al., ; Lai et al., ).

On the other hand, an increasing body of research indicates that visual tokenization (Dosovitskiy et al., ; Ge et al., ; Jin et al., ; Li et al., ; ) significantly impacts MLLM capabilities in vision tasks.
The fundamental approach involves encoding the input visual content into feature representations via a visual encoder (e.g., Clip-VIT Radford et al. ()) and mapping these to an LLM, thus enabling a language-based LLM to understand vision.
The corresponding method involves patchifying the original visual images of various sizes into smaller fixed-size patches (Dosovitskiy et al., ; Bavishi et al., ; Liu et al., ; Sun et al., ), treating these as tokens, and encoding each patch/token to obtain corresponding embeddings, which are then fed into the LLM.
Subsequent research (Jin et al., ; Ge et al., ), aiming further to unify the training objectives of language and visual modalities by introducing codebook techniques, where visual elements are represented as discrete tokens.
This allows visual training to be treated similarly to language training, i.e., conducting *next token prediction* (Ge et al., ).
Unfortunately, whether in the above visual encoding or tokenization techniques, there is a significant bottleneck of MLLM performance: the integrity of visual semantic units, either visual objects or compositional regions, is compromised during the patchifying process.
This results in a less effective semantic alignment between vision and language within the LLM.
This paper is the first to propose a solution to this problem, introducing a novel Semantic Equivalent Tokenization for MLLM.

In addition, this work is also related to scene decomposition (Yang et al., ; Niu et al., ; Locatello et al., ), which involves segmenting a scene into objects. Typically, these methods use a fixed number of query tokens (Kirillov et al., ; Suzuki, ) and apply cross-attention (Yang et al., ; Qi et al., ) to aggregate visual features implicitly. However, this fixed-token approach may not only correspond to the actual visual content but also requires complex network architectures (Caron et al., ; Gansbeke et al., ) and extensive data for optimization.
When combined with LLMs, such complexity significantly increases computational resource demands. Conversely, we learn a dynamic number of semantic objects and do not require complex model structures for optimization, thereby enhancing resource efficiency and providing a more adaptable solution for integrating visual and language modalities.

## 6 Conclusion

In this paper, we introduce SeTok, a viable semantic-equivalent tokenizer, that enables to tokenize automatically patch-level visual features into a variable number of semantic-complete concept visual tokens.
Then, we integrate SeTok with a pre-trained LLM to build an MLLM, Setokim, optimized using a unified autoregressive objective and a two-stage training strategy.
Extensive experiments demonstrate that our model performs better on a broad range of comprehension, generation, segmentation, and editing tasks, highlighting the effectiveness of Setok.

## Appendix A Ethic Statement

This work aims to build semantic equivalence tokenization to segment input images into semantic complete tokens to enhance the MLLMs in vision understanding, generation, segmentation, and editing capabilities.
Here we discuss all the possible potential impacts of Setokim.

#### Use of Generative Content.

The Setokim, limited by the quantity of fine-tuning data and the quality of the base models, may generate some low-quality content.
Also, as a generative model, the LLM will produce hallucinated content in multimodal formats that may be harmful to society.
We have reminded users to interpret the results with caution.
Anyone who uses this LLM should obey the rules in a license.
And also commercial use of our system is not allowed.

#### Data Privacy and Security.

Our research utilizes datasets that are either publicly available or collected with explicit consent. We adhere to strict data privacy and security protocols to protect the information and ensure it is used solely for this research.

#### Bias Mitigation.

Recognizing the potential for bias in AI models, particularly in vision-language tasks, we rigorously test our tokenizer across diverse datasets. This approach is designed to identify and mitigate biases that may affect the model’s performance or lead to unfair outcomes in its applications.

## Appendix B Limitation

While Setokim has achieved further improvements across various language-driven vision tasks, becoming a zero-shot general specialist, it still faces several limitations.

#### Model Scale.

The evaluation of our model is currently constrained to configurations with 7B parameters.
As shown in (Laurençon et al., ), the performance of MLLMs is limited by the scale of the core backbone LLM.
Despite the impressive results achieved, the potential benefits of employing significantly larger models, such as 65B or 130B, are worth exploring in future studies.

#### The Resolution of Image.

Regarding image resolution, our model currently handles images up to a maximum resolution of 384$\times$384.
While there have been improvements in understanding visually fine-grained content, challenges remain when processing higher-resolution images.
Therefore, enhancing the model to support images of any resolution could further improve its visual perception capabilities.

#### Hallucination.

Although our model has made some progress in mitigating hallucination through fine-grained vision-language alignment, as demonstrated in experiments on the POPE dataset, hallucinations remain inevitable. This area continues to pose challenges and is crucial for future exploration and enhancement.

## Appendix C Detailed Method

### C.1 Token Cluster

The formal token clustering algorithm is described in Algorithm 1.
Specifically, a scope $\bm{z}=[0,1]^{h\times w}$ is initialized to a matrix of ones $\bm{1}^{h\times w}$ to track the degree to which visual embeddings have been assigned to clusters.
In addition, the seed scores are initialized by combining the local density in Eq.(1) and distance in Eq.(2) to perform the selection of visual embeddings.
At each iteration, a single embedding vector $\bm{x}_{i,j}$ is selected at the spatial location $(i,j)$ which corresponds to the argmax of the element-wise multiplication of the seed scores and the current scope.
This ensures that cluster seeds are sampled from pixel embeddings that have not yet been assigned to clusters.
An alpha mask $\alpha_{c}\in[0,1]^{h\times w}$ is computed as the distance between the cluster seed embedding $\bm{x}_{i,j}$ and all individual pixel embeddings according to a distance kernel $\varphi$.
The output of the kernel $\varphi$ is one if two embeddings are identical and decreases to zero as the distance between a pair of embeddings increases.
Additionally, a negative penalty $\beta\bm{s}$ is applied to the alpha mask by misusing the seed scores, where $\beta$ is a hyper-parameter.
This encourages the selection of elements similar to the current feature with lower information density.
The associated concept mask $\bm{M}_{c}$ is obtained by the element-wise multiplication of the alpha masks by the current scope.
An element-wise multiplication with the complement of the alpha masks then updates the scope.
This process is repeated until a stopping condition is satisfied, at which point the final scope is added as an additional mask to explain any remaining embeddings.

Figure: Algorithm 1 Token Clustering Algorithm

### C.2 Concept-level Image-text Contrastive Loss

To enable effective visual concept token learning, we propose concept-level image-text contrastive loss.
Specifically, we randomly select K objects in the image, acquire the corresponding object labels, and then prompt each of them with a set of handcrafted sentence templates, e.g., ‘A photo of a {object label}.’
The motivation for selecting objects is that they are the smallest units of image representation with complete semantics and have a corresponding relationship with the semantic units in the text.
Next, we employ contrastive
losses between the new sets of image-‘prompted text’ pairs $\{(I,{T_{1}}),(I,{T_{2}}),\cdots,({I},{T_{K}})\}$ where $\{{T_{k}}\}_{k=1}^{K}$ are all prompted sentences generated from the objects sampled from the image $I$.
Among the batch $B$, each image has $K$ positive text pairs and $B(K-1)$ negative pairs.
Similarly to the standard image-text contrastive loss (Radford et al., ), we define the concept-level image-text contrastive loss as a sum of two two-way contrastive losses:

$$ $\displaystyle\mathcal{L}_{I\rightarrow\{T_{k}\}_{k=1}^{K}}$ $\displaystyle=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\sum_{k=1}^{K}\exp(\bm{V}_{i }^{I}\cdot\bm{V}_{i}^{T_{k}}/\tau)}{\sum_{k=1}^{K}\sum_{j=1}^{B}\exp(\bm{V}_{i }^{I}\cdot\bm{V}_{j}^{T_{k}}/\tau)},$ (5) $\displaystyle\mathcal{L}_{\{T_{k}\}_{k=1}^{K}\rightarrow I}$ $\displaystyle=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\sum_{k=1}^{K}\exp(\bm{V}_{i }^{T_{k}}\cdot\bm{V}_{i}^{I}/\tau)}{\sum_{k=1}^{K}\sum_{j=1}^{B}\exp(\bm{V}_{j }^{T_{k}}\cdot\bm{V}_{i}^{I}/\tau)},$ (6) $\displaystyle\mathcal{L}_{I\leftrightarrow\{T_{k}\}_{k=1}^{K}}$ $\displaystyle=\mathcal{L}_{I\rightarrow\{T_{k}\}_{k=1}^{K}}+\mathcal{L}_{\{T_{ k}\}_{k=1}^{K}\rightarrow I},$ (7) $$

where the concept representation $\bm{V}_{i}^{T_{k}}$ is extracted by the pre-trained CLIP-based text encoder, which is frozen during training.

## Appendix D Detailed Experiments Settings

### D.1 Implementation Details

For the SeTok, we apply pre-trained SigLIP-SO400M-patch14-384 (Zhai et al., ) as our vision encoder, and the numbers of inner-cluster and inter-cluster transformer layers are set as 12, and 8, respectively.
The dimension of the semantic-equivalent token is 512.
For the detokenizer, we adopt $L=12$ blocks for mask tokens to query visual information contained in
the semantic-equivalent tokens, and then an upsampler inspired by the architect of Yu et al. () is employed as the pixel decoder.

In the Setokim framework, we employ the LLaMA-2-7B (Touvron et al., ) to initialize our LLM backbone.
Following Kirillov et al. (), we take the image embedding extracted in the vision encoder in SeTok and the visual tokens generated by LLM as inputs, which are both fed into the mask decoder.
This decoder uses prompt self-attention and cross-attention in two directions (prompt-to-image embedding and vice-versa) to update all embeddings.
After running two blocks, we upsample the image embedding and an MLP maps the output token to a dynamic linear classifier, which then computes the mask foreground probability at each image location.
Following Li et al. (), we employ a small MLP consisting of three residual blocks (He et al., ) for computing the diffusion loss.
Each block sequentially applies a LayerNorm (LN) (Ba et al., ), a linear layer, SiLU (Elfwing et al., ), and another linear layer, merging with a residual connection.

### D.2 Training Data

Here, we detail the training data utilized for training SeTok and Setokim in Table 7.

**Table 7: The training data used in our experiments.**
|  | Name | Size |
| --- | --- | --- |
| SeTok | ImageNet-1K (Deng et al., ) | 1.2M |
| OpenImages (Kuznetsova et al., ) | 9M |  |
| Stage-I | CC12M (Changpinyo et al., ) | 12M |
| LAION-aesthetics-12M (Schuhmann et al., ) | 12M |  |
| ALLaVA-Caption-4V (Chen et al., ) | 715K |  |
| InstructPix2Pix (Brooks et al., ) | 313K |  |
| LLaVA-595K (Liu et al., ) | 665K |  |
| MSCOCO (Lin et al., ) | 313K |  |
| Visual Genome (Krishna et al., ) | 108K |  |
|  | OpenImages (Kuznetsova et al., ) | 9M |
|  | SlimPajama (Soboleva et al., ) | - |
| Stage-II | ALLaVA-Instruct-4V (Chen et al., ) | 661K |
| ShareGPT4V (Krishna et al., ) | 80K |  |
| Alpaca (Taori et al., ) | 5K |  |
| LLaVA-v1.5-mix-665K (Liu et al., ) | 665K |  |
| VQA^v2 (Goyal et al., ) | 83K |  |
| GQA (Hudson & Manning, ) | 72K |  |
| OKVQA (Marino et al., ) | 9K |  |
| AOKVQA (Schwenk et al., ) | 50K |  |
|  | RefCOCO/+/g (Kazemzadeh et al., ; Mao et al., ) | 65K |
|  | InstructPix2Pix (Brooks et al., ) | 313K |
|  | MagicBrush (Zhang et al., ) | 10K |

### D.3 Training Receipt

In Table 8, we list the detailed hyper-parameters setting at three stages, i.e., Setok training and two-stage Setokim training.
All training is conducted on 16$\times$ H100 (80G) GPUs.

**Table 8: Training recipes for SeTok, Setokim of Stage-I: Multimodal Pretraining and Stage-II: End-to-end Instruction Tuning.**
| Configuration | SeTok | Stage-I | Stage-II |
| --- | --- | --- | --- |
| Optimizer | AdamW | AdamW | AdamW |
| Precision | bfloat16 | bfloat16 | bfloat16 |
| Peak learning rate of LLM | - | 5e-5 | 5e-5 |
| Peak learning rate of Visual Part | 5e-4 | 1e-4 | 2e-4 |
| Weight Decay | 0.05 | 0.1 | 0.01 |
| Learning Rate Scheduler | Cosine | Cosine | Cosine |
| LR Warmup Steps | 10K | 2K | 5K |
| Input image resolution | 384 $\times$384 | 384$\times$384 | 384$\times$384 |
| Batch Size Per GPU | 16 | 16 | 16 |
| Gradient Accumulation Steps | 8 | 8 | 8 |
| Maximum Token Length | - | 2048 | 2048 |

## Appendix E Extended Experimental Analysis

**Table 9: The effect of unlocking vision encoder in training Setok and Setokim.**
| Setting | ImageNet (rFID$\downarrow$) | Flickr30(CIDEr.$\uparrow$) | VQA^v2 (Acc.$\uparrow$) |
| --- | --- | --- | --- |
| Frozen | 123.6 | 85.4 | 77.5 |
| UnFrozen | 2.07 | 86.9 | 78.7 |

#### The Impact of Unfreeze Vision Encoder.

To evaluate the impact of unfreezing the vision encoder, we conduct an ablation experiment where the vision encoder is kept frozen, and only the token merger and detokenizer are optimized.
We observe that SeTok fails to reconstruct the image as freezing the vision encoder hinders its ability to learn the low-level features required for accurate reconstruction.
In this scenario, the vision decoder alone is tasked with reconstruction, but it is unable to do so effectively using only high-level semantic features. Interestingly, freezing the vision encoder did not noticeably impact SeTok’s performance in vision-language semantic understanding.

**Table 10: Comparison between Setok and other vision tokenization approaches, all of which generate continuous visual tokens that are subsequently fed into the LLM.**
| Mechanism | #Tokens | TFLOPs | Flickr30K | VQA^v2 | OK-VQA |
| --- | --- | --- | --- | --- | --- |
| SigLIP + MLP (Liu et al., ) | 256(Fixed) | 15.8 | 80.6 | 72.4 | 56.1 |
| SigLIP + Q-former (Zhu et al., ) | 32(Fixed) | 12.4 | 81.3 | 71.0 | 54.6 |
| SigLIP + Resampler(Alayrac et al., ) | 64(Fixed) | 13.4 | 83.4 | 72.5 | 54.9 |
| \cdashline1-6<br>SeTok | Dynamic | 8.2 | 86.9 | 78.7 | 60.2 |

#### The Comparison of Vision tokenizer.

To evaluate whether our proposed SeTok effectively integrates with LLMs to enhance model performance, we experimented with different connector strategies, such as MLP (Liu et al., ), Q-former (Zhu et al., ) and Resampler (Alayrac et al., ).
Using the same vision encoder (i.e., SigLIP-SO400M-patch14-384), we construct various MLLM architectures.
We follow a two-stage training process on the same dataset. Finally, we assessed the models’ performance on vision-languages tasks, and the results are presented in Table 10.
As observed, SeTok demonstrates higher efficiency, achieving lower TFLOPS while delivering superior vision understanding capabilities.
These findings validate that SeTok is capable of learning more aligned and compact visual tokens, leading to better semantic integration and improved performance.

Figure: Figure 8: The image reconstruction results from the visual detokenizer in Setok.
Refer to caption: x8.png

#### The Quantitative Reconstruction of SeTok.

In Figure 8, we visualize some reconstructed examples by Setok.
It can be seen that, given the tokenized visual tokens, the original input images can be successfully recovered.
The reconstructed examples exhibit a high degree of the construction of the method.

#### Visual Generations.

Figure 9 visualizes the images generated by Setokim.

Figure: Figure 9: The visualization of generation images from Setokim.
Refer to caption: x9.png

Figure: Figure 10: The Setokim’s performance visualization of image captioning (a) and VQA (b) task.
Refer to caption: x10.png

#### Visual Understanding.

Figure 10 presents additional examples of vision-language understanding and reasoning tasks.
Notably, as shown in Figure 11, Setokim exhibits strong in-context learning and multi-image reasoning capabilities.

Figure: Figure 11: Illustration of Setokim performing in-context learning in (a) with two image-text pairs and a third image as context to prompt the model, and reasoning across multiple images in (b) with two images with the question as context to guide the model.
Refer to caption: x11.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.In Table 2 that SETOKIM and some similar methods still have a gap in performance with SOTA under some data sets, please give some explanations.
soundness: 3
strengths: 1.This paper points out that the main bottleneck of current MLLMs is visual tokenization method, which cannot effectively align semantic and image features. Current visual tokenization methods only divide the image into a fixed number of patches, which not only destroys the visual context information, but also makes it difficult to align the visual semantic units.

2.During SeTok training, this work introduces concept-level text-image contrastive loss and image reconstruction loss. The former ensures the semantic independence and integrity of each patch, and semantically alignes visual tokens with the corresponding text concepts. The latter ensures that the token retains enough pixel-level detail by calculating the loss when feeding the token into the detokenizer to reconstruct the original image. Finally, the sum of the two losses is calculated using weighting.

3.The ablation experiments of key modules are sufficient, and the performance of the method on various language image tasks is fully demonstrated, demonstrating the superiority of the method from multiple perspectives.
summary: This work purpose a novel dynamic Semantic-Equivalent Vision Tokenizer(SeTok)，using adynamic cluster method to group visual features into semantic units, , flexibly determining the number of tokens based on image complexity. The proposed SETOK-based MLLM (SETOKIM) shows superior performance on various text-image tasks.
weaknesses: 1.Table 2 compares the performance of SETOKIM method and some similar methods under the task of zero-shot text-to-image generation and editing under several data sets. The results show that the indicators of some data sets are not optimal, and there is still some gap with SOTA.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - How is the visual detokenizer trained? As the tokenized visual tokens have dynamic numbers, it is very confused for me on how the visual tokenizer is trained based on the dynamic visual tokens. And I found no description on this in the paper.

- I also have an open question for the author. Any answers should be ok. The proposed tokenization method groups the vision inputs without any semantic instruction. But as we know, vision has infinite divisibility, which means that all objects can always be decomposed into smaller parts. Based on this, maybe a more reasonable way is to perform clustering during the reasoning process inner the MLLM according to corresponding contexts, instead of the input part. So, I would like to hear the authors’ thoughts on this.

I intend to raise my score if the authors can properly address my concerns in the weakness.
soundness: 2
strengths: - The motivation of dynamic SeTok makes a lot of sense. As the paper mentioned, fragment visual input can corrupt the visual semantic integrity, which is a long-lasting problem in visual tokenization. This paper provides a simple solution though clustering but exhibit high performance, which could inspire future works.
- Sufficient experimental evaluation. Various tasks are included, like referring expression segmentation, visual understanding, and text-to-image generation. This could take much efforts. Also, this paper present visualization of the token masking results of SeTok in Fig. 7, which I think is a strong support for the motivation.
summary: This paper proposes a new visual tokenization method SeTok, which groups visual features into semantic units via a dynamic clustering algorithm. This design is aimed to address the information fragmenting problem and the semantic alignment problem in previous visual tokenization methods. A MLLM called SETOKIM is trained based on the proposed tokenization method.
weaknesses: - Unfair comparison. 1) The model is finetuned on a lot of multimodal instruction datasets, whereas the compared works, like the ones in Tab. 1, use smaller data sizes for finetuning. Performing comparison on different training data hardly demonstrates the priority. 2) The visual encoder introduces more parameters for clustering (totally 20 transformer layers). 3) A more advancing pretrained encoder SigLIP-384 is used. In conclusion, many settings, including the data, the model parameters, and the pretrained encoder, exist unfair comparison. These settings conceal the true effectiveness of the proposed tokenization method. I understand that it is hard for an MLLM to perfectly align with previous settings, and it is ok. But my intension is to make sure the authors can explicitly present these differences in the tables, which helps prevent misleading and ensure fair comparisons in future research.

- Potential data leak. The benchmarks used in Tab. 1 all have released answers, which may be potentially seen during training. It is suggested to perform additional evaluation on some addition benchmarks that do not have released answers, like MMB.

- Potentially poor performance on fine-grained visual perception. The clustering-based visual tokenization may be unfriendly to some fine-grained, like OCR, because clustering always fuses information for coarse representation.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Is SeTok trainable during multimodal pre-training stage?
2. How much additional text data needs to be added to avoid degrading the performance of LLMs?
soundness: 3
strengths: 1. The proposed SeTok is interesting and valuable. 
2. The experimental details in the paper are thorough, ensuring reproducibility.
summary: This paper focuses on the design of visual tokenizer for MLLMs. Current MLLMs do not consider the visual semantic integrity when processing images into patches. To address this, the authors propose SeTok that employs a dynamic clustering strategy to groups visual features into semantic units. Furthermore, they integrate SeTok into MLLM, creating SeTokIM, and demonstrate promising results on both generation and understanding tasks.
weaknesses: 1. The performance of MLLMs is greatly influenced by SFT data and LLMs. Therefore, when comparing image understanding capabilities, it is important to make sure under the same settings, such as using the same settings as LLaVA, to better reflect the effectiveness of the proposed methods. However, there is lack of this comparison in the experiments.
2. In the comparison of image understanding capabilities, there is a lack of evaluation on more fine-grained benchmarks such as TextVQA, OCRBench, SEED-Bench-2-Plus, as well as more general benchmarks like MMBench and SeedBench. A potential concern is that SeTok's dynamic clustering strategy might lead to the loss of fine-grained information, making it challenging to complete more difficult tasks.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Why do we choose ImageNet-1K and OpenImages for reconstruction and alignment learning separately?
2. How to do fixed clustering? k-means?
soundness: 3
strengths: 1. The paper is well-written and easy to follow.
2. The experiments are well-conducted and quite comprehensive.
3. The ablation studies of SeTok is quite clear and valuable, highlighting the effectiveness of the proposed components.
4. I think that the use of dynamic clustering to generate semantic-equivalent visual tokens in MLLM is quite interesting, compared to standard operations like average pooling and query-based compression. The comparison of dynamic and fixed clustering demonstrates the superiority of this clustering mechanism.
summary: In this paper, the authors propose a dynamic Semantic-Equivalent Vision Tokenizer (SeTok) to preserve semantic integrity in visual features. SeTok can produce different numbers of visual tokens for various images. The authors introduce concept-level image-text contrastive loss and image reconstruction loss to balance the high-level understanding and fine-grained detail preservation abilities of SeTok. SeTokIM, an MLLM equipped with SeTok, shows superior performances on image-text understanding and generation tasks.
weaknesses: I have some concerns regarding the comparisons in Table 1. The authors claim that SeTok is superior to Q-former or cross-attention framework. However, SeTokim and the other compared MLLMs do not share the same training scheduler, base ViT, LLM, or training datasets. A fair comparison is to evaluate various kinds of visual tokenizers in an environment with tightly controlled variables.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: [-] Training/inference efficiency. Due to the clustering of visual tokens, the number of tokens obtained from each image is dynamic. So, how to perform parallel training on a batch, without affecting the efficiency of training and testing due to clustering?
soundness: 2
strengths: [+] For MLLMs, the pursuit of a more reasonable tokenizer to unify visual tasks and generate visual tokens that are accurately aligned with text semantics has always been a key challenge that deserves further research. 

[+] The paper is easy to follow and understand, having clear formulation.

[+] Many experiments are conducted to demonstrate the idea’s performance, across comprehension, generation, segmentation, and editing tasks, showing good generality.
summary: This paper introduces one viable semantic-equivalent tokenizer for MLLMs, namely, SeTok. It tokenizes automatically patch-level visual features into a variable number of semantic-complete concept visual tokens, further achieving accurate alignment between vision and text. 
And then use a unified autoregressive objective to perform two-stage optimization on SETOKIM (one MLLM). Experiments covering comprehension, generation, segmentation, and editing tasks, demonstrate the effectiveness.
weaknesses: [-] Complex images. SeTok seems to be effective in handling simple images, but when faced with complex images, the quality of clustering seems difficult to guarantee (for example, there are dense people, objects, or significant repetition between people and objects in the image). As a result, clustering errors are injected into visual tokens, which corrupt the final output results.

[-] Source of gains. For MLLMs, there are usually two types of visual bottlenecks: A. maintaining detail resolution to form hierarchical information [1]; B. one reasonable tokenizer. When demonstrating the value of the tokenizer, this paper still uses the detokenizer (in Fig. 2) to reconstruct pixel level images, i.e., keeping visual details. Can you decouple these two to demonstrate the effectiveness of each component separately? For example, report the performance with only \mathcal{L}_{citc}, or make more comparisons with [1].

[1] Eagle: Exploring the design space for multimodal llms with mixture of encoders

Meta-review: This work purposes a novel dynamic Semantic-Equivalent Vision Tokenizer (SeTok)，using a dynamic cluster method to group visual features into semantic units, flexibly determining the number of tokens based on image complexity. The proposed SETOK-based MLLM (SETOKIM) shows superior performance on various text-image tasks. Although reviewers have raised some concerns about performance comparison, missing experimental results and so on. The authors have addressed these concerns well in the discusion period. Therefore, the AC recommends this paper as accept.

Decision: Accept (Poster)