Subobject-level Image Tokenization
Authors: Delong Chen
Samuel Cahyawijaya
Jianfeng Liu
Baoyuan Wang
Pascale Fung
Venue: ICML 2025 poster (2025)
Keywords: image tokenization
vision-language models
image captioning
boundary detection
watershed segmentation
Primary area: applications->computer_vision
TL;DR: A subword style image tokenizer for VLM

Abstract: Patch-based image tokenization ignores the morphology of the visual world, limiting effective and efficient learning of image understanding. Inspired by subword tokenization, we introduce subobject-level adaptive token segmentation and explore several approaches, including superpixel, SAM, and a proposed Efficient and PanOptiC (EPOC) image tokenizer. Our EPOC combines boundary detection--a simple task that can be handled well by a compact model--with watershed segmentation, which inherently guarantees no pixels are left unsegmented. Intrinsic evaluations across 5 datasets demonstrate that EPOC's segmentation aligns well with human annotations of both object- and part-level visual morphology, producing more monosemantic tokens and offering substantial efficiency advantages. For extrinsic evaluation, we designed a token embedding that handles arbitrary-shaped tokens, and trained VLMs with different tokenizers on 4 datasets of object recognition and detailed captioning. The results reveal that subobject tokenization enables faster convergence and better generalization while using fewer visual tokens.

Subobject-level Image Tokenization
Delong Chen 12 Samuel Cahyawijaya? ?
Jianfeng Liu
Baoyuan Wang" Pascale Fung 12
Abstract
celli & Olshausen, 2001). Direct modeling on pixels (Chen et al., 2020; Nguyen et al., 2024) tends to be computation- ally inefficient, especially for high-resolution images. The widely adopted approach is patch- patch-based tokenization (Doso- vitskiy et al., 2021), which divides images into fixed-size square patches, each treated as a single token. Although be- ing straightforward and efficient, the non- adn-adaptive nature of patch-based tokenization prohibits its ability to capture the underlying morphological structure of the image (Palmer, 1977). Consequently, larger patches fuse multiple seman- tics within a single token (polysemanticiry), an undesirable property that hinders effective learning of token represen- tations. On the other hand, smaller patches exhibit greater monosemunticity but lead to an excessive number of tokens, resulting in computational inefficiency.
Patch-based image tokenization ignores the mor- phology of the visual world, limiting effective and efficient learning of image understanding. In- spired by subword tokenization, we introduce sub- object-level adaptive token segmentation and ex- plore several approaches, including superpixel, SAM, and a proposed Efficient and PanOptiC (EPOC) image tokenizer. Our EPOC combines boundary detection-a simple task that can be han- dled well by compact model-with watershed segmentation, which inherently guarantees no pix- els are left unsegmented. Intrinsic evaluations across 5 datasets demonstrate that EPOC's seg- mentation aligns well with human annotations of both object- and part-level visual morphol- ogy, producing more monosemantic tokens and offering substantial efficiency advantages. For extrinsic evaluation, designed a token embed- ding that handles wubitrary-shaper tokens, and trained VLMs with different tokenizers on 4 datasets of object recognition and detailed cap- tioning. The results reveal that subobject to- kenization enables faster convergence and bet- ter generalization while using fewer visual to- kens. Project website: https: //thtub.
In Natural Language Processing (NLP), similar challenges of input redundancy also exist, and the solution is to merge characters that frequently co-occur into individual tokens (Sennrich et al., 2016; Kudo, 2018; Gastaldi et al., 2024). Sennrich et al. (2016) demonstrates that a statical- based grouping approach can produce subword tokens that identi- fies the morphological structure, e.g., the word sweetish in English can be chunked into "sweer" and "-ish", while the word "siißlich" in German, can be chunked into "xüf8" and "-lich". This yields better monosemanticity of tokens com- pared to other alternatives such as word-based or character- based tokenization methods which improves generalization to rare or out-of-vocabulle vocabulary words by enabling semantic composition of subwords (Wang et al., 2018; Wilie et al., 2020; Mielke et al., 2021), while also guaranteeing effi- ciency by significantly reducing sequence length (Song et al., 2021; Muller et al., 2022: Cahyawijaya et al., 2024).
1. Introduction
Partitioning the raw observational data stream into a man- ageable number of segments often beneficial for learning, particularly when nearby elements in the data stream exhibit strong correlations and therefore high redundancy (Cover, 1999; Barlow et al., 1961; Biederman, 1987; Marr, 2010; Delétang et al., 2024). In Computer Vision (CV), nearby pixels are often highly correlated (He et al., 2022; Simon-
Patch tokenization on images analogous to "character chunk tokenization" on text, which is conterintuitive and rarely adopted in practice. We hypothesize that adaptive token segmentation, as opposed to static patch tokenization, can facilitate better learning of image understanding. To test this, explore both object-level tokenization, which is akin to word-level tokenization, and subobject-level to- kenization, resembling subwords in images. For subobject segmentation, a promising method is SAM's "segment ev- erything" " mode (Kirillov et al., 2023), but suffers from computational inefficiency and the presence of unsegmented regions. We propose Efficient and PanOptiC (EPOC) to
MMeta FAIR Paris "The Hong Kong University of Sci- ence and Technology "Alibaba Group "Zillow. Correspondence to: Pascale Fung <pascalo@cce.us..kkt Delong Chen <de- long.chen caahcncconntt.ust.k....
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Subobject-Ieve Image Tokenization
addresses these two key limitations. EPOC integrates bound- ary detection and watershed segmentation (Vincent & Soille, 1991). The boundary detection formulation (instead of mask generation) simplifies the learning task (Canny, 1986) and enables us to squeeze the model size to 3.7M parame- ters from SAM's 641M while achieving a similar level of segmentation quality'. Moreover, the revisited watershed algorithm ensures comprehensive segmentation while also providing flexible control over segmentation granularity through single scalar hyperparameter. This design im- proves inference efficiency by making the cost independent of the number of masks (i.e., (() complexity).
much more efficient.
We show that subobject- image tokenizers facili- tate faster VLMs convergence, improves their gener- alization, and achieves better token efficiency, outper- forming patch and object toknizers.
2. Preliminaries
2.1. Problem Formulation
Let X E #xxxx represent an input image and Y repre- sent the target label, the task of image understanding is to learn a function ʃ that maps X to Y. When using sequence models such as Transformers, this mapping is achieved by processing and generating sequences of tokens derived from X and Y. i.e., f(xx1 (1, X2..... xNl)
Studies on text tokenization in NLP typically involve intrin- sic evaluation measuring tokenizer's innate qualities, and extrinsic evaluation measuring its impact on downstream task performance (Gowda & May, 2020; Zouhar et al., 2023; Goldman et al., 2024). Following this practice, we also measure both aspects for holistic evaluation. Intrinsic evaluation encompasses three key aspects: 1) morphology- whether the segmentation align with semantic boundaries, 2) monosemanticiyy whether individual token avoids cov- ering multiple semantics, and 3) efficiency-how much ad- ditional computational overhead are introduced. Results on five datasets covering both object- and subobject-level annotations reveals that both SAM and EPOC yields strong morphology alignment and token monosemanticity, while EPOC enjoys significant advantage on efficiency.
= 121.y2.... V2.... , Nmll where each vector x; E IRd is an individual visual token and [yr) are text tokens. The task of image tokenization is to transform the high-dimensional raw input X into a compact set of visual tokens, i.e., X + [1, X2 .2.....,N... x.]]
The process of image tokenization consists of token segmen- tation and token embedding. Token segmentation aims to produce token index map M E {O..... N - " from X, where each element in this map assigns the correspond- ing pixel to one of N tokens. All pixels assigned to the same token index are grouped together as individual visual tokens. We are interested in developing good token segmentation that enables f to better approximate X + Y (i.e., faster convergence and better generalization). Furthermore, since the computational cost of f usually heavily depends on the number of input tokens, the mapping M that has smaller N is preferred the performance of f is the same. In this paper, we propose and implement different token segmen- tation approaches for generating M, and learn f X + Y with each of them under controlled settings.
For extrinsic evaluation, we first design a vision-language model (VLM) (Liu et al., 2023) architecture to incorpo- rate adaptive token segmentation which produces dynamic spatial arrangement and non-regular shapes, and train se- ries of models with different image tokenizers. Results on ImageNet- 1 (Deng et al., 2009), ShareGPT4V (Chen et al., 2024), Pixmo-cap (Deitke et al., 2024), as well as a new dataset CLEVR-cap generated from CLEVR (Johnson et al., 2017) indicates that VLMs using subobject-level image tok- enization enjoy faster convergence and better generalization with much fewer number of visual tokens. Such advantage can be observed across different L. and visual embed- dings. We also show that EPOC-based method holds more robustness to dropping long-tai tail small tokens, which allow us to further reduce sequence length. To summarize, we make three core contributions in this paper:
2.2. Patch-based Image Tokenization
Patch tokenization divides the image into N = p X p non- overlapping square patches of fixed size, where p controls the number of patches per side. Its effectiveness lies in leveraging the spatial inductive bias that neighboring pixels exhibit high correlations. However, the underlying assump- tion of patch tokenization can be overly strong since visual semantics are not usually distributed in a grid-like structure. The violation of such unrealistic assumption leads to the is- sues of either token polysemanticity or token redundancy. Large patches in crowded regions often encompass multiple semantics. While model f can learn to represent such polysemantic tokens, the unique mixture of semantics is unlikely to appear frequently in the training data, leading to limited sample exposure and insufficient learning. This issue is analogous to the challenge of rare words in word- based text tokenization in NLP, which can be addressed by
We introduce EPOC, a novel subobject tokenization that integrates boundary detection and watershed seg- mentation to achieve efficient panoptic segmentation.
We compare different adaptive token segmentation methods via extensive intrinsic evaluation. EPOC pro- vides comparable segmentation quality while being
'The boundary detection model was referred to as the Direct Segment Anything Model (DirectSAM) in our initial arXiv ver- sions. Here EPOC = DirectS AM + Watershed
2

---

Subobject-Ievel Image Tokenization
increasing the segmentation granularity. The solution for vision models that shares the similar spirit is lower the
tion not only come from its similarity with subword tok- enization, but also from its alignment with the Recognition- Dy-Components (Biederman, 1987) theory in cognitive science, which posits that human recognize objects through their constituent parts (Tversky & Hemenway, 1984; De- haene et al., 2022). Moreover, in previous CV studies, part- based image recognition (Felzenszwalb & Huttenlocher, 2005) has also demonstrated improved robustness (Li et al., 2023; Sitawarin et al., 2023; Li et al., 2024) and sample effi- ciency (Lake et al., 2015). In the following, we introduce three subobject segmentation approaches that we explore.
patch size, as adopted by many recent large ViTs. However, this introduces redundancy. For instance, large region of clean sky can be divided into excessive number of tokens, wasting computation resources.
3. Adaptive Token Segmentation
As natch-based tokenization struggles with the conflict be- tween token polysemanticity and token redundancy. this section introduces adaptive token segmentation methods, which include both segmenting images into object instances ($3.1) and subobject entities ($3.2).
3.2.1. Superpixel Segmentation. Superpixel segmentation is a classical type of method that groups adjacent pixels into local clusters. These clusters, termed superpixels, are formed based on pixel properties such as color intensity, This typically results in irregularly shaped segments that are smaller and more detailed than entire objects, and as such, superpixels are naturally categorized within the subobject level. Over decades, many superpixel segmentation methods have been developed. We consider the k-means- -aaed Sim- ple Linear Iterative Clustering (SLIC) (Achanta et al., 2012) method. The x for k-means is the key hyperparameters that allow control over the granularity of segmentation.
3.1. Object-level Image Tokenization
A straightforward approach that enables adaptive tokeniza- tion to group pixels by object instances. Here, "objects" refers broadly to both discrete "thing" instances (e-g-, cats, humans, cars) and amorphous "stuff" regions (e.g., sky, grass, water). This approach is promising due to the nat- ural coherence of objects in the real world, where their constituent pixels exhibit high internal correlations as they tend to move as a whole. Object-level image tokenization can be implemented using panoptic segmentation (Kirillov et al., 2019) models, where the generated "label map" can be directly utilized as token segmentation M.
3.2.2. Segment Anything Models. Segment Anything Mod- els (SAM) are trained on the large SA-1B dataset, which contains 11 million images and 1 billion mask annotations of varying granularitics, encompassing objects, parts, and subparts (Kirillov et al., 2023). The SAM architecture com- bines an image encoder and decoder, enabling segment anything according the given prompt. To generate comprehensive segmentation for a whole image, a regular grid of point prompts is applied to the decoder, producing masks corresponding to each point prompt. These masks are subsequently filtered and merged to yield the final segment everything results.
Panoptic segmentation models are typically trained on the COCO (Lin et al., 2014) and ADE20K (Zhou et al., 2019) datasets, which respectively include 133 and 150 annotated classes of common objects and regions. However, the num- ber of object types in the real world far exceeds this scale ( (Wang et al., 2023). A significant limitation of these models is their reliance on the fixed vocabulary of object cate- gories, which poses challenges for generalization to out-of- vocabulary objects. In addition, image understanding tasks are not always object-centric (Tong et al., 2024). When the target Y involves information of object parts, object-level tokenization will still suffer from polysemanticity.
mask
While SAM demonstrates strong performance across diverse image segmentation benchmarks, its effective application in image tokenization is hindered by the following two signif- icant limitations: Efficiency: To perform segment every- thing on a single image with a default grid of 32 x 32 point prompts requires 1,024 times of forward passes through the mask decoder. Although the mask decoder has been made lightweight, and improvements such as reducing the encoder size (Zhao et al., 2023) and reducing the number of prompts (Zhang et al., 2023) have been introduced, the com- plexity remains O(N). where N is the number of masks to be generated. Comprehensivenes:: As shown in Fig. 1, the result of SAM's segment everything mode is not guaranteed to be panoptic. SAM's design of generating each mask in- dependently introduces potential gaps between individual segments, or certain background regions may remain un- segmented. While these gaps or unsegmented areas can be
3.2. Subobject-level Image Tokenization
In NLP, subword tokenization has demonstrated superior performance in language modeling compared to word-based tokenization. Inspired by subwords and with an analogy of object in image are akin to words in sentences, we in- troduce the notion of subobject-level image tokenization. Subobjects represent an intermediate level situated between objects and pixels, similar to the concept of subwords which is also an umbrella covers various methods.
Subobject entities are semantically coherent and function- ally meaningful units, encompassing object parts, aa--parts, and finer subdivisions. The potential of subobject tokeniza-
3

---

Subobject-Ieve Image Tokenization
Input Image
SAM VIT-H (641M)
EPOC (3.7M)
Boundary Probabilities
Watersheds
Seed Regions
M
Input Image
Boundary Detection
Identify Seed Regions
Watershed Segmenttation
Figure 2. Proposed EPOC. A boundary probability map P [o. 1] x]_* is predicted from the input image X E R xa and treated a topographical surface. The watershed segmentation be- gins by identifying basins in P as seed regions (labeled in different colors) with threshold t. A "flooding" process then progresses until the entire P is submerged. When seed regions meet during flooding. "watersheds" are formed separate them.
Figure 1. Comparing SAM and our EPOC on SA-1B images. The design of independent mask decoding makes SAM often leave thin gaps between segments or background regions unsegmented. Our EPOC inherently guarantees complete coverage while also improves computational efficiency.
cent Soille, 1991), which naturally ensures comprehen- siveness. As shown in Fig. 2, the method treats the predicted P metaphorically as topographical surface, where higher probabilities correspond to elevated regions like peaks ridges and lower probabilities represent basins. The algo- rithm simulates gradual "flooding" of these basins, with boundary of segmentation masks are generated acting as barriers that prevent water from merging between different basins. Watershed typically requires seed regions be- gin the "flooding" process, which we obtain by applying threshold t to the boundary map. Pixels below are as- sumed to be within object interiors and serve seeds. By tuning t. one can seamlessly control the granularity of the resulting segmentation-highee t. merges seeds into larger segments, while lower t. retains finer details. Fig. 10 in Appendix provide an example illustrating such flexibility.
grouped into a single token to avoid information loss, the resulting tokens often suffer from high polysemanticity.
3.2.3. Proposed EPOC. We propose Efficient and Com- prehensive Image Tokenization (EPOC), novel segment everything approach that ensures comprehensive segmenta- tion while significantly improving computational efficiency. Unlike SAM's prompt-conditioned mask decoding, EPOC adopts boundary detection approach, predicting pixel- wise boundary probability map P E [o. 1) that outlines subobject entities across the entire image. This approach avoids the overhead of generating individual masks and ben- efits from the simplicity of boundary detection. Tasks of similar difficulty have historically been effectively handled by non pon-aaametricc algorithms (Canny, 1986) and early lay- ers of CNNs (Zeiler & Fergus, 2014). The task simplicity allow us to employ an extremely compact model. We em- pirically found a lightweight regFormer-be (Xie et al., 2021) with only 3.7M parameters learns well from SA-1B.
EPOC achieves its computational efficiency through three advantages. First, the boundary probability predictor oper- ates on the entire image in a single forward pass, resulting in complexity of ((1) which is independent to the number of tokens N. Second, the compactness of SegF ormer- bb further reduces GPU memory usage, allowing for large batch sizes or multi-process parallelism. Third, the sub- sequent watershed algorithm non-parametric and runs efficiently CPUs (Kornilov & Safonov, 2018). These features together make EPOC efficient image tokenizer.
Based on the predicted boundary probability map P, a naive approach to derive the token segmentation map M might involve thresholding the map and applying connected com- ponent analysis (Wu et al., 2009). However, such methods leave boundary pixels unsegmented, leading to potential information loss or polysemanticity when these pixels are merged. Instead, we revisit the watershed algorithm (Vin-
4. Intrinsic Evaluations
4.1. Evaluation Setup
This section evaluates innate properties of token segmen- tation models (which take image to generate the index
4

---

Subobject-Ieve Image Tokenization
Object-lavel
Subobject-lavel Image Tokenization
Patches
Panoptic Segmentation
Superphxel
Segment Anything Models
Ours
Ground
Patch
Patch 32x33
Mask2Former OneFormer
SLIC k-64
SLIC k--256
SAM V1T-8
SAM
FastSAM VOLDVE-x
MobileSAMv2 VLT-H
EPOC t.0.25
EPOC
Truth
18x10
tiny
Large
VIT-H-MS-64P
t 85
COCO object-level
ADE20K object-evel
Pascal Panoptic
Parts (PPP)
subobjacd-lovel
PartmageeNets
(PIN+*) subobjackievel
SA-1B subobjeck-evel
Figure Intrinsic evaluation dataset examples and token segmentation results. Object-level tokenization based on panoptic segmentation suffers from out-of-vocabulrry problem. Superpixel segmentation relies on bottom-up pixel grouping, which limits its ability to capture underlying structures. The SAM model and its variants generally provide reasonable token segmentation. The quality and style of the segmentation generated by EPOC closely match those of SAM, while utilizing significantly smaller model size.
map M ($3) without training any downstream models (e.g., VLMs). Models. We compare three categories of candi- dates: batch-based: square patches with patch size p vary- ing from 2 to 32; object-level: Mask2Former (Cheng et al., 2022) and OneFormer (Jain et al., 2023) trained on COCO and ADE20K. subobject-level: SLIC superpixel (Achanta et al., 2012), SAM (Kirillov et al., 2023), FastSAM (Zhao al., 2023), MobileSAMv2 (Zhang et al., 2023), and the proposed EPOC method. Datasets. We conduct evalua- tions five datasets, encompassing different annotation granularities, with COCO's s COCONut relabeled validation split (Deng et al., 2024) and ADE-20K (Zhou et al., 2019) validation split provide object-level annotations, and Pascal Panoptic Parts (PPP) (de Geus et al., 2021), PartlmageNer++ (PIN++) (Li et al., 2024) and SA-1B (Kirillov et al., 2023) consist subobject-lvee annotations. Fig. 3 provide visual- ization of converted ground truth boundary (leftmost) and token segmentation generated by different tokenizers. We provide richer details in the Appendix A and B.
detection studies (Arbeláez et al., 2011; Yamagiwa et al., 2024). A token segmentation that accurately captures the semantic structure of an image should align well with the ground truth boundaries, achieving both high precision and recall. The top row of Fig.4 presents the results for an input resolution of 768px. The evaluation confirms that patch- based tokenization exhibits very low alignment with image morphological structures, whereas all adaptive tokenization methods clearly outperform it. Superpixel segmentation underperforms other learned tokenization methods, indicat- ing that bottom-up pixel grouping lacks the holistic under- standing required to capture high-level semantic structures. Object-level tokenization based on panoptic segmentation shows strong in-domain performance (e.g., on COCO and ADE-20K) but struggles in zero-shot settings, where the segmentation models have to generalize unseen datasets. On PPP and PIN++ where all models are in zero-shot set- tings, our proposed EPOC demonstrates clear advantages over panoptic segmentation models and SAM ViT-B mod- els, achieving Pareto optimal performance. EPOC also matches the performance of FastSAM and MobileSAMv2 and closely approaches SAM ViT-H models. Importantly, EPOC achieves this level of performance with significantly smaller model size and offers substantially faster inference speeds, will be presented soon.
4.2. Results and Discussions
Alignment to Morphology. We first measure how well different token segmentation methods capture the seman- tie structures of images. Although the human annotations are provided as mask, traditional mask-based metrics are unsuitable for this evaluation due to their incompatibility with class-agnostic and mixed-granularity segmentation (as have both object- and subobject-level models and notations). Instead, we adopt boundary precision-recall metrics, which are widely used in boundary / edge/ / contour
Token Monosemanticity Score. As discussed in $2.2, one of the key goal of adaptive token segmentation to mini- mize the occurrence of polysemantic tokens. We quantify this by calculating a token monosemanticityy score, defined as the percentage of predicted token segments that lie en-
5

---

Subobject-Ievel Image Tokenization
COCO
ADE20K object-iovel
Pascal Panoptic Parts (PPP) SubobjecHevel
PartimageNaat+ (PIN++) subobjecHevel
SA-1B subabjoct-Iave
abjock-level
SAM
Crafie
Cod
Veskiforme
(C000)
E200
9.5
Prectten
Comtort Pediann
20A
Town)
Juun
Moi Whott
Numbe of "Steral Eagacite
mbrr of Tobera 3000
Namborol of Totera Eopacit
Namrorr Teaces solel
Namber of Texers V. sooel
Figure Intrinsic evaluation of token segmentation. Connected represent same model different sizes or with different hyperparameters. Top: We measure the aligmment between token segmentation and semantic annotations with boundary precision and recall. Our proposed EPOC achieves Pareto optimality compared SAM ViT-B models and matches the performance of FastSAM and MobileSAMV2. Bottom: All subobject-level methods demonstrate clear advantages over object-level (in maximum achievable monosemanticity score) and static patch-base tokenization (in token efficiency).
tirely within a single ground-truth region, i.e,, no crossing of ground truth boundaries that separate semantically distinct regions (see Appendix B for more details). The bottom row in Fig. 4 presents the results, where token monosemantic- ity scores plotted against the total number of tokens. For patch-based tokenization, smaller patches naturally avoid crossing ground-truth boundaries but result in signif- icant token redundancy (large N). All adaptive methods outperform static patch, with subobject-level tokenization (SAM, EPOC, and SLIC) being able to approach high (e.g., 90%) monosemanticity with varying levels of token effi- ciency, while object-level panoptic segmentation remains mostly below 60%. This observation supports our analysis in $3.1, where we hypothesized that the out-ol-vocabulary problem poses significant challenge for panoptic segmen- tation models.
Table 1. Comparing computation efficiency of image tokenizers.
Params (Millions)
Maximum FPS
GPU
Image Tokenizer
Usage
Patch
0.0
+:00
0.0
tiny
47.4
Mask2Former
lazge
215.5
44.6
OneFormer
tiny
50.7
17.3
99.1
large
218.8
11.1
99.4
k-64
15.4
k-128
0.0
13.2
0.0
Superpixel
k=255
11.0
SAM ViT
93.7
0.6
96.5
ViT-H
0.4
Variants
Fast:AMM
72.2
12.2
94.2
MobileSAMv2
711.0
1.2
89.2
t-0
13,4
5,7
t-0.3
3.7
16.2
8.1
(Ours)
t-0.5
17.1
9.0
Computational Efficiency. Image tokenizers must also offer fast inference. We measure throughput with a V100 (32GB) an 30 CPU cores by progressively spawning tok- enizer processes (each with batch size of 10) until total of 30 processes. Table 1 shows the maximum achievable FPS and corresponding GPU utilization. With only 3.7M parameters, EPOC achieves up to 17.1 FPS with under 10% GPU utilization, outperforming all SAM-based methods, allowing minimal impact on VLM's computation. Profil- ing indicates SegFormer's forward occupies only 5-7% of the total time, with CPU-based watershed taking the remainder. This explains why increasing threshold can boost efficiency: the gap between seed regions become thin-
ner and the watershed step shortens. Optimized watershed implementations (Perret et al., 2019) could bring further improvements, but since our implementation already meets the throughput needs of training downstream VLMs, we leave further optimization to future work. More details are provided in Appendix B.
5. Extrinsic Evaluations
While our intrinsic evaluations assessed the innate proper- ties of various image token segmentation methods, a com- plete picture requires extrinsic evaluations that measure their impact on downstream tasks. Compared to static patch,
6

---

Subobject-Ievel Image Tokenization
adaptive image tokenization ($3) introduces new challenges of dynamic spatial arrangements and heterogeneous sizes and shapes. Traditional architecture cannot handle these properties, as they rely on simplified design that assumes predetermined grid structures and static raster-order patch arrangement. In this section, we first detail our methodology of incorporating such information into visual tokens in $5.1, then present the setup and results in $5.2 and $5.3.
define the position embedding as x = ( by where @ denotes concatenation. The encode(-) function converts each cropped mask into a vector, and we implement it with downsampling and flattening. More sophisticated en- codings, such as CNN-based encoders, can be incorporated. However, as the amount of computation is dependent on the number of tokens n, we keep it simple in this paper.
Fusion. Finally, the content embedding x and the position embedding x, are fused via a small MLP: x; == MLP (x} +
5.1. Token Embedding for Adaptive Segmentation
x"). yielding the i-th visual token. Collectively, these tokens [x1 ; X2,.... Xx] serve the input the downstream model for tasks such as image classification or caption generation.
Given an image X and token segmentation map M. the goal to generate visual tokens J.N where each x; corre- sponds to one segment in M. We first encode the content and position of each token into xf and xf, respectively rep- resenting "what it is" and "where it is", and fuse them into visual tokens X,, which are then fed to f to learn X + Y.
5.2. Evaluation Setup
VLMs. We follow the strategy outlined in $5.1 to transform images into visual tokens, then feed them to text-only LLMs with a Llama-style architecture (Dubey et al., 2024). We apply the standard next-token prediction loss only on text tokens, ignoring the loss for the visual tokens. We train the entire LLM and the MLP projection while freezing the feature embedding (Appendix C provides more details on our evaluation setup).
Content Embedding, We allow a vision encoder to extract a feature map from the input image, i.e., feature(X) E R MM,. XW where Hę and We are spatial dimensions, and C' represent channels. For the case where feature() (-) is an identity function, it allows I to directly learn from raw pixels. Despite such special case, the spatial reso- lution of feature(X) is usually lower than M whose resolution is H x W, so we upsample the feature map to match the resolution of the mask M via X' (fsamp)e(featuree()) E xHx
Image Tokenizer. Due to the high cost of training VLMs, we only select a set of efficient representative token segmen- tation approaches to compare, which include patch tokeniza- tion, object-level panoptic segmentation (Mask2Former t tiny and 1 rarge trained on COCO), and subobject-level SLIC and EPOC. We use varying patch's 's p, SLIC's k and EPOC's t to sweep over different granularity.
Given X' and M, we aggregate the pixel features corre- sponding to each segment to obtain the content embed- w|
ding. Specifically, we define x = ppool(X M(k, w| = i)), where ('h,, w] E RCC is the feature vector at pixel coordinates (h, w) and (-) is pooling func- tion (e.g., average pooling). Although more sophisticated strategies (e.g., flattening ROI features or learning a Per- ceiver Resampler (Alayrac et al., 2022)) may further refine these embeddings, we find that simple average pooling is effective if the segments are reasonably monosemantic.
Datasets. We train VLMS on four datasets: ImageNet-1K (Deng et al., 2009): A object recognition dataset widely- used in CV. We treat class names as text target. ShareGPT- 4V (Chen et al., 2024): A detailed captioning dataset com- monly used for vision-language alignment. Pixmo-cap (Deitke et al., 2024): A high-quality dataset featuring rich -nnottted dense captions. CLEVR-cap: A new dataset introduced for fast ablation experiments. The dataset is generated by using CLEVR (Johnson et al., 2017) meta- data to create detailed captions that enumerate the attributes of each object from left to right. Captions follow the template of "A total of {n} objects: a (color) {size} (material) {shape]. ...". CLEVR-cap involves core visual understanding capabilities, including counting, spa- tial reasoning, and recognition of object color, size, material, and shape. By constraining to clean synthetic domain, it significantly facilitates evaluation cycles while maintaining experimental rigor.
Position Embedding. Adaptive tokenization introduces dynamic spatial arrangements and irregular token shapes, requiring position embeddings to preserve such information. Each segment in M has that information, but directly encod- ing each mask at the full image resolution costly. Instead, we exploit the prior that most segments form connected components, which allows more compact representation via box-mask decomposition.
Specifically, we compress the raw mask into token loca- tion (represented as a bounding box), and token shape (a cropped mask inside the bounding box). For each token i, we compute its bounding box b, E [0, 1] using the normal- ized <xywh> format, and we also extract a cropped binary mask m; E {0, 1) Hm xWm that indicates the exact shape of the segment (with Hm < H and W., < W). We then
5.3. Results and Discussions
Adaptive Tokenization Facilitates Image Understanding. Figure 5 presents validation perplexities of VLMs trained
7

---

Subobject-leve Image Tokenization
[4] Imagallll-1K
(b) ShareGPT4V
[c] Plamo-cap
(4) CLEVR-cAp
(4) CLEVR-cop
caloddling
DD1b12 fateddinng
tisoe Enbedsing
UIMNN Enbedding
VSE Enbredting
SoclLng 2110--t-nntttt
LLana 3.2-18
17am0-2-2-18
SeallNG 1200 2.0n-2sttract
SONTLLN LIRGN, 360M, L.79g-Erettrac
Nitch
Paic
Panepti Segneraate
Panopli
1.78
Segnerratioo
Panepi:
Sogg silee
Sepmaration
EPOC 10un)
superpent (AUI)
1.70
Sepapiion (SLK))
112 Sopertes (SUK)
EPOG JOun
1.75 Mkariber
Wuiber Toicess 120 3AX
20g
Figure 5. Extrinsic evaluation of token segmentation. (a-d): Adaptive token segmentation shows clear advantage over patch tokenization, with subohject-leve SLIC and EPOC being able to approach lower perplexity than object-level ones. (e): when using VAE embedding which is less semantically expressive, patch-based model failed to converge, while adaptive tokenizers works fine.
guee (tre.1)
goeR (ests)) paatcs (p:14)
Patch
SPCC
Ha 26-20
Robustness to Token Reduction. We explore the poten- tial of further reducing visual tokens by truncation. We do random drop-out for patch tokenizer, while for object- and subobject-level tokenization, we start dropping from the smaller ones. The result in Figure 6 (right) shows that adaptive tokenization methods exhibit stronger robustness compared to patch tokenization. Their long-tail token size distribution (Appendix B) ensures that discarding the small- est tokens has minimal impact on performance, whereas random patch dropout leads to notable degradation. This suggests that adaptive approaches hold even greater poten- tial for optimizing token efficiency.
0.3
8.5
6/
Number
Figure 6. Left: adaptive subobject-level tokenization outperform patch tokenization consistently across different embeddings. Right: EPOC-based tokenization is robust to token truncation, showing the potential of further improvement in token reduction.
different tokenizers (each point corresponds to an individual run). Across all datasets, most adaptive methods lie to the left or below the patch baseline, particularly in low N regime, indicating they can either reduce the number of tokens for similar performance or improve generalization under comparable token budgets. Among them, subobject- level EPOC and SLIC stand out for matching or surpassing the best performance of very small patches while using much fewer tokens. In contrast, object-level tokenization is more efficient than large patches but still lags subobject- level approaches in minimal achievable perplexity.
6. Related Work
Tokenization, a critical step in the NLP (Gastaldi et al., 2024), has been historically extensively studied (Mielke et al., 2021). Despite active explorations (Pagnoni et al., 2024; team et al., 2024), subword remains the predomi- nant choice for language modeling. Over the years, the effectiveness of subwords has been analyzed from multiple perspectives, ranging from intuitive explanations about re- ducing rare words (Sennrich et al., 2016; Liu et al., 2019) to iibomatiion-hhorceee measurements (Zouhar et al., 2023; Schmidt et al., 2024: Goldman et al., 2024). The common- ality among these analysis is the idea of a good tokenizer should maintain a balanced distribution over a manageable size of vocabulary, while providing reasonable and token sequence length. The principle of balanced distribution parallels our goal of preventing polysemantic tokens with limited appearance frequency-akin to rare words.
Compatibility to Different Token Embeedings. Figure 5 also compares the DINOv2 (Oquab et al., 2024) with VAE embeddings on CLEVR-cap. Notably, patch-based tok- enization struggles to converge when using the weaker VAE features, even with an LLM scaled to 1.7B parameters. In contrast, subobject- based models remain effective, demon- strating that adaptive tokenization can simplify the learn- ing of image understanding. Additionally, Figure 6 (left) presents an ablation study across various token embeddings on CLEVR-cap using three token segmentation methods that produce similar N. Subobject tokenization consistently outperforms static patch tokenization while maintaining comparable number of visual tokens. Interestingly, CLIP (Radford et al., 2021) underperforms DINOv2 in our setup, likely due to its lack of dense supervision and the lower feature-map map resolution of only 7x7.
The idea of adding adaptivity to visual tokenization has also been explored in previous studies. Some methods (Lew et al., 2024; Aasan et al., 2024; Ke et al., 2022) rely on bottom-up superpixel grouping. which lacks semantic awareness. Feature-based methods, such as slot attention (Locatello et al., 2020) and various token pooling methods (Haurum et al., 2023; Ronen et al., 2023; Marin et al., 2023;
8

---

Subobject-Ievel Image Tokenization
Feng & Zhang, 2023; Huang al., 2022), suffer from low- resolution feature maps which limits fine-grained segmenta- tion, and unreliable segmentation quality in early training stages. Our approach of using separate segmentation model, EPOC, avoids these issues, and outperform SAM- based segmentation (Kim et al., 2024) in terms of efficiency. Additionally, there are also works on image tokenization focusing on improving token embeddings for understanding or generation (Ramesh et al., 2021; Gansen-Estruce al., 2025), while leaving token segmentation simply patch grids. Our work can be combined with these efforts.
ern VLMs are still using relative low FPS rate as consequence).
Adaptive tokenization can be interpreted as a learned compression scheme, grouping correlated pixels to re- duce token count. While it has been shown beneficial for VLM, it inevitably discards intra-token spatial struc- ture, which might be valuable in settings that demand ultra-fine resolution pixel-level edits. This issue is similar to the disadvantage of subword- bassd LLMs on character-level task. Adding more adaptivity to adap- tive tokenization (i.e., task conditioned segmentation) can be potential ways to address this.
7. Conclusion
A Acknoledgement
In this work, we introduce the concept of subobject image tokenization and propose EPOC, an efficient and panoptic image tokenizer that combines boundary detection and wa- tershed segmentation to achieve high-quality, monosemantic token segmentation while maintaining computational effi- ciency. Through extensive evaluations, we demonstrate that EPOC not only aligns well with human annotations of vi- sual morphology, achieves better token monosemanticity, but also enabling faster convergence, better generalization of VL.Ms. Our findings highlight the potential of adaptive segmentation in improving vision models.
We sincerely thank Quentin Garrido, Yejin Bang, and Willy Chung for their valuable discussions, insightful feedback, and constructive comments on this work. Their input has significantly contributed to improving the clarity and qual- ity of this paper. Samuel Cahyawijaya, Jianfeng Liu and Baoyuan Wang contributed to this work before Delong Chen and Pascale Fung joined FAIR at Meta.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Limitations
Despite the advantages of EPOC and adaptive (especially subobject-level) tokenization, there are several limitations and areas for future improvement. Below, we outline several key considerations:
References
Aasan, M., Kolbjornsen, O., Solberg, A. H. S., and Rivera, A. R. A spitting image: Modular superpixel tokenization in vision transformers. CoRR, hass22408.06680 2024. doi: 10.48550/ARXIY.2488000
Our boundary prediction model is based on SegFormer, which produces feature maps at quarter of the input resolution. Consequently, very fine or thin structures (e.g., strands of hair texts in small font) may not be segmented precisely. Using different architecture and with higher resolution output can be a potential way to alleviate this issue.
Achanta, R., Shaji, A., Smith, K.,, Lucchi, A., Fua, P., and Süsstrunk, S. SLIC superpixels compared to state-of-the- art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell., (4(11):2274-223 2012. doi: 10. 109/TPAM.. 2012.120
Our extrinsic evaluations employ LLMS with max- imum of 1.7 Billion parameters. While this scale is already substantial for vision backbones (compared to ViT), it remains limited compared to the largest LLMs available.
Alayrac. J., Donahuc, J., Luc, P., Micch, A., Barr, I., Has- son, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z. Samangooei. S., Monteiro, M.. Menick, J. L., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. Flamingo: a visual language model for few-shot learning. In Koyejo, S., Mohamed, S., Agar- wal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Ad- vances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing
On the other hand, although EPOC is lightweight and fast, it still adds separate segmentation process. This overhead may be less critical for computationally heavy VLM training or inference pipelines but war- rants attention in fatency-sensitive applications. We believe, however, that this overhead is more than offset by efficiency gains- particularr in high-dimensional data such as video (Zheng et al., 2025)- where token number explosion is persistent bottleneck (and mod-

---

Subobject-leve Image Tokenization
Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Cover, T. M. Elements of information theory, John Wiley & Sons, 1999.
Arbeláez, P., Maire, M., Fowlkes, C. C., and Malik, J. Con- tour detection and hierarchical image segmentation. IEEE Trans, Pattern Anal. Mach. Intell., 33(5):898 916, 2011. doi: 10.1 2..1099/PAM..200.166
de Geus, D., Meletis, P., Lu, C.,, Wen, X., and Dubbel- man, G. Part-aware panoptic segmentation. In IEEE Conference on Computer Vision and Pattern Recogni- tion, CVPR 2021, virtual, June 19-25, 2021, pp. 5485- 5494, Computer Vision Foundation IEEE, 2021. doi: 10w19//9944477..2202.12..
Barlow, H. B. et al. Possible principles underlying the trans- formation of sensory messages. Sensory communication, 1(01):217-233, 1961.
Dehaene, S., Al Roumi, F., Lakretz, Y., Planton, S., and Sablé-Meyer, M. Symbols and mental programs: a hy- pothesis about human singularity. Trends in Cognitive Sciences, 26(9):551-76,, 2022.
Biederman, I. seeenitton-b--oomonene a theory of human image understanding. Psychological review, 94 (2):115, 1987.
Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J. S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, L., Lu, J., Anderson, T., Bransom, E., Ehsani, K., Ngo, H., Chen, Y., Patel, A., Yatskar, M., Callison-Burch, C., Head, A., Hendrix, R., Bastani, F., VanderBilt, E., Lambert, N., Chou, Y., Chheda, A., Sparks, J., Skjons- berg, S., Schmitz, M., Sarnat, A., Bischoff, B., Walsh, P., Newell, C., Wolters, P., Gupta, T., Zeng, K., Bor- chardt, J., Groeneveld, D., Dumas, J., Nam, C., Le- brecht, S., Wittlif, C., Schoenick, C., Michel, O., Kr- ishna, R., Weihs, I.., Smith, N. A., Hajishirzi, H., Gir- shick, R. B., Farhadi, A., and Kembhavi, A. Molmo and pixmo: Open weights and open data for ofte-offthhear multimodal models. CoRR, abs/2409. 2024. doi: 10448550/ARXIV.240011
Cahyawijaya, S., Lovenia, H., Koto, F., Putri, R. A., Ceng- goro, T. W. Lee, J., Akbar, S. M., Dave, E., Shadieq, N., Mahendra, M. I., Putri, D. A., Wilie, B., Winata, G. I., Aji, A. F., Purwarianti, A., and Fung, dol: Open instruction-tumi
Cen- generative large language models for indonesian languages. In Ku, L.,, Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd An- nual Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 14899-14914, As- sociation for Computational Linguistics, 2024. doi: 10. 11186////222AAAALLLLNGG779999
Canny, F. computational approach to edge detection. IEEE Trans. Patlern Anal. Mach. Intell., 676)667, 1986. doi: LLL10///PAMI..18646777773
Delétang, G., Ruoss, A., Duquenne, P., Catt, E.,, Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchi- son, M., Orseau, L., Hutter, M., and Veness, J. Language modeling is compression. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vi- enna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F. and Lin, D. Sharegpt4v: Improving large modti-modal models with better captions. In Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., and Varol, G. (eds.), Computer Vision ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XVII, volume 15075 of Lecture Notes in Computer Science, pp. 370-387. Springer, 2024. doi: 10. 00.1077997833-31-722653
Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255 IEEE Com- puter Society, 2009. doi: 10.1109/CVP..200.55222
22.
Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. In Proceedings of the 37th International Conference on Machine Learning, ICML, 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1691-1703. PMLR, 2020.
Deng, X., Yu, Q.. Wang. P., Shen, X., and Chen, L. Co- conut: Modernizing COCO segmentation. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. (1863-21873, IEEE, 2024. doi: 10.1109/CVPR22733 2024.02065.
Cheng, B., Misra, I,, Schwing, A. G., Kirillov, A., and Gird- har, R. Masked-attention mask transformer for universal image segmentation. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 1280-1289. IEFE, 2022. doi: 10.1 10.1109/CVPR52688. 2022.00135.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition scale. In 9th International Conference
10

---

Subobject-Ieve Image Tokenization
on Learning Reprexentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. Open 2021.
Findings of ACL, pp. 3953-3964, Association for Com- putational Linguisties, 2020. doi: 10.18653/V1/2020,
FINDINGS-EMNN
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A.,
Hansen-Fstruch, P., Yan, D., Chung, C.-Y., Zohar, O., Wang. J., Hou, T., Xu, T., Vishwanath, S., Vajda, P., and Chen, X. Learnings from scaling visual tokenizers for reconstruc- tion and generation. arXiv preprint arXiv:2501.09755 2025.
Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozière, B., Biron, B., Tang, B., Chem,, B., Caucheteux, C., Nayak, C.,, Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C.,, Wu, C., Wong. C., Ferrer, C. C., Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choud- hary, D., Mahajan, D.,
Haurum, J. B., Escalera, S., Taylor, G. W., and Moeslund, T. B. Which tokens to use? investigating token reduc- tion in vision transformers. In IEEE/CVF International Conference on Computer Vision, ICCV 2023 Workshops, Paris, France, October 2-6, 2023, pp. 773-783. IEEE, 2023. doi: LL109/CCCVW6999.2222300000
D.. Perino, D., Hup- kes, D., Lakomkin, E.,, AlBadawy, E.,. Lobanova, E., Di- nan, E., Smith, E. M., Radenovic, F,, Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L.,, Nail, G., Mialon, G., Pang. G. Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Tou- vron, H., Zarov, I., Ibarra, I. A., Kloumann, I. M., Misra, I. Evtimov, I., Copet, J., Lee, J., Geffert, J.. Vranes, J., Park, J., Mahadcokar, J., Shah, J., van der Linde, J., Bil- lock, J., Hong. J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., and et al. The Ilama 3 herd of models. CoRR, abs/2407.21783, 2024. doi: 00.4850/ARXII.240.21111
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. B. Masked autoencoders are scalable vision learners. In IEEF/CVF Conference Computer Vision and Put- tern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 15979-15988. IEEE, 2022. doi: 10. 10.11999CPR52288822220000 202..2222....22221......2
Huang, H., Zhou, X., Cao, J., He, R., and Tan, T. Vi- sion transformer with super token sampling. CoRR, abs/2211.1116) 2022. doi: 10.48550/ARXIV.22111 11167.
Felzenszwalb, P. F. and Huttenlocher, P. Pictorial struc- tures for object recognition. Int. Comput. Vis., 61()):55- 79, 2005. doi: /10.////.IILL.000444455551.1.11.
Jain, J., Li, J., Chiu, M., Hassani, A., Orlov, N., and Shi, H. Oneformer: One transformer to rule universal image segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver. BC, Canada, June 17-24, 2023, pp. 2989-2998. IEEE, 2023. doi: 10.109/CVPRS5729.22222. 202.32902222.22..00222...
Feng, Z. and Zhang, S. Efficient vision transformer via token merger. IEEE Trans. Image Process., 32:4156- 4169, 2023. doi: 10.1109/TIP220033329330
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., and Girshick, R. B. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vi- sion and Pattern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017, pp. 1988-1997, IEEE Computer Society, 2017. doi: 10. D0.110//CVP..2077..00
Gastaldi, J. L., Terilla, J., Malagutti, L., DuSell, B., Vieira, T., and Cotterell, R. The foundations of tokeniza- tion: Statistical and computational concerns. CORR, abs/2407.11600 2024. doi: 10448550/ARXIV2400. 11606.
Goldman, O., Caciularu, A., Eyal, M., Cao, K., Szpek- tor, I., and Tsarfaty, R. Unpacking tokenization: Evalu- ating text compression and its correlation with model performance. In Ku, L., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computa- tional Linguisties, ACI. 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 2274-2286. Association for Computational Linguistics, 2024. doi: 10. 10.18653/VJ1 FWN.DIAAA.IIIINNSS.S
Ke, T.-W., Mo, S., and Yu, S. X. Learning hierarchical image segmentation for recognition and by recognition. arXiv preprint
0at::222000031444 2022.
Kim, Y. K., Di Martino, M., and Sapiro, G. Vision trans- formers with natural language semantics. arXiv preprint urXiv:2402.1 17863, 2024.
Kirillov, A., He, K., Girshick, R. B., Rother, C., and Dollár, P. Panoptic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 9404 9413. Computer Vision Foundation IEEE, 2019. doi: 10.1109/CVPR.200 201..P....10000110 9.000093..
Gowda, T. and May, J. Finding the optimal vocabu- lary size for neural machine translation. In Cohn, T., He, Y., and Liu, Y. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of

---

Subobject-leve Image Tokenization
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, I.., Xiao, T., Whitehead, S., Berg, A. C.,, Lo, W., Dollár, P., and Girshick, R. B. Segment anything. In IEEE/CVF International Conference on Computer Vi- sion, ICCV 2023, Paris, France, October 1-6, 2023, pp- 3992-4003. IEEE, 2023, doi: 10.1009/CCCV5100000023.. 00371.
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tun- ing. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural In- formation Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- - 16, 2023, 2023.
Kornilov, A. S. and Safonov, I. V. An overview of wa- tershed algorithm implementations in open source li- braries. J. Imaging, 4(10):123, 2018. doi: 10.3390/ JIMAGING4100123.
Liu, Z., Xu, Y., Winata, G. L., and Fung, P. Incorporat- ing word and subword units in unsupervised machine translation using language model rescoring. In Bojar, O., Chatterjee, R., Federmann, C., Fishel, M., Graham,
Y., Haddow, B., Huck, M., Jimeno- Jimeno-Yepes, A., Koehn, P., Martins, A., Monz, C., Negri, M., Névéol, A., Neves, M. L., Post, M., Turchi, M., and Verspoor, K. (eds.), Proceedings of the Fourth Conference on Machine Trans- lation, WMT 2019, Florence, Italy, August 1-2, 2019 Volume 2: Shared Task Papers, Day I, pp. 275-282 Association for Computational Linguistics, 2019. doi: 10.18653/V1/W195555
Kudo, T. Subword regularization: Improving neural network translation models with multiple subword candidates. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Compulational Linguisties, ACI. 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 66-75, Association for Computational Linguistics. 2018. doi: 10.18653/V1/ P18- 1007
Locatello, F., Weissenborn, D., Unterthiner, T., Mahen- dran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention. In Larochelle, H.,, Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Process- ing Systems 33: Anmual Conference on Neural Informa- tion Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. Human-level concept learning through probabilistic pro- gram induction. Science, 350(6266): 133--1338, 2015.
Lew, J., Jang, S., Lee, J., Yoo. S., Kim, E., Lee, S., Mok, J., Kim, S., and Yoon, S. Superpixel tokenization for vision transformers: Preserving semantic integrity in visual tokens. CoRR, abs/2412.04680, 2024. doi: 10.48550/ARXX..2412.006
Marin, D., Chang, J. R., Ranjan, A., Prabhu, A., Rastegari, M., and Tuzel, O. Token pooling in vision transformers for image classification. In IEEE/CVF Winter Confer- ence on Applications of Computer Vision, WACV 2023, Waikoloa, HI, USA, Jamuary 2-7, 2023, pp. 12-21. IEEE, 2023. doi: 10.1 (0.1100/AACV5668..02223
Li, X., Wang, Z., Zhang, B., Sun, F., and Hu, X. Recog- nizing object by components with human prior knowl- edge enhances adversarial robustness of deep neural net- works. IEEE Trans. Pattern Anal. Mach. Intell., 45(7): 8861-8873, 2023. doi: 2..100///AAMM.223322222
Marr, D. Vision: A computational investigation inio the hu- man representation and processing of visual information. MIT press, 2010.
Li, X., Liu, Y., Dong, N., Qin, S., and Hu, X. Partima- genet++ dataset: Scaling up bast- models for ro- bust recognition. In Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., and Varol, G. (eds.), Computer Vision ECCV 2024 18th European Confer- ence, Milan, Italy, September 29- 19-October 4, 2024, Pro- ceedings, Part LXXI, volume 1 15129 of Lecture Notes in Computer Science, pp. 496. Springer, 2024. doi: 10.1007/978- 0-00779978--0031-3220.00 .2..
Mielke, S. J., Alyafeai, Z., Salesky, E.,, Raffel, C., Dey, M., Gallé, M., Raja, A., Si, C.,, Lee, W. Y., Sagot, B., and Tan, S. Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP. CoRR, 1bs/2112..10003 2021.
Muller, B., Gupta, D., Fauconnier, J., Patwardhan, S., Vandyke, D., and Agarwal, S. Languages you know influence those you learn: Impact of language characteris- ties on multi-lingual text-to-text transfer. In Albalak, A., Zhou, C., Raffel, C.,, Ramachandran, D., Ruder, S., and Ma, X. (eds.). Transfer Learning for Natural Language Processing Workshop, 03 December 2022, New Orleans, Louisiana, USA, volume 203 of Proceedings of Machine Learning Research, pp. 802 PMLR, 2022.
Lin, T., Maire, M., Belongie. S. J.. Hays. J., Perona, P., Ra- manan, D., Dollár, P., and Zimickk C. L. Microsoft COCO: common objects in context. In Fleet, D. J., Pajdla, T., Schiele, B., and Tuytelaars, T. (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part V. volume 8693 of Lecture Notes in Computer Science, pp. 740-755. Springer, 2014. doi: 10.1007/978- 3-77883--1991 3-319-1000220 48.
12

---

Subobject-Ievel Image Tokenization
Nguyen, D.-K., Assran, M., Jain, U., Oswald, M. R., Snoek, C. G., and Chen, X. An image is worth more than 16x16 patches: Exploring transformers arXiv preprint arXiv:2406.09411 09415, 2024.
and Chen, Y. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 678-702 Association for Computational Linguisties, 2024.
individual pixels.
Oquab, M., Darcet, T., Moutakanni, T. Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P., Li, S., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jégou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. Dinov2: Learn ing robust visual features without supervision. Trans. Mach. Learn. Res., 2024, 2024.
Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Pro- ceedings of the 54th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2016, August 7-12. 2016. Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguisties, 2016. doi: N0.18653/V//16-1
Pagnoni, A., Pasunuru, R., Rodríguez, P., Nguyen, J., Muller, B., Li, M., Zhou, C., Yu, I.,, Weston, J., Zettle- moyer, L., Ghosh, G., Lewis, M., Holtzman, A., and Iyer, S. Byte latent transformer: Patches scale bet- ter than tokens. CoRR, abs/2412.09871, 2024. doi: 10.45550AAXX..2412.0085
Simoncelli, E. P. and Olshausen, B. A. Natural image statis- tics and neural representation. Annual review of neuro- science, 24(1):1193-1 1216, 2001.
Sitawarin, C., Pongmala, K., Chen, Y., Carlini, N., and Wagner, D. A. Part-based models improve adversarial robustness. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. Open 2023.
Palmer, S. E. Hierarchical structure in perceptual represen- tation. Cognitive psychology, 9(4)444 -4744,, 1977.
Song, X., Salcianu, A., Song, Y., Dopson, D., and Zhou, D. Fast wordpiece tokenization. In Moens, M., Huang, X., Specia, L., and Yih, S. W. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 2089- 2103. Association for Computational Linguistics, 2021. doi: 10. 18.16553V 1tt.://..0022212.... WWW.EEENNNLLN....
Perret, B., Chierchia, G., Cousty, J., aes, S. F. G., Kenmochi, Y., and Najman, L. Higra: Hierarchical graph analysis. SoftwareX, 10:1-6, 2019. ISSN 2352-7110. doi: 10.1016/ 2019. 100335. 2033.0
2.10f
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision. In Meila, M. and Zhang. T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748- 8763. PMLR, 2021.
team, L., Barrault, L.., Duquenne, P., Elbayad, M., Kozhevnikov, A., Alastruey, B., Andrews, P., Coria, M., Couairon, G., Costa-jussà, M. R., Dale, D., Elsa- har, H., Hefferman, K., Janeiro, J. M., Tran, T., Rop- ers, C., Sánchez, E.,, Roman, S., Mourachko, A., Saleem, S., and Schwenk, H. Large concept models: Language modeling in sentence representation space. CoRR, abs/2412.008811 2024. doi: 10.48550/ARXIIV. 2412.08821.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, ICML 2021, 18-24 July 2021, Virtual Event. volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831. PMLR, 2021.
Tong, S., Liu, Z., Zhai Y., Ma, Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA. June 16-22, 2024, pp. 9568-9578. IEEE, 2024. doi: 10. 11111//VVR5233..222..0000
Ronen, T., Levy, O., and Golbert, A. Vision transform- ers with mixed-resolution tokenization. In IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, CVPR 2023 Workshops, Vancouver, BC, Canada, June 17-24, 2023, pp. 4623-4622. IEEE, 2023. doi: 10.1 1AL10///VPRW552222000000
Tversky, B. and Hemenway, K. Objects, parts, and cate- gories. Journal of experimental psychology: General, 113(2):169, 1984.
Vincent, L. M. and Soille, P. Watersheds in digital spaces: An efficient algorithm based on immersion simulations. IEEE Trans. Pattern Anal. Mach. Intell., 13(6):583 598, 1991. doi: 10. 1 103444.87344.
Schmidt, C. W., Reddy, V., Zhang, H., Alameddine, A., Uzan, O., Pinter, Y., and Tanner, C. Tokenization is
more than compression. In Al-Onaizan, Y., Bansal, M.,
13

---

Subobject-leve Image Tokenization
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Linzen, T., Chrupala, G., and Alishahi, A. (eds.), Proceedings of the Workshop: Analyzing and Interpret- ing Neural Networks for NLP, BlackboxNLP@G EMNLP 2018, Brussels, Belgium, November 1, 2018, pp. 353- 355. Association for Computational Linguistics, 2018. doi: 10. 00653//11W11-333
8689 of Lecture Notes in Computer Science, pp. 818-833.
Springer, 2014. doi: 10. 10.1007/978-3-31--110 10590-1110091111 1 53.
Zhang. C., Han, D., Zheng. S., Choi, J., Kim, T., and Hong, C. S. Mobilesamv2: Faster segment anything everything. CoRR, abs/2312.09579, 2023. doi: 10. 10.4550/ARXI22312.5555
Zhao, X., Ding, W., An, Y., Du, Y., Yu, T., Li, M., Tang, M., and Wang. J. Fast segment anything. CoRR, abs/2306. 12156, 2023. doi: 10.48550/ARXIV22306.
Wang, J., Zhang, P., Chu, T., Cao. Y., Zhou, Y., Wu, T., Wang, B., He, C., and Lin, D. V3det: Vast vocabulary visual detection dataset. In IEEE/CVF International Con- ference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 19787-19797. IEEE, 2023. doi: 10. 10.10000CCCS111100222..0000
12156.
Zheng, C.,, Zhang, J., Salehi, M., Gao, Z., Iyengar, V., Ko- bori, N., Kong, Q., and Krishna, R. One trajectory, one token: Grounded video tokenization via panoptic sub- object trajectory. arXiv preprint arXiv:2505. 23617, 2025.
Wilie, B., Vincentio, K., Winata, G. I., Cahyawijaya, S., Li, X., Lim, Z. Y., Soleman, S., Mahendra, R., Fung, P., Bahar, S., and Purwarianti, A. Indonlu: Benchmark and resources for evaluating indonesian natural language derstanding. In Wong, K., Knight, K., and Wu, H. (eds.), Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natu- ral Language Processing, AACL/JCNLP 2020, Suzhou, China, December 4-7, 2020. pp. 843-857. Association for Computational Linguistics, 2020.
Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., and Torralba, A. Semantic understanding of scenes through the ADE20K dataset. Int. J. Comput. Vis., 127 (3):0023321, 2019. doi: 10.1007/S11263-000 1140--0
Zouhar, V., Meister, C., Gastaldi, J. L.., Du, L.., Sachan, M., and Cotterell, R. Tokenization and the noiseless channel. In Rogers, A., Boyd- Graber, J. L.,, and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9- 14, 2023, pp. 5184-5207. Association for Computational Linguistics, 2023. doi: 10. 18613/6// 284.
Wu, K., Otoo, E. J., and Suzuki, K. Optimizing two- pass connected-componeet labeling algorithms. Pat- tern Anal. Appl., 12(2):117-135, 2009, doi: 10.1007/ $10044-008-0109-7
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Álvarez, J. M., and Luo, P. Segformer: Simple and efficient design for semantic segmentation with transformers. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 12077-12099 2021
Yamagiwa, H., Takase, Y., Kambe, H., and Nakamoto, R. Zero-shot edge detection with SCESAME: spectral clustering based ensemble for segment anything model estimation. In IEEE/CVF Winter Conference on Appli- cations of Computer Vision Workshops, WACVW 2024- Workshops, Waikoloa, III, USA, January 1-6, 2024, pp. 541-551. IEEE, 2024. doi: 10.1109/WACVW66836. 2024.00064.
Zeiler, M. D. and Fergus, R. Visualizing and understand- ing convolutional networks. In Fleet, D. J., Pajdla, T., Schiele, B., and Tuytelaars, T. (eds.), Computer Vision ECCV 2014- - 13th European Conference, Zurich, Switzer- land, September 6- 12, 2014, Proceedings, Part I, volume
14

---

Subobject-leve Image Tokenization
Appendix
A. Token Segmentation Models
Patch-based Tokenization. We divide the image into non-overlapping p x p patches. The p ranges from to 32 in intrinsic evaluations ($4) and ranges from 9 16 in extrinsic evaluations ($5).
Panoptic Segmentation. We employ Fask2Former and OneFormer, two popular types panoptic segmentation models. Our implementation is based on their official releases in Huggingface, which include models with different sizes and models trained on both COCO and ADE20K:
Superpixel Segmentation. We vary the k in (29, 32, .2.... ... ]62} for k-means clustering, effectively controlling the number of superpixels (Fig.7). We use the SLIC implementation in s kit--imaee library (url), where the corresponds to the n_segments parameter. Other hyperparameters are kept as default.
Figure 7. SLIC superpixel segmentation results with different k.
SAM, FastSAM, MobileSAMv2. Token segmentation is generated by prompting the SAM on a regular grid of points and merge the resulting masks using the official "automatic mask generation" implementation. We test SAM backbones with different sizes: ViT-B and ViT-H. To explore the limit of fine-grained segment everything, we further increase the prompt density from the default 32 x 32 to 48 x 48 and 64 x 64 and include one-layer multi-scale augmentation (MS) based on the ViT-H model. One the other hand, we scale down the density to 24 x 24, 16 x 16 and 8 x 8 based on the smallest ViT-B model for more conservative and efficient segmentation. We also include two efficient variants of SAM: FastSAM (github url) which uses lightweight CNN detector instead of ViT, and MobileSAMv2 (github url) which replaces grid point prompt with a learned proposal, reducing the number of forward pass through the mask decoder. Fig. 8 compares their segmentation results on SA-1B samples.
Proposed EPOC
For boundary detection, we trained a SegF o rmer-b0 model on SA-IB dataset for 2 epochs, where mask annotations are converted into binary boundary maps as the target. Specifically, for every segmentation in an image, we computed the difference between its dilation and erosion using a circular kernel with the size of 5, and stacked them together as the boundary label.
The training was performed on a single NVIDIA 8xA100 machine. We used a effective batch size of 64, learning rate of le-4 with a constant schedule and 5000 warmup steps. The model was optimized using AdamW with weight decay of 0.05. Images were resized to 1024x 1024, and the loss was computed as a pixel-wise binary cross-entropy. During training, we applied random horizontal flipping and color jittering as data augmentation. Fig. 9 visualizes the boundary probability maps predicted by the trained model on unseen Pixmo-cap samples with 1024px input resolution. We apply watershed algorithm
15

---

Subobject-Ieve Image Tokenization
VIT-8 Mall
VIT-B 300I0
VIT-B 24x24
VAT-B J2X32
VAT-H J2X32
VST-H GECAU
VIT-H 64b644
VIT-H G4bo4 MS
FAST YOLOVE-X
HoM21e5AW/2 ViT-I
BR
Figure "Segment Everything" results from different SAM models.
(with Beekkit-iaage sigmoid normalized boundary probability map to get panoptic segmentation. Fig. 10 visualizes how different threshold t. affects the segmentation granularity.
DANGER
SWITCH BLADES al Faaly
DAPGEE
Herling and Bomts
Aer Lin
40ED
Figure 9, Visualization of predicted boundary probability maps on Pixmo-cap samples.
16

---

Subobject-level Image Tokenization
8.01
0.85
0.1
0.25
0.4
t 0.6
Seed Regions (labeled color)
M
M
M
Input Image
Boundary Detection
Watershed Segmentation
Boundary Probability Map
Token Segmentation M
M
M
M
M
M
M
2.5k
910 tokens
= 223 tokens
= tokens
tokens
tokens
tokens
Figure 10. The threshold provides flexible control over segmentation granularity: lower produces more seed regions and thus finer segmentation, while higher results in fewer, larger tokens.
B. Intrinsic Evaluations (Extended)
Datasets. For COCO, PPP, PIN++ and SAIB, we randomly sample 3k images for efficient evaluation. For ADE-20K, we include all 2k samples in the validation set. As we employ boundary-based metrics for evaluation, we convert their mask annotations into boundaries by applying morphological dilation and crosion operations with a kernel size of 3 on the mask, and then taking the difference to isolate the boundary. Fig. 11 provide extended visualization of dataset examples.
Boundary Precision and Recall. small to tolerancereaall = 5px is applied to the predicted boundaries to compen- sate for slight pixel misalignments when calculating recall.
Token Monosemanticity Score. A token is considered monosemantic it does not include pixels from more than one ground-truth semantic region. Formally, let X HR x3 an image, M E (0...... ...0.....111... N-13 HW the predicted token map, and M {0. .O..... 1-1) #x the ground-truth. The i-th predicted token is T, {(h, w) M[h, w| i}.
Define the indicator:
3k s.t. V(h, E), M" w] k 0, otherwise.
1,
Mono
Then the monosemanticity score is defined as:
Mono(M,,M') N
mono (Tr).
This class-agnostic score quantifies how often a token is wholly contained in single semantic region, capturing the absence of polysemantic tokens. To measure this: 1) we erode each predicted token segment by toleranceeoneemeent iolty 2 25px so that only the "core" of each token remains; 2) Tokens whose eroded region does not intersect a ground-truth boundary are deemed monosemantic. This captures whether token merges multiple semantics into single segment.
Token Size Distribution. To better understand how each segmentation method allocates tokens, we analyze the relative area of each token by sorting them from largest to smallest. Figure 12 plots the average token size as percentage of the total image area on a log scale, with the z-axis denoting the rank of the token in descending order of size. Patch-based segmentation yields uniform token size distribution. panoptic segmentation, only a few tokens occupy large portion
17

---

Subobject-Ievel Image Tokenization
Objeck-eevel
Subobject-level Image Tokenization
Patches
Panoptic Segmentation
Superpixal
Segment Anything Models
Ours
Ground
Patch
Patch
Mask2Former
OneFormer
SLIC
SAM
FastSAM
MobileSAMv2 VIT-H
EPOC
EPOC
tiny
1arge
t=0.05
cocO object-eeve
ADE20K abjjest--ee abjesdi-eeee sel---vvell
Pascal Panoptic Parts subcbject-ieeel
PartimageNet subebject-leve
SA-1B subobject leveel
Nammal
Portes Rocrs
Palbil law Poloeli kg
Partiolars Necre
Figure 11, Intrinsic evaluation dataset examples and token segmentation results.
of the image. EPOC exhibit a pronounced long-tailed distribution. Those observations can explain our findings in the token truncation experiments ($5.3).
Computational Efficiency. We measure throughput on an NVIDIA V100 (32GB) with 30 CPU cores by gradually increasing the number of parallel processes, each running one tokenizer. During these multi-process runs, we record wall-clock time, GPU memory consumption, GPU utilization, and any out-of-memory events. This setup reveals how well each tokenizer scales in concurrent inference setting and highlights potential memory bottlenecks when more processes added. Fig. 13 visualize the dynamics of FPS and average GPU utilization with increasing number of process.
C. Extrinsic Evaluations (Extended)
Datasets. Fig. 14 provide visualization dataset examples. ImageNet- and CLEVR provide official validation splits, we use 5k samples from them for efficiency. For Pixmo-cap, we randomly sample Sk samples as validation, and for ShareGPT-44v, we treat 5k samples randomly selected from the GPT-4V generated captions as validation split.
18

---

Subobject-Ivvel Image Tokenization
Patch
Panoptic Segmentation
Superpixel([LICC
SAM
EPOC (Ours)
10009
100%
300%
300%
175
109%
10%
10%
10%
07.
Weletive Token Size (log
17.
175
170
1.34
234
10008
3038
11M
31M
Token Position
Position
Taken Position
Taken Position
Tokee Position
Figure 12. Visualization of token size distribution.
TENGAY
1P3C
Mamber Processn
Number Precusom
Namber Processes
Namber Procesoes
Risktheneer
Figure 13. Detailed computational efficiency evaluation results.
VLM Architecture and Training We use two-layer MLP as the connector between embeddings and LLM. The width is x4 of LLM's hidden state dimension. We freeze the image feature extractor and do end-to-end fine-tune the small MLP projection plus the LLM. For CLEVR-cap,, ImageNet- Ik, ShareGPT4V, and Pixmo-cap datasets, we respectively train the model for 30, 1, 1, epochs, with a batch size 512, 256, 256, and 256. Max tokens are set to 100 for EPOC and 64 for Mask2Former tokenizer. We use Adam W with learning rate 10-4 cosine decay or constant scheduling, and 500 warmup steps. Mixed-precision 66 is used to accelerate training. We do standard language-modeling next-token prediction text tokens only.
Convergence Speed We measure convergence speed using average training loss, which effectively shows how quickly the model fits the training data. Fig. 15 (top) demonstrates that the convergence speed correlate well with generalization performance (bottom), both showing the advantage of subobject-level image tokenization.
19

---

Subobject-Ieve Image Tokenization
CLEVR-cap (70k samples)
objects: logg gny cube, purple cube, large green pny rubber sphero, lange eyen
ImageNet-Ik (1.28M samples)
ShareGPT4y (1.24M samples)
The capures the breathaking dominats
Harbin sculptures mesiculously cratted blacks
perspective
ice angle, looking details of ice and snow. shove
the buildings whise, reflecting
colors image are geedeminantly blue, peoviding a sriking costrast to the loy landscape below. The image ukkn during the day, and the brigt sunlight illuminstes the solpiases, casting long shudows and hightightng Be texture of the ice. The ovenall effect is one oww and admiration for the urtstry and skill involved in creating such spectuole.
creativity
Pixmo- Saxmo-Cap (707k samples)
this indastial--yyle kitches, builing soene leatures large silver gallon pot the backgoond and stainees steel cooking stniien the Groggound. Ceneer stage two sizable with black motal handlex and stinleas steel bottems, beth brimming with culerful arzy ingrodients. The pmn the left loden with spircch aragode,
chanks Meanvchile, com, lomatocs, Eary rell dhilies, and wome ground mest mony streciop waggesos a lively cooking acssion, crumbl
pieses resamble potato
possibly
nal
right showesses yellow scattered and two sauce bottles one red and ore clear posiiiined to the für right. Overhead stanles steel shelves and pots add to the inductions anbitace the kitchen, This vibrane scene captires the making of bearty breakfast, potentially scrambled mixtare or in omelet, full of frest, colectul ingredients sizzling together.
Figure 14. Examples from extrinsic evaluation datasets that used train VLMs.
CLEVR-cap
ImagoNet-1K
ShareGPT4V
Pixmo-cap
02
1.6
2.4
233
Paich
Parch
Patch
Falthh
Panoplic Sngmenizzttoo
0.5
2.30
1.5
Superpted rotel (SLIC)
Panopiic
Segmentation
(Ours)
03
Supersteal (S.IC)
Panoptic Segmaniarion
Superpieel (SLIC)
EPOC (Ours)
10: Superptel (SLIC)
2.64
Panoolic
2.5
Segmeniation
EPOC (Qam)
5.0
Wilcation Perpeity
Patch
Pasch
Patch
Pazch
Panoplic
4.4
mngmeniation
Panoolic Segmeniatoos
Panoptic
Superptenl (SUCC)
Seg on
Panopric
1.2
Superpas (SUIC)
Superpieal (BLIC)
(Cus)
Superstest (SLIC)
EPOC (Our)
EPOC (Curs)
Tokknn (log acall)
Tokons (og scale)
Tollan (log scale)
Takens (lng scale)
Figure 15, Comparing average training loss (measuring convergence speed) and validation perplexity.
20


Reviews:
[{"review_id": "QzDNWFW3N5", "reviewer": "Reviewer_Ev7G", "summary": "This paper introduces **Subobject-level Image Tokenization**, a novel adaptive image tokenization strategy inspired by subword tokenization in NLP. Previous patch-based image tokenization methods suffer from inefficiencies and polysemanticity. To address these limitations, the paper proposes a new tokenizer called **Efficient and PanOptiC (EPOC)**, which combines boundary detection and watershed segmentation to guarantee comprehensive segmentation and computational efficiency.\n\nMain Contributions:\n1. **Panoptic Segmentation Improvement** The proposed  EPOC integrates boundary detection and watershed segmentation.\n2. **Computational Efficiency**  The proposed  EPOC achieves better token efficiency.\n\nMain Results:\n1. Intrinsic Evaluation: Evaluations on five datasets demonstrate that EPOC tokens align well with human semantic annotations, achieving higher monosemanticity scores.\n2. Extrinsic Evaluation(VLMs): EPOC-based tokenization achieves faster convergence, improved generalization, and better token efficiency.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and insightful review. Below, we provide detailed responses addressing your specific questions and suggestions:\n\n---\n\n### 1. **Formulating Token Monosemanticity Score** (Question 1)\nYes. Below is the explicit formal definition, which will be added to the final version:\n\n**Definition 1 (Token Monosemanticity Score):** Given an image \\\\( \\\\mathbf{X}\\\\in\\\\mathbb{R}^{H\\\\times W\\\\times 3} \\\\), a predicted token segmentation \\\\( \\\\mathbf{M}\\\\in\\\\{0,\\\\dots,N-1\\\\}^{H\\\\times W} \\\\), and a ground-truth segmentation \\\\( \\\\mathbf{M}^{\\\\ast}\\\\in\\\\{0,\\\\dots,K-1\\\\}^{H\\\\times W} \\\\), define the \\\\( i \\\\)-th predicted token \\\\( \\\\mathbf{t}_i \\\\) as the set of pixel coordinates assigned token index \\\\( i \\\\), i.e., \\\\( \\\\mathbf{t}_i=\\\\{(h,w)\\\\mid \\\\mathbf{M}(h,w)=i\\\\} \\\\).\n\nWe say a token \\\\( \\\\mathbf{t}_i \\\\) is **monosemantic** if it lies entirely within exactly one ground-truth semantic region. Formally, the indicator function of token monosemanticity is defined as follows:\n\n\\\\[\n\\\\mathbb{I}_{\\\\text{mono}}(\\\\mathbf{t}_i)=\n\\\\begin{cases}\n1, & \\\\text{if } \\\\exists k \\\\text{ such that } \\\\forall (h,w)\\\\in \\\\mathbf{t}_i, \\\\mathbf{M}^{\\\\ast}(h,w)=k \\\\\\\\\n0, & \\\\text{otherwise.}\n\\\\end{cases}\n\\\\]\n\nThen, the **Token Monosemanticity Score** is defined as the fraction of all predicted tokens that are monosemantic:\n\n\\\\[\n\\\\text{monosemanticity}(\\\\mathbf{M}, \\\\mathbf{M}^{\\\\ast})=\\\\frac{1}{N}\\\\sum_{i=0}^{N-1}\\\\mathbb{I}_{\\\\text{mono}}(\\\\mathbf{t}_i).\n\\\\]\n\nIntuitively, this metric quantifies how effectively the predicted segmentation avoids polysemantic tokens, which span multiple semantic regions.\n\n---\n\n### 2. **Boundary Detector Backbone** (Question 2 & 4)\n\nWe chose SegFormer primarily due to its popularity, simplicity, and computational efficiency. However, EPOC is definitely not confined to SegFormer. Boundary detection itself is a historically well-studied and relatively straightforward task, effectively handled by various convolutional or Transformer backbones [1-4].\n\n> [1] DeepContour: A deep convolutional feature learned by positive-sharing loss for contour detection. CVPR’15\n>\n> [2] Richer convolutional features for edge detection. CVPR’17\n>\n> [3] EDTER: Edge detection with transformer. CVPR’22\n>\n> [4] DiffusionEdge: Diffusion Probabilistic Model for Crisp Edge Detection. AAAI’24\n\nTo directly address your questions regarding scalability and alternative backbones, we conducted an additional scaling experiment comparing the original SegFormer-b0 to a substantially larger SegFormer-b5. Both models were trained on SA-1B dataset for 2 epochs, and the followings are performance evaluation (boundary recall) results:\n\n| Model | Parameters  | Object-level (ADE20k) | Subobject-level (PPP) |\n|---|---|---|---|\n| SegFormer-b0 (Submission) | 3.7M  | 58.88 | 69.22 |\n| SegFormer-b5 (New)  | 87.5M (x24) | 65.73 (+6.85%)  | 71.46 (+2.24%)  |\n\nThese results clearly show that while EPOC can indeed scale up with larger models, the performance gains are marginal relative to the substantial increase in computational overhead. This further justifies our original choice of the efficient SegFormer-b0.\n\n---\n\n### 3. **Ablation on Position and Content Embeddings** (Question 3)\n\nWe agree that an ablation study on position and content embeddings provides valuable insights into how VLMs utilizing adaptive token segmentation interpret images. We conducted an experiment with our EPOC-based VLM trained on CLEVR-cap, where we shuffled positional or content embeddings across visual tokens, and then evaluated the impact on performance.\n\nSince CLEVR-cap is synthetic and the captions follow a structured template—e.g., *“Total 5 objects: a small gray metal cube, a small red rubber sphere, …”*—it allows us to parse the generation and calculate average accuracy (%) for individual attributes. The results are summarized below:\n\n|  | Count | Size | Color | Material | Shape |\n|---|---|---|---|---|---|\n| No Ablation | 57.0 | 71.4 | 59.3 | 77.7 | 69.3  |\n| Shuffle Position Embeddings | 35.0 | 49.9 | 16.8 | 47.3  | 50.5  |\n| Shuffle Content Embeddings | 48.0 | 66.4 | 20.8  | 52.0  | 59.3  |\n| Chance | 10.0 | 50.0 | 12.5 | 50.0 | 33.3 |\n\nThese results show that both positional and content embeddings contribute to accurately capturing semantic attributes and correctly associating them with their corresponding objects, confirming their complementary roles in adaptive token segmentation-based image understanding.\n\n---\n\n### 4. **Comments and Suggestions on Section 5.2 and Fig. 5**\n\nThank you very much for these practical suggestions. Indeed, Section 5.2 is densely packed due to the strict page limits of the ICML submission format. The final version is allowed an extra page, enabling us to reorganize the presentation. Additionally, we will ensure Figure 5 includes consistent grid lines across all subplots.\n\n---\n\nFinally, thank you again for your constructive and detailed feedback, which significantly improves the quality and clarity of our paper.", "claims_and_evidence": "The claims made by the paper are strongly supported by a clear proposed method (EPOC), well-designed experiments (intrinsic and extrinsic evaluations), and quantitative and qualitative experimental results.", "methods_and_evaluation_criteria": "The proposed methods make clear sense for the problem of image tokenization. The paper carefully compares the proposed methods with the previous one (patch-based tokenization and object-level segmentation).", "theoretical_claims": "There are no theoretical claims that require correctness in this paper.", "experimental_designs_or_analyses": "The experiments in this paper are well-designed.\n1. Intrinsic evaluations are performed across five datasets. The authors focus on boundary precision-recall metrics and monosemanticity score.\n2. Extrinsic evaluations are performed on four datasets which are high-quality and widely-used. Besides, the authors use different feature embeddings to demonstrate the robustness of the proposed method.", "relation_to_broader_scientific_literature": "1. This paper is related to vision transformers using patch-based image tokenization.\n2. This paper is related to the image segmentation models.", "essential_references_not_discussed": "The essential references are well discussed in Section 6.", "other_strengths_and_weaknesses": "Paper Strengths:\n1. The paper is well-written and easy to follow.\n2. The paper introduces a novel analogy from NLP to CV, which is conceptually innovative.\n3. The proposed EPOC achieves computational efficiency, which could lead to high-resolution images with fewer computational resources.\n\nWeaknesses:\n1. Limited novelty. \n2. Limited ablation studies on position and content embeddings.", "other_comments_or_suggestions": "Here are some comments and suggestions for the authors:\n\n- Highly recommend reorganizing the highlighted or underlined words in section 5.2.\n- As for Figure 5, I highly recommend consistently plotting with grids.", "questions_for_authors": "Questions for authors:\n\n1. Can the authors formulate the token monosemanticity score?\n2. Why do you choose **SegFormer** as a boundary prediction model? Did the authors try to use other models to perform the same role, and what could be the performance? Can the authors provide some experiment with other boundary prediction models?\n3. Can the authors provide some ablation on position and content embeddings?\n4. Can the EPOC be scaled up?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IKkcSkIiFf", "reviewer": "Reviewer_avnM", "summary": "This paper proposes sub object-level image tokenization, which tokenize image based on the morphological structure of the image. Compared to other potential subobject tokenizers, EPOC improves efficiency. Experiments on multiple VLMs demonstrate the advantages of the subobject tokenizer.\n\n## update after rebuttal\nDuring the rebuttal, the authors provided additional experiments that addressed my and other reviewers' concerns. I keep my score as accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging review and thoughtful questions. Your recognition of the strengths of our method is greatly appreciated. Below are our clarifications on your valuable questions:\n\n---\n\n### 1.**Extrinsic evaluations on caption data rather than general VQA** (in “Methods And Evaluation Criteria”)\n\nOur main goal was to isolate image understanding performance clearly from complex reasoning tasks. Standard VQA datasets tend to fuse perception with significant **knowledge** and **reasoning** capabilities of LLMs. For example, questions like “Where is he looking?”, or “What time is it on the clock?” which commonly occurs on VQA data requires the VLM to understand concepts like “third-person pronoun”, “sense of direction”, and “time” which can potentially **confound the assessment of the effectiveness of adaptive token segmentation**. Thus, we focused on detailed captioning datasets to precisely evaluate token-level perception capability.\n\n---\n\n### 2. **Practical value regarding the comparison between DINO and CLIP** (in “Methods And Evaluation Criteria”)\n\nThe observed advantage of DINO over CLIP in our experiments primarily results from the higher resolution and dense supervision available in DINO embeddings. CLIP, especially the original OpenAI version, lacks resolution flexibility with a 7x7 feature map, limiting fine-grained perception that is essential for subobject tokenization. We agree CLIP still holds significant practical value, especially when high-resolution features are integrated, which can further enhance subobject tokenization.\n\n---\n\n### 3. **Future direction towards video tokenization** (in “Questions For Authors”)\n\n\nIndeed, as we stated in Appendix E, subobject-level tokenization's greatest potential may lie in video understanding, where computational efficiency and token management are even more critical due to the temporal dimension. Our work provides foundational insights that naturally extend to video data, and we consider this a key future direction.\n\nThank you once again for your valuable input and for clearly identifying the broader applications of our method.", "claims_and_evidence": "All the claims made in the submission supported by clear evidence.", "methods_and_evaluation_criteria": "Regarding the question on extrinsic evaluation, why is the comparison only made on caption data and not on general VQA benchmarks? Is it because it doesn't perform as well? \n\nThe second question concerns Section 5.3, \"Results and Discussions - Compatibility to Different Token Embeddings.\" It mentions that when using the subobject tokenizer, the DINO model performs better than CLIP. However, in mainstream models, DINO does not outperform CLIP. The explanation provided in the paper is that this is due to the lower resolution. Does the author believe that, for now, the subobject tokenizer still has limited practical value?", "theoretical_claims": "yes, I check the correctness of proof in the paper.", "experimental_designs_or_analyses": "Please see Methods And Evaluation Criteria section", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The writing is clear and convincing.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "In my view, under image-LLM, the image tokenizer doesn't really need significant improvements, as even ultra-high resolutions are still manageable for current GPUs. I believe the subobject tokenizer is more suitable for use in video.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zGzKBlInw0", "reviewer": "Reviewer_7brk", "summary": "Tokenization is an important step for any transformer-based model. For the vision transformers, it is often performed on a patch-level, where locally neighboring parts of the image are tokenized together in the form of small square patches. However, patch-based tokenization is not adaptive, that is it tokenizes any given patch equivalently. This work proposes an adaptive image tokenization strategy based on sub-object level features, aiming to address the drawbacks of the existing image tokenization strategies. The work further includes intrinsic (with respect to image) and extrinsic experimental analysis for highlighting the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed and comprehensive review. We sincerely appreciate the considerable effort and depth of analysis you provided! Below, let us address each of your concerns point-by-point and outline concrete steps we will take to improve the final manuscript:\n\n---\n\n### 1. **VLM Architecture in Extrinsic Evaluation** (in “Methods And Evaluation Criteria”)\n\n> **TL;DR**  Our approach is precisely a **minimal adjustment** of the LLaVA architecture. The modification involves adding positional embeddings to **support adaptive token segmentation**.\n\nWe appreciate your comment and realize that our original description might have caused confusion. To clarify, our VLM architecture directly builds upon the well-established **LLaVA framework** (visual encoder → MLP connector → LLM). We do not introduce any new paradigm, such as *\"separate visual encoder and a VLM on top\"* as suggested. In the final version, we will explicitly emphasize this fact in Sec 5.\n\nThe sole reason for the modification is to handle **adaptive token segmentation**, which inherently involves variations in token size, position, and arrangement—an aspect not supported by existing architectures. Apart from this, our training settings and datasets (e.g., using ShareGPT-4V, Pixmo-cap) **align well with common practices** in VLM pretraining.\n\n---\n\n### 2. **Perplexity-based Metric in Extrinsic Evaluation** (in “Methods And Evaluation Criteria”)\n\nWe appreciate this important suggestion. To clarify, we used validation perplexity mainly to **maintain consistency and clarity** across different datasets. We fully agree that additional metrics are valuable. For all data points in Fig. 5, we calculated their accuracy or BLEU as suggested, and measured their correlation with perplexity. As below, two metrics **strongly correlate** with each other. We will include these results in the appendix of the final manuscript:\n\n| |CLEVR-cap | ImageNet | ShareGPT4v | Pixmo-cap |\n|---|---|---|---|---|\n| Metric  | Accuracy  | Accuracy  | BLEU  | BLEU |\n| Pearson Correlation with Perplexity | -0.86$^*$  | -0.97  | -0.93  | -0.91 |\n| p-value | p < 0.0001 | p < 0.0001 | p < 0.0001 | p < 0.0001 |\n\n$^*$The correlation on CLEVR-cap seems weaker compared to others. This is because they exhibit a log-linear relationship according to our visualization. The correlation between perplexity and log(accuracy) is -0.95.\n\n---\n\n### 3. **Boundary-based Metric in Intrinsic Evaluation**  (in “Experimental Designs Or Analyses”)\n\n> **TL;DR** Mask-based metrics are fundamentally unsuitable for evaluating image tokenizers. Boundary-based metrics accurately assess tokenization quality in **class-agnostic** and **multi-granularity** settings and have been widely adopted in CV and NLP.\n\nMask-based metrics, such as mIoU, depend heavily on clearly defined semantic classes, which makes them inappropriate for evaluating our tokenizer. They also struggle when segmentation granularities differ between predictions and ground truth. Accurately subdividing an annotated object into several meaningful subparts may produce **misleadingly low IoU scores**.\n\nBoundary-based metrics naturally overcome these limitations. These metrics have also long been standard in boundary detection literature and have been used by SAM-related studies. They are also applied in NLP tokenizer evaluation, assessing the alignment to linguistic morphological boundaries. \n\nAdditionally, the **Token Monosemanticity Score** is essentially a mask-based metric (see response 1 to reviewer Ev7G).\n\n---\n\n### 4. **Reporting Training Loss in Fig.6.** (in “Questions For Authors”)\n\nThe current Fig.6 is presenting **average training loss**, which is an indicator of convergence speed. To align with Fig.5, we will update it to report **validation perplexity** following your suggestion. In fact, the two metrics closely correlate with each other, as it can be seen in Fig. 15. With validation performance in Fig.6, the relative performance relationship stays **exactly the same** and the conclusions in Sec. 5.3 remains unchanged. \n\n---\n\n### 5. **Relation to Recent Works**  (in “Questions For Authors”)\n\nThank you for highlighting these strongly related recent works. The submission already included [B] (Aasan et al., 2024), and we will cite and discuss [A] in our final version. Both studies explore **adaptive token segmentation**, closely aligning with our research, yet with important differences:\n\n- Compared to EPOC, the superpixel approach in [B] performs bottom-up pixel grouping without **semantic understanding**, and SAM-based segmentation in [A] cannot ensure **efficient** and **panoptic** segmentation. They are already involved as baseline methods in our paper.\n\n- These works focus on **ViTs** and **image classification**, while our approach extends to **VLMs** and evaluates performance on the more challenging task of **detailed image captioning**.\n\n---\n\nOnce again, thank you very much for your comprehensive and insightful comments!", "claims_and_evidence": "The proposed tokenization method claims to improve the patch-based tokenization with respect to token polysemanticity and token redundancy. In addition, it also claims to improve other adaptive tokenization methods with respect to computational efficiency and effective segmentation of subobject level regions.\n\nWith respect to the first claim, the authors present compelling verbal arguments, such as the issue of large patches encompassing multiple concepts (object classes, sub-object level details) and the redundancy coming from patchifying everywhere equivalently, e.g diving a background region into multiple tokens even though it is not very informative. In addition, the visualizations provided in Figure 3 and the quantitative analysis on Figure 4 demonstrate these issues more concretely. Given the fact that these arguments are also well-discussed and demonstrated in both the language and vision literatures, these claims are well-supported.\n\nFor the second claim, the authors present extrinsic evaluation results while utilizing an adjusted encoder-decoder model to accommodate for the dynamic nature of tokens following their tokenization process. While the discussion over the experiments is brief (and there is a lack of clarity in the presented figures, such as Figure 5), it can be seen that the proposed approach achieves better validation perplexity over certain alternatives while matching that of superpixel tokenization. In addition, the authors discuss the efficiency of the proposed approach and provide quantitative results backing this part of the claims. However, as detailed in the methods and evaluation criteria, there are several points that are preventing me from stating that this claim is overall well-supported.", "methods_and_evaluation_criteria": "- While it is understandable that it is non-trivial to utilize dynamically tokenized patches with off-the-shelf VLMs, I find the proposed methodology in Section 5 to be confusing, especially the rationale behind using a completely separate visual encoder and a VLM on top. Given the plethora of models with their own visual encoder and language decoder on top [A, B, C, D], I am not fully sure why the authors performed the analysis the way it is, instead of performing small adjustments on the aforementioned models.\n\n- In addition, only the validation perplexity is presented for the utilized benchmarks. However, various other metrics associated with these benchmarks could have been reported (e.g BLEU scores [E], CIDEr [F] for captioning-based ones, classification accuracy for Imagenet) but were instead omitted.\n\nOther than these points, the methodology seems reasonable.\n\n[A] Liu, Haotian, et al. \"Visual instruction tuning.\" Advances in neural information processing systems 36 (2023): 34892-34916.\n\n[B] Li, Junnan, et al. \"Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.\" International conference on machine learning. PMLR, 2022.\n\n[C] Yu, Jiahui, et al. \"Coca: Contrastive captioners are image-text foundation models.\" arXiv preprint arXiv:2205.01917 (2022).\n\n[D] Tschannen, Michael, et al. \"Image captioners are scalable vision learners too.\" Advances in Neural Information Processing Systems 36 (2023): 46830-46855.\n\n[E] Papineni, Kishore, et al. \"Bleu: a method for automatic evaluation of machine translation.\" Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 2002.\n\n[F] Vedantam, Ramakrishna, C. Lawrence Zitnick, and Devi Parikh. \"Cider: Consensus-based image description evaluation.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.", "theoretical_claims": "The paper does not include detailed theoretical discussions or theorems. However, the intuitions and motivations behind the work are clearly explained.", "experimental_designs_or_analyses": "The experimental analysis presented seems sound, though most of the results follow a rather ad-hoc structure for the work. Specifically, the instrinsic evaluation only discusses the edge-related metrics while omitting mask-level ones. While this is partially understandable given the nature of the work, it is still expected that the patch-based tokenization methods or those which do not necessarily rely on edges may not perform as the presented method. As detailed under the methods and evaluation criteria part, extrinsic evaluation also contains several ad-hoc choices.", "relation_to_broader_scientific_literature": "The work aims to tackle a timely problem in the vision literature, namely adaptive tokenization for vision transformers, for allowing efficient allocation of computational resources to more informative areas of images. Given the prominence of recent similar approaches in the language domain [A], if evidenced strongly, the work could be interesting to the broader transformers/deep learning community.\n\n\n[A] Pagnoni, Artidoro, et al. \"Byte latent transformer: Patches scale better than tokens.\" arXiv preprint arXiv:2412.09871 (2024).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The proposed idea based on the watershed algorithm and edge detection is interesting and novel for image tokenization. The goals of the work are also significant and are very timely. With respect to clarity, I think that the work could benefit from a diagram or a more fluent description of the extrinsic evaluation set-up, otherwise the thoughts are presented fluently.", "other_comments_or_suggestions": "- Minor typo on L105: toknizers -> tokenizers\n- Minor typo on L369: coverts -> converts\n- The Figures 4 and 5 are not very easy on the eye and do not quickly convey the message of the work. I think that the work could benefit from moving some of the sub-figures to Appendix and emphasizing the most significant results from them.", "questions_for_authors": "-  I am not sure what the Figure 6 is presenting: Is it just the convergence with respect to training loss? Would not it benefit from the presentation of generalization performance too?\n- As a very minor question, where do you think this work stands in comparison to [A] and [B]? \n\n\n[A] Kim, Young Kyung, J. Matías Di Martino, and Guillermo Sapiro. \"Vision transformers with natural language semantics.\" arXiv preprint arXiv:2402.17863 (2024).\n\n[B] Aasan, Marius, et al. \"A Spitting Image: Modular Superpixel Tokenization in Vision Transformers.\" arXiv preprint arXiv:2408.07680 (2024).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z5nULSMDLH", "reviewer": "Reviewer_D5bb", "summary": "This paper presents a method to encode an image at sub-object level. Specifically, it first detects edges and boundaries in the image with a small model, then utilizes the watershed algorithm to segment the image into sub-object parts. The authors conduct both intrinsic evaluations to validate the segment quality and extrinsic evaluations to verify the derived sub-object token embeddings.\n\n## update after rebuttal\nThank the authors for the detailed response. However, the rebuttal did not satisfactorily address my concern regarding the lack of VQA evaluation. Given that the primary objective of token segmentation is to improve token embeddings for visual understanding, it is crucial to assess performance on diverse understanding-oriented VQA benchmarks, not just simple image captioning tasks. I am NOT requesting that the authors develop a SOTA VLM, but rather that they provide a fair comparison with CLIP embeddings that do not involve token segmentation on VQA benchmarks. The authors’ reluctance to provide such an evaluation leads me to question the true effectiveness of the proposed method. Therefore, I will maintain my rating as weak reject.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback! We realize that there are some concerns that arise from the interplay between token segmentation and token embedding. Therefore, before addressing your comments point-by-point, we first clarify how and why we disentangle these two components in our paper:\n\nIn this study, we explicitly separate the image tokenization into two stages: 1. **token segmentation** and 2. **token embedding**. Our paper emphasizes the segmentation step, which has been largely overlooked in existing research focusing primarily on improving embeddings. **Our claim** is that: **adaptive segmentation** facilitates better image understanding, and this is consistent across **different embedding** methods.\n\nWe adopted this separation due to several considerations:\n\n- Segmentation and embedding have fundamentally different roles and characteristics. They are agnostic to each other and can be flexibly combined.\n\n- Disentangling these two components allows controlled experiments and clearer attribution of improvements. \n\n- This separation aligns well with previous works investigating token segmentation methodologies, both in language modeling and image understanding literature. \n\nBelow, we provide detailed responses addressing each of your specific comments.\n\n---\n\n### 1. **Token Embedding for Subobject Segmentation** (Weakness 1)\n\n> **TL;DR** Rather than overlooking token embeddings, our work demonstrates that adaptive segmentation consistently enhances image understanding **across various embedding methods**. \n\nExisting studies typically focus on improving token embedding quality while defaulting to simple, uniform patch-based segmentation. In contrast, our paper explicitly addresses the overlooked aspect of adaptive token segmentation. This segmentation method is designed to remain **agnostic** of embedding choices, facilitating a controlled evaluation of segmentation effectiveness across diverse embedding schemes, as in Fig.6 (left).\n\nAlthough popular ViT-based encoders are convenient and commonly used, they are by no means mandatory for our method. Indeed, as in Fig. 5(e), our EPOC achieves **stronger advantages** when coupled with **convolutional VAE** encoders or using raw **RGB pixels**.\n\n---\n\n### 2. **Extrinsic Evaluation Settings** (Weakness 2)\n\n> **TL;DR**  We indeed largely adopt the LLaVA framework, but with some minimal sufficient modifications. Without these changes, the original LLaVA framework **cannot provide meaningful evaluation** for adaptive token segmentation methods.\n\nThe goal of this extrinsic evaluation is to probe the effectiveness of adaptive segmentation, not to propose any new VLM framework. The VLM evaluation setup in our paper closely follows common practices already established in the literature. The commonalities includes:\n\n- We adopt the standard \"Visual Encoder → MLP Connector → LLM\" architecture from LLaVA. \n\n- We adopt visual encoders (CLIP, DINOv2, VAE), MLP settings (2-layers), and LLM (autoregressive decoder-only Transformer) that is well aligned with existing studies.\n\n- We adopted ShareGPT4V and Pixmo-cap, which are both widely used for pretraining VLMs.\n\nHowever, minimal changes were necessary:\n\n- Standard LLaVA assumes fixed-size and raster-ordered patches, thus omitting positional embeddings. We introduced positional embeddings specifically to handle adaptive tokenization, which involves irregular segment shapes and arrangements.\n\n- Standard VLM benchmarks commonly evaluate visual reasoning through tasks like VQA, reflecting both vision and LLM capabilities. Since our goal is exclusively to measure improvements in image understanding capability, we only measure the caption quality—thereby minimizing confounding from LLM knowledge and reasoning factors.\n\n---\n\n### 3. **Throughput Comparison** (Questions For Authors)\n\nAccording to the disentanglement between token segmentation and token embedding, we interpret *“patch-wise tokenizer”* in the question as combining patch segmentation with CLIP or DINOv2 embeddings. A fair throughput comparison thus requires combining the same embedding backbones with subobejct segmentation. Since **embedding methods do not impact segmentation latency**, the throughput conclusions from Table 1 directly apply – *although patch segmentation is negligible in latency, the overhead introduced by our EPOC is minimal and practically insignificant relative to overall VLM training*.\n\n---\n\n### 4. **Structure and Clarity** (Other Comments Or Suggestions)\n\nThank you for this constructive suggestion. While our original intention was to clearly separate the token segmentation from embedding strategies for conceptual clarity, we acknowledge that they are methodologically relevant. We will reorganize our paper in the final version to move \"Token Embedding for Adaptive Segmentation\" (Section 5.1) into the \"Method\" section (Section 3) as suggested.\n\n---\n\nOnce again, we sincerely appreciate your detailed review and valuable suggestions!", "claims_and_evidence": "The claims in this paper supported by experimental results or prior studies.", "methods_and_evaluation_criteria": "The extrinsic evaluation benchmarks used in the paper are not common for the VLM evaluation.", "theoretical_claims": "There is no proof or theoretical claim in this paper.", "experimental_designs_or_analyses": "The experimental designs are solid and fair.", "relation_to_broader_scientific_literature": "This paper studies the image tokenization problem from a novel perspective, i.e., at sub-object level, which is analogous to sub-word tokenization in NLP.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths:**\n- The idea of tokenizing an image at sub-object level is intuitive and intriguing. It reduces the number of tokens required to encode an image compared to traditional patch-wise tokenization, and facilitates fast convergency in downstream tasks.\n- The paper is well-written and easy to follow.\n\n**Weaknesses:**\n- Sub-object image tokenization involves two stages: first, partitioning an image into sub-object parts, and second, encoding each part into tokens. This paper overlooks the design of the latter stage. It still relies on a patch-wise encoder to generate 2D feature maps, and then pool the features corresponding to each segment. This design could fail to fulfill the monosemanticity of sub-object tokens.\n- The extrinsic evaluation benchmarks used in this paper are not common in VLM evaluation. The authors are encouraged to conduct evaluation under standard settings, i.e., the LLaVA framework.", "other_comments_or_suggestions": "I suggest that the authors move the introduction of 'Token Embedding for Adaptive Segmentation' (Section 5.1) to the 'Method' section (Section 3), as deriving sub-object token embeddings should be an indivisible part of the proposed method.", "questions_for_authors": "How does the throughput of the sub-object tokenizer compare to that of the patch-wise tokenizer (e.g., CLIP or DINOv2) for a similar model size, including the phase of deriving token embeddings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel adaptive image tokenization strategy inspired by subword tokenization in NLP to align with visual morphology, improve token quality and efficiency, and enable vision-language models to converge faster and generalize better using fewer tokens.

It has received 4 reviews, with 2x accepts, 1x weak accept, and 1x weak reject.  Authors have provided detailed rebuttals which are successful at raising reviewers' final ratings.  Reviewers find the paper well written, conceptually innovative, and delivered practical efficiency.  

The negative reviewer had a lingering concern on the performance on VQA, which authors did provide additional results in a further response (after the reviewer's rebuttal acknowledgement).   

There is also an interesting discussion on the separation of token segmentation and token embedding that the authors should include in the final version.   The following work is highly relevant -- there are several other recent related works in computer vision that deal with end-to-end adaptive segment tokens such as HGFormer, SSN, SimSeg etc: 

Learning Hierarchical Image Segmentation For Recognition and By Recognition, Tsung-Wei Ke, Sangwoo Mo, Stella X. Yu, ICLR 2024
(Earlier version: CAST: Concurrent Recognition and Segmentation with Adaptive Segment Tokens, ArXiv 2210.00314, 2022)

It would be helpful to add results on the more challenging Cityscapes dataset.
Decision: Accept (poster)