ICML 2025

Subobject-level Image Tokenization
Delong Chen 1 2 Samuel Cahyawijaya 2 Jianfeng Liu 3 Baoyuan Wang 4 Pascale Fung 1 2
Abstract
Patch-based image tokenization ignores the mor-
phology of the visual world, limiting effective
and efficient learning of image understanding. In-
spired by subword tokenization, we introduce sub-
object-level adaptive token segmentation and ex-
plore several approaches, including superpixel,
SAM, and a proposed Efficient and PanOptiC
(EPOC) image tokenizer. Our EPOC combines
boundary detection–a simple task that can be han-
dled well by a compact model–with watershed
segmentation, which inherently guarantees no pix-
els are left unsegmented. Intrinsic evaluations
across 5 datasets demonstrate that EPOC’s seg-
mentation aligns well with human annotations
of both object- and part-level visual morphol-
ogy, producing more monosemantic tokens and
offering substantial efficiency advantages. For
extrinsic evaluation, we designed a token embed-
ding that handles arbitrary-shaped tokens, and
trained VLMs with different tokenizers on 4
datasets of object recognition and detailed cap-
tioning.
The results reveal that subobject to-
kenization enables faster convergence and bet-
ter generalization while using fewer visual to-
kens.
Project website:
https://github.
com/ChenDelong1999/subobjects.
1. Introduction
Partitioning the raw observational data stream into a man-
ageable number of segments is often beneficial for learning,
particularly when nearby elements in the data stream exhibit
strong correlations and therefore high redundancy (Cover,
1999; Barlow et al., 1961; Biederman, 1987; Marr, 2010;
Del´etang et al., 2024). In Computer Vision (CV), nearby
pixels are often highly correlated (He et al., 2022; Simon-
1Meta FAIR Paris
2The Hong Kong University of Sci-
ence and Technology 3Alibaba Group 4Zillow. Correspondence
to: Pascale Fung <pascale@ece.ust.hk>, Delong Chen <de-
long.chen@connect.ust.hk>.
Proceedings of the 42 nd International Conferenc...