Differentiable Hierarchical Visual Tokenization
Authors: Marius Aasan
Martine Hjelkrem-Tan
Nico Catalano
Changkyu Choi
Adín Ramírez Rivera
Venue: NeurIPS 2025 spotlight ()
Keywords: ViT
Partitioning
Tokenization
Segmentation
Vectorization
Adaptive
Hierarchical
Model Selection
Primary area: deep_learning
TL;DR: An end-to-end learnable tokenizer for Vision Transformers that enhances spatial and semantic learning by allowing retrofitting of pretrained models to use pixel-level tokens

Abstract: Vision Transformers rely on fixed patch tokens that ignore the spatial and semantic structure of images. In this work, we introduce an end-to-end differentiable tokenizer that adapts to image content with pixel-level granularity while remaining backward-compatible with existing architectures for retrofitting pretrained models. Our method uses hierarchical model selection with information criteria to provide competitive performance in both image-level classification and dense-prediction tasks, and even supports out-of-the-box raster-to-vector conversion.

Differentiable Hierarchical Visual Tokenization
Marius Aasan',
Martine Hjelkrem-Tan'
Nico Catalano",
Changkyu Choi Adín Ramírez Rivera'
University of Oslo
Polytechnic University of Milan
UiT The Arctic University of Norway
Depiartment of Informatics
Artificial Intelligence and Robotics Lab
Department of Physics and Technology
marluase www.aaii........ matanduio no, nico. eatalano@polini changkyu. chiiuuit.zo. adinrQuio.no
Abstract
Vision Transformers rely on fixed patch tokens that ignore the spatial and semantic structure of images. In this work, we introduce an end-to-end differentiable tok- enizer that adapts to image content with pixel-level granularity while remaining backward-on wokkwaddcoopatblle with existing architectures for retrofitting pretrained models. Our method uses hierarchical model selection with information criteria to provide competitive performance in both image-level classification and dense- cense-prediction tasks, and even supports out-of-the-bol raster-to-vector conversion.
Introduction
Transformers [I] have become the de facto architecture for all but a few data modalities [2-8]. The architecture is, however, contingent on the process of tokenization. Tokenizers for natural language [9, 10] are designed to compress text into morphemes and semantic subwords-mining units aligned with meaning. Yet in vision, tokenization is comprised of partitioning images into uniform square patches, ignoring semantic content and object boundaries in favor of computational convenience. This highlights key incongruity; text tokenizers align with semantic units while patch-based vision tokenizers fragment objects without regard for their structure. Figure illustrates how patch tokens lack the semantic coherence and granularity necessary for dense predictions.
Efforts to move beyond the grid paradigm include clustering or merging [11, 12] for grouping features dynamically, deformable patches [13, 14] to improve spatial adaptivity. Recent work propose subobject tokenizers [15-17] to partition images into semantically coherent regions, providing gains in classification, segmentation accuracy, and interpretability. However, each method tackles only one or two facets of the larger tokenization problem, and none are fully end-to-end learnable.
An effective visual tokenizer must unify precise semantic alignment, diffcrentiability. and adaptive gramularity. Our key insight is that hierarchical pixel-level partitioning can be formulated multi- scale model selection prohlem, and can be combined with differentiable mechanisms for end-to-end learning. We propose differentiable hierarchical tokenization (7HT) for Vision Transformers (ViTs), emphasizing modularity [18, 19] for reuse and backward compatibility.
Original
vT (2)
Quadiree [20]
SLIC [16]
2HT (Ours)
Figure 1: Comparing spatial granularity in visual tokenizers. 0HT (right) provides end-to-end learnalee framework for multi-scale tokenization. We provide examples in Figure E.6.
Code and model weights: https: https://github... wwt../ooood.iiiiiiieeeeeee.....e 39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Festure Exrrator
Tokenieer
Input
Peatures ((x)
Hierarchy N
Mesn Iejection
Information Criterion
Subtraction
Roaduul
Features S.
F(5)
Gradiene
Feature
M
- No Godieu
Extractor
Seperposels
Figure Illustration of 0HT tokenization and feature extraction pipeline. From an input image we produce a hierarchy of superpixel representations. An optimal segmentation is then selected from the hierarchy using information criteria. and features are extracted for each superpixel. Features can then be used in any VIT backbone. (Right) We also depict the feature extraction process of superpixel S where its features mixed based on foreground, M background, M and shared background features, Details in Section 2.4.
Our contributions can be summarized as follows:
novel end-to-end differentiable tokenization method which adapts training data, and ensures effective tokenization for classification and segmentation tasks.
Learnable tokens:
Retrofitted tokenizers: flexible fine-tuning strategy to adapt pretrained ViTs for superpixel tokens with pixel-level granularity, enhancing versatility across tasks.
Multiscale model selection: A lattice theoretic extension of information criteria to multilevel hierarchical partitioning for selecting the most informative image partitions.
Image vectorization: out-of-the-bon method for adapting hierarchical superpixel tokenizers to raster-to-vector graphics conversion, without being specifically trained for this task.
1.1 Motivation
Visual tokenization means discovering a discrete set of regions from a continuous, high-dimensional image, under strict compute and memory budgets-effecttine and representation learning all at once. Unlike 1D sequences, where you have natural ordering and can pick breakpoints by cues such as whitespace and morpheme statistics, spatial data offer no canonical order. The space of possible region shapes, sizes and connectivity explodes, and tokens must respect multi-scale structure and spatial invariances. Moreover, full integration with ViTs require differentiable tokenization aakenization-adaativvely choosing token count, placement, and shape to adhere semantic boundaries to extract meaningful atomistic units from scene with end-to-end learning.
solving segmentation, compression
1.2 Related Work
Existing approaches adaptive visual tokenization be organized into three broad categories.
Dynamic Grouping and Merging: Several works [11, 20-22] propose to dynamically cluster or merge grid tokens within the transformer's layers. These approaches reduce redundancy and adapt token budgets per image, but remain tethered to the original grid primitives and do not introduce truly off- off-grid partitions. This prevents discovery of true subobject boundaries for semantic coherence.
Deformable Sampling: A different approach stems from replacing fixed patches with learned sampling with deformation and dynamic positions. DPT [13] and Deformable Attention Transformer (DAT) [14] learn per-token centers and scales via differentiable bilinear sampling, granting geometric flexibility. While fully end-to-end learnable, these are still fundamentally patch based, leaving semantic alignment and object boundaries implicit.
Subobject and Superpixel Tokenizers:
Superpixel-based methods partition images into seman- tically coherent regions prior to encoding. SuiT [16] pools CNN features over SLIC superpixels; EPOC [17] combines learned boundary cues with watershed grouping to form panoptic tokens; SP-

---

former [23] uses hybrid rross-attention pixel-to-toker clustering between attention blocks; SPiT [15] performs parallel hierarchical graph merging.
While SuiT works on a downsampled version the input image, other methods provide true pixel-level granularity for extracting subobject tokens. These yield clear gains in classification, segmentation, and interpretability, but the non-differentiable grouping step prevents the tokenizer itself from being trained end-to-end.
2 Method: Differentiable Hierarchical Tokenization
Our proposed method builds on previous subobject level approaches to provide the first fully end- to-end learnable tokenizer for ViTs. We emphasize that 0HT is not another ViT variant, but fully modular tokenizer that can serve as a aang-and-play extension for pretrained models. Our design emphasizes precise semantic alignment with pixel-level granularity, multi-scale awareness via hierarchical pruning with information criteria, end-to-end differentiability, and modularity.
High Level Overview:
0HT produces adaptive tokens through four stages, illustrated in Figure
Feature projection (Section 2.1): We embed each pixel into learned d-dimensional feature space using lightweight CNN, establishing a similarity metric for subsequent grouping.
Hierarchical partitioning (Section 2.2): Starting from individual pixels, we iteratively merge similar adjacent regions construct complete hierarchy, capturing structure multiple scales.
Optimal selection (Section 2.3): Information criteria identify the partition that best balances model fit against complexity, eliminating manual threshold tuning.
Differentiable extraction (Section 2.4): A mean- -ajennnjettion mechanism produces token features compatible with standard ViTs while enabling end-to-end gradient flow.
Consider graph G (V, E) where the vertices represent pixel positions in grid of width w and height h, and edges E are connections of horizontally and vertically adjacent vertices. We view image signal function V R mapping each vertex E V e-channel pixel feature. A connected partition of V is a set π = (S C V} which satisfies:
Preliminaries:
ii Non-overlapping: For any pair S, E their intersection is empty, i.e., S S = =
( Covering: The union all E is the full vertex set, i.c., USEr S
(ii) Connected: For any vertices
E S, there exists path of adjacent vertices in S starting and ending at v, where each pair of consecutive vertices is connected by an edge in E.
Let IIVV) be the space of all partitions of V and consider two partitions T1, 42 E II().. If for every region S₁ E ni there exists a region S E T2 such that S₁ S S2, we say that the two partitions form a hierarchy H. (T1, π2) ordered by refinement.
2.1 Subobject Feature Projection
We use a lightweight CNN encoder f: R -> Rdd to embed each pixel to an initial feature space fo(u) (x(()). We design f with residual branch that applies x convolution lift the c-channel input to dimensions, and a main branch that downsamples via two successive convolutions (3 x or 2 strided) followed by bilinear upsampling.
2.2 Hierarchical Vertex Merging
We expand on existing methods [15] to construct hierarchical partitions by iterative vertex merging. On a high level, the procedure pairs vertices with their most similar neighbor and merges them via parallel connected components, ensuring that each region is connected superpixel.
More formally, given a positive semi-definite kernel 6: x V + R22, we define the maximally similar vertex Umax arg max k(u, v) such that
Emax = {(v, Umax) U V3 E,
pairs each vertex with its most similar neighbor within the neighboring vertices M(v) defined by E.

---

Let C1: V + V1 denote mapping of vertices to the connected components of the spanning subgraph' G[Emax] Then, (v) denotes connected superpixel region S Ví such that vertices within the same component of G [maax] are assigned to S. In particular, S contains v, Umax, and potentially other vertices based on similarity. natural choice for updating the vertex features for Vi is to simply take the average feature values in each component
1 Σ fo(u), - S
[v())
For given level t, the mapping C induces graph contraction G3-1 G.. Each vertex Vi-1 is assigned to a connected component C,()) S. E Tt, which yields a hierarchy of partitions H = (. WT). Importantly, each superpixel St at any level in the hierarchy is always grounded in the original pixel positions in V. We formalize the construction in Appendix A.2.
2.3 Hierarchical Pruning via Information Criteria
Having constructed full hierarchy, we now look to select optimal partition. The construction of H is designed to tackle two objectives; multi-scale adaptability and redundancy management. We compute the hierarchy up to singleton such that all regions
connected up root region representing the full image. This allows us to frame the search for an optimal partition тı E [I(V) as model selection problem using information criteria (IC) [24], removing the need for threshold calibration or gating [20, 25, 26]. Recall that an IC has the generalized form
IC(#) 2 log L(A) g(dfo),
where I(0) is the likelihood of the data under the parameter 0, and g(dfo) is penalty function based on the statistical degrees of freedom dfg that discourages overly complex models. A choice of IC typically determines the form of g. Our goal is then to derive an estimate for IC(m.).
Likelihood: A superpixel representation E S E тu can be considered a piecewise constant model of the image for which each region S is assigned a constant value μ.s. We model each pixel feature fo as i.i.d. samples from Gaussian distribution, i.e., fo S ~ NNus, Es). Under this assumption, the ligllikelhoodd log [(T.) simplifies to terms only involving Es. This aligns with variance reduction criteria [25], but augmented with additional parsimony constraints. See Appendix B.1 for details.
Degrees of freedom:
Traditionally, degrees of freedom dfg correspond the number of parameters estimated in a model. For each superpixel S, Ms is parameter in piecewise constant model of image x, making df, proportional to the total number of regions in the optimal partition т. However, this cannot be determined exactly without search of all possible combinations in H. Instead, we leverage the lattice structure [27] of partition space II(V). and find that dfg are inversely proportional to the total number of connected edges within each S. This gives proportional estimate
df =
Σ dfs x Σ Volc(5))
SER-
SER.
where Volg (S) represents the total number of edges (u,v) E E such that u, u E S. This formulation effectively penalizes partitions that consist of smaller regions (with fewer internal edges), encouraging larger, more informative regions while still capturing essential structural information. We provide a formal derivation of this result in Appendix B.2.
In turn, 0HT prunes a full hierarchy H according to the selected IC, balancing model fit and complexity to select optimal partitions. We explore the effects of different information criteria on method's performance in Section 3.3.
2.4 Differentiable Token Embeddings
There are two main approaches to feature extraction with superpixel tokenization. The most common approach to perform aggregation of regions through separate encoder [16, 17]. However, this prevents drop-in replacement in ViT model, making retrofitting tokenizers to pre-trained models non-trivial. In contrast, SPIT [15] generalizes the ViT feature extraction process to irregular regions. Each region's 's bounding box is interpolated to a fixed patch size while masking out the background
'See Definition A.2 for details.

---

features of the surrounding vertices. This is backward compatible with standard ViTs, but suffers from being inherently non-differentable. öHT makes generalized ViT features fully differentiable by introducing weighted aggregation and novel mean-injection trick reminiscent of straight-through estimation [28, 29].
At each level t, we collect learning signals from the contraction process by computing features via
|Sa
fu+1(1)= = |S RES.
Σ
k(u, Umaxx) fe(u):
where |S,] and [St+1] denote the number of pixels in the respective superpixel regions at two consecutive levels in the hierarchy. This update is a weighted variant of (2) with each vertex's contribution weighted by the kernel score of their most similar neighbor at step t.
under some optimal partition ~.. To extract features from interpolated regions of the original image the idea is to inject the pruned vertex features into the original image channels without altering its local texture properties. For each pixel E S E T. we adjust the original pixel feature z(v) by computing
Additionally, let (v) denote the feature of vertex
i(v)= = sx(r) + W.-(v) F. (v).
where W: Rd -> Ie is a learnable linear mapping. This effectively replaces the mean of each superpixel region F. (v) [S] Eues =u) with learnable estimate from the tokenization process, enabling full gradient flow from tokenization the VIT backbone.
Positional Embedding: Previous work has shown that positional embedding computed as linear combination of kernelized joint-histogram positions for each region generalize standard learnable positional embeddings [15]. We extend previous work by allowing for higher resolution, which provides more fine grained detail to token embeddings, and ablate the effect in Table 8.
Modularity and Retrofitting: Modularity central to developing complex systems [18, 30), and allows architectures to be broken down into reusable components. Just as transfer learning lets you fine-tune pretrained task heads, modular tokenizer enables you to transfer pretrained models across distinct tokenization schemes without retraining from scratch. To apply this principle to a pre-trained patch-based ViT, we initialize 0HT as follows. From (6), we observe that having (f( = F. (v) results in å perfect reconstruction of the original image. We find that pretraining the encoder f and the linear mapping W jointly with
Lree
Σ [>(0) WA VEV
IV
provides maximally aligned features for fine-tuning a VIT backbone with öHT in place of the canonical tokenizer, allowing for fully differentiable tokenizer retrofitting.
Background Masking: During our experiments on retrofitting, we observe that masking out background features leads sparsity when highly irregular regions are interpolated fixed size. Intuitively, masking leads to a minor domain shift in token embeddings, since the original features are dense within patch and masking naturally leads to sparser representations. This can result in slower convergence and loss in performance.
In response, introduce dynamic adaption of background masks during training. Let denote the feature patch size, and let E [0, 1), 3 E Rexxxx be learnable parameters in a feature extractor. Let M E {0, 1|999 be the interpolated foreground mask of a superpixel S, with M5 = 1 M denoting the background mask. We extract token features F(S) Rexxxq via
F(S) = (M² + AM5) 3 &(S) + (1 2) 3
(8)
where @ is element- wise product. This tweak allows the model to blend the foreground and back- ground elements within each region through X. The parameter 3 serves shared background feature, which mitigates sparsity by preventing zeroing out background regions completely. An illustration is provided in the right section of Fig. 2.
We ablate the effect of masking, encoder f, choice of kernel and information criteria IC, as well as other hyperparameters and architectural specifics of 0HT in Tables 6 and 7.
5

---

Table 1: Classification top-1 and KNN accuracies (221 x 221). The top six rows (above the midline) show results for the baseline models and retrofitted (RF) ØHT counterparts. The bottom four rows show results for models trained from scratch. Models are trained exclusively on ImageNetlk (IN).
IN- [N-V
IN Roal. [32]
IN-22 [33]
Cahtcc (34]
CUB CUB'LS
Cass' [36]
Tiken
Pretraining
Model
Acc
KNN
Acc
KNN
A1c
ENN
Acc
KNN
Acce1 I
Acc81
Parch
DETS [37]
VIT-16
80.4
50.7
86.1
86.9
69.7
58.3
87.0
77.3
54.6
36.4
24.9
6.5
DRITS
VIT-R16
53.1
87.7
88.2
73.6
76.9
OIIT
DEITS-FF
VIT-S16
80.1
78.4
84.9
83.7
68.2
58.1
88.
77.4
8.3
DEIT3 RP
V16
83.9
$2.9
$8.3
87.7
73.2
63.0
91.2
79.1
74.4
$2.5
34.7
9.2
TIMM-R RF
VIT B32
83.1
$1.6
87.3
87.4
72.2
62.4
90.2
78.3
72.8
50.1
39.8
Parch
From Seroich
VIT-S16
79.9
77.5
83.9
84.9
57.6
86,
76.4
57.2
49.7
21.4
6.3
VIT-B16
81.9
80.4
86.7
87.3
71.1
62.1
88.4
78.0
73.3
52.0
31.3
8.4
OIT
S16
80.0
77.6
84.1
85.9
89.2
57.2
87.5
76.3
09.6
53.8
23.2
1.9
V1T-B16
83.2
80.3
87.4
88.2
72.8
62.1
89.3
78.1
73.2
51.6
33.0
9.3
Trosen backbone and lineur probing.
Note adaptive tokeniration resalts in higher numbers tokens compared baseline.
3 Experiments and Results
We design our experiments to investigate the representation capabilities of our method's extracted tokens in multiple settings; including end-to-end learning with classification on ImageNetIk [31], transfer learning as drop-in tokenizer replacement for pretrained ViTs (cf. Section 3.1). and demon- strating decoder-free segmentation models with learnable tokenization (cf. Section 3.2). Moreover, öHT can also be evaluated on learnable image vectorization, and we compare our method to learnable image vectorization models (cf. Section 3.2). Training setup is detailed in Appendix D.
3.1 Image Level Predictions
We focus transformer baselines trained exclusively on ImageNetlk [31], and validate various downstream tasks [32-36]. In addition to reporting top-1 accuracy scores, we perform a KNN evaluation assess the quality of the representation space. kNN scores computed by taking the max score over & E {10, 20, 50, 100, 150, 200} [39].
Retrofitting: We evaluate the effect of retrofitting öHT to pretrained models [37]. including the less common B32 capacity [38] for completeness. We align the ØHT tokenizer by pretraining using (7). Then, finetune pretrained models to replace the canonical tokenizer in the ViT that will be retrofitted with üHT tokenizer. All results are uniformly re-evaluated for fair comparison, explaining minor differences from previously reported results [37].
Table (top) shows that tokenizer retrofitting has a generally positive effect on linear evaluation models with base capacity, while maintaining competitive performance for small capacity models. Interestingly. our kNN results for ImageNetlk indicate that DEiT3 [37] models generally perform better with kNN than the linear evaluation, which is surprising one typically expects the opposite. Contrarily, our retrofitted counterparts yield an expected result; the linear head produces better results than kNN. On the other hand, our retrofitted models are better aligned with both linear probing and kNN on Caltech256 [34] and CUB200 [35], indicating that 0HT provides token representations that useful for generalizing beyond the training data.
In general, we observe that retrofitting models with 0HT enables models to adapt tokens individual images, maintaining or slightly improving overall classification performance compared to baselines.
We compare transform- ers using 0HT to standard ViT patch tokeniza- tion [2] by training models from scratch under the same training regime, such that training is guaranteed to be equivalent. This eliminates confounding factors such as hardware and mi- nor implementation differences. Contrary the retrofitted setup, models trained from scratch use local gradient features, which have been shown to improve performance for both canonical and superpixel tokenization [15] with minimal com- putational overhead (+0.07 GFLOPs, +256 pa-
Training from Scratch:
Table 2: Comparison ViT-S16 models trained from
scratch on ImageNet with different tokenizers. Mod. denotes commensurability with ViTs, while Diff. de- notes end-to-end differentiability.
Acc
Method
Mod.
DifT.
Small
Base
Patch/ [40]
79.9
Patch / DEIT3 [37]
80.4
82.6
SPForme [12)
81.7
82.7
SPYT [15]
75.0
80.4
SuiT [16]
80.9
3HT
80.0
83,2

---

Table 3: Single Scale Semantic Segmentation mloU
results on ADE20k [41] and COCO-Stuff1645 [42].
Doraset
Bocibone
Meethod
(1)
ADEIR
RN30 [43]
UperNa [44]
640
40.1
Swas-B [45]
Mask2Furmer [66]
640
$2.4
MIT-B5 [23]
Seaforee [23)
640
51.0
SeaVi B [47]
ATM
512
51.3
BiFrrma-S [68]
UperNan [44]
640
49.8
BiFiema-BB [48]
UguNNa [44]
643
51.0
CasoNest-l
SP-Trandomer [40]
640
43.7
SPFumees-S S [12]
SPFtemer
640
46.5
MAF-R (80)
UperNin (44)
640
47.1
MAF-H [50]
Lincse
640
33.3
DINOw2-S [S]]
Linese
643
44.3
DINOv2-B [S1]
Lises
643
47.3
Rag
DINOv2-B [5i]
Lisez
640
47.3
Beick
Wood
Window
PNT-VVI-
MIP
512
47.1
SHT-VIT-B
MILP
512
53.2
Figure 3: 0HT tokenized images with feature corre- spondences (top) and semantic segmentation (bottom).
COCO 64k
MIE-BS
Segformer (23)
640
46.7
MIT-B5
Lawin [53]
640
47.5
Uperhie: RK
580
68.2
WIT-L
Segmenter
48.4
2HT-VVT-B
MIP
512
48.9
Table 4: Zero-shot segmentation results on salient de- tection datascts using token-cut 0HT outperforms isting approaches, including other adaptive tokenization
frameworks.
ECSSD [S5]
DUTS [X]]
DUT-OMRON (57)
Backhoue
IoU
1
Fasx
Toll
Acce1
Fresx
[oU
Acc
Figure 4: Comparison of image vectorization with
DENO-I [SS]
R1.5
71.2
91.8
60.2
57.6
00.3
60.0
$1.3
$8.0
Dif'VG [59] (left) and öHT (right).
DENO-B [S9]
$7.4
17.2
95.4
75.5
62.4
91.4
69.7
61.8
59.7
SPIT-B [1:5]
90.3
77.3
93.4
77.1
63.9
89.4
71.1
55.4
$6.8
SuIT-B [16]
K2.0
MU.
95.8
6810
00:0
03-4
67.2
GHT-B
92.4
79.9
77.
64.4
$9.8
post processing Hioecal spearpiing..
Table 5: Results on raster-to-vector conversion. We compare with off-the-shelf baseline (Adobe) and differentiable model specifically trained for the task.
DaTVG-5
Method
Zero-shot MSB(1)
PSNR(1)
SSIM(1)
0.00712
21.84
0,7318
DCEVG-MC
0.00316
25.54
082877
Figure 5: Image vectorization from ÖHT token extrac- tion with zooms to show the finer details, comparing the original image (left) and vectorized image (right).
DHIVG-AR AP OLT
0.00265
26.22
0.319:
0,00178
27.50
0.8441
rameters). Our results show that superpixel tokenization generally provides stronger classification results than canonical tokenization with patches, cf. Table 1 (bottom) in a strict apples-to-apples comparison. We also perform a comparison with existing superpixel-based tokenization approaches in Table 2. Our results show that öHT for ViT-B outperforms other methods while preserving modular compatibility with ViT architectures.
3.2 Dense Predictions
We assess our method on dense tasks by evaluating the fully trained üHT models for semantic segmentation, zero-shot segmentation performance on selected benchmarks, well as the non- standard task of image autovectorization-ccooeee converting rasterized images to vector images.
Semantic Segmentation: We fine tune our fully trained öHT models on semantic segmentation baselines [41, 42], without a dense decoder for upscaling. Instead, each superpixel is individually classified as single segment using simple MLP head for each token. The results in Table 3 show that 0HT provides raw token embeddings that are well suited for image segmentation. A qualitative analysis with examples can be found in Appendix E, with extended results in Appendix C.
Zero-Shot Segmentation: Table 4 shows results for fully trained 0HT models using the Token- Cut [58] method over three selected salient segmentation tasks [55-57]. The results demonstrate that äHT provides strong results in zero-shot salient segmentation, and shows that tokens can be used out-of-the-box with no post- post-processing or specialized training required.
7

---

Table 6: Tokenizer hyperparamelers evaluated reconstruction based pretraining on ImageNet.
Dimension d'
Kernel function x
Information criterion
Metric
3
8
16
24
32
Cosine
Tanimoto
Gansslan
AIC
AICC
BIC
CIC
MSE (1)
0.09
0.08
0.06
0.07
0.06
0.06
0.10
0.08 0.55
0.06 0.60
0.08
0.06
0.07
0.11
SSIM [60) (t)
0.56
0.60
0.58
0.57
0.54
0.52
0.60
0.57
0.49
Table 7:
Architectural ablations for 0HT components. Evaluated with VIT-S16 over ImageNet.
Table 8: Ablation on sizes for kernel positional embeddings, Acc@1 I on ImageNet and mloU on ADE20K.
Acc@1
Ablation
DEST3 RF
From Seratch
Capacity
Resolation / GFL.OPs
Lin@1 I
mlotl
8HT haseline
80.1
80.0
S16
16x 16x06/
78.4
36.4
CNN
76.9 (.2)
75.5
(1.4.)
24.24/00.15
80.0
43.1
blending
71.8 (1 8.3)
L.))
32x 0026
80.0
47.2
3-musk features
79.7 (1.0.)
79.1
14.0.9
48 / 10.59
80.1
47.3
kernel aggr.
80.0 (1.0.))
(1.0.))
B16
16x 16 0,07
81.3
40.6
pruning'
80.1 (== 0.01
(1.0
24 0.15
83.3
46.9
Kemoving model selection generally increases
32 0.26
50.4
tokeas image and computational cost.
48 / 00.59
83.5
53.5
Autovectorization and Image Tracing: öHT provides out-of-the-box superpixel partitions that are able to represent images with very high levels of detail. By converting these regions into vectorized representations, our pretrained models can serve as a high quality raster-to-vecte graphics pipeline. We extract the optimal superpixel partition, and convert each region into vectorized path using potrace [61]. Since previous work on learnable image vectorization [59, 62] provide few quantitative baselines, we compare our method to the five examples provided by Li al. [59] in Table 5, This shows that öHT yields high-quality raster-to-vector conversion, illustrated in Figures 4, 5 and E.3.
3.3 Ablations and Hyperparameters
In our ablative study, we by measuring the effect of architectural mechanisms in 0HT-S for both training paradigms. For ablating f as a CNN, we note that f is the foundational source of gradients from the image, so it cannot simply be dropped. We therefore replace the CNN with linear projection (i.e., 1 convolution) over the input channels.
Table 7 shows that each component contributes to a drop in accuracy compared to the baseline, with the exception of model selection. This can be attributed to the fact that model selection represents a constraint on deduplication. When such constraints are removed, the number of tokens increases during training, and the model ends up using much more tokens without increase in performance.
Tokenizer Hyperparameters: We ablate the effect of tokenizer hyperparameters by evaluating image reconstruction quality after tokenization, using mean squared error (MSE) and structural similarity index measure (SSIM) [60]. We focus on the reconstruction of the tokens (which only requires training the tokenizer) instead of their predictive properties (which requires training the full ViT) due to our computational limitations. The ablations compare the cosine, Tanimoto [63], and Gaussian kemels,, well as the Akaike, corrected Akaike, Bayesian, and correlation information criteria (AIC, AICC, BIC, and CIC, respectively) [24].
The results in Table 6 show that Gaussian kernel with d = 8 and AICC produces the best scores. Moreover, we evaluate the effect of different resolutions for the positional embeddings in Table 8. These results indicate that increasing the resolution of positional embeddings generally improve results. However, the effect saturates slightly at 24 x 24 for classification, and at 48 x 48 for segmentation. Given that increasing the resolution adds to the computational complexity (GFLOPs), we use these resolutions for our final models.
Scale Invariance: A central feature of 0HT is that the model can select subset of tokens that is most informative to represent each individual image. As result, the number of tokens differs from image to image, adapting to variations in information and scale. To evaluate the effect of this behavior, we perform an experiment where we compare results over various image scales while keeping the number of tokens equivalent between models. We do this by performing an extra step of merging after the model selection step, merging regions with high similarity to produce similar numbers of tokens for each model. This test is performed comparing the retrofitted models to the basclines.
8

---

80
109
Prich-S16
Parch 8316
# Tokems
Acce
Parch 8332
103
O9TT-S16
ØHT-B16
ØHT-B32
64x
96x
128
234 224
250 266
384 194
512
768 76
64x
96x
128 128
234
256 208
384 334
512
768 76
128
Resolation
Resolation
Figure Scale invariance and token granularity retrofitted models over ImageNet [31] with extended low-
resolution evaluations (64-768px). The additional points highlight how both patch and adaptive tokenizations degrade gracefully with coarser sampling. showing stronger invariance for OHT.
Our results in Figure 6 show that öHT scales considerably better to higher resolutions by adapting tokens to image content, notably without modification of the resolution of positional embeddings.
In very low resolution settings (e.g. CIFAR's 32 x 32), canonical ViT tokenization may outperform ØHT. In these settings, a few of pixels can end up covering an entire region and öHT may yield block-like regions similar to square-patch tokenizer, as there are no clear inter-pixel edges to delineate. The uniform grid patches aligns well with the reduced information content, allowing the standard tokenizer to perform adequately. 0HT's adaptive token selection offers less ad advantage in this scenario there is less fine-gres grained information to exploit. However, we note that at moderately low resolutions, 0HT generally performs quite well out-of-the terttthebbor box compared to patch- based ViTs.
At higher resolutions, images contain more detailed and fine-graines features, and 0HT's ability select and adapt tokens based on the most informative regions becomes highly beneficial. It can effectively capture critical details without being constrained by fixed grid, leading improved performance over the canonical tokenizer. öHT achieves this advantage without modifying the positional embeddings, demonstrating its inherent scalability to higher resolutions.
Discussion and Conclusion
We propose 0HT, differentiable tokenizer with pixel-level granularity that uses information criteria to dynamically select optimal partitions from hierarchical representations. Our experiments demon- strate that öHT achieves strong performance both image-level classification and dense prediction tasks while maintaining modularity when retrofitting pretrained models.
Our work establishes tokenization as an adaptive, learnable component in VIT architectures. As models and datasets scale, this modularity becomes increasingly valuable for adapting representations to specific tasks and domains. Given the broad applicability of superpixels in vision modeling [64-67], integrating adaptive tokenization could unlock performance gains across various applications, from medical imaging to video understanding where redundancy management is eritical.
4.1 Limitations
While 0HT shows promise, it should still be taken early exploration of fully adaptive tok- enization. Our hierarchical pruning with informa- tion criteria effectively manages redundancy but relies on modeling assumptions detailed in Ap- pendix B. The computational overhead of su- perpixel tokenization currently results in lower
Table 9: Computational efficiency of aHT with 24 x 24 pos. embeddings. Throughput measured ImageNet over 8x MI250x.
Cap.
Res.
Tokenizer
Params
Tok./s
Tok./im.
Patch
22.1M
192.3k
197
S16
224
0HT
22.3M
94.2k
240
B16
224
Patch
86.6M
89.7k
197
throughput than canonical ViTs, though this gap narrows at higher resolutions where adaptive to- kenization provides greater benefits. At very low resolutions, reduced semantic information limits adaptive partitioning benefits, though the method adapts well to practical image sizes.
B16
224
aHT
86.9M
68.2k
246
S16
384
Patch
22.1M
112.4k
577
S16
384
9HT
22.3M
58.1k
494
B16
384
Patch
86.6M
61.6k
577
B16
384
0HT
86.9M
48.9k
432

---

Our feature extraction maintains backward compatibility with standard ViTs, but may not be optimally designed for irregular superpixel regions. Developing specialized feature extraction methods while preserving differentiability represents an important research direction.
4.2 Further Work
As mentioned in the limitations, architectural design choices for edge contraction and feature aggre- gation are promising avenues for optimal design of adaptive tokenization in vision transformers. In this work, we gather implicit learning signals weighted by the similarity kernel (5), but extending this to explicit learnable aggregation with graph neural networks could provide more expressive modeling. However, such extensions should preserve symmetry and positive semi-definiteness to ensure consistency and well defined edge contractions in the hierarchy (Appendix A.2).
Extending differentiable tokenization to self-supervised learning represents natural extension. This was previously explored by Lew et al. [16], which trains DINO variant with strong results. In self-supervised settings, masked image modeling (MIM) paradigms have potential for synergy with differentiable tokenization mechanisms, providing more direct learning signals via invariants such as translation, scale, and rotation.
Vision-language models represent another promising research direction, where alignment with language could help inform the edge contraction process. Adaptive tokenization be beneficial for document- foccusd tasks where fixed patches poorly align with heterogeneous text and layout structures. Preliminary investigations suggest 0HT could address key limitations in current VLM approaches by generating tokens that better capture semantic boundaries on a per-sample basis.
Additionally, video transformers present compelling application domain. The quadratic attention complexity makes redundancy management crucial, and spatiotemporal superpixel tokenization could significantly improve efficiency while preserving semantic coherence across frames. More broadly, our findings suggest that for tasks where high dimensionality is bottleneck, better adaptive tokenization such as öHT can provide tractable dimensionality reduction by exploiting inherent local redundancies. Such redundancies cannot be meaningfully exploited tokens are invariant to image content, such in the case of square patches.
Acknowledgments
This work was funded by RCN (the Research Council of Norway) through Visual Intelligence, Centre for Research-based Innovation (309439), and in part by the RCN-NRF (National Research Foundation of Korea) joint project AURoRA (359216, RS-2025-03522980) We acknowledge Sigma2 (Project NN8 (N8104K) for access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through Sigma2, Norway. We acknowledge ISCRA for access to the LEONARDO supercomputer, owned by the EuroHPC Joint Undertaking hosted by CINECA, Italy.
References
[1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in Adv. Neural Inf. Process. Sys. (NeurIPS), I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and Garnett, Eds., 2017, pp. 5998-6008. [Online]. Available: https:/proceddnns..erii...o/ httperrripsccccappr/////a/aaaa 315ee243547dee9) wtw.ddeeeeIIIdddo (alod053cc448455a0 Abstract.html
[2] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16 x 16 words: Transformers for image recognition scale," in Inter. Conf. Learn. Represent. (ICLR). OpenReview.net. 2021. [Online]. Available: https://peeneviee.ne/ffouu Pid-ooummmidiiidddile YicbFFaNTT 1, 6, 30
[3] Y. Gong, Y.-A. Chung, and J. Glass, "AST: Audio spectrogram transformer," in Interspeech Conf., 2021, pp. 571-575.
[4] C. Ying. Cai, Luo. Zheng, G. Ke. D. He, Y. Shen, and T.-Y. Liu, "Do transfommers really perform badly for graph representation?" in Adv. Neural Inf. Process. Sys. (NeurIPS), vol. 34. Curran Associates. Inc., 2021, pp. 28 877-27782 -8 888. [Online]. Available: https: /prrccediing nwwrips.cc/papee. httppe/eei..ieeeeee////////eeeea 1ww.///88811002220400000000 1002 Papeer,p
10

---

[5] S. Islam, H. Elmekki, A. Elsebai, J. Bentahar, N. Drawel. G. Rjoub. and W. Pedrycz, "A comprehensive survey on applications of transformers for deep learning tasks," Expert Syst. Appl., vol. 241, p. 122666. 2024.
[6] J. Lee, Y. Lee, Kim, A. Kosiorek, S. Choi, and Y. W. Tch, "Set transformer: A framework for staeetion-based permutation-invariaa neural networks." Inter. Conf. Mach. Learn, (ICML), 2019, pp. 3744-3755 [Online]. Available:
[7] H. Zhao, Jiang, J. Jia, P. Torr, and V. Koltun, "Point transformer." IEEE Inter. Conf. Comput. Vis. (ICCV), 2021, pp. 16 16255-16 268. [Online]. Available:
html/Zhao L_iitt_oiin___eeeeeeeeeee Transformer, _CC_ 20200022111 _02r___ppr.
[8] B. Lim, S. O. Ank, N. Loeff, and T. Pfister, "Temporal fusion transformers for interpretable multi-horizon time series forecasting," International Journal of Forecasting, vol. 37, no. 4, pp. 1748-1766 2021. 1
[9] R. Sennrich, B. Haddow, and A. Birch, "Neural machine translation of rare words with subword units," in Conf. Assoc. Comput. Ling, (ACL). The Association for Computer Linguistics, 2016. [Online]. Available: https://oi..rrg/1.
1655/v1/pp16 1162
[10] M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, F. B. Viégas, M. Wattenberg, G. Corrado, M. Hughes, and J. Dean, "Google's multilingual neural machine translation system: Enabling zero-shot translation," Trans. Assoc. Comput. Linguistics, vol. 5, pp.
2017. [Online]. Available:
https://doi.org/100 1.. 162/tacl_ a 00065
[11] F. Locatello, K. Strokorb, M. Tschannen, D. Weissenborn, J. Rauber, D. Klindt. T. Rainforth, and P. Battaglia, "Object-centric lcarning with slot attention," in Adv. Neural Inf. Process. Sys. (NeurIPS), vol. 33, 2020, pp. 11 594-11 606. [Online]. Available: https:/procceding.neurrp.. hww.errrppsccccppppp////aaaaaaa 851 8511699800aa660cccall bb2aaao0e000aabb6ee0ee 62356c01 3bcOf Abstract.html 1,2
[12] J. Mei, L.C. Chen, A. Yuille, and C. Xie, "SPFormer: Enhancing vision transformer with superpixel representation." " Trans. Mach. Learn. Res., 2025. [Online]. Available: https: /lopeneeieew.n//fooummttc Pid=nu ISj VjygSuy 1, 6,7
[13] Z. Chen, Y. Zhu, C. Zhao, Hu, W. Zeng. Wang, and M. Tang, "DPT: Deformable patch-based transformer for visual recognition," in ACM Inter. Conf. Multimedia (MM), 2021, pp. 2899-288900 2907. [Online]. Available: https://doi.or// 10. 0111334444883475544
1.2
[14] Z. Xia, X. Pan, S. Song, L. E. Li, and G. Huang, "Vision transformer with deformable attention." in IEEE/CVF Inier. Conf. Comput. Vis. Puttern Recog. (CVPR). 2022, pp. 4784 4793. [Online]. Available:
https://doi.org/100 100.1109//// TTT.R26688.0222.000 2022.00475 1,2
[15] M. Aasan, O. Kolbjornsen, A. Schistad Solberg, and A. Ramírez Rivera, "A spitting image: Modular superpixel tokenization in vision transformers." in European Conf. Comput. Vis. Wksps. (ECCVW), 2024.
[Online]. Available: https:///oi.org// 1007/9977 3-078----333-----0 1000-33100----110 1_906-1_11 1, 3, 4, 6, 7,27
[16] J. Lew, S. Jang, J. Lee, S. Yoo, E. Kim, S. Lee, J. Mok, S. Kim, and S. Yoon, "Superpixel tokenization for vision transformers: Preserving semantic integrity visual tokens." " 2024. [Online]. Available: 1, 2, 4, 6, 7, 10, 30
[17] D. Chen, S. Cahyawijaya, J. Liu, B. Wang, and P. Fung, "Subobject-level Image Tokenization," " in Inter. Conf. Mach. Learn. (ICML), vol. 267, 2025, pp. 7719-7738. [Online]. Available: https:/proceediings.s prapsprrrs//22/2hee22222
2,
[18] Y. Bengio, A. Courville, and P. Vincent, "Representation Learning: A Review and New Perspectives," " IEEE Trans. Pattern Anal. Mach. Intell,, vol. 35, no. 8, pp. 1798-1828, 2013. [Online]. Available: http:://do..or//0.
109/TPAMM.2013550 1,5
[19] B. L. Happel and J. M. Murre, "Design and Evolution of Modular Neural Network Architectures,' Neural Netw, vol. 7, no. 6. pp. 985-1004, 1994. [Online]. Available; http://dii.org/10.1000 11166/0883-6000000 80155-8
[20] T. Ronen, O. Levy, and A. Golbert, "Vision Transformers with Mixed-Resolution Tokenization." in IEEE Inter. Conf. Comput. Vis. Wksps. (ICCVW), 2023, pp. 4612-4621. [Online]. Available:
1, 2, 4, 24, 30
[21] L. Yuan, Y. Chen, T. Wang. W. Yu, Y. Shi, Z. Jiang, F. E. H. Tay, J. Feng, and S. Yan, "Tokens-to- Tooken VIT: Training vision transformers from scratch on imagenet," in IEEE Inter. Conf. Comput. Vis. (ICCV). IEEE, 2021, pp. 538-547. [Online]. Available: https://doi.or/110. 11111////C1/V92222222....
11

---

[22] D. Bolya. C.-Y. Fu, X. Dai. P. Zhang. C. Feichtenhofer. and J. Hoffman. "Token Merging: Your VIT But Faster," in Inter. Conf. Learn. Represent. (ICLR), 2023. [Online]. Available:
[23] F. Xie, W. Wang. 7. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, "SeeFormer: Simple and efficient design for semantic segmentation with transformers." in Adv. Neural Inf. Process. Sys. (NeurlPS), Ranzato, A. Beygelzimer, Y. Dauphin. P. Liang, and W. Vaughan, Eds., 2021. [Online]. Available: https://proceedings.. wwwp:/innnnnnnusssee.m www.iip.ccpppppr/200000.. hwt.s///661122266644222222222260
2at.s5000252--strat..ll. Abstract.html 3,
[24] G. Claeskens and N. I.. Hjort, Model selection and model averaging-
Cambridge University Press, 2008.
4, 8, 24, 25
[25] L. Breiman, "Classification and regression trees." The Wadsworth & Brooks/Cole, 1984. 4, 24
[26] J. D. Havtorn, A. Royer, T. Blankevoort, and B. E. Bejnordi, "MSViT: Dynamic mixed-scale tokenization for vision transformers," " in IEEE Inter. Conf. Comput. Vis. Wksps. (ICCVW), 2023, pp. 838-848, [Online]. Available: https://doi.org/// 10. 1199/CCCW60700 20T.. 2023.0009 4
[27] G. Grätzer, General lattice theory. Springer Science & Business Media, 2002. 4,
[28] Y. Bengio, "Estimating or propagating gradients through stochastic neurons," CoRR, vol. 105/1305.
2013. [Online]. Available: https://doi.rrg/ 48550/arXiv.. 1338.3432 5
[29] E. Jang, S. Gu, and B. Poole, "Categorical reparamcterizatiion with gumbel-softmax." in Inter. Conf. Learn. Represent. (ICLR), 2017. [Online]. Available: https//opeemrvvicw.ee/ffommm 2ar
[30] D. Parnas, "On the criteria to be used in decomposing systems into modules," Commun. ACM, vol. 15, no. 12, p. 1053-1058, 1972. [Online]. Available: http::////oooo///11// 1011111.111111 145/361 598.366163333 5
[31] J. Deng, W. Dong, R. Socher, I..J. Li, K. Li, and I.. Fei-Fei, "ImageNet: A large-scale hierarchical image database." in IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, 2009, pp. 248-255. [Online]. Available: https://doi.org//00 wtw..oogg/...1///CC.. 11w.001119/////..2200 20045556688 6.9
[32] L. Beyer, O. J. Hénaff, A. Kolesnikov, X. Zhai, and A. van den Oord, "Are we done with ImageNet?" CoRR, vol. abs/2006.07 000006.71120 071.9,15999 2020. [Online]. Available: https//arrix.oryhhs2206...777
6
[33] B. Recht, R. Roclofs, L. Schmidt, and V. Shankar, "Do ImageNet Classificrs Generalize to ImageNet?" in Inter. Conf. Mach. Learn. (ICML), 2019. [Online). Available: https:
//proceedings.mlrr
6
[34] G. Griffin, A. Holub, and P. Perona, "Caltech 256," Apr 2022.
[35] P. Welinder, S. Branson, T. Mita, C. Wah, Schroff, S. Belongie, and P. Perona, "Caltech-UCSD Birds
200," California Institute of Technology, Tech. Rep. CNS-TR-201, 2010. 6
[36] J. Krause, M. Stark, J. Deng, and L. Fei-Fei, "3D Object Representations for Fine-Grained Categorization," in IEEE Inter. Conf. Comput. Vis. Wksps. (ICCVW), 2013. [Online]. Available: https://do..or//00. 109/CCVW..013..7
[37] H. Touvron, M. Cord, and H. Jégou, "DeiT III: Revenge of the VIT." in European Conf. Comput. Vis. (ECCV), ser. Lecture Notes Computer Science, S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and Hassner, Eds., vol. 13684. Springer, 2022, pp. 516-533. [Online]. Available: https://doi.or//I1 http://bb...og110000000779---000
27
[38] A. Steiner, A. Kolesnikov, Zhai, Wightman, J. Uszkoreit, and L. Beyer, "How train your VIT? data, augmentation, and regularization in vision transformers," " CoRR, vol. abs/2106. 10270, 2021. [Online].
Available: hutpss//arriv.orr//bb//2 1ttp///2......0000 10270 6, 27
[39] Caron, H. Touvron, I. Misra, H. Jégou, Mairal, P. Bojanowski. and A. Joulin, "Emerging properties in self-supervised vision transformers," in IEEF/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, 2021, pp. 9630-9640. [Online], Available: https://doii.o org//10 0. 109/ICCV48922.202 100951
[40] H. Touvron, M. Cord, Douze, F. Massa, A. Sablayrolles, and H. Jégou. "Training data-efficient image transformers & distillation through attention." in Inter. Conf. Mach. Learn. (ICML). ser. Proceedings of Machine learning Research, M. Mcila and T. Zhang, Eds., vol. 139. PMI.R 2021, pp. 107 [Online]. Available: http://proccedings..lrr prrppi/sssmmrrrrss.////......// 39/touvron22 a.hmml
[41] B. Zhou, H. Zhao, X. Puig. T. Xiao, S. Fidler, A. Barriuso, and A. Torralba, "Semantic Understanding of Scenes Through the ADE20k Dataset," Inter. J. Compur. Vis., vol. 127, no. 3, pp. 302-321, 2019. [Online]. Available: https://doi-org/10.. 1007//
1000000/111620000 1263-018-8811111100 018- 010---008--111-----11 1140-0
12

---

[42] T.-.. Lin, M. Maire, S. Belongie. J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft COCO: Common Objects in Context," in European Conf. Comput. Vis. (ECCV), D. Fleet, T. Pajdla, B. Schiele, and Tuytelaars, Eds. Springer International Publishing, 2014, pp. 740-755. [Online]. Available: https://doi- org/10. 1007/978- 3-319 10602-1_44 7,25
[43] K. He. X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition." in IEEF/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), 2016, pp. 770-778. [Online]. Available:
hups://doi.or//10 10.1 /0//CVPR.200.... 7
[44] T. Xiao, Liu, B. Zhou, Jiang, and J. Sun, "Unified Perceptual Parsing for Scene Understanding." in European Conf. Comput. Vis. (ECCV), 2018. [Oaline]. Available: https:
//doiorr//10 0007/97-33000-0122
26
[45] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong, F. Wei, and B. Guo, "Swin Transformer V2: Scaling Up Capacity and Resolution," " in JEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), 2022, pp. 11 199 12009. [Online]. Available: hupp:///oi..org/10 109/C1/RS2688 20022000700020011100
[46] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar, "Masked-Attentin Mask Transformer for Universal Image Segmentation," in IEEE/CVF Inter. Conf. Comput. Vis. Pottern Recog. (CVPR), 2022, pp. 1290-1299. [Online]. Available: https://doi.or// 109/CVPR52688.2. 202..220008822220001111
[47] B. Zhang, Z. Tian, Q. Tang, X. Chu, X. Wei, C. Shen et al., "SegViT: Semantic Segmentation with Plain Vision Transformers," in Adv. Neural Inf. Process. Sys. (NeurlPS), vol. 35, 2022, pp. 4971-4982. [Online]. Available: https://proceedingss. httpss/pooceddinnnnuurrppccccppp neurips.cc/paper 20189b1 20189blaaa8coo wtw.aaaaoooooo000000066600000 www.:66d80bbo1000665555560 Abstract Conference.hum httpppsccccppppeeeeeeppee/222210
[48] L. Zhu, X. Wang. Z. Ke. W. Zhang, and R. W. Lau, "Biformer: Vision Transformer with Bi-Level Routing Attention," in JEEF/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), 2023, pp. 10 10323-10 333. [Online]. Available: https://doi.org: htr////0.oo//...1o1/// 109/CVPR52229 (tt.PP/22222...22....
7
[49] A. Zhu, Mci. S. Qiao, Yan, Y. Zhu, L.-C. Chen, and H. Kretzschmar, "Superpixel Transformers for Efficient Semantic Segmentation," " IEEE/RSJ Inter. Conf. Intell. Robots Syx. (IROS), 2023. [Online]. Available: https://doi.org// 10. 10000100005552.000
10341519 7
[50] K. He, X. Chen, S. Xie, Y. P. Dollár, and R. B. Girshick, "Masked autoencoders are scalable vision learners," in JEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog, (CVPR). IEEE, 2022, pp. 15 [Online]. Available: https://doi..or/10.. http://o.........000000cR2222.2 2022000022882222201553... 2022.01553
[51] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang. S.-W. Li, I. Misra, M.
Rabbat, V. Sharma, G. Synnacve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski, "DINOv2: Learning Robust Visual Features without Supervision." Trans, Mach. Learn. Res., 2023.
[Online]. Available:
[52] H. Yan, M. Wu, and C. Zhang, "Multi-scale representations by varying window attention for semantic segmentation," in Inter. Conf. Learn. Represent. (ICLR), 2024. [Online]. Available: http://pwwwwwweeeeoe//omdddaaaaa
[53] Cui, Yuan, Z. Zhong, Tian, H. Hu, S. Lin, and Jia, "Region rebalance for long-tailed semantic segmentation," 2022. [Online]. Available: https:/aariiv.or///b///20......0
[54] R. Strudel, R. G. Pinel, I. Laptev, and C. Schmid, "Segmenter: Transformer for semantic segmentation," in IEEE Inter. Conf. Comput. Vis. (ICCV), 2021. [Online]. Available: https: /dooiorr//0..
[55] Q Yan, Xu, J. Shi, and J. Jia, "Hierarchical Saliency Detection," " IRFF/CVE Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), 2013, pp- 1155-1162, [Online]. Available: http://......tt....1///////c
[56] L. Wang, H. Lu, Y. Wang. M. Feng, D. Wang, B. Yin, and X. Ruan, "Learning Detect Salient Objects with Image-level Supervision." " in IEEE/CVF Inter. Conf. Comput. Vis, Pattern Recog. (CVPR). 2017. [Online]. Available: http::/ddb.oog/1o..09vvcc......
[57] C. Yang, L. Zhang, H. Lu, X. Ruan, and M.-H. Yang, "Saliency detection via graph- based manifold ranking." in IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, 2013, pp. 3166-3173
13

---

[58] Y. Wang. X. Shen, S. X. Hu, Y. Yuan, J. L. Crowley, and D. Vaufreydaz, "Sell-supervised Transformers for Unsupervised Object Discovery using Normalized Cut," in IEEE/CVF Inter. Conf. Comput. Vis. Patter Recog, (CVPR). 2022. [Online]. Available: https://oii.rgg ott/ Www.11002/C
http://ooooogyll....ecc/s//s2222 2022.00144 7.27
[59] T.-M. Li. M. Lukáč, G. Michaël, and Ragan-Kelley, "Differentiable Vector Graphics Rasterization for Editing and Learning." " ACM Conf. Spec. Inter. Group Graphics Interact. Techn. (SIGGRAPH), vol. 39, no. 6. pp. 193:1-193:15, 2020. [Online]. Available: https://doi- org. 10. 145/3414685.34 3317871 7, 7.8, 2.8,8
[60] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, "Image Quality Assessment: From Error Visibility to Structural Similarity," IEEE Trans. Image Procexs., vol. 13, no. 4, pp. 600-611 012, 2004. [Online]. Available:
https///oi..or//100 109/T11.2003.81111111 819861 8
[61] P. Selinger, "Potrace: A Polygon-Based Tracing Algorithm."" " 2003. 8, 27
[62] X. Ma, Y. Zhou, Xu, B. Sun, V. Filev, N. Orlov, Fu, and H. Shi, "Towards Layer-wise Image Vectorization," in JEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), 2022. [Ouline]. Available: https///doi.org/10..
http:///bb.oogg/1..11/////c/e// 111111099///S25S 2022.0001688 8, 28
[63] T. T. Tanimoto, Elementary mathematical theory of classification and prediction. International Business Machines Corp., 1958. 8
[64] C. Playout, Z. Legault. R. Duval, M. C. Boucher, and F. Cheriet, "A Region- Approach to Diabetic Retinopathy Classification with Superpixel Tokenization." in IEEE Inter. Conf. Med. Image Comput. Comput. Assist. Interv. (MICCAI), 2024, pp. 36-45. [Online). Available: https://oi.org/// //551000000/977---031----7--
[65] M. Lee, C. Park. S. Cho. and S. Lee, "SPSN: Superpixel Prototype Sampling Network for RGB-D Salient Object Detection," European Conf. Comput. Vis. (ECCV), 2022. [Online]. Available: https://di..rr// 1007/978-3-001--19988 -118188---1
[66] Y. Gao, Wang, Y. Zhang, and B. Tu, "Efficient active domain adaptation for semantic segmentation by selecting information rich superpixels." European Conf. Comput. Vis. (ECCV), 2024. [Online]. Available: https://doi.or// 10700007777777----777777575755
[67] A. Zhang, Ren, Y. Liu, and X. Cao, "Lightweight Image Resee--Rsolution with Superpixel Token Interaction," in IEEE Inter. Conf. Comput. Vis. (ICCV), 2023, pp. 12 682-122699 691. [Online]. Available: http:://doi.or/1100 10.
[68] A. P. Moore, S. J. D. Prince, J. Warrell, U. Mohammed, and G. Jones, "Superpixel Lattices," in IREF/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), 2008. [Online]. Available:
https://do..or//00
24
[69] J. Jain, A. Singh, N. Orlov, Z. Huang. J. Li, S. Walton, and H. Shi, "SeMask: Semantically Masked Transformers for Semantic Segmentation." " in IEEE Inter. Conf. Comput. Vis. (ICCV), 2023. [Online]. Available: https://doi.org// 10. 109911CC/6009999990 000.09932002000888
[70] Y. Huang, D. Kang, L. Chen, Zhe, W. Jia, L. Bao, and X. He, "CAR: Class-awarc regularizations semantic segmentation." in European Conf. Comput. Vis. (ECCV), 2022. [Online]. Available: https://doi.org/I0.. http://di..rrrg11..000////88//
25
[71] Bousselham, Thibault, I.. Pagano, A. Machireddy, J. W. Gray, Y. H. Chang, and X. Song, "Efficient
self-ensemble for semantic segmentation," in British Mach. Vis. Conf. (BMVC), 2022. [Online]. Available: https//bmwc22222.ogg 1pp.m.mppppppm inf .mg..pp......m de/8g// 25
[72] X. Dong, J. Bao, T. Zhang, D. Chen, S. Gu, W. Zhang, L. Yuan, D. Chen, F. Wen, and N. Yu, "CLIP itself is strong fine-tuner: Achieving 85.7% and 88.0% top-1 accuracy with ViT-B and ViT-L on imagenet," 2022. [Online]. Available:
26
[73] H. Zhang, M. Cisse, Y. N. Dauphin. and D. Lopez-Paz, "MixUp: Risk Minimization." in Inter. Conf. Learn. Represent. (ICLR), 2018. [Online]. Available: hutps//opennreview.eee//// www.:/enwwviiieeee/rooouuumm Dir 27
Beyond Empirical
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We briefly outline the main contributions of the work, shown in the main paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Our conclusion has section on the general limitations. We also point out and discuss the independence assumptions on residuals within individual regions in the supplementary.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification:
The theoretical results are clearly outlined in the supplementary.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We detail the methodology to promote reproducibility. We include algorithms and a section on implementation and training details, including hyperparameter settings.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide a GitHub repository with code and model weights.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//niss httppsccccpbccccccceececcccccccc /wt.dee/oooeSuumiiieeeee /Co/eeeodiissseeeeeeeea cassoooooicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide list of hyperparameters and architecture parameters the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the extent of experiments and computational resources of the models trained, we have not added confidence intervals for the main results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We estimate the computational resources in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https: /neurisursp..// cc cctpublic///cccccccccccccccccltt htthhie/eehhcchieeeleenneenee
didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
We have read the code of ethics, and adhere to these.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The code has permissive MIT license.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /atasetss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code includes docstrings adhering to the NumPy format. This will be improved for the camera-ready version, which will include more extensive documentation and example notebooks.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.M policy (https: htt///rrpps.cccooneeeeeeeeeeeeee /LLM) for what should or should not be described.
21

---

A Theoretical Results
In this section, we provide a formal construction for monotonic edge coloring with respect to graph connectedness, and show that this construction yields a hierarchical partitioning of V in Proposition A.13. We leverage this result to define a hierarchical graph partitioning in Definition A.14.
A.1
Notation and Preliminaries
[n]
Discrete sequence (0, .0,,....))) 12). Power set of set X. An undirccted graph G EV, e-channel image x
Partition IV(V).
2
H
Hierarchical partition (T+ te E 1441)mm1
Gr
Graph G2 (V,, 3 in hier. graph seq. An information criteria.
a
R
IC
(1())
Neigborhood of v G.
Likelihood function.
C(v)
Connected components of v G.
df
Degrees of freedom.
G[F]
Spanning subgraph of G under F' C E.
Volo(X) x(u, v)
Number of (u, v) E s.t. U. X.
II(V)
Set of partitions over V.
Pos. def. kernel x R 12 + R20.
Definition A.1 (Neighborhood). Let G = (V, E) be graph. The neighborhood of a vertex v E V is defined by (T) {u {u, v) E}.
Definition A.2 (Subgraphs). Let G = (V, E) be graph, and let S V. Let G[S (S, E(S)) be graph such that E|S] {u,, o) E E u, E S}. Then G[S] subgraph induced by S. Symetrically, for F' C E, subgraph G[F] = (V, F)) is a spanning subgraph under F.
Definition A.3 (Graph Connectivity). Let G (V, E) be graph. For V, let Na(v) = {v} and (v) = M(v). By recursion, define
Mu+1(1) (2)
U N(u). (E))
(A.1)
Then C(v) lim R.(v) is called the connected component of u. If C(v) = for any E V then G is connected graph.
Definition A.4 (Reachability). Let G = (V, F) be a graph. We say that two nodes u, v E V are reachable in G if and only if E C(v).
Definition A.S (Equivalence Relations and Classes). Let V be a set. Let ~ S x V be a binary relation that reflexive, transitive, and symmetric. Then equivalence relation on V. Furthermore, for some fixed element E let
[u] = fo' V V.NNo'
(A.2)
Then [v] is an equivalence class of under r..
Definition A.6 (Quotient Set). Let V be set, and let be an equivalance relation V. Then
V/~ = [w) u e
(A.3)
the quotient set of V induced by Definition A.7 (Partition of Sets). Let V be Let ITVV) 22" such that for all E IT(V)
(i) 0 n,
(ii π covers V. i.c. Usex S V.
(iii) for all S' E n, S", then S S' 0.
Then is a partition of V, and we call IIVV) the set of all partitions of V.
Definition A.8 (Refinement Partitions). Let V be set, and n, n' II(V). If for all S there exists 5' E n' such that S S S', then we say that æ is refinement of x'. denoted by π [ x'. Furthermore, we say that finer than x', and equivalently that qO is coarser than π.
22

---

Definition A.9 (Hierarchy of Partitions). Let V be set, and let H. = (IT II(V) : Th E AT41) Then H. is a hierarchy of partitions ordered by refinement.
Theorem A.10 (Fundamental Theorem Equivalence Relations). Let V be set, and let be equivalence relation on V. Then V/~ E (V)...
Definition A.11 (Weighted Graph). Let G = (V, F) be graph. Let E - R be function on the
edges of G. Then G' (V, E, x) is called a weighted graph.
A.2 Hierarchical Graph Partitions
In this section, we construct the main result Definition A.14, which formalizes the construction of hierarchical partitions via monotonic binary edge coloring.
Proposition A.12 (Partition by Edge Coloring). Let G = (V, E,x) be a weighted graph, and let x: E + {0, 1} a binary coloring of the edges such that the edge set
Ex = (Lu,0) v) E x( =1
(A.4)
is invariant under transitive closure; i.e., F Exx Then, the coloring x induces partition of into connected components, where each component connected via Exx
Proof. From Ex we construct relation ~ on V such that u ~ if and only if u reachable by in the subgraph G|Ex]. Note that is symmetric since G is undirected, and transitive due to x Ex- Since u E (()) for all u E V, then v ~ v so ~ is necessarily also reflexive. Hence, is equivalence relation V, and by the fundamental theorem of equivalence relations, the quotient set V/ is a partition corresponding to the connected components of G[Ex]-
Proposition A.13 (Hierarchical Partitioning by Monotonic Edge Coloring). Let G (V, E, x) be weighted graph, and let X: x .o....,. T} + 10 be a binary coloring of the edges satisfying
(i) Monotonicity, x(u, t) S x(u, v, for all {u, o} E and t = O..... T 1. (i) For each step t = .....: T, the edge set (t) = (uu E E: x(u, u. = 1} invariant under transitive closure, i.e., E (t) (t).
V ordered by refinement; i.e., the partition induced by X Then, x induces hierarchical partition V
is a refinement of the partition induced by Xx for all l S L', where we have that Xe(u, v) x(u, U, for all {u, v) E.
Proof. By Proposition A.12, we have that each Xt induces equivalence relation <us. partitioning V into equivalence classes at level L. Denote this partition by πτ. We will show that for all < t'. the partition ग, is a refinement of N.' The monotonicity criteria (i) X(u, t) S x(u, è + 1) implies (t) C Ex (t + 1). By induction, (t) S (t')) for all S t', Since (t) C (t)), any path using edges in Ex (t) also path (t'). Therefore, if m u,
then u ~e Now, let [u], denote the equivalence class of under Then [u], S [u]e e for all u V and all t < t'. Since n2 V/ ~ [[u], : E V}, we have that ne C We' we wanted to show.
Definition A.14 (Hierarchical Graph Partition). Let G EV, x) be an weighted graph where X E x (0....,77 T + {0. 1) is a binary coloring as in Proposition A.13, i.e., monotonic and invariant under transitive closure. Let X ((u, v, 0) O for all {u, v) E E and let ~u, denote the equivalence relation induced by Ex( for 1 = O...,, Then the sequence
(D) = (V/ Ex),, t= 0..,.., is called hierarchical graph partition, where for each E V we have that each equivalence class [w], = 5 E ne denotes a connected region for n, E H. For notional convenience, we write G[) G. Et).
(A.5)
B Modeling Assumptions and Estimators
In this section, we discuss details regarding methodological assumptions of 0HT from Section 2.4. We cover the i.i.d. Gaussian assumption on the distribution of pixels, show that this approximation has precedence, and empirically verify that this a reasonable modeling choice, and derive the estimator for df., via atomistic properties of the partition lattice.
23

---

-10
Empirical Quantiles
6
Table B.1: Ablation sensitivity Gaussian Assumptions.
Density
Shape parameter
Mettic
b = 2.0
b = 1.0
= 0.5
MSE SSIM
0.06
0.07
0.09
0.60
0.57
0.55
-1
-0.2
-0.1
0
0.1
0.2
Theoretical Quantiles
Residual
Figure B.1: QQ-plot (left) and residual density (right) for pixel residuals ImageNet-Val Val for ØHT. The residuals closely follow Generalized Normal Distribution.
B.1 Distrubutional Assumptions
superpixel model can be viewed as a spatially piecewise-constant approximation image data, constrained by connectivity requirements [68]. Specifically, for an image x superpixel model M is on the form
Ma (v) = 1|c Ss
(B.1)
Given an image defined on pixels v V, we assume the existence of a partition m, into connected regions that minimize
1 IV
MMSE
||-(1) Mr. (v)
(B.2)
where Mz. (v) takes constant value Ms for all pixels E S, and the optimal μs is given by the arithmetic mean of pixel intensities within S. Given fixed partition, this choice of estimator is optimal under the Gauss-Markov theorem as the best linear unbiased estimator (BLUE).
Within each region S, the estimator jas is BLUE under the assumption that the noise affecting pixels independent, identically distributed (i.i.d.) with finite variance and zero mean. Further, adopting Gaussian distribution for pixels conditioned on region membership 2, S ~N Σs) aligns the squared-error minimization directly with maximum likelihood estimation. Specifically, for diagonal Es. maximizing the Gaussian log-likelihood corresponds exactly to minimizing within-region vari- ance, connecting clearly with rariatie-reduction criteria in regression trees and quadtrees [20, 25], supporting our argument in Section 2.4. This is clear from the log-likelihood for multivariate Gaussian dimension for region S, which given by
|S| log L(0s) (og( log det Es) 2 vES
1 1s,,
Ms) Ms) ps)
(B.3)
|S| (dlog(2Te) log det Σs)
(B.4)
where (B.4) follows by the MLE for Es given pss.
We emphasize that these assumptions are made explicitly and are not automatically justified by the existence of an optimal partition under variance minimization. The choice of empirical variance as criterion to evaluate model fit is commonly employed in nonparametric contexts without distributional assumptions, hence the assumption of Gaussianity is not strictly necessary but implicit by the choice of risk minimizer. Claeskens and Hjort [24] show that, even if the candidate models are not parametric distributions, IC approaches remain asymptotically valid in selecting model that minimizes expected prediction error.
To empirically examine the appropriateness of these assumptions, we evaluated the pixel in- tensity distributions within representative superpixels over ImageNetlk. Figure B. show residuals follow generalized Gaussian distribution s.t. S C5((551,, us, 0011. As GN((2, uss, E.)) = = (us, Es), the model is reasonable, however the residuals are somewhat sharper and heavier tailed than the approximation.

---

Table C.I: Additional Single Scale Semantic Segmentation mloU on COCO-Stuff [42] for the original 10k fold. Note the lack of comparative baselines for base capacity models.
Dataset
Backbonc
Method
Size
mioU (t)
COCO1OK
Swin-L
SeMask [69]
47.4
CouvNext-L
CAR [70]
640
49.0
Swin-L
Senformer [71]
640
49.8
VIT-L
Segmenter [54]
512
47.1
WTFB
0HI' + MLP
512
49.2
We also assess the robustness to the Gaussian assumption. We trained tokenizers under alternative Generalized Normal distributions with shape parameters E {2, 1, 0.5),, where = 2 corresponds to the Gaussian bascline. Table B. shows that the Gaussian assumption yields the best reconstruction quality, despite final residuals being closer to se 0.551. This occurs because the distribution is closer to Gaussian in early training iterations, providing initial stability for model fitting.
The Gaussian assumption primarily affects pruning through residual variance estimation; heavier- tailed residuals inflate variance, making the criterion more conservative. Importantly, the complexity penalty remains invariant distributional misspecification as it depends only on graph topology, and 0HT demonstrates robustness consistent with theoretical results on information criteria under model misspecification [24].
B.2 Atomistic Properties of (I)
We informally outline some fundamental lattice theory [27] and describe how can derive an estimate of the degrees of freedom for a partitioned graph under the constraint of connectivity.
The set ITV)) is partially ordered set (poset) under refinement (Definition A.8), and contains two seemingly trivial elements; one is the minimal partition 1 {{u} u E V}. called the bottom where all elements of V are individual blocks. Dually, the top T {V}. is partition in which all clements are grouped in a single block. Any finite nonempty subset V will necessarily satisfy 1, T E II(.). In the partition lattice [IV,,, atoms defined as the minimal non-trivial partitions that cover the bottom element I,, where each vertex are isolated singletons. Dually, the co-atoms are the elements covered by T where all vertices comprise single set. Independent sets of atoms form what is equivalent to a basis (independent sets) in constructing more complex partitions that define each superpixel. Under connectivity in G, the atoms of (IV are precisely the edges E of G.
piecewise constant model, the complexity the model decreases for courser partitions such that the level of complexity is maximal at T. Then dfx, is necessarily inversely proportional to the number of independent atoms each superpixel encompasses within II(V). Dually, it is necessarily also proportional to the number of possible bipartitions required to form the partition of U {o a E V, & o'). Unfortunately, this quantity can be considered more or less intractable, however, estimate can be derived by considering the dualistic nature of the partition lattice.
By assumption
Recall that (Sl for some S S V defined [ffu, v} E u, E S}- Then (ola is the maximal number of atoms required to form S, which yields direct measure of the number of steps between 5 and 1. This typically formalized via a rank function IT(V) + Zzo which turns out to be equivalent to the number of atoms in partition. However, we are instead interested in the number of steps between S and T. Noting that for our construction, we have that estimate degrees of freedom of S n. (V)) by letting
r(T) = Vola (V) EE] is maxima,
di df. (S) ~ [E| Vola
(B.5)
This serves to penalize partitions S that has lower volume and are closer L. In effect, the estimate penalizes higher parameter complexity w.r.t. the piecewise constant model of the image by inducing a preference for larger connected regions in the superpixel partition.
C Extended Results
In the interest of completeness, we include results for COCO-Stufi on the 10k fold to complement Table 3. Note that COCO10k is preliminary release with much smaller number of datapoints,
25

---

Algorithm D.1 Single Merge Iteration
Algorithm D.2 Feature Extraction
Require: Feature map fe E RR
Require: Image tensor E RREXXXXW
Require: Region map St. Edge Et
Require: Region features f E &NN'C''
Require: Similarity kernel r)
Require: Region mup 5
Require:
Information criteria -C(
Require: Grid resolution N
Ensure:
Updated feature map f++1
Require: Positional resolution p E N
Ensure: Updated region mup 1
Require: Projection matrix W E Rxxo
Ensure: Updated edge set Fa
Require: Background token E xxxxx
Ensure: IC penalty Lic
Require: Mixing weight À E [0, 1]
Emaoe +
Ensure: Tokenized features F E R xxxxxx
for U E St do
Ensure: Kernel pos. features E (0, I] NxxX for do
Emax [u] + arg max, (u.))E, k(u,0)
end for
Se41 + CONNICTEDCOMPONIENS/ Emax)
Au +- m
Ru W f[w]
f++1 ZBROS(N" C)
for u E S141 do
for each pixel u do
(1 8. ((1) v) do
a(p) x [p] + Pa. pa
Vmax 4- Emax (0)
end for
w x- [S+(r)]/ [St+1 (w)l (max))
end F +- ZEROS( C. x g) r +- ZEROS x x p)
fet1 1e felo]
end for for Ext1 +- (SDATEEDDEES((--1)
do
M <- DwW.SAMPLEM = u. g)
for u E S411 da
for (4,j) E x g do BILINEARSAMPL hox(u). v MK.
Ci(lu [C( Ee+1) end for return f1+1, St-1, Bit1, Dx
+- X- 8 (1:)) 11:
F Flu,,, [] +- m + m) Sais
P(u]
KERNELPOSEMBEDDS = xp)
return F,
Figure D.I: Core algorithms for ØHT. Left: Single iteration of hierarchical vertex merging with kernhl- aggregation. Right: Differentiable feature extraction with mean-injection and adaptive masking.
which were updated to include the full COCO164k fold at later time. Consequently, there are fewer baselines available. To the best of our knowledge, out of the works reporting results COCO10k, there are no instances of base- or small capacity models available. Nevertheless, we include relevant results for COCO1OK in Table C.I, which illustrate that 0HT performs relatively well compared to larger models with larger capacity (305M parameters for large (L) compared to 87M for base (B) capacity models).
D Implementation and Training Details
We provide a full overview of our experimental setup and training configuration. Training and inference was performed on AMD MI250x and Nvidia A100. We detail central algorithms in Fig. D.I, and provide code and checkpoints in our GitHub repo Our experiments were conducted as follows:
0HT modules were pretrained to optimally reconstruct images from ImageNetIk, using (7). We train for 10 epochs using AdamW with 1c-4 learning rate and le-2 weight decay, but find that performance saturates between epoch 5-6. We test the performance with different hyperparameter setting--ff. cf. Table 6.
Tokenizer Pretraining:
We select three baseline models trained exclusively on ImageNetlk. Each model is then retrofitted with our pretrained tokenizer, using the configuration in Table Db.. Models are fine tuned with layer- learning rate decay of 0.65 [72]. which improves learning for earlier layers. We evaluate the models over various downstream tasks, yielding the results in Table 1.
Retrofitting:
Given how the baseline VIT-B32 model produces very few regions, we perform a comparative evaluation by adding more fine-grained control over the number of tokens over different image resolutions. We add merging mechanisms which serves to limit the total number of tokens in model, and evaluate baselines and retrofitted models over various image sizes. The results are provided in Figure 6.
Scale Invariance:
26

---

Table D.1: Configuration parameters for different stages
(a) Pretraining
(b) Tokenizer Retrofitting
condig
value
config
value
batch size
2048
batch size
2048
opochs
400
epochs
100
dataset
ImageNetlk
dataset
ImageNettk
img.siie
192 x 192
img.size
192 x 192
pos.emb.
x
peo.emh.
loss
(0.1 smooth.)
loss li..
(E1 smooth.)
optimizer
LAMB
optimizer
LAMB
Ir sched.
des.yecay (5 w..)
Trschod.
decay war)
Ir (start base / stop)
3c-3 33: 1c-6
(start / base / stop)
1c-7/6e-51 1c-6
momentum
momentum
dropout path
0.1 (S) (B) le-?
dropout path
0.1 (S)
opt
opt.
le-8
cutmix
1.0
sugmmnnt
rand aunda / aug3
angment
rand.aug. aug3
lrr
0.65
(c) Finetuning
(d) Segmentation Finctuning
config
value
config
value
batch size
512
batch size
512
epochs
epochs
400
datasct
ImageNetik
dataset
COCO-Stuff, ADE20K
img.size
224 x 224
img.size
512 512
pos emb..
24 x 24
eos.emh.
48 48
loss fn.
(O.1 smonth.)
loss fn.
BCE Focal AdamW
optimizer
AdamW
optimizer
Ir.ched.
cos.decay (5 w...)
Insched.
cosdecay (5 w.u.)
Ir (start base stop)
le-6/ 6/ 1e-5/ 1e-5
(start / base / stop)
1c-6/ 1c-5/ le-5
dropout path
(S) (B)
dropout path
(8) (B)
opt. e
le-8
opt. e
le-8 randaug. aug (0.5, 1.0) (0.8, 1.2)
angment
rand.aug. 3
augment
Irr
0.9
crop scale / ratio
lird
0.85
Full Training: We extend these experiments by evaluating a full training procedure, follow- ing the training process outlined by Touvron et al. [37]. Steiner et al. [38], notably without the use of MixUp [73] augmentation. as blended images produces inaccurate boundaries for learning coherent regions. Following previous works [15], models trained from scratch apply gradient histogram features. As is generally recommended, the training was performed in two steps, outlined in Table (a) and Table (C)(c) respectively. The results are featured in the lower half of Table 1.
Segmentation Fine Tuning: Given our fully trained 0HT models, we perform fine tuning for semantic segmentation. We replace each head with single hidden-layer MLP with hidden ratio of 4x. The fine tuning is performed using the configuration in Table D.)(d), and results are reported in Table 3.
We evaluate our fine-tuned classification model on zero- shot salient segmentation. We emphasize that the model has not been trained for this task. Following Wang et al. [58], we compute the graph Laplacian of the token representations, and compute bipartition using the Fiedler vector. Foreground masks are selected by passing masked tokens through the transformer, and selecting the mask that sees the least drop in performance under occlusion. Results are featured in Table 4.
Zero-shot Salient Segmentation:
Our öHT tokenizer provide high fidelity superpixels, which can be directly applied for image vectorization out-of-the- From our pretrained tokenizer we extract both an optimal, well as low granularity partition, noting that lower granularity partitions has much fewer superpixels. We then extract paths for each superpixel in the lower granularity region, and layer high granularity paths on top using potrace [61], resulting an SVG image. We compare results with quantitative baselines in Table 5.
Image Vectorization:
27

---

Figure E.1: Segmentation examples for ØHT over ADE20K, showing fine grained segmentation labels only using simple ML.P head without upscaling. Top: Original images (512 512). Middle: Annotated target images. Bottom: Predicted labels from @HT.
Figure E.2: More examples of semantic segmentation on ADE20k. Top: Original images. Middle: Annotated target images. Bottom: Predicted labels from ØHT.
E Qualitative Results and Visualizations
In this section, we extend the visualizations and results from dense predictions and image vectorization. Figures 1 and E.2 shows a selection of segmentation results on ADE20k validation. öHT performs generally well, particularly considering that the predictions are produced with single scale, i.e. we only use the last level of tokens produce predictions. The results are in large part due to the granularity of the model, and the ability to adapt the partitions the image data. The example in the second column of Figure E.2 shows an example where the granularity of the prediction arguably better than annotations. The examples also demonstrate some typical failure cases, particularly with undersegmentatiion for people in the distance, e.g. column four and five of Figure E.
In Figure E.3, we show more raster-to-vectee graphics conversions on example images from COCO- Val. We emphasize that our method produces high quality results, despite comparatively simpler approach to image vectorization. Unlike other approaches [59, 62), our method does not yield differentiable paths, and does not optimize the vector graphics for each individual image. Instead, we simply use the results of our tokenizer, trained with unrelated downstream tasks, to produce vectorized images.
Figure E.5 illustrates the properties of superpixel hierarchies constructed with different images, showing how over- and under-segmennaation necessitates adaptive model selection mechanisms, similar to mixed-scale tokenization strategies. By selecting partitions dynamically, öHT can adapt to image data over different scales, and adapt regions fit image content with pixel level granularity.
28

---

Figure E.3: Examples of vectorized images using 0HT. The top rows shows original images, while the bottom rows shows the results of image vectorization. The vectorized images contain (on average) ~ 5000 paths.
Figure E.4: Vectorization results with different numbers of paths, increasing from 500 (left) 5000 (right).
29

---

EAS
Figure E.5: Top: Detailed view of superpixel hierarchies for different images, cf. cf.-Fig. 2. Note that the higher levels often yield oversegmented regions, while lower levels yield undersegmented regions. By using model selection with information criteria, model selects the most informative tokens the full hierarchy. Bottom: Original images, included for reference.
Original
VIT [
Quadtree [20]
SLIC [16]
0HT (Ours)
Figure E.6: Comparison of spatial granularity in tokenization methods. Our proposed 0HT (right) provides an end-to-end learnable framework for tokenization.
30


Reviews:
[{"review_id": "Eo6Q41yOp1", "reviewer": "Reviewer_m47p", "summary": "The author presents dHT, namely differentiable hierarchical tokenization, aiming to address the problem in vision tokenization that text tokenizers align with semantic units, yet patch-based vision tokenizers fragment objects without regard for their structure. dHT provides a tokenization with pixel-level granularity by constructing an optimal partition via hierarchical vertex merging. While the tokenization’s multiscale adaptability and redundancy management are achieved through pruning using the information criterion. dHT also achieves differentiability through weighted aggregation and a novel mean-injection trick. Moreover, dHT can retrofit any vision transformer without altering its layers or positional embedding shape. Experiments show that dHT has competitive performance in both image-level classification and dense-prediction tasks, also supports out-of-the-box raster-to-vector conversion. An extensive ablation study proves the effectiveness of each proposed module towards the whole network.", "questions": "Instead of the most similar neighbor in image coordinate space, could the merging process be based on the most similar neighbor in feature space so that the resulting image patch might not necessarily be continuous? In other words, could the entire graph be built on feature space?\n\nAs described in the introduction section, the main purpose of this work is to make the image tokenization similar to the text token by precise semantic alignment. Will the authors try such an image tokenization method on VLM, where text comes together with an image? Will dHT perform better in aligning image and text tokens?", "limitations": "See weakness and the question part.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strength:**\nMainly based on graph theory, the author provides strong theoretical support for every design in the network architecture.\n\ndHT treats the searching for optimal partition as a model selection problem using information criteria offers a clear objective without the need for threshold calibration.\n\nThe author uses a novel mean-injection trick that provides a clever and easy way to make the generalized ViT features fully differentiable\ndHT is a plug‑and‑play visual tokenizer that can retrofit any vision transformer, offering high modularity for reuse and backward compatibility\n\n**Weakness:**\nLine 158 typo ‘sytstems’  -> systems\n\nIn dHT, while selecting tokens is made differentiable, the merging process remains handcrafted. A learnable merging process might result in further gains in performance.\n\nThe extra cost for constructing such a deformable image token should also be reported compared with fixed patch methods in terms of inference efficiency.", "was_revised": true, "final_justification": "The authors addressed my concerns. I recommend accept.", "rebuttal": "## Response to m47p\n\nWe thank the reviewer for insightful comments and insights, and for their time invested in reviewing our work.\n\n---\n\n### Strengths\n\n> *the author provides strong theoretical support for every design in the network architecture*\n\n> *information criteria offers a clear objective without the need for threshold calibration*\n\n> *mean-injection trick that provides a clever and easy way to make the generalized ViT features fully differentiable*\n\nWe appreciate the reviewers remarks on the theoretical foundation of our work, as well as the novelty of our contributions to the field.\n\n---\n\n### Weaknesses\n\n> *Line 158 typo ‘sytstems’ -> systems*\n\nWe thank the reviewer for pointing out the typo we missed at the submission of our manuscript. We will correct this, and perform additional proofreading for the camera ready version.\n\n---\n\n> *The extra cost (...) should also be reported compared with fixed patch methods in terms of inference efficiency*\n\nWe agree that reporting computational overhead is important. We will add a throughput analysis to the camera‑ready version.\n\nGiven the adaptive of dHT, token counts are different depending on the image. Following reviewer rgEH's suggestion, we report *tokens-per-second* for dHT compared to baseline ViTs with base capacity, along with average tokens per image. This provides balanced evaluation of the computational overhead associated with dHT.\n\n| Capacity| Resolution| Tokenizer | Tok./s. | Tok./im.| \n|---------|-----------|-----------|---------|---------|\n| S16     | 224       | Patch     | 192.3k  | 197.0   |\n| S16     | 224       | dHT       |  94.2k  | 240.3   |\n| B16     | 224       | Patch     |  89.7k  | 197.0   |\n| B16     | 224       | dHT       |  68.2k  | 245.7   |\n| S16     | 384       | Patch     | 112.4k  | 577.0   |\n| S16     | 384       | dHT       |  58.1k  | 494.2   |\n| B16     | 384       | Patch     |  61.6k  | 577.0   |\n| B16     | 384       | dHT       |  48.9k  | 431.8   |\n\n\n*Throughput is computed over ImageNet-val (50k images), averaged over 4$\\times$MI250x*.\n\nThe results show that dHT incurs a moderate reduction in tokens processed per second compared to the standard patch tokenizer, as we discuss in our limitations section [L303]. However, as mentioned in our paper, the reduce in throughput is significantly lower at higher capacities and resolutions, which shows promise for high resolution tasks and video.\n\n---\n\n### Questions\n\n> *Instead of the most similar neighbor in image coordinate space, could the merging process be based on the most similar neighbor in feature space so that the resulting image patch might not necessarily be continuous?*\n\nExploring non-continuous extensions to dHT is an intriguing proposal and a promising direction for future work. However, our current framework would need substantial modifications before such an approach could be made viable.\n\n- Our construction guarantees that every region remains a single connected component in the grid graph, which is a condition for the theoretical results in Appendix A (Prop.A.13) and the degrees of freedom estimate (Appendix B, Eq.B.5) by reciprocal graph volume. These are central for IC pruning. If we allow non-planar edges, these results no longer hold, so an alternative IC penalty term would need to be derived.\n- The mean-injection trick (Sec.2.2, Eq.6) via interpolated feature extraction requires regions to be reasonably dense. If a region is scattered over the image, the resulting mask becomes exceedingly sparse, which could potentially lead to vanishing gradients in end-to-end training. Alternative feature extraction frameworks could be explored, but this would make modularity and retrofitting more challenging.\n- Allowing non-contiguous patches increases the search space for merging from $O(N) \\to O(N^2)$ for $N=HW$, which makes early merges significantly less tractable. Tree-based partitioning could provide an interesting approach, but would require a fundamentally different approach than dHT. \n\nThis does not prevent a global disjoint merging strategy in the tokenizer, but key components of our proposed framework would have to be redesigned to accommodate such an approach. Alternatively, non-local merging within the ViT (e.g. adapting Bolya et al. [22]) could be combined with dHT without altering the tokenizer itself, but lies beyond the scope of this submission.\n\nWe thank the reviewer for the suggestion and will add the discussion on non-contiguous tokenisation as a promising avenue for further work in the revised manuscript.\n\n---\n\n> *the main purpose of this work is to make the image tokenization similar to the text token by precise semantic alignment. Will the authors try such an image tokenization method on VLM, where text comes together with an image? Will dHT perform better in aligning image and text tokens?*\n\nThe extension to VLMs is indeed a research direction we are pursuing. Our preliminary investigations have centered around OCR and document-focused VLMs (docVLMs) [[Faysse et al. 2025](https://arxiv.org/abs/2407.01449), [Souibgui et al. 2025](https://arxiv.org/abs/2505.07496)], which has become a standard pre-training objective for modern VLMs. BLIP‑2 [[Li et al. 2023](https://arxiv.org/abs/2301.12597)], LLaVA-NeXT [[Liu et al. 2024](https://arxiv.org/abs/2407.07895)], and CogVLM [[Wang et al. 2024](https://arxiv.org/abs/2311.03079)] all pre‑train on DocVQA‑style corpora alongside natural image captions. \n\nAdaptive tokenization can effectively address key limitations of current docVLM frameworks. We observe that square-patch tokens often become a bottleneck, as these are poorly aligned with the more heterogeneous structure inherent in scanned documents. Preliminary findings suggest that dHT can help resolve this by generating adaptive tokens that better align with the semantics on a per-sample basis. \n\nMore generally, we expect adaptive tokenization to be beneficial for general VLMs as well, but our initial investigations focus on the docVLMs, since this is becoming a canonical pretext task. While we do not present experiments on VLMs in this submission, we believe adaptive tokenization is well suited to VLM tasks, and we are in progress with further development on this line of work."}, {"review_id": "Znu09HNCd9", "reviewer": "Reviewer_rgEH", "summary": "The authors propose a hierarchical approach to image tokenization which is differentiable end-to-end. This has been a topic of discussion since long, given patch based tokenization for images is largely unintuitive. While there have been earlier approaches to this, the authors claim that they present the first differentiable tokenizer that tokenizes according to semantics of the image rather than adhering to a grid based structure. The authors present their method which involves partitioning of tokens for different heirarchies, vertex (token) merging and pruning, and feature extraction (in a differentiable way). The authors present strong results on image classification, semantic segmentation, and image vectorization.", "questions": "Please see the strengths and weaknesses section", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nThis is a refreshing take on tokenization for transformer based image processing, with a neatly defined intuitive method and strong results. The paper makes a lot of engineering choices and explains them well via ablations and theoretical motivations. The clever straight-through estimation trick enables e2e differentiation which makes the tokenizer really clean to train. \n\nWeaknesses and Questions:\n\n1. Currently, the merit of dHT for low resolution images, and subsequently tasks like image superresolution that rely on good abstraction of low res images is unclear. In Fig 6, authors mention that the default patch based tokenizer does better than their method for low resolution images as the grid based tokenization aligns well with semantic features. But shouldn't dHT also do the same? If not, does it mean that it becomes difficult to train dHT for lower resolution images? In that case, what does it mean for tasks like image superresolution which involves constructing a high res image from low res image(s)?\n2. Latency numbers are missing in the paper. Having a tokenizer that works on multiple heirarchies can be costly. Can the authors present the throughputs of their and the default ViT methods at various resolutions? I understand tokenization might be different for different images, in which case the authors can consider reporting avg number of tokens and avg throughput over a stream of images for their method. Additionally, it is currently unclear how this latency requirement would scale with image size, it would be useful to understand that.\n3. The core tokenization process, including the feature extractor and steps like mean injection, while effective in its current form, could be potentially simplified, or the steps can be swapped for something more end-to-end. While the question whether this leads to superior quality remains open, I believe it would be a direction worth exploring.", "was_revised": true, "final_justification": "The paper introduces a strong, heirarchical tokenization approach which is well supported with experiments. The rebuttal also addressed all of my (mostly minor) concerns, and therefore I will continue to hold my Accept rating", "rebuttal": "## Response to rgEH\n\nWe thank the reviewer for all comments, suggestions, and contributions, which have served to improve the overall quality of our manuscript.\n\n---\n\n### Strengths\n\n> *authors present strong results on image classification, semantic segmentation, and image vectorization*\n\n> *refreshing take on tokenization for transformer based image processing, with a neatly defined intuitive method and strong results*\n\n> *makes engineering choices and explains them well via ablations and theoretical motivations*\n\n> *clever straight-through estimation trick enables e2e differentiation*\n\nWe are grateful for the reviewer's remarks on the novelty of our motivation, and appreciate the comments on the central technical contributions in our work.\n\n---\n\n### Weaknesses and Questions\n\n> *the merit of dHT for low resolution images, and subsequently tasks like image super resolution that rely on good abstraction of low res images is unclear*\n\nWe understand the reviewers concern. For very low resolutions (e.g. CIFAR's $32 \\times 32$) a few of pixels can end up covering an entire region. In these settings, dHT will yield block-like regions similar to a square-patch tokenizer, since there are no clear inter-pixel edges to delineate,.\n\nHowever, we note that real world SR tasks are rarely defined at such low resolutions. \n- DIV2k: 4x task upscales from $510 \\times 270$ to $2040 \\times 1080$. \n- Urban100: 4x task typically upscales from $320 \\times 320$ to $1280 \\times 1280$.\n\nIn these regimes, dHT performs well, and typically benefits from fewer overall tokens than square patch tokens with similar performance. Previous work on super resolution using superpixels (e.g., SPIN [[Zhang et al. 2023](https://openaccess.thecvf.com/content/ICCV2023/papers/Zhang_Lightweight_Image_Super-Resolution_with_Superpixel_Token_Interaction_ICCV_2023_paper.pdf)]) show strong results with reduced compute.\n\nWe note that at *moderately* low resolutions, dHT generally performs quite well out-of-the-box compared to patch-based ViTs. To demonstrate this we include a simple small-scale test where we compare patch tokenization (DEITv3) to dHT on ViT-B16 on lower resolutions for ImageNet-val. \n\n\n| Tokenizer| $64 \\times 64$ | $96 \\times 96$  |\n| -------- | -------------- | --------------- |\n| Patch    | 26.01 (16 tok.)| 60.39 (36 tok.) |\n| dHT      | 59.64 (39 tok.)| 70.09 (58 tok.) |\n\n\nThe result of this small scale experiment shows that dHT adapts better to lower resolution images, albeit with a higher token count due to adaptive tokenization. We emphasize that this is out-of-the-box with no fine-tuning, i.e. the ViT interpolates the learnable positional embedding dynamically, whereas dHT’skernelized positional embeddings doesn't require any adjustments. \n\nWe thank the reviewer for their comment, and we will include this point in the camera ready version.\n\n---\n\n> *Latency numbers are missing in the paper (...) tokenization might be different for different images, in which case the authors can consider reporting avg number of tokens and avg throughput over a stream of images*\n\nWe thank the reviewer for their comment, and agree that latency numbers should be reported. We will include this in the camera ready version. As the reviewer points out, token counts can indeed be different between images.  Following your suggestion, we report *tokens-per-second* for dHT compared to baseline ViTs with base capacity, along with average tokens per image. \n\n| Capacity| Resolution| Tokenizer | Tok./s. | Tok./im.| \n|---------|-----------|-----------|---------|---------|\n| S16     | 224       | Patch     | 192.3k  | 197.0   |\n| S16     | 224       | dHT       |  94.2k  | 240.3   |\n| B16     | 224       | Patch     |  89.7k  | 197.0   |\n| B16     | 224       | dHT       |  68.2k  | 245.7   |\n| S16     | 384       | Patch     | 112.4k  | 577.0   |\n| S16     | 384       | dHT       |  58.1k  | 494.2   |\n| B16     | 384       | Patch     |  61.6k  | 577.0   |\n| B16     | 384       | dHT       |  48.9k  | 431.8   |\n\n\n*Throughput is computed over ImageNet-val (50k images), averaged over 4$\\times$MI250x* with `fp32` precision.\n\nThe results show that dHT incurs a moderate reduction in tokens processed per second compared to the standard patch tokenizer, as we discuss in our limitations section [L303]. However, as mentioned in our paper, the reduce in throughput is significantly lower at higher capacities and resolutions, which shows promise for high resolution tasks and video.\n\n---\n\n> *The core tokenization process (...) could be potentially simplified (...) it would be a direction worth exploring.*\n\nWe acknowledge that the design has potential for improvement, and mention this as future work and limitations in Sec.4.1 [L309]: \n\n*Our differentiable feature extraction method adheres to modularity and backward compatibility with ViTs. Although this provides unique advantages, the features are not necessarily optimal for adaptive regions. Further work is required on optimal feature extraction for superpixel tokens while maintaining fully differentiable partitioning mechanisms*\n\nIn this work, our design was guided by our focus on retrofitting, and modeling decisions were made towards standard ViT commensurability to directly allow backward compatability with existing models. If one were willing to forgo this requirement, a more parsemoneous implementation could indeed be possible, but at the expense of backward compatability, which we claim as one of our central contributions. However, we agree with reviewer's sentiment, this is a clearly a direction worth exploring in future work."}, {"review_id": "YSLXsWzNWS", "reviewer": "Reviewer_hZMf", "summary": "This paper proposes a novel adaptive differentiable hierarchical tokenizer for ViT models, named $\\partial$HT. The method first extracts pixel features using a lightweight CNN module, constructs a hierarchical division of pixels via a graph-based paradigm, searches for the best partition using information criteria, and finally aggregates features through a specialized extraction technique. The tokenizer supports various scenarios, including retrofitting pre-trained patch-based ViTs and training from scratch. Extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "1. I am curious about the function $f$ in Eq. 2 and Eq. 5. What is the relationship between $\\bar{f}_1$ ​ and $f_1$? Is Eq. 2 just a special case of Eq. 5? \n2. I am also wondering whether the performance could be further improved if the averaging operation in Eq. 2 and Eq. 5 were replaced with a learnable weighted aggregation, similar to that used in graph neural networks.", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength:\n1. This paper is highly complete and rigorous. The limitations and future directions are discussed in detail as well. It further explores the feasibility of end-to-end differentiable approaches based on existing irregular tokenizers, reducing the reliance on heuristic rules and adopting a more flexible data-driven design. Moreover, the limitations section discusses various aspects from multiple perspectives, making it quite comprehensive.\n\n2. Extensive experiments and ablations demonstrate the effectiveness of the proposed method. The experiments compare the proposed method with the canonical tokenizer across tasks of different granularities, and the visualization results are impressive.\n\n\n### Weakness:\nThe organization of this paper can be further improved for better understandability. Specifically:\n  - the \"High Level Overview\" in Section 2 is not comprehensive enough; instead of only listing what each step does, it should present a clearer logical flow, potentially moving some contents from the following subsections upward, as the overall logic is currently only reflected in Figure 2 and should also be articulated in the text. \n  - A clearer explanation of the computation logic and algorithm implementation in Section 2.3 should be provided. \n  - Additionally, while conceptual, symbolic, and formulaic definitions are necessary, it would be better to minimize the introduction of new definitions where possible.", "was_revised": true, "final_justification": "I believe the authors' response has resolved my concerns and questions well. And I hope the authors can further refine their work in the camera ready version as they promised.", "rebuttal": "## Response to hZMf\n\nThank you for your time invested in reviewing our work. We appreciate your comments, and look to incorporate your suggestions to improve the quality of our manuscript.\n\n---\n\n### Strengths\n\n> *This paper is highly complete and rigorous (...) limitations and future directions are discussed in detail (...) discusses various aspects from multiple perspectives, making it quite comprehensive*\n\n> *Extensive experiments and ablations demonstrate the effectiveness of the proposed method (...) compare the proposed method with the canonical tokenizer across tasks of different granularities, and the visualization results are impressive*\n\nWe appreciate the reviewer’s positive remarks and their summary of the key strengths of our work.\n\n---\n\n### Weaknesses\n\n> *the \"High Level Overview\" in Section 2 is not comprehensive enough (...) instead of only listing what each step does, it should present a clearer logical flow*\n\nWe appreciate the reviewer highlighting the potential gap in our presentation, and agree that the overview should be improved.\n\nIn the camera ready version, we will restructure subsection 2.1 to include a more conceptual overview, briefly recalling the problem statement from the introduction before introducing the high level concept of the main idea. \n\nThis will bring the logical flow implicit in Figure 2 into the prose. Steps (ii)–(iv) will be previewed in one paragraph each, with forward references to their detailed subsections.\n\n---\n\n> *a clearer explanation of the computation logic and algorithm implementation in Section 2.3 should be provided*\n\nWe agree that expanding on core logic and algorithmic steps could improve the clarity of our work, and make it more available to readers.\n\nIn addition to the code in the repo, we add a boxed “Algorithm 1” pseudocode showing a single iteration of the merge loop. \n\n```\n────────────────────────────  Algorithm 1  —  Single merge iteration\nInputs :    feature map          f_t ∈ ℝ^{N×C}\n            region map           S_t\n            edge set             E_t\nParameters: similarity kernel    κ(·,·)\n            information criteria IC(·,·)\nOutputs:    updated feature map  f_{t+1}\n            updated region map   S_{t+1}\n            updated edge set     E_{t+1}\n            IC penalty           ℒ_IC\n            \n 1: E_max ← ∅                                           # init empty merge graph\n 2: for v in S_t:\n 3:     E_max[v] ← argmax_{u: {u,v} ∈ E_t} κ(u, v)\n\n 4: S_{t+1} ← ConnectedComponents(E_max)                # N → N' merged regions\n 5: f_{t+1} ← zeros(N'×C)                               # init new feature map\n \n 6: for u in S_{t+1}:                                   # for each new region\n 7:     for v in {v | S_{t+1}(v) == u}:                 # for each child region\n 8:         v_max ← E_max[v]\n 9:         w ← |S_t(v)| / |S_{t+1}(u)| · κ(v, v_max)\n10:         f_{t+1}[u] += w · f_t[v]                    # weighted aggregation\n\n11: E_{t+1} ← UpdateEdges(S_{t+1})\n\n12: for u in S_{t+1}:\n13:     ℒ_IC[u] ← IC(f_{t+1}[u], E_{t+1})\n\n14: return f_{t+1}, S_{t+1}, E_{t+1}, ℒ_IC\n────────────────────────────────────────────────────────────────────────────────\n```\n\nAdditionally, we add a boxed \"Algorithm 2\" pseudocode to clarify details on feature extraction.\n\n```\n───────────────────────  Algorithm 2  —  Feature extraction\nInputs :    image tensor            x ∈ ℝ^{B×C×H×W}\n            region features         f ∈ ℝ^{N×C′}\n            region map              S                    # pixel → region index\nParameters: grid resolution         q ∈ ℕ\n            positional resolution   p ∈ ℕ\n            projection matrix       W ∈ ℝ^{C×C'}         # learnable\n            background token        β ∈ ℝ^{C×q×q}        # learnable\n            mixing weight           λ ∈ [0,1]            # learnable\nOutputs :   tokenized features      F ∈ ℝ^{N×C×q×q}\n            kernel pos. features    P ∈ [0,1]^{N×p×p}\n\n 1: for u in S:\n 2:     μ_u ← mean_{v ∈ u}(x[v])                         # empirical mean\n 3:     μ̂_u ← W ⋅ f[u]                                   # predicted mean\n 4:     for each pixel p in u:\n 5:         x̂[p] ← x[p] + μ̂_u - μ_u                      # Equation (6)\n\n 6: F ← zeros(N×C×q×q)\n 7: P ← zeros(N×p×p)\n \n 8: for u in S:\n 9:     M ← DownsampleMask(S == u, q×q)                  # foreground occupancy\n10:     for (i,j) in q×q:\n11:         s ← BilinearSample(x̂ , bbox(u), i, j)\n12:         m ← M[i,j]\n13:         s_mix ← λ · s + (1−λ) · β[:,i,j]             # pixel mask blending\n14:         F[u,:,i,j] ← m · s + (1−m) · s_mix           # Equation (8)\n15:     P[u] ← KernelPosEmbed(S == u, p×p)\n\n16: return F, P\n────────────────────────────────────────────────────────────────────────────────\n```\n\nWe thank the reviewer for their suggestion, and agree that additional pseudocode helps communicate the computational logic in our manuscript. It may also serve as a helpful companion to the code in the project repo. \n\n---\n\n> *while conceptual, symbolic, and formulaic definitions are necessary, it would be better to minimize the introduction of new definitions*\n\nWe will audit Section 2 and the appendix and remove one-off symbols or terms that appear only once or twice. When a quantity is used in just a single derivation, we will look to reuse existing symbols. Any lattice-specific terms that are needed only for the proofs (join, meet, rank) will stay in Appendix A.\n\nWe thank the reviewer for their comments, and their contribution to improving the clarity of our work. \n\n---\n\n### Questions\n\n> *I am curious about the function in Eq. 2 and Eq. 5. What is the relationship between $\\bar f_1$ and $f_1$ Is Eq. 2 just a special case of Eq. 5?*\n\nYes, Eq. 2 is the *uniform-kernel* instance of the more general update in Eq. 5.\n\n$\\bar f_{1}(v)$ (Eq. 2) is the pure arithmetic mean of the raw pixel embeddings $f_{0}(u)$ over the first connected component $S=C_{1}(v)$, which demonstrates how aggregation is performed by Aasan et al. [15]. $f_{1}(v)$ in Eq. 5 shows our proposed approach with dHT. \n\nAt $t=0$, every “region” $S_{0}$ is a single pixel, so $|S_{0}|=1$, the ratio $|S_{0}|/|S_{1}|=1/|S_{1}|$. For later levels ($t\\ge 1$) Eq. 5 keeps accumulating information with *kernel-weighted averages* and the size ratio $|S_{t}|/|S_{t+1}|$ so that features stay unbiased while still carrying the similarity signal. The extra weighting is what the barless $f_{t+1}$ symbol is meant to capture.\n\nWe see how the this notation could potentially be unclear to readers, and we will more clearly specify that the vertex merging in Eq. 2 is a general case of Eq. 5. \n\n---\n\n> *I am also wondering whether the performance could be further improved if the averaging operation in Eq. 2 and Eq. 5 were replaced with a learnable weighted aggregation, similar to that used in graph neural networks.*\n\n\nThat is an interesting suggestion. Eq. 5 already performs an *edge-weighted aggregation*\n\n$$\nf_{t+1}(v)=\\sum_{u\\in S_t(v)}\n\\tfrac{|S_t(u)|}{|S_{t+1}(v)|}\\,\n\\kappa(u,u_{\\max})\\,\nf_t(u),\n$$\n\nwhere the *kernel weight* $\\kappa(u,u_{\\max})$ is recomputed from the current feature vectors at every training step (Section 3.1). Because those features receive gradients, the weights are *implicitly learnable*. Introducing an *explicitly parameterised attention kernel* would be an interesting extension, but a few practical constraints arise.\n\n- *Symmetry:* Our setup assumes $\\kappa(u,v)=\\kappa(v,u)$. A key–query attention map would break that symmetry, doubling the number of kernel evaluations and requiring an additional normalisation step to keep the update stable.\n- *Positive-definiteness:* Eq. 1 assumes $\\kappa$ is a positive semi-definite (PSD) similarity kernel so that the edge set $E_{\\max}$ remains well-defined. A freely parametrized kernel should preserve PSD to keep the theoretical guarantees (Appendix A) of the hierarchy construction.\n\nWe agree that exploring parametric extensions (e.g. an adaptive PSD kernel or a learned Mahalanobis metric) is an appealing avenue for future work, and will add a discussion on learnable extensions for weighted aggregation in the camera ready version."}, {"review_id": "L06Xz0SJBo", "reviewer": "Reviewer_foiG", "summary": "Vision Transformers (ViTs) traditionally use fixed grid-based image patches for tokenization, which ignores semantic object boundaries and spatial hierarchies. This paper introduces ∂HT (Differentiable Hierarchical Tokenization), an end-to-end learnable tokenizer that adapts to image content at pixel-level granularity while maintaining backward compatibility with existing ViTs.", "questions": "It is also essential to compare the model's parameter count and computational time in the experiments.\n\nThe \"decoder-free\" segmentation uses a simple MLP head. Is ∂HT’s performance bottleneck the tokenizer or the MLP head?\n\nThe IC formulation assumes i.i.d. Gaussian pixels, but Fig. B.1 shows heavier-tailed residuals. How sensitive is ∂HT to this assumption?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\na). The idea makes sense. ∂HT bridges the semantic gap in visual tokenization by unifying adaptability, differentiability, and modularity. It demonstrates that tokenization can be a learnable component in ViTs, enabling dynamic, content-aware representations without architectural overhaul.\n\nb). Rigorous Methodology. The paper introduces a novel differentiable tokenization framework (∂HT) with well-defined theoretical foundations. The hierarchical partitioning via monotonic edge coloring is mathematically sound.\n\nc). Comprehensive Experiments. Extensive evaluation across tasks  and datasets demonstrates robustness. Ablation studies validate design choices (e.g., kernel selection, positional embedding resolution).\n\n\nWeaknesses:\n\na). The writing is not very clear. The organization of the sections is a bit messy, such as 1.1 and 4.1.\n\nb). It is also essential to compare the model's parameter count and computational time in the experiments.\n\nc). In the Main Results of Table 1, improvements over baselines are modest,  if not negative, raising questions about practical impact.", "was_revised": true, "final_justification": "The authors' rebuttal has fully resolved all my concerns. I have therefore raised my overall rating of the paper.", "rebuttal": "## Response to foiG\n\nWe appreciate the reviewers time in reviewing our submission. Their constructive comments have contributed towards refining and enhancing our manuscript.\n\n---\n\n### Strengths\n\n> *The idea makes sense. ∂HT bridges the semantic gap in visual tokenization by unifying adaptability, differentiability, and modularity (...) demonstrates that tokenization can be a learnable component in ViTs*\n\n> *well-defined theoretical foundations (...) hierarchical partitioning via monotonic edge coloring is mathematically sound.*\n\n> *Extensive evaluation across tasks and datasets demonstrates robustness. Ablation studies validate design choices*\n\nWe thank the reviewer for their summary, and appreciate their comments on the motivation, theoretical grounding, and empirical evaluation of our approach.\n\n---\n\n### Weaknesses\n\n> *the organization of the sections is a bit messy, such as 1.1 and 4.1.*\n\nWe understand why the reviewer found these two sections a little unclear, since each combines different purposes (motivation and related work in 1.1, further work and limitations in 4.1) into single sections. We will separate these into different sections in our final revision. \n\n---\n\n> *essential to compare the model's parameter count and computational time in the experiments* \n\nWe agree on both points; parameter counts and computational overhead are indeed important metrics that would be of interest to readers.\n\nIn terms of parameter count, dHT adds 53k parameters compared to the DEiT3-B16 baseline, which is 4 orders of magnitude less than the total number of parameters (.06%). We will include explicit parameter counts for all baselines and models, and report the results in our final revision.\n\nGiven the adaptive nature of dHT, token counts differs depending on the image. Following reviewer rgEH's suggestion, we report *tokens-per-second* for dHT compared to baseline ViTs with base capacity, along with average tokens per image. This provides balanced evaluation of the computational overhead associated with dHT. We will include the results in the camera ready version of our manuscript.\n\n| Capacity| Resolution| Tokenizer | Tok./s. | Tok./im.| \n|---------|-----------|-----------|---------|---------|\n| S16     | 224       | Patch     | 192.3k  | 197.0   |\n| S16     | 224       | dHT       |  94.2k  | 240.3   |\n| B16     | 224       | Patch     |  89.7k  | 197.0   |\n| B16     | 224       | dHT       |  68.2k  | 245.7   |\n| S16     | 384       | Patch     | 112.4k  | 577.0   |\n| S16     | 384       | dHT       |  58.1k  | 494.2   |\n| B16     | 384       | Patch     |  61.6k  | 577.0   |\n| B16     | 384       | dHT       |  48.9k  | 431.8   |\n\n*Throughput is computed over ImageNet-val (50k images), averaged over 4$\\times$MI250x* with `fp32` precision.\n\nThe results show that dHT incurs a moderate reduction in tokens processed per second compared to the standard patch tokenizer, as we discuss in our limitations section [L303]. However, as mentioned in our paper, the reduce in throughput is significantly lower at higher capacities and resolutions, which shows promise for high resolution tasks and video.\n\n---\n\n\n> *In (...) Table 1, improvements over baselines are modest, if not negative, raising questions about practical impact*\n\nWe acknowledge that improvements vary by task and model configuration, and therefore opted to characterize our results as *competitive* while clearly specifying that dHT does not necessarily improve performance in *every* comparison [L199-L206].\n\nTo put the results in context, Table 1 summarizes 30 comparisons on top-1 linear accuracy and kNN performance across 6 datasets and five model configurations. In the majority of cases, dHT is competitive or better than the baselines. Specifically, for top-1 accuracy, *dHT outperforms the patch baseline in 80% (24/30) cases*, and *70% (42/60) when kNN is included*. The mean improvement is +3.4 pp. for top-1 and +2.9 pp. when including kNN.\n\nWhen dense-prediction tasks are included (Tables 3–5), we note a trend of consistent performance gains on instance level as well as dense downstream tasks. \n\n---\n\n### Questions\n\n> *The \"decoder-free\" segmentation uses a simple MLP head. Is ∂HT’s performance bottleneck the tokenizer or the MLP head?*\n\nThis is a very interesting question. We believe there are gains to be had in both MLP head as well as the tokenizer. In our experiments, we opted to keep the architecture consistent between the tasks (classification and segmentation), and the capacity of the CNN in the tokenizer could potentially be expanded to improve performance in dense tasks. Likewise, we do comparision on single scale (output) baselines for the MLP, and results could potentially be improved by expanding to a multi-scale approach (i.e., concatenating outputs from multiple blocks) which would expand the MLP capacity.\n\nWe emphasize that the intent of the decoder-free segmentation head is to measure the representational quality of tokens \"out-of-the-box\". More expressive decoders are likely to improve both baselines and dHT proportionally. If we were to select between the two, we believe adding more capacity to the MLP head is more likely to improve segmentation results.\n\n---\n\n> *The IC formulation assumes i.i.d. Gaussian pixels, but Fig. B.1 shows heavier-tailed residuals. How sensitive is ∂HT to (Gaussian) assumption?*\n\nIn Appendix B.1, we connect Gaussian assumptions to empirical variance as a risk minimizer, and discuss the validity of non-parametric (e.g., ERM) models for information criteria (IC). Claeskens and Hjorth [24] also discuss robustness of IC under misspecification, showing that information criteria are *relatively robust to misspecification*. \n\nIn our case, the Gaussian assumption is equivalent to a Generalized Normal ($\\mathcal{GN}$) but with a higher shape parameter $b$ than the final residuals show. The choice of Gaussian likelihood only affects pruning through the residual variance estimate. While heavier‑tailed residuals inflate variance the effect is that the pruning is more moderate, making the IC more selective. The complexity penalty itself is immune to misspecification, but adding a penalty term for local kurtosis could potentially correct for this at the cost of computing kurtosis per region.\n\nTo estimate dHT's robustness to the Gaussian assumption, we train tokenizers with $b \\in \\{2,1,.5\\}$ following the ablations in Table 6. The results yields slightly worse results compared to the Gaussian baseline ($b=2$). \n\n| Metric | $b=2$    | $b=1$     | $b=.5$    |\n|--------|----------|-----------|-----------|\n| MSE    | 0.06     | 0.07      | 0.09      |\n| SSIM   | 0.60     | 0.57      | 0.55      |\n\nGiven that the distribution of residuals end up closer to $b=0.551$, this may initially seem somewhat surprising. However, we find that the distribution is closer to standard Gaussian ($b = 2$) in *early iterations of training*, providing initial stability in fitting the model. \n\nTo directly answer the reviewer's question; *dHT is not particularly sensitive to assumptions of Gaussianity*. We thank the reviewer for an engaging question, and will include this discussion in our final revision."}]

Author rebuttal:

Meta-review: The authors proposed an algorithm for differentiable image tokenization. All the reviewers agree the novelty of the paper. The experiments are well and the paper is well written. The problem they tackled is important and fundamental. I believe it will be an impactful work.
Decision: Accept (spotlight)