Title: Differentiable Hierarchical Visual Tokenization
Venue: NeurIPS 2025
Accepted: yes
Keywords: ViT

Partitioning

Tokenization

Segmentation

Vectorization

Adaptive

Hierarchical

Model Selection
Abstract: Vision Transformers rely on fixed patch tokens that ignore the spatial and semantic structure of images. In this work, we introduce an end-to-end differentiable tokenizer that adapts to image content with pixel-level granularity while remaining backward-compatible with existing architectures for retrofitting pretrained models. Our method uses hierarchical model selection with information criteria to provide competitive performance in both image-level classification and dense-prediction tasks, and even supports out-of-the-box raster-to-vector conversion.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nVision Transformers (ViTs) traditionally use fixed grid-based image patches for tokenization, which ignores semantic object boundaries and spatial hierarchies. This paper introduces ∂HT (Differentiable Hierarchical Tokenization), an end-to-end learnable tokenizer that adapts to image content at pixel-level granularity while maintaining backward compatibility with existing ViTs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\na). The idea makes sense. ∂HT bridges the semantic gap in visual tokenization by unifying adaptability, differentiability, and modularity. It demonstrates that tokenization can be a learnable component in ViTs, enabling dynamic, content-aware representations without architectural overhaul.\n\nb). Rigorous Methodology. The paper introduces a novel differentiable tokenization framework (∂HT) with well-defined theoretical foundations. The hierarchical partitioning via monotonic edge coloring is mathematically sound.\n\nc). Comprehensive Experiments. Extensive evaluation across tasks  and datasets demonstrates robustness. Ablation studies validate design choices (e.g., kernel selection, positional embedding resolution).\n\n\nWeaknesses:\n\na). The writing is not very clear. The organization of the sections is a bit messy, such as 1.1 and 4.1.\n\nb). It is also essential to compare the model's parameter count and computational time in the experiments.\n\nc). In the Main Results of Table 1, improvements over baselines are modest,  if not negative, raising questions about practical impact.\n\nQUESTIONS:\nIt is also essential to compare the model's parameter count and computational time in the experiments.\n\nThe \"decoder-free\" segmentation uses a simple MLP head. Is ∂HT’s performance bottleneck the tokenizer or the MLP head?\n\nThe IC formulation assumes i.i.d. Gaussian pixels, but Fig. B.1 shows heavier-tailed residuals. How sensitive is ∂HT to this assumption?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel adaptive differentiable hierarchical tokenizer for ViT models, named $\\partial$HT. The method first extracts pixel features using a lightweight CNN module, constructs a hierarchical division of pixels via a graph-based paradigm, searches for the best partition using information criteria, and finally aggregates features through a specialized extraction technique. The tokenizer supports various scenarios, including retrofitting pre-trained patch-based ViTs and training from scratch. Extensive experiments demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength:\n1. This paper is highly complete and rigorous. The limitations and future directions are discussed in detail as well. It further explores the feasibility of end-to-end differentiable approaches based on existing irregular tokenizers, reducing the reliance on heuristic rules and adopting a more flexible data-driven design. Moreover, the limitations section discusses various aspects from multiple perspectives, making it quite comprehensive.\n\n2. Extensive experiments and ablations demonstrate the effectiveness of the proposed method. The experiments compare the proposed method with the canonical tokenizer across tasks of different granularities, and the visualization results are impressive.\n\n\n### Weakness:\nThe organization of this paper can be further improved for better understandability. Specifically:\n  - the \"High Level Overview\" in Section 2 is not comprehensive enough; instead of only listing what each step does, it should present a clearer logical flow, potentially moving some contents from the following subsections upward, as the overall logic is currently only reflected in Figure 2 and should also be articulated in the text. \n  - A clearer explanation of the computation logic and algorithm implementation in Section 2.3 should be provided. \n  - Additionally, while conceptual, symbolic, and formulaic definitions are necessary, it would be better to minimize the introduction of new definitions where possible.\n\nQUESTIONS:\n1. I am curious about the function $f$ in Eq. 2 and Eq. 5. What is the relationship between $\\bar{f}_1$ ​ and $f_1$? Is Eq. 2 just a special case of Eq. 5? \n2. I am also wondering whether the performance could be further improved if the averaging operation in Eq. 2 and Eq. 5 were replaced with a learnable weighted aggregation, similar to that used in graph neural networks.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a hierarchical approach to image tokenization which is differentiable end-to-end. This has been a topic of discussion since long, given patch based tokenization for images is largely unintuitive. While there have been earlier approaches to this, the authors claim that they present the first differentiable tokenizer that tokenizes according to semantics of the image rather than adhering to a grid based structure. The authors present their method which involves partitioning of tokens for different heirarchies, vertex (token) merging and pruning, and feature extraction (in a differentiable way). The authors present strong results on image classification, semantic segmentation, and image vectorization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis is a refreshing take on tokenization for transformer based image processing, with a neatly defined intuitive method and strong results. The paper makes a lot of engineering choices and explains them well via ablations and theoretical motivations. The clever straight-through estimation trick enables e2e differentiation which makes the tokenizer really clean to train. \n\nWeaknesses and Questions:\n\n1. Currently, the merit of dHT for low resolution images, and subsequently tasks like image superresolution that rely on good abstraction of low res images is unclear. In Fig 6, authors mention that the default patch based tokenizer does better than their method for low resolution images as the grid based tokenization aligns well with semantic features. But shouldn't dHT also do the same? If not, does it mean that it becomes difficult to train dHT for lower resolution images? In that case, what does it mean for tasks like image superresolution which involves constructing a high res image from low res image(s)?\n2. Latency numbers are missing in the paper. Having a tokenizer that works on multiple heirarchies can be costly. Can the authors present the throughputs of their and the default ViT methods at various resolutions? I understand tokenization might be different for different images, in which case the authors can consider reporting avg number of tokens and avg throughput over a stream of images for their method. Additionally, it is currently unclear how this latency requirement would scale with image size, it would be useful to understand that.\n3. The core tokenization process, including the feature extractor and steps like mean injection, while effective in its current form, could be potentially simplified, or the steps can be swapped for something more end-to-end. While the question whether this leads to superior quality remains open, I believe it would be a direction worth exploring.\n\nQUESTIONS:\nPlease see the strengths and weaknesses section\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author presents dHT, namely differentiable hierarchical tokenization, aiming to address the problem in vision tokenization that text tokenizers align with semantic units, yet patch-based vision tokenizers fragment objects without regard for their structure. dHT provides a tokenization with pixel-level granularity by constructing an optimal partition via hierarchical vertex merging. While the tokenization’s multiscale adaptability and redundancy management are achieved through pruning using the information criterion. dHT also achieves differentiability through weighted aggregation and a novel mean-injection trick. Moreover, dHT can retrofit any vision transformer without altering its layers or positional embedding shape. Experiments show that dHT has competitive performance in both image-level classification and dense-prediction tasks, also supports out-of-the-box raster-to-vector conversion. An extensive ablation study proves the effectiveness of each proposed module towards the whole network.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\nMainly based on graph theory, the author provides strong theoretical support for every design in the network architecture.\n\ndHT treats the searching for optimal partition as a model selection problem using information criteria offers a clear objective without the need for threshold calibration.\n\nThe author uses a novel mean-injection trick that provides a clever and easy way to make the generalized ViT features fully differentiable\ndHT is a plug‑and‑play visual tokenizer that can retrofit any vision transformer, offering high modularity for reuse and backward compatibility\n\n**Weakness:**\nLine 158 typo ‘sytstems’  -> systems\n\nIn dHT, while selecting tokens is made differentiable, the merging process remains handcrafted. A learnable merging process might result in further gains in performance.\n\nThe extra cost for constructing such a deformable image token should also be reported compared with fixed patch methods in terms of inference efficiency.\n\nQUESTIONS:\nInstead of the most similar neighbor in image coordinate space, could the merging process be based on the most similar neighbor in feature space so that the resulting image patch might not necessarily be continuous? In other words, could the entire graph be built on feature space?\n\nAs described in the introduction section, the main purpose of this work is to make the image tokenization similar to the text token by precise semantic alignment. Will the authors try such an image tokenization method on VLM, where text comes together with an image? Will dHT perform better in aligning image and text tokens?\n\nLIMITATIONS:\nSee weakness and the question part.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}