Title: Native Segmentation Vision Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision Transformers

Segmentation

Perceptual Grouping
Abstract: Uniform downsampling remains the de facto standard for reducing spatial resolution in vision backbones. In this work, we propose an alternative design built around a content-aware spatial grouping layer that dynamically assigns tokens to a reduced set based on image boundaries and their semantic content. Stacking our grouping layer across consecutive backbone stages results in hierarchical segmentation that arises *natively* in the feature extraction process, resulting in our coined Native Segmentation Vision Transformer.
We show that a careful design of our architecture enables the emergence of strong segmentation masks solely from grouping layers, that is, without additional segmentation-specific heads. This sets the foundation for a new paradigm of *native*, backbone-level segmentation, which enables strong zero-shot results without mask supervision, as well as a minimal and efficient standalone model design for downstream segmentation tasks.


Peer reviews:
comments: SUMMARY:
This paper proposes a novel segmentation architecture Native Segmentation Vision Transformer (SeNaTra), which replaces traditional uniform downsampling in vision backbones with content-aware spatial grouping layers. The spatial grouping layer creates hierarchical segmentation that emerges naturally during feature extraction, eliminating the strict need for dedicated segmentation heads.

STRENGTHS AND WEAKNESSES:
**Strengths**
1. This paper is well-organized and easy to follow.
2. The paper propose a novel architecture that replaces grid-based uniform downsampling with semantic-aware spatial grouping, intrinsically encoding segmentation capabilities within the backbone's hierarchical feature extraction without requiring dedicated segmentation heads.
3. Significant outperformance on zero-shot segmentation tasks, even against models trained on much larger datasets.

**Weaknesses**
1. While the "native" model is effective, the paper's best results still rely on a powerful segmentation head (Mask2Former). This weakens the argument for decoder-independence and demonstrates that such heads remain essential for performance.
2. The paper would be strengthened by a discussion and empirical comparison with ToMe [1].
3. How sensitive is the model to the number of grouping iterations L, a key hyperparameter that was fixed at 3? An ablation study analyzing the performance vs. efficiency trade-off for varying L would be a valuable addition.


[1] Token Merging: Your ViT But Faster, ICLR 2023

QUESTIONS:
See weakness

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper challenges the current standard for spatial downsampling in vision backbones, replacing uniform downsampling with a learnable image-adaptive downsampling based on differentiable grouping. Thanks to this novel spatial grouping layer, super-pixel-like naturally arises in feature maps, enabling zero-shot segmentation tasks without additional task-specific heads or mask supervision. Being a general backbone, the proposed method can also be improved by leveraging mask supervision and segmentation heads, while not being strictly required.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- the paper is well written and easy to follow;\n- it challenges the current standard in vision, proposing a novel spatial downsampling technique;\n- segmentation naturally arises from the architecture, without mask-level supervision;\n- solid experimental section and ablation studies;\n- this new method can either improve existing segmentation pipelines or pave the way for new paradigms not using task-specific heads.\n\nWeakness:\n- the only minor weakness I can find is the limited set of competitors included in table 2, many others backbones and segmentation heads could be included.\n\nQUESTIONS:\nComing back to the previously mentioned weakness, for example, having the backbone native segmentation capabilities, I would have liked to see if lighter segmentation heads (e.g. \"You Only Segment Once: Towards Real-Time Panoptic Segmentation\" or \"PEM: Prototype-based Efficient MaskFormer for Image Segmentation\".) could benefit more from this paradigm wrt the heavy M2F head. Since the backbone natively supports segmentation, I suppose the heads would need to do much less heavy-lifting.\n\nI wonder how this approach, leading to native segmentation arised thanks to architectural design, could be leveraged together with vision foundation models (e.g. DINO) where similar properties wrt dense predictions arise through training, considering that these models are usually ViT-based and non-hierarchical.\n\nLIMITATIONS:\nThe only limitation, as also stated in the paper, is the additional overhead introduced by the grouping layer. Indeed, this is likely a minor limitation, since this overhead is negligible wrt the overhead introduced by task specific modules, not strictly needed anymore for this model.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents native group layer, enables tracking pixels from high-resolution during downsampling. The method can be seamlessly integrated with several segmentation tasks. The experiments demonstrate its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\n(1) It is interesting regarding the trackable downsampling of the Group layer, which is a potential way to balance segmentation quality (high-resolution) and memory costs.\n\n(2) The paper is well-written, providing a comprehensive discussion with related works.\n\nQUESTIONS:\n(1) The description of M_loc in the algorithm section is too brief, yet this mechanism is arguably the core feature that distinguishes this method from query-based approaches. Based on the limited information provided, is it correct to assume that it simply masks out all regions that fall outside of a predefined rectangular window?\n\n(2) Furthermore, since M_loc is non-learnable, this fixed masking rule might be unable to properly handle objects that span a very large area, such as a sky that occupies half of the screen. Does this local constraint pose a limitation for segmenting large-scale objects?\n\nLIMITATIONS:\n(1) Given the current trend where open-vocabulary image segmentation is increasingly becoming a sub-task for Large Language Models (LLMs) and Large Vision Models (LVMs), I am curious about the potential for the native transformer architecture to be integrated with state-of-the-art VLMs, such as LISA.\n\n(2) I understand that native segmentation appears to offer a different paradigm from the conventional segmentation head approach. However, I have reservations about its scaling potential. For instance, in Table 6, the model shows no performance advantage over Swin at the Large scale. Furthermore, Table 2 does not provide performance results for a \"Large + native\" configuration.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed native segmentation vision transformer. It uses the spatial grouping layer to replace uniform grid-based downsampling with learned dynamic assignment of visual tokens to semantically coherent groups based on image content. Successive grouping operations across backbone stages naturally compose into a mapping from input pixels to final tokens, effectively creating a multi-scale hierarchy of segmentation masks for tokens at each backbone stage. Experiments conducted on zero-shot segmentation tasks across multiple benchmarks demonstrate that the proposed model significantly outperforms prior art, including models trained on an order of magnitude larger datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe paper introduces the concept of \"native segmentation,\" which generates high-quality segmentation masks solely through the backbone network without requiring additional segmentation heads. The spatial grouping layer is ingeniously designed, leveraging a differentiable iterative clustering algorithm, demonstrating strong innovation..\n2.\tThe paper is well-written and easy to follow.\n3.\tThe paper includes evaluations across classification, zero-shot, and supervised segmentation, alongside detailed ablations, which thoroughly validate the approach.\n4.\tThe method shows strong performance in both zero-shot and supervised segmentation with fewer parameters and FLOPs than many baselines.\t\n\nWeakness:\n\n1.\tThe grouping tokens are initialized via strided convolution. How impactful is this initialization approach compared to random or learned initializations as used in Slot Attention?\n2.\tHave the authors evaluated whether tokens from intermediate stages (e.g., stage 2 or 3) are semantically meaningful enough for classification or segmentation? Could this allow for controllable granularity or multi-scale segmentation?\n3.\tGiven that grouping tends to merge spatially close tokens, does the model struggle with small object segmentation?\n\nQUESTIONS:\n1.\tThe grouping tokens are initialized via strided convolution. How impactful is this initialization approach compared to random or learned initializations as used in Slot Attention?\n2.\tHave the authors evaluated whether tokens from intermediate stages (e.g., stage 2 or 3) are semantically meaningful enough for classification or segmentation? Could this allow for controllable granularity or multi-scale segmentation?\n3.\tGiven that grouping tends to merge spatially close tokens, does the model struggle with small object segmentation?\n4.    How about the results compared to SAM2?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}