Native Segmentation Vision Transformers
Authors: Guillem Braso
Aljosa Osep
Laura Leal-Taixé
Venue: NeurIPS 2025 poster ()
Keywords: Vision Transformers
Segmentation
Perceptual Grouping
Primary area: applications
TL;DR: We introduce a segmentation-centric backbone architecture built around a spatial grouping layer, capable of generating high-quality segmentation masks through feature extraction.

Abstract: Uniform downsampling remains the de facto standard for reducing spatial resolution in vision backbones. In this work, we propose an alternative design built around a content-aware spatial grouping layer that dynamically assigns tokens to a reduced set based on image boundaries and their semantic content. Stacking our grouping layer across consecutive backbone stages results in hierarchical segmentation that arises *natively* in the feature extraction process, resulting in our coined Native Segmentation Vision Transformer.
We show that a careful design of our architecture enables the emergence of strong segmentation masks solely from grouping layers, that is, without additional segmentation-specific heads. This sets the foundation for a new paradigm of *native*, backbone-level segmentation, which enables strong zero-shot results without mask supervision, as well as a minimal and efficient standalone model design for downstream segmentation tasks.

Native Segmentation Vision Transformers
Guillem Brasó
Aljoša Ošep
Laura Leal-Taixe
NVIDIA
rescarch.nvidia.com//lu
Input panches:
Stago 2 output
Stage 3 ourput resolution Mooos
Stage outpat: resolation 2x7
Input petches:
Stago 2 ourput
Stage 3 output:
Stage 4 curput resolution
resollttio S6x36
resalation 28128
resolation Stac6e
resolution 25x.8
resclution Mixid
Pooling and bilinear interpolation
Our learned gouu hicrarshy
Figure Downsampling vision backbones via uniform downsampling (top) v.s. learned downsampling (bottom, this work): Vision backbones downsample feature maps using uniform-grid operators (e.g., pooling, top) and rely on uniform upsampling (e.g., bilinear interpolation, top) for image segmentation tasks. Our reduced set of tokens, aligning with image boundaries during downsampling (bottom). This enables scalable backbone-level native segmentation, i.e., without the need for dedicated segmentation heads.
backbone with spatial grouping layers learns to map pixels to
Abstract
Uniform downsampling remains the de facto standard for reducing spatial reso- lution in vision backbones. In this work, we propose an alternative design built around a content-aware spatial grouping layer, that dynamically assigns tokens a reduced set based on image boundaries and their semantic content. Stacking our grouping layer across consecutive backbone stages results in hierarchical seg- mentation that arises natively in the feature extraction process, resulting in our coined Native Segmentation Vision Transformer. We show that a careful design of our architecture enables the emergence of strong segmentation masks solely from grouping layers, that is, without additional segmentation-speet -speciffi heads. This sets the foundation for new paradigm native, backbone-leve segmentation, which enables strong zero-shot results without mask supervision, as well as minimal and efficient standalone model design for downstream segmentation tasks.
1 Introduction
Status quo. Modern hierarchical vision backbones [1, 2, 3] mirror the design principles of early convolutional networks [4], organizing feature processing across multiple stages at progressively lower spatial resolutions. While feature processing has been challenged, e.g., convolutions v.s. self- attention, the downsampling stage has largely remained unchanged. Typically implemented via ubiquitous pooling or, more recently, strided convolations [1], these operations treat all spatial locations in grid uniformly. irrespective of the image content. Such hierarchical feature extraction forms the foundation for state-of-the-- art image segmentation methods, where dedicated segmentation heads [5, 6] learn to upsample and group the resulting features into semantically meaningful regions.
The uniform spatial treatment of features during downsampling manifests feature misalignment during upsampling operations, placing an additional burden on decoder heads to compensate for inherent limitations in the backbone design [7, 8). To this end, recent works [9, 10, 11, 12] explore
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

alternative segmentation network designs and strategies for based on their semantic content. Despite their conceptual appeal, these methods fall short against modern architectures due to either (i) algorithms with quadratic computational complexity relative to input resolution [9, 10], or (ii) non-differentiable grouping operations that limit their scalability and widespread practical use [11, 12], and necessitate the use of dedicated segmentation heads for downstream segmentation tasks, instead of capitalizing on their pixel-groupin grouping capabilities.
bottom-up pixel grouping
Native segmentation. We introduce Native Segmentation Vision Transformer (SeNaTra), a back- bone architecture whose core component, the spatial grouping layer, replaces uniform grid-based downsampling with learned dynamic assignment of visual tokens to semantically coherent groups based on image content. Successive grouping operations across backbone stages naturally compose into mapping from input pixels to final tokens. effectively creating a multi-scale hierarchy of segmentation masks for tokens at each backbone stage. We call this capability native segmenta- tion, as it emerges from the backhone's inherent region-aware representation, rather than external heads [13, 6, 5J. makes such external heads no longer strictly required, although empirically they can still be beneficial.
Our design has two main methodological advantages over prior backbone-level grouping work: (i) unlike methods using vanilla cross-attention [9, 10] or non-differentiable clustering [11, 12] we employ differentiable, iterative clustering inspired by perceptual grouping algorithms [14, 15]. embedding a structured inductive bias that enables coherent groups to arise without direct supervision; (ii) we ensure scalability through local grouping layers with restricted context windows in early enageing linear scaling with input coolutton-whil employing dense grouping only in the final stage to efficiently produce whole-image segmentation masks. Overall, our design enables scalable native segmentation while retaining efficiency and remaining end-to-end differentiable.
Key findings. We observe that in the absence of any mask supervision, super-pixel-lik structures emerge as a consequence of our network design (Figure 1, bottom), akin to classical superpixel algorithms [16, 17, 18. 14], rather than being hand-crafted [19], or explicitly used input [1]]. These are further grouped into semantically meaningful regions in the final, dense grouping layer. We validate our native segmentation capability on zero-shot segmentation tasks across multiple established benchmarks and show that our model significantly outperforms prior art, including models trained on an order of magnitude larger datasets, suggesting our architecture data-efficient, thanks to our grouping layer. When trained with explicit mask supervision for semantic and panoptic segmentation on ADE20k [20] and CanO-panoptic [21] our method outperforms several strong baselines without any dedicated segmentation heads, e.g. Rol heads [5] or Transformer decoders [6], with significantly reduced parameter and FLOP count. Furthermore, when used in conjunction with such heads, SeNaTra consistently improves the performance of top-performing backbones.
In summary, we (i) propose a Native Segmentation Vision Transformer, that learns hierarchical segmentation of the visual input in the absence of any pixel/mask supervision. The key building block of our network is (ii) our grouping layer that performs image-content-uuee adaative feature downsampling. effectively replacing uniform, grid-based feature nown/up-samplinng layers, employed in consolidated segmentation networks. Finally, (iii) we unveil streamlined native segmentation network that obtains masks in the absence of any dedicated heads, and excels at zero-shot segmentation, trained without any pixel/mask supervision, as well as on standard semantic/panoptic segmentation benchmarks.
2 Related Work
Vision backbones Since the pioncering work of Neocognitron [22] and LeNet [23], Convolutional Neural Networks (CNNs) have been propelling the advancements in data-driven computer vision. These networks typically employ a hierarchy of convolutional layers that apply set of learnable filters to the input feature map, which are alternated with feature downsampling operations, yielding hierarchy of multi-scale feature maps. Despite the rise of plain transformer- based architectures [24], modern hierarchical backbones [1, 3, 25] are still dominant in dense prediction [26] and still adhere to the same underlying design principle: they are organized in multiple feature extraction stages, with uniform downsampling operations among them. In this work, we put our focus on the largely overlooked downsampling operation, and show that by replacing it with our proposed spatial grouping module, we can obtain a backbone with native segmentation capabilities.
2

---

Dense prediction. In the last decade, we witnessed a Cambrian explosion in network design for dense prediction. Notable examples include Fully Convolutional Networks [27], encoder-decoder architectures [28], and the pioneering work of [29, 30]. More recently, DETR [31] tackled end-to-end detection as set prediction using Transformers, treating object proposals or segments as learnable queries. MaskFormer [13, 6] capitalized on this design, and added a pixel decoder to upsample feature maps, and trained it jointly with a backbone and transformer decoder to process queries. SeNaTra can be used in conjunction with such segmentation heads to improve segmentation accuracy, or produce high-quality native masks in the absence of such dedicated heads. Recent large-scale efforts such SAM and SAM2 [32, 26] have focused on introducing promptable segmentation and scaling, combining massive data generation pipelines with standard ViT- backbones [24, 33]. In contrast, work targets architectural innovation at the backbone level, introducing native segmentation capabilities with potential to complement or simplify such frameworks.
Perceptual grouping. Prior to the advent of end-to-end segmentation methods, combinatorial optimization was the main algorithmic tool for this task. Notable examples include the seminal work of [17], which introduced efficient graph-based segmentation to adaptively merge regions based on internal variation, and normalized cuts [34]. Traditional superpixel algorithms, such as SLIC [14], emerged as efficient tools to obtain segments based on color similarity and proximity. Recognizing segmentation's 's inherent ambiguity, several methods explored progressively merging regions into hierarchies of segments across multiple scales [18, 35]. Our approach draws inspiration from these but reformulates them in the context of modern, end-to-end trainable vision backbones.
Several methods proposed learning- based mechanisms for pixel grouping. [36] introduced dif- ferentiable variant of the SLIC algorithm for task-specific superpixels. Similarly, [15] proposed a differentiable variant of K-Means for unsupervised object discovery, which iteratively assigns image pixels to a set of slots. While these approaches inspire our spatial grouping layer, we instead propose sparse and efficient design, and integrate it as a fundamental building block for modern backbones.
Grouping in vision backbones. GiT [9] and ClusterFormer [10] pioneered the design of data-driven backbones with learnable downsampling operations. They group image constituents into reduced set of tokens using (dense) cross-attention layers, which hinder their scalability due the quadratic complexity of the attention operation w.r.f. input size. An orthogonal line of work [37] accelerates vision transformers for classification tasks with heuristic-based token merging strate- gies. By contrast, our approach is general, fully learnable, and scalable to large input resolutions as early local layers rand educe input token set cardinality on which dense layers operate. This enables our approach to be used in variety of segmentation tasks and also significantly outper- forms cross-attention-base grouping [10] in text-supervised semantic segmentation. Alternatively, [11, 12] mitigate this issue using non-differentable super-pixel method [19] to obtain initial image segmentation followed by data-driven grouping. while TCFormer [12] relies on an external clustering method to group image constituents across multiple network layers. Our approach does not require such non-diniee -in-dfferentiabl clustering methods and consists solely of differentiable grouping layers. Our streamlined design performs favorably compared to prior art in zero-shot segmentation. Moreover, unlike the aforementioned works, we show it performs favorably both with and without dedicated segmentation heads in downstream segmentation tasks.
3 Native Segmentation Vision Transformers
Our Native Segmentation Vision Transformer (SeNaTra) follows the standard structure of modern hicrarchical vision backbones [1, 2. 3], consisting of four stages that progressively reduce the spatial resolution of feature maps while doubling their channel dimensions (Figure 2). Given an input image of size H x W, the initial stage splits it into 4 x 4 patches to obtain initial token embeddings, and each subsequent stage S, i = 2,...,4 4 produces tokens at resolution of (H/2++1) x (W/2++1..
In Section 3.1, we describe our spatial grouping layer that replaces uniform downsampling layers in in-between network stages. By composing these grouping layers our backbone builds a hierarchical image representation that organizes pixels into increasingly large, semantically meaningful regions (Figure 2 (a)). While our approach is general and task-agnostic, our learned downsampling operation further enables noundary-preserviin feature upsampling, especially beneficial in downstream dense prediction tasks, such segmentation, presented in Section 3.2.
3

---

3.1
Content-aware Spatial Grouping Layer
11-122
Locral Sclte- SelT-hinie Mitiniee
Locall Saft- Artentic
Decoe Scate- AcAlilitiiii
Local Grospingg
Local tresaingg
Dencce Creepping
Solmna
Weigted
Assigrmnnt
Avange
Stage
Stagp 2
Stape 3
Stagg 4
Dipot-Gempp Assigneent
Group Fastures Update
A A
(a) Architecture overview
(b) Spatial Grouping Layer (c) Learned upsampling
Figure 2: Overall model design. Visualization of our hierarchical architecture and its key components. (a) Our backbone architecture consists of four processing stages interconnected by content-aware grouping layers for downsampling.
(b) Core operations of our Spatial Grouping Layer, which computes soft token assignments and updates group features iteratively (detailed in Algorithm I).
(c) The composition of learned assignment matrices across grouping layers in consecutive backbone stages enables principled feature upsampling.
Learning semantically meaningful pixel groups. Uniform downsampling operations such as pooling or strided convolutions, which de facto standard in current architectures, treat all feature locations in an image equally regardless of their feature content, and apply a fixed operation for all input tokens. This approach inherently limited in its ability to distinguish between high and low-frequency regions and capture relevant details. To address this limitation, we propose to learn a mapping between input and downsampled tokens that dynamically adapts to input features, instead of solely relying on feature positions in a grid. Specifically, we map tokens with similar feature embeddings, hence belonging to the same object or semantically meaningful region, the same output token in downsampled representation. By learning such mapping, our model preserves semantically meaningful boundaries within the image across its consecutive network stages.
Grouping algorithm. Building on this intuition, we frame our task as differen- tiable clustering process inspired by the K-means [38, 39] and its modern differen- tiable variant [15] where our output down- sampled tokens act as centroids, and input tokens are iteratively assigned to them. For- mally, let X R R-N"x xd denote set of Nin d-dimensional input tokens, which corre- spond to either pixel embeddings tokens
Algorithm 1 Grouping layer over an input feature map X for L iterations with sparsity.
Input: Feature map {0. 1)
X
Mase E
Learnable Modules: Strided Conv Conv; linear projections Q. V: MLP, LN; rel. pos. bias B; temp. T.
1:
X + LN(Conv(X))
2: for I = I,.... L do
3:
A + TM x 2(XOuTT B
4:
+- A+M1oc
from a previous stage. We aim to produce a reduced set of /oou 2d-dimensional tokens with reduced spatial dimensions. Follow- ing standard architecture designs, we set Nowr = Nin /4 for all layers.
4 + softmax (A)
6:
for
L
Now do
7:
A
down
IN 1
8:
end for
9:
X X (.N([[Acoaa) x v(X
Our full approach is outlined in Algo- rithm 1. We first initialize Xour with a strided convolution, it is common prac- tice [25, 3]. Then, for L iterations (L=3
10: X out + LN(MLP(XXun)
II:
end for
12:
return X Adwun, ups
in our experiments), we alternate between two key steps: (i) computing soft assignment matrix from input tokens with cross-attention-iic operation (L3-5), and (ii) renormalizing this matrix over columns to update xou with weighted mean over input tokens (L6-9). Intuitively, since AMPS E [0, 1 Nx doaan
is row-normalized, each element A can be interpreted as the probability that each input token X gets mapped to output downsampled token X These assignment probabilities are then used to update the corresponding features of oou (I.9), which act as centroids.
For clarity, we show a naive implementation with sparsity via M1.:(0 for enabled pairs, -oo otherwise). See Appendix E. for our efficient implementation.

---

By repeating this process over L steps, we iteratively refine both the assignment probabilities as well as the resulting features X
Local and dense grouping. A key limitation of Algorithm 1 lies in the cost of computing A (1.3) due to the quadratic complexity w.r.f. the cardinality of the input token set, N making it impractical for high-resolution feature maps. Inspired by the SLIC algorithm for superpixel generation [14, 36], for high-resolution feature maps we restrict the computation of cross-attention coefficients a small 3 x 3 local window centered around each output token in X (see Figure 2 b). Intuitively, this mechanism retains the flexibility of a learned downsampling operator, where input tokens can be dynamically mapped to their downsampled counterparts, and injects locality prior: input tokens will be mapped to tokens that will be close in the resulting output space. This enables the notion of locality in output tokens, allowing us to leverage commonly used local attention mechanisms [1, 3]. Computationally, this prior results in highly sparse 4 and Adown matrices that can be efficiently computed with CUDA kernels (see Appendix E. and, overall, reduces the computational complexity of our Grouping Layer from dI. down to (I.Nd) making practical for high-resolution maps. In our architecture, we use local grouping in the second and third stages, where higher- resolution feature maps are processed. In the final stage, we enable dense, i.e., non-sparse, grouping, which ensures that our model's output tokens can merge regions and objects over the entire input image.
Connection to Slot Attention. The core operations in our grouping layer are akin to those introduced in Slot Attention [15]. Our downsampled tokens can be interpreted as slots that, instead of being sampled from a random distribution, are initialized by strided convolution layer over input tokens X. Additional technical differences include replacing the GRU originally used to update slots, iee. pixel groups, with a simpler skip connections (Algorithm 1, L9) and the use of relative positional encodings to encode spatial relationships between input and output tokens (Algorithm 1, L3). More importantly, the sparsity constraints in the cross-attention operation introduced in the previous paragraph enable efficient processing of high-resolution inputs, making this differentiable grouping mechanism practical for hierarchical vision backbones.
3.2 Native Segmentation
Composing assignments via Markov chain. By forwarding an image through our model, the combined output of all grouping layers yields two sets of matrices (447h) =i=1- and (Adown) J==1- where each matrix A (resp. Adown) corresponds to the output of the grouping layer at stage i i 1, with dimensions N x Nqout As grouping layers applied in consecutive stages, N: = N for each = 1, 1.... , n 1. Now, recall that by construction, A is row-stochastic matrix, where entries can be interpreted as the probability of each input token being mapped to a subsequent downsampled state transition matrix, and the overall
token. Each matrix A. can therefore be interpreted mapping from tokens at stage I to tokens at an earlier stage I - k E {1....n n - 1} can be interpreted as Markov chain with state transition probabilities given by:
A
A A- X...
A
A AL-++ (Adown )) x... (ApownyT.
(1)
Where, analogously, since Adown a column-stochaste matrix, A defines mapping for tokens from stage I to + k. Therefore, any set X of arbitrary of Nyou d-dimensional token embeddings at stage / can be upsampled to stage - k (resp. downsampled to l + k) resolution via dot product Aups A X, (resp. 1 X). Since all except for our last grouping layers utilize local grouping, at most one matrix in the product of assignment matrices will be non-sparse. The product of all sparse matrices involved is also block-sparse and can be efficiently computed (see Appendix E.
Backbone-level segmentation. The observations made in the previous paragraph enable a proba- bilistic interpretation of hierarchically decomposing an image into segments. At each stage i, A maps input tokens, i.e., image patches, to No disjoint tokens, i.e., segments, where No decreases with i. Our final stage 4 enables dense grouping, allowing tokens to encode segmentation masks spanning the entire image. Notably, this can be achieved without explicit supervision of intermediate transition matrices or their composition. Since grouping layers are differentiable, our entire archi- tecture remains end-to-end trainable on standard image-level objectives through global pooling of final-stage tokens. At inference time, applying a learned classification head or text embeddings to final tokens, followed by upsampling via ATT enables zero-shot input-level predictions suitable
5

---

for semantic segmentation. Despite the absence of mask supervision, our grouping layer's strong inductive bias yields high-quality masks in this setup, as we show in Section 4.1.
Leveraging mask supervision. Image segmentation tasks can be divided into partitioning an image into S disjoint segments, and doing per-segment classification. While contemporary methods rely on specialized heads to enable instance-level high-resolution predictions [6, 40], our model directly encodes image partitions through input-output token mappings ATT at the backbone level. This enables minimalistic purely native approach: training only MLPs to classify our final tokens with bipartite-matie matching losses. Furthermore, our model can be integrated into standard segmentation frameworks with a key improvement: feature map upsampling and downsampling operations. commonly used in pixel decoders, can be replaced with our grouping- -ased operations. leading to improvements over the segmentation accuracy of state-of-the- methods (Section 4.2).
4 Experiments
Overview. In the following, extensively evaluate SeNaTra with w.t.t different supervision regimes and task complexity. In Section 4.2, we start with mask-free supervision and study emerging segmentation from image-class (Section 4.1.1) and image-caption (Section 4.1.2) supervision, com- paring our model to state-of-the-art zero-shot segmentation methods. In Section 4.2, we train and evaluate our model standard datasets and benchmarks for semantic (Section 4.2.1) and panoptic (Section 4.2.2) segmentation, comparing our direct segmentation model and backbone as drop-in replacement against state-of-the-art, We analyze our design choices and contributions in Section 4.3.
Models. We evaluate three SeNaTra models: tiny (T), base (B), and large (L), with output embedding dimension of 512, 1024, and 1536, following [3]. Full configurations are provided in Appendix D.
4.1 Learning Without Mask Supervision
4.1.1
ImageNet Classification
We train SeNaTra on ImageNet- Ik and ImageNet-22k [41], following the training setup of [1]. We visualize the learned group representations at different backbone stages in Figure 3, along with final per-group activations for the predicted class (5th col), and refer Appendix D.1 for quantitative analysis and comparison with standard backbones [3]. While our network performs on on-par with state-of-the-art on ImageNet classification task, we observe that as by-product of our network design, our network produces a hierarchy of ppreeerign super-pixel-ite groups, combined in the last, dense grouping layer into meaningful semantic regions (4th col). We emphasize we train our models using output-level class supervision only. Our model retains state-of- state-ofthhe-rr the--rt performance w.r.t classification, and, remarkably, learns per-pixel localization of objects withou mask supervision as a direct consequence of our proposed architectural changes.
4.1.2 Zero-shot Segmentation from Vingon-Language Supervision
Setup. We pre-train SeNaTra with image-text pairs using softmax contrastive objective [42, 43], borrowing hyperparameters from [44]. We evaluate our models in zero-shot semantic segmentation. To obtain image group embeddings, we apply linear projection layer the final image (resp. text) output tokens and apply global pooling, followed by L2 normalization. To classify, we feed class names (for each dataset) through the text encoder with standard template prompts and pick the class with maximum cosine similarity for each group embedding, followed by our upsampling operation (Section 3.2). For details, see Appendix D.2.
Datasets. Following [44], we train our model for 20 epochs from seratch on the union of the CC3M [45] and CC12M [46] datasets (20M semi-curated image-text pairs), and union including Red- Caps 2M dataset [47] (+12M additional pairs). Following [48], we evaluate trained models on Pascal VOC [49], Pascal Context [50], COCO [51], COCO-Stuff [52], ADE20k [20] and Cityscapes [53]. These span diverse scenarios, ranging from urban street scenes (Cityscapes), general object categories (COCO, Pascal VOC), and densely annotated fine-grained scenes (ADE20K, Pascal Context). We discuss results in terms of standard mean intersection-ore crarsection-over-uiion (mloU).
Discussion. As can be seen in Table 1, our SeNaTra outperforms specialized otate-of-the-ar methods across most benchmarks, including models leveraging CLIP's large-scale pre- mre-training on 400M

---

Pred: Bald cngte
Pred: strawberry
Pred: dhole
Prod: monntr
lowf
randpuper
Pred: moving vun
Input Image
Stage Groups
Stage 3 Groups
Final Stage 4 Groups
Class Activations
Figure 3: Segmentation emerges from ImageNet pre-training, We visualize group decompositions across each backbone stage, along with their upsampled activations over the predicted class. We observe that even in the absence of mask supervision, super- super-pixel-to pixel-like structures emerge in earlier layers, and are eventually grouped into semantically coherent regions dense grouping layers.

---

Method
Training data
Postproc.
voc
Cix
Ohj.
Swuf
City
ADF:
Avg.
CLIP Pretrannd Methods
Vil.-Seg [54]
CC12M
37.3
18.9
18.1
SegCLIP [55]
CC3M+COCO
52.6
24.7
26.5
TCL [48]
CC3M+CCI2M
PAMR
55.0
30.4
31.6
22.4
24.0
17.1
30.1
CoDe [56]
CC3M+CCI2M
PAMR
57.7
30.5
32.3
23,9
28.9
17.7
31.8
Models triined from scratch
Group [9]
CCMMCC12M+YFCCCI
49.5
19.0
24.3
12.6
6.9
8.7
20.2
ViewCo [57]
CC12M+YFCCI4M
52.4
23.0
23.5
CoCu [58]
CCM+<C12M-YYCCIINN
51.4
23.6
22.7
15.2
22.1
12.3
24.6
PGSeg [59]
CC12M-RedCAss12M
53.2
23.8
28.7
SimSeg [44]
CCAM-CCI2M
CRF
57.4
26.2
29.7
SeNaTTr-B (Ours)
CCAM+CCI2M
61.3
30.2
32.6
21.1
30.0
16.4
31.9
SeNaTra-B (Ours) CCMM#CC12M-RRdlCgs222)
61.4
31.2
33.2
23.2
32.1
17.4
33.1
Table 1: Zero-shot, text-supervised semantic segmentation. We compare our method to state-of- the-art methods six datasets, and report average mloU across datasets where applicable. We bolden lop-performers, and underline 2nd and indicate postprocessing techniques (CRF [60], PAMR [61]).
image-text pairs, 20x larger than our training set. We observe large improvements (4+) mloU over all datasets w.r.f. methods not utilizing CLIP. We note that top-performing methods (TCL [48],
CoDe [56], and SimSeg [44]), rely postprocessing techniques such as PAMR [61] and dense CRFs [60], that increase their performance by 4 mloU as reported in [48, 44]. contrast, we obtain strong results due to our network design, without applying any postprocessing- Our approach also surpasses methods leveraging CLIP on most datasets, except ADE20k and COCO-stuff (150 and 133 classes, respectively), where are second to CoDe. The increased semantic granularity of these datasets benefits from extensive CLIP pre-training. Remarkably, by expanding our training data with just 12M additional image-text pairs from RedCaps 12M, significantly narrowed this gap, showcasing the potential for further scaling.
4.2 Training with Mask Supervision
Overview. We train SeNaTra with mask supervision standard semantic [49] and panoptic segmentation [21] datasets. Following common practice, we initialize weights from ImageNet pre-training (Section 4.1.1). Appendix D.3 provides extended results and implementation details.
Segmentation paradigms. For each task, evaluate (i) our minimal native masks model that generates masks via backbone-level pixel assignments, and (ii) drop-in backbone replacement in conjunction with a Mask2Former (M2F) [6] dedicated head (see Table 2[c]).
Native segmentation: We make per-pixel class predictions by feeding our backbone's final group token embeddings through a 2-layer (512 dim.) MLP. We then upsample these (at stride 32) input resolution using our learned pixel assignments (Section 3.2), and class predictions using cross-entropy loss. For panoptic models, we use additional 2-layer MLP targeting objects. We apply it over the top-100 final group tokens with largest assignment values, representing object candidates. We follow [6] and supervise instance mask and class predictions with bipartite matching loss [31].
Ours+ MMakk2FFrmee Our network versatile and can also be used as a drop-in replacement with networks, such as the widely used M2F, that combine dixcl-decoder using multi-scale deformable attention with a segmentation Transformer decoder. In our version, we replace standard upsampling operations with the assignment matrices obtained through our learned assignments (Section 3.2).
Baselines. As hackbone baselines, we report methods that follow a consolidated design with uniform downsampling. including well-established SwinTranssoo Transformer [1] and NAT [3], as well as recent bottom- up grouping approaches [11, 12, 10]. We report these in conjunction with dedicated segmentation networks, including: UperNet [40], commonly used for benchmarking vision architectures [1, 3, 25, 62, 63), and widely-used MaskFormer (MF) [13] and Mask2Former (M2F)(6). We evaluate SeNaTra as both backbone and to generate native masks without dedicated segmentation heads.
4.2.1 Semantic Segmentation
Setting. We train models to classify pixels into 150 semantic classes on ADE20k dataset [20], and, following common practice, report results the validation set. We follow similar hyperparameter
8

---

Backbone
Seg. Head mloU Paarams
FLOPs
Backbone
Seg. Head
PQ
#Params
Backbones w/ Uniform Downsampling
[win-T[[]]
MF
47.7
42M
Swin-Tr [1]
UperNet
44.5
60M
946G
SeNaTra-T
Native
49.2
32M
Swin-T [1]
M2F
47.2
47M
Swin-T [1]
M2F
53.2
47M
NAT-T [3]
UperNet
47.1
58M
934G
NAT-T" [3]
M2F
54.3
46M
NAT-T" [3]
M2F
49.1
46M
ClusterFm. [10]
[10]
54.7
Swin-B' [1]
M2F
53.9
107M
SeNaTra-TT
M2F
55.0
47M
Swin-L [1]
M2F
56.1
215M
Swin-B [1]
MF
51.8
102M
Backbones w/ Grouping based Downsampling
SeNaTra-B'
Native
52.6
96M
CAST-S(11)
Segmenter
43.1
26M
Swin-B [1]
M2F
56.4
107M
TCFm. VC-S (12)
SemFPN
47.1
29M
370G
SeNaTra-B'
M2F
57.1
112M
SeNaTra-T
Native
49.7
30M
113G
Swin-L [1]
M2F
57.8
216M
TCFm. V2-S [12]
M2F
49.1
42M
SeNaTra-L
M2F
58.1
228M
-lusterFm.TT
[10]
49.1
(b) Panoptic segmentation COCO-val.
SeNaTra-T
M2F
51.3
47M
TCFm. V2-B [12]
SemFPN
50.0
66M
332G
Transfomes Decoder
SeNaTra-B
Native
51.3
95M
347G
TCFm. V2-B [12]
M2F
53.8
80M
Backbone
SeNaTra-B
M2F
54.6
112M
Backbone
SeNaTra-B'
M2F
56.0
112M
SeNaTra-IL
M2F
56.7
228M
M2F head
Native Segmentation
(c) Segmentation paradigms.
(a) Semantic segmentation on ADE20K-val.
Table 2: Downstream semantic and panoptic segmentation after fine-tuning, (a) mloU on ADE20K. (b) PQ COCO va12017. (c) Conceptual visualization of segmentation paradigms. Models marked with are pre-trained on ImageNet-22K. NAT-T" is our implementation.
configurations as baselines (details Appendix D.3), except for reduced number of iterations from 160k to 80k due increased convergence speed with our model.
Discussion, Table 2a observe: (i) our native masks yield substantial improvements over both standard and grouping-basee backbones using well-established segmentation heads (UperNet [40], Semantic FPN [64], Segmenter [65]), with remarkable compute and parameter-efficient smaller variants. SeNaTra-T achieves 49.7 mloU, +2.6 w.z.. NAT w/ UperNet (47.1 1 mloU, NAT-T), with only 12% of its FLOPs and 50% of its parameters. When (ii) using M2F head, our grouping-basse representations consistently improve performance across variants: +1 mloU w.r.t. M2F + Swin, and +2.7 mloU w. M2F + NAT. Overall, (iii) our backbone adds a modest 5- 5-10% increase parameters and FL.OPs over standard backbones. While combining with M2F slightly increases computational costs over NAT, this cost is effectively amortized in the native setup where the segmentation head removed, making the overall approach more parameter- and FLOP-efficient.
in our
4.2.2 Panoptic Segmentation
Setting. We train and evaluate models on COCO-panoptic [21], which consists of 80 object (things) and 53 background (stuff) classes, requiring models to predict semantic classes and instance IDs for things. Our models are trained for 50 epochs, using M2F's original hyperparameters for integrated models. For our native results, we use the same hyperparameters as in semantic segmentation.
Discussion, We observe in Table 2b: (i) our tiny native results (49.2 PQ) outperform MaskFormer w/Swin-T (47.7 PQ) by a sizeable margin, despite fewer parameters (32M v.s. 42M). This trend is consistent across different model sizes, with Table 2a. (ii) M2F + NAT-T backbone (54.3 PQ) outperforms our barebone native masks, however, our SeNaTra-T M2F (55 PQ) achieves top performance, and further improves with a larger backbone (SeNaTra-L, 58.1 PQ). Overall, our native
results surpass consolidated baselines, and our backbone enhances state-of oftte-o-ttee-ee the-art when paired with dedicated segmentation heads.

---

S2
ADE20K
ZS-VOC
ADE20K
ZS-VOC
Baseline
x
x
x
41.3
40.1
SeNaTra
49.7
57.3
SeNaTra
x
x
47.2
51.9
Absolute pos. encoding
48.8 (-0.9)
56.7 (-0.6)
x
/
48.7
54.2
GRU instead skip
44.9 (-4.8)
55.0 (-2.3)
(local)
47.3
55.8
Embeeding group init.
47.2 (-2.5) 54.1 (3.2)
49.7
57.3
Prev. three (Slot Attn. (15))
43.6 (-6.1) 52.3 (-5.0)
(a) Impact of grouping at each backbone stage.
(b) Low-level design choices in our grouping layer.
Table 3: Architecture-level ablations. We report native masks mloU on ADE20k and Sero-Shot(ZS) mloU on Pascal VOC. In (a), evaluate the effect of replacing grouping layers with uniform downsampling at each stage. In (b), we study low-level design decisions inside our grouping layer.
4.3 Ablation Studies
Grouping at different backbone stages. Table 3a compares our spatial grouping layer to uniform downsampling with strided convolution (as in NAT [3], without grouping) over each backbone stage (SI, S2, $3). The Baseline underperforms compared to our approach, in both supervised (41.3 mloU, -8.4) and zero-shot (40.1 mloU, 17. settings. Instead of learned pixel assignments, this approach relies on bilinear interpolation to predict high-resolution masks from coarse stride 32 feature maps. Moreover, we observe that introducing grouping spatial layers across stages increases performance monotonically. Local grouping in the last stage significantly decreases performance in both metrics. Our design enables whole-image masks by leveraging efficient local grouping in early stages.
Grouping layer design. Table 3b compares our grouping layer design (Section 3.1) relative to slot attention [15]. Replacing the GRU with skip connections yields an improvement of +4.8 mloU, In practice, we observed that it addressed numerical instabilities during ImageNet pretraining and reduced memory requirements. Similarly, sampling initial embeddings from a learned Gaussian distribution, as [15], also compromised stability. Using learnable embeddings for initialization, as in [66], still drops performance by 2.5/3.2 mloU. Further using relative positional encodings yields an additional mloU. Altogether, these yield significant improvements 6. 6.1/5.0 mloU on ADE20k and ZS- VOC, respectively, while enhancing training stability and memory footprint.
Segmentation paradigms. In Table 4, we ab- late: (i) backbone choice (ours with native seg- mentation capabilities vs. baseline [3]), and (ii) two key Mask2Former components: pixel decoder for multi-scale feature fusion, and Transformer decoder for producing mask embed- dings. In the first two rows, we compare NAT (w/o grouping) with ours, without any additional components. Our baseline fails at this task (PQ 15.9, row 2) and underperforms in semantic seg- mentation (-8.4 mloU). Adding a pixel decoder (MSDeformAttn from Faskerrrmer, rows 3&:4) minimally impacts our approach, but significantly improves NAT baseline (+6.4 mloU). Finally, rows 5&6 show that segmentation decoder is crucial for NAT to segment instances (54.3 mloU), and benefits semantic segmentation (+1.7 mloU). Dedicated decoder also benefits our approach in terms of panoptic segmentation (55.0 PQ, +5.8 PQ), showing potential for improvement.
Pix Dec.
Dec.
mloU
PQ
SeNaTra Notive
49.7
49.2
w/o grouping
41.3
15.9
ScNaTra
49.7
48.8
w/o grouping
47.4
17.3
SeNaTra M2F
51.3
55.0
w/o grouping
49.1
54.3
Table 4: Segmentation paradigms. We ablate adding Pixel Decoder and Transformer Decoder on ADE20k (mloU) and COCO-Panoptic (PQ).
Conclusions
This work introduces a novel architecture particularly suited for segmentation tasks centered around our proposed spatial grouping layer. Our design offers significant methodological advantages over prior art, being fully differentiable with strong inductive bias and scalable to large input resolutions. Through empirical results, we demonstrated the emergence of meaningful segments without explicit mask supervision and a streamlined paradigm for downstream segmentation. Our work shows that segmentation fundamental perception task-can be inherently encoded in a model's internal representations rather than delegated to specialized decoder modules, opening new directions in segmentation-centie backbone architectures.
10

---

References
[1] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[2] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In CVPR, 2023.
[3] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. CVPR, 2023.
[4] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, (14::541-551, 1989.
[5] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask R-CNN, In ICCV, 2017.
[6] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022.
[7] Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[8] Shihua Huang, Zhichao Lu, Ran Cheng, and Cheng He. FaPN: Feature- aligned pyramid network for dense image prediction. In ICCV, 2021.
[9] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and X. Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, 2022.
[10] James C Liang, Yiming Cui, Qifan Wang, Tong Geng, Wenguan Wang, and Dongfang Liu. Clusterformer: Clustering as a universal visual learner. In Ady. Neural Inform. Process. Syst., 2023.
[11] Tsung- Wei Ke, Sangwoo Mo, and X Yu Stella. Learning hierarchical image segmentation for recognition and by recognition. In ICLR, 2023.
[12] Wang Zeng, Sheng Jin, Lumin Xu, Wentao Liu, Chen Qian, Wanli Ouyang, Ping Luo, and Xiaogang Wang. Teformer: Visual recognition via token clustering transformer. IEEE Trans. Puttern Anal. Mach. Intell., 2024.
[13] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification all you need for semantic segmentation. In Adv. Neural Inform. Process. Syst., 2021.
[14] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua. and Sabine Süsstrunk. Slic superpixels compared to state-of-the-an superpixel methods. IEEE TPAMI, 34(11):2274-2282, 2012.
[15] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Adv. Neural Inform. Process. Syst., 2020.
[16] Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. IEEE TPAMI. 2465):603-619, 2002.
[17] Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Efficient graph-based image segmentation. IJCV, 5972):177-181, 2004.
[18] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE TPAMI, 33(5):898-9)) 9968,991,12100 2011.
[19] Michael Van den Bergh, Xavier Boix, Gemma Roig, Benjamin de Capitani, and Luc Van Gool. Seeds: Superpixels extracted via energy-driven sampling. In ECCV, 2012.
[20] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019.
11

---

[21] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR, 2019.
[22] Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 36(4):193-202, 1980.
[23] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE,
1998.
[24] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[25] Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022.
[26] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything images and videos. arXiv preprint vrXiv:2408.00714 2024.
[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
[28] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and compuler-assisted intervention, 2015.
[29] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik, Rich feature hierarchies for accurate object detection and semantic segmentation. CVPR, 2014.
[30] Ross Girshick. Fast r-cnn. In ICCV, pages 1440-1448, 2015.
[31] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[32] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, 2023.
[33] Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer. Hiera: A hierarchical Vision Transformer without the bells- and-whistles. In International Conference on Machine Learning (ICML), 2023.
[34] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE TPAMI, 22(8):888-905, 2000.
[35] Pablo Arbeláe Jordi Pont-Tuset, Jon Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping. CVPR, 2014.
[36] Varun Jampani, Deqing Sun, Ming-Yo Yu Liu, Ming-Hsuan Hsuan Yang, and Jan Kautz. Superpixel samping networks. In ECCV, 2018.
[37] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In ICLR, 2023.
[38] S. P. Lloyd. Least squares quantization in pem. IEFE Transactions on Information Theory, 28(2):129-133) 137, 1957.
12

---

[39] J. B. MacQueen. Some methods for classification and analysis of multivariate observations. Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability. 1:281-297, 1967.
[40] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV. Springer, 2018.
[41] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 115(3):21 1-222, 2015.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn., 2021.
[43] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Int. Conf. Mach. Learn., 2021.
[44] Muyang Yi, Quan Cui, Hao Wu, Cheng Yang. Osamu Yoshie, and Hongtao Lu. A simple framework for text-supervised semantic segmentation. In CVPR, 2023.
[45] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Assoc. for Comp. Ling.. 2018.
[46] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Push- ing web-scale image-text pre-training to recognize long-tail visual concepts. arXiv preprint arXiv:2102.088 08981, 2021.
[47] Karan Desai and Justin Johnson. Redcaps: Web-crawled image-text data created by the people, for the people. In Adv. Neural Inform. Process. Syst., 2021.
[48] Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In CVPR, 2023.
[49] M. Everingham, L. Van Gool, C.K.I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (VOC) challenge. IJCV, 88(2):303-338, 2010.
[50] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam- Nam-Gyu Cho, Scong-Whaid Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation the wild. In CVPR, 2014.
[51] T.Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[52] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018.
[53] M. Cordts, M. Omran, Ramos, Rehfeld, M. Enzweiler, R. Benenson, U. Franke, Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[54] Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang. Open- world semantic segmentation via contrasting and clustering vision-language embedding. In ECCV, 2022.
[55] Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segelip: Patch aggregation with learnable centers for open- open-vocabulary semantic segmentation. In Int. Conf. Mach. Learn., 2023.
[56] Ji-Jia Wu, Andy Chia-Hao Chang, Chieh- Yu Chuang, Chun-Pei Chen, Yu-Lun Liu, Min-Hung Chen, Hou-Ning Hu, Yung- Yu Chuang, and Yen-Ye Yu Lin. Image-text co-decomposition for text-supervised semantic segmentation. In CVPR, 2024.
13

---

[57] Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiaodan Liang. Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency. In ICLR, 2023.
[58] Yun Xing, Jian Kang, Aoran Xiao, Jiahao Nie, Shao Ling, and Shijian Lu. Rewrite caption semantics: Bridging semantic gaps for language-supervisg semantic segmentation. In NeurIPS, 2023.
[59] Fei Zhang, Tianfei Zhou, Boyang Li, Hao He, Chaofan Ma, Tianjiao Zhang, Jiangehao Yao, Ya Zhang, and Yanfeng Wang. Uncovering prototypical knowledge for weakly open-vocabulary semantic segmentation. In NeurlPS, 2023.
[60] Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected erfs with gaussian edge potentials. In Adv. Neural Inform. Process. Syst., 2011.
[61] Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In CVPR, 2020.
[62] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye. Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. In A. Globerson, L.. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 103031-103063 Curran Associates, Inc., 2024.
[63] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pages 24185-24198, 2024.
[64] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollar. Panoptic feature pyramid networks. In CVPR, June 2019.
[65] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7262-7272, October 2021.
[66] Ondrej Biza, Sjoerd Van Steenkiste, Mehdi S. M. Sajjadi, Gamaleldin F. Elsayed, Aravindh Mahendran, and Thomas Kipf. Invariant slot attention: object discovery with slot-centric reference frames. In Int. Conf. Mach. Learn., 2023.
[67] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. ECCV, 2024.
[68] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position rep- resentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languuge Technologies, Volume (Short Pupers), pages 464-468, 2018
[69] Hangbo Su, Zhi Yang, Zhiwei Gao, Nannan Zheng, Song Bai, Wei Tan, Huan Li, Chao Qian, and Jianlong Fu. Roformer: Enhanced transformer with rotary position embedding. ICLR, 2021.
[70] Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In ECCV, 2024.
[71] Thang Dao, Han Hu, Hyungwoo Kwon, Xuehai Zhang, Lisha Song, Dmitriy Vasilenko, Yi Gu, Xinyu Yang. Khac Day Nguyen, and Kyunghyun Lee. Flashattention: Fast and memory-efficient exact attention with io-awareness. NeurlPS, 2022.
[72] Ali Hassani, Wen mei Hwu, and Humphrey Shi. Faster neighborhood attention: Reducing the o(n^2) cost of self attention at the threadblock level. In Adv. Neural Inform. Process. Syst., 2024.
14

---

[73] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training aata-efficient image transformers; distillation through attention. In Int. Conf. Mach. Learn., 2021.
[74] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), pages 646-661. Springer, 2016.
15

---

Appendix
Overview. We structure the appendix follows: in Appendix A and Appendix B, we provide general discussion and broader impact statement, respectively. In Appendix C we show visualizations of our model's learned hierarchical decompositions and final segmentation masks for image-text supervision. In Appendix D we provide extensive implementation details and additional experi- mental results. Lastly, in Appendix E we discuss low-level details on our spatial grouping layer's implementation and its runtime and compute considerations.
A Discussion
SeNaTra introduces a new family of backbone architectures enabling native segmentation through spatial grouping layers. As demonstrated in our experiments, our approach outperforms strong baselines and previous grouping- based works both in the purely native setting (including zero-shot), as well as with additional segmentation heads. Despite its promising results, SeNaTra has limitations. While our model scales approximately linearly with respect to input resolution (see Appendix E.2), grouping layers introduce computational overhead compared their de-facto counterpart, strided convolutions, given the lightweight nature of the latter. This added complexity is largely amortized when leveraging native segmentation capabilities but remains a limitation when integrating our model with external heads. As explained in Appendix E.I, we provide efficient CUDA-based implementation, however, there remains room for improvements both in terms of low-level CUDA optimizations and general module design. Another consideration is that while our native results perform favorably in both semantic and panoptic segmentation, our model yields larger gains in semantic segmentation. A plausible explanation lies in biases acquired during ImageNet pre-training. During this stage, our model is not incentivized to separate different instances of the same class in grouping layers, but rather to focus overall semantics. Therefore, the adaptation needed for a pre-trained model to transfer knowledge through grouping layers for semantic segmentation is likely smaller than that required for panoptic segmentation, where instance separation is required. Throughout our experiments, we focused on using off-the-shelf pre-training recipes to highlight advantages induced solely by our architectural design. However, we believe there are multiple exciting opportunities for future work to address this observation and design object-oriented pre-training schemes, such as work focusing on visual grounding [67].
B Broader Impact Statement
Our work proposes new vision backbone architecture with applications mainly in the field of segmentation. Given the broad scope and potential applications of this task, and computer vision systems overall, our model inherits both opportunities and challenges common this field. Like any general-purpose data-driven model, it may exhibit biases present in training data and could potentially be adapted for concerning applications. However, some of our empirical results demonstrate improved parameter and compute efficiency compared to existing solutions, as well as increased data efficiency. These properties could enable positive impact in coourre-constrainee settings where access to large datasets or computational resources is limited, such as applications in life sciences.
C Zero-Shot Qualitative Results
In Figure 4, we show both per-stage groups as well as final predicted semantic masks for SeNaTra- pretrained on image-text pairs, i.e., CC3M and CC12M datasets. The results are obtained on validation images of the PASCAL VOC dataset [49]. As with class-supervised models, we observe a hierarchy of boundary pereddrry groups across stages. We notice that our model's final groups have a larger tendency towards oversegmenting objects and regions. This can be explained due to the richer and denser semantic content present in text embeddings. which may benefit from a more granular visual representation. Remarkably, by querying final tokens with text embeddings we obtain high-quality semantic masks (column 5), suggesting that our model's 's pixel partitions carry semantic awareness.
16

---

Input Image Stage 2 Groups Stage Groups Final Stage GroupsPredicted MasksGround Truth Masks Figure 4: Qualitative zero-shot segmentation learned from image-text contrastive pre-training. We visualize hierarchical final decompositions along with their predicted semantic masks, obtained in zero-shot setting Pascal VOC validation images [49], and corresponding ground truth masks. Note that these models did not receive any form mask supervision during training, and were trained with standard contrastive objective on image-text pairs. Final masks are obtained without any form of heuristic postprocessing.
D Additional Results and Implementation Details
Overview.
the following, we provide additional results, and implementation details for each of the experimental setups described in the main paper. Note that our code and pre-trained models will be made publicly available.
Model variants. As explained in Section 4.1, we present results with three different model variants, each corresponding to an increased parameter count. Table 5, we specify the configuration each variant, including (i) number of transformer encoder layers, i.e., blocks consisting of self-attention
17

---

# layers
MLP ratio
# params
S
3,4, 18,
512
3
29M
SeNaTra-B
3, 4, 18,5
1024
94M
ScNaTra-L 3,4, 18,
1536
21
Table 5: Model variants. We summarize the model configuration of each of our backbone variants: number of transformer encoder layers used at each stage layers), output token dimension (dim), and MLP hidden dimension ratio in transformer encoder layers (MLP ratio).
followed by an MLP with residual connections, (ii) final embedding dimension, and (iii) MLP ratio, i.., hidden dimension of the aforementioned MLPs in transformer encoder layers. The overall configuration used for each model variant is borrowed from [3], with the exception of large, which was not presented in its original work. We choose the design of [3], due to its strong baseline performance among existing networks in both classification and dense prediction.
Rotary Positional Encodings. In the original work of [3], the self-attention layers used at each backbone stage were implemented with relative positional encodings [68]. Recently, Rotary Positional encodings (RoPe) [69, 70J have gained popularity, due to their ability to encode pairwise positional information in principled way and without the need for explicit access to the self-attention matrix. This can be largely beneficial in terms of runtime and memory consumption, it allows the use of fused implementations of self-attention [71, 72], resulting in significant runtime improvements. For the sake of efficiency,
replace the relative position biases in self-attention layers originally used by [3] with RoPe, and leverage the recently proposed fused kernels from [72]. Empirically, we observe a negligible decrease of downstream classification and segmentation performance, and significant increase in speed. Particularly on newer hardware, e.g., A100s, this change results in an approx 30% decrease runtime, helping offset the overhead introduced by our spatial grouping layers.
D.1 ImageNet Classification
ImageNet-Ik Ik pre-training: imple- mentation details. We train our mod- els from scratch for 300 epochs at res- olution 224 x 224, following all train- ing hyperparameteres, including op- timizer, learning rate scheduler, and augmentation setting of [1]. How- ever, we disable MixUp augmenta- tion as degrades our results. The drop is likely caused by the ambigu- ity that alpha composite images in- troduce in our grouping layer. Un- like [3], we do not train for additional cooldown epochs. Following [1, 3], we use stochastic depth for regulariza- tion [74], with default survival prob- abilities of 0.3 and 0.5 for our tiny and base variants, respectively. Train- ings take approximately hours on 8 A100 GPUs
method
image size #params. FLOPs
top-1 acc.
ImageNet- Ik trained models
VIT-B/16 [24]
384
86M
55.4G
77.9
DeiT-S [73]
2242
22M
4.6G
79,8
DeiT-B B [73]
2247
86M
17.5G
81.8
ClusterFormer-T [10]
224
28M
81.3
Swin-T [1]
224
29M
4.5G
81.3
ConvNeXi-T [25]
2242
28M
4.5G
81.3
TCFormerV2-S [25]
2242
26M
4.5G
NAT-T [3]
2242
28M
4.5G
83.3
SeNa (Ours)
2242
29M
4.9G
Swin-B B [1]
2247
88M
15.4G
83.5
ConvNeXt-B [25]
2242
89M
15.4G
83.8
NAT-B SeNa -B (Ours)
2242
90M
13.7G
84.3
2242
90M
14.9G
84.0
ImageNNtt-224 pre-trained models
V/16 [24]
3842
86M
55.4G
84.0
Swin-B B [1]
3842
88M
47.0G
86.4
ConvNeXi-B [2]
3842
89M
45.1G
86.8
SeNaTra -B (Ours)
3842
94M
47.9G
86.9
ImageNet-222 pre-tnainin:: imple- Following 25], we pre-train our larger model variants for 90 epochs on the larger- scale ImageNet-22k dataset, which consists of approximately 16M im- ages labeled over 22k classes. In this setup, we reduce stochastic depth probabilities to 0.2, following [1]. We follow prior art [1, 25] and fine-tune these models for 30 additional epochs on the ImageNet- dataset at 384 384 resolution and report top-1 accuracy
VIT-L/16 [24]
3842
307M
190.7G
mentation details.
Swin-L [1]
3842
197M
103.9G
87.3
ConvNeXt-L [2]
3842
198M
101.0G
SeNaTra -I. (Ours)
3842
211M
107.3G
87.3
Table 6: Image classification on ImageNet-1k and -22k.
We compare various standard and grouping- based backbones both trained from scratch on 1 and pre-trained on 22k.
18

---

on SmaaeNeet-1k-vall As in the previous setup, we follow the training recipe and parameters of [1], both for pre-training and fine-tuning, with the exception of Mix augmentation. Pre-training runs are conducted on 16 A A100 GPUs for approximately a week, and fine-tuning on ImageNet-I6 1k takes approximately hours on 8 A100s.
Discussion. In Table 6, we provide quantitative results of our model against slate-of-tee-ue methods following comparable supervised setup. As noted in the main text, our approach performs on par with sate-of-the-art, while enabling the emergence of strong pixel-level localization properties, as it can be observed qualitatively in Figure 3.
D.2 Image-Tert Text Pretraining
Pre-training: implementation details. As explained in the main paper, we adhere the training setup of [44]. We note, however, that [44] open-sourced its evaluation code, but did not provide scripts nor instructions to train its model. Our attempts at reproducing their reported results lag 1- mloU behind the reported numbers. Following the hyperparameters specified in their original paper, we train for 20 epochs with initial learning rate 3 x 10 and a use cosine decay scheduler leading to minimum learning rate of 3 x 10-5, We apply linear learning rate warmup during the first 3k iterations, and train for a total of 20 epochs with batch size of 4096 (approx. 68k iterations). As with ImageNet models, we use stochastic depth for regularization [74], with survival probabilities set to 0.2 and 0.3 for our tiny and base model, respectively. For our configuration using additional data from RedCaps 1MM, we keep all hyperparameters identical, and train for 20 epochs on the union of all datasets (CC3M, CC12M, and RedCaps 12M), leading to a total of 126A iterations. Trainings are conducted on 16 A100 GPUs, lasting approximately two days for default configurations, around four days when using the RedCaps 2M dataset.
Backbone-level comparison. As ex- plained, we follow the experimental setup described [44] and replace its backbone (VIT. [24]) with our proposed SeNaTra backbone. We highlight the impact of this change in Table 7. We note that SimSeg [44] produces coarse patch-class activa- tions and relies on postprocessing with Conditional Random Fields [60] to obtain pixel-precise masks. Our method does not require such heuristic postprocessing and instead utilizes our upsampling operations (Section 3.2) to produce pixel-level output. For base, our approach yields 3.5 avg. mIoU increase, which further increases to 6.7 mloU when comparing postprocessing-Tin free outputs.
Backbone
CRF
VOC
Context
COCO-Obj.
Avg.
VIT-S
x
53.8
23.5
25.7
34.3
56.4
25.8
27.2
36.5
SeNaTra -T
x
57.3
27.3
29.7
38.1
VIT-B
x
53.1
23.3
27.4
34.6
V
57.4
26.2
29.7
37.8
SeNaTra-B -B
X
61.3
30.2
32.6
41.4
Table 7: Backbone comparison for text-supervised zero-shot segmentation. Our approach significantly outperforms SimSeg [44] trained with a ViT backbone, without relying on CRF post-processing.
Zero-shot Segmentation Inference. As explained in the main text, we quantitatively evaluate the performance of our image-text pre-trained models on zero-shot semantic segmentation. Our inference details are the following: given a dataset with C. target classes, we obtain C corresponding text embeddings by feeding template prompts as "An image of [CLASS )" ttrouuhh our text encoder (we use the original templates of [44]). We then compute the dot-product similarity between each of our Nout final projected image tokens and target class embedding. By repeating this process over all C' text embeddings, we obtain an Noai C unnormalized similarity map. which we upsample to input resolution, i..., patch-level, with the transition matrices described in Section 3.2. By applying an ar argmax over classes, we obtain final class prediction for each input patch. For datasets with an additional backgroumd class (Pascal VOC [49], Pascal Context [50] and COCO-obj [51]), we need to set a threshold for mask values. To do so, we use the original method of [44], consisting of computing the image-level-texx embedding similarity of the top-k classes in the dataset, and computing the mean value with an additional standard deviation. For the remaining datasets, we just apply pixel-wise argmax over all classes. Lastly, we obtain a small performance boost by computing overall class similarities as the average of image-level similarities (after pooling), and the maximum spatial similarity over final tokens, i.e., max similarity over our final groups
19

---

D.3 Native Segmentation Models with Mask Supervision
implementation details. As explained in the main paper, our native segmentation for semantic segmentation obtains masks by feeding final token embeddings through 2-layer MI., followed by upsampling the results to input resolution with our learned upsampler. Our model is implemented with imseementation. For simplicity, we follow the default configuration of the original FCN [27], which further concatenates final token embeddings after feeding them through the MLPs, akin to skip connections. We have not explored this design thoroughly and better alternatives are likely possible. Further following the original configuration of [27]. during training add auxiliary loss at our penultimate stage, consisting of an additional MLP that's used analogously to the one one in our final layer. This auxiliary MLP is ignored at test-time. Models trained with the same training hyperparameters used by Swin Transformer used in conjunction with UperNet, with the only difference being an increased weight decay of 0.05, and shorter schedule of 80k iterations, instead of the original 160k. Trainings are conducted on 8 A100 GPUs for approximately 6 hours.
Semantic Segmentation:
implementation details. As explained in the main paper, main MLPs for panoptic segmentation targeting things and classes separately. The logic behind this division is to avoid penalizing oversegmentation errors for background regions. Therefore we use the MLP targetting stuff regions as in our semantic segmentation model and produce class predictions over each imput token individually. The MLP targetting things is only applied over the top 100 final tokens with largest assignment values in the last stage, representing potential object candidates. This MLP classifies tokens into either a things class label or no object. While it is possible to obtain a mask for each object candidate by directly using its corresponding input-level assignment, we find it beneficial to refine instance mask predictions by computing the dot-product between object candidates' final embeddings and our penultimate stage output embeddings, effectively recomputing the assignment matrix in our last grouping layer. In addition, before computing the dot-product, we linearly project final token embeddings and upsample them to the previous stage resolution, aiming to provide global context Stage features before re-computing the assignment. This procedure enables correcting common veesseggmnttttoon errors and introduces no significant overhead. We supervise the resulting masks and class predictions with the same bipartite matching loss as [6]. Lastly, during training, we find it beneficial to apply both MLPs and their corresponding losses to the intermediate outputs of our last stage (5 in total), akin to the intermediate losses used by [31, 13, 6]. At test-time, intermediate predictions are not used. Lastly, models are trained on 8 A 100s following the same configuration from our semantic models, but for longer schedule of 50 epochs as in Mask2Former [6]-and taking around 2.5 days.
Panoptic Segmentation:
two
E
Efficiency and Performance Considerations
E.1
Efficient Sparse Implementation
Naive implementation. In Algorithm 1, for clarity when describing our grouping layer, we depict a naive implementation with sparsity constraints. Formally, MiacE {0, N is defined as O for input-output edges that are enabled (see Figure 2), and -00, i.e., a large negative constant, for the rest. By being added to A in L4, right before applying sof tmft in I.5, entries set to - -co effectively become O. In this setup, dense grouping naturally corresponds to Mise containing zeros in all of its entries. This formulation accomodates both local and dense grouping, but is not the one we utilitze in practice.
Optimized implementation. The problem with the naive implementation described in Algorithm 1 that given N input tokens and Noout output, tokens, it requires storing a dense Nín Nout output matrix, which becomes impractical for input sets with large cardinality as the ones we process in the early layers of our backbone. To exploit sparsity and avoid computing non-zero entries in the cross- attention matrix, we leverage the sliding window attention CUDA kernels introduced in the natten [3] library. In Algorithm 2, we outline the high-level implementation of the sparse variant of the cross- attention and re-normalization operations described in Algorithm (L3-8, excluding the use of LN) using PyTorch. Abusing notation, we denote ( k = ((xiin), v = v(Xi), corresponding to linearly projected output and input token embeddings. The main idea of the the implementation is to repurpose the natten primitives na2d_ _qk, corresponding to query-key cross-attention multiplication, and na2d_ _v, corresponding to computing a weighted sum of values from the query-key matrix.
20

---

To do so, given our input feature ten- sor ' with spatial dimensions H x W, and an initial set of of output downsam- pled tokens Xour with spatial dimensions (H/2) (W/2), we first Unfold X into 2 x patches, resulting in a x (H/2) (W/2) tensor. Now by expanding, L...
Algorithm Efficient implementation of sparse Spatial
Grouping Layer cross-attention operation
Input: k, v 9_idx, T,
// Compute sparse cross-attention
attn na22__q(k,, q. B,3)
3: // Softmax over "groups"
4: attn + softmax(attn) т, dim -1] +
creating view with 'multiple copies', our target downsampled tokens X into 4 (H/2) x (W/2). and linearly projecting them described, we obtain a tensor repre- sentation in which applying sliding window cross-aftention yields expected group- ing operation over local windows. In Algo- rithm 2, further refer to gather opera-
5: // Reindex, reshape, renormalize over inputs
attn_q
q..
denom f- sum(attn_q, dims = (1,3))
8: attn_q f atta_q denoom
9: attn_q reshape(attn_q. attn.sshape.
10: // Aggregate updates
II: updates na2d_ avtaattt__...""
12: updates f sum(updates, dim = 1)
tions based on q_idx (L6). At a high-level, q_idx is an index tensor that enables rein- dexing the cross-attention matrix produced in L2 to enable computing the weighted mean from groups over imputs efficiently (corresponding to Algorithm 1, L6-8). The resulting sparse matrix attn_q can then be used as needed in na2d_av. These same ideas enable us to efficiently upsample and downsample feature maps with our resulting assignment matrices by leveraging the aforementioned primitives.
13: return updates, attn, attn_q
While it is possible to further optimize these operations by defining, fused kernels that merge the computation cross-attention, renormalization, and reindexing without storing intermediate tensors, our proposed implementation already provides a large improvement over naive implementations, or pure Py PyTorch-base operations based on Unfold, as we show in Table 9. Overall, while our current implementation fully optimal it enables the practical usage of our grouping layer within modern backbones on large resolution feature maps without exploding memory requirements. As already mentioned, we will release our code and models.
E.2 Runtime and Memory Analysis
Model
Seg. Head
FPS
Time (ms)
Mem. (GB)
mloU
NAT-B [3]
UperNet [40]
37.4
26.7
0.9
48.5
SeNaTra-B
Native
43.6
22.9
0.6
51.3
Table 8: End-to-end model performance and resource usage. We compare native masks against
the NAT baseline (with a UPerNet [40] decoder) in terms of throughput, latency, GPU memory, and final downstream mloU on ADE20k.
Setup. We evaluate the computational efficiency of our method on NVIDIA A100 GPU with 40GB of VRAM using batch size one and full FP32 precision for our base model variant across multi- ple input resolutions in Table 8, and at standard 512 x 512 for our Table 9.
Resolution 256
Impl.
FPS
Time (ms)
Mem. (GB)
None
72.9
13.7
0.4
Naive
49.7
20.1
0.6
CUDA
51.0
19.6
0.4
512
None
57.8
17.3
0.4
Naive
17.9
55.8
3.2
CUDA
44.7
22.4
compare three implementation approaches: con- ventional uniform downsampling (None, equivalent to our NAT [3] baseline), naive, pure PyTorch implementa- tion of our grouping layer (Naive), and our CUDA-optimized implementa- tion, leveraging natten (CUDA). Our CUDA-optinized spatial grouping en- ables practical deployment at high res- olutions- where the naive implementa- tion becomes prohibitively slow beyond
Discussion. In Table
768
None
36.7
27.2
0.5
Naive
6.8
146.8
14.8
CUDA
29.0
34.5
0.6
1024
None
23.3
43.0
0.7
Naive
0OM
CUDA
18.2
54.9
0.7
Table 9: Backhone-level throughput and resource us- age. We report FPS, per-image latency, and peak GPU memory for different input resolutions and grouping im- plementations. "OOM" indicates out-of-memory.
21

---

512 x 512 and runs out of memory on a 40GB GPU at 1024 x 1024 resolution. Critically, our local grouping design ensures that both memory consumption and runtime scale approx. linearly with imput resolution, making our approach practical for real-world applications. While our method introduces latency overhead ranging approx. 20 40% compared uniform downsampling baselines when used solely as feature extractor (Table 9), the overall cost of our grouping layers is amortized when considering end-to-end segmentation performance, as shown in Table 8: our native segmentation capability eliminates the need for heavy decoder heads, ultimately reducing overall latency while simultaneously improving segmentation quality (mloU), Moreover, the relative overhead of grouping layers is reduced at higher resolutions, due to the scalability of our approach, and the relatively smaller compute being dedicated to downsampling vs. the rest of the backbone. In future work, additional optimizations over our implementation and design could further bridge this gap.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Through extensive experiments, and qualitative examples, further extended in the appendix, we thoroughly validate all claims regarding our proposed architecture.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include a discussion section in Appendix A where we explicitly address limitations our approach. We further discuss limitations whenever appropriate in our main experimental section (Section 4) e.g. dedicated heads still bringing performance improvements over native results.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
23

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This work does not provide new theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes].
For each experimental configuration, we aimed provide many details as possible to enable the reproducibility of our approach in Appendix D, including detailed training hyperparameters, model configurations, and inference logic. Moreover, we provide pseudo-code of our efficient implementation in Appendix E. Upon acceptance, we will release all code and pre-trained models.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
24

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No].
Justification: All the datasets used by this paper are publicly available, but we are not releasing code at the time of submission. We are, however, committed to releasing code and trained models upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. wwt.cc/ppppppppuiieeeeee http:/uudes//eee/ https//oooSSeeessseeeeee asionooiccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new reproducible, they proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide extended descriptions of all configurations used for the results presented in our main experimental section (Section 4) in Appendix D. These include detailed training hyperparameters, model configurations, and datasets used, well as references to previous work with open-source code [1] from which we borrow experimental setups.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
25

---

Answer: [No]
Justification: Following prior work, we do not report error bars. Most of our experiments are in large-scale segmentation datasets, in which error bars are not commonly used in the literature. Moreover, we empirically observe little to no noise in our experimental results, and little variance among train runs, hence enror bars would offer limited value experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix D, we detail the hardware used in each experimental setup, together with the approximate compute time required.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics attpps//neeurr http://wwwrppp..cccpuil/tttiiiia hwtppp/bllc/..hiccccceeeeeeeeeee ines?
Answer: [Yes]
Justification:
We have read the code of ethics and believe our work adheres to it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
26

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include an explicit broader impacts section in Appendix B.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeetm of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA].
Justification: Segmentation tasks, which this paper focuses on, do not pose risks that can typically be addressed with safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-uss should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
27

---

Justification:
All datasets used in this work are public and we cite their original publications.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
28

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA].
Justification:
This work is unrelated to LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy petps://neurips: cc/ co//oonferencce/2225//////////// for what should or should not be described.
29


Reviews:
[{"review_id": "VbZzrZbN6Q", "reviewer": "Reviewer_7tgC", "summary": "This paper proposed native segmentation vision transformer. It uses the spatial grouping layer to replace uniform grid-based downsampling with learned dynamic assignment of visual tokens to semantically coherent groups based on image content. Successive grouping operations across backbone stages naturally compose into a mapping from input pixels to final tokens, effectively creating a multi-scale hierarchy of segmentation masks for tokens at each backbone stage. Experiments conducted on zero-shot segmentation tasks across multiple benchmarks demonstrate that the proposed model significantly outperforms prior art, including models trained on an order of magnitude larger datasets.", "questions": "1.\tThe grouping tokens are initialized via strided convolution. How impactful is this initialization approach compared to random or learned initializations as used in Slot Attention?\n2.\tHave the authors evaluated whether tokens from intermediate stages (e.g., stage 2 or 3) are semantically meaningful enough for classification or segmentation? Could this allow for controllable granularity or multi-scale segmentation?\n3.\tGiven that grouping tends to merge spatially close tokens, does the model struggle with small object segmentation?\n4.    How about the results compared to SAM2?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper introduces the concept of \"native segmentation,\" which generates high-quality segmentation masks solely through the backbone network without requiring additional segmentation heads. The spatial grouping layer is ingeniously designed, leveraging a differentiable iterative clustering algorithm, demonstrating strong innovation..\n2.\tThe paper is well-written and easy to follow.\n3.\tThe paper includes evaluations across classification, zero-shot, and supervised segmentation, alongside detailed ablations, which thoroughly validate the approach.\n4.\tThe method shows strong performance in both zero-shot and supervised segmentation with fewer parameters and FLOPs than many baselines.\t\n\nWeakness:\n\n1.\tThe grouping tokens are initialized via strided convolution. How impactful is this initialization approach compared to random or learned initializations as used in Slot Attention?\n2.\tHave the authors evaluated whether tokens from intermediate stages (e.g., stage 2 or 3) are semantically meaningful enough for classification or segmentation? Could this allow for controllable granularity or multi-scale segmentation?\n3.\tGiven that grouping tends to merge spatially close tokens, does the model struggle with small object segmentation?", "was_revised": true, "final_justification": "My concerns have been addressed.", "rebuttal": "We are thrilled that the reviewer finds our paper well-written, our method well-performing and thoroughly evaluated, and our grouping layer to be a “strong innovation” and “ingeniously designed”. We are grateful for detailed comments and respond to detailed questions below. \n\n**The reviewer asks how the initialization strategy of group tokens compares to random or learned initializations.**\n\nWe compare our initialization strategy to both randomly sampled embeddings (from Slot Attention [15]) and learned embeddings [63] in Section 4.3 and Table 3 (b). We empirically observed that random sampling led to training instability and divergence under standard learning rate configurations, and learned embeddings performed significantly worse (row #4 in Table 3 (b)). Our approach, utilizing strided convolutions, provides fast and input-dependent initialization with greater flexibility while remaining computationally efficient.\n\nWe will expand on the existing discussion in Section 4.3. in our manuscript\n\n**The reviewer asks how semantically meaningful the tokens from intermediate stages are, and asks about their potential to enable multi-granularity or multi-scale segmentation.**\n\nIn Figures 4 and 5 in the appendix, we observe that intermediate representations from stages 2 and 3 preserve semantic boundaries among objects and their parts, and can often correspond to fine-grained structures and small objects in the input (e.g., small fruits in Figure 4, row #1). However, due to the local grouping constraint applied in these stages, these intermediate tokens have limited receptive fields and therefore do not necessarily capture complete objects or semantic parts in their entirety.\n\nOverall, the segments from these intermediate stages provide boundary-preserving representations that facilitate progressive, hierarchical upsampling. As demonstrated in Table 3(a), employing grouping in intermediate layers significantly outperforms using a single dense grouping layer (row #2).\n\nFurthermore, the reviewer's insight regarding controllable granularity and multi-scale segmentation is well-founded. The hierarchical nature of our token representations is inherently suited for such applications. For example, in interactive annotation scenarios, intermediate-stage tokens could enable annotators to correct segmentation errors at various levels of detail, presenting a promising area for future research.\n\n**The reviewer asks how our method performs in the segmentation of small objects.** \n\nOur grouping algorithm incorporates both spatial proximity cues (through relative position encodings) and semantics via feature embeddings. Thus, it has all the necessary information needed to segment small objects. \n\nWe quantify this observation by computing Average Precision (AP) for our panoptic segmentation models separately over different object sizes (Large, Medium, and Small, as determined by the original COCO tools package), and report results compared to baselines for both native and Mask2Former (M2F) based results below:\n\n| Backbone | Head   | PQ   | AP   | AP$^{\\text{large}}$ | AP$^{\\text{medium}}$ | AP$^{\\text{small}}$ |\n|----------|--------|------|------|----------------------|-----------------------|----------------------|\n| Swin-T    | MF     | 47.7 | 33.6 | 55.8                 | 38.6                  | 15.2                 |\n| Ours-T     | Native | **49.2** | **38.2** | **60.3**                 | **42.1**                  | **18.2**                 |\n| Swin-T     | M2F    | 53.2 | 43.3 | 65.4                 | 47.0                  | 23.6                 |\n| Ours-T     | M2F    | **55.0** | **44.9** | **66.9**                 | **49.1**                  | **25.6**                 |\n\nAs shown in the table, our method demonstrates strong performance on small object segmentation, achieving absolute improvements of +3.0 AP$^{\\text{small}}$ in the native setting (18.2 vs 15.2) and +2.0 AP$^{\\text{small}}$ with M2F heads (25.6 vs 23.6) compared to Swin baselines.\n\nAs can be seen, our method outperforms significantly both methods in terms of AP$^{\\text{small}}$, showing its ability to segment objects of small scale. We will add these results to the appendix. \n\n**The reviewer asks how our method compares to the Segment Anything (SAM) model.**\n\nWe appreciate the reviewer's important question regarding the comparison with SAM and SAM2. We would like to clarify the distinct focus of our work. SAM and SAM2 introduce a comprehensive framework comprising three key components: the promptable segmentation task; the associated dataset and its generation engine; and a segmentation model, which employs a conventional ViT/Hiera backbone image encoder.\n\nIn contrast, our work focuses on the development of novel image segmentation backbone architectures that can replace conventional encoder/decoder networks for training image (segmentation) foundation models, such as SAM.\n\nOur model’s core contribution, the spatial grouping layer, allows segmentation masks to emerge directly from the backbone, enabling segmentation without the need for dedicated decoder heads. This native segmentation capability could streamline SAM's model architecture, potentially offering a more efficient decoder design. This may lead to better scaling properties and comparable performance while requiring fewer annotations, as demonstrated by our image-text pre-training results in Section 4.1.2; however, such scaling efforts are out of the scope of this paper.  We will elaborate on this distinction in the revised version of our paper."}, {"review_id": "JAIbzPStix", "reviewer": "Reviewer_dauE", "summary": "This paper presents native group layer, enables tracking pixels from high-resolution during downsampling. The method can be seamlessly integrated with several segmentation tasks. The experiments demonstrate its effectiveness.", "questions": "(1) The description of M_loc in the algorithm section is too brief, yet this mechanism is arguably the core feature that distinguishes this method from query-based approaches. Based on the limited information provided, is it correct to assume that it simply masks out all regions that fall outside of a predefined rectangular window?\n\n(2) Furthermore, since M_loc is non-learnable, this fixed masking rule might be unable to properly handle objects that span a very large area, such as a sky that occupies half of the screen. Does this local constraint pose a limitation for segmenting large-scale objects?", "limitations": "(1) Given the current trend where open-vocabulary image segmentation is increasingly becoming a sub-task for Large Language Models (LLMs) and Large Vision Models (LVMs), I am curious about the potential for the native transformer architecture to be integrated with state-of-the-art VLMs, such as LISA.\n\n(2) I understand that native segmentation appears to offer a different paradigm from the conventional segmentation head approach. However, I have reservations about its scaling potential. For instance, in Table 6, the model shows no performance advantage over Swin at the Large scale. Furthermore, Table 2 does not provide performance results for a \"Large + native\" configuration.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "(1) It is interesting regarding the trackable downsampling of the Group layer, which is a potential way to balance segmentation quality (high-resolution) and memory costs.\n\n(2) The paper is well-written, providing a comprehensive discussion with related works.", "was_revised": true, "final_justification": "Thanks for authors' further explanation, which addressed my concerns. I keep my original scores.", "rebuttal": "We thank the reviewer for their positive assessment of our work, including recognizing our method's ability to perform segmentation efficiently, its effectiveness demonstrated through experiments, and the clarity of our writing. We are pleased to address their questions and concerns below.\n\n**The reviewer points out that the description on $M_{\\text{loc}}$ in the algorithm section requires further elaboration.**\n\nWe appreciate the reviewer's request for further clarification on $M_{\\text{loc}}$. As the reviewer correctly points out, $M_{\\text{loc}}$ indicates that input tokens only cross-attend over group tokens over a pre-defined rectangular patch, which we refer to as local grouping.  We note that $M_{\\text{loc}}$ is used solely to illustrate that local grouping can be understood as a masking operation in the cross-attention-like operation introduced in Algorithm 1, line 3. In our implementation, as explained in Appendix E.1, $M_{\\text{loc}}$ is not explicitly used, and the dot-product is only computed for non-masked pairs.\nWe also note that we visually illustrate the behavior of $M_{\\text{loc}}$ in Figure 2 (b), and further expand on it in Section 3.1 “Local and Dense Grouping”. In our revised manuscript, we will update the text to clarify the explicit connection between $M_{\\text{loc}}$ further, as introduced in Algorithm 1, and the local grouping discussion to improve clarity.\n\n**The reviewer asks if $M_{\\text{loc}}$ comes with potential limitations for segmenting large-scale objects.**\n\nThe reviewer's question touches on a fundamental aspect of our grouping design. We would like to clarify that local grouping (formalized through $M_{\\text{loc}}$) is only applied in the two intermediate grouping layers of our architecture. In the last grouping layer, we perform global (or dense) grouping without masking. Therefore, final groups can span large-scale objects without any constraint. The distinct behavior of different grouping stages is extensively visualized in Figures 4 and 5. We observe that our model’s output segments often represent entire background regions, such as ‘sky,’ spanning large regions. The reviewer is correct that if $M_{\\text{loc}}$ were to be applied to our final grouping layer too, our model’s ability to segment large objects would have been compromised. \nIn Table 3 a), row #4, we ablate quantitatively the role of disabling $M_{\\text{loc}}$ (i.e., local grouping) in the last stage, and observe a drop of 2.5 mIoU on supervised fine-tuning on ADE20k, and 1.5 mIoU on zero-shot segmentation on Pascal VOC, confirming the expected behavior.\n\n**The reviewer asks how our method could be integrated into VLMs.**\n\nWe thank the reviewer for pointing out this exciting research direction. We believe SeNaTra is particularly well-suited to be integrated into VLMs, and highlight two exciting directions:\n- Our output group tokens could enable an improved learnable image tokenization scheme, where visual tokens are better aligned with semantically meaningful concepts, by explicitly representing regions, objects, and their parts. It could further enable efficiency gains by reducing the number of redundant tokens in background regions (i.e., utilizing a single token to represent ‘sky’, instead of multiple rectangular patches). \n- For models such as LISA, which explicitly aim to obtain segmentations and use explicit mask decoders, our proposed native segmentation architecture could enable a more efficient design that could significantly reduce the burden on such mask decoders.\n\nMoreover, based on the remarkable performance obtained for zero-shot segmentation under CLIP-style text-supervision in Section 4.1.2, and the success of CLIP-based models as vision encoders in the context of VLMs, we hypothesize that SeNaTra could be a great candidate to be integrated into VLMs for future work.\n\n**The reviewer asks about the potential of our method under parameter scaling.**\n\nWe appreciate the opportunity to clarify our parameter scaling results. Empirically, we observed that our model provides consistent improvements over Swin-Large; see Table 2. To be specific, in Table 2 (a), Ours-Large + M2F yields +0.6 mIoU for Semantic Segmentation, and in Table 2 (b), in the same setup, we observe +0.3 PQ. \nBelow, we further provide results for Ours-Large+Native for both semantic segmentation on ADE20k and panoptic segmentation on COCO. As in Table 2, we compare our native results against two strong baselines: Swin-L+UperNet and Swin-Large + MaskFormer. All models are pretrained on ImageNet-22k and evaluated on the same input resolution:\n\nSemantic Segmentation on ADE20k\n| Backbone | Head        | mIoU | Params |\n|----------|-------------|------|--------|\n| Swin-L     | UperNet     | 52.1 | 234M   |\n| Swin-L     | MaskFormer  | 54.1 | 212M   |\n| Ours-L     | Native      | **54.7** | 211M   |\n\nPanoptic Segmentation on COCO-Panoptic\n| Backbone | Head        | PQ   | Params |\n|----------|-------------|------|--------|\n| Swin-L     | MaskFormer  | 52.7 | 212M   |\n| Ours-L     | Native      | **53.5** | 211M   |\n\nThese results further demonstrate that our model consistently outperforms baselines under larger parameter regimes. We will include these comparisons in Table 2 in our revised manuscript."}, {"review_id": "j8Uz5GJvRl", "reviewer": "Reviewer_RCbm", "summary": "The paper challenges the current standard for spatial downsampling in vision backbones, replacing uniform downsampling with a learnable image-adaptive downsampling based on differentiable grouping. Thanks to this novel spatial grouping layer, super-pixel-like naturally arises in feature maps, enabling zero-shot segmentation tasks without additional task-specific heads or mask supervision. Being a general backbone, the proposed method can also be improved by leveraging mask supervision and segmentation heads, while not being strictly required.", "questions": "Coming back to the previously mentioned weakness, for example, having the backbone native segmentation capabilities, I would have liked to see if lighter segmentation heads (e.g. \"You Only Segment Once: Towards Real-Time Panoptic Segmentation\" or \"PEM: Prototype-based Efficient MaskFormer for Image Segmentation\".) could benefit more from this paradigm wrt the heavy M2F head. Since the backbone natively supports segmentation, I suppose the heads would need to do much less heavy-lifting.\n\nI wonder how this approach, leading to native segmentation arised thanks to architectural design, could be leveraged together with vision foundation models (e.g. DINO) where similar properties wrt dense predictions arise through training, considering that these models are usually ViT-based and non-hierarchical.", "limitations": "The only limitation, as also stated in the paper, is the additional overhead introduced by the grouping layer. Indeed, this is likely a minor limitation, since this overhead is negligible wrt the overhead introduced by task specific modules, not strictly needed anymore for this model.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- the paper is well written and easy to follow;\n- it challenges the current standard in vision, proposing a novel spatial downsampling technique;\n- segmentation naturally arises from the architecture, without mask-level supervision;\n- solid experimental section and ablation studies;\n- this new method can either improve existing segmentation pipelines or pave the way for new paradigms not using task-specific heads.\n\nWeakness:\n- the only minor weakness I can find is the limited set of competitors included in table 2, many others backbones and segmentation heads could be included.", "was_revised": true, "final_justification": "Amazing new architecture with lot of possible follow up works. It can be a groundbreaking innovation in segmentation.", "rebuttal": "We appreciate the reviewer’s thorough evaluation and positive feedback on our paper, methodology, and “comprehensive evaluation”. We address the reviewer’s questions below:\n\n**The reviewer asks for a comparison with additional lightweight segmentation heads.**\n\nThe reviewer notes that additional backbones and segmentation heads could strengthen Table 2 further. In its current version, we focused on representative models, including widely used transformer-based backbones (e.g., Swin) and established heads (e.g., Mask2Former). Below, we provide results on COCO-Panoptic with the additional head proposed: YOSO. For our implementation, we replace transposed convolution layers in the pixel decoder with our learned upsampling operators. Results are summarized in the table below, where we include our native and Mask2Former-based results as references. \n\n| Backbone | Head        | PQ   |\n|----------|-------------|------|\n| Ours-T   | Native      | 49.2 |\n| Ours-T   | YOSO        | 52.1 |\n| Ours-T   | Mask2Former | 55.0 |\n\n\nWe observe that YOSO partially bridges the gap between our native setup and Mask2Former. These results align with the reviewer’s intuition, showing that with our backbone, this lightweight head can perform favorably, providing middle-ground between the minimal native paradigm and the heavier Mask2Former head.\nWe commit to including additional heads (including PEM, as suggested by the reviewer) in Table 2 in our updated manuscript.\n\n**The reviewer asks about the potential of training our model using self-supervised objectives, such as those employed in DINO.**\n\nThe reviewer raises an excellent point about the synergy between our architectural design and self-supervised pre-training schemes. As the reviewer correctly highlighted, DINO (and DINOv2) typically use ViTs. We believe our hierarchical design could be adapted by: (1) introducing a global CLS token in our final stage, where we use global attention, and (2) applying DINOv2's objectives to our final group embeddings, as opposed to patch embeddings used in ViTs. We believe that these modifications could further enhance the emergence of semantically meaningful object-centric representations in our final group tokens.\n\nOverall, we agree that this represents a compelling direction for follow-up works, and could unlock large-scale pre-training of native segmentation architectures.\n\n\nLastly, we also thank the reviewer for noting the label overlap in Figure 5 of the supplementary material, which will be corrected in our updated manuscript."}, {"review_id": "9MO9RVfIh7", "reviewer": "Reviewer_Z1HJ", "summary": "This paper proposes a novel segmentation architecture Native Segmentation Vision Transformer (SeNaTra), which replaces traditional uniform downsampling in vision backbones with content-aware spatial grouping layers. The spatial grouping layer creates hierarchical segmentation that emerges naturally during feature extraction, eliminating the strict need for dedicated segmentation heads.", "questions": "See weakness", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. This paper is well-organized and easy to follow.\n2. The paper propose a novel architecture that replaces grid-based uniform downsampling with semantic-aware spatial grouping, intrinsically encoding segmentation capabilities within the backbone's hierarchical feature extraction without requiring dedicated segmentation heads.\n3. Significant outperformance on zero-shot segmentation tasks, even against models trained on much larger datasets.\n\n**Weaknesses**\n1. While the \"native\" model is effective, the paper's best results still rely on a powerful segmentation head (Mask2Former). This weakens the argument for decoder-independence and demonstrates that such heads remain essential for performance.\n2. The paper would be strengthened by a discussion and empirical comparison with ToMe [1].\n3. How sensitive is the model to the number of grouping iterations L, a key hyperparameter that was fixed at 3? An ablation study analyzing the performance vs. efficiency trade-off for varying L would be a valuable addition.\n\n\n[1] Token Merging: Your ViT But Faster, ICLR 2023", "was_revised": true, "final_justification": "The response of author have been addressed my all concerns", "rebuttal": "We thank the reviewer for their constructive feedback and overall positive assessment of our approach, highlighting its architectural novelty and presentation clarity. We address the reviewer's comments below. \n\n**The reviewer points out that a dedicated segmentation head is still needed to achieve state-of-the-art performance for downstream segmentation results.**\n\nThe reviewer correctly points out that our best performance on downstream segmentation tasks is still achieved by combining our backbone with Mask2Former (M2F). However, we would like to highlight the following points:\n- In  Table 2(a), we show that for Semantic Segmentation, the gap is already relatively small. For instance, Ours-T + native only lacks 1.6 mIoU behind Ours-T + M2F, despite a ~36% reduction in trainable parameters. \n- While in Table 2(b) we observe larger gaps between native and M2F-based results for our backbone for Panoptic Segmentation, we would like to highlight that M2F is a highly optimized model that has benefited from significant architectural contributions over its predecessor, MaskFormer. In contrast, in our work, we show a first iteration of our novel lightweight native paradigm, with streamlined design choices. Despite its minimalistic nature, it already outperforms a strong baseline such as MaskFormer, which we believe is a significant milestone.\n\nOverall, we aim for our results to show the potential of our native paradigm, and believe that with community adoption and similar future contributions, such as those applied over the original MaskFormer, native architectures will be able to bridge this gap.\n\n**The reviewer requests a discussion of ToMe**\n\nWe thank the reviewer for pointing out the \"Token Merging: Your ViT But Faster\" (ToMe) paper. We will include a dedicated discussion in our revised paper's related work section to clarify its relationship to our approach. We highlight the most important differences below:\n- Distinct Focus: *Segmentation vs. Classification Throughput*. ToMe primarily aims to increase throughput with minimal accuracy degradation for existing ViTs by heuristically merging tokens for image/video classification tasks, specifically showing results on ImageNet, K400, and AudioSet-2M. It does not report results for downstream segmentation, which is the key focus of our work. In contrast, our SeNaTra introduces a novel architectural paradigm based on differentiable, content-aware spatial grouping directly within hierarchical backbones, comprehensively evaluated on multiple downstream segmentation tasks (semantic and panoptic).\n- Architectural Integration for Native Segmentation. ToMe is a \"drop-in\" method that can be applied to existing ViTs without retraining, merging tokens in each block based on similarity. While efficient for throughput, it's an optimization applied to pre-existing architectures. Our work fundamentally replaces the uniform downsampling operation in hierarchical backbones with learned grouping layers. This architectural innovation enables \"native segmentation,\" meaning masks emerge directly from the backbone, offering a more intrinsically segmentation-aware representation, with benefits across zero-shot and downstream segmentation tasks.\n- Comparison Challenges. Directly comparing ImageNet classification results is difficult because ToMe primarily reports performance on pre-trained ViTs using mostly self-supervised recipes (including MAE fine-tuning), whereas our ImageNet pre-training for segmentation is standard classification-based. This prevents a truly \"apple-to-apple\" comparison in that domain. Moreover, as already mentioned, ToMe's focus is not on accuracy improvements but rather on throughput improvement, which is beyond our scope.\n\nOverall, while both approaches explore token reduction in vision transformers, their scope and primary objectives are orthogonal. ToMe optimizes throughput for classification in existing models while aiming for minimal accuracy drops, while our work focuses on a new architectural paradigm for native, data-efficient segmentation, offering foundational benefits for the development of future segmentation-centric models. We will include this discussion in the revised paper.\n\n**The reviewer asks about the impact of the number of grouping iterations.**\n\nThe reviewer correctly points out that the number of iterations for grouping was fixed at 3 during our experiments, and ablating this hyperparameter could be a valuable addition to our work. Below, we provide an experiment where we analyze the role of this parameter for both downstream Semantic Segmentation with fine-tuning on ADE20k, and zero-shot segmentation from text-supervision on PASCAL VOC, as in Table 3, and retrain models with a varying number of iterations. To study efficiency, we additionally report latency (in ms) during inference for a SeNaTra-B model using batch size 1 and on 512x512 images, measured on an NVIDIA A100 GPU (using fp32 precision). Results are summarized below:\n\n| \\# iterations | ADE20k (w/ fine-tuning) | Pascal VOC (zero-shot) | Latency (ms) |\n|--------------|-------------------------|------------------------|--------------|\n| 1            | 48.4                    | 55.5                   | 19.1         |\n| 2            | 49.5                    | 56.9                   | 20.8         |\n| 3            | 49.7                    | 57.3                   | 22.4         |\n\n\nOverall, we observe that our models obtain the highest segmentation accuracy when performing three grouping iterations. This indeed introduces a moderate overhead compared to fewer iterations; however, this parameter can always be adjusted for downstream applications that require a different tradeoff between segmentation accuracy and latency. We will expand on this this analysis in our final manuscript."}]

Author rebuttal:

Meta-review: Four reviewers recommend to accept (3 borderline) the paper. The main reasons for acceptance are the novel architecture for segmentation, the strong results (especially in zero-shot settings), and the potential for impact in research and practical applications. The main weaknesses are relatively minor and relate to the breadth of experimental comparisons, ablation studies, and some open questions about scaling and integration with other paradigms. The reviewers agree that the paper is well-written, innovative, and makes a valuable contribution.
On balance, the AC sees not basis to overturn the reviewer suggestions and recommends acceptance. The paper presents an interesting approach to native segmentation in vision transformers, with strong empirical validation and good potential for future impact. The AC highly recommends the authors to address the concerns of the reviewers and take into account their suggestions of improvement when preparing a revised version.
Decision: Accept (poster)