Object-Centric Refinement for Enhanced Zero-Shot Segmentation
Authors: Srinivasa Rao Nandam
Sara Atito
Zhenhua Feng
Josef Kittler
Muhammad Awais
Venue: ICLR 2026 Poster (2026)
Keywords: Zero-Shot Learning
Vision-Language models
Semantic Segmentation
Computer Vision
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose an object-centric framework to improve zero-shot segmentation.

Abstract: Zero-shot semantic segmentation aims to recognize, pixel-wise, unseen categories without annotated masks, typically by leveraging vision-language models such as CLIP. However, the patch representations obtained by the CLIP's vision encoder lack object-centric structure, making it difficult to localize coherent semantic regions.
This hinders the performance of the segmentation decoder, especially for unseen categories. To mitigate this issue, we propose object-centric zero-shot segmentation (OC-ZSS) that enhances patch representations using object-level information. 
To extract object features for patch refinement, we introduce self-supervision-guided object prompts into the encoder. These prompts attend to coarse object regions using attention masks derived from unsupervised clustering of features from a pretrained self-supervised~(SSL) model. Although these prompts offer a structured initialization of the object-level context, the extracted features remain coarse due to the unsupervised nature of clustering. To further refine the object features and effectively enrich patch representations, we develop a dual-stage Object Refinement Attention (ORA) module that iteratively updates both object and patch features through cross-attention. Last, to make the refinement more robust and sensitive to objects of varying spatial scales, we incorporate a lightweight granular attention mechanism that operates over multiple receptive fields. OC-ZSS achieves state-of-the-art performance on standard zero-shot segmentation benchmarks across inductive, transductive, and cross-domain settings.

Published as a conference paper at ICLR 2026
OBJECT-CENTRIC REFINEMENT FOR ENHANCED ZERO-SHOT SEGMENTATION
Srinivasa Rao Nandam' Sara Atito Zhenhua Feng² Josef Kittler² Muhammad Awais
Surrey Institute for People-Centred AI (PAT), University of Surrey, UK RCVSSP, University of Surrey, UK "School of Al and Computer Science, Jiangnan University, China {s nandam, sara.atito, j. kikttler, muhammad. awais @surrey.ac.uk
fengzhenhuaf siangnan.cdii cnu.cn
ABSTRACT
Zero-shot semantic segmentation aims to recognize, pixel-wise, unseen categories without annotated masks, typically by leveraging vision-language models such CLIP. However, the patch representations obtained by the CLIP's vision encoder lack object-centric structure, making it difficult to localize coherent semantic re- gions. This hinders the performance of the segmentation decoder, especially for unseen categories. To mitigate this issue, we propose object-centric zero-shot segmentation (OC-ZSS) that enhances patch representations using object-level information. To extract object features for patch refinement, we introduce self- supervision-guidder object prompts into the encoder. These prompts attend to coarse object regions using attention masks derived from unsupervised cluster- ing of features from pretrained self-supervised (SSL) model. Although these prompts offer structured initialization of the object-level context, the extracted features remain coarse due to the unsupervised nature of clustering. To further refine the object features and effectively enrich patch representations, we develop dual-stage Object Refinement Attention (ORA) module that iteratively updates both object and patch features through cross-attention. Last, to make the refine- ment more robust and sensitive objects of varying spatial scales, we incorporate a lightweight granular attention mechanism that operates over multiple receptive fields. OC-ZSS achieves state-of-the-ar performance on standard zero-shot seg- mentation benchmarks across inductive, transductive, and cross- cross-domain settings. Code is available at https: //gi thub. com/ confuploa//ccccc 2SS.
INTRODUCTION
Semantic segmentation aims assign semantic label every pixel in image (Xie al., 2021; Guo et al., 2022). While current methods perform well in closed-world settings, they rely on large- scale pixel-level annotations and struggle to generalize to unseen categories. Zero-shot segmentation (ZSS) addresses this limitation by training on seen classes and leveraging vision-language models such as CLIP (Radford et al., 2021) to generalize unseen ones. Although CLIP enables zero-shot classification by aligning global image and text embeddings, it lacks mechanisms for fine-grained alignment between visual regions and textual concepts, an essential capability for segmentation.
Recent ZSS methods (Zhou et al., 2023; 2022a; Li et al., 2024; Zhang et al., 2024; Ding et al., 2022) build on frozen CLIP backbones and focus on decoder-side improvements, including cross-attention modules (Zhou et al., 2023), cascading decoders (Li et al., 2024), and sinkhorn- based attention (Kim al., 2024b). However, these approaches overlook a fundamental limitation: CLIP's patch-level features lack object-centric structure and are less suitable be naturally grouped into semantically coherent regions (Jiao et al., 2023). This weak visual grounding impairs fine- grained localization and limits segmentation performance, limitation largely unaddressed by prior ZSS methods.
We argue that refining patch features to be more object-centric, without retraining the encoder, is ben- eficial for improving zero-shot segmentation. Methods that mine object features have shown promise in supervised (Xu et al., 2022a: Locatello et al., 2020; Zhang et al., 2023) and open- open- vocabulayy set- tings (e.g., OVSegmentor (Xu al., 2023)), typically using Slot Attention produce slot/group

---

Published as a conference paper at ICLR 2026
tokens (Locatello al., 2020). However, in the ZSS setting, the goal is not generate group to- kens but to align patches with text-based semantic classes, rendering these techniques not directly applicable. While CLIP-RC (Zhang et al., 2024) introduces region prompts fixed square grids, does not encourage object grouping and does not influence patch-level representations beyond simple feature fusion.
To address this, we propose Object-Centrie Zero Shot Segmentor (OC-ZSS), an object-centric zero- shot segmentor that refines patch representations using dynamically discovered object features. A central challenge is identifying object locations without supervision. We introduce a novel mech- ssupervision-guidee
object prompts: frozen prompts injected into the CLIP en- coder that attend to distinct object locations via masked attention. These masks are generated by clustering features from a pretrained DINO model (Caron al., 2021), allowing the prompts to capture coarse object features without requiring annotations or encoder finetuning. Unlike CLIP- RC (Zhang et al., 2024), which applies fixed region prompts, our approach adaptively captures semantically meaningful regions. Additionally, rather than open-vocabulary methods such as CLIP- DINOiser (Wysoczańska et al., 2024b) and ProxyCLIP (Lan et al., 2024), we do not mimic DINO features; DINO is used only to guide the attention of object prompts, which capture object features for refining CLIP patch tokens into more object-centric representations.
anism called
These object prompts provide initial but coarse representation of objects. Meanwhile, CLIP patch features remain object-agnostic. To improve patch features using object cues, we introduce dual-stage Object Refinement Attention (ORA) module that iteratively refines object and patch features via cross-attention. Unlike standard slot attention (Locatello et al., 2020), which focuses on mining object tokens, ORA enhances patch representations using updated object features while also refining the object features themselves. This mutual refinement promotes tighter semantic grouping and improves aligmment with textual concepts. Additionally, in contrast to OV-Segmentor (Xu et al., 2023), which requires end-to-end training of the vision-lane language stack and generates group tokens, our method operates entirely on frozen CL.IP backbones and directly improves patch-text alignment.
Standard cross cross-attention in ORA focuses on global context aggregation and lacks the ability to cap- ture objects varying spatial scales, which is crucial for effectively refining both patch and object features. While multi-scale context has traditionally been used in semantic segmentation decoders to improve dense predictions (Chen et al., 2018; Xie al., 2021; Guo et al., 2022), we incorporate it into the object and patch refinement process itself. Specifically, we propose granular attention mechanism within ORA that applies depthwise separable atrous convolutions with multiple dilation rates to extract features at different receptive fields. This enables ORA to effectively model object variability in scale and structure during iterative refinement, improving both patch enrichment and object representation.
Our main contributions are:
Refine CLIP patch features to be more object-centric, improving segmentation especially of unseen categories.
Introduce self-supervision-guuieeee object prompts to extract coarse object features from frozen encoder using unsupervised DINO clustering.
Design dual-stage ORA module that jointly refines object and patch features to enhance object-awareness in CLIP representations.
Incorporate granular attention mechanism into ORA capture multi-scale features and improve segmentation robustness.
OC-ZSS achieves state-of-the-art performance on standard zero-shot segmentation benchmarks, in- cluding inductive, transductive, and cross-domain settings, highlighting the value object-centric refinement in vision- languae models.
2 RELATED WORK
Zero-Shot Semantic Segmentation: Zero-shot semantic segmentation (ZS3) aims to segment ob- ject categories not seen during training by leveraging external knowledge, typically in the form of class descriptions or text embeddings. Early methods mapped image features and class seman- ties into shared space using word embeddings (Gu et al., 2020; Back al., 2021; Cheng al.,
2

---

Published as a conference paper at ICLR 2026
2021). More recent work adopts transductive setting (Pastore et al., 2021; Bucher et al., 2019), where unseen class names are available during training and pseudo-labes are used to boost per- formance through self-training. The advent of vision-language models like CLIP (Radford et al., 2021) has enabled more direct pixel-level alignment via text embeddings. Two-stage models such as ZegFormer (Ding al., 2022) and 7SSeg (Xu et al., 2022b) generate class-agnostic masks before classifying them using text, but incur redundancy due to repeated encoder passes.
segmentation with broader, multi-task recipe based on complex Mask2Former (Cheng et al.. 2022) architecture. MAFT (Jiao et al., 2023) fine-tunes the CLIP image encoder with mask- aware and distillation losses to improve proposal sen- sitivity in dual-stage methods. These heavier multi-stage systems motivated single-stage, parameter- efficient alternatives that align CL.IP features and text in single setting. MaskCLIP+(Zhou et al., 2022a) modifies CLIP's penultimate layer to better match textual embeddings. ZegCLIP (Zhou et al., 2023) and CLIP-RC (Zhang et al., 2024) introduce prompts or spatial tokens to adapt CLIP's vision encoder. Cascade-CLID (Li et al., 2024) proposes hierarchical decoders, while OT-Seg (Kim et al., 2024b) replaces standard attention with Sinkhorn attention. MVP-SEG+(Guo et al., 2023) adds orthogonality constraints to enhance part-level distinction. SPT-SEG (Xu et al., 2024b) in- troduces spectral prompts with a opectral-guider decoder. Unlike parameter efficient methods de- scribed earlier, a different direction taken by AlignZeg (Ge et al., 2024), which trains an auxiliary vision encoder to reduce the domain gap between image-level and pixel-level features. It also ap- plies prompt learning to adapt the CLIP vision encoder, and uses two decoders: a pixel decoder for class-agnostic mask proposals and semantic decoder built from CLIP's top layers, fusing these streams end to end. In contrast, we keep CLIP frozen and use single lightweight decoder, remain- ing parameter-efficient like prior decoder methods (Kim et al., 2024b; Xu et al., 2024b; Zhang et al., 2024). Rather than pursuing increasingly intricate decoders, we further differentiate by refining patch features through ORA with object-level cues derived from object prompts, which improves object localization and promotes better generalization to unseen classes.
FreeSeg (Qin et al.. 2023) targets unifeed/open-vocauule
Open-vocabulary semantic segmentation methods often pair CLIP with self-supervised models, pri- marily DINO (Caron et al., 2021) or DINOv2 (Oquab et al., 2024), to improve token grouping. FOSSIL (Barsellotti et al., 2024) retrieves synthetic visual references and aligns them to masks pro- duced from DINO features using OpenCut. Lazy Visual Grounding builds an affinity graph over DINO tokens and segments it with normalized cuts before assigning text labels (Kang & Cho, 2024). CLIP-DINOiser distills DINOs localization priors into CLIP, teaching CLIP to produce DINO-like local features (Wysoczańska et al., 2024b). Similarly, ProxyCLIP produces DINO-like grouping by forming proxy attention from correspondences computed by DINO, with adaptive normalization and masking (Lan et al., 2024). In contrast, we do not mimic or distill DINO: we introduce object-centric structure into CLIP features, using DINO only to guide object prompts, then use the object features from these prompts within Object Refinement Attention (ORA) to iteratively refine both patch and object features, yielding more generalisable, object-centric CLIP patch representations.
Several other works also explore semantic segmentation in more general open- open- vocabulary settings, where their training and evaluation protocols differ from the standard induetive/transddetive zero- shot segmentation setup. CLIP-DIY (Wysoczańska et al., 2024a) performs training-free open- vocabulary semantic segmentation by applying multi-scale dense CLIP inference guided by unsu- pervised object localization. SAM-CLIP (Wang et al., 2024) merges SAM and CLIP into a unified vision backbone via multi-task distillation on large-scale web data, and targets open- vocabulary seg- mentation. Shape-aware ZSS (Liu et al., 2023) promotes shape-aware segmentation by fusing CLIP- based predictions with eigensegments from self-supervised spectral decomposition. SCD (Qiu et al., 2025) fine-tunes CLIP ViTs with a Spatial Correlation Distillation framework to improve region- language alignment for open-vocabulary dense prediction. SimZSS (Stegmüller et al., 2025) aligns a text encoder to frozen vision-only backbone using imagecaption datasets such as COCO Cap- tions and LAION, and is evaluated under open-voue vocabulary segmentation protocols. In contrast, OC- ZSS is strictly studied under the reddsuiv//ranssuuctive zero-shot segmentation setting with explicit seen/unseen splits, as in standard ZSS methods such as OTSeg and CLIP-RC.
Object-Centric Representation and Slot Attention Slot Attention (Locatello et al., 2020) intro- duced an iterative attention mechanism for learning object-centric representations by binding latent slots to visual inputs. While effective for unsupervised object discovery, it suffers from issues such as slot collapse, addressed in later work using optimal transport-baan -based objectives (Zhang et al., 2023). The framework has also been extended to video segmentation: Guided Slot Attention (Lee et al.,
3

---

Published as a conference paper at ICLR 2026
photo of
P2
oegmennt
K Classes
NAK Text Embeddingg
Encoder
Badkgroundd
Retatorship Desemptor
MPRA
MPSA
GRT (PF Prwv, PP)
:PP PPev.
Decodor
Image Encoder
D3s. Token
Object Outouts
Object Refinemet Stage
Patch Refinmmnt
Crosa Albs Gean Gran.
Chuss Q. K.K.V)
Dr-
Image
SSL
Glastr
Maak
Stage
ters
OF_PPrer
+ PY_Prev
Encoder
Object
Patcn Refin
Object Refinement Altention
Refin.
PF
Figure I: Architecture overview of OC-ZSS. Object prompts (O), guided by an attention mask from frozen SSL encoder, are passed with learnable prompts (P) and the input image through an image encoder to produce patch (PF) and coarse object features (OF). Patch refinement is performed via a dual-stage ORA module with S iterations. Refined patches then passed to decoder for segmentation. Patch/object refinement with granular attention is shown on the right.
2024) enforces temporal consistency by propagating slots across frames, while Adaptive Slot Atten- tion (Fan al., 2024) adjusts the number of slots dynamically over time. Recent approaches like EAGLE (Kim et al., 2024a) and SOLV (Aydemir et al., 2023) further explore object-centric mod- eling by discovering spatial or temporal object masks through clustering or reconstruction-base objectives. In the open-vocabulary setting, OVSegmentor (Xu et al., 2023) applies Slot Attention within a trainable vision encoder to mine group tokens but does not refine patch-level features and requires full end-to-end training. In contrast, externally guided object prompts to enhance patch representations. We further introduce multi-scale refinement, direction not explored in prior slot-based methods.
method operates frozen CLIP features, using
METHODOLOGG
We propose OC-ZSS, an object-centric framework for zero-shot segmentation that improves CLIP patch representations by modeling and refining object-level structure within frozen backbone. Ex- isting zero-shot segmentation methods (Zhou et al., 2023; Zhang al., 2024; Kim et al., 2024b) typically focus aligning CLIP features to text embeddings via decoder-side attention, but over- look the fact that the underlying patch features lack object-centric grouping. While some methods apply auxiliary supervision like OTSeg (Kim et al., 2024b) or fuse with static region prompts like CLIP-RC (Zhang et al., 2024), they do not explicitly extract or refine object features, limiting their ability support anniigrieedd segmentation of unseen classes. To address this, introduce frozen selfssupervisionguudde object prompts that are injected into the CLIP encoder that attend to distinct object regions via masked attention. These attention masks constructed by clustering features from pretrained DINO model, allowing each prompt to focus on coarse object region without requiring labeled masks encoder finetuning.
However, these object prompts provide only coarse object features and do not directly influence the patch features used for segmentation. To refine both representations, we introduce the Object Refinement Attention (ORA) module, which performs iterative cross-attention between object and patch features. ORA gradually enhances object tokens and uses them enrich patch semanties. promoting tighter object-level grouping and improved alignment with text embeddings. Standard attention layers used refinement struggle to capture objects with varying spatial extents. To ad- dress this, we incorporate a granular attention mechanism into ORA by replacing linear projections with multi-scale feature extractors based on depthwise separable atrous convolutions (Chen et al., 2018). This enables both object and patch features to be refined in scale-aware manner, improving robustness to object size variation. An overview of OC-ZSS is shown in Figure 1. We describe each component in detail in the following subsections.
4

---

Published as a conference paper at ICLR 2026
3.1 PRELIMINARIES
CLIP (Radford et al., 2021) is used as the vision-language backbone, with a prompted vision en- coder and a multi-prompt Sinkhorn attention decoder for the task of zero-shot segmentation (Kim et al., 2024b). During training, the model is exposed only to a set of seen classes C., and evaluated on both seen and unseen categories the CLIP text encoder using K prompt templates, yielding T = Text Enc(C') IR xxNxx where N is the number of classes and D is the embedding dimension. The CLIP vision encoder produces outputs {g. P. H: the global class token 9, deep prompts P = {Po.... Pap). and patch embed- dings H = {hu.... ... h. Following (Kim et al., 2024b), a relationship descriptor for each class constructed as T concat(T 3 9. 7), which serves as the query input to a decoder composed of three Multi-Promy Sinkhorn Attention (MPSA) blocks, where MPSA uses multi-prompt Sinkhorn (MPS) attention (details in appendix), Each block performs:
C. Cu- Class-level
embeddings are obtained from
MPSA(Q,ER K, V) = MPS(QK)) VV,
An auxiliary prediction Y is also generated directly from the relationship descriptor T before the decoder and on better align patches with the final segmentation, where Upsample (bilinear interpolation) restores input resolution and Sigmoid is applied element-wise.
Upsample Siigmoid (MPS(TH")))
(2)
The final output during inference is the average of Y and Y. The total training loss is defined as:
Liot = Lang Y Y gt) + Leng (Y, Y gt);
where Caeg combines Dice and focal loss, and y denotes the ground-trutt segmentation map.
Input
Region P.
Object Prompts
Input Image
Figure 3: Region prompts attend to squares; object prompts focus dy- namic objects. Colored regions distin- guish areas.
Figure 2: Clustered image and patch locations used to gen- erate the attention mask. Clusters color-coded in both the image and mask; white indicates -0, and other colors indicate in the mask.
3.2 SELF-SUPERV ISIPPRRVISION-OO SIIDNGGDDDD OВJECT PROMPTS FOR CAPTURING RICH OBJECT FEATURES
Our framework is built on enhancing patch representations through object-level features. To enable object-based refinement, first need to capture plausible object features that provide an initial basis for further refinement. At each transformer layer of the CLIP vision encoder, the input is denoted as X {g' P H', O'), where g' is the class token, P are learnable prompts, H' are are frozen object prompts appended at that layer. O
patch embeddings, and O' = (0) "
The goal is to guide each object prompt to attend to distinct, semantically meaningful object location in the image. Prior work such as CLIP-RC (Zhang et al., 2024) has used region prompts to focus fixed square locations via attention. However, fixed grids cannot accurately capture the non- rigid and often irregular shapes of objects, which is essential for enabling object-centric refinement. Figure. shows difference between attention of region and object prompts.
To overcome this, generate custom attention mask Attn mask IR +1++ x (appeno that selectively enables each object prompt to focus only on a particular subset of patch tokens. Figure. shows the attention generation process. The mask is initialized zeros, and rows and
5

---

Published as a conference paper at ICLR 2026
columns corresponding to object prompts are first set -00 to block self-attention. To allow each object prompt to attend to its corresponding region, selected entries are set back to zero.
Constructing such masks for square regions is trivial. but capturing objects requires dynamic, non- rigid regions. To this end, we use a pretrained self-supervised model (DINO (Caron et al., 2021)) to generate object-aware patch groupings. The input image I is first patchified and passed through DINO to obtain patch-level features X These features are then clustered into no groups using Voronoi clustering (Aurenhammer, 1991), where implementation follows (Zhang et al., 2022) (de- tails in suppl.). Let C' = {el,..., CGo be the sets of patch indices assigned to each object prompts.
C= (ci (ci...., Gro = aaronoi_clustering X*)
The attention mask is updated for each object prompt j to allow attention to patch tokens in cy:
Attn sin__mask[] [ + np + M + j. + ap + = 0 E oj
(5)
Attn_mask[ (1 + np +
np + M 00 Ei
(6)
This attention masking strategy is applied across all selfaattennio layers in the encoder. As a result, each object prompt Oj learns to focus on its assigned region across layers, gradually capturing object-level information. The final set of object prompts from the last layer, OF = O², serves as coarse object features for our Object Refinement Attention (ORA) module.
3.3 OBJECT REFINEMENT ATTENTION MODULE
To refine patch features using the coarse object features extracted from object prompts, we propose a dtal-stage Object Refinement Attention (ORA) module, which also refines these object features themselves. This module operates iteratively, with each iteration consisting of two stages; object refinement followed by patch refinement.
In the object refinement stage, the initial object features OF are taken from the final layer of the CLIP encoder, i.e., OF = O", and refined based on the patch features PF H Proper initializa- tion of OF is critical, as random initialization has been shown to degrade performance in iterative refinement methods such as slot attention (Zhang et al., 2023). In our case, the object prompts intro- duced into the encoder not only serve extract object-specific features but also provide semantically meaningful initialization for OF, which is essential for stable refinement. This stage begins with a gross-attention operation where object features attend patch features, producing refined object features OF Ref:
OF Prev= = OF Ref
OR(OF, PF)
These features are then dynamically aggregated with the previous object features using a recurrent layer, specifically Gated Recurrent Unit (GRU) (Cho al., 2014). This recurrent update is crucial for stabilizing the refinement process over iterations by maintaining continuity in object representa- tions, as shown effective in prior iterative attention frameworks (Locatello et al., 2020).
OF GR Prev, OF Ref)
Next, in the patch refinement stage, the updated object features OF are used refine the patch features PF. This stage mirrors the object refinement step but inverts the attention direction. Patch features attend object tokens via a second cross-attention block:
PF; PF_Rei Cross_Attente AttPentiooo_RPPP,,, OF)
The refined patch features are again passed through a GRU yield the updated stable patch repre- sentations:
PF SLUUUPF_Preys PF_Ref)
(10)
These two refinement stages are repeated for S iterations. The final patch features PF serve as object-aware representations and are passed to the segmentation decoder. Specifically, the key and value matrices K, V in the Multi-Prompt Sinkhorn Attention decoder (Equation 1) are derived from PF instead of the original encoder outputs H. The rest of the decoding and training setup, including loss computation and prediction aggregation. follows OTSeg (Kim et al., 2024b) as described in Section 3:1: Equation 1 through Equation 3.

---

Published as a conference paper at ICLR 2026
3.4 GRANULAR ATTENTION FOR BETTER OBJECT CAPTURE
In our ORA module, let K Vor be the key and value for the object refinement attention Cross_Attr Cross_Attention__ OR, and Q"" be the query for the patch refinement attention Cross_Attention _PR, which are typically derived from patch features PF using linear projections. However, such pro- jections are limited their ability to capture context at varying spatial resolutions, an essential requirement for refining object and patch-level representations corresponding to objects of different scales.
To address this, we incorporate a multi-scale feature extractor into ORA that replaces the linear projections to make the attention more granular. Motivated by prior work in semantic segmenta- tion (Xie et al., 2021; Guo et al., 2022; Chen et al., 2018), we adapt lightweight module inspired by (Chen et al., 2018), but apply it within our object and patch refinement stages rather than in a decoder. This module uses parallel atrous depthwise convolutions (DW_ (onvv with dilation rates d1, d2, d3, d4 and a global average pooling (GAP) branch. The outputs are concatenated along chan- nels and passed through a 1 x 1 convolution reduce channels from 5D to D, where D is the original embedding dimension.
X Conea(DWWCCnv(3 x 3,r= =dddd,, d1), DW_ WW__onv(3 x 33, =d),
DW_Conv(3 x 3, = da), DW_Conv(S x 3 d4), GAP)
(11)
X Xgran Conv(1 x 1,5D,D)(Xin)
(12)
We apply this module to generate the granular key and value for object refinement, Gran K Gran Vor multi-scale(PF). and the granular query for patch refinement, Gran_ Q = multi-scale(PFF. Here multi-scale layer is application Equations 11, 12.
By replacing the standard projections in scale-aware representations, the ORA module becomes more granular and robust to spatial variation, enabling finer object capture and more discriminative patch features.
OR and Cross Lotss_Attetiin__ee ___ with these
EXPERIMENTS
We evaluate our proposed OC-ZSS framework on three widely used zero-shot semantic segmen- tation (ZSS) benchmarks: PASCAL VOC 2012 (20 object cutegoriess et al., 2010), PASCAL Context (59 (latses) et al., 2014), and COCO-Stuff 164K (171 classes)
et al., 8), using standard seen/unseen splits from prior works(Ding et al., 2022; Zhou et al., 2023;
Zhang et al., 2024). Performance is measured using mean Intersection-over-Unniii (mloU) for both unseen (U) and seen (S) classes, with the harmonic mean (hloU) to capture the balance between the two. We report results under both the inductive setting where training uses only annotations from seen classes and the transductive setting, which allows access to unlabeled images containing unseen class instances during training (Zhou et al., 2023). OC-ZSS uses CLIP ViT-B/16 (Radford et al., 2021) as the backbone, keeping the text encoder and DINO-B/16 (Caron et al., 2021) SSL encoder frozen, while applying visual prompt tuning (VPT) (Jia al., 2022) on the image encoder. The decoder a 3-layer lightweight transformer using Multi-Prompt Sinkhorn Attention (MPSA). We set 6 object prompts for VOC and COCO-Stuff, and 8 for PASCAL Context, with dual-stage refinement iterations by default. The model is trained using AdamW (Loshchilov & Hutter, 2019) with learning rate of 2.5e-5 and batch size of 16, for 20K, 40K, and 80K iterations VOC 2012. PASCAL Context, and COCO-Stufi respectively. All experiments are run on single NVIDIA A100 or four RTX 3090 GPUs
Inductive & Transductive Setting. We evaluate OC-ZSS under both inductive and transductive zero-shot segmentation settings across three widely used benchmarks: PASCAL VOC 2012, PAS- CAL Context, and COCO-Stuff 164K. In our comparisons with OTSee (Kim et al., 2024b), we report results using its best-performing variant, OTSeg+. In the inductive setting, where the model is trained only with annotations from seen classes and without access to unseen class names, OC- ZSS achieves the best overall performance. As shown in Table 1, OC-ZSS attains 85.3% mloU for unseen classes and 93.6% for seen classes on VOC 2012, resulting in harmonic mean of 89.2%, outperforming OTSeg and CLIP-RC. On PASCAL Context, achieves 62. 1% (U) and 55.6% (S), while on COCO-Stuff 164K it reaches 42.8% and 42.2%, both higher than prior methods.
7

---

Published as a conference paper at ICLR 2026
Table Comparison under the inductive setting (* denotes our reproduction, official code).
Methods
VOC 2012
PASCAL Context
COCO Stuff164K
mloU(U)
mloU(S)
mloU(U) mloU(S)
hioU
mloU(U)
mloU(S)
hloU
ZegFormer (Ding al. 2022)
63.6
86.4
73.3
33.2
34.8
ZSSeg (Xu et al., 2022b)
72.5
83.5
77.6
36.3
39.3
37.8
MAFT (ZSSeg) (Jiao et 2023)
66.2
88.4
75.7
40.1
40.6
40.3
ZegCLIP (Zhou
2023)
77.8
91.9
84.3
54.6
46.0
49.9
41.4
40.2
40.8
SPT (Xu et al., 2024h)
81.9
91.1
86.2
57.3
52.8
55.0
39.8
41.5
40.6
CLIP-RC (Zhang
2024)
80.7
91.6
85.8
57.3
47.5
51.9
41.6
40.9
41.2
OTSeg (Kim et al., 2024b)
81.6
93.3
87.1
60.4
53.2
57.7
41.8
41.3
41.5
OC-2SS
85.3
62.1
55.6
58.7
42.8
42.5
Table 2: Comparison under transductive setting (* denotes reproduction, official code).
VOC
PASCAL Context
COCO-StuiT164K
Methods
mloU(UU
mloU(S)
hloU
mloU(U)
mloU(S)
bloU
mloU(U)
mloU(S)
hloU
ZSSeg (Ku 20226)
78.1
79.2
79.3
43.6
39.6
41.5
MaskCLIP+ (Zhou et al., 2022a) 2022a)
88.1
86.1
87.4
66.7
48.1
53.3
$4.7
39.6
45.0
FreeSeg (Qin
2023)
45.3
MVP (Guo et al., 2023)
87.4
89.0
88.0
67.5
48.7
54.0
$5.8
39.9
45.5
ZegCLIP (Zhou et 2023)
89.9
92.3
91.1
47.2
55.6
40.7
48.5
SEG (Xu al. 2024b)
93.4
93.5
93.4
68.2
53.4
59.9
60.4
42.2
49.7
CLIP-RC (Zhang et al,, 2024)
92.2
93.9
93.0
64.5
48.1
55.1
60.8
42.0
49.7
OTSeg (Kim et al., 2024b)
94.3
67.0
54.0
59.8
62.6
49.8
OC:7SS
95.6
94.9
95.2
70.0
55.6
61.9
64.2
43.0
51.5
In the transductive setting, where the model is trained with access to unseen class names (no un- seen class seg. ground truth), OC-ZSS further improves across all benchmarks. achieves 95.6% and 94.9% mloU for unseen and seen classes respectively VOC 2012, with harmonic mean of 95.2%. On PASCAL Context and COCO-Stufi reaches hloU scores of 61.9% and 51.5%, setting benchmark. These consistent improvements be attributed finement framework. Unlike prior methods, our approach focuses on generating patches that capture entire object regions rather than arbitrary parts. OC-ZSS first identifies coarse object-level features, which are then refined in subsequent stages to form more accurate and object-focused patch group- ings. This structured refinement process helps the model better learn object semantics and improves generalization especially in unseen classes, which is important in zero-shot segmentation.
object-centric patch re-
building (o)
Broacle
fence (
Seank (u)
Soewelk (+)
Figure 4: Mask predictions: input image with ground truth mask (left), OTSeg pred. (middle), and ours pred. (right)
Figure 5: Input image (left), clustered patches without ORA (middle), and clustered patches with ORA (right)
Figure presents a comparison between the predictions of the previous state-of-ther teffthe-ar art method, OT- Seg (Kim et al., 2024b), and our proposed OC-ZSS. The predictions from OTSeg exhibit significant noise and reduced accuracy, whereas our method produces cleaner and more precise segmentation maps across both seen and unseen categories. Figure 5 illustrates the impact of our Object-aware Refinement and Aggregation (ORA) module on patch clustering. We visualize clusters by assign- ing different colors patches within input image. Without the ORA module, the patch clusters appear scattered and unstructured. In contrast, with ORA, the clusters are more coherent and local- ized around meaningful objects. This demonstrates that object-centric patches not only contribute to improved quantitative performance but also lead to more semantically meaningful and visually accurate segmentation outputs.
8

---

Published as a conference paper at ICLR 2026
Table 3: Cross-dataset generalization using COCO as the source.
Table Model efficiency and resource usage.
Method
Params (M)
GFLOPs
GPU Mcm J
Method
PASCAL Context
VOC 2012
7sseg
61.1
1916.7
ZegFormer
60.3
1829.3
Inductive setting
CLIPRC
36,9
73.8
3,47G
ZegCLIP
41.2
93.4
OTSeg
13.8
61.9
3.25G
OC-ZSS
27.2
64.0
3.35G
OTSee
49.1
93.8
OC-ZSS
49.6
94.2
Table 5: VOC12 with very few seen classes
Transductive sefting
ZegCLIP
45.8
94.2
OT 53.4
94.2
hlol
U
5
hloU
OC-ZSS
54.0
94.5
CLIP RC (Zhang al.,, 2024)
28.8
71.9
41.1
55.1
86.3
67.3
OT (Kim et 2024bb
17.7
88.3
29.4
53.0
90.2
66.8
OC-25S
92.7
59.1
90.6
71.5
Table 6: Ablation of main components for VOC 2012 and Con- text (inductive setting) represents component used.
Table 7:
Attention masks genera- tion strategies for object prompts.
Components
VOC 2012
PASCAL Context
Variant
hlu
DINO B/16 (Caron et al., 2021)
89.2
Obj prompts mloU(U) mloU(S) hloU
mloU((U) mioU(S)
DINO-B/S (Caron et al., 2021)
x
x
80.6
92.5
86.1
59.4
55.0
57.1
DINOx2-B/14 B/14 (Oquab al., 2024)
89.1
CLIP (Radford et al,, 2021)
87.6
X 84.0
93.2
88.3
61.7
55.4
58.4
Region (Zhang et al., 2024)
/ 85.3
89.2
55.6
No Mask
87.1
Cross-Dataset Generalization. OC-ZSS demonstrates strong cross-dataset generalization under both inductive and transductive settings. As shown in Table 3, when trained on COCO and evalu- ated on PASCAL Context and VOC 2012, OC-ZSS outperforms prior methods across both datasets. In the inductive setting, it achieves 49.6% mloU on PASCAL Context and 94.2% on VOC 2012, surpassing ZegCLIP and OTSeg. Under the transductive setting, performance further improves to 54.0% and 94.5% respectively. These results validate the generalization ability of our object-centric refinement framework across distribution shifts.
Maintaining Efficiency with Object-Centric Design. OC-ZSS introduces additional parameters to support its refinement architecture, yet remains highly efficient compared to other methods. As shown in Table 4, it uses only 3.35G GPU memory and 64.0 GFLOPs, staying close to OTSeg de- spite delivering significantly better segmentation. The slight increase parameter count (27 (27.2M) deliberate and lightweight addition, enabling object-centric reasoning without relying tra encoders or complex pipelines. Unlike CLIP-RC, which incurs higher memory and compute costs, OC-ZSS achieves strong segmentation performance with minimal overhead, demonstrating that substantial gains can be achieved without significant increases in complexity.
Strong Performance in Low-Shot Regimes. We further evaluate OC-ZSS under labeled data scarcity, where only 25% or 50% of PASCAL VOC classes are treated as seen during training, compared the default 75% setting (15 seen classes). As shown in Table OC-ZSS achieves 49.8% mIoU on unseen classes with just 25% supervision and 59.1 % with 50%, outperforming both CLIP-RC and OTSeg. The strong performance under the 25% setting particular demonstrates the models ability to generalize effectively from minimal supervision, validating the impact of our patch refinement framework.
ABLATIONS
Impact Each Component in OC-ZSS. conduct a stepwise ablation on VOC 2012 and PAS- CAL Context to quantify the contribution of each core module. As shown in Table 6, adding only the ORA module (row 2; object features randomly initialized) improves unseen mloU by +1.4 on VOC and +2.0 on Context over the baseline. Enabling DINO-guided prompts (row 3) supplies richer object features to ORA, mitigating poor- poo--iitialiizaionn effects noted in Slot Attention (Zhang et al., 2023) and yielding further gains seen and unseen splits. Finally, incorporating granular atten-

---

Published as a conference paper at ICLR 2026
tion further improves both seen and unseen performance, achieving the highest scores across both datasets. These results confirm that all components are required for better generalization.
Comparing Mask Generation Strategies. We compare strategies for generating attention masks that guide the object prompts (Table 7). SSL-based clustering (DINO-B/16, DINOv2-B/14 DINO- B/8) achieves the best hloU, with DINO-B/ slightly outperforming DINOv2, and surpasses masks from CLIP features (Radford et al., 2021) and fixed Region prompts from CLIPRC (Zhang et al., 2024). This trend underscores the need for good feature initialization (Zhang al., 2023) relative to No Mask (random-init object features in ORA). This supports leveraging external self-supervised representations to initialize more effective object prompts.
LIMITATIONS AND FUTURE WORK.
OC-ZSS relies external SSL backbone (DINO in our experiments) to provide object cues, which introduces extra computation and can propagate biases when masks are strongly misaligned with true objects. Although our Voronoi clustering and ORA refinement mitigate some over-/under- coverage issues, heavily overlapping or highly ambiguous objects remain challenging, and gains on near-saturated categories sometimes modest. In addition, for the number of object prompts and ORA refinement iterations, without adapting them per image or dynamically re-estimating/spliting masks during refinement. Exploring SSL-free ways to guide object prompts or extract object features, as well as improved refinement architectures, is an im- portant direction for future work. Further, adaptive schemes (e.g., entropy-based for choosing the number of object prompts and iterations, and mechanisms that can re-split or update object masks online, are promising avenues to enhance robustness and flexibility.
fixed global hyperparameters
CONCLUSION
We presented OC-ZSS, novel object-centric framework for zero-shot semantic segmentation that operates entirely on a frozen CLIP backhone. By introducing self-superission-uiidee object prompts and dual-stage Object Refinement Attention (ORA) module, our approach captures and refines object-level features without requiring additional supervision or encoder tuning, Through it- erative refinement and multi-scale attention, OC-ZSS promotes stronger semantic grouping in patch representations and improves alignment with textual concepts. Our method achieves state-of-the- art performance across multiple benchmarks and settings, demonstrating that object-centric patch refinement is powerful yet efficient strategy for advancing zero-shot segmentation.
8 REPRODUCIBILITY STATEMENT
All details required to reproduce our results are provided in the Experiments section and Appendix (Implementation Details), including datasets, hyperparameters, training schedules. Upon accep- tance, we will release code, configuration files, and checkpoints to enable exact replication.
ACKNOWLEDGEMENT
This work was supported by the EuroHPC Joint Undertaking through the EuroHPC Development and the EuroHPC Benchmark Access Call
Access Call (Proposal ID: EHPC-DEEV2005511000 We are grateful to Umar Marikkar, Iqra Farooq, and Silpa
(Proposal ID: CHIPC-BEN-0025B0-- Vadakkeeveetil (University of Surrey), and Carl Udora (University of Bristol), for their valuable feedback during the rebuttal process/compute.
REFERENCES
Franz Aurenhammer. Voronoi diagramsa survey of a fundamental geometric data structure. ACM Computing Surveys, 23(3):345405, September 1991. ISSN 0360-0300.
Görkay Aydemir, Weidi Xie, and Fatma Guney. Self-supervised cejecc-centric learning for vidcos. Advances in Neural Information Processing Systems, (ei.32879-32899 2023.
10

---

Published as a conference paper at ICLR 2026
Donghyeon Baek, Youngmin Oh, and Bumsub Ham. Exploiting joint embedding space for general- ized zero-shot semantic segmentation. In Proceedings of the IEEE/CVF international conference computer vision, pp.
2021.
Luca Barsellotti, Roberto Amoroso, Lorenzo Baraldi, and Rita Cucchiara. Fossil: Free open- vocabulary semantic segmentation through synthetic references retrieval. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1453-1462, 2024. doi: 10.1109/WACV57700..22 20224000000..22.00044449
Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmenta- tion. Advances in Neural Information Processing Systems, 32, 2019.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1209-1218, 2018. doi: 10. 10.110//VPR.22088000
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650-9660, 2021.
Liang-Chies Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018.
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. 2022.
Jiaxin Cheng, Soumyaroop Nandi, Prem Natarajan, and Wael Abd-A Ald-gedd Sign: Spatial- information incorporated generative network for generalized zero-shot semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9556-9566, 2021.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Meth- ods Natural Language Processing (EMNLP), pp. 1724 1734. Association for Computational Linguistics, 2014.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. URL
pettss:/openreeeee
.et//
Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
11583-11592, 2022.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Ke Fan, Zechen Bai, Tianjun Xiao, Tong He, Max Horn, Yanwei Fu, Francesco Locatello, and Zheng Zhang. Adaptive slot attention: Object discovery with dynamic slot number. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23062-23071, 2024.
Jiannan Ge, Lingxi Xie, Hongtao Xie, Pandeng Li, Xiaopeng Zhang, Yongdong Zhang, and Qi Tian. Alignzeg: Mitigating objective misalignment for zero-shot semantic segmentation. In European Conference on Computer Vision, pp. 142-161. Springer, 2024.
Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature gener- ation for zero-shot semantic segmentation. In Proceedings of the 28th ACM International Confer- on Multimedia, pp. 1921-1929, 2020.
11

---

Published as a conference paper at ICLR 2026
Jie Guo, Qimeng Wang, Yan Gao, Xiaolong Jiang, Shaohui Lin, and Baochang Zhang. Mvp-seg: Multi-viww prompt learning for open-vocabulary semantic segmentation. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp. 158- 1711. Springer, 2023.
Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. Advances in neu- ral information processing systems, 35:1 140-1156, 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference Computer Vision (ECCV), 2022.
Siyu Jiao, Yunchao Wei, Yaowei Wang, Yao Zhao, and Humphrey Shi. Learning mask-aware clip representations for zero-shot segmentation. Advances in Neural Information Processing Systems, 36:35631-35653, 2023.
Dahyun Kang and Minsu Cho. In defense of lazy visual grounding for open-vocabulary semantic segmentation. In European Conference on Computer Vision, pp. 143-164. Springer, 2024.
Chanyoung Kim, Woojung Han, Dayun Ju, and Seong Jae Hwang. Eagle: Eigen aggregation learn- ing for object-centric unsupervised semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3523 22333, 2024a.
Kwanyoung Kim, Yujin Oh, and Jong Chul Ye. Otseg: Multi-prompt sinkhorn attention for zero-shot semantic segmentation. In ECCV 2024. pp. 200-217, Cham. 2024b.
Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang. Litong Feng, and Wayne Zhang. Prox- yelip: Proxy attention improves clip for open-vocabulary segmentation. In European Conference Computer Vision, pp. 70-88. Springer, 2024.
Minhyeok I Lee, Suhwan Cho, Dogyoon Lee, Chaewon Park, Jungho Lee, and Sangyoun Lee. Guided slot attention for unsupervised video object segmentation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 3807-3816, June 2024.
Yunheng Li, Zhong- Yu Li, Quan-Sheng Zeng, Qibin Hou, and Ming-Ming Cheng. Cascade-clip: Cascaded lingun-language embeddings alignment for zero-shot semantic segmentation. In Inter- national Conference on Machine Learning. 2024.
Xinyu Liu, Beiwen Tian, Zhen Wang, Rui Wang, Kehua Sheng, Bo Zhang, Hao Zhao, and Guyue Zhou. Delving into shape-aware zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2999-3009, 2023.
Francesco Locatello, Dirk Weissenbom, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in neural information processing systems, 33:1 13:11525-11538 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Con- ference Learning Representations (ICLR), 2019. URL https:/ http:::///peneevee..ooo forummid--Bg6Ricoom
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmen- tation in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 891-898, 2014. doi: 10.1 0..109//VPP.2204.1115
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervi- sion. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
12

---

Published as a conference paper at ICLR 2026
Giuseppe Pastore, Fabio Cermelli, Yongqin Xian, Massimiliano Mancini, Zeynep Akata, and Bar- bara Caputo. A closer look at self-training for zero-label semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2021.
Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang, Rui Wang, Shilei Wen, Xin Pan, et al. Freeseg: Unified. universal and open-vocabulary image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
19446-19455, 2023.
Congpei Qiu, Yanhao Wu, Wei Ke, Xiuxiu Bai, and Tong Zhang. Refining CLIP's spatial awareness: A visual-centric perspective. In The Thirteenth International Conference on Learning Represen- tations, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. PmLR, 2021.
Thomas Stegmüller, tim Lebailly, Nikola Dukić, Behzad Bozorgtabar, Tinne Tuytelaars, and Jean-Philippe Thiran. A simple framework for open-vocabulary zero-shot segmentation. In The Thirteenth International Conference Learning Representations, 2025. URL https:
//openzeview net/ /tttte/ooaeee tttp://rrmmiiddssseeee
Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3635-3647, 2024.
Monika Wysoczańska, Michaël Ramamonjisoa, Tomasz Trzciáski, and Oriane Siméoni. Clip-diy: Clip dense inference yields open-vocabulary semantic segmentation for-free. In Proceedings of the JEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1403-14444 1413, 2024a.
Monika Wysoczańska, Oriane Siméoni, Michaël Ramamonjisoa, Andrei Bursuc, Tomasz Trzciñski, and Patrick Pérez. Clip-dinoiser: Teaching clip few dino tricks for open-vocabulary semantic segmentation. In European Conference on Computer Vision, pp. 320-337. Springer, 2024b.
Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077-12090, 2021.
Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In The Twelfth International Conference on Learning Representations, 2024a. URL https://
openreview net http:///wweee/tooodddd-d------ee www.foommmddcceennn....
Jiarui Xu, Shalini De Mello, Sifci Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18134-18144 2022a.
Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng. Yi Wang, Yu Qiao, and Weidi Xie. Learning open- vocabulary semantic segmentation models from natural language supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pp. 2935-2944 2023.
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open- vocabulcyyy semantic segmentation with pre-trained vision- language model. In European Conference on Computer Vision, pp. 736-753. Springer, 2022b.
Wenhao Xu, Rongtao Xu, Changwei Wang, Shibiao Xu, Li Guo, Man Zhang, and Xiaopeng Zhang Spectral prompt tuning: Unveiling unseen classes for zero-shot semantic segmentation. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 38, pp. 6369-6377, 2024b.
Jian-We Zhang, Yifan Sun, Yi Yang, and Wei Chen. Feature-proxy transformer for few-shot seg- mentation. Advances in neural information processing systems, 35:6575-6588. 2022.
13

---

Published as a conference paper at ICLR 2026
Yan Zhang, David W. Zhang, Simon Lacoste-Julien, Gertjan J. Burghouts, and Cees G. M. Snoek. Unlocking slot attention by changing optimal transport costs. In Proceedings of the 40th Inter- national Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 41931-41951. PMLR, 23-29 Jul 2023.
Yi Zhang, Meng-Hao Guo, Miao Wang, and Shi-Min Hu. Exploring regional clues in clip for zero- shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3270-3280, June 2024.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pp. 696-712. Springer, 2022a.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representa- tions (ICLR), 2022b.
Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11175-11185, 2023.
A MORE IMPLEMENTATION DETAILS
OC-ZSS uses CLIP ViT-B/16 as the backbone, keeping both the text encoder and DINO-B/16 SSL visual encoder frozen. Visual Prompt Tuning (VPT) is applied to the image encoder using 10 learn- able prompts. The decoder is a 3-layer lightweight transformer with Multi-Prompt Sinkhorn Atten- tion (MPSA). We set 6 object prompts for VOC and COCO-Stuff, and for PASCAL Context. The Object-Centric Refinement module performs 4 dual-stage refinement iterations by default; for PAS- CAL Context, we use 5 iterations during both training and testing to handle its higher class diversity. All models are trained using the Adam W optimizer with a learning rate of 2.5e-5, weight decay of 0.01, and batch size of 16. Training is conducted for 20K, 40K, and 80K iterations on VOC 2012, PASCAL Context, and COCO-Stuff, respectively. All experiments are run on single NVIDIA A100 or four RTX 3090 GPUs Evaluation is performed using single-scale inference. The values for number of object prompts no, number of ORA iterations S is selected based on grid search for each where the values for VOC are shown in further sections.
We compare against standard baselines in both inductive and transductive settings, using re- ported numbers or official results: ZegFormer (Ding et al., 2022), ZSSeg (Xu et al., 2022b), MaskCLIP+ (Zhou et al., 2022a), ZegCLIP (Zhou et al., 2023), CLIP-RC (Zhang et al., 2024), Cascade-CLIP (Li et al., 2024), OT-Seg (Kim et al., 2024b), MVP-SEG+ (Guo et al., 2023), SPT- SEG (Xu et al., 2024b), MAFT (Jiao al., 2023), and FreeSeg (Qin et al., 2023). For fairness, we report MAFT combined with ZSSeg in the inductive table, instead of FreeSeg (Qin et al., 2023). FreeSeg adopts a more complex Mask2Former-style query mask design (Cheng et al., 2022) with a broader multi-task recipe for unified/open-vocabulary segmentation, is included only in our trans- ductive comparison, following prior works (Zhang et al., 2024: Li et al., 2024). Before applying the multi-scale adapter, we reshape CLIP patch tokens back to an H x W grid so that the depthwise atrous convolutions operate on contiguous spatial neighborhoods. The adapter is implemented as parallel 3 x 3 depthwise convolutions with fixed dilation rates {1, 6. 12, 18), and 1 x 1 projec- tion to produce the final Q/K/V projections. Both the CLIP and DINO encoders are deptfrozee in all experiments; the only trainable components we add are ORA and visual prompts. Finally, the optimal-transpp decoder (including the Sinkhorn iterations, temperature, and log-domain stabi- lization) is taken unchanged from OTSeg, so any performance differences stem from our refinement modules rather than from changes to the decoder. The GRU used in ORA is shared for every re- finement iteration including cross-attention layers. All code, hyperparameters, checkpoints will be made available on acceptance.
B DECODING PROCESS WITH MPS AND MPSA
OTSeg (Kim et al., 2024b) introduces decoding strategy that combines Optimal Transport- based prompt-pixel alignment with a transformer-style attention mechanism. It consists of two key compo- nents: the Multi-Prompp Sinkhorn (MPS) module and the Multi-Prompis Sinkhorn A Attention
14

---

Published as a conference paper at ICLR 2026
Algorithm Mask generation from DINO last-layer features (FPS Voronoi + prompipatch wiring)
Input: F E PRRxx (DINO last-layer per image), clusters K, #VPT tokens P, (opt.) target spatial size
Output: (altention mask), labels
1: if S given then
2:
reshape F to SxSxC, bilinear upsample to SxS, flatten to NxC 3: end if
FPS: pick 61 uniformly; for t=2..K: d, min ||F: Fey
<t
[i C t- arg mag, d,
5: Voronoi labels: & arg (1. KY
||
Foz for
Layout: total length A 1+P+N --
7: indices: c1s=0;
= vp part
8: Init: .M + -00- LAXA
9: for i i= 1 K de
10:
p patch patch f
patch tokens in cluster prompt <
Unmask: M (1)rmpt()) 0: (rppppt + 0; Mpprompt(t) promppt
12: end for
13:
return M,
(MPSA) module, both aimed at enhancing multimodal alignment between pixel embeddings and prompt embeddings.
Multi-Promppts Sinkhorn (MPS). Given M pixel embeddings X E RMxd and N text prompt embeddings T IR-xxd MPS first computes a cosine similarity matrix:
S S
and defines cost matrix regularized optimal transport problem:
S. To align pixels with prompts, OTSeg solves an entropy-
P arg min b)
(P. (H),
where (P)) is the entropy of the transport plan, and II(a, b) denotes the set of joint distributions with given marginals. The optimal transport plan P is computed using the Sinkhorn algorithm.
The final prompt-guided score map is obtained as:
S OOS S.
which acts as a refined segmentation prediction.
Multi-Promptt Sinkhorn Attention (MPSA). To integrate prompt-pixel transport directly into
transformer decoding, OTSeg replaces standard cross-attention with MPSA. Unlike softmax-based
attention:
Attention Q K, V) = softmax softmax(QK) V,
MPSA computes the transport matrix via Sinkhorn:
MPSA(Q, (Q, K. V) = Sinkhomn (KK V.
This approach promotes diverse prompt activation and better semantic separation in the generated attention maps.
Decoder Design. The decoder used in OTSeg consists of three transformer layers, each applying MPSA instead of standard multi-head cross-attention. The final output is upsampled using bilinear interpolation operator L4 to match the original image resolution:
Y final u(Y).
where E " and K the number of semantic classes. This decoder output can be used independently fused with the MPS-based prediction for improved performance.
15

---

Published as a conference paper at ICLR 2026
Table 8: ORA.
Effect of different design choices for
Table 9: classes.
Pascal Context with very few seen
Variant
ml)U(U)
ml)U(S)
hloU
Methods
Conteu 25%
Cantest 509
5
blll
11
5
ball
ORA
85.3
93.6
89.2
CLIP RC (Zhung et 20124)
11.5
51.1
18.8
18.01
45.7
25.8
Slot Attention
83.4
93.1
87.9
OTSa (Kim 20246)
11.4
57.2
19.0
21.2
54,4
50.5
ORA w/o GRU
83.1
90.9
86.8
OC-2SS
12.5
58,9
20.6
235
$5.6
33.0
1x Cross Attention
82.6
93.4
87.7
2x Cross Attention
82.2
93.2
87.3
4x Cross Attention
81.6
92.9
86.8
Table 10: Ablation on ORA iterations (S) in the inductive setting.
Table 11: End-to-end throughput (iterations/sec; higher is better).
Iterations
0
2
3
4 (ours)
5
Config
it/s
hloU (%)
86.1
87.2
87.7
88.0
89.2
88.9
87.6 87.8
Baseline (OTSeg) Baseline + Object Prompts
15.8
15.2
Baseline + ORA
13.9
Baseline + Granular Attn
13.0
C VORONOI AND MASK GENERATION ON DINO FEATURES.
Given last-layer DINO tokens E NxC per image (optionally resized to fixed grid), we select K centers by farthest-point sampling in feature space and assign each patch token its nearest center (squared (2), inducing a Voronoi partition (Aurenhammer, 1991). We then form the sequence [CLS], 1PT Patch1.N. ObjPrompt,,k and build sparse attention mask that unblocks only the connections between ObjPrompt, and the patch tokens in cluster i (plus the prompt self self-edge); all other entries remain oc. The implementation is GPU-based PyTorch with tensor indexing (following (Zhang et al., 2022)), so the per-iteration cost is negligible relative to CLIP forward; empirically, enabling prompts changes throughput from 15.8 to 15.2 it/s, see Table 11.
D MORE ABLATIONS AND VISUALISATIONS
Effect of Different Design Choices for ORA. Table 8 presents an ablation study on the design choices of the proposed Object-Region Attention (ORA) module. The full ORA configuration, which includes both object and patch refinement stages, achieves the best performance across un- seen, seen, and harmonic mean IoU. Replacing ORA with Slot Attention (Locatello al., 2020), which refines patch features from coarse object representations without refining objects first, leads to a noticeable drop, particularly in unseen class performance. Removing the GRU module from ORA (ORA w/o GRU) also reduces performance, validating the role of recurrent refinement. Lastly, we compare against standard cross-attention modules with 1, 2, and 4 stacked layers. Increasing the number of cross-attention layers does not improve performance and instead results in gradual degradation, highlighting the importance of both object and patch refinement stages.
Ablation on number of ORA iterations. We vary the number of refinement iterations (Table 10) S E {O..... 7} (with S-0 disabling ORA). hloU improves up to S-4 (from 86.1 to 89.2, +3.1), showcasing improved generalisation, then plateaus and slightly declines for larger S (88.9 at 87.6 at 6, 87.8 at 7), suggesting overfitting and reduced generalisability. We therefore fix S=4 in the main method as an accuracylatency sweet spot.
Runtine/throughput ablation. We report end-to-end throughput (iterations/sec; higher is better) for each configuration under identical hardware and resolution where DINO features are extracted and stored for all images. Relative to the baseline of OTSeg (15.8 it/s), Object Prompts incur a small overhead (15.2 it/s; 3.8% drop; includes clustering). ORA moderate (13.9 it/s; 12.0% drop), and Granular Attn is the heaviest (13.0 it/s; 17.7% drop). Note that when DINO extraction is performed sequentially, the throughput decreases accordingly due to the additional ceature-extraction overhead. This value is provided for reference only and does not represent practical real-world operation, where
16

---

Published as a conference paper at ICLR 2026
Table 12: OC-ZSS with different CLIP vision hacubones/sources (inductive setting).
Table 13: Attention-mask sources for ob-
ject prompts (inductive). is our default setting
Backbone Weights/ Source
hloU (%)
Mask
hIoU (%)
ViT-B/16 OpenAl CLIP (Radford et al., 2021)
89.2
VIT-B/16 MetaCLIP (Xu et al., 2024a)
90.1
No Mask
87.1
VIT-L/14
OpenAl CLIP (Radford et al., 2021)
90.3
*DINO-B/16 (Caron et al., 2021)
89.2
iBoT (Zhou et al., 2022h)
88.3
DINOv2-B/14 + Reg (Darcet al., 2024)
89.2
Ablation on the number object prompts n, on VOC. moderate number of prompts yields the best trade-off, and we therefore adopt n. = 6 as the default.
Table 14:
Object prompts ne
2
3
4
5
6 (ours)
7
8
hloU (%)
87.2
87.4
88.3
88.8
89.2
88.1
87.8
DINO feature extraction executed in parallel across future batches avoid bottlenecks. As shown in Table 11, these overheads remain modest; taken with the accuracy gains in the main ablation, Table they support including each module the final model.
Effect of different CLIP models We vary the CLIP vision backbone while keeping OC-ZSS fixed DINO-B/16 masks and similar number of object prompts. As shown Table 12, MetaCLIP. B/16 (Xu et al., 2024a) improves over OpenAl CLIP--B11 (Our default) (Radford et al., 2021) (89.2 + 90.1 hloU), and OpenAl CLIP-L/14 yields further gain of 90.3 hloU. These results suggest that our object-centric refinement transfers across CLIP variants and backbones. Larger backbones and better variants perform better.
More experiments on Attention Mask As in Table 13, SSL-derived masks consistently outperform No Mask (random initialization). confirming that better feature initialization improves guidance for ORA. In particular, DINO-B/ and DINOv2 Registers (Darcet et al., 2024) reach 89.2 hloU, while iBoT attains 88.3, all exceeding the 87.1 hloU obtained without masks. We use DINO-B/16 instead of MINOv2-B/14 with registers (Darcet et al., 2024) due to better efficiency from smaller patch size of DINO-B/16. This aligns with the trend observed Main paper, Table 7: self-supervised models provide stronger spatial grouping cues for mask generation than unguided initialization.
Generalization with Limited Seen Classes Pascal Context. We evaluate OC-ZSS under ex- treme low-shot settings on Pascal Context by restricting the number of seen classes to only 25% or 50% during training, compared to the standard zero-shot setting which uses 49 seen and 10 unseen classes out of 59 total. This dataset is inherently more challenging than PASCAL VOC due to its larger number of classes, higher scene complexity, and increased semantic overlap between cate- gories. As shown in Table 9, the performance under the 25% setting, with minimal or no gains. Despite this, OC-ZSS still improves over both CLIP-RC and OTSeg, achieving the best unseen and harmonic IoU. These results demonstrate the robustness of OC-ZSS and its ability to generalize to complex and varied unseen categories, even with severely limited supervision.
unseen classes tends to stagnate for other methods
Effect of the number object prompts. We also vary the number of object prompts n. on VOC while keeping all other settings fixed. Table 14 reports the harmonic mean (hloU) between and unseen mloU, Performance improves when moving from very few prompts (n, = 2,3) to a moderate number (n, 4,5), peaks at n,, 6, and then degrades slightly for larger values.
17

---

Published as a conference paper at ICLR 2026
Table 15: Very-few-seen class ablation on seen unseen IoU. We disentangle the benefit of each major component (object prompts, ORA, and alternative aggregation mechanisms) and how they affect low-shot generalization. All numbers are percentages.
Method
Unseen IoU
Seen IoU
hioU
Baseline (OTSeg replication)
17.70
88.30
29.40
Baseline object prompts (SSL guided) No ORA
17.40
88.75
29.10
Baseline + object prompts (SSL guided) ORA
49.80
92.70
64.80
Bascline + ORA (random init object fcats) + no object prompts
42.69
89.67
57.84
Baseline + object prompts (mask from CLIP) ORA
49.40
90.39
63.89
Baseline object prompts (SSL guided) slot attention
38.35
70.85
49.76
Baseline + object prompts (SSL guided) cross-attention
36.97
90.04
S2.42
grass
Wo
lic
chei
floer
Grees mobortt
Surtoin (+)
(
perso
sofa
flace
Figure 6: More Mask predictions: input image with ground truth mask (left), OTSeg prediction (middle), and OC-ZsS prediction (right).
Figure 7: More Visualisation: Input
image (left), clustered patches without ORA (middle), and with ORA (right).
Additional Qualitative Examples. Figure presents additional comparisons between the predic- tions of OTSeg (Kim et al., 2024b) and our OC-ZSS across diverse scenes. Consistent with the observations in the main paper, OC-ZSS produces more accurate and spatially coherent segmenta- tion masks, particularly for complex object boundaries and less common categories. In contrast, OTSeg predictions often exhibit over-segmentation or coarse localization. Figure further visual- izes the impact of the Object-aware Refinement and Aggregation (ORA) module. When ORA is removed, the resulting patch clusters remain noisy and dispersed. With ORA enabled, patches align more clearly with object contours, resulting in more compact and semantically consistent group- ings. These examples qualitatively support the effectiveness of our design choices improving both segmentation quality and region-level consistency.
Further analysis. While our main comparisons on VOC, PASCAL Context and COCO-Stufi and the ablations components, mask sources, and CLIP backbones already show that stronger object features combined with ORA systematically improve zero-shot segmentation, here we study the same components in more controlled very-few-seen regime better understand how they affect
18

---

Published as a conference paper at ICLR 2026
Table 16:
Example per-class IoU on COCO-Stuff comparing OTSeg and OC-ZSS. OC-ZSS brings
larger gains on several fine-graines or cluttered categories (top rows), while improvements on near- saturated or ambiguous categories (bottom rows) are smaller.
Class
OTSec loU
OC-ZSS IoU
A
wall-concrete
17.96
30.75
+12.79
skateboard
40.17
46.31
+6.14
baseball bat
33.87
37.41
43.54
stairs
17.93
20.55
+2.62
road
38.45
42.12
+3.67
clouds
32,90
34.12
+1.22
giraffe
68.17
68.85
+0.68
elephant 87.84
87.95
+0.11
Table 17: Reported performance of recent open-vocabulary segmentation methods on VOC, PAS- CAL Context, and COCO-Stuff. Rows above the line report mloU under their respective open- vocabulary protocols; the OC-ZSS row (") reports transductive hloU under the standard zero-shot segmentation setting with seen/unseen splits. Protocols, training data, and supervision differ substan- tially from our setting, so these numbers are not directly comparable to our inductive/transductive results.
Method
VOC
PASCAL Context
COCO-Stuff
SAM-CLIP (Wang al., 2024)
60.6
29.2
31.5
CLIP-DIY (Wysoezańska al., 2024a)
59.9
OLIP-DINOiSe (Wysoczańska et al., 2024b)
62.1
32.4
24.6
ProxyCLIP (Lan et al., 2024)
61.3
35.3
26.5
OVSegmentor (Xu al., 2023)
53.8
20.4
FOSSIL (Barselloti et al., 2024)
35.8
24.8
SimZSS (Stegmüller et al., 2025)
58.4
37.2
OC-ZSS (ours, ZSS hloU)?
95.2
61.9
51.5
generalization. Table 15 decomposes the contributions of SSL/CLIP-guided object prompts, and (ii) the ORA refinement itself.
Starting from our OTSeg baseline (17.70% unseen 88.30% seen 29.40 hloU), adding SSI.-guided object prompts withour ORA leaves unseen IoU and hloU almost unchanged (17.40% unseen / 88.75% seen 29.10 hloU), indicating that stronger object features alone are not sufficient, as the patches are not yet refined with these object features. In contrast, ORA with randomly initialized object features already yields a large boost in unseen IoU and hloU (42.69% unseen 88.677% seen / 57.84 hloU), showing that the refinement mechanism itself is crucial for low-shot generalization. Guiding ORA with SSL masks (DINO) further improves performance to 49.80% unseen / 92.70% seen/ / 64.80 hloU, and replacing SSL masks with CLIP masks yields similar gains of 49.40% unseen / 90.39% seen / 63.89 hioU, confirming that our framework is not tied to a particular encoder but consistently benefits from better object features.
Finally, when we keep the same SSL-guided object prompts but replace ORA with slot attention or cross-altention, we obtain clearly weaker trade-offs: slot attention reaches 38.35% unseen 70.85% seen / 49.76 hloU, while cross-attention reaches 36.97% unseen 90.04% seen / 52.42 hloU, To- gether with the main-paper results and ablations, these experiments show that (i) We need stronger object features, (ii) making patch representations more object-centric is crucial, and (ii) ORAs As de- sign exploits strong object prompts more effectively than standard slot- or cross-attention, leading to robust and balanced generalization in both the standard and very-few-seen regimes.
Per-class analysis on COCO-Stuff. To better understand how refinement behaves beyond aggre- gate mIoU, we report per-class loU on COCO-Stuff for few representative fine-grained / cluttered categories and near-saturated or ambiguous classes. As shown in Table 16, OC-ZSS yields sub- stantial gains several challenging structures where they thin, small finegrained (e.g., wall- concrete, skateboard, baseball bat, stairs, road), while improvements on already high- hegh-performing ambiguous categories (e.g., clouds, giraffe, elephant) are understandably modest.
19

---

Published as a conference paper at ICLR 2026
Comparison with open-vocabulary segmentation methods
For completeness, we report a coarse comparison to several recent open vocc-uoby (OV) segmenta- tion methods shared datasets, using the mloU numbers reported in their respective papers. Note that these results are not directly comparable to our transductive ZSS hloU, since OV methods typi- cally train large imagetext corpora and do not follow the standard seen/unseen ZSS protocol.
E LLM USAGE STATEMENT
We used a large language model only for editorial assistance: polishing prose, correcting grammar and typography, tightening phrasing, and standardizing terminology. The tool was not used to gen- erate ideas, methods, analyses, figures, tables, results, and not used write code modify data.
20


Reviews:
[{"review_id": "MMBB2dASc7", "reviewer": "Reviewer_3xo5", "summary": "The paper Object-Centric Zero-Shot Segmentor (OC-ZSS) proposes a framework to enhance zero-shot semantic segmentation by introducing object awareness into CLIP’s patch representations. While CLIP aligns global image and text features well, it lacks fine-grained correspondence between visual regions and textual concepts, limiting segmentation accuracy for unseen categories.\n\nOC-ZSS addresses this by injecting self-supervision-guided object prompts into the CLIP encoder. These prompts, informed by unsupervised clustering from a pretrained self-supervised model such as DINO, attend to distinct object regions through masked attention. This enables the extraction of coarse object-level features without annotations or encoder finetuning.\n\nTo refine these features, the method introduces a dual-stage Object Refinement Attention (ORA) module, which iteratively updates both object and patch representations through cross-attention. This mutual refinement strengthens semantic grouping and improves text alignment while keeping the CLIP backbone frozen. To handle objects of varying scales, OC-ZSS further employs a granular attention mechanism using multi-scale atrous convolutions, allowing more precise modeling of object structures.\n\nThe approach demonstrates competitive performance across standard zero-shot segmentation benchmarks.", "questions": "Please see the comments in the Weaknesses section.\n\nAdditional:\n\n1. Please discuss following design choices:\n\n\n     a) Number of object prompts: 6 for VOC/COCO, 8 for Pascal Context.\n\n\n      b) Refinement iterations: 4 for most datasets, but 5 for Pascal Context.\n\n\n      c) Clustering parameters not thoroughly analyzed.\n\n\n      d) Training is conducted for 20K, 40K, and 80K iterations on VOC 2012, PASCAL Context, and COCO-Stuff, respectively.\n\n#\n\n2. Why dual-stage? No explanation for why object refinement must precede patch refinement.\n\n#\n\n3. Why GRU specifically? Borrowed from slot attention without justification.\n\n#\n\n4. Why these attention patterns? The masked attention prevents object prompts from seeing the full image that can hurt global context.\n\n#\n\n5. No analysis of what makes patches \"object-centric\" beyond qualitative clustering visualizations (Figure 5 and 7).\n\n#\n\n6. Missing ablation on number of object prompts.\n\n# \n7.  The code or implementation has not been released, which limits reproducibility, as several technical details in the paper remain underspecified and cannot be fully verified or replicated without access to the source code.\n\n#\n\n8. Several implementation aspects remain ambiguous, particularly the masking semantics, GRU sharing strategy, and multi-scale convolution ordering. Clarifying these would improve reproducibility and theoretical transparency.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is clearly written and well-structured, effectively highlighting key challenges in zero-shot semantic segmentation and addressing them through an object-centric refinement approach. It maintains clarity and logical flow while presenting a focused and well-motivated solution that enhances region-level semantic alignment and segmentation accuracy for unseen categories.\n\n#\n\n1. The paper takes a principled step toward introducing object-centric reasoning into zero-shot segmentation, addressing a key shortcoming of CLIP-based models that rely only on global or patch-level embeddings without structured object awareness.\n\n#\n\n2. The proposed framework adaptively refines patch features using object-level cues, enabling it to effectively handle objects of diverse shapes, scales, and spatial layouts, a capability that some existing CLIP-based segmentation models lack.\n\n#\n\n3. The use of self-supervised clustering from a frozen encoder to guide object prompts provides a label-efficient way to utilize the inherent structure of visual features for identifying coarse object regions without requiring annotations or fine-tuning.\n\n#\n\n4. The dual-stage Object Refinement Attention (ORA) module enables bidirectional interaction between patch and object features, allowing their representations to be iteratively refined for improved semantic consistency. The inclusion of multi-scale granular attention, implemented through depthwise separable atrous convolutions with different dilation rates, helps capture contextual information across multiple receptive fields.\n\n#\n\n5. The paper includes experiments across multiple benchmarks and settings (inductive, transductive, cross-domain), showing consistent performance improvements.", "weaknesses": "1. The degree of novelty is moderate, as the method primarily combines established components such as self-supervised clustering, cross-attention refinement, and multi-scale feature aggregation. Moreover, the paper lacks a deeper analysis of why mutual refinement improves object-centricity or generalization in zero-shot settings, offering empirical validation without corresponding conceptual depth or theoretical analysis.\n\n#\n\n2. The approach relies on features from a pretrained self-supervised model, which may limit its performance if the external model (e.g., DINO) does not provide strong object cues. If the DINO-based clustering fails to correctly identify an object (e.g., over-segments a single large object or mistakenly merges two separate small objects), this error is propagated and refined by ORA. The refinement module can only improve the representation given the clusters it receives; it cannot fundamentally fix a poor initial object grouping. This makes the entire pipeline vulnerable to the inherent biases and failure modes of an un-analysed, unsupervised upstream process.\n\n#\n\n3. The granular attention mechanism (Section 3.4) shows an architectural inconsistency by applying CNN-style atrous convolutions to transformer patch embeddings without proper adaptation. Since atrous convolutions rely on contiguous spatial neighborhoods, their use on non-contiguous transformer patches may limit effectiveness. The paper also omits details on dilation rates and lacks comparisons with transformer-native multi-scale approaches, making this design choice appear misaligned with transformer principles and weakening the claim of a lightweight refinement module.\n\n#\n\n4. The use of entropy-regularized Sinkhorn attention introduces sensitivity to the regularization parameter ε and iteration count.\nWhile smaller ε leads to gradient instability and near-discrete couplings, larger ε causes over-smoothing and loss of discriminative structure in the transport plan (Cuturi, 2013; Genevay et al., 2018; Peyré & Cuturi, 2019).\nMoreover, the method incurs iterative additional computational cost and requires log-domain stabilization to prevent numerical underflow. These factors raise concerns about training stability, computational efficiency, and generalization sensitivity compared to standard softmax attention.\n\n      [1] Cuturi, M. Sinkhorn Distances: Lightspeed Computation of Optimal Transport. NeurIPS 2013.\n\n      [2] Genevay, A., Peyré, G., & Cuturi, M. Learning Generative Models with Sinkhorn Divergences. AISTATS 2018.\n\n      [3] Peyré, G., & Cuturi, M. Computational Optimal Transport. FnT in ML 2019.\n\n#\n\n5. The paper claims to be \"parameter-efficient\" and \"lightweight,\" yet:\n\n\n     a) Requires running an additional frozen DINO encoder for every input image.\n\n\n\n     b) Adds Voronoi clustering operations for mask generation.\n\n\n\n     c) Performs 4 iterative dual-stage refinements.\n\n#\n\n6. No comparison with foundation models (SAM, recent large-scale foundation models).\n\n     [1] SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding, Wang et. al, CVPR 2024, Workshops.\n\n\n#\n\n7. Missing comparisons with closely related open-vocabulary segmentation/ZSS methods (CLIP-DINOiser, ProxyCLIP, OVSegmentor, FOSSIL) that also combine CLIP with self-supervised features. The paper mentions these methods (lines 130 - 154) to distinguish its approach conceptually, but provides no quantitative benchmarking against them. This omission makes it difficult to assess whether the proposed dual-stage object refinement genuinely outperforms simpler CLIP+DINO fusion strategies or end-to-end trained alternatives.\n\n#\n\n8. The performance gains over the baseline (OTSeg) are often modest despite nearly doubling the parameter count (27.2M vs 13.8M).\n\n#\n\n9. It would strengthen the paper to include a numerical or methodological comparison with the following related approaches, and to cite them appropriately:\n\n     a) Delving into Shape-aware Zero-shot Semantic Segmentation, Liu et. al, CVPR 2023.\n\n     b) CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Free, Wysoczanska et. al, WACV 2024.\n  \n     c) Refining CLIP's Spatial Awareness: A Visual-Centric Perspective, Qiu et. al, ICLR 2025.\n\n     d) A Simple Framework for Open-Vocabulary Zero-Shot Segmentation, Stegmüller et. al, ICLR 2025.\n\n#\n\nMinor:\n\n1. Sensitivity to hyperparameters, such as the number of clusters and choice of feature layer in the self-supervised model, is not thoroughly analyzed, leaving questions about robustness across configurations.\n\n#\n\n2. The method focuses on segmentation benchmarks but lacks evaluation on broader vision-language tasks where object-centric refinement might also be relevant. Below would strengthen the claim of generalization across varied scene structures.\n\n     a) ADE20K \n\n     b) Cityscapes\n\n#\n\n3. Cross-dataset generalization (Table 3): Gains are minimal (0.5% on Pascal Context, 0.4% on VOC).\n\n#\n\n4. How were the dilation rates {r1, r2, r3, r4} chosen? Whether they are fixed or data-driven is not clear. \n\n#\n\n5. Line 132, FOSSIL reference is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jdDWMKqPa6", "reviewer": "Reviewer_wdgB", "summary": "The paper proposes OC-ZSS, a novel framework that builds on OTSeg’s framework and adds an object-centric front-end: DINO features are clustered to form attention masks that gate object prompts toward coherent patch groups in a frozen CLIP encoder; features are then iteratively refined by ORA before entering the OTSeg-style decoder. Results are solid across protocols, with especially strong numbers in the very-few-seen setting.", "questions": "1. It would be helpful to also report the computational costs of online end-to-end inference, including the DINO feature extraction and clustering/masking.\n2. The improvements in the very-few-seen setup are impressive, and it would be very helpful to provide more insights into what contributes to this gain. An ablation study under this setup would be very helpful.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The modular design of the newly proposed components should be straightforward to adapt to existing CLIP-based frameworks. The experiments are thorough, and the qualitative results align with the object-centric motivation. The ablation studies are helpful and generally consistent.", "weaknesses": "1. One of the main proposed components is the SSL DINO encoder for attention mask generation. This adds significant memory and computation costs. However, the authors reported this part in a somewhat tricky way: Params/GFLOPs and throughput are reported with DINO features pre-extracted and with GPU-side tensor indexing, so the cost of **online** DINO inference + clustering/masking isn’t in the headline numbers. That makes the method look closer to OTSeg than an end-to-end deployment would. The inference of DINO is likely to be costly in some deployment environments where special GPU acceleration algorithms are not available.\n2. When the attention mask for object prompts is off, performance looks about the same as OTSeg; it is possible that the main performance gain comes from the strong semantics of the DINO encoder rather than other proposed components.\n3. The results in Table 5 are compelling, but they don’t show readers where the few-seen gains come from. A reasonable guess is that DINO’s semantics carry a lot of the weight there, but the paper doesn’t actually test that. Running an ablation study under this setting would be very helpful for more insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "207ORGE7tK", "reviewer": "Reviewer_JToE", "summary": "The paper proposes OC-ZSS, a method for zero-shot semantic segmentation that injects object-centric priors into a frozen CLIP visual encoder. It introduces object prompts guided by self-supervised DINO features, which generate attention masks corresponding to distinct objects. These prompts guide CLIP’s attention toward object-centric regions.\nA novel Object Refinement Attention (ORA) module is designed to iteratively refine both object and patch features in a two-stage manner (object → patch). Additionally, granular attention integrates multi-scale receptive fields via dilated depthwise convolutions to improve scale robustness.\nExtensive experiments on PASCAL VOC, PASCAL Context, and COCO-Stuff show state-of-the-art performance across inductive, transductive, and cross-dataset settings while maintaining low parameter and compute overhead.", "questions": "1.Choice of object prompt number (no): How is the number of object prompts determined? Could it be dynamically adapted based on image complexity or feature entropy?\n2.ORA iteration steps (S): Why is S = 4 chosen as default? Could an adaptive stopping criterion (e.g., convergence of feature distance) reduce computation?\n3.Handling overlapping/merged objects: How robust is the method when DINO clusters merge multiple objects into one? Is there any mechanism to re-estimate or split masks during refinement?\n4.Backbone transferability: How does OC-ZSS behave when plugged into stronger backbones (e.g., DINOv2-Large, EVA-CLIP)?\n5.Runtime analysis: Can the authors provide wall-clock inference times (ms / image) and throughput comparison with OTSeg or CLIP-RC?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Originality: The idea of dynamically generated, self-supervised object prompts is novel and effectively bridges self-supervised and multimodal learning.\n2.Empirical quality: SOTA performance on inductive, transductive, and cross-dataset tasks; solid ablations and visualizations (cleaner masks, sharper object boundaries).\n3.Clarity: The modular structure (Object Prompt + ORA + Granular Attention) is well-motivated and easy to follow.\n4.Significance: Achieves high accuracy with only 27M parameters and 64 GFLOPs, showing strong potential for practical deployment and low-supervision adaptation.", "weaknesses": "1.Dependence on SSL masks: The approach relies heavily on DINO-generated clusters. Performance robustness across different SSL models and clustering hyperparameters (K, no) could be further analyzed.\n2.Limited theoretical insight: The convergence and dynamics of the iterative ORA process are not deeply analyzed. More discussion on when ORA may fail (e.g., ambiguous or overlapping objects) would strengthen the paper.\n3.Evaluation granularity: Results on small/overlapping objects or fine-grained categories are underexplored.\n4.Comparisons with stronger backbones: Although experiments with DINOv2-Reg and other CLIP variants are provided, additional integration with high-capacity decoders could better reveal the upper bound and generality of OC-ZSS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)