OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts
Authors: Shiting Xiao
Rishabh Kabra
Yuhang Li
Donghyun Lee
Joao Carreira
Priyadarshini Panda
Venue: NeurIPS 2025 spotlight ()
Keywords: Image segmentation
Segment anything model
Vision-language model
Primary area: deep_learning

Abstract: The ability to segment objects based on open-ended language prompts remains a critical challenge, requiring models to ground textual semantics into precise spatial masks while handling diverse and unseen categories. We present OpenWorldSAM, a framework that extends the prompt-driven Segment Anything Model v2 (SAM2) to open-vocabulary scenarios by integrating multi-modal embeddings extracted from a lightweight vision-language model (VLM). Our approach is guided by four key principles: i) Unified prompting: OpenWorldSAM supports a diverse range of prompts, including category-level and sentence-level language descriptions, providing a flexible interface for various segmentation tasks. ii) Efficiency: By freezing the pre-trained components of SAM2 and the VLM, we train only 4.5 million parameters on the COCO-stuff dataset, achieving remarkable resource efficiency. iii) Instance Awareness: We enhance the model's spatial understanding through novel positional tie-breaker embeddings and cross-attention layers, enabling effective segmentation of multiple instances. iv) Generalization: OpenWorldSAM exhibits strong zero-shot capabilities, generalizing well on unseen categories and an open vocabulary of concepts without additional training. Extensive experiments demonstrate that OpenWorldSAM achieves state-of-the-art performance in open-vocabulary semantic, instance, and panoptic segmentation across multiple benchmarks. Code is available at https://github.com/GinnyXiao/OpenWorldSAM.

Open WorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts
Shiting Xiao Yale University
Rishabh Kabra Google DeepMind rkabra@google.. com
Yuhang Li Yale University yuhang. le@yalllleee... edu
ginny. www.xxiaiiieee...o
ginny.xiao@yale.
Donghyun Lee
João Carreira Google DeepMind joaoluis@google.c com
Priyadarshini Panda
Yale University hew.yyuw..eeeyyleee...oo
Yale University priya. paniyaa.and@@@al.... edu
Abstract
The ability to segment objects based on open-ended language prompts remains a critical challenge, requiring models ground textual semantics into precise spatial masks while handling diverse and unseen categories. We present Open WorldS framework that extends the prompt-driven Segment Anything Model v2 (SAM2) to open-vocabulary scenarios by integrating multi-modal embeddings extracted from lightweight vision- languaee model (VLM). Our approach is guided by four key principles: i) Unified prompting: Open WorldSAMM supports diverse range of prompts, including category-level and sentence-level language descriptions, providing flexible interface for various segmentation tasks. ii) Efficiency: By freezing the pre-trained components of SAM2 and the VL.M, we train only 4.5 million parameters on the COCO-stuff dataset, achieving remarkable resource efficiency. iii) Instance Awareness: We enhance the model's 's spatial understanding through positional tie-breaker embeddings and cross-attention layers, enabling effective segmentation of multiple instances. iv) Generalization: OpenWorldSANN exhibits strong zero-shot capabilities, generalizing well on unseen categories and an open vocabulary of concepts without additional training. Extensive experi- ments demonstrate that OpenWorldSAN achieves state-of- of-the-art performance in opca-vocabulary semantic, instance, and panoptic segmentation across multiple benchmarks. Code is available at http://github.com/innnny
Introduction
settings, where models can only recognize objects from predefined taxonomy [1, 2, 3, 4, 6, 7, However, real-world applications, e.g., Embodied AI [9, 10], demand systems that understand open-ended language descriptions (from single nouns like "pedestrian" to rich referring expressions such as "the man in a red shirt") and segment novel objects unseen during training. This open-vocabulary segmentation problem poses two core challenges: (1) Semantic grounding mapping free-form text to visual entities, and (2) Instance awareness distinguishing multiple objects that match the same description.
Image segmentation has long been constrained to
Detection-centric methods [11, 12] relied on two-stage pipelines, first detecting class-agnostic mask proposals then classifying them with vision- language models (VLMs), e.g.. CLIP [13] and ALIGN [14]. While effective, such approaches struggle with complex queries and specialize exclu- sively in semantic segmentation, lacking versatility. Recent generalist models [15, 16] explore unified architectures that jointly handle vision and language, allowing single model to perform detection, segmentation, and grounding tasks. These generalist models demonstrate impressive flexibility, but they typically involve essurcce-ittensvv pre-training. The emergence of promptable segmentation models like the Segment Anything Model (SAM) [17, 18] offered new possibilities introduced
vision-language
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Visual Understanding
Language Understanding
Open- Vonabuluarury Category-level Prompts
Visual Prompts
Instance
Panoptic
Semantic
Sentence- Level Prompts
Interactive (Point, Box)
Tent aramoa
Referring Expression
SAM2
Open WorldSAM
Figure 1: Overview of the proposed framework. The green region highlights the SAM v2 baseline, supporting visual prompts (e.g., boxes, points) for interactive segmentation. Our OpenWorldSAM extension integrates open-vocabulary language understanding, enabling hoth category-level segmentation across semantic, instance, panoptic tasks and referring expression segmentation.
paradigm shift by allowing users to segment arbitrary objects using simple visual prompts (e.g., points, boxes). Trained on an extensive dataset, these models exhibit remarkable generalization and interactive capabilities. However, they inherently lack semantic understanding. Subsequent attempts to combine SAM with large language models (LLMs) [19, 20, 21J achieved language awareness but at prohibitive computational costs, imposing overwhelming overhead.
We posit that an ideal open-vocabulas segmenter should: (i) Natively support textual prompts without cascaded classification components, (ii) Preserve the knowledge of the vision foundation models like SAM without adding large overhead, and (iii) Segment multiple possible instances that could correspond to a single query. To this end, we propose Open OpenWorldSAM, an open-vocabulary extension to the SAM v2 (SAM2) architecture that satisfies these requirements. Open AM injects language understanding while retaining SAM2's core strengths through lightweight language adapter (=4.5M trainable parameters), unifing category-leve instance, semantic, and panoptic segmentation, and sentence-level referring expression segmentation (Figure 1).
Specifically, feed the image and descriptive text input into a frozen multi-modal encoder and obtain fused semantic representations. These serve as prompts to SAM2' mask decoder that produces masks for any described object or region. We introduce positional tie-breaker mechanism to resolve ambiguities when a text query could apply to multiple regions, allowing the model to perform multi-instance segmentation. Furthermore, our adapter employs soft prompting technique that uses tross-attention between textual queries and image features, sharpening localization by allowing semantic contexts to focus toward relevant image areas. By combining these design innovations, Open WoreWSoMSSMMM can accurately identify and segment arbitrary objects described by text, all while using only frozen pre-trained encoders and a tiny trainable adaptation module.
In summary, Open AMen represents a new paradigm of seggmett anything in the open world". It inherits SAM's interactiveness while being guided by flexible language prompts. Our contributions include:
VOC-20 (mlo1
ADD-857 (mloU)
1. We introduce that supports various open-vocabulary segmen- tation tasks. We propose an efficient language adapter with tie-breaker and cross-attention soft prompting, improving multi-object localization.
a unified interface
PC-59
ADE-150
(imol))
45
(mloU)
PC-459
ADB-150
(mloU)
(PQ)
13
10
2. OpenWorldSAl achieves state-of-the-art zero- shot performance across six benchmarks (Fig- ure 2), setting new standard for open-vocabulary segmentation (e.g., 60.4 mloU on ADE20K [22]). OpenWorldSAl also acheives strong perfor- mance in referring expression segmentation (74.0
50
SUN-37 (mloU)
SCAN-10
41
31
(mloU)
SCAN-20
SCAN-20
(mdoU)
(PQ)
OpenWorldSAM
APE
MaskCLIP
cloU on RefCOCOg (23]) with substantially fewer resources compared to recent models.
D-Deerrer
OVSeg
OpenSeeD
Figure 2: OpenWorldS AM achieves new state-of- the-art on six datasets with one suite of parameters.
Our work demonstrates that lightweight architec- tural interventions can unlock zero-shot segmenta- tion capabilities rivaling specialized models while preserving SAM2's efficiency and interactivity.

---

"zebra"
"zebra"
(a) outputs point prompt
OpenWorIdSAM w/ tie-breaker
(6) tie-breuker
Figure 3: (a) SAM takes a visual click and outputs valid masks on the same person (the person, the backpack, and a backpack region) [17]. It will not output masks for the person standing next to her. (b) Tie-breaken shift the queries to distinct regions, enabling simultaneous segmentation of all three "zebra" instances. (c) Naïve approach [43]: single language query for "zehra" causes SAM2 to segment only the most salient instance.
2 Related Work
Open-vocabulary segmentation. Recent advances in open-vocabulary segmentation have lever- aged vision- vision-language models (VLMs) [13, 14] to overcome the constraints of traditional closed-set segmentation models. Early approaches like LSeg [24], RegionCLIP [25] and OWL-ViT [26] estab- lished a baseline by introducing contrastive learning framework to align image embeddings with CL.IP-based text embeddings for zero-shot detection/segmennttioo Subsequent methods [11, 27] scaled effectively by using weak supervision of large-scale images with captions (up to millions of regions) or text-only signals, enabling more flexible and broader semantic coverage. Two-stage approaches like MaskCLIP [28] and OVSeg [12] further refined this paradigm by generating mask proposals using MaskFormer [29] followed by CLIP-based classífication, notably boosting accuracy through mask-adapted fine-tuning. Another line of works formulated this task visual grounding problem and established region-text fusion [30, 31, 32, 33]. More recently, unified architectures such ODISE [34], X-Decoder [15], SEEM [35]. OpenSeeD [36], HIPIE [37], Cemantic-SAM.. [38] and APE [16] have integrated multiple segmentation tasks into single framework, showing significant progress towards ceneerl-purposs, models, but they typically required resource intensive pre-training.
Extending SAM for text-prompted segmentation. The Segment Anything Model (SAM) [17, 18]
achieved breakthrough in promptable segmentation by training on billion masks, enabling it to generate high-quality masks for visual prompts. flurry of recent works have explored infusing SAM with semantic or language understanding to move beyond its original prompt types. Grounded- SAM [39] is a pioneering effort that combines an open-vocabulas
detector DrONndingDINO [31] to generate bounding boxes from text query, then feeds those boxes as prompts into SAM. Fast- SAM [40] matches CLIP embeddings with regions of interest. LLM-centric works [41, 20, 19, 21] attempt to map large LLMs or VLMs language embeddings into SAM or SAM-like decoder's prompt latent space to enable referring expression segmentation. Among these, LISA [19] pioneered "mask- as-text-embeddinng" approach but was limited single-object queries. LISA [42] introduced instance awareness through additional instruction-tuniig data, though it requires LLMS to explicitly enumerate objects-22 computationally expensive process. EVF-SAM [43] recently demonstrated lightweight alternative, integrating SAM with a multi-modal BEiT-3 encoder [44] (O.7B parameters). While achieving state-of-the-ar referring segmentation accuracy with minimal parameters. it remains constrained to single-object queries. Inspired by the success of EVF-SAM, we enhance SAM further into the domain of open-vocabulary segmentation, where the goal is segment and label all objects ("things" and "stuff") in the scene with open-set categories.
3 Methodology
Motivation and key challenges. A fundamental limitation of SAM-like architectures! is their inability to resolve multi-instance ambiguity from a single prompt. While visual prompts (e.g., points) may occasionally lack granularity specificity for instance, click on a backpack could imply segmentation of either the backpack or the entire person (Figure 3a)-they inherently localize to single spatial region. Language prompts, however, introduce a distinct challenge: a text query like "zebra" may correspond to multiple spatially disjoint objects (Figure 3b), with no prior knowledge of instance counts. Prior attempts to add language capabilities either rely on segmentation-thee classification pipelines (losing end-to-end training) or require costly region-level text grounding
3

---

Prompea "giraffee" "rock"
[Ground Truth Masks]
Humgarian Matching
MLP
Input: (.2)
Input: (F)
Norm
SAM2 Mask Decoder
2 Multi-Medal Encoder
Image
Encoder
Cross Atteation
SAM2 Prompt Encoder
JiPing)
xv
T
MLP Projector
Output
Image festures]
Quiput: (1s]
Prompting Transformer
Self-Attention
[Visual-cest amnhedding)
(pex54)
K)
f
Copy KX
Norm
(Tie-brenkers)
Airullc
Mrock
Copy 2N
Rg. R2
3x
(a) Inputs & Encoder Outputs
(b) Model Pipeline
(c) Soft Prompting Zooming in
Figure 4: (a) Preliminaries on the inputs and outputs of the vision and multi-modal encoders. (b) Open World-
SAM pipeline. (c) Detailed soft prompting Transformer architecture.
during pre-training. Our key insight addresses this gap: SAM2's mask decoder can inherently segment multiple instances if equipped with diverse positional guidance, i.e., learned cues that disentangle identical semantic queries into spatially distinct segmentation targets.
Architecture overview. Figure depicts our framework which comprises: (i) hierarchical SAM2 image encoder that extracts image features, (ii) a multi-moda vision-language encoder that jointly ingests the image and text prompt, (iii) lightweight MIPP projector, (iv) learnable positional tie- breakers for multi-instance queries, (v) a soft prompting Transformer block that aligns text-image features with SAM2's image features, and (vi) the SAM2 mask decoder producing final masks. Only small language adapter with components (iiivv) is trained; all other backbones remain frozen.
Multi-modal encoder. We leverage BEiT-3 [44] encode the input description into a semantic embedding. Given an image and a text prompt T (e.g., category name or referring expression), we feed both modalities into BEiT-3's 's encoder to obtain joint visual-text embeddings. Concretely, tokens of T and patch embeddings of a downsampled I are concatenated and processed by BEiT-3, yielding set of feature vectors [ficLs]. .i....; }. We take the classification token f(CLS) as a compact summary denoted as Plang of the prompt conditioned on the image content.
We adopt BEiT-3 because its carly-fusion training on image-text pairs equips it with rich, bidirectional cemuntics-cruuial for reasoning about unseen classes. Compared with CLIP-style contrastive image- text matching using only the features from the last encoder layers, BEiT-3 exposes finer cross-modal interactions. By embedding the text while it sees the image, the encoder already localizes the concept loosely (e.g.. "giraffe" vs. "rock" in Figure 4) before any downstream segmentation, preventing the mask decoder from learning semantics from scratch.
Prompt projection. BEiT-3 emits 1,024-D tokens, whereas SAM's prompt channels 256-D. A two-layer MLP acts as a projector that (i) preserves the coarse semantics of Plang E Rdi024 and (ii) learns to highlight dimensions that are most useful for mask prediction: u MLP E d2s0,
Positional tie-breaker and multi-instance queries generation. The projected visual-text embedding captures what to segment but lacks awareness of how many instances exist and where they are. To enable multi-instance segmentation, we propose K learnable positional tie-breaker vectors (t....., tx) C Rdaes that perturb u into K distinct queries:
qi u -utte t., i= K.
(I)
These perturbations serve two purposes: 1) Positional disambiguation: Each t; nudges the query towards different spatial regions (Figure 3b), mimicking how human annotators might click different points to segment each zebra. 2) Instance diversity: The tie-breakers are optimized during training to maximize coverage distinct instances, preventing query collapse. Conceptually these queries play the role of the "object queries" in DETR [45]. Crucially, they impose segmentation distinction for the same language semantics, making positional tie-breaking novel and key feature for OpenWorldSAN In practice K 20 covers 999% images in COCO [46]; for larger scenes K can be increased trivially.

---

Soft-prompting via cross-attention. The perturbed queries } interact with SAM2's image features through 3-layer Transformer [47] in Figure 4c, which alternates self-attention (queries talk to each other, promoting diversity) and cross-attention (queries look at image features). Each language-awaa query is refined on-the-fly by cross-attention with the frozen SAM2 features. SAM2's image encoder follows hierarchical vision Transformer ("Hiera" [48, 49]) that outputs three features (2256x256 F F128x F64x61) with 256², 1282, and 64 spatial resolutions, respectively. We operate on the level-3 features with 64 resolution as they optimally balance precision for retainaing boundary details and computational efficiency (16x cheaper than full-resolution attention). They are also used by SAM2 for mask decoding by default [18]. The soft prompting Transformer computes q' CrossAttn(( (q. F F61x61),
1..... K, whose key/value inputs are the flattened level-3 features F61x61 R R4096x256 256 This step grounds the language-awe aware queries in SAM2's high-resolution visual features, resolving ambiguities (e-g,, distinguishing adjacent zebras by stripe patterns).
Mask decoding and class assignment. The refined queries {q{} are input to SAM2's mask decoder alongside level-3 image features. We inject the queries as the prompt tokens in place of, e-g- point box prompts in the original SAM2's prompt encoder to obtain prompt embeddings. The prompt embeddings are then passed to the mask decoder which outputs K masks and corresponding confidence scores. We assign each mask the original text prompt 7 as its class label, since the generation is fully conditioned on T and thus inherits the semantic identity.
Training, All heavy visual (Hiera) and vision-language (BEiT-3) encoders are kept frozen to preserve their pre-trained knowledge and avoid costly retraining. Only the MLP projector, beebbrakers, and the soft prompting Transformer are learnable. For each training sample and prompt, we match the K predicted masks to the ground-truth masks of class T via Hungarian matching [45], then apply a focal loss, encouraging precise segmentation of all instances described by the prompt. The tie- t, E Rdase are implemented as learnable parameters randomly initialized from a normal distribution. During training. the Hungarian matching loss naturally encourages each t, to specialize in different spatial regions. Notably, this mechanism requires no explicit supervision about instance counts.
Inference. From the predicted K masks, we derive results for three segmentation tasks: semantic, instance, and panoptic. For semantic segmentation, we merge masks sharing the same class label, weighted by their confidence scores. For instance segmentation, we apply confidence-scom filtering to remove masks below certain threshold, followed by non-maximum suppression (NMS) to eliminate highly overlapping masks and retain distinct object instances. Similarly, for panoptic segmentation, we perform confidence-bart based filtering and NMS, ensuring each pixel is uniquely assigned to either a "thing" (instance) or "stuff"" (semantic) label.
Optionally, we perform a two-stage inference. In this setup, masks obtained from the first inference stage are used as visual prompts fed back into SAM2's mask decoder, which refines mask contours. Qualitatively, two-stage inference improves the precision of mask boundaries for correct predictions (Appendix B). However, quantitative analysis (Table 1) reveals that the second inference stage provides minimal improvements in segmentation metries, suggesting it mainly enhances visual quality rather than overall accuracy.
4 Experiments
Datasets and metrics. We train OpenWorI on the COCO2017 [46] dataset with panoptic annotations following X-Decoder [15]. The training set contains 104k images. We evaluate the model in a zero-shot setting eight segmentation tasks across five diverse datasets: ADE20K- 150/857 [22], PASCAL VOC-20 [50], PASCAL C09text-59/459 [51], ScanNet-20/40 [52], and SUN-RGBD-37 [53]. Evaluation metrics include panoptic quality (PQ). mean average precision (mloU), corresponding to panoptic, instance, and semantic segmentation tasks, respectively. For referring segmentation, we finetune RefCOCOg UMD training split. Following prior works, we report the cumulative intersection over the cumulative union (cloU) metric on the RefCOCOg UMD validation split.
(mAP), and mean intersection-over-ume
Implementation. We implement our model in PyTorch, We initialize the visual model with the weights of SAM22-iera--argg [18] and the VLM encoder with the weights of EVF-SAM BEIT-3- Large [43]. It is trained for 25 epochs on COCO-Stuff using the AdamW optimizer with a learning rate of le-4, batch size 8, on single NVIDIA A100 GPU. Image resolution is set to 1024 for
SAM2 and 224 for BETT-3. Number of postional tie-breaks is set to 20 for COCO dataset. Our implementation details can be found in Appendix A.
5

---

Ground truch
O22
X-Decoder
X-Decoder (Oracle)
(8)
XTTL DoneOTT, Serictory BISTTA SK11
218l 90 Tigirr Aurraoody, (TAtta 3201
NETRAR S21007
211
Ffantl:
MERCH IESleiia aomaataa scimeaasy MO
Figure 5: Qualitative comparisons on ADE20K-857. In many cases, (c.g.. (c) road, field), X-Decoder predicts semantically related but incorrect labels due to ambiguity in the category list. The final column shows X X-Decoder predictions using oracle prompts, which reduces confusion. OpenWorldSAM, conditioned on the correct prompt, produces faithful masks and avoids semantic mismatches. Color maps each model vary. Please refer to the predicted lahels. Best viewed with zoom in. We use two-stage inference for the visualization.
4.1
Open-Vocabulary Segmentation Evaluation Protocols and Challenges
Ambiguity of open vocabulary evaluation. Most prior open-vocabulary segmentation meth- ods-including X-Decoder [15]. OVSeg [12], and MaskCLIP [28]-adopt Global-Matching protocol: for each predicted mask, model matches it against the entire dataset vocabulary using precomputed text embeddings and selects the best-aligned class. However, this strategy can be problematic when applied to datasets like ADE20K, which contain hundreds of fine-grained and overlapping labels. As observed in OVSeg [12], this leads to semantically reasonable predictions being marked incorrect under exact label matching: "The ground- category is 'building' while our model predicts 'skyscraper'. This ambiguity stems from the inherent subjectivity of language: synonymous closely related concepts may be indistinguishable in visual context, yet only one is accepted by the ground truth. We observe similar issues in our own qualitative analysis. As shown in Figure 5, X-Decoder predictions ADE20K-85 often produce valid but non-canonical labels (e.g.. 'road' instead of 'runway', or 'screen' instead of 'arcade machine'), ) resulting in unfair penalization.
Oracle-Promee evaluation. To address this, we introduce alternative evaluation strategy: Oracle Prompts-during evaluation, we explicitly provide the ground-truth class names as prompts. This mimics the intended use of prompt-based models like SAM, which are inherently interactive and conditioned on user input. Under this protocol, the model does not have to resolve linguistic ambiguity across the full label space; it segments what the user asks for. We report results under both settings: Table shows baseline performance using the global matching protocol, consistent with prior works. Table 2 revisits X-Decoder under the oracle-prone macle-prompt protocol for comparison to Open A.M, which by design is evaluated under oracle prompts. We believe this approach provides more fair assessment of SAM-style models in open-vocabulary segmentation.
equitable
4.2 Open-Vocabulary Segmentation Performance Analysis
Zero-shot open-vocabulary transfer. OpenWSol generalizes out-of-the-bor to a broad set of segmentation tasks without any weight adaptation. As shown in Table 1, it achieves of- the-art performance across almost all datasets and evaluation metries. Its performance consistently surpasses strong baselines such as X-Decoder and APE, despite using only 4.5M trainable parameters. ADE20K-857, OpenWorldS achieves 33.1 1% mloU, outperforming the previous best (X- Decoder) by +23.9 absolute points (9.2 + 33.1). On PASCAL Context-459, it achieves 47.5% mloU,

---

Table 1: Zero-shot performance of open-vocabulary segmentation models across multiple benchmarks. For COCO, different methods use different supervisions of mask (m), class label (cls) and caption (cap). "ITP" indicates whether model uses image-texi pairs/referring data. "DET" indicates extra detection data (e.g- Objects365, LVIS, OpenImages, efc.) denotes the model has the capability for the task but does not have number reported. "-" means the model does not have the ability for the specific task. Purple color means fully supervised approach, and tan semi-supervised learning approach. Two-stage inference means we refine mask contours by re- peoproppingg SAM using the raw mask predictions, Bold entries indicate the best performance.
COCO (pl)
ADE-150
ADE-857
VOC-20
PC-59
PC-459 mloU
SUN-37
SCAN-20
SCAN-40
Model
Train Params
ITP DET
m els cap
PQ MAP mloll
mioli
mioli
mloli
mloli
PQ
Intoli
MSeg (B) [54)
(M)
V
V
V
x
33.7
326
19.1
73.4
43.4
29.5
33.4
GrouPVT (S) (27)
44 (M)
x
x
X
1 [84]
(M)
X
388
ZegFormer [55]
(M)
V
V
X
8.1
80.7
OpenSeg (B) [11]
86 (M)
V
X
V
V
X
26.4
8.1
70.2
44.8
11.5
OVSeg [12]
0.6 (M)
V
X
X
29.6
9.0
94.5
55.7
12.4
MaskCLIP (L) (28)
428
X
X
15.1
6.0
23.7
8.2
10.0
Ope (366
39 (M)
V
X
V
19.7
15.0
23,4
X-Deodde-Seg" Seg" (B) [1.5]
28 (M)
X
X
16.9
9.5
23.8
4.6
97.8
64.7
12.1
32.2
35.1
33,8
18.5
X-Decoder (15)
38 (M)
21.8
13.1
9.2
97.7
16.1
43.0
49.5
39,5
29.7
APF-B [16]
42(M)
26.4
23,5
9.2
95.88
ESC-Net [56)
451 (M)
x
x
X
41.8
18.1
98.3
65.6
27.0
OpenWorldSAM
(M)
x
X
35.2
16.9
60.4
33.1
98.0
73.7
47.5
67.7
65.0
41,9
55.6
reo-stage inference
4M)(M)
x
36.3
15.6
58.0
32.6
97.6
72.6
45.8
68.2
64.8
39.9
54.1
Table 2: Oracle-Prompei evaluation of open-vocabule segmentation models. We report the otatc-of-the-art (SOTA) model X- Deeooderr [15]'s performance under both evaluation protocols. Other methods are omitted either because: 1) they are not SOTA, or 2) they do not support pracle-prompe evaluation.
ADE-150
ADE-857
VOC-20
PC-59
PC-459
SLN-37
SCAN-40
Model
Evaluation Protecol
mloU
mloU
mJoU
mloU
mloU
mloU
mloU
X-Desoder (L) [15] Global-Mathinng (default)
29.6
9.2
97.7
64.0
16.1
43.0
29.7
X-Decoder (L)
Oraclo-Prompts
51.5
29.1
98.1
75.5
42.3
67.1
49.1
AM AM
Oracle-Prompss (default
60.4
33.1
98.0
73.7
47.5
67.7
55.6
improving over APE's 21.8% by +25.7 points, and on ScanNet-40, it reaches 55.6% mloU, +25.9 point improvement over X- X-Decoder's s 29.7%. On AP score we under-perform APE, which included extra detection datasets, e.g., Objects365 [57], in their training recipe for better localization.
We attribute our strong performance to the model's prompt-conditioned decoding mechanism, which directly leverages language input to guide mask prediction. This is particularly advantageous when the target concept is known at query time. In contrast, global retrieval-based models such as X-Decoder must resolve ambiguity across the entire vocabulary space, which introduces classification error. While one might argue that differing evaluation protocols confound the comparison, it's important to note that both families of models require the same semantic input- the only difference lies in when and how that input is used.
Oracle-Prompts evaluation. As SAM-style models are designed for interactive segmentation, oracle prompts closely reflect practical use cases- such as cuman-in-the-lowe annotation, robotic object search, or dynamic UI feedback. To fairly compare with the state-of-ther oh-the-art art generalist model X- Decoder [15], we also evaluate under oracle prompts: we restrict its vocabulary to the ground-truth classes for each image. As shown in Table 2. Open these controlled conditions. Notably, on large-vocabulary datasets such as ADE20K-857 and PASCAL Context-459, OpenWorl achieves 33.1% and 47.5% mloU, surpassing X-Decoder by +4.0 and +5,2 points, respectively. This highlights our model's 's superior language grounding ability in long- tailed, fine-grained category distributions. On smaller datasets like PASCAL Context-59 Comtxt_C and PASCAL VOC-20, where most categories overlap with COCO, X-Decoder slightly outperforms our model (75.5% vs. 73.7% mloU and 98.1% vs. 98.0%), suggesting it benefits more from class memorization in such settings. Moreover, Figure illustrates that global matching often fails despite producing correct masks. Conditioning on oracle prompts significantly reduces this ambiguity, highlighting the robustness of our evaluation protocol and the effectiveness of prompt- based segmentation.
continues to outperform even under
Qualitative Results Figure 5 presents example outputs of Open WoelWSr on challenging scenes, with comparisons to X-Decoder under both evaluation protocols. In one example 5(a), an image from ADE20K-88 857 containing game room scene is segmented by our model using prompts for
7

---

Figure 6: Qualitative results RefCOCOg. OpenWoridSAM is capable of understanding spatial relationship, colors, actions, and shapes, etc.
Table 3: Referring segmentation performance (cloU) comparison on RefCOCOg benchmark validation set between proposed OpenWorldSAM and prior SOTA methods. We abbreviate the datasets: C (COCO), RC (Re(COCO/+), RCg (RefCOCOg), PL (PACO-LVIS), O365 (Objects365), V (Video segmentation datasets), OID (OpenImages Detection), (Visual Genome), ADE (ADE20K), PP (PASCAL--Part),
(PASCAL-VOC), We compare model trainable parameters, model capabilities (OV seg (open-vocabulary segmentation) and Inter denotes an estimate of the trainable parameters, LoRA [58] with rank- 8/16 adapters for finetuning.
Seg (interactive segmentation),, and training data required.
since these models
Method
Foundation Model
Train Params w/ SAM? OV Seg? Inter Seg? Training Data
cloU
X-Deeoder (L) [15]
CLIP-B [13] (63M)
38 (M)
RCg. Cap4M
SEEM (L) [35]
CLIP-B [13] (63M)
39 (M)
X
C. RC, RCg. PL
63.6
PolyFer (L. [59]
BERT.B [60] (104M)
450 (M)
X
X
RC, RCy
71.2
UNINEXT (H) (32)
BERT-B [60] (104M)
673 (M)
X
V
C. RC, O365, V
74.4
APE-B (1.)
CLIP-L. (123M)
42 (M)
C. PC, 0365, OID, VG, RC, RCg
63.5
PixeLLM
LLaMA2 (13B)
29 (M)"
ADE,
LISA [19]
Vicuna [63] (7B)
(M)"
X
RC, ADE, PI, PP
66.4
GLaMM [20]
Vieuna (7B)
GranD
u-LLaVA [2]]
Vicuna [63] (7B)
(M))
C, RC, ADE, PL, PC
u-LLaVA [21]
Vieuma [63] (7B)
(B)
X
C. RC, ADE, PL, PC
74.8
Sa2VA [64]
InternVL2 [65] (1B)
22 (M)"
X
RC, RCg. V. GranD
72.3
Sa2VA [64]
VnterVL.2 [65] (4B)
35 (M)"
X
V
RC, RCg. V. GranD
74.1
SAMWISE [66]
RoBERTa [67] (125M)
5.6 (M)
X
RC
67.3
EVF-SAM [43]
BFIT-3-1. [44] (0.7B)
674
RC
77.0
OpenWorldSAM
BETT-3-L
(M)
RCg
various objects ("ceiling, light, seat, person, arcade machine"). Open WorlWWoM accurately masks each object and stuff region, whereas X-Decoder misclassifies the "arcade machine" due to confusion between similar semantic objects under Matbal-Matching, and produces fragmented masks for the person and seat under Oracle-Prompts. Similarly in example 5(b), X-Decoder misclassifies the "wall" and proposes object masks for prompts that did not exist in the ground truth (e.g., "window glass") under Mooobll-atchhig,. and failed to segment "plant" under Dracle-Prompt, This showeases our model's 's clear understanding of category semantics (thanks to the VLM prompt) combined with precise mask delineation (thanks SAM2's capability). More qualitative results in Appendix C.
4.3 Referring Expression Segmentation Performance Analysis
Performance. As shown in Table 3 and Figure 6, OpenWorldSAM achieves strong performance on the RefCOCOg validation set, obtaining cloU of 74.0%, significantly outperforming earlier generalist models like SEEM and X- X-Decoder (=65%) and competitive with specialized models such as GLaMM (71.2%) and UNINEXT (74.4%). Notably, OpenWorldSAM reaches this accuracy using just BEiT-3 encoder with 673M parameters and an additional 4.5M trainable parameters, substantially fewer than recent large-scale models like LISA, GLaMM, and u-LLaVA, which rely on much larger vision-language foundations (7B+ parameters) and multiple additional datasets. While EVF-SAM achieves higher cloU (77.0 (7.0%)) WorldSA inherits SAM's interactive features, offering unique flexibility interative segmentation tasks. Furthermore, with its tie bie-breaker and soft- poft-prompting modules, Open AM can also perform general segmentation tasks, distinguishing it from higher-scoring yet narrower models.
4.4 Ablation Studies
We systematically validate OpenWorldSAMss design through zero-shot transfer on ADE20K-150PP benchmark and fintuning on RefCOCOg benchmark.

---

Table 4: Ablation the VLM choice, e.g., CLIP [13] model from OpenAL. 'V' denotes modality used during training, and 'X' unused. Only the adapter modules are trainable, and the VLMs are kept frozen. Late fusion means we concatenate text/image features from the last layers of CLIP's text/image encoder. Early fusion means BETT-3 processes both modalities in all 24 Transformer layers.
ADE-150
ReiCOCOg
Encoder
Params
Text
Image
Modality Fusion
PQ
mloU
mloU
cloU
CLip-Large
123 (M)
13.5
2.9
25.7
12.8
25.2
CLIP Large
(M)
(Last layer Concat)
25.3
BEBT: Largg
(M)
13.6
BErT-3-Large
673 (M)
Early (All-layer Attcution)
35.2
16.9
60.4
33.1
74.0
Table S: Ablation on trainable and inference modules. For training, frozen. For inference, 'v' denotes activate, and 'x' denotes non-activate.
V denotes trainable, and 'x' denotes
Inference Modules
ADE-150
Exp
Train Params
Tie-breaker BETT- Cress-Attn
MLP Projector
Tie-breaker BETT- Cross-Atth
PQ
AP
mloU
mloU
(M)
x
(M)
x
(M)
x
35.1
56.8
32.2
E4
X
674.0 (M)
x
13.6
3,5
24.4
10.6
E5
4.5 (M)
v
35.2
16.9
60.4
33.1
86
V
V
677.2 (M)
13.9
388
23.6
11.2
Multi-modal encoder analysis. In Table 4, compares performances using different VLM encoders and fusion methods (early fusion vs. late fusion). BEiT-3's early cross-modal fusion (joint text-image processing across all layers) outperforms CLIP's late fusion (last-layer concatenation) by +33.9 mloU, +21.2 PQ, and +13.3 AP on ADE-150, demonstrating that deep semantic integration is critical for aligning language concepts with visual regions, echoing findings by EVF-SAM [43].
Visual Context Matters. Table 4 demonstrates that removing visual inputs to BEiT-3 (text-only) causes catastrophic performance collapse (-34.4 mloU on ADE-150), This confirms that SAM's segmentation backbone cannot ground textual semantics without explicit visual-textual co-encoding.
Optimal Training Strategy. In Table 5, we varied the trainable modules in OpenWorldS AM (thus varying total new parameters from 1.2M to 770M). We found in E5 that freezing BEiT-3 and training only the language adapter module (tie-breaker pross-attention, 4.5M parameters) yields optimal performance (60.4 mloU ADE-150) Notably, comparing E6 vs E5 and E4 vs E3, we found fine-tuning the entire BETT-3 encoder (673M parameters) significantly degrades accuracy (mloU drops from 60.4 to 23.6), likely due to underfitting on sparse category label prompts compared to its original web-scale pretraining.
Positional tie-breaker vs. none. Comparing E3 vs E1 in Table 5, positional tie-breaker boosts AP from 1.0% to 17.1% %. As shown in Figure 3, without the tie-breaker, the model usually collapses on one instance of the class (especially if the one was particularly salient among others). This confirms the necessity of this component for reliable instance segmentation.
Cross-Attente layer removal. As shown in Table 5, E5 E3, removing the cross-attention layers expectedly led to inferior performance (-1.5 mIoU on ADE-150 and -0.9 mloU on ADE-857). This indicates that cross- cross-attentiee attention helps align prompts the intended visual regions.
5
Conclusion
bridges the gap between promptable segmentation and open-vocabulary understand- ing by unifying SAM's segmentation prowess with vision-language models' semantic grounding. This approach generalizes across tasks semaatic/isstanc//anngon ing practitioners a unified tool for real-world scenarios where novel objects and ambiguous queries are the norm. Three innovations drive this success: (1) Positional tie-breakers enable multi-instance segmentation from single-text queries, resolving a critical limitation of SAM-like architectures. (2) Cross-mode soft prompting dynamically aligns language semantics with SAM's visual space, ensuring precise localization without costly L.LMs. (3) Frozen foundation synergy leverages pre- trained knowledge from SAM and BEiT-3, proving that dense prediction tasks benefit much as
and prompts (nouns/sentences) offer-

---

classification from parameter- parameter-efffiiient adaptation. Beyond technical contributions, OpenWorldSANN advances paradigm for extending segmentation foundations: instead training monolithic models, strategic adaptation of frozen components achieves open-world readiness at minimal cost.
Acknowledgement. This work supported in part by CoCoSys, a JUMP2.0 center sponsored by DARPA and SRC, the National Science Foundation (CAREER Award, Grant #2312366, Grant #2318152), the DARPA Young Faculty Award and the DoE MMICC center SEA-CROGS (Award #DE-SC0023198).
References
[1] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[2] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention- MICCCA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. pages 234-241. Springer, 2015.
[3] Jlang-Chieh Chen, George Papandreou, Jasonas Kokkinos, Kevin Murphy, and Alan I. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions pattern analysis and machine intelligence, 40(4):834-848, 2017.
[4] Liang-Chiel Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018.
[5] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2966 2017.
[6] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077-12090, 2021.
[7] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- mask transformer for universal image segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1290-1299, 2022.
[8] Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung- Yeung Shum. Mask dino: Towards a unified transformer-bast based framework for object detection and segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3041-3050, 2023.
[9] Lei Fan, Jianxiong Zhou, Xiaoying Xing, and Ying Wu. Active open- open- vocabulary recognition: Let intelligent moving mitigate clip limitations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16394-16403, 2024.
[10] Sebastián Barbas Laina, Simon Boche, Sotiris Papatheodorou, Simon Schaefer, Jaehyung Jung, and Stefan Leutenegger. Findanything: Open-vocabulary and object-centric mapping for robot exploration in any environment. arXiv preprint arXiv:2504.08603, 2025.
[11] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmen- tation with image-level labels. In European conference on computer vision, pages 540-557. Springer, 2022.
[12] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. roccbuaoybyulary semantic segmentation with mask-adapted clip. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 7061-7070, 2023.
10

---

[13] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, PmL.R, 2021.
pages
[14] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Kun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- saiiunla representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[15] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15116-15127, 2023.
[16] Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all once for universal visual perception. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 13193-13203, 2024.
[17] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015 1026- 2023.
[18] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408. 00711, 2024.
[19] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579 9589, 2024.
[20] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji. AbdeIrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 13009-13018 2024.
[21] Jinjin Xu, Liwu Xu, Yuzhe Yang. Xiang Li, Fanyi Wang, Yanchun Xie, Yi-Jie Huang, and Yaqian Li. u-llava: Unifying multi-modal tasks via large language model. In ECA/ 2024, pages 618-625, IOS Press, 2024.
[22] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
[23] Varun Nagaraja, Vlad Morariu, and Larry Davis. Modeling context between objects for referring expression understanding, In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 792-807, Springer, 2016.
[24] Boyi Li, Kilian Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language- driven semantic segmentation. arXiv preprint S0Niv:22011.03444 2022.
[25] Yiwu Zhong, Jianwei Yang. Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- based language-image pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16793-16803. 2022.
[26] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, al. Simple open- ppccbboaayllary object detection. In European conference on computer vision, pages 728-755. Springer, 2022.
11

---

[27] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 18134-18144 2022.
[28] Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskelip: Masked deff-istillltion advances contrastive sunguage-image pretraining. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11995-11000 2023.
[29] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems, 34:17864- 17875, 2021.
[30] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang. Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10975, 2022.
[31] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li. Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-trainin vre-training for open-set object detection. In European Conference on Computer Vision, pages 38-55. Springer, 2024.
[32] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception object discovery and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15325-15336, 2023.
[33] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detelip: Dictionary-enrichel visual-concept paralleled pre-training for open-world detection. Advances in Neural Information Processing Systems, 05:9125-9135 2022.
[34] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2955-2966, 2023.
[35] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in neural information processing systems, 36: 19769-99782, 2023.
[36] Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1020-1031, 2023.
[37] Xudong Wang, Shufan Li, Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Hierarchical open-vocabulary universal image segmentation. Advances in Neural Information Processing Systems, 36:21429 01221429-21453 21453, 2023.
[38] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything at any granularity. arXiv preprint arXiv:2337.00476, 2023.
[39] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open- open-world models for diverse visual tasks. arXiv preprint 1.X1:22001.14159) 2024.
[40] Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything. arXiv preprint arXiv:2306.12156 2023.
[41] Yonglin Li, Jing Zhang, Xiao Teng, Long Lan, and Xinwang Liu. Refsam: Efficiently adapting segmenting anything model for referring video object segmentation. arXiv preprint axX1v:2307.009977.000 2023. 2023.
12

---

[42] Senqiao Yang. Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. An improved baseline for reasoning segmentation with large language model. CoRR, 2023.
[43] Yuxuan Zhang, Tianheng Cheng, Lianghui Zhu, Rui Hu, Lei Liu, Heng Liu, Longjin Ran, Xiaoxin Chen, Wenyu Liu, and Xinggang Wang. Evf-sam: Early vision-language fusion for text-prompted segment anything model. arXiv preprint arXi::2406.20076 2024.
[44] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as foreign lan- guage: Beit pretraining for vision and Asion-Language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19175-19186, 2023.
[45] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[46] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755, Springer, 2014.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[48] Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po- Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, et al. Hiera: A hierarchical vision transformer without the learning, pages 29441-29454. PMLR, 2023.
In International conference on machine
[49] Daniel Bolya, Chaitanya Ryali, Judy Hoffman, and Christoph Feichtenhofer. Window attention is bugged: how not to interpolate position embeddings. arXiv preprins #rXiv:2311.05613, 2023.
[50] Mark Everingham and John Winn. The pascal visual object classes challenge 2012 (voc2012) development kit. Pattern Analysis, Statistical Modelling and Computational Learning, Tech. Rep, 8,5):2-5, 2011.. 2011.
[51] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattem recognition, pages 1011-998.221 2014.
[52] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: annottead 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828 1828-5839, 2017.
[53] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567-576, 2015.
[54] John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi-domain semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2879-2888, 2099228882288 2020.
[55] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11583-1 1183-11592, 2022.
[56] Minhyeok Lee, Suhwan Cho, Jungho Lee, Sunghun Yang, Heeseung Choi, Ig-Jae Kim, and Sangyoun Lee. Effective sam combination for open-vocabulary semantic segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 26081-26090. 2025.
13

---

[57] Shuai Shao, Zeming Li, Tianyuan Zhang. Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale. high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8430-8439, 2019.
[58] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhut Yuanzhi Li, Shean Wang. Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[59] Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18653-18663. 2023.
[60] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume I (long and short papers), pages 4171 1171-4186, 2019.
[61] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 26374-26383, 2024.
[62] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[63] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-asa--juge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 044466955466622 2023.
[64] Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming- Hisngn Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501 04001, 2025.
[65] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo. Zheng Ma, et al. How far are to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101, 2024.
[66] Claudia Cuttano, Gabriele Trivigno, Gabriele Rosi, Carlo Masone, and Giuseppe Averta. Sam- wise: Infusing wisdom in sam2 for text-driven video segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3395-3405, 2025.
[67] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[68] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. nttps://gtt thw.b...tttuuuu/e wttc http:/ooworreeeecceeettetttttoot 2019.
[69] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset. In CVPR Workshop on the Future of Datasets in Vision, volume 2, page 1, 2015.
[70] Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, Trevor Darrell, et al. Bdd100k: A diverse driving video database with scalable annotation tooling. arXiv preprini 18X5::8805.04688 2(5)) 2018.
[71] Petra Bevandić, Marin Oršić, Ivan Grubišić, Josip Šarie, and Siniša Šegvić. Multi-domain semantic segmentation with overlapping labels. Proceedings of the IEEF/CVE winter conference on applications of computer vision, pages 2615-2624, 2022.
14

---

[72] Hangbo Bao. Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision- laioo-languag pre-training with cexture-of-modaltty-eee 32912, 2022.
Advances in Neural Information Processing Systems, 35:32897-
[73] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-ter aequence-to-sequenn to-sequence learning framework. In International conference on machine learning, pages 23318-23340 PMLR, 2022.
[74] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for variety of vision tasks. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pages 48 4818-4829, 2024.
[75] Quan Sun. Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale, arXiv preprint arXiv:2303. 15389, 2023.
[76] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-lane lsioonlanguuge encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025.
[77] Jiahui Yu, Zirui Wang. Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917. 2022.
[78] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.0673 06794, 2022.
[79] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[80] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. /4824, 2023.
[81] Shilong Zhang, Xinjiang Wang, Jiaqi Wang, Jiangmiao Pang, and Kai Chen. What are expected queries in end-to-end object detection? arXiv preprint arXiv:2206.012.. 2022.
15

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In our abstract and introduction, we talk about our contribution on proposing novel framework for open- open-vocabulary segmentation. We provide extensive experiments on comprehensive datasets to support this claim. We also conduct in-depth ablation studies that verifies the effectiveness of our model design.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations in Appendix D, which is about the model general- ization quality to outdoor scenes and self-driving scenes.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our work does include theoretical assumptions and proofs.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide the detailed methodology and experimental setup in Sections 3 and 4. Moreover, we provide all source codes to reproduce the results, including training scripts (detailed configurations included) and evaluation scripts (model checkpoints included). We will open source the code on GitHub after acceptance.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide the source codes as supplementary material. We provide instruc- tions that contain the exact command and environment needed to run to reproduce the results. We provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Justification: Training and test details can be found in Sections 3 and 4, and Appendix A.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
16

---

Justification: We did not include the error bars as we fix the random seed for every experi- ment, reducing the impact from data loader and other parameters initialization.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the details of computer resources Section 4. All experiments can be run single A100 GPU. We also provide analysis on trainable parameters in Section 4.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httts://neurrpal wwtppplicctttcciiiccccce.cccccte htt.p//p..c..thhtc......eeeeeee.
Answer: [Yes]
Justification: Our experiments conform the NeurIPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is social impact of this work.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Our model and its code development are based on baseline works which are credited in the paper. Our datasets are the standard benchmarks that are widely used in academia.
Justification:
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. Institutional review board (IRB) approvals or equivalent for research with human subjects
17

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
18

---

A Experimental Settings
A.1 Pre-training
We implement our model in PyTorch, building on the Detectron2 [68] framework. We initialize the base models with the public weights of SAM2-HieraLLarge and BErn-3-Large'. The model is pre-trained for 25 epochs on the COCO-2017 training split (104K images) [46]. We use the panoptic annotations, which provide pixel-accurate masks and category labels for all 132 thing and stuff classes. Training is conducted Optimization employs AdamW (learning rate le-4). A step decay scheduler drops the learning rate by factor of 0.1 at 89% and 96% of the total iterations. Compared with recent generalist models, our recipe is markedly more data-cti eefta-fficcent (see Table 6).
single NVIDIA A100 (80 GB) GPU with a batch size of 8.
Table 6: detailed list of training data for generalist models and OpenWorIdSAM, 0365: Objects365. OID: OpenImages Detection, VG: Visual Genome. INB: ImageNetBoxes. RefC: RefCOCO/ +/g.
Data (grouped by annotation types)
Image Consumption
Method
Instance-level
Image-level
Batch Size
#Epoch
#Image
Batch Size #lter
X-Decoder [15]
COCO, RefC
Cap4M
32. 1024
200M (50 Ep Img)
OpenSeeD [36]
COCO,
32,
48M (30 Ep
Img)
(B) [16]
COCO, O365, OID, VG,
(16 x Tter)
OpenWorldSAM
COCO
8 2.50M (25 Ep x 104M Img)
A.2 Zero-Shot Evaluation
We evaluate semantic, instance, and panoptic segmentation in a zero-shot setting. For instance segmentation and panoptic segmentation, we apply confidence-score filtering remove masks with scores below 0.7, followed by non-maximum suppression (NMS) with IoU threshold 0.5 to remove duplicate detections and retain distinct object instances. The confidence scores, originally termed "estimated IoU scores" in SAM [17, 18], are direct outputs from SAM2's mask decoder. These scores were optimized during SAM2's 's pre-training to select high-quality (i.e., confident) mask outputs.
Table 7: Open-Vocaleeeee Vocabulary Segmentation Benchmark Statistics.
Annotations
Evaluation Dataset
type
Images
Classes
Semantic
Instance
Panoptic
ADE-150
common
V
2000
150
ADE-847
2000
847
Pascal Voc
common
X
X
1449
20
Pascal Context-59
X
5105
59
Pascal Context-459
common
5105
459
SUN RGBBDD
in-door
X
X
5050
37
ScanNet-20
in-door
5436
ScanNet-40
in-door
X
5436
40
The open-vocabulary benchmark comprises 5 datasets covering 8 different segmentation tasks; statistics are summarized in Table 7. We show a comprehensive evaluation protocol for open- vocabulary segmentation in various vocabulary sizes and image domains.
A.3 Finetuning
segmentation we fine-tune the pre-trained checkpoint on RefCOCOg UMD training split for 10 epochs. Because images from RefCOCOg were seen during pre-training (with category labels substituted for referring expressions ground truth), we adopt conservative learning rate of le-5. We use batch size of during training.
For
19

---

B Qualitative Comparison on Two-Stage-Inference
During inference, we perform an optional two-stage inference. First, the model predicts multi-instance masks. These masks then fed back visual prompts, and SAM2's mask decoder is a second time to refine the contours. Figure 7 illustrates the visual improvement. However, quantitative gains marginal across segmentation metrics (see Sec. 4.2 of the main paper), suggesting it mainly enhances visual quality rather than overall accuracy. The reasons are twofold: (1) Two-stage inference only refines mask contours; ItU-style metries saturate once coarse localization is accurate, so small contour tweaks seldom raise mloU/PQ/AP: (2) Errors will be amplified on hard examples. On incorrectly localized masks from stage 1, refinement anchored to incorrect regions further degrade metrics. Given that the two-stage inference serves as an optional, low-cost post-processing step, conveniently enable or disable it based their preference.
One-Stage Inference
Two-Stage Inference
cEiling
ceiling
Itor, monitoring dev
manitor, monitoring goo devi
Joor
boor
flooring
exhibitor someone, somebody, mortal, Soul
exhibitor
somconc, somebody. mortal soul
ticket
counter
Iloring
ceiling
colling
windowpane, window
Nindowpans nos window
Jam urtain, drage. drapery. mantle, pel.
urteinn drape, dranesy, V. mante, palit
Tpor. Toorii
Floor, Rioorn
carpet,
Figure 7: Qualitative results comparisons on using two-stage inference refinement on ADE20K-857.
C Additional Zero-Shot Qualitative Results
Figure 8 showcases multiple challenging indoor scenes drawn from ADE20K-150/857 [22], and PASCAL Context-459 [. In each sub-panel, we compare example outputs of Open WorldS with comparisons X-Decoder under both global-matching and oracle-prompts evaluation protocols.
Panel (a) (ADE20K- - 150) top row depicts cluttered bedroom. OpenWorldSA cleanly delineates thin structures such the "closet" edge and the narrow "lamp stem". and assigns single coherent mask to the "cushion". X-Decoder fragments the closet and mis-classifies the cushion as a generic "pillow" under global matching. Under oracle-prompee larly, the bottom row depicts an airport conveyor belt. X-Decoder mis-classifies the "bulletin board" the "crt screen", the "box" as the "trade name" under global matching, and still mis-classifies the "box" under oracle-prompts.
X-decoder fails to predict "cushion" Simi-
Panel (b) (ADE20K-857) top row shows dining area. Under the global-matching protocol, X- Decoder hallucinates "rug"/"rocking chair" labels and fragments the "sofa bed" pixels. The bottom row shows cluttered living room where X-Decoder outputs fragmented low-quality masks and false predictions under both evaluation protocols. indelity-inrroducc
comparison, our model preserves category extra labels-aa and produces noticeably cleaner chair boundaries, illustrating the synergy between BEiT-3 language grounding and SAM2's high-resolution masks.
Punel (c) (PASCAL Context-459) top row shows that X-Decoder fails to predict the "cloth" object. The bottom row is an indoor scene crowded with small objects ("'d", "speaker", "chair"). OpenWorld World-
SAM retrieves almost every queried category (except for "cd") and suppresses false positives such as
20

---

"calendar" and "ladder" that appear in X-Decoder's output, demonstrating stronger open open-vocabulary grounding and sharper instance separation.
Ground truth
OpenWoridSAMB
X-Decoder
X- X-Decoder (Oracle)
TXX
Lr. Iatan bocee
SEaeyo EXIiE (STA922 7214 POTT EeB
EOTTANY Sal CONNNES CONURTST
(a) ADE20K-150
Ground truth
OpeaWorldSAke
X-Decoder
X-Decoder (Oracle)
L.X. TAEES
eirg
collin
arting DRANO EATREAANIE
0119) re/Oui
DIT PATE UEO CEA.AAAAI AT3SA 2903
curtain, drape, drapery.
RDE SEC
Rooring
(b) ADE20K-857
Ground truth
Open WorldSAM2
X-Decoder
X-Decoder (Oracle)
AA
wali
300
Dellin
celling
cictune
picture
calendar
LISHTE
light
piment
book
book
person
cd
person
eabinel
abinet
abiinet
cable
abinet
chair
chair
chair
chair
flcor
floor
floor
floor
(c) PASCAL 455
Figure 8: Qualitative comparisons between X-Decoder [15] and OpenWarldSAM on ADE20K-150, ADE20K- 857, and PASCAL C5n.
21

---

D Limitations Outdoor Generalization
Despite strong results on indoor and everyday photographs, OpenWorldSAN under-performs on driving datasets such as Cityscapes [69] and BDD10K [70] (Table 8). Fine-tuning on Cityscapes narrows the gap, yet performance still trails methods explicitly exposed to multi-domain data. Understanding the source of this shortfall essential for future extensions.
Observed failure modes. Figure 9 shows high IoU for broad stuff regions (e.g. road, sky), but sharp drop for small elongated thing instances. Correspondingly, AP remains low for motorcycle, person, bicycle, etc.
Table 8: Outdoor performance. Open-vncabulary models are evaluated zero-shot on Cityscapes and BDD10K; the last row is fine-tuned on Cityscapes.
Cityscapes AP
BDD10K
Model
Evaluation
mloU
PQ
mloU
PQ
X-Decoder (L) [15]
zero-shot
52.0
24.9
38.1
47.2
17.8
Op
zero-shot
39.4
10.1
26.4
31.3
15.6
OpenWorldSAM
Finetune on Cityscapes
57.4
12.0
36.1
38.0
17.4
Pen-class IoU and AP on Cityscapes (sorted by loU)
1.0
basling irpesation
grifii sign
Meyele
nadoropria
Brtfic
Figure 9: Per-class loU And AP Cityscapes (sorted by IoU), Performance collapses thin or distant thing classes (e.g. person, traffic light).
Hypotheses.
(i) Domain shift. COCO images are mostly handheld and indoor, whereas Cityscapes/BDDIOQ 10K contain forward-looking dash-cam frames with motion blur, glare and night scenes. X-Decoder was co-trained on web-scale image-text pairs that include many outdoor photos, so its visual encoder has wider coverage. Large-scale multi-domain training is known mitigate domain shift [71].
(ii) Resolution bottleneck. Cityscapes frames are 2048x 1024. Rescaling to 1024x (SAM default) reduces poles and traffic lights nearly one pixel the feature stride of 16x. X-Decoder keeps an FPN branch at 8x, preserving thin structures.
Take-away. COCO-only pre-training for Open WorldSAM leaves blind spot for urban driving paassery-pattiuularr for distant, thin or cluttered objects under challenging lighting. Bridging the gap likely requires explicit exposure to outdoor domains and (ii) higher-resolution feature branches. We leave large-scale outdoor pre-training and depth-aware augmentation for future work.
E
Model Structure Details
Table 9 summarizes the architectural differences between OpenWorldSAM and competing models, detailing each method's visual backbone, segmentation head, text encoder, and training-image resolution.
22

---

Table 9: Architectural choices for recent open-vocabulary models: visual backbone, base segmentation model, text/multi-moda encoder, and training image size.
Image Size
Method
Visual Backbone
Base Model
Text Encoder
Short
Long
MSeg (B) [54]
HRNet-W48 (65 M)
HRNet-Seg
1024
1024
ViT (S) [27]
ViT-S/16 (22 M)
Group VIT
Transformer
224
224
ISee+ (B) [24]
CLIP ViT-B/16 (86 M)
DenseCI.Ir
CLIP
512
512
ZegFormer (B) [55]
CLIP ViT-B/16 (86 M)
ZegFormer
CLIP
640
640
OpenSeg (B) [11]
ResNet-101 (45 M)
OpenSeg
CLIP/ALIGN
640
640
OVSeg (B) [12]
CLIP VIT-B/16 (86 M)
MaskFormer
CLIP
640
640
MaskCLIP (L) [28]
CLIP ViT-L/14 (307 M)
MaskCLIP
CLIP
1024
1024
X X-Decoder [15]
DaVIT-L (196 M)
X-Decoder
CLIP
1024
1024
OpenSeeD [36]
Swin-L. (197 M)
MaskDINO
UniCL
1024
1024
SEEM [35]
DaViT-L (196 M)
X-Decoder
CLIP
800
1333
APE (B) [16]
VIT-L (307 M)
DETA
CLIP
1024
1024
PolyFormer (L) [59]
Swin-L (197 M)
PolyFormer
BERT
1024
1024
UNINEXT (H) [32]
VIT-H (632 M)
DINO
BERT
320~-80
1333
PixeILM [61]
CLIP ViT-L/14 (307 M)
PixeILM
LLaMA2-133
448
448
LISA [19]
SAM VIT-H (636 M)
SAM
Vicuna-7B
1024
1024
GLaMM [20]
SAM VIT-H (636 M)
SAM
Vicuna-78 7B
1024
1024
u-LLaVA [21]
SAM VIT-H (636 M)
SAM
Vicuna-78 7B
1024
1024
EVF-SAM [43]
SAM VIT-H (636 M)
SAM
BETT-3
1024
1024
EVF-SAM2 [43]
SAM2 Hicra-L (224 M)
SAM2
BEÎT-3
1024
1024
SAM2 Hiera-L (224 M)
SAM2
BETT-3
1024
1024
E.1 Possible Text Encoder Alternatives
We argue that the key ingredients for open-vocabulary segmentation are backhone-agnostic: any strong interactive segmenter can supply high-resolution mask decoding. while any pretrained vision-language encoder can provide semantics. What is missing is a lightweight adaptor that (i) aligns the two embedding spaces, (ii) scales multiple object instances from single text query, and (iii) preserves the efficiency that makes interactive segmentation attractive in the first place.
Our Open WorldSMAAAA is general plug-in architecture that satisfies these desiderata while keeping all heavy backbones frozen. Although we instantiate the framework with SAM2 and BEiT-3 in this paper, neither component is required by design; alternative interactive decoders vision-language encoders can be swapped in with only minor re-training of the adapter.
Table 10 surveys representative VLM encoders that could replace BEiT-3 in OpenWorldSAAl with <5 M adaptor parameters. All rows assume the heavy backbone is frozen; only the 256-D projector and tie-breakers are fine-tuned.
Adaptor fine-tuning recipe (all encoders). Freeze all VLM weights and SAM2 decoder; initialize dn 256 MLP projector and K 256-D tie- brekere embeddings (default K = 20, total =55M params). For training, one could use unchanged Hungarian matching loss on COCO.
Takeaway. Farly-fusion encoders (VLMo, OFA, Florence-2) require zero architectural change beyond projector resizing and therefore the most promising immediate swaps. Dual-encoders (CLIP family) need shallow cross-attention adaptor to overcome missing image context. Larger hybrids (BLiP-2, Kosmos-2, PaLI) open research directions (multi-query tie-breakers, OCR) the cost of real-time guarantees.
F Additional Ablation Studies
We provide additional ablation studies on the number of tie-breaker tokens and the number of cross-attention layers.
23

---

Table 10: Candidate vision- lang encoders. "TFM" stands for Transformer. "Pooled dim" is the size of the single semantic vector exposed to the adaptor: GFLOPs/Img" computed
GFLOPs/Lng
18.625
17.9/24.7
19.0
37.2
8.1
23.7
564
31.5
34,8
closest to BBTT-3 - just replace tokenizor di- mension projectors keep breakers changed
adjiss tnlenizer and change inyua dim in projector; reports slightlly weaker alignment than BETT-3 none beyond changing tokenizzer and impoat dim in projector
semantic ablation saw weaker performance Mingatio: add a 2-layer cross-atts ndapter that re- re-iijeeet image tokens before the projector: expeet AP drop if cross alin
need to tap the unimmolal decnder lniklen state memorry heavy: need to tap the unimodal decoder hidden
pon/aveegg queries or exteedd S SAM prompt len. Requires a one- one-step step decode per grongt (latency) and an additionl MT. .. to location
menory heary, expect drop if no cross attn expect AP drop if cross cross-aitt
not image- mmage--oodditions -
Adaptor speeetti tweaks
Encrnde: Devoder Contrastive Head (pooled decader)
same interface as BEIT-: smaller model: multi-- lingual
inuriction tumedf handy we ever expese cap- tioning
Daal entcoder Conmustiiv (iest vector nox Smage conditione))
unlimittdd vocabulary: latency, many open checkpoinns
Early fusion Pronsfommery (drop-at in closest BETT-3
SOTA zero- shor retrieval: (0-lang support
Query former Hybrids (mulliipc vector))
optional box tokens UX studies
stronger semantis than CLIP-L >dge-fricndly multiinggua
100- 100-lang lon-lang OCR; cobust semiaatics
Pros for Open WorldSAM
better long sail semmatics
maliv multi query
Parsons
230/41MM
18433 2M
30HM
610M
86M
M
3.0
1.1B
1.68
Pooled dim
x2x
768
768
768
1024
512
768
1024
768
VET+ Fonner LLM
VET emc. + TFM dec. dec. VITE enc. eno.
TS doc. doc.
VET ene. + LLM dec.
VIT-G/14 TEx VIT/TT
Join enc. IFM
Joins enc. TFM
join enc. TFM
Arch. type
ViT+Tett
at 224 resolution for the visual branch.
BLIP 2. OPT:2.77 2.78 [79]
Family Exemplaas
Florence---Bartt 2-Base [74]
CLIP-VI-L/IA
EVA-CLIP-E (75]
VLMo-BIL 172]
S [76]
CoCo-Base Bood (77)
OFA-B/L [73] [73)
Kosmos-2 [80]
PaLI-3B (78)
24

---

F1 Effect of varying tie-breaker tokens
We set the hyperparameter K = 20, meaning for each prompt (c.g., a category name), our model identify up to 20 distinct objects. For crowded scenes containing category, increasing K' is straightforward and advisable. In practice, COCO images typically contain moderate number of distinct categories and instances (the original COCO paper reports "on average, our dataset contains 3.5 categories and 7.7 instances per image." [46]). The chosen value should match exceed the maximum expected number of objects per category. For reference, DETR [45] used 100 total queries, aligning roughly with the maximum number of objects per image. Our choice (K 20) results, on average, in approximately 70 queries per image (20 queries x 3.5 categories). providing ample coverage for typical scenes.
than 20 objects per
Further, [81] observed that increasing queries initially improved Average Precision (AP), but then plateaued or even slightly declined when queries became excessive, indicating redundancy in higher query counts. However, recall does improve with more queries, since more detection slots increase the chance to find each object.
We conducted additional ablation experiments in Table 11 by varying K, pretrained on COCO and evaluated on ADE20K instance segmentation.
Table 11: Ablation on the number of tie-breakers K.
Metric
K
10
K
20
K 30
Average Precision (AP)
14.2
16.9
16.5
Average Recall 100 (AR)
21.6
28.8
29.4
Observations. (1) Increasing K from 10 to 20 improves recall and AP; beyond 20 gains saturate.
mirroring the behavior reported for DETR-style object queries; (2) Average Recall with max 100 detections per image (AR 100) improve when increasing K from 20 - 30; (3) 20
optimal for balancing precision and recall in standard datasets.
F.2 Effect of varying number of cross-attention layers
In Table 12, we observe consistently higher accuracy with 3-layer cross-attention across datasets, confirming the importance of multi-layer cross-attention. However, single-layer variant significantly narrows the gap with fewer parameters (2.4M vs. 4.5M), suggesting a practical compromise between parameter count and accuracy.
Table 12: Ablation on cross-attention depth across datascts. Metrics are PQ/AP/mloU for ADE-150 and mloU for the others.
Variant
Params
ADE-150 (PO/APintoL)
ADE-857 (mloU)
VOC-20
SUN-37
SCAN-40
no cross-attn
1.7 (M)
35.1/17.1/56.
32.2
70.4
44.2
97.3
63.6
53.8
l-ayyer cross- altn
2.4 (M)
35.1/16.8/59.0
32.8
72.6
46.3
97.5
66.4
54.0
3-layer cross-attn
4.5 (M)
35.2//6.9/60.4
33.1
73.7
47.5
98.0
67.7
53.6
G Inference Speed Analysis
We conducted detailed profiling quantify the impact of adding the VLM and our adapter modules to SAM. In Table 13 and 14, we present inference timing breakdowns for processing single 1024 x 1024 image on an NVIDIA A5000 GPU, averaged over five independent runs.
Summary (single prompt). SAM modules total time: 373.35 ms (81.0%), Non-SAM modules total time: 87.21 ms (18.9%), Non-SAM overhead: 87.21
Summary (six prompts). SAM modules total time: 539.72 ms (78.4%), Kon-SAM modules total time: 148.65 ms (21.6%), Non-SAM overhead: 148.65 ms.
25

---

Table 13: Inference timing breakdown for single text prompt (20 queries).
Module
Time (ms)
Percentage
Category
san_
Dacciboae_featurr_ppe
329,83
71.6%
SAM
prompt_.tkkeniza
0.43
0.1%
Non-SAM
beit3_forvard
70.84
15.4%
Non-SAM
mip_projection_ _aaeer
6.68
1.4%
Non
prepare_batched. wwt.battceedde. tie, tte_eetteeeeeebre___e____ tokens
0.13
0.0%
Non-SAM
cross_attention
8.45
1.8%
Non-SAM
san.promptt_
0.11
0.0%
SAM
san_mask_decoder
43.41
9.4%
SAM
postproceseing
0.68
0.1%
Non-SAM
TOTAL TIME TIME
460.69
100.0%
Table 14: Inference timing breakdown for six text prompts (120 queries).
Module
Time (ms)
Percentage
Category
san_ sann_bacbbone_eee
334.42
48.6%
SAM
prompt_tokenizeett
1.02
0.1%
Non-SAM
beita_forvard
123.73
18.0%
Non-SAM
rip.projection__1g
4.48
0.6%
Non-SAM
corparebbatched_..
0.20
0.0%
Non-SAM
Eross_attention__ _layers
18.17
2.6%
Non RAM
san_prompt__eee
pet.prrrpp_enooeer encoder
0.12
0.0%
SAM
san_nask_dedee decoder
205.18
29.8%
SAM
postprocessing
1.06
0.2%
Non SAM
TOTAL TIME
688.50
100.0%
%
Takeaway.
The profiling results show that adding the VLM and adapter modules results in only moderate increase in inference time (approximately 19-22% overhead). Most computational cost remains within SAM's backbone and mask decoder.
Mask Decoder scaling. sam_ cat__ask_deccooe cost grows almost linearly with (K x P).
Going from 20 queries (same prompt) adds +41
Going from prompt - 6 prompts (120 queries) adds further 162 ms.
Note that one text prompt mimics user clicks 20 times on an image. If automatic mask generation is dense 32 32 grid of
desired without user intervention, SAM's built-in auto-mask generator point prompts, incurring significantly higher costs compared to our text-based prompting approach.
Overall overhead. Relative for a single prompt (332 - 461 ms). However, it becomes approximately 3x more efficient when handling three or more prompts, as the backbone and VLM overhead are amortized. Thus, our enhancements introduce manageable overhead, maintaining practical usability in real-world applica- tions.
vanilla SAM2 call, pipeline is approximately 39% slower
26


Reviews:
[{"review_id": "reTXRIbyKF", "reviewer": "Reviewer_CF7s", "summary": "The paper presents OpenWorldSAM, extending SAM2 for open-vocabulary multi-instance segmentation in a resource-efficient manner. The methodology includes: (1) jointly encoding image and text prompts, together with multilevel aggregation of input  for richer information, and aiding with reasoning over ambiguity and unseen classes. (2) a novel tie breaking mechanism to equip the model with awareness of different object instances. Experiments show that OpenWorldSAM outperforms SOTAs across multiple benchmarks.", "questions": "1. Figure 4 illustrates inputting two object queries concurrently (giraffe and rock) into the model; while the textual explanation only refers to a single query (e.g. line 130: “a category name“), maybe for simplicity in writing. How is it actually done in the main experiments? What is the performance difference between feeding each query separately vs. multiple queries at once? If multiple queries, how is positional information (non tie-breaker) incorporated into the soft-prompting transformer module?\n2. What does soft-prompting in soft-prompting transformer module mean? Is there any prefix used during the training?\nWe acknowledge the difficulty in keeping up with the rapid pace of the field. Do the authors plan to discuss concurrent works (e.g. [1,2])?\n3. What’s the relationship between tie-breaker tokens and learned positional encodings (e.g. [3])?\n\n[1] https://arxiv.org/abs/2501.04001\n\n[2] https://arxiv.org/pdf/2411.14723\n\n[3] https://arxiv.org/abs/1810.04805", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- OpenWorldSAM consistently performs strongly relative to prior arts, across a broad range of evaluation settings. Apart from comprehensive quantitative comparison, the authors also qualitatively analyze the resulting segmentation masks.\n- Unified architecture while being very lightweight and parameter efficient.\n- The tie-breaker solution is elegant and effectively addresses the multi-instance problem.\n- The introduced Oracle-Prompts, an alternative evaluation protocol, addresses the ambiguity issue for better performance assessment. Through this, they also identify deeper characteristics into existing models.\n- Design choice of the model is well validated through detailed ablations.\n- The upfront honesty of the model’s limited performance on outdoor datasets is appreciated.\n- The discussion on two-stage inference for segmentation map refinement is also appreciated.\n\nWeaknesses:\n\n- Line 154: “ In practice K = 20 covers >99% images in COCO; for larger scenes K can be increased trivially.”, though the paper doesn’t showcase the effect of varying the number of tie-breaker tokens (K), either during training or post-training.\n- Line 178: “During training, the Hungarian matching loss naturally encourages each t_i to specialize in different spatial regions.” While graceful; without explicit encouragement, this could lead to problems (e.g. query collapse, forgetting, or overfitting) when finetuning to small downstream datasets with low instance counts, which isn’t studied in this work.", "was_revised": true, "final_justification": "The paper provides strong performance against past and concurrent baselines. The methods are elegant and properly justified.", "rebuttal": "We thank Reviewer 1 for the strong endorsement of our paper and insightful feedback! We are encouraged that you found our method effective, efficient, and well validated. We address reviewer comments below and will incorporate all feedback.\n\n### 1. Effect of varying tie-breaker tokens ($K$)\n\nThanks for highlighting this point. To clarify, setting $K=20$ means for each prompt (e.g., a category name), our model can identify up to 20 distinct objects. For crowded scenes containing more than 20 objects per category, increasing $K$ is straightforward and advisable.\n\nIn practice, COCO images typically contain a moderate number of distinct categories and instances (the original COCO paper reports \"on average, our dataset contains 3.5 categories and 7.7 instances per image.\" [1]). The chosen value should match or exceed the maximum expected number of objects per category. For reference, DETR [2] used 100 total queries, aligning roughly with the maximum number of objects per image. Our choice ($K=20$) results, on average, in approximately 70 queries per image (20 queries × 3.5 categories), providing ample coverage for typical scenes.\n\nFurther, Zhang *et al.* (2022) [3] observed that increasing queries initially improved Average Precision (AP), but then plateaued or even slightly declined when queries became excessive, indicating redundancy in higher query counts. However, **recall** does improve with more queries, since more detection slots increase the chance to find each object.\n\nWe conducted additional ablation experiments by varying $K$, pretrained on COCO and evaluated on ADE20K instance segmentation:\n\n| Metric | $K=10$ | $K=20$ | $K=30$ |\n| --- | --- | --- | --- |\n| Average Precision (AP) | 14.2 | **16.9** | 16.5 |\n| Average Recall@100 (AR) | 21.6 | 28.8 | **29.4** |\n\nObservations. (1) Increasing $K$ from 10 → 20 improves recall and AP; beyond 20 gains saturate, mirroring the behavior reported for DETR‑style object queries; (2) Average Recall with max 100 detections per image (AR@100) improve when increasing $K$ from 10 → 20 → 30; (3) $K=20$ is optimal for balancing precision and recall in standard datasets.\n\nWe'll include these findings clearly in the supplementary materials.\n\n### 2. Query collapse, forgetting, or overfitting on small datasets\n\nWe appreciate this concern and provide a detailed analysis below:\n\n**A. Query collapse**:\n\nDuring fine-tuning on RefCOCOg, each text prompt corresponds to exactly one target object, making it effectively a small-instance-count dataset scenario. We found our model generalizes robustly: cIoU scores demonstrate strong performance. We explicitly measured query collapse—the fraction of images where multiple tie-breakers predicted identical masks (IoU > 0.9). Results show minimal query collapse (<1%), even when fine-tuning on the smaller RefCOCOg-Mini (2k images):\n\n| **Setting** | Frozen encoders? | Train images | Avg. inst./img | Collapse?* |\n| --- | --- | --- | --- | --- |\n| RefCOCOg (full) | ✔ | 26 k | 2.1 | 0.3 % |\n| RefCOCOg (mini) | ✔ | 2 k | 2.2 | 0.7 % |\n\n**B. Catastrophic forgetting**:\n\nTo mitigate forgetting and small-dataset overfitting, we follow established adapter-style fine-tuning strategies. PSF‑SAM [4] observes that fully fine‑tuning SAM on a few hundred images “frequently results in catastrophic forgetting”, and proposes freezing most weights plus a lightweight adapter to preserve upstream skills. For OpenWorldSAM, all heavy encoders remain frozen, and only our lightweight 4.5M-parameter adapter is trained. This approach empirically preserves COCO-level zero-shot capabilities while adapting effectively to RefCOCOg.\n\nFurthermore, we assessed the zero-shot performance on ADE20K following fine-tuning. After RefCOCOg‑Mini fine‑tuning, ADE‑150 mIoU drops by only **4.2 pts** (60.4 → 56.2), indicating minimal loss of zero‑shot capability.\n\n| Setting | ADE-150 (PQ, AP, mIoU) | ADE-857 (mIoU) |\n| --- | --- | --- |\n| COCO pretraining (original setting in paper) | 35.2, 16.9, 60.4 | 33.1 |\n| COCO pretraining + RefCOCOg (full) fine-tune | 35.7, 16.5, 56.6 | 32.2 |\n| COCO pretraining + RefCOCOg (mini) fine-tune | 35.2, 16.3, 56.2 | 32.0 |\n\n**C. Overfitting**:\n\nWe explicitly examined potential overfitting on RefCOCOg-Mini (2k images), finding no evidence of such:\n\n| Setting | Train Images | cIoU (Validation Set) | cIoU (Training Set) |\n| --- | --- | --- | --- |\n| RefCOCOg (full) | 26k | 74.0 | 75.5 |\n| RefCOCOg (mini) | 2k | 73.4 | 77.5 |\n\nThis comprehensive analysis shows that our Hungarian matching objective implicitly enforces query diversity through one-to-one assignment, while the cross-attention layers (lines 155–166) further guide queries towards distinct visual regions. We will clarify this *implicit* enforcement of diversity in L178 and provide detailed experimental results in supplementary materials.\n\nThank you again for your insightful questions; we will incorporate these clarifications into the revised manuscript.\n\nReferences\n\n[1] Lin  et al. \"Microsoft coco: Common objects in context.\" *ECCV 2014.*\n\n[2] Carion  et al. \"End-to-end object detection with transformers.\" *ECCV*, 2020.\n\n[3]Zhang et al. \"What are expected queries in end-to-end object detection?.\" *arXiv preprint arXiv:2206.01232* (2022).\n\n[4] Wang et al.. \"An efficient fine tuning strategy of segment anything model for polyp segmentation.\" *Scientific Reports* 15.1 (2025): 14088.\n\n### Additional Questions\n\n#### 1. Clarification on processing multiple text prompts per forward pass\n\nOur implementation supports **an arbitrary number of text prompts per image**, processed simultaneously in a single forward pass. Specifically, during the forward pass:\n\n- We first tokenize all text prompts for each image simultaneously and repeat the visual tokens from BEiT-3 for each prompt, generating visual-text embeddings in one BEiT-3 forward pass (e.g., embeddings for \"giraffe\" and \"rock\" concurrently).\n- Tie-breaker embeddings are replicated for each prompt independently, and the resulting queries are concatenated (e.g., concat[queries for \"giraffe\", queries for \"rock\"]). This combined stack is then processed through SAM’s prompt and mask decoders simultaneously. Thus, Figure 4(b) accurately represents this concurrent processing.\n\n**Evaluation setup in Tables 1–5:** For comparability with previous studies (each image is seen in one forward pass), all text prompts per image were processed simultaneously in one forward pass in the reported evaluations.\n\n**Impact of batching prompts on accuracy:** We empirically tested batching prompts versus separate processing on ADE20K-150:\n\n| Prompt mode | Forward-passes / image | mIoU |\n| --- | --- | --- |\n| Single-prompt per pass | = #labels | 60.0 (-0.4) |\n| All prompts together (default) | 1 | 60.4 |\n\nThe minor difference (0.4 mIoU) indicates that batching prompts neither significantly harms nor benefits segmentation quality. The reason is that BEiT-3 already provides a distinct CLS token for each prompt, and tie-breakers are independently replicated per prompt, minimizing cross-interference. The slight improvement from batching arises from the self-attention layers in our soft-prompting Transformer module, promoting query diversity.\n\n**How positional information is handled when multiple prompts are batched.** Positional tie‑breakers are added independently to every prompt before the prompts are stacked. No extra “prompt‑id” encoding is introduced or needed; Hungarian matching during training ensures that the K queries attached to each prompt specialize on distinct spatial modes for *that* prompt only.\n\n#### 2. “What exactly is ‘soft‑prompting’ in the soft‑prompting transformer? Is there a trained prefix?”\n\n“Soft‑prompting” here means that each language‑aware query $q_ᵢ$ is **refined on‑the‑fly by cross‑attention** with the frozen SAM2 level‑3 feature map (Fig. 4c).  No static prefix or extra tokens are prepended as in LLM prompt‑tuning; instead the three Transformer layers are *entirely learnable* and act as an adapter that lets text semantics “look at” high‑resolution vision tokens before the mask decoder sees them. \n\n#### 3. Relationship between tie‑breaker tokens and positional encodings?\n\nTie-breaker tokens serve as **instance-diversity embeddings**, fundamentally different from standard positional encodings (learned or sinusoidal) that encode absolute pixel coordinates. While traditional positional encodings map pixels to unique spatial coordinates, our tie-breaker embeddings globally nudge identical semantic queries toward distinct latent spatial subspaces. During training, the Hungarian loss encourages each tie-breaker token to represent distinct, data-driven spatial modes (e.g., differentiating multiple \"zebras\" in Figure 3). The SAM2 image encoder still provides its standard 2‑D positional encodings, so spatial locality is preserved; tie‑breakers complement these by resolving instance ambiguity rather than spatial positioning itself.\n\n#### 4. Related work discussion\n\nWe appreciate the suggestions and will include references to the mentioned concurrent works in the revised Related Work section (§2). Additionally, we will explicitly compare these works quantitatively in Tables 1 and 3:\n\n| Model | ADE‑150 mIoU | RefCOCOg |\n| --- | --- | --- |\n| OpenWorldSAM (902M) | **60.4** | **74.0** |\n| ESC-Net (451.3M) | 41.8  |  -  |\n| Sa2VA-1B (1B) | - | 72.3 |\n| Sa2VA-8B (8B) | - | 74.1 |\n\nWe sincerely thank the reviewer again for their detailed and constructive feedback, which will greatly enhance the clarity and comprehensiveness of our manuscript."}, {"review_id": "o6dhbOVKyv", "reviewer": "Reviewer_AeTy", "summary": "This paper presents OpenWorldSAM, a unified interface designed to support a range of open-vocabulary segmentation tasks. It introduces an efficient language adapter that enables parameter-efficient fine-tuning while maintaining the model's generalization and interactivity. Additionally, a tie-breaking mechanism is proposed to generate instance-level segmentation masks. To further enhance multi-object localization, the paper incorporates a cross-attention soft prompting strategy.", "questions": "See weakness", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe core contribution of this paper is an efficient, learnable positional tie-breaker that enables SAM to segment multiple object instances rather than a single one, significantly improving recall in complex, multi-object scenes.\n\nThe introduction of a soft prompting Transformer block facilitates alignment between language and image features via cross-attention.\n\nBoth components—tie-breaker and soft prompting—are shown to positively impact performance. In particular, the tie-breaker demonstrates strong results on instance segmentation tasks, where images typically contain multiple instances per class.\n\nWeaknesses:\n\nAs acknowledged by the authors (lines 293–295), the SAM encoder’s feature space offers limited semantic richness. This raises concerns about the effectiveness of the proposed cross-attention mechanism and tie-breaker in accurately aligning language and image features. Based on my prior experience, SAM’s image features often struggle to distinguish fine-grained semantics. Clarification is needed on how the tie-breaker achieves robust performance under this limitation.\n\nWhile the visual-text embedding may provide coarse spatial cues, much of the semantic reasoning likely stems from BEiT-3, not direct alignment with language inputs. In this context, the operational role and limitations of the tie-breaker should be examined in more detail, especially since it is a central contribution.\n\nOne of the key justifications for using BEiT over CLIP is to enable early multimodal fusion. However, the paper would benefit from a deeper discussion of this design choice, especially regarding the EVF-SAM setup. A comparison with SAM4MLLM would also be valuable, as it presents a relevant and potentially competing approach.", "was_revised": true, "final_justification": "Thanks for the detailed response; the rebuttal addresses most of my concerns. I keep my inital rating.", "rebuttal": "We sincerely thank Reviewer 2 for the positive feedback, particularly appreciating your recognition of our contributions such as the efficient learnable positional tie-breaker mechanism and the soft prompting Transformer block, significantly improving SAM2’s open-vocabulary segmentation performance in complex, multi-object scenes.\n\nBelow, we address the reviewer’s constructive questions and concerns:\n\n### 1. Semantic richness of SAM’s encoder and the role of cross-attention and tie-breakers\n\n**Concern:** SAM features have limited semantic richness, raising concerns about the effectiveness of the proposed cross-attention and tie-breaker mechanisms.\n\n**Response:**\n\nWe fully agree with the reviewer that SAM’s encoder alone possesses limited semantic richness. The BEiT-3 encoder indeed plays a critical role by providing robust semantic alignment. The early fusion strategy of BEiT-3 provides richer semantics closer to SAM's features.\n\nHowever, we argue that our OpenWorldSAM is a general plug-in architecture that aligns SAM’s embedding spaces with VLM’s embedding space while keeping all heavy backbones frozen. Although we instantiate the framework with SAM2 and BEiT-3 in this paper, neither component is required by design; alternative interactive decoders or vision-language encoders can be swapped in with only minor re-training of the adapter (Please refer to Appendix E.1 & Table 5). Appendix Table 5 surveys representative VLM encoders that could replace BEiT-3 in OpenWorldSAM with $\\approx$ 5(M) adaptor parameters. For example, early-fusion encoders like BEiT-3 (e.g., VLMo, OFA, Florence-2) require zero architectural change beyond projector resizing and are therefore the most promising immediate swaps. Dual-encoders (CLIP family) need a shallow cross-attention adaptor to overcome missing image context.\n\nOf course, with better VLMs whose embedding spaces are closer to SAM’s encoder space, our model will perform better.\n\nOn the role of cross-attention. **Cross-attention injects BEiT-3 semantics *into* SAM’s high-resolution grid.** Our proposed 3-layer soft-prompting Transformer (Fig. 4c) performs cross-attention over the high-resolution 64×64 feature map used by SAM2 for mask decoding. This strategy provides every query direct access to BEiT-3’s fused semantics at pixel-level granularity (~16K tokens), rather than relying solely on the coarse global [CLS] embedding discussed in Line 133. Empirically, removing the cross-attention block (while keeping other components intact) degrades performance: ADE-150 mIoU drops by 1.5 points, and ADE-857 mIoU by 0.9 points (Table 5, E3 vs. E5).\n\nOn the role of Tie-breakers. **Tie-breakers address SAM’s limited semantics by *spatially diversifying* identical semantic queries.** When BEiT-3’s semantics are present, disabling positional tie-breakers leads to a severe performance collapse. Specifically, ADE-150 AP drops from 17.1% to 1.0%, and mIoU plummets from 56.8% to just 1.2% (Table 5, E3 vs. E1). This highlights the critical role of tie-breakers in differentiating multiple instances spatially, especially in the presence of weaker intrinsic semantics from SAM.\n\n### 2. Operational role and limitations of tie-breakers relative to BEiT-3\n\n**Concern:** Semantic reasoning primarily comes from BEiT-3 rather than direct language-image alignment, raising questions about the operational role and limitations of tie-breakers.\n\n**Response:**\n\n**Operational role of tie-breakers:** BEiT-3 provides robust **class-level semantic cues (\"what\")**, whereas tie-breakers supply **instance-level spatial cues (\"where/how many\")**. During training, Hungarian matching compels each tie-breaker vector to specialize in spatially distinct regions, thereby enforcing diversity and preventing redundancy. \n\n**Limitations:** Tie-breakers are fundamentally positional and thus cannot rectify semantic misclassifications arising from BEiT-3. If BEiT-3 misinterprets a semantic concept, tie-breakers cannot correct such semantic errors. Their effectiveness is contingent on BEiT-3’s correct semantic interpretation.\n\n### 3. Justification for BEiT vs. CLIP and comparisons with EVF-SAM and SAM4MLLM\n\n**Concern:** The paper would benefit from a deeper discussion of the choice of BEiT-3 over CLIP (especially regarding EVF-SAM) and a comparison to SAM4MLLM.\n\n**Response:**\n\nWe appreciate this suggestion and further clarify below:\n\n**Relation to EVF-SAM:** In the submitted manuscript, we (1) gave a shout out to EVF-SAM in Related Work L108 (“Inspired by the success of EVF-SAM, we enhance SAM further into the domain of open-vocabulary segmentation…”), (2) identified the problem of EVF-SAM in Figure 3c (“Naive solution [EVF-SAM]: A single language query for ‘zebra’ causes SAM2 to segment only the most salient instance.”), and (3) gave credits to EVF-SAM’s preliminary experiments on replacing CLIP with BEiT-3 in our ablations L292 (“…deep semantic integration is critical for aligning language concepts with visual regions, echoing findings by EVF-SAM”). \n\nGiven that EVF-SAM’s primary innovation lies in utilizing BEiT-3, we believe that our multiple citations throughout the submitted manuscript sufficiently acknowledge EVF-SAM's contributions, avoiding redundant discussion on the choice of BEiT-3 over CLIP.\n\nWhile EVF-SAM also leverages BEiT-3, it is strictly limited to single-instance queries. As illustrated in Figure 3, EVF-SAM segments only the most salient instance when given category-level prompts, ignoring other instances. Our design (tie-breakers plus cross-attention) significantly advances EVF-SAM’s single-instance capability into a robust multi-instance, multi-task model, which is clearly demonstrated by our detailed ablations (Table 5).\n\n**Comparison with SAM4MLLM [Chen et al., 2024]:** Despite employing a powerful 7B-parameter Qwen-VL backbone with ~29M LoRA-tuned parameters, SAM4MLLM achieves only a minor improvement (+0.5 cIoU) over OpenWorldSAM (74.5% vs. 74.0%) on RefCOCOg. Moreover, SAM4MLLM relies on a two-stage pipeline (\"LLM-dialogue → point prompts → SAM\"), incurring higher inference latency and failing to demonstrate multi-instance segmentation capabilities or results on challenging multi-instance benchmarks (ADE20K, PASCAL, ScanNet). In contrast, our lightweight 4.5M-parameter adapter with early vision-language fusion via BEiT-3 and positional tie-breakers consistently achieves state-of-the-art results across multiple open-vocabulary segmentation tasks and datasets.\n\nWe deeply appreciate your valuable suggestions and will incorporate all feedback into the manuscript."}, {"review_id": "x8P1b7nYDV", "reviewer": "Reviewer_viZZ", "summary": "This paper extends the SAM2 framework for open-vocabulary semantic segmentation by integrating multi-modal embeddings from a Vision-Language Model (VLM). The authors propose four key design principles,i.e., Unified Prompting, Efficiency, Instance Awareness, and Generalization, to enhance the method from multiple perspectives. Experimental results demonstrate the superior performance of the proposed approach.", "questions": "Please refer to the weaknesses for the specific questions.\nMinor: In Table 1, the value \"16.9\" for OpenWorldSAM should be in bold.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+The paper is well-prepared and easy to follow.\n+The proposed framework appears effective, achieving improved accuracy compared to baseline methods.\n+Inspired by DETR, the authors propose a novel and effective tie-breaker module for querying multiple instances from target images.\n\nWeaknesses:\n-Tables 3 and 4 suggest that replacing the foundational model from CLIP/Vicuna with BEiT significantly improves segmentation. It is unclear whether the state-of-the-art results are primarily attributable to the BEiT model or the proposed methodology. A justification from the authors on this point is needed.\n-A two-stage inference method is proposed. However, according to Table 1, the results with two-stage inference do not achieve better accuracy compared to the single-stage OpenWorldSAM. Furthermore, Figure 1 in the Appendix provides only two comparative images, which is insufficient to convincingly explain the contradiction between lower evaluation metrics and seemingly better visualization effects.\n-Given that one of the paper's claimed merits is its multi-instance segmentation capability, it is strongly recommended to conduct a dedicated multi-instance segmentation evaluation to more robustly demonstrate its superiority.\n-In Table 5, E5 appears only slightly better than E3. It would be beneficial to see a comparison of these two settings on other evaluation datasets. If this outcome remains consistent, confirming the effectiveness of the cross-attention module, the authors should discuss the possibility of further simplifying the cross-attention module to reduce training parameters.", "was_revised": true, "final_justification": "I have carefully read the authors' responses and other reviewers' comments. I believe that most of my concerns have been addressed in the rebuttal. I will therefore maintain my initial rating.", "rebuttal": "We sincerely thank Reviewer 3 for the positive and insightful comments, especially appreciating your recognition of our clear exposition, the effectiveness of our proposed tie-breaker module inspired by DETR, and the improved accuracy achieved by OpenWorldSAM over baseline methods.\n\nBelow, we address your constructive questions and concerns:\n\n### 1. Attribution of state-of-the-art results to BEiT-3 vs. proposed methods.\n\n**Concern:**\n\nReviewer points out that replacing CLIP/Vicuna with BEiT-3 significantly improves performance, raising a question about the relative contributions of BEiT-3 versus our proposed modules.\n\n**Response:**\n\n**Short answer — both are required.** \n\nWe fully concur with the reviewer that BEiT-3 primarily contributes semantic reasoning. Our ablation study (Table 4) indicates this, where replacing BEiT-3 with late-fusion CLIP drastically decreases performance. The early fusion strategy of BEiT-3 indeed provides richer semantics closer to SAM's features. Early fusion lets every Transformer layer mix visual and textual evidence, yielding finer localization that late‑fusion CLIP cannot retrieve from a global text embedding.\n\nHowever, Table 5 already shows that **BEiT‑3 alone is not sufficient**: if we remove tie‑breakers the model collapses. **Tie‑breakers compensate for SAM’s weak semantics by *spatially* diversifying otherwise identical semantic queries.** With language semantics present, removing the positional perturbations causes near‑complete failure: ADE‑150 AP plunges from 17.1 → 1.0 and mIoU from 56.8 → 1.2 (Table 5, E3 vs E1). Tie-breakers are thus essential in enabling multi-instance segmentation. Therefore, both the powerful BEiT-3 encoder and our novel tie-breaker strategy jointly enable our state-of-the-art performance.\n\n### 2. Clarifying two-stage inference results and visualization discrepancy.\n\n**Concern:**\n\nThe reviewer correctly notes that despite improved visual mask boundaries in our two-stage inference, Table 1 shows no substantial quantitative improvement, and the provided visualization in the appendix is limited.\n\n**Response:**\n\nIndeed, quantitative metrics (mIoU, PQ, AP) show negligible improvements primarily because:\n\n1. Two‑stage inference only refines mask contours; IoU‑style metrics saturate once coarse localization is accurate, so small contour tweaks seldom raise mIoU/PQ/AP.\n2. Amplified errors on hard examples. On incorrectly localized masks from stage 1, refinement anchored to incorrect regions can further degrade metrics.\n\nAs the rebuttal submission does not allow visuals, we will add some failure examples to the revised Appendix.\n\nTo substantiate the visual improvement despite stable metrics, we conducted a **user-study**. We ran a blind preference test on **50 random image pairs** from ADE‑857 val:\n\n- Participants: 10 graduate volunteers familiar with vision tasks.\n- Task & options: For each of the 50 pairs, respondents answered a single forced‑choice question: “Which mask looks more accurate?” Choices: A, B, No discernible difference.  Average completion time ≈ 15 min.\n- Result: Across 10 × 50 = 500 judgements, Two‑stage preferred: 43.6 % (218/500); One‑stage preferred: 37.2 % (186/500), No difference: 19.2 % (96/500). Excluding “No‑difference”, Two‑stage wins 218/ (218+186) = **54.0 %**.\n- Takaway: Although the gain is modest, humans still favor the refined masks more often than the originals.\n\nGiven that the two-stage inference serves as an optional, low-cost post-processing step, users can conveniently enable or disable it based on their preference.\n\n### 3. Dedicated multi-instance segmentation evaluation.\n\n**Concern:**\n\nGiven multi-instance segmentation is a core claim, Reviewer 3 recommends a dedicated evaluation.\n\n**Response:**\n\nWe agree on the importance of explicitly evaluating multi-instance segmentation. Notably, our primary results already include extensive multi-instance evaluation through panoptic and instance segmentation benchmarks (ADE20K-150, ScanNet-20). Specifically, OpenWorldSAM achieves new state-of-the-art panoptic (combining instance-level and semantic-level) segmentation performance on ADE20K-150 (35.2 PQ) and strongly outperforms baselines on instance-level metrics (16.9 AP on ADE20K-150). These results directly validate our multi-instance capabilities. Nonetheless, we will further emphasize these findings explicitly in the final manuscript to highlight this strength.\n\n### 4. Ablation comparison between E3 and E5 and simplifying cross-attention.\n\n**Concern:**\n\nThe reviewer observes that E5 (full three-layer cross-attention, 4.5M parameters) performs only slightly better than E3 (no cross-attention, 1.7M parameters) on ADE-150, recommending additional datasets comparison and exploring simpler cross-attention architectures.\n\n**Response:**\n\nThanks for this valuable suggestion. We performed additional comparative experiments across multiple datasets, summarized below:\n\n|  | Params (M) | ADE-150 (PQ/AP/mIoU) | ADE-857 (mIoU) | PC-59 (mIoU) | PC-459 (mIoU) | VOC-20 (mIoU) | SUN-37 (mIoU) | SCAN-40 (mIoU) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| **E5 (3-layer)** | 4.5 | **35.2** / 16.9 / **60.4** | **33.1** | **73.7** | **47.5** | **98.0** | **67.7** | **55.6** |\n| **E3 (no cross-attn)** | 1.7 | 35.1 / **17.1** / 56.8 | 32.2 | 70.4 | 44.2 | 97.3 | 63.6 | 53.8 |\n| **E5.1 (1-layer cross-attn)** | 2.4 | 35.1 / 16.8 / 59.0 | 32.8 | 72.6 | 46.3 | 97.5 | 66.4 | 54.0 |\n\nWe observe consistently higher accuracy with E5 (three-layer cross-attention) across datasets, confirming the importance of multi-layer cross-attention. However, a single-layer variant (E5.1) significantly narrows the gap with fewer parameters (2.4M vs. 4.5M), suggesting a practical compromise between parameter count and accuracy. We will include this detailed analysis in the revised Appendix and explicitly discuss potential simplified variants as a recommended direction for practitioners seeking more parameter-efficient adaptations.\n\n### Minor formatting suggestion:\n\nThank you for noting the formatting oversight in Table 1 (\"16.9\" should be bold). We will correct this in the final manuscript.\n\nWe greatly appreciate your thorough and constructive feedback and will incorporate all suggestions to improve our manuscript clarity and robustness."}, {"review_id": "5EhKbg5Z0d", "reviewer": "Reviewer_8njg", "summary": "This paper introduces OpenWorldSAM for Open-Vocabulary Segmentation tasks. OpenWorldSAM extends the SAM2 to open-vocabulary scenarios by integrating multi-modal embeddings extracted from a lightweight vision-language model (VLM). The experimental results show that the proposed method achieves consistent gains on several benchmarks.", "questions": "Question:\n1. What about the Global-Matching prompts for OpenWorldSAM?\n2. How much does the performance decrease when replacing SAM2 with SAM or a light-weight SAM2 backbone?\n3. How does OpenWorldSAM work when both text and point/box prompts are given?\n4. Many open-vocabulary object detection methods with SAM can do open-vocabulary segmentation, including GroundedSAM[1][2] and VL-SAM[3], and they typically conduct experiments on more challenge datasets, LVIS, in a zero-shot manner. So, what about the performance of OpenWorldSAM on LVIS?\n\n[1] Liu, Shilong, et al. \"Grounding dino: Marrying dino with grounded pre-training for open-set object detection.\" ECCV 2024. \n[2] Ren, Tianhe, et al. \"Grounded sam: Assembling open-world models for diverse visual tasks.\" arXiv preprint arXiv:2401.14159 (2024).\n[3] Lin, Zhiwei, Yongtao Wang, and Zhi Tang. \"Training-free open-ended object detection and segmentation via attention as prompts.\" NeurIPS 2024.", "limitations": "This paper inherits the shortcomings of SAM2, and the inference speed may be slow for real-world applications.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. OpenWorldSAM supports a diverse range of prompts.\n2. This paper designs tie-breaker embeddings to address the multi-object issue.\n3. The training is efficiency.\n\nWeaknesses:\n1. OpenWorldSAM inherits the shortcomings of SAM2 in multi-scale ambiguous segmentation, i.e., predicting several masks for a single prompt.\n2. The inference speed may slow.\n3. In Table 1, using Oracle-Prompts for OpenWorldSAM and Global-Matching prompts for other methods is misleading.", "was_revised": true, "final_justification": "The rebuttal addresses most of my concerns. I want to keep my original rating.", "rebuttal": "We sincerely thank Reviewer 4 for the positive feedback and valuable insights, especially appreciating your recognition of OpenWorldSAM’s diverse prompting capabilities, our novel tie-breaker design for handling multi-object scenarios, and the efficiency of our training method.\n\nBelow, we address your constructive concerns and questions:\n\n### 1. Multi-scale ambiguous segmentation inherited from SAM2\n\n**Concern:** OpenWorldSAM inherits SAM2’s issue of predicting multiple masks for single prompts in multi-scale ambiguous scenarios.\n\n**Response:**\nIndeed, our approach builds upon SAM2, and thus inherits its architectural characteristics. However, our method specifically addresses instance-level ambiguity, not scale-level ambiguity. Unlike visual prompts (points/boxes), text prompts typically introduce instance ambiguity rather than scale ambiguity (see Figure 3). \n\nThe reviewer correctly pointed out that by default, SAM outputs 3 valid masks given one query prompt (`multimask_output=True`). Importantly, we **disable SAM's default `multimask_output` setting during training, ensuring the model predicts only one mask per query** (1 text prompt → $K=20$ queries → 20 mask predictions, instead of $20 \\times 3= 60$ masks). We are encouraging the model to predict what exactly the text prompt asks for. Thus, OpenWorldSAM does not inherit the multi-scale ambiguity directly. \n\nAdditionally, future improvements to SAM2 will naturally enhance OpenWorldSAM, given our modular, plug-and-play architecture.\n\n### 2. Inference speed concern\n\n**Concern:** Reviewer points out the potential slowdown in inference speed.\n\n**Response:** We conducted detailed profiling to quantify the impact of adding the VLM and our adapter modules to SAM. Below, we present inference timing breakdowns for processing a single 1024×1024 image on an NVIDIA A5000 GPU, averaged over five independent runs:\n\n**Single Text Prompt (20 queries):**\n\n| **Module** | **Time (ms)** | **Percentage** | **Category** |\n| --- | --- | --- | --- |\n| sam_backbone_feature_prep | 329.83 | 71.6% | SAM |\n| prompt_tokenization | 0.43 | 0.1% | Non-SAM |\n| beit3_forward | 70.84 | 15.4% | Non-SAM |\n| mlp_projection_layer | 6.68 | 1.4% | Non-SAM |\n| prepare_batched_tie_breaker_tokens | 0.13 | 0.0% | Non-SAM |\n| cross_attention | 8.45 | 1.8% | Non-SAM |\n| sam_prompt_encoder | 0.11 | 0.0% | SAM |\n| sam_mask_decoder | 43.41 | 9.4% | SAM |\n| postprocessing | 0.68 | 0.1% | Non-SAM |\n| **TOTAL TIME** | **460.69** | **100.0%** |  |\n\n**Summary:**\n\n- SAM modules total time: 373.35 ms (**81.0%**)\n- Non-SAM modules total time: 87.21 ms (**18.9%**)\n- Non-SAM overhead: 87.21 ms\n\n**Six Text Prompts ($20\\times6=120$ queries):**\n\n| **Module** | **Time (ms)** | **Percentage** | **Category** |\n| --- | --- | --- | --- |\n| sam_backbone_feature_prep | 334.42 | 48.6% | SAM |\n| prompt_tokenization | 1.02 | 0.1% | Non-SAM |\n| beit3_forward | 123.73 | 18.0% | Non-SAM |\n| mlp_projection_layer | 4.48 | 0.6% | Non-SAM |\n| prepare_batched_tie_breaker_tokens | 0.20 | 0.0% | Non-SAM |\n| cross_attention_layers | 18.17 | 2.6% | Non-SAM |\n| sam_prompt_encoder | 0.12 | 0.0% | SAM |\n| sam_mask_decoder | 205.18 | 29.8% | SAM |\n| postprocessing | 1.06 | 0.2% | Non-SAM |\n| **TOTAL TIME** | **688.50** | **100.0%** |  |\n\n**Summary:**\n\n- SAM modules total time: 539.72 ms (**78.4%**)\n- Non-SAM modules total time: 148.65 ms (**21.6%**)\n- Non-SAM overhead: 148.65 ms\n\n**Takeaway:**\n\n1. The profiling results show that adding the **VLM and adapter modules results in only a moderate increase in inference time** (approximately 19–22% overhead). Most computational cost remains within SAM’s backbone and mask decoder.\n2. **Mask Decoder scaling:** `sam_mask_decoder` cost grows almost linearly with (K × P).\n    - Going from 1→20 queries (same prompt) adds ~41 ms.\n    - Going from 1 prompt→6 prompts (120 queries) adds a further ~162 ms.\n    \n    Note that one text prompt mimics user clicks 20 times on an image. If automatic mask generation is desired without user intervention, SAM’s built-in auto-mask generator uses a dense 32×32 grid of point prompts, incurring significantly higher costs compared to our text-based prompting approach.\n    \n3. **Overall overhead:** Relative to one vanilla SAM2 call, our pipeline is approximately 39% slower for a single prompt (332 → 461 ms). However, it becomes **approximately 3$\\times$ more efficient** when handling three or more prompts, as the backbone and VLM overhead are amortized.\n\nThus, our enhancements introduce manageable overhead, maintaining practical usability in real-world applications.\n\n### 3. Evaluation protocol clarity in Table 1\n\n**Concern:** Using Oracle-Prompts for OpenWorldSAM but Global-Matching prompts for other methods in Table 1 may be misleading.\n\n**Response:**\nWe appreciate this clarification. Table 2 already provides both protocols explicitly for X-Decoder, but we agree that clearer labeling and alignment are necessary. We will explicitly state the evaluation modes (\"Oracle-Prompt\" for OpenWorldSAM, \"Global-Matching\" for other methods) in Table 1 in the revised manuscript to ensure an accurate, fair comparison. Importantly, OpenWorldSAM remains superior even under aligned evaluation conditions in Table 2.\n\n### Additional Questions:\n\n#### 1. Global-Matching prompts for OpenWorldSAM\n\nOpenWorldSAM, like SAM, is specifically trained for interactive, prompt-conditioned segmentation (Oracle-Prompt mode). It is inherently unsuitable for Global-Matching evaluation, which involves prompts that may not correspond to objects in the image, causing issues as negative sampling was not incorporated in SAM’s original training recipe. Thus, evaluating OpenWorldSAM with global prompts would not accurately reflect its intended usage or strengths.\n\n#### 2. Performance with alternative SAM backbones\n\nDue to time and computational constraints, we could not perform additional training with SAM or lighter SAM2 variants. Nonetheless, since our approach is modular, retraining our lightweight adapter to align with other backbones (e.g., SAM2-small/base) is straightforward. Given the minimal performance gaps reported among SAM2 variants in the table bellow (taken from SAM2's Github repo), we expect our method's performance to remain robust across different backbone scales:\n\n| **Model** | **Size (M)** | **SA-V test (J&F)** | **MOSE val (J&F)** | **LVOS v2 (J&F)** |\n| --- | --- | --- | --- | --- |\n| sam2_hiera_tiny | 38.9 | 75.0 | 70.9 | 75.3 |\n| sam2_hiera_small | 46 | 74.9 | 71.5 | 76.4 |\n| sam2_hiera_base_plus | 80.8 | 74.7 | 72.8 | 75.8 |\n| sam2_hiera_large | 224.4 | 76.0 | 74.6 | 79.8 |\n\n#### 3. Combining text and point/box prompts\n\nCurrently, OpenWorldSAM does not simultaneously support text and point/box prompts. Our goal was precisely to reduce or eliminate user intervention by using positional tie-breakers that emulate spatial clicks. However, incorporating dual-modality prompts is an interesting direction we intend to explore in future work.\n\n#### 4. Performance on challenging datasets like LVIS\n\nWe agree that evaluating on challenging large-scale datasets like LVIS-1203 would further validate our method’s scalability and robustness. Given our strong performance on similarly challenging datasets (e.g., ADE-857), we anticipate competitive generalization to LVIS. Furthermore, incorporating strategies like Grounding DINO's large-scale pretraining (datasets: Objects365, OpenImages, GoldG, Cap4M, COCO, RefCOCO) could significantly enhance our model's capabilities. And VL-SAM’s preprocessing stage where a 17B parameter multimodal reasoning model (CogVLM-17B) is asked to list all the objects in the image could help transition OpenWorldSAM from oracle-prompt conditions toward fully automated open-vocabulary segmentation. We plan to explore these promising avenues in future research.\n\nWe deeply appreciate your valuable suggestions and insights, which will significantly enhance the final manuscript."}]

Author rebuttal:

Meta-review: All four reviewers gave positive ratings (3 borderline accepts and 1 strong accept), and the rebuttal successfully addressed their concerns. In summary, this paper proposes OpenWorldSAM, a method extending SAM2 for open-vocabulary segmentation via language prompts. Strengths acknowledged across reviews include its unified and lightweight architecture, tie-breaker mechanism for multi-instance segmentation, and strong zero-shot generalization across diverse benchmarks. Reviewers acknowledged the author’s thorough responses and kept their ratings after rebuttal. Given the technical contributions and solid experimental results, the AC recommends acceptance.
Decision: Accept (spotlight)