Title: OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts
Venue: NeurIPS 2025
Accepted: yes
Keywords: Image segmentation

Segment anything model

Vision-language model
Abstract: The ability to segment objects based on open-ended language prompts remains a critical challenge, requiring models to ground textual semantics into precise spatial masks while handling diverse and unseen categories. We present OpenWorldSAM, a framework that extends the prompt-driven Segment Anything Model v2 (SAM2) to open-vocabulary scenarios by integrating multi-modal embeddings extracted from a lightweight vision-language model (VLM). Our approach is guided by four key principles: i) Unified prompting: OpenWorldSAM supports a diverse range of prompts, including category-level and sentence-level language descriptions, providing a flexible interface for various segmentation tasks. ii) Efficiency: By freezing the pre-trained components of SAM2 and the VLM, we train only 4.5 million parameters on the COCO-stuff dataset, achieving remarkable resource efficiency. iii) Instance Awareness: We enhance the model's spatial understanding through novel positional tie-breaker embeddings and cross-attention layers, enabling effective segmentation of multiple instances. iv) Generalization: OpenWorldSAM exhibits strong zero-shot capabilities, generalizing well on unseen categories and an open vocabulary of concepts without additional training. Extensive experiments demonstrate that OpenWorldSAM achieves state-of-the-art performance in open-vocabulary semantic, instance, and panoptic segmentation across multiple benchmarks. Code is available at https://github.com/GinnyXiao/OpenWorldSAM.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces OpenWorldSAM for Open-Vocabulary Segmentation tasks. OpenWorldSAM extends the SAM2 to open-vocabulary scenarios by integrating multi-modal embeddings extracted from a lightweight vision-language model (VLM). The experimental results show that the proposed method achieves consistent gains on several benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. OpenWorldSAM supports a diverse range of prompts.\n2. This paper designs tie-breaker embeddings to address the multi-object issue.\n3. The training is efficiency.\n\nWeaknesses:\n1. OpenWorldSAM inherits the shortcomings of SAM2 in multi-scale ambiguous segmentation, i.e., predicting several masks for a single prompt.\n2. The inference speed may slow.\n3. In Table 1, using Oracle-Prompts for OpenWorldSAM and Global-Matching prompts for other methods is misleading.\n\nQUESTIONS:\nQuestion:\n1. What about the Global-Matching prompts for OpenWorldSAM?\n2. How much does the performance decrease when replacing SAM2 with SAM or a light-weight SAM2 backbone?\n3. How does OpenWorldSAM work when both text and point/box prompts are given?\n4. Many open-vocabulary object detection methods with SAM can do open-vocabulary segmentation, including GroundedSAM[1][2] and VL-SAM[3], and they typically conduct experiments on more challenge datasets, LVIS, in a zero-shot manner. So, what about the performance of OpenWorldSAM on LVIS?\n\n[1] Liu, Shilong, et al. \"Grounding dino: Marrying dino with grounded pre-training for open-set object detection.\" ECCV 2024. \n[2] Ren, Tianhe, et al. \"Grounded sam: Assembling open-world models for diverse visual tasks.\" arXiv preprint arXiv:2401.14159 (2024).\n[3] Lin, Zhiwei, Yongtao Wang, and Zhi Tang. \"Training-free open-ended object detection and segmentation via attention as prompts.\" NeurIPS 2024.\n\nLIMITATIONS:\nThis paper inherits the shortcomings of SAM2, and the inference speed may be slow for real-world applications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the SAM2 framework for open-vocabulary semantic segmentation by integrating multi-modal embeddings from a Vision-Language Model (VLM). The authors propose four key design principles,i.e., Unified Prompting, Efficiency, Instance Awareness, and Generalization, to enhance the method from multiple perspectives. Experimental results demonstrate the superior performance of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+The paper is well-prepared and easy to follow.\n+The proposed framework appears effective, achieving improved accuracy compared to baseline methods.\n+Inspired by DETR, the authors propose a novel and effective tie-breaker module for querying multiple instances from target images.\n\nWeaknesses:\n-Tables 3 and 4 suggest that replacing the foundational model from CLIP/Vicuna with BEiT significantly improves segmentation. It is unclear whether the state-of-the-art results are primarily attributable to the BEiT model or the proposed methodology. A justification from the authors on this point is needed.\n-A two-stage inference method is proposed. However, according to Table 1, the results with two-stage inference do not achieve better accuracy compared to the single-stage OpenWorldSAM. Furthermore, Figure 1 in the Appendix provides only two comparative images, which is insufficient to convincingly explain the contradiction between lower evaluation metrics and seemingly better visualization effects.\n-Given that one of the paper's claimed merits is its multi-instance segmentation capability, it is strongly recommended to conduct a dedicated multi-instance segmentation evaluation to more robustly demonstrate its superiority.\n-In Table 5, E5 appears only slightly better than E3. It would be beneficial to see a comparison of these two settings on other evaluation datasets. If this outcome remains consistent, confirming the effectiveness of the cross-attention module, the authors should discuss the possibility of further simplifying the cross-attention module to reduce training parameters.\n\nQUESTIONS:\nPlease refer to the weaknesses for the specific questions.\nMinor: In Table 1, the value \"16.9\" for OpenWorldSAM should be in bold.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents OpenWorldSAM, a unified interface designed to support a range of open-vocabulary segmentation tasks. It introduces an efficient language adapter that enables parameter-efficient fine-tuning while maintaining the model's generalization and interactivity. Additionally, a tie-breaking mechanism is proposed to generate instance-level segmentation masks. To further enhance multi-object localization, the paper incorporates a cross-attention soft prompting strategy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe core contribution of this paper is an efficient, learnable positional tie-breaker that enables SAM to segment multiple object instances rather than a single one, significantly improving recall in complex, multi-object scenes.\n\nThe introduction of a soft prompting Transformer block facilitates alignment between language and image features via cross-attention.\n\nBoth components—tie-breaker and soft prompting—are shown to positively impact performance. In particular, the tie-breaker demonstrates strong results on instance segmentation tasks, where images typically contain multiple instances per class.\n\nWeaknesses:\n\nAs acknowledged by the authors (lines 293–295), the SAM encoder’s feature space offers limited semantic richness. This raises concerns about the effectiveness of the proposed cross-attention mechanism and tie-breaker in accurately aligning language and image features. Based on my prior experience, SAM’s image features often struggle to distinguish fine-grained semantics. Clarification is needed on how the tie-breaker achieves robust performance under this limitation.\n\nWhile the visual-text embedding may provide coarse spatial cues, much of the semantic reasoning likely stems from BEiT-3, not direct alignment with language inputs. In this context, the operational role and limitations of the tie-breaker should be examined in more detail, especially since it is a central contribution.\n\nOne of the key justifications for using BEiT over CLIP is to enable early multimodal fusion. However, the paper would benefit from a deeper discussion of this design choice, especially regarding the EVF-SAM setup. A comparison with SAM4MLLM would also be valuable, as it presents a relevant and potentially competing approach.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents OpenWorldSAM, extending SAM2 for open-vocabulary multi-instance segmentation in a resource-efficient manner. The methodology includes: (1) jointly encoding image and text prompts, together with multilevel aggregation of input  for richer information, and aiding with reasoning over ambiguity and unseen classes. (2) a novel tie breaking mechanism to equip the model with awareness of different object instances. Experiments show that OpenWorldSAM outperforms SOTAs across multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- OpenWorldSAM consistently performs strongly relative to prior arts, across a broad range of evaluation settings. Apart from comprehensive quantitative comparison, the authors also qualitatively analyze the resulting segmentation masks.\n- Unified architecture while being very lightweight and parameter efficient.\n- The tie-breaker solution is elegant and effectively addresses the multi-instance problem.\n- The introduced Oracle-Prompts, an alternative evaluation protocol, addresses the ambiguity issue for better performance assessment. Through this, they also identify deeper characteristics into existing models.\n- Design choice of the model is well validated through detailed ablations.\n- The upfront honesty of the model’s limited performance on outdoor datasets is appreciated.\n- The discussion on two-stage inference for segmentation map refinement is also appreciated.\n\nWeaknesses:\n\n- Line 154: “ In practice K = 20 covers >99% images in COCO; for larger scenes K can be increased trivially.”, though the paper doesn’t showcase the effect of varying the number of tie-breaker tokens (K), either during training or post-training.\n- Line 178: “During training, the Hungarian matching loss naturally encourages each t_i to specialize in different spatial regions.” While graceful; without explicit encouragement, this could lead to problems (e.g. query collapse, forgetting, or overfitting) when finetuning to small downstream datasets with low instance counts, which isn’t studied in this work.\n\nQUESTIONS:\n1. Figure 4 illustrates inputting two object queries concurrently (giraffe and rock) into the model; while the textual explanation only refers to a single query (e.g. line 130: “a category name“), maybe for simplicity in writing. How is it actually done in the main experiments? What is the performance difference between feeding each query separately vs. multiple queries at once? If multiple queries, how is positional information (non tie-breaker) incorporated into the soft-prompting transformer module?\n2. What does soft-prompting in soft-prompting transformer module mean? Is there any prefix used during the training?\nWe acknowledge the difficulty in keeping up with the rapid pace of the field. Do the authors plan to discuss concurrent works (e.g. [1,2])?\n3. What’s the relationship between tie-breaker tokens and learned positional encodings (e.g. [3])?\n\n[1] https://arxiv.org/abs/2501.04001\n\n[2] https://arxiv.org/pdf/2411.14723\n\n[3] https://arxiv.org/abs/1810.04805\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}