Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Authors: Zhihua Liu
Amrutha Saseendran
Lei Tong
Xilin He
Fariba Yousefi
Nikolay Burlutskiy
Dino Oglic
Tom Diethe
Philip Alexander Teare
Huiyu Zhou
Chen Jin
Venue: ICML 2025 poster (2025)
Keywords: Visual Prompt Learning
Grounded Segmentation
Multimodal Prompt Learning
Diffusion Model
Primary area: applications->computer_vision
TL;DR: We introduce Segment Anyword, a training-free visual prompt learning framework with test-time inversed adaption for open-set language grounded segmentation, where visual prompts are simultaneously regularized by linguistic structual information.

Abstract: Open-set image segmentation poses a significant challenge because existing methods often demand extensive training or fine-tuning and generally struggle to segment unified objects consistently across diverse text reference expressions. Motivated by this, we propose Segment Anyword, a novel training-free visual concept prompt learning approach for open-set language grounded segmentation that relies on token-level cross-attention maps from a frozen diffusion model to produce segmentation surrogates or *mask prompts*, which are then refined into targeted object masks. Initial prompts typically lack coherence and consistency as the complexity of the image-text increases, resulting in suboptimal mask fragments. To tackle this issue, we further introduce a novel linguistic-guided visual prompt regularization that binds and clusters visual prompts based on sentence dependency and syntactic structural information, enabling the extraction of robust, noise-tolerant mask prompts, and significant improvements in segmentation accuracy. The proposed approach is effective, generalizes across different open-set segmentation tasks, and achieves state-of-the-art results of 52.5 (+6.8 relative) mIoU on Pascal Context 59, 67.73 (+25.73 relative) cIoU on gRefCOCO, and 67.4 (+1.1 relative to fine-tuned methods) mIoU on GranDf, which is the most complex open-set grounded segmentation task in the field.

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Zhihua Liu' Amrutha Saseendran' 2 Lei Tong? Xilin He Fariba Yousef? Nikolay Burlutskiy? Dino Oglic
Tom 2iethe Philip Teare
Huiyu Zhou Chen Jin
Abstract
(UCLIP besed mebhods
Seginenr Anywood (Chos)
Image
"The rad
Open-set image segmentation poses significant challenge because existing methods often demand extensive training or fine-tuning and generally struggle to segment unified objects consistently across diverse text reference expressions. Moti- vated by this, we propose Segment Anyword, a novel training-free visual concept prompt learning approach for open-set language grounded segmen- tation that relies token-level cross-attention maps from frozen diffusion model to produce segmentation surrogates or mask prompts, which are then refined into targeted object masks. Initial prompts typically lack coherence and consistency as the complexity of the image-text increases, re- sulting in suboptimal mask fragments. To tackle this issue, we further introduce novel linguistic- guided visual prompt regularization that binds and clusters visual prompts based on sentence dependency and syntactic structural information. enabling the extraction of robust, noise-tolerant mask prompts, and significant improvements in segmentation accuracy. The proposed approach is effective, generalizes across different open-set set segmentation tasks, and achieves state-of-the-art results of 52.5 (+6.8 relative) mIoU on Pascal Context 59, 67.73 (425.73 relative) cloU on gRef- COCO, and 67.4 (+1.1 relative to finc-tuned meth- ods) mloU on GranDf, which the most complex open-set grounded segmentation task in the field,
Text
and the dog
Encoder
Mask
Eneodder
Insgge
Contrala Mobrh
Texl and the dop
db
Augama
Detaly
lage Language
Image
Selecton
(3) Proraee based methods
Image
Segnerter
Prosegts
SAM
SAM
VEM
Syganier
and
Figure 1. Key comparison of multi-modal open-set image seg- mentation architectures. (a) CL.IP hased methods (Wang et al., 2022: Xu et al., 2023c) (b) MLLM based methods (Rasheed et al., 2024; Lai et al., 2024; Zhang et al., 2024). (c) Previous prompt learning based methods (Chen et al., 2024; Lin et al., 2024) (d) Our Segment Anyword's architecture, which possesses simple design for effective arbitrary syntax level grounded segmentation capability with minimal visual prompt optimizing efforts. Project page, code, and data are available at https: //zhihua Lued, gi github. io/ http//egmettttteeeeeeeeeeeeeeeee _eyyerdtaaaywrrrdd
1. Introduction
Image segmentation plays critical role in intelligent sys- tems, enabling perceptual applications in various domains such as autonomous driving (Cordts et al., 2016; Geiger et al., 2013), robotic manipulation (Butler et al., 2017; Bruce et al., 2000) and computer-assisted diagnosis (Menze et al., 2014: Litjens et al., 2017). Traditional neural networks are supervised-trainee to perform dense pixel-wise classi- fication to delineate the object mask (Long et al., 2015; Ronneberger et al., 2015; He et al., 2017). Although these networks succeed in most cases, they struggle at handing novel objects novel domains, e.g. classifying wild ani- mals or new biomarkers, due to fixed close-set set training data samples and labels (Liu al., 2020; Wu et al., 2024).
'School of Computing and Mathematical Sciences, Uni- versity of Leicester, UK "Centre for AI, Data Science & Artificial Intelligence. BioPharmaceuticals R&D, As- traZeneca, Cambridge, UK "Shenzhen University. Correspon- dence to: Huiyu Zhou 4hol44@@iicsstcracc.ukk Chen Jin <chen.jin@ astrazeneca.com>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
In Distribution Imagee
Ou: DustDiiibbttin Imagee
cwetl dlcc some saudd bulkecky The dow.
sillquue lekking lally
can Phott
Veld rale Tallii aoooeded by Todle
inth kecken
The youge pllie sale diudalonaa
aadd pai Broeet Madeee
cep Vicnas rakkus calig Scade bedigioudd
with
ploot becedd df
ale Karao i toa ta wach tud St
Thi wand
Thett of
Als siddag duikkard be
(haleg and pailg (ar lanes, Theiss
Attrenccad mn soes
packced cart locp
mask rand bnd villi solle
haak Berbert nene
plean Jaix bill tant cesee
tentiiis teckgraan
Landscape/ Architecture Portnai
Close-up/ / Snopshot
Syntheric/ Medical
Figure 2. We propose Segment Anyword for Open-Set Grounded Segmentation, Segment Anyword is multi-modal promptable image segmentor solely huilt on the semantic prior knowledge extracted from frozen diffusion model. Segment Anyword demonstrate superior performance across multiple multi-modal modal segmentation task, including 1) reference image segmentation (refCOCO, gRefCOCO). 2) complex grounding segmentation (GranDf) and 3) OOD Medical Image Segmentation. Best viewed in colors and zoomed-in.
Recent advancements in vision-language models (VLMs) and multi-modal large language models (MLLMs) have demonstrated a remarkable progress on learning transferable multi-modal representations (Radford et al., ;0 Ho et al., 2020). Related works utilized VLMs (Zhou et al., 2022a; Wang et al., 2022; Xu et al., 2022; 2023c; Sun et al., 2024) or MLLMs (Lai et al., 2024; Rasheed et al., 2024; Xia et al., 2024; Zhang et al., 2024), pre-trained on large-scala scale image-text datasets, to address open-set segmentation and classify novel objects and categories by training or fine- tuning additional encoders and decoders (Figure (a-c), detailed in Appendix B). Such methods commonly require expensive training or fine-tuning resources and struggle with inferior performance when encountering words or phrases unseen in the training data where diverse terms are used to describe the same object, e.g. everyday terms versus domain domain-specific terminology. We verified this through a motivational study (Figure 3), which highlights the need for flexible and adaptive multi-modal segmentation framework that is robust to text variation and minimizes ambiguous segmentation of unified visual concepts.
the scalability of the diffusion model, which allows seamless inverse adaptation to new words and visual concepts with minimal effort (Figure 2). By collecting averaged cross- attention maps, which facilitates text-image interaction as segmentation prior, we randomly sample points from each map as mask prompts for downstream mask generators, such as Segment Anything Model (SAM) (Kirillov et al., 2023). We observe that the initial visual prompts may lack coherence and consistency, resulting inaccurate word-to- mask mapping with suboptimal segmentation fragments. To tackle this, we introduced novel linguistics-based regular- ization to efficiently bind and cluster mask prompts. The main contributions of this work are summarized as follows:
(1) Our motivational studies revealed that the current VLM and MLLM-bbssed approaches struggle with diverse terms during open-set segmentation, resulting in confounded mask outputs, underscores the need for adaptive methods to learn new visual concepts low cost.
(2) We propose Segment Anyword, a novel training-fice prompt-learning framework for open-set language grounded segmentation performing purely at test-time. Segment Any- word utilizes the cross-attention from a pre-trained diffusion model, which encodes coken-leve localization of visual con- cepts, as object prompts to guide downstream segmentation.
Generative models show promise in adapting to unseen ob- jects by treating them as few shot concepts, linking them to learnable textual embeddings and generating customized im- ages (Gal et al., 2022). In this paper, we aim to leverage this strong adaptive capability for image segmentation and pro- pose Segment Anyword, which is an efficient framework that takes an off-the-shelf diffusion model for visual concept prompt learning to perform open-set language grounded seg- mentation (Figure (d)). Segment Anyword capitalizes on
(3) To improve word-to-mask mapping accuracy, we pro- pose novel linguistics-guider visual prompt regularization techniques for prompt binding and clustering. This approach allows the integration of linguistic knowledge into visual prompts, enhancing the effectiveness of incorporating text
2

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
(15/4
giatts (94.90
left gurll wila and
Ferfest left miaands Racouin (460))
Terthel mytadd (260)
Keroor ie Nortin un Ioar eiggnal (71.04)
and treocalivn art
teage-6t
plaae on og
Casher and player 175.0)
en in boatiginu and cacceer
right 1915/9
Figure 3. Motivational Study on the multi-object reference segmentation ReLA (I.iu et al., 2023). We categorize data points into easy (green), medium (blue), and hard (red) samples to achieve accurate and stable segmentation, based on the retained IoU mean and standard deviation, which are calculated across the caption dimension. The IoU for each image-text pair, compared to the ground truth, is shown brackets. Our study validates the presence of segmentation variability, highlighting the challenges in generating accurate and stable segmentation masks with free-form text reference deseriptions.
sion model for downstream mask prompt adaptation. To improve coherence and completeness of mask prompts, we propose novel regularization techniques that directly inte- grate linguistic dependency and syntax knowledge for vi- sual prompt binding and clustering (see Section 2.4). Our method demonstrates effortless inverse adaptation to new words and visual concepts with minimal efforts, showcasing its efficiency and adaptability in Figure 2 and Table 1.
Table 1. Comparison of existed multi-modal image segmentation technologies with our proposed method. From left to right: Open- Vocabulary (V), Open- Dpen-Reference (R), Word-Grounding word with object region (G), graining-Free (T), Fine Tuming-Free Free (F), Localize Novel Concept that has not been encountered during training (C), # Trainable Parameters (#P).
index
Method
V
G
T
F
C
#P
MattNet (Yu et al., 2018)
X
X
X
27.57M
CRIS (Wang et al., 2022)
X
X
X
x
40.42M
ETRIS (Xu et al., 2023c)
V
V
V
X
x
1.39M
2.1. Preliminaries
GLaMM (Xu et al., 2023c)
V
V
V
X
x
130M
LISA (Lai et al., 2024)
V
X
X X
405M
Text-guided Diffusion Models are a class of generative
OMGLLaVA (Zhang et al., 2024)
V
V
X
X
X
220M
S
models that generate images by progressively denoising ran- dom Gaussian noise conditioned on a text prompt. Specif- ically, we are interested in utilizing a pre-trained denois- ing network eg. Given an initial random Gaussian noise é ~~ NO, I). example image x and text condition S, cg can generate an image v = <o V) closely resembing x. Within Eg, the conditioned textual embedding V = (S) interacts with the noised image embedding 2; o,z + ore via coss-attention layer at time step t, where is encoded from frozen image encoder E, and and oz are noise schedulers. The token-wise cross-attention maps, M = softmax (KT / /Vd), correlate to the similarity be- tween image embedding query Q = fQ((+) and textual embedding key K = fV). ). In Segment Anyword, we average the attentin maps across all time steps to identify high-confidence areas for object localization.
GSVA (Xia et al., 2024)
X
X
X
13B
ReLAA (Liu et al., 2023)
X
X
3B
SAM4MILLM (Chen et al., 2024)
X
X
7B
ODISE (Xu et al., 2023h)
28.1M
OVDiff (Karazija et al., 2025)
EmerDiff (Namekata et al., 2024)
Segment Anyword
<
We report the average trainable parameters, where normal cases, our Segment Anyword only updates the visual concept corresponded textual embedding parameters.
E(x)
syntax and dependency structual information.
(4) Extensive experiments on six molli- segmentation datasets demonstrate Segment Anyword establishes a new state-of-the-ar art performance among training-free methods, even outperforming fully trained or fine-tuned approaches across several metrics. Furthermore, Segment Anyword generalizes effectively to open-set, reference, and out-of- distribution segmentation, attributed to the novel prompt- learning and alignment design.
Inversion aims to invert a given image back into the latent space, such that the input image can be faithfully reconstructed from the inverted deterministic latent repre- sentation =* = [ z Recent techniques focus on utilizing a textual inversion regenerate customized im- ages from few concept shots (Gal al., 2022). We predicte of based on the presumption that the denoising process can be reversed in the limit of infinite small steps with the noise scheduler (Zhu et al., 2016; Xia et al., 2022):
2. Methods
We start with preliminaries in Section 2.1 and then move to the results of our motivational study in Section 2.2. Sec- tion 2.3 describes Segment Anyword, a novel visual concept prompt learning framework that leverages a frozen diffu-
#6-1 Via (v
colst-ge 1). (1)
3

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
"a bey in blue sweatshirt eating donut"
frozen text encoder
text embeddings
frozen text-to-image denoising network
20
Step I: Optimize y
At one diffarion step
Allgnment
"a A (buy) (blue)
dencisiig U-nel
(sweatshirl) cating & (donu))
Imaage embtedddng
Step 2: Test-time Cross-Attentin Collection
fo(21)
QK
QK
Laiee
lext embeehhing
+
K = fx(V)
M
Attention Mark (AneMasy) average crusn-atiation maju stogy
boy (root)
*+ A *
Negativ Miraal Exclusive
Symon
Prompt Dinding
NI: "a blue sweatshirt (root)
VP: "est donut (root)"
Deperdency
Prezppi Ensenbii
(boy)
(blwe (swwarokko))
eatling
(slonord)
blue
swaatshirt
Segruerntien
*
*
M
Positive Adjective Procutt Dinding
(+)
Linguistic Guided Visual Prompt Regularization
Figure 4. Pipeline overview. Segment Anyword leverage the inversed scalability of frozen text-to-image denosing diffusion model ea for training-[ree free open-set language grounded segmentation. First, Segment Anyword regards the segmentation reference expression (lop-left) image generation text condition and reconstruct the imput image zp. Within image reconstruction process, Segment Anyword only update the textual embedding V = "u',...," V. & of visual concepts (coloured texts) while the rest of network parameters (Ca, eg) remain frozen. At test time, the averaged cross-attention maps are collected through an diffusion process of Z with optimized V. We further introduce novel finguistic-guidee visual prompt regularization to bind and cluster mask prompts for mining and improve downstream segmentation accuracy by integrating sentence denpendency and syntax structual information directly.
prompts
We utilize direct inversion (Ju et al., 2024) to preserve the original image, where the distance between the inverse and the denoising chains minimized by the addition operator.
study. We modify the subject noun-phrase in each text ref- erence while preserving the semantic meaning of original sentence to create an open-set learning space. Thus we ex- tend each image associating with additional generated 2-5 mutated expressions (e.g. apple pieces]-> apple pieces, apple slices, cut up apples]). We retain the Intersection-over Union (IoU) of each image-text pair and calculate the mean (IoU Mean) and Standard Deviation (IoU Std) across all associated expressions. As shown in Figure 3, we observe that: 1) current state-of-the-art VLMs struggle with stable segmentation performance from inaccurate boundaries or mis- mis-localizing the object. 2) such unstable performance arises from misalignment hetween visual mask representa- tion and linguistic embedding, leading to confounded seg- mentation results when text reference expression varies.
Segment Anything Model (SAM) (Kirillov al., 2023) is general segmentation model consisting of an image encoder, a prompt encoder, and a mask decoder. Given an image with a set of visual prompts p = age features and prompt tokens to interact with mask tokens in a two-way transformer. The final segmentation is gen- erated by multiplying mask tokens with image features. A vanilla SAM cannot automatically control the mask category and granularity for segmented objects, resulting in numer- ous irrelevant segmentation results. Here we simply apply frozen SAM as a pure post-processing module by feeding in object location priors as point prompt p E IR X2 from our prompt-learning framework, with automatic label assign- language grounded
]. SAM extracts im-
2.3. Segment Anyword
ment inherited from text expression segmentation. Compared with related works that utilized SAM as mask generator (Chen et al., 2024), Segment Anyword did not require additional training or fine-tuning, which is computationally lightweight and resource effective.
As illustrated in Figure 4, Segment Anyword is prompt learning framework, generating localization prior-Mask Prompts-for for mask segmentation, by utilizing an off-the- shelf diffusion model to retrieve per-token cross-attention maps. Given an input image-text pair, Segment Anyword learns list of textual embeddings V = [E* ...., & ini- tialized from frozen text encoder cs such as BERT (Devlin, 2018). This optimization is guided by image-level recon- struction from denoising diffusion process, but only make
2.2. Motivational Study
To understand the capabilities of existing methods for open- set segmentation, we begin with a large scale motivational

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Algorithm 1 Segment Anyword (pseudo code)
Tngut
CrossAttn
AttnMask
@
CrossAtte #
AttmMask
Mask
1: Input: image 2, text expression s, pre-trained Co).
2: Output: aligned embeddings V1= = [Vj,..., 1/ mask surro-
gates MM = [mr....... J.
3: updatinn Vi= [0g 1 o4 with LDM
4: for step = 1 do
5: Encode E(2)
(e.t")...... (x))) 7: Vi:= arg miny F...llllle (2 Vi[l
6: Compute V = 1o"
8: end for
9: 4 test-time cross- on col lection
Figure 5. Illustration of the averaged cross-attention map and at- tention mask. By leveraging a frozen denoising text-to-image diffusion network, the word-level cross-attention map offers lo- cation prior. In contrast, the hard mask produces a sub-optimal segmentation result, lacking fine- fineggrained details such as object shape, posture, and boundaries.
10: Inversion noisy latents Z = Ii s,z! with Eqn.
11: Z = Denoise(ro(2** V)))
12: (m...... m, 16-1,...7 = CrossAttention(,, 2, V4)
M = Avg([mn......... m (........)))
13: Return (M, Vi)
V V=Iu" -Iu".... & as trainable parameter while keeping and eg frozen. This test time embedding updating strategy is a simple yet effective to minimizes the alignment gap between text expression and image. Here, each text expres- sion sentence s can be parsed into a set of noun words of target concepts
models, where the network prioritizes image-level recon- struction to achieve a photo-realistic layout of objects, often at the expense of boundary precision as a training shortcut. To address this issue and achieve accurate mask boundaries, we employ a frozen SAM as post-processing module. Specifically, we get the cross-attention mask from the av- eraged aross-attention map with threshold 0.7. A prompt point is then randomly sampled from the largest connected mask region and used as SAM positive mask prompt input.
[n* :....: n & }, which are the visual entities to be segmented within the input image. The rest of sentence can be uub-grouped into descriptive words such as adjectives a = [a"..... a & and non-semantic neutral texts = [e*,.... {. We adopt multi-concept textual inversion (Jin et al., 2024) for optimizing with:
2.4. Linguistics Guided Visual Prompt Regularization
LDM .) E. - V)I
(2)
The averaged cross-attention map M serves as a localiza- tion prior, which is often noisy and coarse. To better mine noise-tolerant visual prompts, we propose two novel prompt regularization strategies that directly guided by sentence syntax and dependency structual information. Some current state-of-the-at methods, such as GLaMM (Rasheed et al., 2024), LISA (Lai et al., 2024), and OMG-LLaVA (Zhang et al., 2024), already incorporate similar linguistic informa- tion into their input pipelines such as integrating complete noun phrases during feature fusion or translating a special [SEG] token with context into segmentation masks. These models achieve impressive results but require significant training efforts. Other baseline methods focus on test-time optimization, such CLIPasRNN (Sun al., 2024), OVD- iff (Karazija et al., 2024), and Peekaboo (Burgert al., 2022), could theoretically leverage this linguistic informa- tion as auxiliary input. However, empirical observations suggest that their performance remains suboptimal under these conditions. In contrast, our proposed method explicitly formalizes linguistic knowledge through prompt regulariza- tion. This strategy enables robust mining of noise-tolerant mask prompts, yielding refined and higher-quality segmenta- tion masks, even without extensive training or configuration overhead.
Following Equation (1), we invert the input image to gener- ate the deterministic noising latents as test time input. The inverted latents x* are sent to a frozen denoising network to reconstruct the original input image. During the de- noising process, the updated text embedding V4 interacts with the image features through cross-attention M. As il- lustrated in Figure 4, we regard the high resporrrspsse attention as the visual concepts activated by the text embedding. We obtain the averaged cross-attention map Al across all de- noising ttmesttps and sample points from high- resporese area, demonstrated in Algorithm 1. The object class label can be retained from the original text expression, where we parse the text expression into Noun-Phrases (NP) and Verb-Phrases ( (VP) and identify each rooted noun subject (root) within the phrase.
Limitation of Plain Segment Anyword. The averaged cross-attention map M can be regarded as a pseudo mask, reflecting the token-level correlation between textual prompt and visual object entity. As illustrated in Figure 5, plain Segment Anyword can output M as the segmentation mask through a hard threshold. These results indicate that while the cross-attentia maps effectively capture the concept- prompt correlation, they lack fine- -ne-griined boundary details. This limitation arises from the training objective of diffusion
As illustrated in Figure 6, our visual prompt regulariza- tion aim to (1) reinforce the correlation between prompts,
5

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
boy
M
Swardfer
exing
CorLEC
Negative Mutual Exclusive Prompt Binding. Positive prompts reinforce the correlation between visual concepts, prompts, and segmentation masks. However, it only oper- ates within the target noun-phrase which does not inherently promote the mark boundary separation between different noun entities. Leveraging the mutual exclusive relationships between objects, we propose a negative prompt binding strat- egy guided by sentence syntax structure, where identifiable entities are incompatible with each other at both the linguis- tic and visual levels. Concretely, for target noun object, we sample additional points to serve negative prompts for remaining noun objects. These negative points guide the mask generator by highlighting areas excluded from the current object. cases where only one object needs seg- mentation, we randomly sample 1~3 negative points outside the object mask, which serve as background prompts. By parsing the text expression into noun-phrases, such positive and negative binding strategy can be constructed recurrently, where the clustered positive prompts from mutual exclusive noun-phrase can be grouped as negative prompts for current target as the iteration of noun- noun-phrase progresses.
Chow Anreziion
iisutty
puby)
doly
Depeddacy
det
boy
in
blue x
sweatshirt rk
ewting
dorut
smoo
Noun Phrase: "s boy (root)
*
Noun Phrraper blue sieettshirt (root)"
*
*
"eeting donut (root)"
Verb Phrese:
Segneet
Woodleve Porngt Sqgnenitiinn
Builie Adjative Rodingg Negutive Maraal Excknive Realing
Figure 6. Linguistic guided visual prompt regularization. Ini- tial visual prompts may lack detailed mask description both coherence and consistency resulting in suboptimal mask fragments (bottom-left: <boy>, <donut>). Our linguistic guided visual prompt regularization is simple yet effective, leveraging sentence dependency and syntax information. The target object can cluster associated adjective modification word, improving mask complete- ness (bottom-mid <sweatshirt>> + <bluc>>). Each entity can be recurrently bind negative prompts from other entities, eliminating felse-negative regions for distinct mask boundaries (bottom-right: <boy>> + <blue sweatshirt>> <donut>, <donut>> <boy>>.
3. Experiments
We perform extensive experiments six multi-modal im- age segmentation datasets, including open-set language grounded segmentation dataset GranDf (Rasheed et al., 2024), multi object reference image segmentation dataset gRefCOCO (Liu et al., 2023), single object reference im- age segmentation dataset RefCOCo, RefCOCO+ and Ref- COCOg (Kazemzadeh et al., 2014), open-vocabulary seman- tic segmentation Pascal Context (Mottaghi et al., 2014), During evaluation, we follow the official setups for each of the datasets and report: i) the AP with threshold of 50% (AP50), mean IoU (mloU), and Recall on GranDf, ii) the cumulative IoU (cloU) and generalized IoU (gloU) on gRe- iCOCO, and iii) the cloU over all splits on ReiCOCO/+/g.
concepts, and masks by clustering positive prompts with descriptive words such as adjectives and (2) eliminate false- negative area by recurrently binding each object with neg- ative prompts from extuulee entities and their at- tributes.
Positive Adjective Prompt Clustering. Recent research works validate that pre-trained generative model can also discover descriptive words like adjectives for certain objects and subjects (Rombach et al., 2022; Ruiz et al., 2023; Vinker et al., 2023; Jin et al., 2024). This suggests that leverag- ing adjective words can reinforce the triplet correlation be- tween the visual concepts, visual prompts, and segmentation masks. For noun words (n*,..... n rooted within parsed noun phrase, the embedding of the corresponding adjective [a".....,a & is also updated during test-time optimization [n' in",,...," n & is adjective modified by [a",.... a &], We retrieve and average the associated cross-attention maps of such adjective across all time steps in denoising process. The adjective rross-ateenioon map follows the same process- ing pipeline as the noun words to generate point prompt from its high-activation area. Such prompt from adjective is clustered with the point from noun word as a positive point pair. In cases where the noun word is not accompanied by any adjective words, we sample points twice from the same noun word cross-atted attention map.
We present implementation details in Appendix C. In ad- dition to accelerate textual embedding update speed, we first use 500 image-text pairs from each dataset training split to LoRA fine-tune BERT text encoder of Segment Any- word with 1 1100 steps. With LoRA fine-tuned BERT text encoder, Segment can achieve fast textual embedding update within 50 steps during inference on validation and test splits (Segment Anyword,). This step does not require large-scale training or supervision and provides a practical way to effi- ciently align the text encoder with the target domain, which aligns with our goal of avoiding full training-sel -ett traverse and supports fast, sample-efficient test-time optimization.
3.1. Main Results
Open-Set Grounded Segmentation. GranDf is one of the most challenging datasets in the field of multi-modal
6

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Table Open-Set I Language Grounded Segmentation Performance on GranDI Dataset validation and test split. "#lmages" indicates the number of training images. "_" indicates the result is not reported from original paper.
Table 4. Single Object Reference Image Segmentation Perfor- mance on RefCOCo, ReiCOCO+ and RefCOCOg Dataset. The evaluation metric cloU
Method
Pinsaes
ReCUCCO
RefCOCO+
Val
TedA
TIeA
Tbatt
Method
timages
APSO
GranDi Val
minU
Recall
APSO
Granii mieU
Recall
Teatitional nothooss
1. LAVT
12K
72.7
75.5
68.8
62.1
63.4
55.1
61.2
62.1
Training End-to-Ein End MLLMM
VLM based sxcttadd
1. RuhoCiPT
130M
19.1
54.0
29.4
17.3
54.1
27.0
2. CRIS
12K
70.5
73.2
56.1
65.3
63.1
S3.7
50.9
60.4
2. Konmon-2 2
9M
17.1
55.6
28.3
17.2
56.8
29.0
BIRUS
12K
71.00
06.00
04
DM
3. LISA
0.12M
25.2
62.0
26.3
24.8
61.7
35.5
VPD
12K
75.25
62.09
11M
28.9
65.8
39.6
27.2
64.6
MILM lusved moniludd
5. OMG-LLaVA
74K
26.9
64.6
26
62.8
USA
0.12M
74.9
79.1
72.3
65.1
70.8
56.
67.9
70.6
Fine Toming MLLM
GSVA
0.12M
73.2
59.8
73.3
GLAMM
1IM
79.5
74.9
6.
2MM,
200K
30.8
66.3
41.8
29.2
65.6
40,8
7.
OMG-LLaNA,
200K
29.9
65.5
28.6
64.7
Training methods
CL-CUP
26.56
05.64
24
35.92
33.67
Training- Free Prompr Learnnn
CLIPARNN
36.57
Segment Anyword
31.3
40.7
63.4
34.7
Segnemn Angwoed,
55.33
00199
9. Segment Anyword, woot
500
30.2
65.9
42.4
31.1
64.1
33.9
Table 3. Multi Object Reference Image Segmentation Performance on gRefCOcO Dataset
Table S. Open- Vocabulary Segmentation Performance PASCAL
Context 59 (PC-59) Dataset
Method
elmage
Val
TestA
TestB
Method
Training Data
PC-59
mloU
cloU
sloU
cloU
gloU)
cloU
glol
Truditional methods
Traditional methods
2.
LTS
MatiNet
115K
24K
52.30
47.51
48.24
52,70
61.87
59.30
62.64
49.96
45.33
46.14
50.42
2.
1.
GroupVIT
O'Segmenter
4.3M
12M
23.4
20.4
3.
VLT
24K
52.00
62.19
63.20
50.52
CLIP based methoos
4.
LAVT
12K
57.64
58.40
65.32
65.90
55.04
55.83
3.
SeyCLIP
3M
24.7
VLM haved mathods
MaskCLIP
0
26.4
5.
CRIS
12K
55.34
56.27
63.82
63.42
$1.04
$1.79
Fine-tuning difflasion based methods
6.
ReLA
12K
62:42
63.60
69.26
70.03
59.88
61,02
5. ODISE
83K
$7.3
MLLM based methods
Tradning-free difftsiion bused methods
7.
LISA
0.12M
38.72
32.21
52.55
48.54
44.79
39.65
6. OVDilT
0
32,9
8.
GSVA
0.12M
61.70
63.32
69.23
70.11
60.26
61.31
7. CaR
0
39.5
Fine-tumed methods
EmerDDiff
0
9.
LISA,
256
61.76
61.63
68.50
66.27
60.63
58.84
9.
Segment Anyword
0
52.5
GSVAA
16K
63.29
66.47
69.93
71.08
60:47
62.23
SAMAMLLM
110K
66.33
68.96
70.13
70.54
63.21
63.98
Training- Free Methods
CLIPacRNN
16.8
13. PSALM (Zero-Shot)
1 130M
42.00
43.30
52.40
54.50
50.60
52.50
ment Anyword on reference image segmentation, which aim to segment binary mask of text referred objects in im- age without requiring grounded mask-phrase mapping. We report multi-object reference image segmentation on gRef- COCO dataset in Table 3 and single-object reference image segmentation on RefCOCO, RefCOCO+ and RefCOCOg in Table 4. Following the original setup of GLaMM, we per- form cross-matching between predictions and ground truths based on IoU. Compared with zero-shot methods including CaR (Sun et al., 2024) and PSALM (Zhang et al., 2025), we achieve new state-of-the-arre splits, ranging from 18.40 gloU to 25.73 cloU. Despite fine- tuned methods such as SAM4MLLM (Chen et al., 2024) utilizing SAM to extract object localization information, Segment Anyword demonstrates superior performance with a notable margin of 1.40 cloU to 6.92 gloU, These results highlight that by simply leveraging visual concept prompts from
Segment Anyword,
500
67.73
66.08
73.57
74.63
image segmentation, as it includes multi-domain images from MSCOCO (Lin et al., 2014), PSG (Yang et al., 2022a), and SA-1B (Kirillov et al., 2023), along with detailed man- ual text descriptions. This diversity presents significant challenges in understanding and aligning multi-modal in- formation, matching object segmentation with associated noun-phrase. Our proposed Segment Anyword demon- strates strong perception capabilities compared to com- prehensive baseline methods. As shown in Table 2, our approach outperforms most recent MILMM methods and achieves results comparable to saute-of-the-ard models such as GLaMM (Rasheed et al., 2024) and OMG-LLaVA (Zhang et al., 2024). Notably, we establish new state-of-the-art results in AP50 and mIoU on the GranDf validation set. Compared to GLaMM, which also employs SAM as post- processing module, our method benefits from grounded vi- sual concept prompt learning, effectively preserving vision- language inference capabilities.
gReCCOCO across all
maps generated by a frozen diffusion model, our method effectively bridges discriminative seg- mentation tasks with generative models, offering significant advantages in designing perceptual systems capable of han- dling complex task such as simultaneously referring and segmenting multiple objects.
Reference Image Segmentation. We also evaluate Seg-
7

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Table 6. Ablation Study of core components of Segment Anyword on GranDf dataset Validation Split. PL: test-time prompt learning, RI: positive adjective prompt clustering from dependency regu- larization. R2: negative mutual-exclusive prompt binding from syntax regularization.
Components
GranD Validation
w/SAM
PL.
R2
mAP
mioU
X
x
x
8.2
13.7
X
x
x
16.9
32.4
(550 steps)
x
x
19.1
38.8
V (1100 steps)
x
x
22.1
42.6
(1000 steps)
V
x
24.9
62.2
(11100 steps)
28.5
63.1
<11000 steps)
31.3
67.4
V
/ (LoRA+550 steps)
V.
30.2
65.9
Figure 7. Predicate Mask Visualization. For predicate words with- out explicit semantic meaning, Segment Anyword can be prompted to identify the predicate masks, such as (a) a relation between sub- ject and object entities and (b) human-object interaction.
Semantic Segmentation. We also com- pare Segment Anyword with previous open-vocabulary seg- mentation methods using Pascal Context 59 (PC59), which contains 59 object categories (Table 5). Segment Anyword reports a new state-of-the-art result, surpassing other diffu- sion based training free methods such as OVDiff (Karazija et al., 2024) and EmerDiff (Namekata et al., 2024) by largin margin (19.6 and 6.8 mloU). This result validates that without generating synthetic support image sets or storing intermediate features, retrieve visual prompts from cross- attention map from a frozen diffusion model can still achieve superior segmentation result. Despite ODISE (Xu et al., 2023b) trained additional mask encoder and decoder with large number of samples, Segment Anyword shows compa- rable semantic segmentation capability. which provides feasible solution for deploying training free open-vocabulary semantic segmentation methods in low resource conditions.
suboptimal performance for segmentation task, which requires pixel-level mask delination.
Textual Embedding Update. We validate the contribution of test-time textual embedding updating, which only up- dates a very small number of parameters for aligning token embedding with image features. As reported in Table 6, we obtain 6.4 mIoU increase with 550 optimizing steps and fur- ther 10.2 mIoU increase with 1 1100 optimizing steps. This result validates the findings from our motivational study that without test-time textual embedding optimization, vanilla Segment Anyword suffers from terminology variance. We also validate that using LoRA to fine-tune text-encoder can achieve fast text domain adaptation, decreasing the updating steps of textual embedding updating from 1100 to 50 while preserving mask surrogates effectiveness for downstream promptable segmentation, providing a reasonable trade-off between inference speed and accuracy.
3.2. Ablation Study
We perform ablation studies to fully understand each fac- tor contribution using GranDf Validation split. We focus on the best performing variant, Segment Anyword with textual embedding update (PL), positive adjective prompt clustering from dependency regularization (R1), negative mutual-exclusive prompt binding from syntax regulariza- tion (R2), promptable segmentor post-processing (w/SAM) and text encoder LoRA fine-tune for fast textual embedding updating within 50 steps.
Linguistic Guided Visual Prompt Regularization. We ex- plored the influence of linguistic guided visual prompt regu- larization. As reported in Table 6, Segment Anywordpy. achieves best performance with final 31.3 AP50 and 67.4 mloU on GranDf validation split. The results suggest that injecting linguistic information, both dependency structure and syntax structure, can generate robust visual prompt combination with best capability for noisy tolerance, which helps promptable segmentation generator such as SAM elim- inate false- possiivvs areas, bind inclusive object attribute frag- ments and generate boundary distinct segmentation masks.
SAM Proc-pgr,, We first validate the boosted per- formance from SAM as a post-processing module. We examine the proposed method, Segment Anyword, which uses a thresholded cross- aats-nattnn mask as the segmenta- tion output. We show boosted segmentation performance improved from 13.7 to 32.4 mloU. This verifies our obser- vation of plain Segment Anyword, where the cross-attention map can capture the poucept-promp correlation but lacks fine-graines grained details at object mask boundaries, leading to
8

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Iopux
Croen Thuak
Amroga Crow Anenics
Sugnect Anproed (Dha))
mance as the text reference varies. In contrast, Segment Any- word is a novel visual concept prompt learning framework that leverages frozen diffusion model leveraging intermedi- ate cross- cross-attention maps with optimized texutal embedding to localize token-level visual concepts for promptable seg- mentation. We propose novel prompt binding and clustering regularization that enable the direct transfer of linguistic structure knowledge into visual prompts, which enhancing the effectiveness of the mask prompts with dependency and syntax information. Extensive experiments across differ- ent tasks and datasets demonstrate our approach's 's superior performance, confirming its robustness and adaptability.
Rb st anlatsenun
with www.rreeeaaa add sipe
langing fiom
Figure 8. Failure Case Visualization. Our method struggles with very tiny objects composed with thin structures, due restricted cross-attention resolution.
3.3. Predicate Segmentation
A Acknowledgment
Language-driven visual concept discovery is human- computer interaction where a user describe an image us- ing multiple unfamiliar concepts, The entities of interest are often concrete linguistic subjects and objects, with ex- plicit visual geometric and semantic characteristics. In con- trast, predicate words such as verbs or gerunds with abstract meanings, have been less explored in previous multi-modal segmentors. However, predicate words are essential for link- ing, reasoning, and understanding multi-modal information. Without training or tuning additional modules, we demon- strate that Segment Anyword can also effectively learn a transferable visual prompts for abstract predicate words. As shown in Figure 7, Segment Anyword can not only prompt concrete object masks but also reveal the correlation be- tween object-object ('pulling' trailer) and human-objies ('holding' racket). To the best of our knowledge, Segment Anyword is the first approach capable of handling both con- crete and abstract visual concepts in open-set segmentation. By learning predicate correlations representations of visual entities, it can reduce hallucinations in image generation and editing. Furthermore, it holds great potential to ac- celerate scientific knowledge discovery by learning entity relationship from experimental observations or textbooks.
We thank the area chair and reviewers for their constructive feedbacks. Most experiments were performed using the Sulis Tier 2 HPC platform hosted by the Scientific Com- puting Research Technology Platform at the University of Warwick. Sulis is funded by EPSRC Grant EP/T022108/1 and the HPC Midlands+ consortium.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Bruce, J., Balch, T., and Veloso, M. Fast and inexpen- sive color image segmentation for interactive robots. In Proceedings. 2000 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS 2000)(Cat. No. 00CH371 13), volume 3, pp. 2061-2066. IEFE, 2000.
3.4. Failure Case Analysis
Although Segment Anyword achieves promising result, we acknowledge that it can still encounter failure segmentation. Similar to other image segmentation methods, Segment Any- word struggles to segment tiny objects with tiny structures (Figure 8), as the averaged cross-attention offers limited detail for precise shape contours due to its restricted reso- lution size (16x 16). Analyzing Segment Anyword with a high-resolution backbone is promising direction in future.
Burgert, R., Ranasinghe, K., Li, X., and Ryoo, M. S. Peek- aboo: Text to image diffusion models are zero-shot seg- mentors. arXiv preprint w:Xiv:2211.13222 2022.
Butler, D. J., Elliot, S., and Cakmak, M. Interactive scene segmentation for efficient human-in-the-loop robot ma- nipulation. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2572-2579, IEEE, 2017.
4. Conclusion
Chen, Y.-C., Li, W.-H., Sun, C., Wang, Y.-C. F., and Chen, C.-S. Sam4mllm: Enhance multi modal large language model for referring expression segmentation. In European Conference on Computer Vision, pp. 323-340, 2024.
We introduce Segment Anyword for open-set language grounded segmentation. Our motivational study reveals that current VLMs or MILLMs struggle to achieve stable perfor-

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Chiang, W.-L.,, Li, Z.,, Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:/ ht/p/mmmmpp http///sssssssllll//eee org/blog/ 2023-03-22 100-33-viiiun 30-vicuna/.
Jing, Y., Kong, T., Wang, W., Wang, L.,, Li, L.,, and Tan, T. Locate then segment: A strong pipeline for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9858-9867, 2021.
Ju, X.,, Zeng, A., Bian, Y., Liu, S., and Xu, Q. Pnp inversion: Boosting diffusion-based editing with 3 lines of code. International Conference on Learning Representations (ICLR), 2024.
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M.. Benenson, R., Franke, U.. Roth, S., and Schiele, B. The cityscapes dataset for semantic urban scene un- derstanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Karazija, L.,, Laina, I., Vedaldi, A., and Rupprecht, C. Dif- fusion models for open-vocabulary segmentation, 2024. URL https::t///... /ttpttp://aaxx..oo/ hatp/vxivvvvvv.
aaa//bb/////0/ 2306. 09316. 6.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv: 1et..08800.00 2018
Karazija, L.., Laina, I., Vedaldi, A., and Rupprecht, C. Dif- fusion models for open-vocabulary segmentation. In Eu- ropean Conference on Computer Vision, pp. 299-317. Springer, 2025.
Ding, H., Liu, C., Wang. S., and Jiang, X. Vision-language transformer and query generation for referring segmen- tation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp.
Kazemzadeh, S., Ordonez, V., Matten, M., and Berg, T. Referitgame: Referring to objects in photographs of natu- ral scenes. In Proceedings of the 2014 conference on em- pirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
2021. Eger, S., Sahin, G. G., Rücklé, A., Lee, J.-U., Schulz, C.,, Mesgar, M., Swarnkar, K., Simpson, F., and Gurevych, I. Text processing like humans do: Visually attacking and shielding nlp systems. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics. pp. 1634-1647, 2019.
Kenton, J. D. M.-W. C. and Toutanova, L. K. Bert: Pre- training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLLT, volume 1, pp. 2. Minneapolis, Minnesota, 2019.
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint (xXxx:2008.01618, 2022.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision 2023.
pp.
Geiger, A., Lenz, P. Stiller, C., and Urtasun, R. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(1 1):1211-1237, 2013.
Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., and Jia, J. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579 9589, 2024.
He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask r- cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Le, T., Ye, Y., Hu, Y., and Lee, D. Cryptext: Database and interactive toolkit of human-written text perturbations in the wild. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pp. 3639-3642. IEEE, 2023.
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-or, D. Prompt-to-prr toootpptppppppoooo image editing with cross-attention control. In The Eleventh Interna- tional Conference on Learning Representations, 2023.
Lin, T.-Y,, Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitncck C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Ho, J., Jain, A., and Abbeel, P. Denoising diffuuson proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Jin, C., Tanno, R., Saseendran, A., Diethe, T., and Teare, P. A. An image is worth multiple words: Discovering prompt learn-
object level concepts using ing. In Forty-first International Conference on Machine Learning, 2024.
Lin, Z., Wang. Y., and Tang, Z. Training-free free open open-ended ob- ject detection and segmentation via attention as prompts. arXiv preprint arXiv:24 10.05663, 2024.
10

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A., Van Gin- neken, B., and Sánchez, C. I. A survey on deep learning in medical image analysis. Medical image analysis, 42: 60-88, 2017.
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., and Cohen- Or, D. Null-text inversion for editing real images us- ing guided diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6038-6047, 2023.
Liu, C., Ding, H., and Jiang. X. Gres: Generalized re-
Mottaghi, R., Chen, X., Liu, X.,, Cho, N.-G., Lee, S.-W., Fidler, S., Urtasun, R., and Yuille, A. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 891-898, 2014.
ferring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 23592-23601, 2023.
Liu, L.,, Ouyang, W., Wang. X., Fieguth, P., Chen, J., Liu, X., and Pietikäinen, M. Deep learning for generic object detection: A survey. International journal of computer vision, 128:261-318, 2020.
Mukhoti, J., Lin, T.-Y., Poursaeed, O., Wang, R., Shah, A.,
Torr, P. H., and Lim, S.-N. Open vocabulary semantic segmentation with patch aligned contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19413-19423, 2023.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V.
Roberta: A robustly optimized bert pretraining approach. arXiv preprint arX.v:1907.1 2019.
Namekata, K., Sabour, A., Fidler, S., and Kim, S. W. Emerdiff: Emerging pixel-level semantic knowledge in diffusion models. arXiv preprint arXiv:2401.11739 2024.
Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3431-3440, 2015.
Patashník, O., Garibi, D., Azuri, I., Averbuch-Elor, H., and Cohen- Or, D. Localizing object-level shape variations with text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 23051-23061, 2023.
Luo, G., Dunlap, L., Park, D. H., Holynski, A., and Darrell, T. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems, 36, 2024.
Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma. S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 2nn::2366.18244 2023.
Luo, H., Bao, J., Wu, Y., He, X., and Li, T. Segelip: Patch aggregation with learnable centers for open- vocabulary semantic segmentation. In International Conference Machine Learning, pp. 23033-23044 PMLR, 2023.
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641- 2649, 2015.
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A. L., and Murphy, K. Generation and comprehension of unam- biguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 11-20, 2016.
Marcos-Manchón, P., Alcover-Couso, R., SanMiguel, J. C.,, and Martínez, J. M. Open-vocabulary attention maps with token optimization for semantic segmentation in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9242- 9252, 2024.
Radford, A. Improving language understanding by genera- tive pre-training. 2018
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. $748-8763. PMLR, 2021.
Meng, B., Xu, Q., Wang, Z., Cao, X., and Huang, Q. Not all diffusion model activations have been evaluated discriminative features, 2024. URL https: //axiv.. org/ htg//bbs/ae /410. 03558.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In International conference on ma- chine learning. pp. 6821-8831, Pmlr, 2021.
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramet J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging, 34(10):1993 1994-204,, 2014.
Ranasinghe, K., McKinzie, B., Ravi, S., Yang, Y., Toshev, A., and Shlens, J. Perceptual grouping in contrastive vision-language models. In Proceedings of the IEEF/CVE

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
International Conference on Computer Vision, pp. 5571- 5584, 2023.
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J. Yfcc100ml The new data in multimedia research. Communications of the ACM, 59(2): 73, 2016.
Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R. M., Xing, E., Yang, M.-H., and Khan, S. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13009- 13018, 2024.
Tian, J., Aggarwal, L.., Colaco, A., Kira, Z., and Gonzalez- Franco, M. Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion. arXiv preprint arXiv:2308. 12469, 2023.
Vinker, Y., Voynov, A., Cohen-Or, D., and Shamir, A. Con- cept decomposition for visual exploration and inspiration. ACM Transactions on Graphics (TOG), 42(6):1-13, 2023,
Ren, Y., Xia, X., Lu, Y., Zhang, J., Wu, J., Xie, P., WANG, X., and Xiao, X. Hyper-SD: Trajectory segmented consis- tency model for efficient image synthesis. In The Thirty- eighth Annual Conference on Neural Information Pro- cessing Systems, 2024.
Wang. Z., Lu, Y., Li, Q., Tao, X., Guo, Y., Gong, M., and Liu, T. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11686-11695, 2022.
Rombach, R., Blattmann,
A., Lorenz, D., Esser, P., and Ommer, B. rigghrresoltion image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Wu, J., Li, X., Xu, S., Yuan, H., Ding, H., Yang, Y., Li, X., Zhang, J., Tong, Y., Jiang, X., et al. Towards open vocabulary learning: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024.
Ronneherger, O., Fischer, P., and Brox, T. U-net: Convolu- tional networks for biomedical image segmentation. In In- ternatoonal Conference on Medical image computing and computer-assisted intervention, pp. 234-241. Springer, 2015.
Xia, W., Zhang, Y., Yang, Y., Xue, J.-H., Zhou, B., and Yang, M.-H. Gan inversion: A survey, IEEE transactions on pattern analysis and machine intelligence, 45(3):1121- 3138,20 2022.
Xia, Z., Han, D., Han, Y., Pan. X., Song, S., and Huang, G. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 3858-3869, 2024.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500-22510, 2023.
Xu, J., De Mello, S., Liu, S., Byeon. W., Breuel, T., Kautz, J., and Wang. X. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 18134-18144, 2022.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
Sun, C., Shrivastava, A., Singh, S.,, and Gupta, A. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843-852, 2017.
Xu, J., Hou, J., Zhang, Y., Feng, R., Wang, Y., Qiao, Y., and Xie, W. Learning open-vocabulary semantic seg- mentation models from natural language supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2935-2944. 2023a.
Sun, S., Li, R., Torr, P.,, Gu, X., and Li, S. Clip rnn: Seg- ment countless visual concepts without training endeavor. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 13171-13182, 2024.
Xu, J., Liu, S., Vahdat, A., Byeon, W., Wang, X., and De Mello, S. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pp. 2955-2966, 2023b.
Tang, R., Liu, L., Pandey, A., Jiang, Z., Yang, G., Kumar, K., Stenetorp. P., Lin, J., and Türe, F. What the daam: Interpreting stable difffusion using cross attention. In Pro- ceedings of the sl Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 2023.
Xu, Z., Chen, Z., Zhang, Y., Song, Y., Wan, X., and Li, G. Bridging vision and language encoders: Parameter- efficient tuning for referring image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 17503-17511 2023c.
12

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Yang, J., Ang. Y. Z.., Guo, Z., Zhou, K., Zhang, W., and Liu, Z. Panoptic scene graph generation. In European Conference on Computer Vision, pp. 178-196. Springer, 2022a.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning prompt for vision-language models. International Jour- nal of Computer Vision, 130(9):233--2348 2022d.
Zhou, T., Zhang, F., Chang, B., Wang, W., Yuan, Y., Konukoglu, E., and Cremers, D. Image segmentation in foundation model era: A survey. arXiv preprint arXiv: 240.x 21..2244...22222 12957, 2024.
Yang. Z., Wang, J., Tang, Y., Chen, K., Zhao, H., and Torr, P. H. Lavt: Language-ave aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18155-18165, 2022b.
7hu, C. and Chen, L.. A survey on open-vocabulary detec- tion and segmentation: Past, present, and future. IEEE Transactions Pattern Analysis and Machine Intelli- gence, 2024.
Yu, L., Lin, Z., Shen, X., Yang. J., Lu, X., Bansal, M., and Berg. T. L. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1307-1315, 2018.
Zhu, J.-Y., Krähenbühl, P., Shechtman, E., and Efros, Generative visual manipulation on the natural image man- ifold. Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11- 14, 2016, Proceedings, Part V 14, pp. 597-613. Springer, 2016.
Yu, S., Sco, P. H., and Son, J. Zero-shot referring im- age segmentation with global-local context features. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 19456-19465, 2023.
Zhang, T., Li, X., Fei, H., Yuan, H., Wu, S., Ji, S., Loy, C. C., and Yan, S. Omg-llava: Bridging image-level, object- level, pixel-level reasoning and understanding. arXiv preprint ar Xrxi::2406. 19389, 2024.
Zhang, Z., Ma, Y., Zhang, E., and Bai, Psalm: Pixelwise segmentation with large multi-modal model. In European Conference Computer Vision, pp. 74-91. Springer, 2025.
Zhao, W. Rao, Y., Liu, Z., Liu, B., Zhou, J., and Lu, J. Unleashing text-to-image diffusion models for visual per- ception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5729-5739, 2023a.
Zhao, Y., Lin, Z., Zhou, D., Huang, Z., Feng, J., and Kang, B. Bubogpt: Enabling visual grounding in multi-modal 1lms. arXiv preprint arXiv:2307.0888) 2023b.
Zheng, L., Chiang, W.-L., Sheng. Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, 7.,, Li, D., Xing, E., et al. Judging alm--s-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Zhou, C., Loy, C. C., and Dai, B. Extract free dense labels from clip. In European Conference Computer Vision, pp. 696-712. Springer, 2022a.
Zhou, C., Loy, C. and Dai, B. Extract free dense labels from clip. In European Conference on Computer Vision (ECCV), 2022b.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Conditional prompt learning for languae models. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18816-16825, 2022c.
13

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
A. Additional Motivational Study Results
loll Mean vs. loU Std ETRIS refroca+
80
SoU Mean
40
30
50
Iu Std
Figure Motivational Study the stale-of-the-art parameter efficient fine-tuning model ETRIS (Xu al., 2023c) for reference segmentation.
Mean odl Dencat PetA gwwlooco
Istl Out BeTTAED Tecms
MAA
(a)
(b)
Figure 10. (a) Scatter plot IoU mean versus IoU standard deviation per image sample on the gRefCOCO dataset against ReLA. (b) IoU standard deviation per image sample the ReiCOCO+ dataset against ETRIS. Yellow dots
Scatter plot of loU represent results from our method, while blue dots represent results from baseline methods. The plots show that by focusing on test-time optimization, method effectively improves segmentation accuracy and maintains stability under diverse textual expressions, pushing the results toward the top-left corner.
We present additional motivational study using another state-of-the-arr model, ETRIS (Xu al., 2023c), in Fig. ETRIS focuses aligning representations from pre- pre-trained visual and language encoders by introducing intermediate fine-tuned adapters. The results further validate our previous findings: without test-time alignment, current open-set segmentation models are vulnerable to input variance, leading to unstable segmentation results.
As demonstrated through comprehensive experiments and additional results in Fig. 10, our method effectively improves both accuracy and stability by leveraging an off-the-shelf diffusion model extract mask prompts via an inversion process. This approach is modular and pluggable, offering several advantages including the ability to handle novel visual and linguistic concepts without requiring additional supervision complex training procedures.
14

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
B. Related Works
Vision Language Representation Learning Learning a generalized and transferable vision-language representation has been extensively explored, driven by particular tasks such as visual question answering (Antol et al., 2015) and recently rocket-rising Artificial Intelligence Generated Content (AIGC) (Ramesh al., 2021; Rombach et al., 2022). Building on the advancements of pre-trained language models (Kenton & Toutanova, 2019; Radford, 2018), large-scale VLMs were proposed for training with image-text pair contrastive learning (Radford et al., 2021) over large-scale multi-mode datasets (Sun et al., 2017; Thomee et al., 2016). Such strong VLMS transformed the paradigm of visual task setting, with task complexity increasing from pre-defined close-set categorical labels to free-form open-set natural language references (Zhou et al., 2024). Recent works such as CLIPpy (Ranasinghe et al., 2023) and PACL (Mukhoti et al., 2023) assign labels by contrasting image patch embeddings with object text label embeddings. However, these methods still require specific training configurations. Peekaboo (Burgert et al., 2022) is closely related our approach, as both aim to learn visual concepts using aff-the-shelf diffusion models. However, Peekaboo heavily relies on alpha map initialization and is sensitive to its quality. Other seminal works in prompt learning, such CoOp (Zhou et al., 2022d), introduced prompt tuning to adapt vision-language models like CLIP for few-shot tasks. However, CoOp is limited when handling unseen classes. Its successor, CoCoOp (Zhou et al., 2022c), extended this approach by providing input-conditioned adaptability, making it more suitable for open-world segmentation tasks involving diverse textual expressions. While CoCoOp demonstrates robustness under rigorous testing, it introduces additional computational complexity.
While the results are promising, our motivational study described in Section 2.2 highlights that discovering and aligning visual concepts between dense pixel-level mask representations and word-level text embeddings remains challenging, leading to unstable and confounded segmentation results.
Open-Set Image Segmentation. Unlike traditional image segmentation which operates within a close-set pre-defined labels, open-set image segmentation requires recognizing previously unseen objects and categories descrihed using novel words or sentences (Zhu & Chen, 2024; Wu et al., 2024), which is essential for real-world applications where adaptability to diverse terminology is paramount. Early efforts leveraged powerful pre-trained VLMs such as CLIP (Radford, 2018) to bridge the gap between visual and textual modalities by contrastive learning, prototype matching, and similarity clustering (Zhou et al., 2022b; Wang et al., 2022; Xu et al., 2023b;c). Although effective to some extent, they often fall short in handling complex and descriptive sentences encountered. Recent advancements introduced multi-modal large language models to improve the understanding of natural language descriptions and decoding final mask tokens (Lai et al., 2024; Rasheed et al., 2024; Xia et al., 2024; Zhang et al., 2024). However, these approaches require substantial computational costs for training or fine-tuning, where performance deteriorates when faced with vocaaulooca words not covered by pre-trained models. Methods including (Xia et al., 2024; Chen et al., 2024) can generate multiple object masks within a single binary segmentation map. However, these masks are not explicitly associated with specific word indices. Moreover, neither model is capable of handling novel concepts, as both require training or fine-tuning on the target dataset. This limits their ability to generalize to unseen concepts during test time. In practical applications, text references tend to be highly complex and detailed as users provide intricate descriptions to guide segmentation tasks (Eger et al., 2019; Le et al., 2023). To overcome these limitations, we introduce Segment Anyword, a novel framework that adaptively aligns to diverse terminologies to describe unified visual concepts or objects. By leveraging the inverse scalability of an sff-the-shelf pre-trained diffusion model, Segment Anyword achieves superior adaptability with minimal visual prompt optimization effort, offering a practical and efficient solution for open-set segmentation.
Diffusion Models for Image Segmentation. Diffusion models have significantly advanced image generation, recognized for their proficiency in modeling data distributions through forward noising and backward denoising processes (Ho et al., 2020). Beyond powerful generative capabilities, diffusion models offer valuable insights into discriminative tasks such as image classification and segmentation (Luo et al., 2024; Meng et al., 2024). Previous works, such as EmerDiff (Namekata et al., 2024), DAAM (Tang et al., 2023) and OVAM (Marcos-Manchón et al., 2024), have successfully detected objects by identifying salient perturbations in attention maps. However, they often struggle with robust language grounded object instantiation. Other approaches, such as VPD (Zhao al., 2023a), have sought to enhance segmentation by fine-tuning mask decoder pre-trained diffusion network, which requires issturceinttessvve pixel-wise annotations. Zero-shot methods like OVDiff (Karazija et al., 2024) and DiffusionSeg (Tian et al., 2023) typically require intricate post-processing steps, including prototype matching and iterative refining. In contrast, Segment Anyword employs frozen text-to-in to-image diffusion model that operates without additional training or fine-tuning, which is not only more computationally efficient, simpler, and more straightforward, but also remains highly effective.
15

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
C. Dataset and Implementation
C.1. Implementation Details
In our motivational study, we generate the mutated expressions by querying ChatGPT4o, with a template prompt as follows:
Prompt:
As a NLP expert. please genrate a list of n synonyms of the noun phrases in the following [sentence] and output the list separated by" '&'
where n is in randint(2, 5) and [sentence] is set to original labeled text referring expression.
Unless specified, we retain the original hyper- nyper-parameter of latent diffusion model (LDM) (Patashnik et al., 2023) as our diffusion backbone. We follow the visual concept discovery process specified in multi-concept prompt learning (MCPL) (Jin et al., 2024). Differently from MCPL that uses pseudo placeholder for concept learning, we use the pre-defined text expression from the data as condition prompt where all nouns, adjectives and prepositions are explicitly described by human annotator. Our experiments were executed on a single 40G A 100 GPU with batch size of 8. The base learning rate for textual embedding was set to 0.005. The hyper-parametee of textual embedding updating remains the same in LDM and MCPL, with the temperature and scaling term (T. of (0.3, 0.00075). We use BERT (Devlin, 2018) to generate token embeddings. For words included in BERT's pre-trained vocabulary, we directly use their pre-trained embeddings. For vocaulauyy words, token embeddings are randomly initialized. By leveraging the pre-trained token embedding initialization, we reduce the optimization steps to 1,100, achieving speedup of 6x compared to LDM and MCPL. To accelcrate inference time textual embedding update speed, we further introduce LoRA fine-tuned text encoder for fast inference time textual embedding update (Segment Anyword,). For each evaluation dataset, we randomly sample 500 image-text pairs from training set to perform a LoRA fine-tune on BERT encoder only, with LoRA hyper-parameter r = 16. With LoRA fine-tuned BERT text encoder, Segment Anyword, achieve a fast inference time text domain adaptation, decreasing textual embedding update steps from 1100 to 50 by sacrificing a relative small accuracy.
We choose the fine-tuned version of Vicuna-7B-v1.55 (Zheng et al., 2023) as our large language model (LLM) parse the text prompt and generate the noun phrases, which is adopted by previous state-of-the- grounding segmentation method (Rasheed et al., 2024) with a low-rank adaptation scale = 8. We keep LLM parameters frozen. For noun phrases parsed from the original text prompt, we use pfrt-of-speech tagging to identify the root noun as the object to be segmented and the adjective modification (amod) as the positive prompt binding.
We use cross-attention maps at the 16x16 resolution, averaged across all denoising time steps, to obtain the final cross- attention. This setup follows prior works such as MCPL (Jin al., 2024) and Prompt2rrompt (Hertz et al., 2023), ensuring fair and consistent implementation for updating textual embeddings. For the post-processing module, utilize frozen SAM with ViT-H as the promptable mask generator.
C.2. Dataset Details
GranDf (Rasheed et al., 2024) is motivated by the need for higher-quality data during the fine-tuning stage. It comprises 214K image-grounded text pairs, along with 2.5K validation samples and 5K test samples, sourced from an extension of open-source datasets, including Flickr-39K (Plummer et al., 2015), RefCOCOg (Kazemzadeh et al., 2014), and PSG (Yang et al.. 2022a). The original aim of GranDf is for Grounded Conversation Generation (GCG), involving both reference generation and grounded image segmentation. As a segmentation model, we only focus on segmentation capability evaluation by using ground truth text expression as segmentation reference.
RefCOCO (Kazemzadch al., 2014) dataset contains 142,210 referring expressions for 50,000 objects across 19,994 images. Collected from MSCOCO (Lin et al., 2014) through two-player game, it is divided into 120,624 training, 10,834 validation, 5,657 testA, and 5,095 testB samples. On average, each referring expression has mean length of 3.6 words.
ReSCOCO+ (Kazemzadeh et al., 2014) dataset consists of 141,564 referring expressions linked to 49,856 objects in 19,992 images. follows the same rrain-validation-ees split as RefCOCO, with 120,624 training, 10,758 validation, 5,726 testA, and 4,889 testB samples. Unlike RefCOCO, RefCOCO+ excludes absolute- -lcatton words, making it a more challenging benchmark for referring image segmentation with averaged sentence length of 3.53 words.
16

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
RefCOCOg (Mao al., 2016) comprises 104,560 referring expressions for 54,822 objects across 26,711 images. Unlike RefCOCO and RefCOCO+, its natural expressions more descriptive phrases, averaging 8.4 words per expression. Additionally, RefCOCOg contains more detailed location and appearance- descriptions, and in this work, we adopt the UNC validation and test partition for evaluation.
collected from Amazon Mechanical Turk, resulting in longer and
gRefCOCO (Liu et al., 2023) dataset comprises 278,232 referring expressions associated with 19,994 images, including 80,022 multi-target and 32,202 empty-target expressions. Following the UNC partition RefCOCO, the dataset is divided into training, validation, testA, and testB subsets. The validation set contains 1,485 images with 5,324 sentences, while testA includes 750 images with 8,825 sentences, and testB consists of 749 images with 5,744 sentences.
PASCAL Context (Mottaghi al., 2014) dataset is extension of the PASCAL VOC 2010 detection challenge, with pixel-level segmentation mask annotation. We used a subset, which contains 59 frequent classes (PC59) for evaluation, with 5,100 images in validation set.
17

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
D. Competing Baselines
Below we present related works we selected as comparison baselines, where we focusing on comparison with previous state-of-the-an methods, also fair comparison based on same training frree pipeline, SAM as mask refine module and frozen text-to-image diffusion model as backbone. For task definition, please refer Figure II.
Image
Reference
Sogmentation
ls standing on the Tooe showe
dddrggonshape fring. hoticeably unjaceett the
Open- Sel Language Grounded Segmeniation
(GranDDi)
Th
Cross-Attention and Promptable Mask from Sepment Anyword
man
sand
dragonshaped
kite
See
Test Reference
Segmentation
Multi-Objoc
dog
Reference Image Segmentation (gRefCOCO, RefCOCO, RefOOCO+, RefCOCOg)
Lags left
Single Objec
Crass-Alienion and Promptable Mask from Segment
AngrinntAyyward Anyward
dog
chair
leg
Image
Labels
Segmentation
Cat, Woll, Tiee, Wond
Conestarate
Open- Vocabulary Sogmentation (Pascal Context 59)
Reference
An image of and and free and wood.
Cross-A Altention and Promptable Mask from Sogment Anyword
tree
wood
Figure 1.. Open-set Image Segmentation Comparison. From top bottom, the complexity decrease from free-form descriptive expression to key-word concatenation. Note that some sentence description in RefCOCO could he amhiguity, as the dataset designs for implicit reasoning and referring.
18

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
D.I. Open-Set Image Segmentation
BuboGPT (Zhao et al., 2023b) is one of the earliest attempts working multi-model open-set grounded image segmentation. By fine-tuning on novel instruction dataset, BuboGPT achieves fine-grained object-level understanding in both conversation generation and grounded recognition using an entity integrated text prompt template *( (List of Entities) ). ( List of Text)--
Kosmos-2 (Peng et al., 2023) enables the perception of object annotation text into visual understanding, with format string instruction template 'it List of annotations ), List of Text ). Kosmos-2 advances the multi-modal understanding capability with novel object location tokens for various downstream tasks including grounded segmentation.
LISA (Lai et al., 2024) further improve MLLM learning and reasoning with novel question-answering instruction template, predicting the final object segmentation mask token as an answer token 'Sure, it is SEG y. Trained on datasets from both semantic segmentation, reference image segmentation and visual question answering, LISA showsing promising results on domain generalization and reasoning segmentation with world knowledge.
GLaMM (Rasheed et al., 2024) further improves MILM understanding by accommodating both text and visual prompts. GLaMM added region level encoders and decoders trained on GranD dataset to extract mask information during MLLM training and fine-tuning.
OMG-LLaVA (Zhang et al., 2024) extends MILM reasoning capability at pixel-level by integrating image information, perception priors, and visual prompts into visual tokens. OMG-LLaVA achieves universal segmentation with a fine-tuned molti-modal projector and visual decoder.
D.2. Reference Image Segmentation
MattNet (Yu al., 2018) decomposes the reference expression into three submodules relates to object appearance, location and relationship using a Bi-LSTM, improving localization and segmentation performance.
LTS (Jing et al., 2021) extract the textual embedding through a GRU and multiply the bottleneck layer image feature within ConvNet, mimic the perceptual process of 'Locate the Segment'.
VLT (Ding et al., 2021) trained a transformer based vision-language matching network to query reference expression with image features.
LAVT (Yang et al., 2022b) replacing the backbone with a vision transformer and inject textual embedding from BERT with attention module.
CRIS (Wang et al., 2022) utilize CLIP model to contrast reference expression and semantic information by training a multi-modal projector with text-to-pixel constrastive learning.
ETRIS (Xu et al., 2023c) utilize CLIP model as backbone. In addition, it trained an additional bridging module between vision encoder and text encoder to minimize the aligmment gap between mudti- representations.
ReLA (Liu et al., 2023) is the first model for multi object reference segmentation framework featureing region-to-text cross-attention.
GSVA (Xia al., 2024) utilized fine-tuned MLLM for reference segmentation. A segmentation mask decoder generates the segmentation mask with joint input from mask token and image features.
SAMAMLLM (Chen al., 2024) provide detailed instruction template for MLLM using SAM to filter object-of- interests, integrating location information into textual template List of BBox )..
GL-CLIP (Yu et al., 2023) injecting the global and local features using mask proposals for both image and textual embedding, where the final mask the filtered by the cosine-similarity based feature matching.
CLIPasRNN (Sun et al., 2024) recurrently filter out in irrelevant visual concepts without any training or fine-tuning which preserves the original vocabulary space of off-the-shelf model.
PSALM (Zhang et al., 2025) extends the MLLM with expert crafted input with images, task instructions, conditional prompts, and mask tokens, which shows promise zero-shot generalization on multi-object reference segmentation.
19

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
VPD (Zhao al., 2023a) adopts diffusion model as backbone and trained an additional task-specific decoder to generate output based intermediate features from diffusion model input.
D.3. Open-Vocabulle Semantic Segmentation
OVSegmentor (Xu et al., 2023a) trains a multi-modal transformer to jointly learning image features with raw caption, masked caption and prompted entity for open- open-vocabulary semantic segmentation.
GroupViT (Xu et al., 2022) clusters vision transformer intermediate features into superpixel representations, which is further contrastive learned with textual embedding for open-vocabulary segmentation.
ODISE (Xu et al., 2023b) is of the earliest attempts utilizing frozen diffusion model for open-vocabulary segmentation, where the frozen diffusion model generates mask proposals, which is optimized by matching masked-out object features with textual embeddings of object labels.
MaskCLIP (Zhou et al., 2022b) is one of the earlies attempts proposing train-free and tuning- free methods for open- vocabulary semantic segmentation. MaskCI.IP reorganizing CLIP text encoder classifier to filter image embedding for specific object label.
SegCLIP (Luo et al., 2023) utilize pre-trained CLIP model to learn and contrastive aggregate image patches into superpixels for image reconstruction, which facilities CLIP model with pixel-level understanding.
OVDiff (Karazija et al., 2024) take advantage of promising generation capability of diffusion model to generate synthetic support set of images for target object to be segmented, where the object feature are filtered by support set image feature and textual embedding.
EmerDiff (Namekata al., 2024) leveraging the intermediate features of a frozen diffusion model, where the segmenta- tion mask are first constructed by k-means clustering feature maps and further refined by evaluating with semantic correspondence through feature-level perturbation operation.
20

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
E. Additional visualization on Suboptimal Mask from Plain Segment Anyword
Below we present additional visualization on suboptimal segmentation mask from plain Segment Anyword. Through text-to-image reconstruction, we can observe that token-level atssettttntionn map reflects the correlation between visual concepts and textual representation. We can obtain a naive cross-attention mask through a hard threshold (here is 0.5) initial segmentation surrogates. Such mask is often coarse and noisy, lacking fine-grained object shape and boundary refinement. Thus we regard such cross-attention as a well candidate for downstream promptable segmentation, where we can randomly sample points as object location prompts, further mined by our novel linguistic guided dependency and syntax structural information regularization.
AttnMask AttnMask
Input
Mask
Token-wise Cross-Attention Map
"A @(person)) in gray lean chair and black #(dog))
"A silver ((truck) mext to blue shirt (gguy))
"A #(voman) green shirt (wwoman) uniform" "
"#(Athlete) in a red uniform and ((referee) in black suit"
"@(Gir)) with blonde hair and #(Woman) in blue shirt" "@(Kid) in pink jacket and H ((woman)) red glove"
A red "@(ambulance) parked dirt #(roady
Figure 12. Additional Suboptimal segmentation masks from plain Segment Anyword. We show that by reconstructing the input image, Segment Anyword can leverage the cross-attention map from frozen diffusion model as a localization prior to generate mask surrogates. However, hard threshold cross-altention mask suffer from inaccurate shape recognition and boundary delination.
21

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
F. Additional Ablation Study Results
F.I. Ablation study on pos-tagging methods
The primary focus of the proposed Segment Anyword is to improve the quality automatic mask prompts without relying on complex training configurations. The external language model is merely used parse and index object-related words for cross-attention map retrieval. Importantly, the use of fine-tuned LLM (e.g.. Vicuna (Chiang et al., 2023)) is required-our method is compatible with state-of-the-art language models such as GPT-4o, which can provide strong reasoning capabilities out of the box. Additionally, standard NLP libraries such as NLTK and SpaCy can be used for text pre-processing well. These tools are widely adopted in prior work, known to be fast and reliable.
The primary focus of the proposed Segment Anyword is to enhance the quality automatic mask prompts without relying on complex training configurations. The external language model is used solely to parse and index object-related words for cross-attention map retrieval. Notably, fine-tuned LLM (e.g.. Vicuna (Chiang al., 2023)) not required our method compatible with state-of- the-art language models such as GPT-4o, which offer strong reasoning capabilities out of the box. Additionally, standard NLP libraries such NLTK and SpaCy can be used for text pre- rre-processing. These tools, widely adopted in prior work, are known for their speed and reliability.
We conducted the study using 100 randomly selected image-text pairs from RefCOCO (Kazemzadeh et al., 2014). For the SpaCy', we used the en core web trf pipeline based on RoBERTa (Liu et al., 2019). We filtered tokens with pos-tag NOUN and A ADJ, and indexed the token with the nsubj dependency label as the referred object. For GPT-4o and Vicuna-7B, we used the following prompt:
Prompt:
As a NLP expert, you will be provided caption describing an image. Please do pos tag the caption and identify the only one referred subject object and all adjective altributes. Your response should be in the format of 1((ttribbute, attribute2, attribute3, ...), objectl]"
Conditions:
(1) If the attribute is long, short it by picking one original word.
(2) Please include one original word possessive source into the attributes for the subject.
mloU SpaCy(RoBERTa)
Parsing methods
GPT4o
68.2
46.9
Vicuna-7B
59.7
Table 7. Segmentation results of our Segment Anyword accompanied with different pos pos-tagging tools.
We present the final segmentation results in Table 7. While SpaCy offers fast, offline parsing, it often misses key adjectives such as color terms like "white." In contrast, GPT--oo provides more accurate parsing, reliably capturing fine-grained attributes. Based on our empirical observations, we offer the following recommendation. for large-scale processing where speed is critical, static NL.P libraries such as SpaCy are more suitable due to their efficiency. However, for detailed interactions involving concept learning and prompt refinement, advanced language models like GPT-4o are preferred for their superior reasoning and parsing capabilities.
22

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
F.2. Ablation study using Ilm generated text description
Image GT
"ditiog uboe"
Alulic Sower
Image
Frad ecrwigh
"sellow schoodles
STOP
STOP 22
STOP
STOP
mage shawoase oouch slooed right besdo clringg tible the vose (here is writee liswer, the souch to aiss posilioned front of wall. Imago
Test Thes Image Teatu rSS stopsign pese ancd in tront of yos schoolbus.
"dining olble*
flowe"
STOP
STOP 22
STOP
STOP
There
prominently from
sackground
Figure 13. Qualitative result comparison between Segment Anyword using ground truth description and GLaMM generated text description.
GranDf Val
Segment Anyword
AP50
mloU
Recall
w/ GT Text
30.2
65.9
42.4
w/ GLamm Text
27.1
62.5
37.7
Table Quantitative result comparison between Segment Anyword using ground truth text description and GLaMM generated text description.
We conducted an additional experiment using GLAMM-generated captions as textual input for our method, reporting both quantitative results (Table 8) and qualitative results (Fig. 13). In general, since GLAMM- text is not always accurate, can affect the alignment of prompt embeddings our method. Nevertheless, our approach remains competitive, as it capable of refining and adapting prompt embeddings- even from noisy or imprecise text-to to better match the target object during test-time optimization.
23

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
F.3. Ablation study with Pure Text Prompt
"Bear"
"Wall"
loods
baside
Image
GT
Ours
LangSAM
Ours
LangSAM
mecul
ocatod bca0s
Image
GT
Ours
LangSAM
wallle atseve
Some Jacon beffer
Image
GT
Ours
LangSAM
Parging The vo be
Image
GT
Ours
LangSAM
Figure Qualitative result comparison between Segment prompi(LangSAM).
Anyword
Segment Anything Model text description and
GranDf Val
mAP
mloU
Segment Anyword
31.3
67.4
LangSAM
17.6
33.5
Table 9. Quantitative result comparison between Segment Anyword and Segment Anything Model with text description and prompt(LangSAM).
We presnet additional experiments of how different prompt influence the downstream mask generation. We compare our Segment Anyword mask prompt against pure text input, using official implementation of LanguageSAMM2. We present both quantitive and qualitative results on GranDf validation set. Results show that our method is very effective on improving mask prompt quality.
24

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
F.4. Ablation study using Ilm generated text description
"Cat"
"Mirree"
An image ofa cat lering nex to minor Image
Cross-Attmtion
Ours
Crese-Altemion
Ours
"Squirre""
"Burger"
Image
(a)
Stable Diffusion-vi-5 DDIM Inversion
A.RS BOCS
pevemene
Imagee
Cross-Altemtion
Durs
CooseAttttiinn "Sky"
Ours
"Aupplane"
ainalane under
the sky.
Image
Crosi-Atiention
Quus
Cross-Atiztton
Ours
(b) Stable Diffusion-vi--4 Null- Text Inversion
Figure 15. Qualitative result comparison between Segment Anyword using different diffusion backbones (Stable Diffusion v1.4 and v1.5) and different inversion algorithms (DDIM inversion and null-text inversion).
MCPL primarily serves as the inversion backbone in method. However, our approach is not limited to MCPL and can be seamlessly integrated with other inversion or concept-discovery methods. We present qualitative results (Fig. 15) using different cross-attention sources, demonstrating that our method is composable with various diffusion models and inversion algorithms. These include replacing LDD with Stable Diffusion(Rombar et al., 2022) versions 1.4 and 1.5, and using alternative inversion techniques such as vanilla DDIM inversion (Song et al., 2021) and Null-Text Inversion (Mokady et al., 2023).
25

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
G. Additional Stress Testing Results
G.I. Grounded Segmentation with Novel Visual Concepts
Image
Ours Attention
Ours
GSVA
SAMAMLLM
Building paint on the mug.
An image of an elf toy.
An image of kittytoi.
image of brown bull,
Figure Stress Testing Results: show qualitative results from stress testing study, where test data involves novel visual concepts.
We present qualitative comparison of stress testing involving several novel concepts, such as building paint, kittytoi, and brown bull. For GSVA (Xia et al., 2024), we use the LLaMA-7B base weights with LLa VA LLLaV--Lighiing--/ -eahhminng---ddlla--ll.... and SAM VIT-H. For SAM4MLLM (Chen et al., 2024), we employ LLaMA--LLVVA-nexx... and EfficientVIT-SAM-XI... While both methods are capable of localization, they struggle to produce accurate masks, particularly in capturing fine important to note that both GSVA and SAM4MILLM require training or
details around object parts and boundaries. fine-tuning of LLaVA and SAM the training set, which demands substantial computational resources. In contrast, our method operates purely test time without accessing the full training set, making it both simple and effective for handling novel concepts.
26

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
G.2. Grounded Segmentation under Noisy Text Description
Cross Attention
Ours
Peekaboo Mask & Result
Image
Prompt & Result
An image of mug.
Builddnng point the mug.
image
elf toy.
An image of catstalue.
An image of kittytoi.
The red wine the left.
Figure Stress Testing Results: show qualitative results from stress testing study, where the test data involves noisy test descriptions.
We acknowledge that noisy inputs, including sentences with incorrect grammar, may be present from customized user input during test time. However, handling noisy text parsing is not the primary focus of our work. Instead, our objective is to generate mask prompts that are robust to such noise, without relying on supervised training. As shown in Fig. 17, our method can effectively handle typos such as "catstatue" and "kittytoi." In extreme cases, word-level cross-attention mask can be directly matched against ground- truth segmentation masks to determine the best ranking and pairing, thereby compensating for parsing inaccuracies.
27

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
G.3. Inference-Time Speed Comparison
Methods
Average Inference Time /image
Denoising Steps
Peekaboo (Burgert et al., 2022) CLIPasRNN (Sun al., 2024)
150s
300 steps
180s
Segment Anyword
470s
1100 steps
Segment Anyword,
28s
50 steps
Table 10. Inference time speed comparison between Segment Anyword and other training-free baseline methods including CLIPas-
RNN (Sun al., 2024) and Peekaboo (Burgert al., 2022).
We acknowledge that a trade-off exists between test-time optimization speed and mask prompt quality. Specifically:
Previous methods typically require substantial computational resources and manual effort to train or fine-tune large vision- lasion-lanuuage models on curated training datasets. While effective in controlled settings, these approaches often lack generalization capabilities and are resource-intensive..
In contrast, our proposed test-time prompt optimization method is more efficient and better suited for real-world open-set scenarios, where test samples may include novel linguistic and visual concepts. In such cases, textual embeddings must be dynamically updated and aligned with the target object during inference. Thus, the trade-off between inference speed and embedding alignment inherent and cannot be entirely eliminated.
Nevertheless, we demonstrate that the number of test-time steps can be substantially reduced to improve inference speed. Table 10 presents a comparison inference time with related training-free free baselines, including CLIPasRNN (Sun et al., 2024) and Peekaboo (Burgert al., 2022). All experiments were conducted on single NVIDIA A A100 40GB GPU. Our method shows significant acceleration-redueee small number of target-domain samples and decreasing the number of inference steps from 1100 just 50, with minimal performance degradation. Future work may further enhance speed through engineering optimizations, such as replacing the current backbone with Hyper Stable Diffusion (Ren et al., 2024).
inference time from 470s to 28s by fine-tuning the text encoder
28

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
H. Additional Qualitative Segmentation Results
Image
Ground
The image shows horsel walking on the grass, bes it horse2 is standing on the same grassy surface
Ground Truth
"computer
"dining table"
"paper"
The image displays a computer mouse placed next to cup on a diring tab e. oup is positioned next to sheet paper.
Image
Ground Truth
"couch"
diniing table"
"aase"
"iiite flower"
"wall"
The image showcase couch placed right beside dining tacle a vase is also beside the table. Inside the vase there is white flower, the couch positioned front wall.
Image
Ground Truth
"wrall"
"chair
"gray tiger cat"
The image depicts gray tiger cat sitting on chair next table which is on rug. the cat looking book, the chair is positioned
"penson"
"Gog"
A person walking with dog. The dog is sos positioned beside some gress.
Figure 18. Additional qualitative open-set language grounded segmentation results (GranDf validation set). We show that Segment Anyword can achieve accurate open-set language grounded segmentation with well-localized mask prompt, even can generate fine grained object part-aware masks such "chair", "vase" and "rug"
29

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Image
Ground Truth
"girafle"
"Iree"
The image shows giraffe an enclosed space. The giraffe reaching for branch of a tree. Image
Ground Truth
"yellow and white trinn"
"railroud"
"dirt"
"grass" "M
"buildingg
"sky"
A yellow and white train is driving on a railroad. The railroad situated beside dirt and grass. A building stands near this dirt area along side the sky.
Image
Ground Truth
"inn
"counter"
"bortle"
"wall"
"towel" t
"cabines"
"rgg"
The image show cases home bathroom with toilet sink and a shower area. The sink is embedded counter with bottle atop the counter. The the adjacent
mirror accessory
cabinot rug positioned the tiled floor,
Image
Ground Truth
"shoop!" |M
"shoep2"
"house"
"sky
Sheep1 and sheep2 are seen eating grass. A house located on the grass. The sky is visible above. Image
Ground Truth
"white boat"
"lake"
"greenery"
"sky"
large white boat in a marina on the serene lakes edge circled by beautiful greenery under clear blue sky.
Figure 19. Additional qualitative open-set language grounded segmentation results (GranDf validation set). We show that Segment Anyword can achieve accurate open-set language grounded segmentation with well-localized mask prompt, even with complex indoor placements "counter" and outdoor infrastructure "railroad".
30

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Image
Ground Truth
guy white
"black shirt"
Guy white on left side and right black shirt
Image
Ground Truth
person
"woman"
20
20
20
The person wearing black and a woman wearing glasses throwing frisbee
Image
Ground Truth
"bunana lop"
"bumana bottom"
banana on top banana on bottom
Image
Ground Truth
"pizza"
"slice"
Whole pizza and slice top left
Image
Ground Truth
whiite car"
"brown stationwagen"
White car far left and brown stationwagen
Image
Ground Truth
"player"
"guy white"
Left player and guy white
Figure 20. Additional qualitative reference image segmentation results (gRefCOCO validation set). We show that Segment Anyword achieve accurate reference image segmentation with well-localized mask prompt. With updated textual embedding, it further distinguish "slice" from "pizza"
31

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Image
Ground Truth
"vod
"hiite bus"
The red bus and white bus Image
Ground Truth
"bag"
"suitcase"
The bag and the suitcase Image
Ground Truth
"carrots"
"food"
A bowl of carrots in the middle and bow of food on the right
Image
Ground Truth
"phone"
"green book"
phone middle of book on the right and green book on the left.
Image
Ground Truth "fridge"
"black thing"
Black thing on the right and a fridge
Image
Ground Truth
"lass cup"
"donut"
glass cup and donut
Figure 21. Additional qualitative reference image segmentation results (gRefCOCO validation set).
32

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Image
Ground Trath
"aeroplane"
"sky
image of aeroplane and sky
Image
Ground Truth
"dog""
image of dog and grass Image
Ground Truth
"Gaf"
"Boot
image of cal and floor
Image
Ground Truth
"borse"
"ground"
image ground sky
Image
Ground Truth
"person"
"book"
"Caal"
"bedclathes"
imaaee of person and book and beddothes
Imuge
Ground Truth
"inoturbikee"
"petson"
"building"
"road"
An image of motorbike and person and building and road
Figure Additional qualitative open- vocabularyy semantic segmentation results (Pascal Context validation set). By concate- nating word label into sentences, we show that Segment Anyword can achieve accurate vocabbacyy image segmentation with well-localized mask prompt.
33

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
Image
Ground Truth
"caviry"
Ground Truth
"myocardium"
an image of round cavity encircled by circle myocardium
Image
Ground Truth
"transition"
Image
Ground Truth
"transition"
image of round transition
Figure 23. Exemplar qualitative OOD Medical Image segmentation results. We show that Segment Anyword has potential prompt learning and segmentation capability for out-of-distribution medical image segmentation, such as "cavity", "myocardium' " prostate "transation" zone.
34

---

Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
I. Additional Results Failure Cases
Image
"arrboord"
"cap"
X
Gronn
ChoneAlttation
"penso""
Failrre Case
over
Image
"skill"
Ground Touls
Cross-Altetiin
"seran"
Failuro Casa
young dhild standing sym gymbbllding
"skatcheand"
Ground Truth
Cross Attenionn
"srass"
Falture Case
riding person postioned
sky gress image
Image
"person"
"grass"
"pavereet"
standing slsteeoard which paverentt he The evathead covering
there
Ground Touth
Cuno-Alientin
"Jultdoond""
Failure Case
Figure Additional qualitative failure case results. acknowledge that Segment Anyword can still encounter failure segmentation, particularly when the target object is small and visual ambigious, such as "skis" and "skateboard". This is mainly due to the cross-attention map resolution is small (16x 16), which may lead wrong localization of mask prompt back the orginal image, although the diffusion backbone can capture the visual concept.
35


Reviews:
[{"review_id": "zKAvqk4brG", "reviewer": "Reviewer_jmV5", "summary": "This paper introduces Segment Anyword, a novel framework for open-set grounded segmentation. The key idea is to invert the mask prompting process by leveraging a pre-trained diffusion-based text-to-image model (e.g., Stable Diffusion) to generate high-quality, grounded segmentation masks. The method achieves competitive performance on both in-distribution and out-of-distribution datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable questions and insights. We are pleased that they find our proposed method novel, resource-efficient, and highly generalizable.\n\nWe have made several amendments and addressed the reviewer's specific queries as detailed below:\n\n__Q1__: \"__Inference time cost / Slow speed.__\"\"\n\nWe appreciate the reviewer highlighting the concern regarding test-time computational costs. We acknowledge there is indeed a trade-off in test-time optimization speed and mask prompt quality. Specifically:\n\n1. Previous methods typically require extensive resources and effort to train or fine-tune large vision-language models on training datasets. This approach is resource-intensive with limited generalization capabilities.\n2. In contrast, our proposed test-time prompt optimization method is more efficient and better suited to real-world open-set applications, where test samples may contain novel linguistic and visual concepts. In such scenarios, textual embeddings need to be dynamically updated and aligned with the target object during testing. Consequently, this necessary trade-off between inference speed and embedding alignment cannot be entirely avoided.\n3. However, we have demonstrated that it is possible to substantially reduce the number of test-time steps from 1100 to 50 by adapting the text encoder to the target dataset domain fine-tuned by only a samll number of samples. Furthermore, future work could focus on additional engineering optimizations, such as replacing the current backbone with HyperDiffusion, to further enhance inference speed.", "claims_and_evidence": "The paper claims that Segment Anyword works effectively on both in-distribution and out-of-distribution datasets. This is supported by quantitative results on multiple benchmarks (GranDf, gRefCOCO, PC59, synthetic/medical datasets), showing strong performance, particularly in handling novel categories without retraining.\n\nThe paper introduces positive adjective binding and negative mutual-exclusive binding as key components, which are well-supported by clear and convincing evidence in Table 6. The ablation results directly isolate and quantify their impact on segmentation performance, confirming their utility.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, there is no proof for theoretical claim.", "experimental_designs_or_analyses": "The experimental designs are sound", "relation_to_broader_scientific_literature": "Yes", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper introduces a novel approach by inverting the typical mask prompting paradigm. Instead of designing prompts manually, it optimizes them at test-time, leveraging diffusion models' cross-attention layers for precise mask generation.\n\nA significant advantage is that the method requires no additional training of the diffusion model or SAM, making it resource-efficient and appealing for real-world deployment.\n\nThe approach demonstrates robust results on both in-distribution and out-of-distribution datasets, confirming its effectiveness in open-set scenarios.\n\nWeakness:\n\nWhile the method smartly avoids retraining large models, it heavily depends on test-time optimization, involving multiple gradient steps to optimize textual prompts and collect cross-attention maps from a diffusion model. This inference-time overhead can be significant, especially considering the use of diffusion models that already involve multiple denoising steps.", "other_comments_or_suggestions": "See weakness", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sYYTGRF34V", "reviewer": "Reviewer_ixNi", "summary": "This work proposes Segment Anyword, an approach for language-guided open-set segmentation. It uses a diffusion model to create initial correspondence between words in the text prompt and points in the image, refines the point prompts based on linguistic analysis, and prompts SAM to generate the final segmentation masks. Empirical results on tasks like referring image segmentation show promising results.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful questions and detailed feedback, which have significantly helped improve the quality of our manuscript. Below, we address the reviewer’s concerns point by point.\n \n__Q1__: \"__Definition on \"Word-Grounding\" and \"Novel-Concept\"__\"\n\n__R1__: We refer \"word-grounding\" to the model’s ability to associate or align words from a sentence prompt with specific object regions in an image.\nWe refer \"novel-concept\" to that the model can recognize or handle new concepts it has not encountered during training.\n\nWe will make amendements on Table caption.\n\n__Q2__: \"__Additional motivation study.__\"\n\n__R2__: We thank the reviewer for raising this question. We present additional motivational study with another state-of-the-art model ETRIS (ICCV23) ([link1](https://anonymous.4open.science/api/repo/ICML-Submission3702-Rebuttal/file/Imgs/ETRIS-RefCOCO+_TestB-Scatter.pdf)), which focus on aligning representation from pre-trained visual and language encoder by adding intermediate fine-tuned adapters. The result also validates our previous findings that without test-time alignment, current open-set segmentation models suffer from input variance, resulting in unstable segmentation results.\n\nWe will extend this discussion to motivation section in our manuscript.\n\n__Q3__: \"__why baseline methods failed with certain conditions.__\"\n \n__R3__: Both GSVA and SAM4MLLM generate multiple object masks within a single binary segmentation map, but these masks are not explicitly associated with specific word indices. Additionally, neither model is capable of handling novel concepts, as both require training or fine-tuning on the target dataset. This limits their ability to generalize to unseen concepts during test time.\n\nWe will add this extended discussion to our manuscript.\n\n__Q4__: \"__Text parsing relies on external model__\"\n\n__R4__: The primary focus of the proposed Segment Anyword is to improve the quality of automatic mask prompts without relying on complex training configurations (Page 2 Contribution 2). The external language model is merely used to parse and index object-related words for cross-attention map retrieval. Importantly, the use of a fine-tuned LLM (e.g., Vicuna) is not required—our method is compatible with state-of-the-art language models such as GPT-4o, which can provide strong reasoning capabilities out of the box.\n\nAdditionally, standard NLP libraries such as NLTK and SpaCy can be used for text pre-processing as well. These tools are widely adopted in prior work, known to be fast and reliable.\n\nWe will add this discussion to our manuscript.\n\n__Q5__: \"__Text could be noisy__\"\n\n__R5__: We acknowledge that noisy inputs, including sentences with incorrect grammar, may occur in the test dataset. However, addressing noisy text parsing is not the focus of our work. Instead, our goal is to generate mask prompts that are robust to such noise, without relying on supervised training. We show an example that our method can still handle typos such as \"_catstatue_\" and \"_kittytoi_\" in [link2](https://anonymous.4open.science/api/repo/ICML-Submission3702-Rebuttal/file/Imgs/Ours%20vs%20Peekaboo.pdf)\n\nIn extreme cases, each word’s cross-attention mask can be matched directly against the ground-truth segmentation masks to determine the best ranking and pairing, thereby compensating for parsing inaccuracies.\n\nWe will add these additional results.\n\n__Q6__: \"__Fairness on GranDf comparison__\"\n\n__R6__: We thank the reviewer for raising this issue. To address it, we conducted an additional experiment using GLamm-generated captions as the textual input for our method, we report both quantitative and qualitative results [link3](https://anonymous.4open.science/api/repo/ICML-3702-Rebuttal2/file/Ours%20GT%20Text%20vs%20GLamm%20Text.pdf)\n\n|                   |      | GranDf Val |        |\n|-------------------|------|------------|--------|\n|                   | AP50 | mIoU       | Recall |\n| Segment Anyword_f |      |            |        |\n| w/ GT Text        | 30.2 |       65.9 |   42.4 |\n| w/ GLamm Text     | 27.1 |       62.5 |   37.7 |\n\nIn general, since GLamm-generated text is not always accurate, it can affect the alignment of prompt embeddings in our method. Nevertheless, our approach remains competitive, as it is capable of refining and adapting the prompt embeddings—even from noisy or imprecise text—to better match the target object during test-time optimization.\n\nWe will update the experiment and result.\n\n__Q7__: \"__In the test-time cross-attention collection step, which layer(s) of the denoising UNet are considered when collecting the cross-attention values?__\"\n\n__R7__: We use the cross-attention maps at the 16×16 resolution, averaged across all denoising time steps to obtain the final cross-attention. This setup is inherited from prior works such as MCPL and Prompt2Prompt to ensure a fair and consistent implementation for textual embedding update. We will add this to our implementation section.", "claims_and_evidence": "- Table 1 introduces some attributes. To the reviewer, \"Word-Grounding\" and \"Novel-Concept\" are not clearly defined. It is also unclear why the most recent models like GSVA and SAM4MLLM do not satisfy these conditions.\n\n- The motivation study is limited to ReLA, a previous model not equipped with large language models. The conclusion of this study may not hold true for more recent MLLM-based models.", "methods_and_evaluation_criteria": "- The method heavily relies on an external model (a fine-tuned Vicuna, as indicated in Appendix B.1) and a tagging method to parse the text prompt, whose parsing accuracy is not validated. In reference image segmentation, some text expressions are written by human annotators with incorrect grammar structures. How to correctly understand these expressions when the parsing is noisy?", "theoretical_claims": "This work does not include theoretical claims.", "experimental_designs_or_analyses": "- The experiment on \"open-set grounded segmentation\" is very misleading. In the original work, GLaMM, GranDf is proposed for the \"grounded conversation generation\" task, where a model is required to generate a detailed description of the image and ground the noun phrases within the description. GLaMM does not mention the task as \"open-set grounded segmentation.\" Furthermore, as detailed in Appendix B.2, this work \"only focus on segmentation capability evaluation by using ground truth text expression as segmentation reference.\" The comparison in Table 2 is not fair and the performance of Segment Anyword cannot be considered as high, given that the text descriptions are from the ground truth.", "relation_to_broader_scientific_literature": "This work introduces a new framework for open-set segmentation with language prompts without leveraging extensive supervision. However, the experiment designs have several flaws and even misleading results, which should be fixed before publication. The proposed method is also a bit too complicated and relies on several external models (diffusion model, parsing LLM and SAM), limiting its applicability in a wider range of tasks.", "essential_references_not_discussed": "No concerns.", "other_strengths_and_weaknesses": "No other concerns.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "- In the test-time cross-attention collection step, which layer(s) of the denoising UNet are considered when collecting the cross-attention values?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2HoxIVMZLs", "reviewer": "Reviewer_4TbZ", "summary": "This paper introduces Segment Anyword, a training-free framework for open-set language-grounded image segmentation. It leverages token-level cross-attention maps from a frozen diffusion model to generate mask prompts, which are then refined by the Segment Anything Model (SAM) for accurate segmentation. To address the lack of coherence and consistency in initial prompts, the authors propose Linguistic-Guided Visual Prompt Regularization, which incorporates syntactic and dependency structures to improve prompt quality. The method shows significant performance gains and strong generalization across diverse datasets without the need for extensive training or fine-tuning.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their detailed feedback. We are glad that the reviewer found our paper to present clear claims with evidence, supported critiques. And our method is computationally lightweight.\n\n__Q1__: \"__training-free warrants scrutiny__\"\n\n__R1__: The term “training-free” refers to the fact that our method does not require access to or processing of the training dataset. In contrast to prior approaches that demand significant computational resources to train or fine-tune on curated datasets, our method avoids such data-dependent training procedures.\n\nInstead, we perform at test-time, which is more suitable for real-world, open-set scenarios—where test samples often involve novel concepts that has not been seen during training. In such cases, updating prompt embeddings during inference is necessary to align them with the target objects effectively.\n\nTo avoid confusion, we will revise our terminology throughout the manuscript, replacing “training-free” with “test-time optimization.”\n\n__Q2__: \"__fine-tune the BERT encoder increase model complexity.__\"\n\n__R2__: We demonstrate that it is possible to reduce the number of diffusion steps during test time textual embedding update from 1100 to as few as 50—by adapting the text encoder using a small number of samples, without accessing the full training dataset. This lightweight adaptation, achieved via LoRA, is intended purely for accelerating test-time optimization, not for conventional training or fine-tuning on a full dataset.\n\nImportantly, this step does not involve large-scale training or supervision and serves as a practical means to align the text encoder with the target domain efficiently. We emphasize that this adaptation remains consistent with our goal of avoiding full model training and supports fast, sample-efficient test-time optimization.\n\nWe will add this discussion to our manuscript.\n\n__Q3__: \"__avoid overstating claim.__\"\n\n__R3__: We are sorry for the confusion terms and we will replace with \"test-time optimization\" for clarification.\n\n__Q4__: \"__Proposed method's impact on performance.__\"\n\n__R4__: We thank the reviwer for raising this issue. We presnet additional experiments of how different prompt influence the downstream mask generation. We compare our Segment Anyword's mask prompt against pure text input, using official implementation of LanguageSAM. We present both quantitive and qualitative results [link1](https://anonymous.4open.science/api/repo/ICML-Submission3702-Rebuttal/file/Imgs/Ours%20vs%20LangSAM.pdf) on GranDf validation set. Results show that our method is very effective on improving mask prompt quality.\n\n|                 | GranDf Val |      |\n|-----------------|------------|------|\n|                 | mAP        | mIoU |\n| Segment Anyword |       31.3 | 67.4 |\n| LangSAM         |       17.6 | 33.5 |\n\nWe will add this additional experiment to our manuscript.\n\n__Q5__: \"__Discuss with CoOp and CoCoOp__\"\n\n__R5__: We thank the reviewer for bringing up CoOp and CoCoOp, two important works in visual prompt learning. We agree with reviewer's comments and will include the discussion in our manuscript.\n\n__Q6__: \"__difference from MCPL__\"\n\n__R6__: MCPL serves primarily as the inversion backbone in our method. However, our approach is not limited to MCPL and can be seamlessly integrated with other inversion or concept-discovery methods as well. We present a qualitative results with different cross-attention source, showing that our method is composible with various diffusion models and inversion algorithms [link2](https://anonymous.4open.science/api/repo/ICML-Submission3702-Rebuttal/file/Imgs/Ours%20with%20different%20cross%20attention%20source.pdf). We will add this discussion to our manuscript.\n\n__Q7__: \"__green, blue, and red represent?__\"\n\n__R7__: These colors illustrates easy/medium/hard samples for achieve accurate and stable segmentation, categorized by its IoU mean and std.\n\n__Q8__: \"__PL, R1, and R2 represent?__\"\n\n__R8__: PL stands for prompt learning. R1 and R2 are two linguistic guided prompt regularizations.\n\nWe will revise the figure and table captions to enhance clarity and avoid ambiguity.\n\n__Q9__: __Typos__\n\n__R9__: We will revise our typos throughout the paper.\n\n__Q10__: \"__novelty of this paper__\"\n\n__R10__: As demonstrated through comprehensive experiments and additional results provided in [link3](https://anonymous.4open.science/api/repo/ICML-Submission3702-Rebuttal/file/Imgs/gRefCOCO_ext_Scatter_comparison.pdf) and [link4](https://anonymous.4open.science/api/repo/ICML-Submission3702-Rebuttal/file/Imgs/RefCOCO_TestB_Scatter_comparison.pdf), our method effectively improves both accuracy and stability by leveraging an off-the-shelf diffusion model to extract mask prompts via an inversion process. This approach is modular and plugable, offering several advantages—including the ability to handle novel visual and linguistic concepts—without requiring additional supervision or complex training procedures.", "claims_and_evidence": "Overall, the paper presents its claims with clear and convincing evidence.\n\nThe authors support their critique of existing open-set segmentation methods—namely, their reliance on extensive training and difficulty in achieving consistent object segmentation across diverse textual expressions—through both quantitative and qualitative analysis, particularly in Figure 3.\n\nTo address the issue of incoherent and inconsistent prompt quality, the proposed linguistic-guided visual prompt regularization enhances alignment between text expressions and visual masks. The effectiveness of this module is demonstrated both qualitatively (Figure 6) and quantitatively (Table 6).\n\nThe claim that the method is computationally lightweight compared to existing approaches is substantiated in Table 1, where the number of trainable parameters is significantly lower than that of most prior methods.\n\nHowever, the repeated emphasis on being \"training-free\" warrants scrutiny. In practice, the method updates textual embeddings for up to 1100 steps during inference, which arguably constitutes test-time adaptation or lightweight training. Additionally, the use of LoRA to fine-tune the BERT encoder further complicates the notion of being entirely training-free. While not the central contribution, this aspect should be more carefully framed to avoid overstating the claim.\n\nFinally, although the method effectively provides initial localization through cross-attention maps, much of the segmentation accuracy relies heavily on the use of SAM as a post-processing module. This dependency raises questions about how much of the final segmentation performance can be attributed solely to the proposed method itself.", "methods_and_evaluation_criteria": "The proposed method is well-aligned with the problem of open-set language-grounded segmentation and effectively addresses its key challenges. The evaluations are conducted on appropriate benchmark datasets using standard metrics, and the results, along with ablation studies, support the method’s validity.", "theoretical_claims": "The paper does not present formal theoretical claims; it is primarily empirical, focusing on experimental results and practical effectiveness.", "experimental_designs_or_analyses": "Most of the experiments are based on empirical results and are considered sound.", "relation_to_broader_scientific_literature": "The proposed method focuses on generating masks more efficiently compared to existing methods. Being able to effectively perform segmentation for various text expressions in an open-world scenario can be highly impactful.", "essential_references_not_discussed": "Prompt learning papers, beginning with \"CoOp: Conditional Prompt Learning for Vision-Language Models\" as the seminal work, should be addressed, followed by its successors. CoOp’s experimental design is empirically sound, introducing prompt learning to adapt vision-language models like CLIP for few-shot tasks, efficiently generating masks and outperforming baselines like zero-shot CLIP across datasets such as ImageNet; however, it struggles with unseen classes. Follow-up works like \"CoCoOp\" (CVPR 2022) enhance this by adding input-conditional adaptability for open-world segmentation of diverse text expressions, maintaining soundness with robust testing but introducing computational complexity.", "other_strengths_and_weaknesses": "The core framework claimed in this paper heavily borrows from the assertions in \"An Image is Worth Multiple Words: Discovering Object-Level Concepts using Multi-Concept Prompt Learning,\" but it lacks sufficient evidence to explain what differentiates the proposed method. It directly adopts the approach of averaging cross-attention maps and merely refines the masks using SAM afterward. Given that this is a training-free method not utilized for learning, I believe a more significant approach is needed.\n\nRelatedly, I think the ablation study for the proposed method is insufficient. There’s little detail on how SAM is applied or what design choices are made in the subsequent regularization step, where deeper design exploration and ablation studies seem necessary.\n\nOverall, the explanations for figures and tables are lacking. In Figure 3, the intent is somewhat clear, but it’s unclear what the green, blue, and red colors represent. Similarly, in Table 6, terms like PL, R1, and R2 are presented without any explanation of their meaning.", "other_comments_or_suggestions": "entirely : off-shelf → off-the-shelf?\nL296 : eight → seven?", "questions_for_authors": "As mentioned earlier, I have doubts about the novelty of this paper. Most of the ideas are derived from existing methods, and reinforcing the noun part of text descriptions with adjectives or adverbs isn’t particularly a new idea either. This concern needs to be addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jQ0Jz3jylL", "reviewer": "Reviewer_SSnW", "summary": "The authors identify an issue in VLMs / MLLMs of unstable segmentation against variations in textual prompt. They propose a text-to-image diffusion model based test-time optimization technique combined with language guided prompt tuning to solve this issue. Their resulting framework, tagged Segment Anyword, is evaluated across diverse tasks and datasets to show strong performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Q1__: \"__Exact model to generate segmentations__\"\n\n__R1__: We use the official implementation and pretrained checkpoint from ReLA. The final segmentation mask is produced by the pixel decoder whose outputs are weighted by a language-supervised object activation map.\n\n__Q2__: \"In plot, is IoU calculated against the ground-truth?\"\n\n__R2__: Yes.\n\n__Q3__: \"__Each dot means?__\"\n\n__R3__: Yes, each dot corresponds to a single image paired with multiple descriptions. The mean and standard deviation are calculated along the caption dimension.\n\n__Q4__: \"__Use different colour__\"\n\n__R4__: We will revise the color coding in the figures.\n\n__Q5__: \"__IoU inside brackets?__\"\n\n__R5__: Yes it is IoU against GT.\n\n__Q6__: \"__how to generate mutated expressions?__\"\n\n__R6__: We generate the mutated expressions by querying ChatGPT4o, with a template \"_Genrate a list of __n__ synonyms of the noun phrases in the following [sentence] and output the list separated by '&'_\", where __n__ is in randint(2, 5) and _[sentence]_ is set to text referring expression.\n\n__Q7__:\"__Figure 3 plot add your model?__\"\n\n__R7__: Please find link3 and link4 in our reply R10 to reviewer 4Tbz.\n\n__Q8__: \"__Table with average std__\"\n\n__R8__: We provide the table with average std below:\n\n| STD Avergage Comparison |          |\n|-------------------------|----------|\n|                         | RefCOCO+ |\n| ETRIS                   |   10.958 |\n| Segment Anyword         |    8.021 |\n|                         |          |\n|                         | gRefCOCO |\n| RELA                    |   13.217 |\n| Segment Anyword         |    4.365 |\n\n__Q9__: \"__the average std should drop for your method?__\"\n\n__R9__: That's correct. We focus on test-time prompt embedding alignment. We show that out method is simple but very effective to reduce the sensitivity of input variance.\n\n__Q10__: \"__Slow inference speed__\"\n\n__R10__: Thank you for your feedback. Please refer our reply to reviewer jmV5.\n\n__Q11__: \"__inference time comparison__\"\n\n__R11__: We present a Table comparing the inference time with related baselines, including CLIPasRNN, Peekaboo.\n\n|                   | Average Speed Per Image | Inference Steps Per Image |\n|-------------------|-------------------------|---------------------------|\n| Segment Anyword   | 470s                    | 1100steps                 |\n| Segment Anyword_f | 28s                     | 50steps                   |\n| Peekaboo          | 150s                    | 300steps                  |\n| CLIPasRNN         | 180s                    | -                         |\n\nAll implementations were evaluated on a single NVIDIA A100 40GB GPU. As previously discussed, test-time optimization methods inherently involve a trade-off between inference speed and mask quality. Nevertheless, we demonstrate that our method can be significantly accelerated—reducing inference time from 470s to 28s—by fine-tuning the text encoder on a small number of target-domain samples and decreasing the number of inference steps from 1100 to just 50, with minimal performance degradation.\n\n__Q12__: \"__Training-free overclaim?__\"\n\n__R12__: Please refer our reply R1, R2 and R3 to reviwer 4Tbz.\n\n__Q13__: \"__Concerns on expression parsing__\"\n\n__R13__: Please refer our reply R4 and R5 to reviewer ixNi\n\n__Q14__: \"__baseline + additional info__\"\n\n__R14__: We observe that certain baseline methods, such as GLamm, LISA, and OMG-LLaVA, already incorporate similar linguistic information into their input pipelines. For instance, by integrating complete noun-phrase during feature fusion, or translating a special [SEG] token with context into segmentation masks. These models achieve impressive results but require significant training efforts.\n\nOther baseline methods focused on test-time optimization, such as CLIPasRNN, OVDiff, and Peekaboo, could theoretically leverage this linguistic information as auxiliary input; however, empirical observations suggest that their performance remains suboptimal under these conditions.\n\nIn contrast, our proposed method explicitly formalizes this linguistic knowledge as prompt regularization. This strategy enables robust mining of noise-tolerant mask prompts and consequently yields refined and higher-quality segmentation masks, even without extensive training or configuration overhead.\n\n__Q15__: \"__Discussing related works.__\"\n\n__R15__: We thank the reviewer for highlighting these important works, which also aim to leverage intermediate features from large vision-language models (VLMs) for downstream segmentation tasks.\n\nBoth CLIPpy and PACL assign labels by contrasting image patch embeddings with object text label embeddings. However, these methods still require training configurations.\n\nPeekaboo is closely related to ours, as both aim to learn visual concepts using off-the-shelf diffusion models. However, Peekaboo heavily relies on alpha map initialization and is sensitive to its quality.\n\nWe will add this discussion to our manuscript.", "claims_and_evidence": "Inadequate evidence on how method solves the motivation problem of how VLMs *struggle with diverse terms* in textual prompt. Provide some simple quantitative evaluation (maybe your own benchmark using the dataset used in Figure 3) to back this claim. This will strengthen the paper. See more info in weaknesses.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, good.", "relation_to_broader_scientific_literature": "Relevant and useful topic explored.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n\n1. Interesting motivation analysis\n2. Tackles a difficult task of reducing VLM sensitivity to input prompts\n3. Through evaluation to show consistent performance improvements across benchmarks.\n\n\n\n**Weaknesses**\n\n\n1. Sec 2.2 / Figure 3 unclear: please provide more details regarding the visualization in the Figure caption. \n    * What is the exact model used to generate segmentations here? \n    * In plot, is IoU calculated against the ground-truth?\n    * What exactly does each dot correspond to? A single image with multiple captions (mean / std along caption dimension)? \n    * Can you use a different colour for the second red? You use red for two examples\n    * What's inside brackets? I'm assuming it's IoU against GT? \n    * \"Thus we extend each image associating with additional generated 2-5 mutated expressions (e.g. [ apple pieces]→ [ apple pieces, apple slices, cut up apples]).\" - how are these generated? Templates? LLM?\n\n2. Accomplishment of motivation.\n    * How does Figure 3 plot look if you add your model there? \n    * Can you provide table with average std (i.e. calculate std for each image as done currently, and then average across all images) of baseline vs your method? \n    * If the problem identified in contributions (1) - *struggle with diverse terms* - is solved by method, the average std (in an experiment like Figure 3) should drop for your method compared to baseline. Is this correct? Discuss this and back this motivation with quantitative results. \n\n3. Inference Speed (time / compute)\n    * This method appears much slower compared to other approaches that do not use test-time optimization\n    * Please provide a table comparing the inference time with that of baselines \n    * Maybe you can compare to slower, similar test-time optimization method like [1] to justify a slow speed\n    * \"Training-free\" claim in abstract / intro - is this fair, since you are training parameters during inference? In fact, you have at table showing the parameters you *train*.  \n\n4. Method details missing\n    * \"we parse the text expression into Noun-Phrases (NP) and Verb-Phrases (VP) and identify each rooted noun subject (root) within the phrase.\" - how is this done? What algorithms / methods are used? How accurate is this? \n    * If baselines are given this additional information (noun - verb phrases), will they perform as well? \n\n\n[1] Burgert, R., Ranasinghe, K., Li, X., & Ryoo, M.S. (2023). Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors. CVPR 2023.", "other_comments_or_suggestions": "Consider discussing related work: \n\n* Ranasinghe, K., McKinzie, B., Ravi, S., Yang, Y., Toshev, A., & Shlens, J. (2023). Perceptual Grouping in Contrastive Vision-Language Models. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 5548-5561.\n\n* Mukhoti, J., Lin, T., Poursaeed, O., Wang, R., Shah, A., Torr, P.H., & Lim, S.N. (2022). Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 19413-19423.\n\n* Burgert, R., Ranasinghe, K., Li, X., & Ryoo, M.S. (2023). Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors. CVPR 2023.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a training-free approach for open-set language-grounded image segmentation that leverages cross-attention maps from diffusion models and introduces language-guided visual prompt regularization to improve mask quality. Despite initial mixed scores (2, 2, 4, 4), the reviewers converged towards acceptance following the author response and discussion period. Concerns raised by the initially less positive reviewers were largely addressed, with one indicating an inclination to raise their score and the other expressing comfort with acceptance conditional on minor revisions, while the positive reviewers maintained their support after their concerns were resolved. 

This paper presents an interesting direction and strong results on challenging benchmarks, making it a valuable contribution to ICML; the authors are requested to incorporate the remaining feedback from all reviewers into the final camera-ready version, paying particular attention to clarifying claims as suggested (especially for ixNi).
Decision: Accept (poster)