RESAnything: Attribute Prompting for Arbitrary Referring Segmentation
Authors: Ruiqi Wang
Hao Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Referring Expression Segmentation
Prompting
Primary area: applications

Abstract: We present an open-vocabulary and zero-shot method for arbitrary referring expression segmentation (RES), targeting input expressions that are more general than what prior works were designed to handle. Specifically, our inputs encompass both object- and part-level labels as well as implicit references pointing to properties or qualities of object/part function, design, style, material, etc. Our model, coined RESAnything, leverages Chain-of-Thoughts (CoT) reasoning, where the key idea is attribute prompting. We generate detailed descriptions of object/part attributes including shape, color, and location for potential segment proposals through systematic prompting of a large language model (LLM), where the proposals are produced by a foundational image segmentation model. Our approach encourages deep reasoning about object or part attributes related to function, style, design, etc., enabling the system to handle implicit queries without any part annotations for training or fine-tuning. As the first zero-shot and LLM-based RES method, RESAnything achieves clearly superior performance among zero-shot methods on traditional RES benchmarks and significantly outperforms existing methods on challenging scenarios involving implicit queries and complex part-level relations. Finally, we contribute a new benchmark dataset to offer ~3K carefully curated RES instances to assess part-level, arbitrary RES solutions.

RESAnything: Attribute Prompting for Arbitrary Referring Segmentation
Ruiqi Wang Hao Zhang
Simon Fraser University
{ruiqi_w, haoz)0sfu.ca
robcol PRIKIct
hide
Poste de
altzale
CUl Poblano
object- part -leve, semantic labels
material/style
functionalty/ddeiinn
logo/packaging label
Figure Open-vocabulary and zero-shot referring expression segmentation with RESAnything. Our method produces accurate object or part masks from general- and free-form text expressions including, from left to right: object part semantic label, material/style properties. function/design descriptions. or logos and packaging labels textual or other graphical in image. For visualization purposes, overlay segmentation regions with color each example.
Abstract
We present an open- vecebuloayyl and zero-shot method for arbitrary referring ex- pression segmentation (RES), targeting more general input expressions than those handled by prior works. Specifically, level labels as well as implicit references pointing to properties or qualities object/part function, design, style, material, etc. Our model, coined RESAnything, leverages Phain-of-Thoouhhe prompting. We generate detailed descriptions of object/part attributes including shape, color, and location for potential segment proposals through systematic prompting of large language model (LLM), where the proposals are produced by foundational image segmentation model. Our approach encourages deep rea- soning about object/part attributes related to function, style, design, etc., to handle implicit queries without any part annotations for training or fine-tuning. As the first zero-shot and LLM-based RES method, RESAnything achieves superior perfor- mance among zero-shot methods on traditional RES benchmarks and significantly outperforms existing methods on challenging scenarios involving implicit queries and complex lert-level relations. We contribute new benchmark dataset of ~3K carefully curated RES instances to assess part-level, arbitrary RES solutions.
inputs encompass both object- and part-
(CoT) reasoning, where the key idea is antribute
Introduction
With rapid developments in Large Multimodal Models (LMMs), visual perception systems have evolved significantly, demonstrating remarkable capabilities in bridging vision and language tasks [16
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent advancements in LMMs have enabled sophisticated understanding of visual content, from object detection to semantic segmentation
20 28 36
One of the emerging segmentation tasks that has drawn a great deal of attention lately is the so-called Referring Expression Segmentation (RES) which aims at obtaining a segmentation mask an image or video that represents an object
-5
instance referred to by a natural language expression 19 60 73
5%
7
23
53 12).
Despite much progress made on RES, two common limitations are often observed. First, while existing approaches excel at identifying and segmenting objects as whole entities, they often fall short when the input expressions refer to specific object parts. Such situations arise frequently in applications such as eCommerce, where sellers and buyers often promote or review product features referring to specific parts, and in robotics, agents must interact with object parts. Second, most works to date on RES have focused on referring expressions that contain semantic labels in one way or another. Even the so-called generalized RES (GRES) 331 only extends the expression coverage to an arbitrary number of (including zero) target objects, with labels. On the other hand, object/part references are often implicit, without semantic labels. Such expressions can refer to properties or qualities related to object/part function, design, style, material, or they may appear in textual or other graphical forms as a logo or packaging label; see Fig. for some samples expressions and segmentations.
computer interaction, and automated systems, where
In this paper, we present an open-vocabulary and zero-shot RES method address both limitations. For lack of a better term, we call our task arbitrary referring segmentation and our model as RESAnything. Our goal is to allow input expressions to be more general than what prior works have been designed to handle, while solving our problem without any training or fine-tuning on specialized datasets. To this end, we leverage the generalization and zero-shot capabilities of modern- day foundational models such as Pixtral [ and Claude []] as Large Language Models (LLMs and
SAM !E! for image segmentation. However, solving the arbitrary RES task demands a deeper understanding of object and part properties, moving beyond traditional object-level and label-centric referencing to more nuanced reasoning for part- and attribute-level perception.
There have been recent works [25 46 26 on reasoning based segmentation through active LLM querying. An implicit query text, such as "the object containing the most Vitamin C." is first analyzed by text LLM and then referenced to the "'orange" object in the provided image. Nonetheless, such methods often fall short when the implicit connections between object/part properties (e.g.. functional or stylistic ones) and their visual manifestations are cascadedly hidden. Even advanced LLMs, with their sophisticated reasoning capability, struggle to ground their understanding without explicit supervision at the part or attribute level. Additionally, existing methods, e.g., LISA 125), typically rely on fine-tuning on specially prepared or curated datasets- they are not zero-shot. Our model for arbitrary RES is training-free. It lever-
ages ofhuunhs-TTohuhhts (CoT) for comprehensive part-level understanding. Our key idea is attribute prompting, which generates detailed descriptions of object/part attributes including shape, color, and loca- tion for potential segment proposals through system- atic prompting of L.MSs (4)
SAM
Expression
Actribute Prompting
MILM
vers
relesence
M. where the proposals
MLLM CUP
are produced by a foundational image segmentation e4 Our approach encourages deep reasoning about object/part attributes related to function, style, design, etc., enabling the system handle implicit queries without any part annotations for training or fine-tuning By bridging abstract de- scriptions with concrete visual attributes through a two-stage evaluation framework (attribute prompting + grouping and selection of segment proposals), as illustrated in Fig. 2 RESAnything achieves robust performance on both traditional expressions and challenging implicit queries for arbitrary RES.
response
model such as SAM
Figure 2: Overview of RESAnything: a two-stage framework for zero-shot arbitrary RES. The at- tribute prompting stage generates reference and candidate texts from input image and referring ex- pression using SAM-generated proposals and an MLLM. The mask proposal selection stage lever- ages MILMM and CLIP to evaluate both candidates and proposals and produce the final response.
In summary, our contributions are as follows:
The first zero-shot and LLM--ased voca-vacyb RES method, targeting input expressions that are more general than those addressed by prior works.
2

---

The novel idea of attribute prompting. as a means for Chain-of- ofTohughtss (CoT) reasoning, achieve SOTA performance on both object- and part-level RES tasks.
A new dataset, ABO-Image-ARE, built upon ABO [14,, offering carefully curated RES instances as a benchmark to assess part-level, arbitrary RES solutions.
Our dataset consists of 2,989 prression-segmeen pairs: 1,360 with object/part semantic labels, 742 depicting logos/packaging labels, 502 referring to functions/designs, and finally, 385 covering material/style properties.
We demonstrate by extensive experiments that RESAnything achieves superior performance among zero-shot methods on traditional RES benchmarks such RefCOCO, RefCOCO+ 78, Ref- COCOg 40 Our method also significantly outperforms existing methods on the recent reasoning segmentation dataset ReasonSeg [25] as well as RES tasks in challenging scenarios involving im- plicit queries and complex part-level relationships such as those from A-BO-Imaee-ARI
With its
zero-shot capabilities, the most important practical advantage of our method lies in the improved scalability and generalizability for real-world applications with diverse referring expressions. In contrast, current supervised methods, e.g., LISA 245 and GLaMM [6 require substantial training resources, with high data collection and annotation costs by humans. While performing well on vanilla RES benchmarks, they are not as scalable and are limited to scenarios in their training data.
2 Related Work
Recently, multimodal LLMs (MLLMs) has brought the success of LLMs to image understanding by integrating the visual and linguistic modalities. Example state-of-the-art proprietary models include Claude Sonnet (1. Gemini (2), GPT-4 series [3] etc. Most existing MLLM architectures connect a
pre-trained vision encoder the LLM decoder with modality connector. For example, Flamingo [
proposed the Perceiver Resample to bridge the modality gap, with follow-up works OpenFlamingo 6
and Otter 2 particularly developed for effective in-context instruction tuning. InstructBBIIP [T
built upon the Querying Transformer as in BLIP2 (29]. The LLaVA models [36 3 and Mini-
GPT4 [9 utilized lightweight MLP and achieved appealing performances in various MLLM
benchmarks. Recent developments include supporting high-resolution image inputs 770 33
optimizing model efficiency 87 and constructing higher-quality datasets [1]]
2.1
Open-Vocabular and RES
aims to segment target image regions based on textual descriptions. The core challenge lies in bridging the gap between image and language modalities. Typically. transformer- based text encoders [18 45 are employed to extract textual embeddings, which are then integrated
RES 23
into segmentation architectures through cross-attention or feature alignment 13
52 62
75
85 67
to achieve language-aware segmentation
30
73
37
68
32
Recently, SAM 24 has
introduced text-guided segmentation 84
39
12
For instance, Grounding-Se Souunding-SAM SAM
4 leverages
to prompt SAM for mask prediction, while 4 to select the final result from class-agnostic masks generated by SAM. However, the majority of these methods have been primarily designed for object-level segmentation based on explicit semantic expressions.
bounding boxes returned by Grounding-DINO
Fast-SAM [o utilizes CLIP similarity scores
To address a broader range of segmentation targets and linguistic inputs beyond semantics, methods based on MLLMs have emerged, leveraging the powerful language understanding capabilities inherited from LLMs 82
One of the pioneering works in this area is LISA 25 which enables MILMMS to segment objects by using text embeddings from LLaVA to prompt a SAM !Z! decoder to predict masks. LISA demonstrated promising performance on new task called Reasoning Segmentation, similar to our Arbitrary Referring Segmentation. While improvements over LISA have been developed for extending it to generalized RES [66] 65 and grounded segmentation restricts the variety of referring expressions but also weakens the reasoning capability of pre-trained MLLMs. In contrast, of the MLLM to reason about the input images.
26 77
80
58 10,
43
R2
6%
15
46 49
fine-tuning MLLMS on fixed segmentation datasets not only
method operates in training- ffree manner, preserving the complete ability
Some methods have demonstrated the feasibility of adopting pre-trained foundation models for RES 54
without additional training 79
22
MaskCLIP obtains pseudo masks by modifying the CaR couples CLIP and GradCAM to generate mask proposals, then
56
88
last attention layer of CLIP
SS
3

---

Prompt Qree
ii booted on the upon corton the che gne in ollx and prorotuu littor, sroed ig bach confat fax
rodar qu soctin the eppears to be par of depitted with bucking frrmed black bottee
bockrest* imput exprossion f
MLLM
reference text yrat
4 sole of the mark therr smmrest angied rovorts
Prompt Q
SAM
MILM
vertice an noneare mubls against a dank bedigreune
Input image!
segmentation proposals M (M8g..). My]
visual prompts
cancidate texts [77 10
Figure 3: Attribute prompting using SAM and MLLM. Given the input image and referring expression. this stage produces two groups of predictions. The first output, reference text 7 is generated from an MLLM with the text prompt Q It describes the visual attributes (e.g.. color, shape. location) of the target region ("mesh backrest" this example). The second group a set of candidate texts 7,"h, generated by an MLLM with the text prompt Q and visual prompts derived from segmentation mask proposals. These texts describe the attributes of their corresponding segmentation region proposals, visualized with the same border color.
employs CLIP classifier to select the final masks, before mask refinement ST in post-processing, Global-Local CLIP
pioneered zero-shot RES using CLIP to extract visual features. Our approach follows a similar design, leveraging SAM for proposal generation and MLLMs for mask selection. Although MILMs already exhibit superior reasoning abilities compared CLIP, our novel attribute promoting technique further amplifies their inferential capabilities for arbitrary RES.
7
2.2 Visual Prompting
Prompting 50 has emerged as a powerful technique for adapting pre- pre-trained language models to downstream applications. By incorporating additional hand-crafted instructions, prompt engineer- ing methods effectively facilitate the adaptation process. For instance, Chain-of-Thhugee (CoT) prompting encourages models explain their step-by-step reasoning while answering questions [6]
Recently, visual prompting 72
4ll
[3 (55) has been proposed enhance the adaptation of CLIP
for open-vocabulary segmentation by overlaying ovals over segmentation targets [54] SAM 24
on the other hand, allows users provide points, boxes, masks as prompts for image segmentation, with the latest version supporting video segmentation 47). Visual prompting has also been applied to MLLMs [44] Overlaying image regions with bounding boxes, masks, circles, scribbles, etc has
enhanced MLLMs' ability to perform region or pixel-level image understanding 774
3 Method
Problem statement. Given an image / and free-form expression E referring to potential target region It in 1, RESAnything first processes the image to generate and refine a set of segmentation proposals M = {m1,- ... ........ my). }. from which it selects the most appropriate binary segmentation mask m, representing R. The imput expression can be either an explicit referring expression (e.g., semantic label of object/part) or implicit expression (e.g., functional or material properties). For targets not directly visible, our method handles two scenarios: a) Irrelevant queries: indicate that the target does not exist in the image; b) Invisible targets: infer their location through their functional and spatial relationships, with explanatory reasoning.
A naive approach for applying MLLMs to solve our task would involve prompting the MLLMs to output a score for each segmentation proposal m,, indicating its similarity to the input expression E. However, current MLLMs struggle with directly connecting the text description to the image region. is possible to fine-tune a MLLM with many paired samples of texts and mask annotations, however, mentioned earlier, this incurs significant computational cost during fine-ture me-tunnng and human effort for data annotation.
Overview. Instead of fine-tuning, we propose novel approach to facilitate reasoning between text descriptions and visual elements, by systematic "attribute prompting." which tasks the MLLMs with generating detailed text descriptions of visual properties including shape, color and location. By doing so, we not only encourages the MLLMs to perform in depth visual reasoning around the target regions, but also circumvents MILMSs weakness handling image-text pairs, by creating additional intermediate text-text pairs that enable more robust comparison metrics.

---

Text-to-text CLIP scare t2t
Text-to-image CLIP score
Euse whe banggno morn copsed tack
locaned an center image
coend on the loper notton the eaal: gryy in mako and za carno and spareens that werthatrr wosliing canfort fansianalty.
ortice oudr budoons positoree in the upper certay in banke
with gray backing furoal by
reference prer
"mesh backrest"
Masked visuaal promiepss
solk up usssurs badkground and
Inout exp resiion E
condidate toxts (7) Ti
@t t.. the condidate 7fth refor desse on reference
g reforence text jefento
MLLM
Text toxtett-eext decision d
Text-to-image decision dd
Figure 4: Multi-metric mask proposal selection using MLLM and CLIP. To select the final mask from mask proposals generated by SAM, we introduce four metrics computed across different modalitics and models to evaluate the similarity between input expression E and the mask proposals. Specifically, the text-to-text MLLM-based binary decision d and CLIP score nd match reference text to candidate texts. The texl-to-image MLLM-based binary decision d and CLIP ش match reference text to masked visual prompts.
Figure 2 provides an overview of RESAnything. which consists two main stages: 1) an attribute prompting stage that generates reference text for the target and candidate texts for generated seg- mentation proposals (Section 3.1 2) a proposal selection stage that employs multiple metrics to robustly analyze the relationship between candidate and reference texts and produce the final response (Section 3.2)
3.1
Text Generation via Attribute Prompting
To facilitate reasoning between the input expression E and the segmentation proposals M, first apply attribute prompting to generate detailed text descriptions: reference text 7 which describes the input expression E in relation to the image candidate texts TT..N. which describe each of the segmentation proposals in a format similar to that of the reference text. We apply MILMs to generate these texts, carefully designing the input prompts encourage the MILMMs to provide description that capture comprehensive object properties and inter-object relationships.
Reference text generation. The reference text 7 functions as extended visual description of the input expression E, providing more concrete visual attributes for challenging expressions such part-level semantic labels and functionality/featurebbn based descriptions. We task MILMM to generate the reference text Tf = fMLLM(I, E Q with a carefully designed reference text prompt Qre! that instructs the MLLM to generate single sentence with detailed visual attributes, such shape, color and location, that describe the region It in I targeted by E. For invisible or irrelevant targets, the re provides reasoned explanation of why the target cannot be localized. We provide the full reference text prompt Qref in the supplementary. An example shown in the top part of the Fig 3 Given the input "mesh backrest". the reference text describes its key attributes: "a gray curved mesh backrest with black lining located at the upper portion the chair".
Candidate text generation. The candidate texts 7 Tean describe the mask proposals mi.
.... mi in a format similar to that of the reference text net Without requiring fine-tuning, our method can directly apply off- thel SOTA image segmentation methods to obtain mask proposals. We adopt SAM 24 in this work. As SAM's raw outputs often contain duplicate or overlapping masks, as well as tiny segments, we configure SAM with sampling points 0.015% of total image pixels and filter out segments smaller than 0.1% of the image area, preventing over-segmentation while maintaining meaningful region proposals. We also filter out duplicate proposals.
Given a mask proposal mi we generate a corresponding candidate text T:aan fVLLM(VV V.?... that similarly asks for visual attributes such as shape, color and location; and V/2 ... V.K are visual prompts that provide distinct visual representations of the mask proposal M.. good visual prompt
Q""") using an MILM,, where Q is the candidate text prompt
5

---

need to guide the MLLM to focus on the mask region, without removing attribute-relater information
or adding distractions.
Figure shows a few possible representations for visual prompis: image retains all information of the original image, but does not cover any mask-specific properties; mask cropped high- lights the visual attributes the masked region, but does not suggest the location of the masked region nor its relation with other parts of the image; in contrast, bounding box, mask contour and blur background provides such relational and locational information, but the bounding box outlines, the mask overlays, and blur background are distractions when it comes to visual prop- erties such color shape. Using multiple visual prompts, intuitively, alleviate the issues of the respective prompting representation. In practice, we find using two visual prompts, bounding box (V*) and mask cropped (V"",, is sufficient for our purpose. This is consistent with the observations of [54]. The complete candidate text prompt Qenn is provided in the supplementary. Fig 3 right part shows examples of generated candidate texts.
imae
ocpped
bounding
bacopround
Figure 5: Example of different visual prompts V; gener- ated from a segmentation proposal my
3.2 Multi-metric Mask Proposal Selection
The generated reference text and candidate texts allow us to assess the similarity between the input expression E and the mask proposals M much more effectively: the reference text 7e provides more detailed information than the original expression E, thus facilitating in depth text-to-image comparisons; in addition, the candidate texts 7 enables an additional modality, allowing direct comparisons between two piece of texts. In this stage, we combine multiple evaluation metrics to perform both text-to-image and text-to-text comparisons to select the mask proposal (or none) that matches the input expression.
Text-to-text comparison. To compare mask proposal m, against the input expression E, we first evaluate the similarity between the reference text describing E, and the candidate text describing m,,
We first use the same MLLM to generate binary decision dfah fMLLM " T:. Q(a) E {0, 1].
where Qt2t is the text-to-text comparison prompt, as shown in the lower left corner of Figure 4
The MILM outputs a yes/no binary decision, as we observed empirically that it often struggles to output consistent scalar scores. However, there are cases where multiple mask proposals receive "yes" response. To disambiguate such cases, we further employ CLIP to generate a scalar similarity score: sfah (T , Tran) E [0, IJ. Although CLIP is generally more error-prone (as we show in the supplementary), its ability to output consistent scalar scores makes it well-suited for further disambiguating among the top candidates filtered by the binary MLLM decision.
Text-to-image comparison. While the text-to-text metrics already enable good candidate selection, potential errors during candidate text generation could degrade their performance. To alleviate this, further perform text-to-image comparisons between the reference text and the mask cropped visual prompt V.n Similar to the text-to-text comparison, we use an MLL. M.L.M-generated binary decision de (T V.m Q E {0, 1}. followed by a CLIP-generated scalar score ofah = SCL ee ViT) E [0. 1]. where Q is the text-to-image comparison prompt as shown in the lower right corner of Figure 4
Grouping and selection. Given the computed metrics, we select the mask candidate that best matches the input expression E, or return the reference text T if none is found. Algorithm summarizes this process.
As MLLM decisions are prioritized over CLIP sores, begin by checking whether any masks receive positive responses for both text-to-text and text-to-image MLLM decisions. In practice, we notice that the correct candidate often the union of all the candidate masks that satisfy this condition, especially in cases where single semantic entity spans multiple segments (e.g., all legs of a sofa). Therefore, we also include the union of these masks as another viable candidate, We then return the mask candidate with the highest combined CLIP score (sum of 8/2 and 812)) If no such masks exist, we then repeat this process, using only the text-to-text MILM decisions as the filter, and then using only the ext-tt--maae MLLM decisions the filter.

---

Table 1: Quantitative results standard RES benchmarks refCOCO/+/g. reported as cloU values.
Method
refCOCO
refCOCO+
reCOOCOg
val
testA
testB
val
lestA
val(U)
val(G)
tesi(U)
felly-supervised on the maaining set
VLT 19
67.5
70.5
65.2
56.3
61.0
55.0
57.7
CRIS 50
70.5
73.2
66.1
62.3
68.1
53.7
59.9
60.4
LAVT 7R1
72.7
75.8
68.8
62.1
68.4
55.1
61.2
62.1
GRES
83
73.8
76.5
70.2
66.0
71.0
57.7
65.0
66.0
pre-frained on the same task
UniRES (59
71.2
74.8
66.0
59.9
66.7
51.4
62.3
63.2
LISA-7B
74.9
79.1
72.3
65.1
70.8
58.1
67.9
70.6
GSVA 666
77.2
78.9
73.5
65.9
69.6
59.8
72.7
73.3
GLaMM 46
79.5
83.2
76.9
72.6
78.7
64.6
74.2
74.9
SAM4MLLM
79.8
82.7
74.7
74.6
80.0
67.2
75.5
76.4
training-free zero-shot
GLCLIP 79
26.2
24.9
26.6
27.8
25.6
27.8
33.5
33.6
33.7
CaR 54
33.6
35.4
30.5
34.2
36.0
31.0
36.7
36.6
36.6
RESAnything
68.5
72.2
70.3
60.7
65.6
52.2
60.1
60.5
60.9
We also prioritize text-to-text over text-to-image decisions, as empirically, we find the former more reliable. As final verification step (lines 17-20 in Algorithm IT., when no candidates receive positive MILMM responses, we check if any of them has a combined CT.IP score over threshold (set to for all experiments), and return the mask with the highest score. This threshold helps identify cases where the target is either invisible or irrelevant to the image, in which case we return the reference text Te explanation that describes why the target cannot be localized.
This algorithm enables method handle clusion cases by combining parts segmentations, while also generalizing multi-object scenarios. Additional discussions and results are available in the supplementary materials.
Algorithm 1 Grouping and Selection Process
I: conditions + {(True, True), (True, False), (False, True)}
2:
3: for (t2f, t2i) in conditions do
121 and (2i then
5:
C C [m: d
=
Experiment
6:
df =113
7:
else if 12f then
8:
C' <- [m² d. =
We use Pixtral 12B 4 as the MILM. SAM ViT-
9:
else if /2i then
H [24] for generating segmentation proposals, and CLIP-VIT-B-I for CLIP scores. Our ex- periments were conducted on server with 8 NVIDIA 32GB V100 GPUs for parallel infer- ence, but the entire inference process can run effectively on just single NVIDIA 24GB 4090 GPU. Additional inference time details pro- vided the supplementary materials.
10:
C {me di 1)
11: CC = 1 then
12: return C
13:
else |C| 1 then
14:
Memb + CoombinMMssks())
15: Compute <mbb Sumb
16: return argmax
(CUUmm))) (s + m))
Public datasets. Following the most previous
18: else
works on referring segmentation 25
[2], we
19: pass
evaluate the performance of RESAnything
20: if max + sm) < then
four public benchmark datasets: RefCOCO, Re-
21: return ae
fCOCO+ RefCOCOg 40 42 and Reason-
22:
else
Seg 25 evaluate the validation and test sets without
Being a zero-shot method, we directly
23: return argmax((22.. 8.).m)
mEM
any fine-tuning.
ABO-Image-ARES benchmark. To further evaluate the capability of RESAnything in handling implicit expressions (e.g., part-level materials, features, and functionalities). we establish the ABO- Image-ARES benchmark for complex reasoning segmentation tasks. We build upon the ABO dataset, which contains product listings with rich metadata, images, and 3D models from Amazon.com. Our benchmark comprises 2,482 high-resolution catalog images spanning 565 product types, with 2,989 referring expressions targeting part-level regions that describe specific materials, features, functionalities, or packaging elements. Fig. 3 shows representative examples, with detailed refer extraction procedures and data annotation provided in the supplementary.
7

---

Table 2: Quantitative results ReasonSeg (left) and ABO-Image-ARRESrright))
val
Methad
gloU
cloU
Method
test
gloU
cloU
GLaMM 46
47.4
47.2
LISA-7B-LLLVALE
25
53.6
52.3
LISA-ISB-LLaVAI5
25
43.3
34.0
1BA-133-LLLaVA 25
57.7
60.3
GLaMM 46
46.2
38.7
SAM4MLLM I
58.4
60.4
RESAnything
78.2
72.4
RESAnything
74.6
72.5
Evaluation metries. We evaluate our method using standard metrics following prior works 25 4o1: generalized IoU (gloU) and cumulative IOU (cloU). gloU computes the scores, while cloU measures the ratio of cu- average per-image Intersection-over-Uniiiss
mulative intersection to cumulative union across all images. We report gIOU for Ref- COCO, RefCOCO+, and RefCOCOg, and both metries for ReasonSeg and ABO-Image-ARES.
4.1 Evaluation on Vanilla RES
We evaluate RESAnything on standard referring seg- mentation benchmarks, as shown in Table [ Our method significantly outperforms existing zero-shot approaches, more than doubling the performance of GLCLIP (68.5% vs 26.2% on refCOCO val set) and achieving comparable results with early supervised methods like VLT. Despite UniRES 59 being de- scribed zero-shot method, it was pre-trained on their proposed MRES-322 32M dataset, which remains unavailable to the public. Furthermore, due to UniRES being closed source, our comparisons limited the accuracy figures reported in their paper. The performance gap compared recent supervised methods can be attributed to our segmentation strategy with smaller mask proposals, which faces challenges when handling large complete objects that common in these datasets. Qualitative results are provided in the supplementary. Furthermore, we evaluate RESAnything with competing methods on more general referring segmentation tasks as detailed in our supplementary.
meterial
design
Landtin
packagg label
Figure 6: Examples of different expressions in A-BO-Imae-ARRe ARES. Best viewed with zoom-in.
4.2 Evaluation on Reasoning Segmentation
We evaluate RESAnything on the ReasonSeg benchmark (Table 2 where our method achieves state-of-the-art performance of 74.6% gloU and 72.5% cloU, surpassing LISA-13B by 17% and SAM4MLLM by 16%. Notably, while LISA variants require fine-tuning on reasoning tasks and GLaMM SAM4MLLM rely on extensive training data, RESAnything achieves this superior performance without any task-specific training, demonstrating the effectiveness of leveraging MLLMS for deep reasoning. Qualitative comparisons are shown Fig
ABO-Image-ARES contains more challenging referring expressions targeting materials, features, functionalities package elements. On this benchmark, RESAnything achieves 78.2% gloU and 72.4% cloU, significantly outperforming GLaMM by over 30% in both metrics, demonstrating our method's strong capability handling complex reasoning queries (See Fig 8
4.3 Evaluation on Affordance-based RES
We further evaluate our method on affoordance-based referring expression segmentation tasks, as shown in Table 3 All compared methods are fully-supervised and fine-tuned on COCO-Tasks training data, while RES RESAnything operates in zero-shot manner. As seen above, our method surpasses TOIST and TaskCLIP, demonstrating its strong generalization capability across affordance-based scenarios. When we incorporate task-specific prompt optimization, e.g., adding "human-object interaction" attributes, the performance improves from 51.2 to 54.6, approaching CoTDet. This demonstrates how prompt engineering can enhance the performance of our framework.

---

U5A
RESAnything
QUEYy
GLaMM
USA
RESA 1g
Figure 7: Qualitative comparisons on ReasonSeg. Our method demonstrates superior performance in both object localization accuracy (rows 3, 4) and seg- mentation precision (rows 5).
Figure 8: Qualitative comparisons on ABO-Image- ARES. RESAnything demonstrates superior general- ization ability across diverse queries, producing fine-grained segmentation.
Table 3: Results on COCO-Tasks(mloU)
Method
mloU@0.5 (14 tasks average)
supervised (trained on training set)
GGNN
51
32.4
TOIST
31
(w distillation)
44.1
Taskelip
9
50.3
CotDet
S
56.9
maining-free zero-shot
RESAnything
51.2
RESAnything prompt optimization
54.6
4.4 Ablation Study
Attribute prompts. Our core contribution attribute prompting, a novel mechanism that empha- sizes reasoning about part attributes to handle implicit queries and complex part-level relationships. To validate its effectiveness, we compare attribute prompting against conventional prompting baselines on the ReasonSeg test set, as shown Table
Table 4:
Ablation study comparing prompting
strategies on ReasonSeg test set.
Method
gloU
cloU
Visual prompts. As shown in Fig we explore different types of visual prompts for generating can-
Standard prompt
50.8
49.3
didate texts Tean and performing text-to-ing -mage com- parison. Table S compares their performance on Ref- COCO test A set. The combination of mask-cropped and bounding box prompts achieves the best performance (72.2% gloU), while using mask alone yields the lowest (47.2% gloU) as obscures contextual relationships. This demonstrates the importance of preserving spatial context through bounding box while maintaining region-specific details through mask cropping. Additional analysis provided in the supplement.
Attribute prompt
74.6
72.5
MILM backbone. To analyze the impact of varying the MLLM backbone, we compare the performance of different MILMS on ReasonSeg. Table 5 summarizes the results. While Pixtral-12B our default choice, both Qwen2-VL and Claude 3.5 Sonnet achieve comparable slightly better performance (74.2-76.25 gloU), demonstrating our method's robustness across different MI.MMs. See supplementary materials for extended analysis.

---

Table Ablation studies visual prompts (left) and MLLM backbone (right).
Dataset
Visual Prompts
gloU
cloU
Image
mask
bbox
contour blur
47.2
42.3
L.MM
gloU
cloU
56.2
53.3
48.4
44.2
Pixtral 12B 4
74.6
72.5
RefCOCO test A
43.5
39.2
Claude3.5
76.2
73.4
67.4
64.1
Qwen 2-VL
74.2
72.1
72.2
69.5
68.5
64.4
50.4
46.6
Conclusion, limitation, and future work
We present RESAnything, zero-shot approach advance open- vece-vocabllary RES by supporting lan- guage expressions referring to highly general concepts. Our method comprises two key components: novel attribute prompting technique to extract detailed attributes as text descriptions by synergizing SAM and MLLM for CoT'' analysis, and a multi-metric mask selection module based on CLIP and MLLM to select the optimal mask from SAM proposals.
Our method demonstrates superior performance over prior zero-shot methods on standard RES benchmarks (RefCoCO///g) More importantly, training-free free approach substantially outperforms existing fine-tuned MLLM methods on both ReasonSeg 25 for reasoning segmentation and our newly augmented ABO dataset, underscoring its comprchensive reasoning capabilities. While RE- SAnything also performs well on object-level RES, attribute prompting excels especially at part-level reasoning since the attributes considered (color, shape, and location) tend to exhibit more consistency over parts, than objects, that share similar functions, styles, material, etc. It would be interesting to explore other attributes for CoT' automate the prompts.
Our method has substantial room for inference efficiency optimization in future work, particularly through Rol filtering and size-based mask proposal pruning to reduce candidate text generation overhead. RESAnything also inevitably inherits limitations common to foundation model-based approaches. Notably, SAM occasionally fails to produce the best mask candidates, potentially degrading RES accuracy, as shown the supplementary materials. In addition, effectiveness of RESAnything depends the specific MLLMs employed. Future work could focus improving the mask proposal generation process and exploring the integration of more advanced LLMs/MLLMs.
Acknowledgements
We thank Kai Wang for his dedicated involvement and commitment throughout this project. We also thank Yiming Qian and Fenggen Yu for their invaluable contributions, and Maria Zontak, Brian P. Jackson, and Eric P. Bennett for their guidance and support.
References
[1] Claude 3.5 sonnet.
[2] Google gemini.
[3] Openai gpt-4o. httess//oeeaaa.cmm////dee//l//1s
[4] Pravesh Agrawal, Szymon Antoniak, Emma Bou Hamna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, et al. Pixtral 12b. arXiv preprint arXiv:2410.07003,, 2024.
[5] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances neural information processing systems, 35:23716-23788 2022.
10

---

[6] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre. Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2208..01333 0/300,, 2023.
[7] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 00.300..29960 /2966, 2023.
[8] Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory Meyer, Yuning Chai, Dennis Park, and Yong Jae Lec. Making large multimodal models understand arbitrary visual prompts. In IEEE Conference on Computer Vision and Pattern Recognition, 2024.
[9] Hanning Chen, Wenjun Huang, Yang Ni, Sanggeon Yun, Yezi Liu, Fei Wen, Alvaro Velasquez, Hugo Latapie, and Mohsen Imani. Taskelip: Extend large vision-language model for task oriented object detection. In European Conference on Computer Vision, pages 401-418. Springer, 2024.
[10] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306.15795, 2023.
[11] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.1279) 2023.
[12] Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu-Chiang Frank Wang, and Chu-Song Chen. Sam4mllm: Enhance multi-modal large language model for referring expression segmentation. In European Conference on Computer Vision, pages 323-340. Springer, 2025.
[13] Yong Xien Chng, Henry Zheng, Yizeng Han, Xuchong Qiu, and Gao Huang. Mask grounding for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26573-26583, 2024.
[14] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik. Abo: Dataset and benchmarks for real-world 3d object understanding. CVPR, 2022.
[15] Ming Dai. Lingfeng Yang, Yihao Xu, Zhenhua Feng, and Wankou Yang. Simvg: simple framework for visual grounding with decoupled multi-moda fusion. Advances in neural information processing systems, 37:121670-121698, 2024.
[16] Wenliang Dai, Junnan Li, D Li, AMH Tiong, Zhao, W Wang. B Li, P Fung, and S Hoi. Instructblip: Towards general-purpose purpose vision-language models with instruction tuning. arxiv 2023. arXiv preprint 13Xi:23305006000 2, 2023.
[17] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for statc-of-the- multimodal models. arXiv preprint 14X1::449..17466 2024.
[18] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04900 2018
[19] Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages
2021.
[20] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. alama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2344.15010, 2023.

---

[21] Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expressions. In Computer Vision- ECCo 2016: 14th European Conference, Amsterdam, The Netherlands, October I 11-14, 2016, Proceedings, Part I 14, pages 108-124. Springer, 2016.
[22] Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero-shot open-vocabulary segmentation. arXiv preprini arXiv:2306. 09316, 2023.
[23] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[24] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026. 2023.
[25] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9579-9589, 2024.
[26] Mengcheng Lan, Chaofeng Chen, Yue Zhou, Jiaxing Xu, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Text4seg: Reimagining image segmentation as text generation. arXiv preprint arXiv:2410.09855, 2024.
[27] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023.
[28] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: multi-modal model with in-context instruction tuning. corr abs/2335.03326 (2023), 2023.
[29] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping eeguaae-immgge pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[30] Muchen Li and Leonid Sigal. Referring transformer: one-step approach to multi-task visual grounding, Advances in neural information processing systems, 34: 14:19652-1964,, 2021.
[31] Pengfei Li, Beiwen Tian, Yongliang Shi, Xiaoxue Chen, Hao Zhao, Guyue Zhou, and Ya-Qin Zhang. Toist: Task oriented instance segmentation transformer with noun-pronout distillation. Advances in Neural Information Processing Systems, 35:17597-17611, 2022.
[32] Yang Li, Xiaoxue Chen, Hao Zhao, Jiangtao Gong, Guyue Zhou, Federico Rossano, and Yixin Zhu. Understanding embodied reference with touch-line transformer. In ICLR, 2023.
[33] Chang Liu, Henghui Ding, and Xudong liang. Gres: Generalized referring expression segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23592-23601, 2023.
[34] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26396, 2024.
[35] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024.
[36] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[37] Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 18653-18666 2023.
12

---

[38] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan
Li, Jianwei Yang, Hang Su, et Grounding dino: Marrying dino with grounded pre-training for open-set object detection, arXiv preprint arXiv:2303.054993 2023.
[39] Sun-Ao Liu, Hongtao Xie, Jiannan Ge, and Yongdong Zhang. Refersam: Unleashing segment anything model for referring image segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 2025.
[40] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016.
[41] Sayan Nag, Koustava Goswami, and Srikrishna Karanam. Safari: Adaptive s equence ns f ormer for we kly supervised eferring expression segmentat i on. In European Conference on Computer Vision. pages 485-503. Springer, 2024.
[42] Varun K Nagaraja, Vlad Morariu, and Larry Davis. Modeling context between objects for referring expression understanding. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 792-807. Springer, 2016.
[43] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
[44] Shraman Pramanick, Guangxing Han, Rui Hou, Sayan Nag, Ser-Nam Lim, Nicolas Ballas, Qifan Wang. Rama Chellappa, and Amjad Almahairi. Jack of all tasks master of many: Design- ing general-purpose coarse-to-fi fine vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14076-14088, 2024.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748- 0163. PMLR, 2021.
[46] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji. AbdeIrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 1309--3018 2024.
[47] Nikhila Ravi, Valentin Gabeur, Yuan-Tine Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma. Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprins wwXiv2408.007714 2024.
[48] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open open-world world models for diverse visual tasks. arXiv preprint arXiv:2401. 14159, 2024.
[49] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 26374-26333 263883, 2024.
[50] Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXhv:2402.0799 2024.
[51] Johann Sawatzky, Yaser Souri, Christian Grund, and Jurgen Gall. What object should usek-task driven object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7605-7614, 2019.
[52] Hengcan Shi, Hongliang Li, Fanman Meng, and Qingbo Wu. wey-word- network for referring expression image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 38-54, 2018.
13

---

[53] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1 11987-11997, 2023.
[54] Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, and Siyang Li. Clip as rnn: Segment countless visual concepts without training endeavor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13171-13182 2024.
[55] Yanpeng Sun, Jiahui Chen, Shan Zhang, Xinyu Zhang, Qiang Chen, Gang Zhang, Errui Ding, Jingdong Wang, and Zechao Li. Vrp-sam: Sam with visual reference prompt. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23565-23574, 2024.
[56] Yucheng Suo, Linchao Zhu, and Yi Yang. Text augmented spatial-aware zero-shot referring image segmentation. arXiv preprint arXiv:2310. 18049, 2023.
[57] Jiajin Tang, Ge Zheng. Jingyi Yu, and Sibei Yang. Cotdet: Affordance knowledge prompting for task driven object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3068-3078, 2023.
[58] Wenhai Wang, Zhe Chen. Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024.
[59] Wenxuan Wang, Tongtian Yue, Yisi Zhang, Longteng Guo, Xingjian He, Xinlong Wang, and Jing Liu. Unveiling parts beyond objects: Towards finer- amer-granularity referring expression segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages -9998-13008, 2024.
[60] Zhaoqing Wang. Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1186-116695 2022.
[61] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thoughh prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-28877, 2022.
[62] Zhichao Wei, Xiaohao Chen, Mingqiang Chen, and Siyu Zhu. Linguistic query-guided mask generation for referring image segmentation. arXiv preprint arXiv:2301.064/ 2023.
[63] Jiannan Wu, Yi Jiang, Bin Yan, Huchuan Lu, Zehuan Yuan, and Ping Luo. Segment every reference object in spatial and temporal spaces. In ICCV, 2023.
[64] Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li, Zhaoyang Xia, Aaron Chang, Tong Yu, Sungchul Kim, Ryan A Rossi, Ruiyi Zhang, et al. Visual prompting in multimodal large language models: A survey, arXiv preprint S4tiv:2409.1531111 2024.
[65] Tsung-Han Wu, Giscard Biamby, David Chan, Lisa Dunlap, Ritwik Gupta, Xudong Wang, Joseph E Gonzalez, and Trevor Darrell. See say and segment: Teaching Imms to overcome false premises. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13459-13400 2024.
[66] Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3858-3869, 2024.
[67] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu. Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for variety of vision tasks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 4818 4829, 2024.
[68] Linhui Xiao, Xiaoshan Yang, Fang Peng, Yaowei Wang, and Changsheng Xu. Oneref: Unified one-tower expression grounding and segmentation with mask referring modeling. In Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024.
14

---

[69] Jiarui Xu, Xingyi Zhou, Shen Yan, Xiuye Gu, Anurag Arab, Chen Sun, Xiaolong Wang, and Cordelia Schmid. Pixel-alignets language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13030-13039, 2024.
[70] Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an Imm perceiving any aspect ratio and high-resolution images. arXiv preprini arXiv:2403. 1703, 2024.
[71] Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arXiv:2310. 11441, 2023.
[72] Lingfeng Yang, Yueze Wang. Xiang Li, Xinlong Wang, and Jian Yang. Fine-grained visual prompting. Advances in Neural Information Processing Systems, 36:24993-25006, 2023.
[73] Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18155-18165, 2022.
[74] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309. 17421, 9():1, 2023.
[75] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 00502-10511, 2019.
[76] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13040-13051, 13051, 2024.
[77] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704,000 2023.
[78] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara I. Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part II 14, pages 69-85. Springer, 2016.
[79] Seonghoon Yu, Paul Hongsuck Seo, and Jeany Son. Zero-shot referring image segmentation with global-local context features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19456-19449 19465, 2023.
[80] Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28202-2821 1, 2024.
[81] Ao Zhang, Liming Zhao, Chen- Chen-Wei Xie, Yun Zheng, Wei Ji, and Tat-Seng Chua. Next-chat: An Imm for chat, detection and segmentation. arXiv preprint arXiv:2311.0m 1.44998, 2023.
[82] Shilong Zhang. Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interess arXiv preprint arXiv:2307.03661, 2023.
[83] Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond hla Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406.0m 084877 2024.
[84] Yuxuan Zhang, Tianheng Cheng, Rui Hu, Lei Liu, Heng Liu, Longjin Ran, Xiaoxin Chen, Wenyu Liu, and Xinggang Wang. Exf-sam: Early lasion-language fusion for text-prompted segment anything model. arXiv preprint arXiv:2406.20070 2024.
15

---

[85] Zicheng Zhang, Yi Zhu, Jianzhuang Liu, Xiaodan Liang, and Wei Ke. Coupalign: Coupling word-pixel with sentence-mai mask alignments for referring image segmentation. Advances in Neural Information Processing Systems, 35:14729-14742, 2022.
[86] Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything. arXiv preprint 2wt.:::2336.115555.. 2023.
[87] Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402. 14289, 2024.
[88] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pages 696-712. Springer, 2022.
[89] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing usioo-languaae understanding with advanced large language models. arXiv preprint a304 /0592, 2023.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Contributions and scope of the paper are clearly claimed in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This paper does not include theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Answer: [Yes]
Justification: Implementation details are presented in the main paper and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct. the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer:
[NA]
Justification:
Code and data are not included in the submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cc/
public/gguides//odeSuubiii http://ddwsuubiissooooooooi...e/ for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https / //nips. wtipssccc/pbb...... wwwplliiiggii/eeeeeeeeeee /ooee// usionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Implementation details including prompts used are presented in the main paper and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Results are presented with details.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details of computer workers are presented in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research was conducted win a paper conform in every respect, with the NeurIPS conde of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper introduces method for referring expression segmentation, which has no societal impact regarding the negative presented in the guidelines.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper cite and state the original paper and dataset clearly.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, |paper swperswiie thcode. com/ cooo/aaasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper proposed new dataset and benchmark in details.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: This paper leverages the power of LLMs to reason the relationship between targets in the images, which support the task in segmentation.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:////eerrrpsscccoooofeeeeees for what should or should not be described.
23

---

RESAnything: Attribute Prompting for Arbitrary Referring Segmentation
Supplementary Material
The supplementary document provides (1) detailed anal- ysis of limitation of current methods, including both MILMM and CLIP in our task, in Section 1; (2) comprehensive details of language and visual prompts used in RESAny- thing in Section 2: (3) additional information about the con- struction of ABO-Image-AREE
ilarity computation. Therefore, we reformulate similarity assessment as binary classification queries, returning yes or no in our selection algorithm, which better aligns with MILMs' natural language understanding capabilities. As shown in Fig 2, our experiments reveal that MLLMs tend to generate similarity scores that appear arbitrary or biased by their training distribution, rather than computing true sim- ilarities between the given elements, and their binary re- sponses prove to be more reliable indicators.
in Section 3; (4) extended quantitative results on part level and multi-object GRES task and qualitative results, including failure cases in Sec- tion 4 and 5, respectively.
1. Limitation of Current Methods
1.2. Limitation of CLIP
Our method leverages Chain-of-Thought (CoT) attribute prompting for detailed descriptions and combines MLLMs and CLIP as mask selector to select optimal segmentation proposals. While this dual-mode approach achieves strong performance, arises from the inherent limitations of both components. In this section, we analyze the constraints of current MLLMs and CLIP that motivate our design choices in attribute prompting and the hybrid evaluation strategy.
The limitations of CLIP in analyzing contextual similari- ties become evident when dealing with complex descrip- tions and image content. As shown in Fig 3, while CLIP's text-to-text similarity scores reveal meaningful comparison, they often fail to capture erucial contextual details like color attributes. Additionally, CLIP's text-to-image similarity scores show limited discriminative power, consistently re- maining below 0.3. These limitations underscore our de- cision to adopt MILMS as our primary mask selector, as they demonstrate superior capability in understanding and comparing detailed contextual content.
1.1. Limitation of MILM
Attribute Prompt. While MLLMs exhibit strong reason- ing capabilities, they often fail to perform systematic CoT reasoning without explicit prompting guidance. As shown in Fig 1, when asked to describe the details of input expres- sion E without specific attribute requirements, MLLMs typ- ically generate oversimplified descriptions that fail to cap- ture the target's essential characteristics and details effec- tively. Therefore, providing MLLMs with explicit attribute requirements is essential guide their reasoning process effectively. RESAnything leverages this insight to gener- ate more comprehensive and accurate descriptions, ensuring that all necessary details of the target expression are prop- erly captured.
1.3. Ablation Study
We further evaluate the effectiveness of adopting both MLLM and CLIP as mask selectors in RESAnything. Ta- ble 1 compares the performance of RESAnything Rea- sonSeg test set with different mask selectors configurations. Using CLIP as the sole mask selector results in poor perfor- mance due to its previously mentioned limitations in un- derstanding complex relationships and abstract concepts. While MLLM demonstrates superior reasoning and contex- tual similarity capabilities compared to CLIP, using MILM alone can lead to incomplete region selection, particularly for expressions targeting multiple parts (e.g., sofa legs or armrests). These results validate our design choice of incor- porating both MILM and CLIP as mask selectors to ensure robust region selection.
Binary Response. As mentioned in our main paper, a naive approach for applying MLLMs to solve our task would involve prompting the MILMMs to output a score for each segmentation proposal m, denoting its similarity with the input expression E. However, MILMS signed to understand and generate text rather than compute precise numerical similarities. While they excel at compar- ing and reasoning about content qualitatively, they struggle to produce reliable numerical similarity scores. Our exper- iments reveal that MLLM-generated similarity scores ex- hibit high variance and poor correlation with actual con- textual similarity, the model essentially samples from its probability distribution rather than performing true sim-
primarily de-
Table Ablation study on different mask selectors.
test
Method
gloU
cloU
CLIP only
42.5
38.4
LLM only
70.5
64.6
both
74.6
72.5

---

2. Prompts
combine all information seamlessly; Focus on creating cohesive descriptions that feel natural and informative without drawing attention to the source of the information.
2.1. Language Prompts in Attribute Prompting
As mentioned in the main paper, we use reference text prompt Q to generated reference text Twe for each refer based on the imput expression E. Given the input image I and referring expression E, we prompt the MLLM using following Q to obtain reference text T
We adjust the Qan based different visual prompts for ablation study, e-g. mask cropped V only: You are presented with a cropped masked view showing detailed visual properties;
For the region described as (b) in the image, provide a single detailed sentence describing an object or part of object by including its location, appearance (color, shape, location), including
Fig 4 shows examples of query (input expression) and generated reference & candidate text.
2.2. Language Prompts in Grouping and Selection
and distinctive characteristics relevant actions, state, or function. Focus on features that would help uniquely identify this specific region from others in the image. Be as succinet as possible and in English only.
We employ MILM as one of the mask selectors in our grouping and selection algorithm. Certainly, for text-to-text decision d use following Q
You are evaluating if the following candidate text describes the input expression region: E. Reference information provided for context if the input expression text is not clear: 7 Here is the candidate text to evaluate: Tean Evaluate if the candidate text refer to the target by checking:
Similarly, given the mask cropped V." and bounding box image V/t visual prompts of segmentation proposal my prompt the MILM using following Quan obtain candidate text T
You are presented with two complementary views of the same region: 1) A cropped masked view showing detailed visual properties; 2) A full view with bounding box showing location and context. Generate single detailed sentence following these guidelines:
Spatial location match;
Visual characteristics match (color, shape, size);
Object/subject identity match; State/action consistency (if applicable).
Return 'yes' 'no' ONLY: 'yes' if most aspects
FOR COMPLETE OBJECTS:
substantially match; 'no' if some significant aspect differs.
Combine visual details and spatial context naturally;
Visual properties (color, shape, texture, size);
For text-to-image decision d we use following Q
Location in the scene;
are evaluating if the following reference text describes the non-black region of the cropped mask image: Tref, The target is E for context if the reference text is inaccurate. You have two images for context: 1) A cropped mask image showing An image with a green bounding box surrounding the region showing the full scene and spatial relationships Evaluate if the reference text describes the non-black region of the cropped mask image by checking:
Relationships with surroundings;
State or action if relevant;
FOR PARTIAL REGIONS:
Describe the part while providing clear context: Part identification and its visual properties: Its position within the larger object/scene; Relevant contextual details; Important Rules: Start directly with the subject: 'A [description]...
region in non-black color; 2)
'The [description]...] Describe only what is visible in the non-black regions for visual properties and the image with green bounding box is for location and relation analysis;
Spatial location match (the location is relative location, not absolute location); Visual characteristics match (color, shape, size)
Never mention masks, boxes, or annotations; Use confident language for clear identifications; Use tentative language when inferring; Create natural, flowing descriptions that
Object/subject identity match (the masked image could be only a part of the target); State/action consistency (if applicable).
2

---

Return 'yes' or 'no' ONLY: 'yes' if most aspects substantially match; 'no' if some significant aspect differs.
maintaining accurate spatial understanding.
3. ABO-Image-ARES Data Preparation
2.3. Visual Prompts Selection
3.1. Image Data
We explore five visual prompts V; in method: (1) origi- nal image, (2) mask-cropped image, (3) bounding box over- laid on image, (4) mask contour overlaid on image and (5) blur background overlaid on image. We choose the combi- nation of mask-cropped image and bounding box overlaid on image as the best visual prompts Vi to obtain candidate text Tean Apart from quantitative results presented in the ablation study, we further analyze the effectiveness and lim- itation of different individuul/commiiatioon of these visual prompts, as shown in Fig. 5:
Our dataset builds upon image data from ABO [1], a dataset collected from worldwide Amazon.com product listings, including their metadata, images, and 3D models. ABO encompasses 147,702 product listings across 576 prod- uct types from various Amazon-owned stores and web- sites (e.g., Amazon, PrimeNow, WholeFoods). Each list- ing is uniquely identified by an item ID and contains struc- tured metadata from its public webpage, including prod- uct specifications such as type, material, color, and di- mensions, along with associated media. The dataset con- tains 398,212 high-resolution catalog images in total. How- ever, to better highlight product properties, we excluded im- ages from 11 categories: phone-related items (phone acces- sories, cellular phone cases, cellular phones, phones, wire- less locked phones), footwear (shoes, shoe inserts, technical sport shoes, boots, sandals), and picture frames. Most im- ages from these categories have no meaningful interest- ing groundable/referraale parts, as shown in Fig. 7. We also selected only the main image of each product, as additional images often show material details or close-up views. As results, ABO-Image-ARRI contains 2,482 high-resolution catalog images spanning 565 product types.
mask cropped only: with mask cropped as the only vi- sual prompt, MLLM is usually failed to infer the ac- tion/relation of the region. Example in Fig 5 shows that from mask cropped image, MLLM generates incorrect description of the region regarding its location and action. blur only: similar to mask cropped only, using blurred background as the sole visual prompt creates challenges for MILMM in distinguishing boundaries between blurred and clear regions, resulting in inaccurate location identi- fication. Critical action-related details may also be ob- scured by blurring, leading to incorrect classification of object activities.
original image with mask-cropped: while adding the orig- inal image helps MLLM better understand location and relationship,, the lack of explicit region guidance causes MLLM to be distracted by irrelevant regions outside the mask cropped area.
3.2. Referring Expression Generation
The referring expressions in ABO-Image-ARES were derived from product metadata, specifically the bulletpoint descriptions that accompany each product listing in ABO. These bulletpoints typically contain detailed information about product features, materials, and functionalities. We processed these descriptions through MLLM, instructing to generate 2-3 referring expressions per product. Prompt for instruction is following:
mask cropped with mask contour overlay: adding con- tour helps MLLM focus on the target region's boundaries, but the choice of overlay color can inadvertently influ- ence MILM's perception of the region's visual attributes. Attempts to show contours without color overlay (Fig 6) often result in ambiguous or confusing visual prompts, particularly for intricate shapes or overlapping regions if the contour is a non-convex shape.
Here is an image of a product. These are the product descriptions for it: (bulletpoints). Please analyze the descriptions and list 2-3 most important features or functionality. Return key words only without any starting or ending statements. Do not include dimension or assembly information. Each feature should be informative. If you cannot extract any relevant product features from both the image and description, return 'N/A'.
bounding box with mask contour overlay: while both ele- ments help localize the target region, their overlay colors can affect MILM's understanding. Even when explicitly prompted to focus on either the bounding box or contour region, both colors influence MLLM's perception of vi- sual attributes, leading to inconsistent descriptions. bounding box with mask-cropped (RESAnything): This combination achieves the best balance- - the bounding box provides spatial context and relationship guidance, while the mask-cropped image offers detailed visual attributes without color interference. By instructing MLLM to fo- cus on the mask-cropped region while using the bounding box for context, we avoid noise from overlay colors while
To ensure quality and visual grounding, we manually filtered out expressions that is 'N/A" and could not be reliably mapped to specific regions in the product images. We also manually reviewed all generated expressions to
3

---

ensure the dataset's 's quality. All manual processing was completed by evaluators. Each evaluator was required to review all the image-expression pairs and judge each expression as either "good" or "bad." To quantify inter- annotator agreement. we employed Fleiss' Kappa [2]. which is suitable for measuring agreement among multiple raters beyond what would be expected by chance. For expressions with low agreement among evaluators (such as 2-2 splits), we either modified the expression manually or removed it from the dataset entirely. The final dataset consists only of expressions that received strong majority approval (3-1 or 4-0 votes) and demonstrated clear visual grounding in the product images. This rigorous curation process yielded 2,989 referring expressions, each target- ing part-level regions and describing specific materials, features, functionalities, or packaging elements.
materials.
4.1. CLIP as RNN
We present quantitative results of CLIP as RNN, the cur- rent SOTA zero-shot method, on both ReasonSeg and ABO- Image-ARES in Table 2.
Table 2. Quantitative results of CLIP as RNN [8], with RESAny- thing's results shown in parentheses for comparison.
test
Datasel
gloU (ours)
cloU (ours)
ReasonSeg[5]
35.2 (74.6)
26.4 (72.5)
ABO-Image-ARES
24.4 (78.2)
15.7 (72.4)
4.2. Part-only RES benchmark
3.3. Mask Annotation
We further evaluate the performance of RESAnything and competing methods on UniRES [9], which contains a subset RefCOCOm for part-level RES. Table shows the quanti- tative results on part-only RefCOCOm. Since the code for UniRES [9] is not publicly available,
Our annotation process leverages SAM [4] to achieve ef- ficient and accurate region segmentation. The annotation workflow consists of two stages: automatic segmentation and manual refinement. In the first stage, we utilize SAM's automatic mode generate comprehensive set of candi- date segmentation masks for each image. GT regions that correspond to our referring expressions are then selected from these candidates. For regions that SAM failed to iden- tify automatically, we proceed to the second stage where we manually annotate them using SAM's interactive mode with point supervision. This semi-automated approach sig- nificantly streamlines the annotation process while ensuring precise region segmentation for our dataset.
directly compare performances using the mloUs reported in their paper. Al- though UniRES is claimed to be a zero-shot method, it is pre-trained on their proposed MRES-32M dataset, which is closed source. Our method significantly outperforms the training-fiee free zero-shot CaR, and generally outforms the su- pervised UniRES and LISA, even though they were both pre- pre-trained on related tasks. GLaMM is the same and is slightly ahead of ours, but this is attributable to its addi- tional fine-tuning on their proposed GranD dataset.
Similar to the evaluation of expressions, we also con- ducted quality assessment for the segmentation annotations. The same panel of 4 evaluators reviewed each segmented region and classified them as either "good" or "bad" based on their accuracy and alignment with the corresponding ex- pressions. We applied Fleiss' Kappa [2] to measure inter- annotator agreement for these segmentation evaluations well. Regions with low agreement scores were flagged for re-annotation using more precise point supervision in SAM's interactive mode. Only segmentations that received strong majority approval (3-1 or 4-0 votes) were retained in the final dataset, ensuring that our ground truth regions accurately represent the visual elements referenced in the expressions.
Table 3. Quantitative results on RefCOCOm Part-only set.
Method
val
testA
tesiB
supervised/ / pre-truined
UniRES [9]
19.6
16.4
25.2
LISA [5]
21.2
19.1
27.4
GLaMM [7]
30.0
27.2
31.8
training-free zero-shot
CaR [8]
10.9
10.6
10.9
RESAnything
27.6
26.5
25.8
4.3. Multi-object GRES benchmarks
Although RESAnything is specifically designed for multi-object RES task, it still effectively handles these cases through the grouping and selection algorithm, demonstrat- ing the generalization on these tasks. Table reports quali- tative comparison on a GRES benchmark, g g-RefCOCO [6]. Among the methods, only GRES is trained on g-RefCOCO. RES Anythngg achieves comparable results as LISA and GLaMM, while significantly outperforming the training- free zero-shot method CaR.
4. Quantitative Results
To ensure statistical robustness and account for potential variability in RESAnythin''s performance, especially for the components involving LLM generation (reference text, candidate text, and similarity analysis). we conducted ex- periments with our approach 8 separate times and report the averaged results in both the main paper and supplementary
4

---

Table 4. Results gRe/COCO (cloU).
sequential processing of each reference and candidate text; and 4) employing batch computation for CLIP similarity scores.
Method
testA
testB
pre-trained on vanilla RES tosks
LISA [5]
48.4
45.1
46.3
Table 6. Runtime comparison.
GLaMM [7]
46.2
46.7
47.2
supervised (trained on gRejCOCO)
Method
Time/iaae
GRES [6]
62.4
69.3
59.9
CaR LISA
5,3
training-fre free zero-shot
7.0
CaR [8]
25.6
22.0
21.5
GLaMM
8.6
RESAnything
52.7
46.2
46.3
RESAnything-Qwar 2-VL
12.1
We conducted additional evaluations of our method against competing methods on R-RefCOCO [10] and Re- fZOM [3]. Images in both datasets are extracted from the RefCOCO, with additional multi-object referring ex- pressions. Table shows the quantitative results on both benchmarks. Both RefSegformer [10] and DMMI [3] are fully supervised method trained on the training set of R- RefCOCO and RefZOM separately. LISA and GLaMM also pre-trained image data from COCO, which serves as the based of both benchmarks. Our method reasonaly underperformed against supervised methods that were ex- plicitly exposed to the training set, but still outperforms the SOTA training-free zero-shot baseline.
5. Qualitative Results
Firstly, Fig 9 17 show qualitative results on RefCOCO test A, test B. RefCOCOg val (G), val (U), test (U), Re- fCOCO+ test A, test B, val set separately. These exam- ples are randomly selected to provide an unbiased as- sessment. RESAnything achieves comparable results to su- pervised methods on vanilla referring segmentation tasks. While our approach effectively handles many occlusion cases, shown in Fig 8, neither our method nor current SOTA approaches can guarantee perfect part detection in every inference. Some failure cases in these results demon- strate challenges in combining parts or handling occlusions. Detailed descriptions and analysis of these failure cases are provided the individual figure captions.
Table 5. Results on R-RefCOCO and RefZOM(mloUU
Secondly, additional qualitative results on ReasonSeg [5] are shown in Figs. 18-20, including comparisons with CLIP RNN (CaR) [8], the current SOTA zero-shot method. Compared to supervised methods, our Chain-of-Thoughts attribute prompting enables clearer input expression iden- tification and more precise segmentation results. Detailed descriptions and failure cases are provided in the captions. Lastly, we present qualitative comparisons on ABO- Image-ARES (Figs, 21-23) to demonstrate our method's ef- fectiveness challenging expressions involving parts, ma- terials, functionality, and package elements. See figure cap- tions for detailed analysis.
Method
R-ReCCOCO ReIZOM
supervised (trained maaining set)
RefSegformer [10]
68.8
DMMI [3]
68.2
pre-truined
LISA [5]
71.1
45.0
GLaMM [7]
72.1
47.4
training-free zero-shot
CaR [8]
30.2
25.7
RESAnything
61.2
40.3
4.4. Runtime Comparison
As stated in the main paper, our method's entire infer- ence process can run efficiently on a single NVIDIA 24GB 4090 GPU. For a fair comparison, we measured the exe- cution times of all competing methods on the same hard- ware. The average per-image processing time was evalu- ated the ReasonSeg test set, with detailed results pro- vided in Table 6. While main results in the main paper were conducted using 8 V100 GPUs for running multiple experiments in parallel during development, we optimized our method's runtime for comparison experiments. These optimizations include: 1) utilizing the bfloat16 data format for the LLM, which is not supported on V100; 2) enabling flash attention for more efficient transformer operations; 3) implementing batch generation for LLM outputs rather than
5

---

standard prompt:
attribute prompt:
For the region described as E the image, provide a single detailed senterce describing an object or part of an object by including its location, appearance (color, shope, locotion), and distinctive characteristics including relevont actions or stote, Focus on features thot would help uniquely identify this specific region from others in the image.
For the region described os in the image, provide a single detailed sentence describing on object or part of on object.
input expression E
reference text Tre (standard prompt)
reference text T (attribute prompt)
image
The person on left a yourng man a crouched, othieric stance wearing bright blue shirt, black shorts, block knee-high socks, and distinctive red bosehali cap, holding a white frisbee his hands cotch it during what appeors to be Ultimate Frishee game on a grassy fild..
person on left
person wearing black socks.
left
prepares
The right girl is weaning black with the lower part visibl, extending from the midsection to the bottom of the image, featuring a slight flore.
right girl
right girl weoring G striped hot.
A ployer wearing a white jersey with the number 15, pasitioned near goal net, their bock to the comero and hair bied in bun, actively engoged in the play as they appear to be moving towards or interacting with the soccer on the ground
The sock of the player
15
numbered 15 is black.
The frant pizza' is large, round Neapolitan- pizzo in the foreground of the image, featuring thin crust with charred edkees, topped with a red tomato souce, scottered herbs, what appear to be olives, sitting white plate with fork beside on a white poper napkin.
The front pizza has charred cnust edge with visible burn marks.
front pizza
The for right region of image shows person standing slightty aport from the group, wearing while shirt with turyard, dark punts, and appearing to observing
The for right mon is holding a drink in right hand.
for right dud
with their hands at their sides, positioned near what looks like on entryway or doorfrome in the background.
The pillow over the baby's head is located the top of the image, partiolly covering the baby's and neck,
Pillow over baby head is a portion of soft pillow that covers the tap of the baby's head.
pillow aver baby head
soft, podded appearance and gentle foids. is white or light-colored, providing comfort and support while the boby drinks from the bottle.
Figure 1. Comparison of Text Generation With and Without Attribute Prompting: Our analysis demonstrates that when attribute prompting not used, MLLM fails accurately identify and reason about input expression attributes. The contrast between standard prompting and attributesppecific prompting highlights this significant limitation attribute recognition.

---

prompt for rating:
For the given mosk-crapped imoge and texts (E and Tref), rate their semontic similarity. Provide two scores between O-T, where 1 meons perfect match.
input expression E: guy on right
(0.8, 0.8) (0, 0)
(0.9, /
Reference text Tref.
The figure on the right is a person wearing a black hoodie and light blue jeans, standing with their back to the camera, holding a skateboard in their right hand while facing a grand, ornate building with a domed roof in the background.
(0.8, 0.9) / (1,
(0.9, 0.8) (0, 0)
input expression E: blue car right
Reference text Tref.
On the right side of the image, a dark blue sedan is partially visible, parked alongside the curb in front of what appears to be a restaurant or bar, with only its rear quarter and taillight visible in the frame.
(0.8, 0.9) / (o, 0)
(0.9, 0.8) (0, 1)
(0.9, 0.8) / (1, 1)
(0.8, 0.8) / (1, 1)
input expression E: block shorts
Reference text rref.
The black shorts are worn by the player on the left, who is running forward with his body leaning slightly to his right, the shorts appearing snug-fitting reaching to just above the knee, controsting sharply with his black and white striped jersey and white socks.
(0.9, (.9)
(0.9, 0.9) / (1, 1)
and
(0.8, 0.9) (0,
(0.9, 0.8) 1)
Figure 2. Analysis of MLLM's Rating and Binary Response Performance: For each mask-cropped region, we compare two types of outputs: numerical scores (score I, score 2) and binary responses (dft d(l) (0='no', l='yes'). The results reveal that MLLM struggles to generate meaningful similarity scores when comparing the input expression E and reference text 7 The assigned scores (typically around 0.8-0.9) appear arbitrary rather than reflecting prove more reliable for assessment purposes.
contextual similarities. In contrast, the model's hinary yes/no responses

---

Reference text Trel;
The figure the right a person wearing black hoodie and light blue jeans, stonding with their bock the camera, holding skateboord their right hand while facing grand, ornote building with domed roof in background.
Candidote text yean;
Candidate
Condldate text yaa;
Condidate yian;
The ground region at the bottom image two individua''s silncuette wolking ligh-colored cobblestone poverent
The man the block hoodle and jeons appears focused walkinng his skateboard through grand. open public square
The young womon, corrying iarge quilted courter bog over shoulder, stonds in the middle of bustling the
A large omate building stands majestically the background, its grand facade featuring lorge windows clossicol arehitectuool details.
plaza, needing directions mon beside her.
0.234 0.241
0.487 / 0.266
0.543 / 0.223
0.406 0.269
Reference text gref. On the right side of the image, dark blue sedan is partially visible, parked alongside the curb in front of what appears to be restaurant or bor, with only its rear quarter and taillight visible in the frame.
Condidate Lext yoa
Condidate text you
Conolidoe text pean, The blue white rectongge region, possibly part of the wall on street beside restouront, block sign
Condidote tex pain
portions the image that showing what dopear to be bench legs individuel sitting on it on povereett povement.
unpoinieed section the door of the car appeors to be mode of plastic, contrasting with smooth metollic surface of the rest of the vehicle.
porked
restourant on cooblestone street bustling around it in the foreground, ciear, sleek metalle surface meticulously mirrors the surrounding urbon environment.
reflecting
0.476 / 0.217
0.577 / 0.264
0.413 / 0.213
0.668 / 0.243
Reference text yref;
The black shorts are worn by the player on the left, who is running forword with his body leaning slightly to his right, the shorts oppeoring snug-fitting and reaching to just above the knee, contrasting sharply with his black and white striped jersey and white socks.
Condidate text yoin,
Condidate text pean, The athlete, dressed black and yellow unifarm, is mation the soccer fieid.
Condidate text pean,
Candidote text pran,
Silnouette centrol sorzer plcyer in vibrant jersey driblling the ball across the lush green fleid, with players, closely traching movements on either side.
soccer olayer weoring red jersey with yellow accents and white shorts with on emblem on the right oppears be in motion.
The player the moroon jersey and white shorts with embossed design appears doring dynomic match.
in mid-or
0.455 0.233
0.539 0.313
0.609 0.241
0.456 / 0.246
Figure 3. Analysis of CLIP's Similarity Evaluation: For each mask-cropped region, compare text-to-text (s and text-to-image (d CLIP scores. Text-to-text scores prove more reliable, while text-to-image scores consistently remain below 0.3, showing limited discriminative power. However, relying solely on text-to-text scores misleading, demonstrated in the last row where description containing "white shorts" receives higher score despite incorrectly matching the reference image showing "black shorts". This highlights the limitation of text-to-text evaluation in capturing crucial contextual details.
8

---

candidate text (selected as output)
query
image
reference text
output
the center-right pertion of the image, iarge lignt bive umbretta with o darker biue underside stands out prominently, shading group sected indhiddools and contrasting with the surrounding multicolored umbretios
A targe beach umbreila with two tored blue conopy, featuring darker blue ponel lighter blue outer panels
blue and blue umbrello
The green vegetabie iocoted between the potato and corrats is a targe head of broccoll, lightty clusteved florets forming rounded, textured dome deep forest green color.
The wheaant head of broocoli with deep forest green florets stonds prominenty in the center of a wanden taay surrownded by on ossortment of vegetables, inciuding sweet potatoes and leafy greens.
green color vegetable In betwean potato corrot
The frosted spreeded cupcoke, located one out of four to the right and frant of the others, is cound, pink and white cupcoke with colorfal spriokles, sitting on lower night plastic trey.
frasted spninkled cupcoke one our of four to the right and front of the others
A colocha donut covered in rainbow sprinites, posibioned in the lower right comer of the image, with a distincive round shope and the center.
of block
long slender lonce pointed tip and a waxden post stands prominenty omidet bustling scene, flonked by two stotely and ornetely equipped horge in front of distingusshed building with blue wolls and terracotta roof.
The target in the image is a wooden post, located centrally between two donkeys, with notable rea frome around hose It vertical weatheved with visible sine openings and appeors be for tethering the onimalls
vitrant,
something that the animals are tied to
The pink suit is wor by on older stordinng the right side of the image feoturing o florol pottern and folling just below the knee, adtinng pop of color to the prederinonttly blue and white cofor scherne of the wedding porty gathered the cabtiestone street.
An oider woman webring belge jocket over flarat-octeened pink dress stords in the right of the image against a block bockground with their hands clesped in frant of them.
pink skirt
When somecne reading book or magazine and wants to take o breok they may need specific object to mork their ploce. What item in the picture commonly used for this purpose?
The target region is o white bookmork positioned between the pages hook the right side the image, which stonas out due to its natrew, rectongulor shope and its placeraent marking a specific page
A cotorfol bookmerk lies the right book thied 'Woeeend Sevingg. It smoll rectongutar card featuring some potterns.
The torget region in the image is cievoted dirculor pletforn with railings situated at the tall, cylishiiall tower, designed to be designated aiea where people con obsenve.
The metal grate at the top of the chimney appears rusted and slightly rounded, suggesting hos been exposed to the elements for extended period.
the area where people con wolk
A smoll cylindricat white plastic companent with multiple tu03ed smolt holes olong lengtn positioned at the rignt end of harizantal rad, nolding lamp in place amios on goltery disoioy
The lights that placed dffferent directions ore white, syfindricat, and mounted on the night end white metal red, festuring light down.
the lights that are placed in different directions
simitor lamps
the area that is first impocted when cur is moving forword end croshes
The torget region is the front of the cor, spesifically front bumper with black oir vents and the rounded headlight arec, lozated at the right of image.
white
The front side of a white toy car, featuning to be the front humper in light gray, providing protection the cor.
Figure 4. Examples of query, reference, and candidate text. For each input expression query (column 1), RESAnything generates detailed reference text describing the input expression's 's attributes (column 3). Our grouping and selection algorithm identifies the most relevant segmentation from candidates. Columns 4 and 5 show Anythnnss 's output segmentation and its corresponding candidate text. Key words of attributes in both texts are highlighted in red color.

---

image
mask cropped
bounding box
contour
blur
visual prompts used
candidate text
A man in bright blue shirt and red coo positioned in centre of image, possibly holding comera, with his left ley extended forword he looks ahead intertly in grassy pork setting.
incorrect location and action
mask cropped
person wearing a red person crouched position the grassy field the centre of the image, warching intently towards the right side of the image, likely focusing on a saccer game activity.
a blue t-shirl.
blur
X
incorrect location and action
Two players playing frisbee gomee in fush green grass field with smoll white flowers scatiered shorts white uniform and blue socks
image
X cistracted inrelevant regions
throohouu,, one a bright t-shirt, and the onother is leoping with one leg extended high in the air.
mask cropped
image
grassy field dotted with small white flowers, two game frisbee. One wears orange t-shirt with black shorts, while their opponent, dressed in a white uniform and blue socks, leaps grocefully with one teg extended skyword.
players engage
X distracted by imelevant regions; X affected by the contour overlay color
contour
mask cropped
A person in bright oronge t-shirt, block shorts, and an orange baseboll cop is shown in o crouched, athletic stance, positioned in the left of image.
X affeected by the contour over lay color
contour
bounding box contour
Positioned the left of assumes crouched stance, wearing brrihtt green shirt coired with black shorts and topped with an orange basebali cap.
an athlete
X affecttd by both hounding box color and contour overlay lay
bounding box mask cropped
A person in bright blue t-shirt, block shorts, and a red baseboll cop shown a crouched, athletic stance, positioned the left image agoinst black background.
correct region, color and action
Figure 5. Comparison of different visual prompt combinations for attribute description generation. Top row shows the four basic visual prompts: original image, mask-cropped region, bounding box overlay, contour overlay blur background. Bottom rows demonstrate how different combinations affect MLLM's generated descriptions. Using mask-cropped or blur alone leads to incorrect location and action inference, while combining with original image causes distraction from irrelevant regions. Contour-based approaches (with either mask- cropped or bounding box) suffer from color overlay interference. Our chosen combination of bounding box and mask-cropped achieves the most accurate descriptions by leveraging spatial context while avoiding color interference.
10

---

mask cropped
contour (without overlay)
Two men are playing lively game of ultimote frisbee on grossy field, with their guardian spirits seemingly trying to inspire them to victory. One mon, clod in a blue shirt, block shorts, long socks, and a cap, is trying to catch an incoming white frisbee. To his left, onother player in a white shirt and socks with black stripes is carrying out athletic maneuver, possibly to block the catch intercept the frisbee.
In the lush green field, dynomic game of frisbee unfolds. mon blue shirt. ond block shorts grips the frisbee securely in his left hand, poised and olert. His opponent, dressed in white shirt. ond groy shorts, caught mid-kick, his right foot extending towards the frisbee in attempt to intercept.
beorded man wearing o dark shirt plays light brown guitar while standing next to fireplace adorned with picture fromes. In the living room, several people sit comfortably, with one man in glosses facing the guitarist, attentively listening the music. A television set. is placed near the fireplace, and a bowl of snacks is visible on the floor next to one of the seated individuals.
A baseball player, wearing o blue and white uniform with the number 8 prominently displayed the back, a wooden bat while a black-clad umpire observes attentively from behind, positioned to the right.
person wearing dark loose jeons and brown shoes crosses their legs, with the top one foot resting on the ankle of the other leg, while seated outdoors on grovel.
The woman blue tank top holds plate with food as she converses with a man casual attire near green metol bench in park, with on old brick building in the background.
Figure 6. When dealing with non-convex shapes, analyzing only the contour without considering the overlaid mask region can lead to ambiguous visual interpretations. This ambiguity often results generated text descriptions that contain mislcading information, where incorrectly identified objects are highlighted in red.
11

---

eono.
Figure 7. Images excluded from the ABO dataset typically lack meaningful or referrable parts. Row shows phone related items that primarily consist of phone cases displaying only the back view of phones. Row 2 features images solely showing product textures materials that fill the entire frame. Images from the footwear and picture frames categories in row 3 & are commonly presented against plain white backgrounds without distinct parts for grounding.
12

---

second guy on right
girl in purple
(SNN
29
lady middle pink
catcher on left
back cowboy
the mom
dark brown horse
guy leg
middle horse hidden
the boot the foreground close
to teh camero dont click the guy in the
in middle
third from front on right
middle bird
black car
only banana that is laying other way
white horse
Cow in background
Figure 8. RESAnything can handle occlusion cases by grouping and selection cases. Results from RefCOCO.
13

---

query
GLaMM
LISA
RESAnything
GT
person bottom left
glasses
gray jeans
orea bottom right
right wornon
red bike
red shirt
Figure 9. Qualitative results on RefCOCO test (randomly selected). The ground truth annotations can be problematic some queries refer only object/region while the GT marks entire person (row "glasses": ; row 4: "area bottom right", row 7: "red shirt"). Row 6 shows a failure case of Anythin.
14

---

query
GLaMM
LISA
RESAnything
GT
right ice glass
right horse
cnib
cop on right
middle bird
toxi
no big couch at bottom of screen
Figure 10. Qualitative results RefCOCO B (randomly selected). Despite being unsupervised, RESAnything achieves comparable results to supervised methods, particularly excelling at crowded regions (row 2). However, it occasionally misses parts when needing to combine multiple masks (row 5).
15

---

query
GLaMM
LISA
RESAnything
GT
bowl befindd the others con only part
pan boy
bock of head of person at the battom
top broccalii
lody on right
very bock tie on top
Figure 11. Qualitative results on RefCOCO val (randomly selected). RESAnything generates ane-grained segmentation input
expression (row 1, 4). As mentioned in Fig 10, it may miss parts when combining multiple masks (row 5).
16

---

query
GLaMM
LISA
RESAnything
GT
rebro running to the left in front of other zebras
white truck with a worker standing it, which front of the man who directing traffic
STOP
STOP
STOP
STOP
frosted sprinkled cupcoke, one out of four to the right and front of the others
the backpack with the water bottle in side pocket
o bluck duffle bag sits the chair
the block choir near the glass wine
middle set of skis
Figure 12. Qualitative results on RefCOCOg val (G) (randomly selected). RESAnything generalizes well on mask with hole (row 3, 5), but may suffering from over-eegmentation (row 1, 4) no good candidate found (row 7).
17

---

query
GLaMM
LISA
RESAnything
GT
skoteboording with grey parits and dark grey sshirt
empty toble spoce at right
with block head wrap talking on phone blue car
plant inside the home
black duffie bag sits the chair
smoll fomb mon's right hand
perfectly sliced piece of sandwich sits wropper
Figure 13. Qualitative results RefCOCOg val (U) (randomly selected).
18

---

query
GLaMM
LISA
RESAnything
GT
green color vegetable between ootato and carrot
the right holf of a long sandwich with bonona peppers and green spices on the side
bottle of chardonnay
the wing of an airplane over bus
troffic lights not obove the
a teddy bear wearing purple jocket stonds next beor dress
Figure 14. Qualitative results on RefCOCOg test (U) (randomly selected).
19

---

query
GLaMM
LISA
RESAnything
GT
botter covering logo
dork blue and purple not bikinl
person with blue jeons and black and white shirt
stretching bay
guy in button up shirt
mon with dark shirt
male doing
skateboord trick
Figure 15, Qualitative results on ReiCOCO+ test A (randomly selected).
20

---

query
GLaMM
LISA
RESAnything
GT
truck by fence
animal facing abead
the most visible sultcose of the three
pizzas foided over on half plate the side
dark blob food above bubnles plate rim
cat on book
choir with bird
Figure Qualitative results on RefCOCO+ test B (randomly selected).
21

---

query
GLaMM
LISA
RESAnything
GT
person by glass
Screen mostly white
pink skirt
tall giroffe
blue and blue urvbrellyy
BLASS
BLAIS
BLUS
BLUS
11
11
11
person in white bench
Figure 17. Qualitative results on RefCOCO+ val (randomly selected).
22

---

query
CaR
GLaMM
LISA
RESAnything
GT
something inoicating the brand of this car
something that the animnats ore dee to
In order to iake clear and stable photo or video, what equipment
in the picture can be used to hold the steady?
the object that helps
to keep the worm
neck
Dogs typically alowed
Beety pubie spaces
they awaly need to be kopt
under control wheen watking
indtors What otject keep
pinove could dog restralnd a hallway?
Smatls fiigle wody
iiqulss motecalls protect them What port in petove con accompiis this tost?
the person who is pushing the baby comraag
When it comes to water sports, people often use various tools to gilde on the surface the water, What item the picture is designed specifically for this purpose?
People con play beautiful music by pressing the keys on the piano keyboord, What port of the piano in the picture con be used for playing?
the part of the vehicle that can opened
Figure 18. Qualitative results ReasonSeg (Part 1). RESAnything outperforms others in correct localization (row 2, 3, 8, 9), refined segmentation (row 1, 4, 5, 6, 7) and plrt-level understanding (row 9, 10).
23

---

query
CaR
GLaMM
LISA
RESAnything
GT
the food with high
the for possengers on the dirshjp
OUTBACK
OUTBACK
TTAM
the area that is first impacted when moving forward and croshes
domaged part the siik stockings
When someone reading book or mogazine
take break, they may need specific object to their place. What item the picture commonty used this purpose?
the where people con walk
MISS
the household appliance used for heating food
Figure 19. Qualitative results on ReasonSeg (Part 2). "MISS" indicates that the method is failed to output segmentation.
24

---

query
CaR
GLaMM
LISA
RESAnything
GT
breothe undewatter dxing requres eavimeen
while undersstw What the peture con nelp numen occemmplih
Bocipaois commanyy used cocying oundoer acticties what the beckpeck iamn acsestones?
formel event
golo aoord ceremwyy, accessory the picuue con wore souurd th nerk to oll muct of elegance to man's sat?
In the picture, there a
type of bird with a distinctive feoture on top of head, which asually indicates its gender. Whet port in picture might have this charactevistic?
otter office, employeas work Whet abjer surface analoyess dccuenents devices mestog?
Irish Post
Irish Post
Irish Post
Irish Post
Irish C Post
the musician
No British move for transplant girl's family
No British move for transplant girl's family
No British move for transplant girl's family
No British move for transplant girl's family
No British move for transplant girl's family
the lights that ore placed in different directions
container that being held by person and is about to pour Biuuid
What object this picture is often used to place coffee cup and postry while sttinn??
Figure 20. Qualitative results on ReasonSeg (Part 3).
25

---

query
CaR
GLaMM
LISA
RESAnything
GT
adjustable strops
MISS
365
365
365
365
Dishwasher prppe
Dishwasher
Dishwasher
Dishwasher
20 detergent packs
MISS
LENON SCENT
LEMON SCERT
LEMON SCENT
LEMON SCENT
curved lever handle
adjustable height
scrotching post
Figure 21. Qualitative results on -Bo--nnge-ARES (Part
26

---

query
CaR
GLaMM
LISA
RESAnything
GT
stoir-climbing wheels
side handles
closp closure
high resolution tens
side strap
lorge center diomond
50LB
wooden oval
Figure 22. Qualitative results on ABO-Image-ARES (Part 2). 27

---

query
CaR
GLaMM
LISA
RESAnything
GT
storage shelf
FRONT LOAD
FRONT LOAD
FRONT LOAD
FRONT LOAD
FRONT LOAD
front load
Presto! DETERGENT
o'
Presto! DETERGENT
Presto! DETERGENT
Presto! DETERGENT
Presto! DETERGENT
woll-mounted
365
365
365
365
365
Clam Strips
Clam Strips
Clam Strips
Clam Strips
Clam Strips
served with
sauce
noilhead trim
convoluted foom layer
night light function
post backs
Figure 23. Qualitative results ABO-Image-ARES (Part 3).
28

---

References
referring expres- [1] Jasmine Collins, Shubham Goel, Kenan Deng. Achlesh- Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik. Abo: Dataset and benchmarks for real-world 3d object understand- ing. CVPR, 2022.
[2] Joseph I. Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378, 1971. 4
[3] Yutao Hu, Qixiong Wang. Wenqi Shao, Enze Xie, Zhenguo Li. Jungong Han, and Ping Luo. Beyond one-to-one: Re- thinking the referring image segmentation. In Proceedings of the IREF/CVF International Conference Computer Vi- sion, pages 4067-4077, 2023. 5
[4] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan- Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015-4026. 2023, 4 [5] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579-9589, 2024. 4,5
[6] Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Gen- eralized referring expression segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 23592 23601, 2023. 4.5
[7] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad Khan. Glamm: Pixel grounding large multimodal model. In Pro- ceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13009-13018, 2024. 4,5 [8] Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, and Siyang Li. Clip rnn Segment countless visual concepts without training endeavor. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Puttern Recognition, pages 13171-13182, 2024. 4,5
[9] Wenxuan Wang, Tongtian Yue, Yisi Zhang, Longteng Guo, Xingjian He, Xinlong Wang, and Jing Liu. Unveiling parts beyond objects: Towards finer-granulen
sion segmentation. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pages 12998-13008 2024.
[10] Jianzong Wu, Xiangtai
Henghui Ding, Yunhai Tong, and Dacheng Tao. Towards robust referring image seg- mentation. IEEE Transactions on Image Processing, 2024.
5
29


Reviews:
[{"review_id": "TX0o07ojmu", "reviewer": "Reviewer_T2sg", "summary": "This paper introduces a zero-shot and training-free approach for referring expression segmentation by addressing the challenge of handling part-level and implicit expressions. The method leverages “attribute prompting” with an MLLM to generate detailed text descriptions for the referring expression and image segmentation proposals generated by SAM. The most appropriate segmentation mask is selected by a multi-stage mechanism that combines text-to-text and text-to-image comparisons using both MLLM-based binary decisions and CLIP scores as criteria. Experimental results show that the proposed method outperforms existing zero-shot methods on benchmark datasets and the introduced dataset focusing on part-level expressions.", "questions": "Please refer to the weaknesses section.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "[Strengths]\n1. The paper is overall well-written and easy to follow.\n2. The method is training-free, eliminating the requirement of extensive annotation costs.\n3. Experimental results show the superiority of the proposed method over existing zero-shot approaches.\n\n[Weaknesses]\n1. The technical novelty and contribution of the paper are limited. The method primarily combines existing models like SAM and MLLM. The integration of these components is straightforward, and the major effort seems to be prompt engineering, which does not introduce significant conceptual novelty.\n\n2. Performance of the method highly depends on meticulously engineered prompts. Performance could vary significantly with different MLLMs or slight prompt modifications, limiting robustness and generalizability beyond the tested configurations.\n\n3. The proposal selection mechanism may struggle with targets comprising multiple objects. While the paper mentions combining masks for cases where a single semantic entity spans multiple segments, the overall selection process prioritizes a single mask with the highest CLIP score. Further clarification might be needed on how robustly it handles inherently multi-object targets that are not necessarily parts of a single semantic entity.\n\n4. Although superior to other zero-shot methods on standard RES benchmarks, the proposed method still shows a performance gap compared to recent supervised methods. This suggests that it might not be the top choice for object-level RES tasks where large objects are prevalent.\n\n5. The two-stage approach is computationally inefficient. Runtime of the method is much higher than existing approaches, as reported in the supplementary material.\n\n6. Details of mask filtering are missing. How are duplicate proposals filtered?", "was_revised": true, "final_justification": "I have read other reviewers’ comments and the authors’ rebuttal. While the rebuttal addresses some of my concerns, I still have reservations regarding the novelty and technical contribution of the work.", "rebuttal": "Dear reviewer T2sg, thank you for your thorough review and helpful suggestions! We appreciate you mentioning that our paper is well-written, the proposed training-free method saves the annotation cost for this task, and the experiments show promising results. We address your concerns regarding the identified weaknesses below.\n\n## Novelty, Robustness and Generalizability\nFor most problems covering diverse test cases, the biggest challenge is tackling the “last mile,” and RES is no exception. Our method is designed for, and excels at, tackling such a challenge under complex cases involving implicit queries and complex part-level relations. To achieve this, we propose the novel idea of attributing prompting, with the emphasis on reasoning about part attributes. Combining SAM and MLLM is not our novel claim, attribute prompting is! To further validate its effectiveness, we conducted an ablation to show advantages over conventional prompting: quantitative results on ReasonSeg test set with different prompts highlight the contribution of our attribute prompting mechanism:\n| Method           | gIoU | cIoU |\n|------------------|:----:|:----:|\n| standard prompt  | 50.8 | 49.3 |\n| attribute prompt | **74.6** | **72.5** |\n\nWe show details of standard prompt, attribute prompt, and corresponding generated descriptions in supplementary materials, Section 6.1 and Figure 9. \nIn addition, we also contribute a challenging new dataset whose characteristics deviate significantly from RefCoco, a dataset most prior methods may have overfit towards. Our new dataset is also catered to real-world application scenarios, e.g., eCommerce.\n\nRegarding robustness and generalizability, Table 5 demonstrates consistent performance across diverse MLLM backbones (Pixtral 12B, Claude 3.5, Qwen2-VL), validating cross-model reliability.  While prompt sensitivity exists, this characteristic actually provides valuable flexibility rather than a limitation: users can adapt the attribute specifications within prompts to optimize performance for specific domains or categories when building unlabeled datasets / in applications. In addition, our training-free approach allows seamless integration of newer, more powerful MLLMs as they become available, providing immediate performance improvements without retraining. \n\nFor multi-object scenarios, our selection mechanism employs comprehensive decision-making analyzing both text-text and text-image similarity rather than single CLIP scores (detailed in supplementary Section 6.2). Section 6.2 and Table 6 in the supplementary further show ablation studies on the effectiveness of this multi-metric mechanism to improve robustness. In addition, we extensively evaluated multi-object performance on the GRES task (target containing multiple objects), as shown in the supplementary materials, Section 9.3. The proposed grouping method actually demonstrates promising results among pretrained/zero-shot approaches, empirically confirming robust handling of multi-object scenarios. Figure 16 in the supplementary materials illustrates our capability for grouping and combining multiple parts of the same object. Additionally, our method successfully adapts to different expression granularities. For example, changing the expression from 'lady middle pink' to 'all players' correctly segments both male players in the image, and similarly, segmenting 'two kids' instead of 'the mom' appropriately selects both kids. Our method's flexibility allows for different grouping settings and configurations, and we acknowledge that future work can further optimize the grouping mechanism to enhance multi-object performance.\n\n## Performance gap with supervised methods\nRegarding the performance gap with supervised methods, we acknowledge this limitation on the RefCOCO series, which represents a fundamental and well-established benchmark in the field. We recognize the continued importance of RefCOCO for evaluating core RES capabilities. However, we would like to highlight that our zero-shot approach is specifically designed to excel in scenarios where supervised methods face limitations. In particular, when labeled training data is scarce or unavailable, or dealing with complex referring expressions that extend beyond traditional benchmark coverage. While RefCOCO provides essential evaluation for common referring expressions involving humans and daily objects, our method demonstrates particular strength in handling sophisticated expressions involving explicit attributes, part-level descriptions, materials, and functionality. These areas represent emerging challenges in real-world applications. Notably, our zero-shot approach achieves state-of-the-art performance on ReasonSeg and our proposed ABO-ARES benchmark, demonstrating superior performance on the sophisticated referring expressions that represent genuine research challenges beyond the scope of traditional benchmarks. \nWe believe the value of our contribution lies in providing an effective solution for domains and applications where extensive supervised training data is unavailable, while simultaneously advancing the field's capability to handle the increasingly complex referring expressions encountered in real-world scenarios beyond traditional benchmark settings.\n\n## Efficiency\nOur current inference speed (12.1s) is not significantly behind the fastest alternative CaR (about 5.3s), which suffers from low accuracy (35.2% vs. ours 74.6% on ReasonSeg). There is an obvious tradeoff between speed and accuracy. A gap of 6.8s is not significant enough to warrant using a more error-prone method. Ultimately, the errors must be corrected in an application and the time it would take to correct a segmentation mask most likely far exceeds 6.8s on average.\n\nFinally, our current method has not been optimized for inference speed. With parallelization on candidate mask generation from SAM, reference text generation, candidate text generation and similarity comparison between the target and candidates, the RESAnything time can be reduced to $\\approx$3s (on multiple 4090 GPUs), as shown below: \nmax(SAM, reference text generation) ($\\approx$2s) + candidate text generation ($\\approx$0.5s) + max(MLLM similarity computation, CLIP similarity computation) ($\\approx$0.5s).\n\nThis optimization leverages the fact that candidate mask generation and reference text generation can be executed in parallel, and similarity computations using MLLM and CLIP can also run concurrently. Furthermore, inference speed could be improved even more by replacing SAM with FastSAM for mask generation.\n## Others\nRegarding the mask filtering, we set a minimum mask region area threshold of 0.4% of the total image area to remove tiny boundary masks commonly produced by SAM. In addition, we apply a prediction confidence threshold of 0.9 to filter out low-confidence duplicate part segmentation masks, as SAM typically generates masks for both whole objects and their constituent parts with varying confidence levels."}, {"review_id": "ZZZJ3BuT1N", "reviewer": "Reviewer_DVej", "summary": "This paper proposes RESAnything, a training-free, open-vocabulary framework for zero-shot referring expression segmentation (RES). Unlike existing RES methods that focus on object-level semantic labels, RESAnything handles arbitrary referring expressions—including part-level, functional, stylistic, and even graphical (e.g., logos)—by leveraging multi-modal large language models (MLLMs) and Segment Anything Model (SAM). The method introduces attribute prompting to describe both the target expression and each mask proposal, and evaluates their alignment using a multi-metric framework involving MLLMs and CLIP. Additionally, the paper contributes a new benchmark dataset, ABO-Image-ARES, focusing on complex, attribute-rich expressions.", "questions": "* How does RESAnything handle cases where multiple objects share similar attributes (e.g., “the black chair” when there are two black chairs)? Have the authors considered evaluating the method’s generalization to embodied references involving human pose? For example, in Understanding Embodied Reference with Touch-Line Transformer (ICLR 2023)", "limitations": "* As acknowledged by the authors (Line 308), the method lacks an explicit mechanism to handle varying object scales. This results in performance degradation on large-object segmentation tasks.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "S:\n\n* The method requires no fine-tuning and works zero-shot, making it broadly applicable across domains.\n* RESAnything supports a wide spectrum of expressions, including implicit references and attributes like material, style, function, etc.\n* The ABO-Image-ARES dataset is a valuable contribution, providing a resource for evaluating reasoning-intensive and part-level RES tasks.\n* The method achieves state-of-the-art zero-shot performance on multiple datasets.\n\nW:\n\n* The method is largely a system-level integration of existing tools (SAM, CLIP, MLLMs), with the main innovation being the design of prompts and evaluation metrics.\n* While the title claims to cover \"anything,\" the method does not address affordance-based or action-grounded queries. See TOIST: Task-oriented Instance Segmentation Transformer, NeurIPS 2022.\n* The pipeline includes multiple calls to SAM and MLLMs for proposal generation, attribute description, and evaluation. This results in high computational cost and limits applicability in real-time or large-scale deployment settings.", "was_revised": true, "final_justification": "For a foundation-model-style system, empirical results are the most important indicator of value, and the new SOTA performance on affordance-based reference tasks convincingly demonstrates the practical utility of this work. I remain unconvinced that the prompting strategy itself is a core scientific contribution, but this is less critical in the current NLP landscape where such design is often the norm. Overall, the strong results justify acceptance.", "rebuttal": "Dear reviewer DVej, thank you for your detailed review and insightful questions! We appreciate your recognition that the proposed zero-shot method is broadly applicable across different domains with support for a wide spectrum of expressions. We are also encouraged by your positive assessment of our proposed benchmark and the outstanding performance demonstrated by our method. Regarding the weaknesses and questions, please find our responses below.\n\n## Novelty\nWhile our method integrates existing foundation models, the key contribution lies in the novel attribute prompting framework that enables systematic reasoning about part-level attributes and implicit queries—capabilities that current methods cannot achieve. Our novelty is not prompt engineering, but the systematic multi-stage reasoning framework that decomposes complex spatial and semantic relationships into interpretable steps, addressing the \"last mile\" challenge in RES.\nTo validate the effectiveness of attribute prompting, we conducted ablation studies showing significant advantages over conventional prompting on ReasonSeg:\n| Method           | gIoU | cIoU |\n|------------------|:----:|:----:|\n| standard prompt  | 50.8 | 49.3 |\n| attribute prompt | **74.6** | **72.5** |\n\nWe show details of standard prompt, attribute prompt, and corresponding generated descriptions in supplementary materials, Section 6.1 and Figure 9. \nIn addition, we also contribute a challenging new dataset whose characteristics deviate significantly from RefCoco, a dataset most prior methods may have overfit towards. Our new dataset is also catered to real-world application scenarios, e.g., eCommerce.\n\n## Affordance-based or action-grounded expression involving human pose\nAlthough we did not explicitly evaluate on any affordance-based or action-grounded benchmarks such as TOIST and the mentioned ICLR paper, both ReasonSeg and our proposed ABO-ARES benchmark contain substantial affordance-based and action-grounded queries that demonstrate our method's capability in this domain. Specifically, ReasonSeg includes expressions such as \"people cross over obstacles,\" \"object used by people to sit down,\" \"where people wash their hands,\" and \"people catch,\" which directly involve human-object interactions and spatial affordances. Similarly, ABO-ARES contains functionality-oriented queries like \"easy pouring,\" (in Figure 8) \"foldable design,\" and \"cushion support\" that explicitly reference object affordances and their intended human interactions. The superior performance of our method on both datasets demonstrates effective handling of these scenarios, validating our approach's applicability to affordance-based reasoning. Our training-free framework can be naturally extended to affordance benchmarks by incorporating embodied attributes such as graspability, stability or manipulation properties into the attribute prompting mechanism, allowing seamless adaptation to action-grounded scenarios without further fine-tuning. We will include these related papers in the final version of the paper. \n\n## Efficiency and cost\nOur current inference speed (12.1s) is not significantly behind the fastest alternative CaR (about 5.3s), which suffers from low accuracy (35.2% vs. ours 74.6% on ReasonSeg). There is an obvious tradeoff between speed and accuracy. A gap of 6.8s is not significant enough to warrant using a more error-prone method. Ultimately, the errors must be corrected in an application and the time it would take to correct a segmentation mask most likely far exceeds 6.8s on average.\n\nFinally, our current method has not been optimized for inference speed. With parallelization on candidate mask generation from SAM, reference text generation, candidate text generation and similarity comparison between the target and candidates, the RESAnything time can be reduced to 3s (on multiple 4090 GPUs), as shown below: \nmax(SAM, reference text generation) ($\\approx$2s) + candidate text generation ($\\approx$0.5s) + max(MLLM similarity computation, CLIP similarity computation) ($\\approx$0.5s)\n\nThis optimization leverages the fact that candidate mask generation and reference text generation can be executed in parallel, and similarity computations using MLLM and CLIP can also run concurrently. Furthermore, inference speed could be improved even more by replacing SAM with FastSAM for mask generation.\n\n## Handling special queries\n“The black chair” is a good example. This actually reflects a general misunderstanding of referring expressions in this task. In linguistics, a referring expression is any noun phrase whose function in discourse is to identify some individual object. When an expression like \"the black chair\" is ambiguous and refers to multiple valid targets, both objects can legitimately be segmented, as the ambiguity lies in the expression itself rather than representing a failure case. However, for referring expressions that intentionally target multiple objects or parts (e.g., \"standing people\", \"all people\", or “all persons except the kid in white”), our proposed grouping and selection mechanism effectively handles these scenarios. We demonstrate this capability through evaluation on the GRES benchmark containing such expressions in our supplementary materials Section 9.3. which specifically includes such multi-target cases and validates our method's robust performance on these linguistically complex expressions. Figure 16 in the supplementary materials shows several multi-part target examples and specifically, if we change the expression from 'lady middle pink' to 'all players' correctly segments both male players in the image, and similarly, segment 'two kids' instead of 'the mom' appropriately selects both kids. Our method's flexibility allows for different grouping settings and configurations, and we acknowledge that future work can further optimize the grouping mechanism to enhance multi-object performance."}, {"review_id": "Qb0itiidEJ", "reviewer": "Reviewer_R2q5", "summary": "This paper proposes RESAnything, a novel open-vocabulary zero-shot method for arbitrary referring expression segmentation, which ingeniously integrates MLLMs, CLIP, and SAM. By generating detailed object/part attribute descriptions via MLLMs and leveraging mask proposals from SAM, RESAnything enables training-free identification of referring expressions encompassing object/part labels, implicit attributes, and logos, achieving remarkable performance. The paper also introduces a new benchmark dataset, ABO-Image-ARES, which bridges a critical gap in RES benchmarks by offering challenging expressions with implicit attribute contexts.", "questions": "As mentioned above, several concerns have been raised in the Weaknesses section.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper ingeniously integrates MLLM, CLIP, and SAM, enabling full exploitation of each component’s zero-shot capabilities for the RES task. The approach achieves performance comparable to existing task-specific models while requiring no training.\n2.The newly introduced benchmark dataset, ABO-Image-ARES, addresses a critical gap in RES benchmarks by offering challenging expressions with implicit attribute contexts.\n3.The proposed Attribute Prompting approach ingeniously repurposes MLLMs’ captioning capabilities for grounding tasks, offering a novel paradigm for extending their applicability to broader task domains.\n\nWeaknesses:\n1.Although the performance improvements are significant, direct comparison between RESAnything with fully training-free methods (e.g., GLCLIP) may appear inequitable. Unlike GLCLIP’s SAM+CLIP pipeline, RESAnything incorporates an MLLM component that has been pre-trained on Referring Expression Comprehension (REC)—a twin task of RES—endowing it with strong REC capabilities.\n2.Accordingly, MLLMs such as QWEN2.5-VL have demonstrated remarkable performance in grounding tasks on the RefCOCO (+/g) dataset. Given that RES primarily involves segmentation within detected bounding boxes, the interplay between REC capabilities and RES performance warrants further investigation to validate RESAnything’s effectiveness.\n3.Considering that Attribute Prompting relies more heavily on the MLLM's understanding capability, it seems necessary to perform more cross-model comparison using MLLMs with varying levels of understanding and grounding performance, in order to illustrate the extent to which RESAnything depends on the MLLM’s pre-trained knowledge.\n4. Regarding the paper’s emphasis on \"Chain-of-Thought\" (CoT), the corresponding implementation is absent. Neither the algorithm flow nor attribute prompting demonstrates explicit application of CoT for MLLM reasoning.\n5.As a core contribution, the authors claim attributes are pivotal to attribute prompts, yet no ablation experiments on different attribute types are provided.\n6. Minor issues:\n-The set of fully-supervised baselines should include more recent methods, as ReLA (GRES) is somewhat outdated.\n-Algorithm 1 may have an indentation error: the condition for checking the total number of elements in set C should be positioned outside rather than inside the loop.", "was_revised": true, "final_justification": "The authors have provided a detailed reply to my concerns and questions. I raise my rating to 4.", "rebuttal": "Dear reviewer R2q5, thank you for your positive feedback and recognition of our ingenious integration of different tools to achieve training-free performance comparable to task-specific models. We appreciate your acknowledgment of our ABO-Image-ARES benchmark and novel attribute prompting paradigm that extends MLLMs' applicability to broader grounding tasks. We address your concerns regarding the identified weaknesses below.\n\n## REC capabilities of current MLLM and RESAnything\nWe appreciate the reviewer's thoughtful observation about the relationship between REC and RES capabilities. While we acknowledge that some MLLMs like Qwen2.5-VL demonstrate strong performance on RefCOCO datasets, we respectfully note several important considerations. First, RefCOCO series, being published 9 years ago, represents a relatively saturated benchmark where both supervised methods and foundation models have been extensively optimized, making strong performance on this dataset less indicative of genuine REC capabilities for novel, complex expressions. As demonstrated in Section 6.1 and Figure 9 of our supplementary materials, when MLLMs are directly asked to provide comprehensive descriptions of target queries in images, they generally fail to give accurate responses, suggesting that they may not have robust REC understanding. \nMore importantly, our method addresses precisely that gap that existing approaches, including well-performing MLLs, cannot handle complex referring expressions involving reasoning and explicit attributes (See the quantitative result comparisons on ReasonSeg and ABO-ARES). These scenarios are absent from traditional benchmarks like RefCOCO series. While the interplay between REC and RES capabilities is indeed worthy of investigation, our focus is on extending both capabilities to challenging scenarios that represent genuine research frontiers rather than optimizing performance on well-established benchmarks. We believe this distinction is crucial for advancing the field toward handling real-world complexity in referring expression tasks. \n\n## Ablation on MLLMs and attributes\nWe have indeed conducted comprehensive ablation studies across MLLMs with varying capabilities and scales, as shown in Table 5 of our paper. Our evaluation includes Claude 3.5 Sonnet (a closed-source commercial model with strong reasoning capabilities), Pixtral 12B (a relatively large open-source model), and Qwen2-VL 7B (a general-scale model), representing a diverse spectrum of MLLM understanding and grounding performance levels.\n\nTo validate the effectiveness of attribute prompting, we conducted ablation studies showing significant advantages over conventional prompting on ReasonSeg:\n| Method           | gIoU | cIoU |\n|------------------|:----:|:----:|\n| standard prompt  | 50.8 | 49.3 |\n| attribute prompt | **74.6** | **72.5** |\n\nWe show details of standard prompt, attribute prompt, and corresponding generated descriptions in supplementary materials, Section 6.1 and Figure 9. Both quantitative and qualitative results highlight the contribution of our attribute prompting mechanism.\n\n## Usage of CoT\nWe appreciate the clarification regarding Chain-of-Thought implementation. To better contextualize our approach, RESAnything employs a step-by-step decomposition strategy that breaks down complex referring expression segmentation into manageable stages: (1) attribute decomposition of the referring expression, (2) mask proposal generation, (3) multi-modal similarity assessment, and (4) optimal selection and grouping. This systematic approach transforms the complex task into interpretable intermediate steps, enabling our method to handle sophisticated referring expressions that challenge end-to-end approaches. Our attribute prompting guides the MLLM through structured analysis at each stage, facilitating robust understanding of spatial and semantic relationships before reaching the final segmentation decision. We are happy to reword this if the reviewer find that this is confusing.\n\n## Others\nWe will update the quantitative comparison with the latest model in the camera ready version, and correct any formatting errors."}, {"review_id": "brxSjT7rjt", "reviewer": "Reviewer_tcpr", "summary": "The paper presents RESAnything, a training-free framework for arbitrary referring-expression segmentation. It first generates SAM mask proposals, then uses a multimodal LLM to write a rich “reference” description for the query and “candidate” descriptions for each proposal. Four cross-modal similarity signals—two from the LLM and two from CLIP—are combined to select or fuse the best mask. Without any fine-tuning, the method more than doubles prior zero-shot accuracy on RefCOCO and outperforms state-of-the-art fine-tuned models on ReasonSeg and the new ABO-Image-ARES benchmark. The paper also contributes that benchmark and detailed ablations of prompts and backbones.", "questions": "Please refer to Weaknesses.", "limitations": "Yes. The authors discuss these limitations fairly thoroughly.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The approach is training-free yet delivers large zero-shot performance gains, demonstrating strong generalization. Its strong performance on part-level RES makes it useful in many downstream tasks and research.\n- It shows that careful prompt engineering plus proposal recycling can close most of the zero-shot gap to supervised models.\n- It introduces a new benchmark for evaluating part-level RES.\n- The experiments are comprehensive, and the paper presentation is clean and organized.\n\n**Weaknesses**\n\n- The method design is systematic and well-thought-out, but it is not elegant for the following reasons:\n    1. Its performance relies on multiple underlying foundation models and, most importantly, on good segmentation proposals from SAM. Any change in prediction performance, sensitivity, or granularity of the underlying foundation models will render the proposed method either insufficient (bad proposals) or redundant (when foundational MLLMs get much better).\n    2. This method relies on handcrafted text prompts (as shown in Supp. Sec. 7.1), which are also potentially sensitive to weight updates or swapping of the MLLM model.\n    3. The inference runtime overhead is substantial (as reported in Table 11 in Supp. Sec. 9.4). \n- It would be super interesting to see these directions in future or modified work:\n    1. How to generate new proposals when current proposals are not good enough.\n    2. Investigating the upper bound and bottlenecks of training-free methods by conducting a comprehensive qualitative study on samples that are predicted correctly by training-based methods but incorrectly by your method.", "was_revised": true, "final_justification": "Thanks for the response, I have also read other reviewers comments & author's rebuttal, I am happy to keep my score. I hope the author will integrate feedback from all reviewers into their final manuscript.", "rebuttal": "Dear reviewer tcpr, thank you for your encouraging feedback and recognition of our training-free approach's substantial zero-shot performance gains, strong generalization capabilities, and comprehensive experimental validation. We particularly appreciate your thorough review of our supplementary materials. Please find our responses to the identified weaknesses below. \n\n## Robustness and usage of foundation models\nWe acknowledge these limitations regarding foundation model dependency, prompt sensitivity, and runtime overhead. Our ablation studies demonstrate robust performance across diverse MLLMs (Pixtral 12B, Claude 3.5, and Qwen 2) using consistent prompts, and we provide comprehensive prompt analysis in the supplementary materials to verify the effectiveness of attribute-based prompting. More importantly, our focus is on solving referring expression challenges that previous methods fundamentally cannot handle. While current powerful MLLMs have extensively learned concepts from standard benchmarks like RefCOCO series through their training data, the critical research challenge lies in handling specialized, domain-specific and explicit, part-level queries in ReasonSeg and proposed ABO-ARES benchmarks. We understand that future foundational MLLMs may eventually be able to handle these queries through scaling, our work offers a viable pathway to obtain such annotated data for training and explores the possibility of learning rich properties, attributes, and other sparse information about target regions of queries in images without requiring extensive fine-tuning. \n\nOur current method has not been optimized for inference speed. With parallelization on candidate mask generation from SAM, reference text generation, candidate text generation and similarity comparison between the target and candidates, the RESAnything time can be reduced to 3s (on multiple 4090 GPUs), as shown below: \nmax(SAM, reference text generation) ($\\approx$2s) + candidate text generation ($\\approx$0.5s) + max(MLLM similarity computation, CLIP similarity computation) ($\\approx$0.5s)\n\nThis optimization leverages the fact that candidate mask generation and reference text generation can be executed in parallel, and similarity computations using MLLM and CLIP can also run concurrently. Furthermore, inference speed could be improved even more by replacing SAM with FastSAM for mask generation.\n\n\n## Future works\nThank you for these insightful suggestions for future research directions. We agree that investigating proposal generation mechanisms when current SAM proposals are insufficient represents a valuable extension, as this directly addresses a key bottleneck in our approach. Our grouping and selection algorithm handles targets consisting of multiple proposals, though we acknowledge this mechanism is not always reliable. As shown in Section 10 of our supplementary materials, we included failure cases that are primarily attributed to poor mask proposals from SAM. This analysis confirms that proposal quality represents the primary bottleneck of our method, making your suggested research direction particularly relevant for advancing training-free referring expression segmentation."}]

Author rebuttal:

Meta-review: This paper proposes RESAnything, a zero-shot and open-vocabulary framework for arbitrary referring expression segmentation (RES). Unlike prior work that primarily addresses object-level queries, the method extends to part-level labels and implicit references such as function, design, style, or material. The key idea is to employ Chain-of-Thought (CoT) reasoning with attribute prompting, where an LLM generates detailed attribute descriptions to refine segment proposals from a foundational segmentation model.

The paper is clearly written and addresses an important problem, pushing RES beyond conventional settings. Results show state-of-the-art performance among zero-shot methods and strong improvements on challenging cases involving implicit queries and complex part-level relations. In addition, the authors release a new benchmark dataset, which is a meaningful contribution for future research.

Nonetheless, reviewers may question the degree of technical novelty, as the method relies heavily on pretrained LLMs and segmentation backbones, with the primary contribution being the prompting strategy. The robustness of attribute prompting also remains uncertain, as performance could be sensitive to prompt design or small variations.

Overall, the strong empirical results, the broader problem formulation, and the dataset contribution outweigh the concerns about novelty. Therefore, the AC recommends Accept.
Decision: Accept (poster)