Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation
Authors: Jian Hu
Jiayi Lin
Junchi Yan
Shaogang Gong
Venue: NeurIPS 2024 poster (2024)
Keywords: Camouflaged Object Detection; Transfer Learning; Test-time Domain Adaptation; Manual-free Promptable Segmentation; Unsupervised Learning
Primary area: machine_vision
TL;DR: We propose using hallucinations as prior knowledge to extract and validate task-related information, which helps generate instance-specific prompts for reducing reliance on manual prompts in promptable segmentation.

Abstract: Promptable segmentation typically requires instance-specific manual prompts to guide the segmentation of each desired object. To minimize such a need, task-generic promptable segmentation has been introduced, which employs a single task-generic prompt to segment various images of different objects in the same task. Current methods use Multimodal Large Language Models (MLLMs) to reason detailed instance-specific prompts from a task-generic prompt for improving segmentation accuracy. The effectiveness of this segmentation heavily depends on the precision of these derived prompts. However, MLLMs often suffer hallucinations during reasoning, resulting in inaccurate prompting.  While existing methods focus on eliminating hallucinations to improve a model, we argue that MLLM hallucinations can reveal valuable contextual insights when leveraged correctly, as they represent pre-trained large-scale knowledge beyond individual images. In this paper, we first utilize hallucinations to mine task-related information from images and verify its accuracy to enhance precision of the generated prompts.  Specifically, we introduce an iterative \textbf{Pro}mpt-\textbf{Ma}sk \textbf{C}ycle generation framework (ProMaC) with a prompt generator and a mask generator.  The prompt generator uses a multi-scale chain of thought prompting, initially leveraging hallucinations to extract extended contextual prompts on a test image. These hallucinations are then minimized to formulate precise instance-specific prompts, directing the mask generator to produce masks that are consistent with task semantics by mask semantic alignment. Iteratively the generated masks induce the prompt generator to focus more on task-relevant image areas and reduce irrelevant hallucinations, resulting jointly in better prompts and masks. Experiments on 5 benchmarks demonstrate the effectiveness of ProMaC. Code is in https://lwpyh.github.io/ProMaC/.

Leveraging Hallucinations to Reduce Manual Prompt
Dependency in Promptable Segmentation
Jian Hu', Jiayi Lin', Junchi Yan', Shaogang Gong' 'School of Electronic Engineering and Computer Science, Queen Mary University of London
Deept. of CSE & School of Al & Moe Key Lab of AI, Shanghai Jiao Tong University [jian.hu, jiayi.lin, s.gong) @qmul.ac.ukk https///wwyyhhghhhhboo/Pooooo
Abstract
Promptable segmentation typically requires instance-specific manual prompts guide the segmentation of cach desired object. To minimize such a need, task- generic promptable segmentation has been introduced, which employs a single task-generic prompt to segment various images of different objects in the same task. Current methods use Multimodal Large Language Models (MLLMs) to rea- son detailed instance-specific prompts from task-generic prompt for improving segmentation accuracy. The effectiveness of this segmentation heavily depends on the precision of these derived prompts. However, MLLMs often suffer hallucina- tions during reasoning, resulting in inaccurate prompting. While existing methods focus eliminating hallucinations to improve model, argue that MLLM hallucinations can reveal valuable contextual insights when leveraged correctly, as they represent pre-trained large-scale knowledge beyond individual images. In this paper, we utilize hallucinations to mine task-related information from images and verify its accuracy for enhancing precision of the generated prompts. Specifically, we introduce an iterative Prompt-Mask Cycle generation framework (ProMaC) with a prompt generator and mask generator. The prompt generator uses a multi- scale chain of thought prompting, initially exploring hallucinations for extracting extended contextual knowledge a test image. These hallucinations are then reduced to formulate precise instance-specift prompts, directing the mask gener- ator to produce masks that are consistent with task semantics by mask semantic alignment. The generated masks iteratively induce the prompt generator to focus more on task-relevant image areas and reduce irrelevant hallucinations, resulting jointly in better prompts and masks. Experiments on 5 benchmarks demonstrate the effectiveness of ProMaC. Code given in https:/lwyyh.githbio//oooor it..oMaaC..///M/C/....
Introduction
Current promptable segmentation methods rely on instance-specifie manual prompts guide segmen- tation, greatly limiting its large-scale application. Recently, a manual-free task-generic promptable segmentation approach was introduced (2I): only single task- mask-generic prompt is needed for all samples under the same task, e.g., "camouflaged animal" is a fusk-generie prompt for all images in a camouflaged object detection task. The model segments task-relevant objects various images based on this generic prompt, significantly reducing the annotation workload.
This work was supported by Veritone, Adohe, OpenAI, the Apocrita HPC facility from the QMUL Research- IT, the China Scholarship Council, NSFC (92370201, 62222607) and Shanghai Municipal Science and Technol- ogy Major Project under Grant 2021SH7DDZX000.. Thanks to Weitong Cai for helpful discussion.
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

untpul prodiction:
lien
Insaani
the
Vorel Conmunte
calerpilaa
cnne of cenoollllgg antmal 181e imag'
Lisn, Deer, Lecpant.
Indage
image
Image
al sodgetees
spin
xpdn
trkoge
(a) Hallucination by co-necurrence prior.
(b) Using hallucinations can henefit accuracte prompt generation.
Figure I: (a) During MLLM pretraining, leopards often co-oceur with grass. the lion is masked, the model incorrectly identifies it as a leopard based on the grass. (b) Directly inputting the image into MILM causes the hidden caterpillar being incorrectly predicted as a leaf. Splitting the image results in interested objects being incomplete or absent, prompting MLLM to induce hallucinations and utilize prior knowledge to predict potential task-related objects within the image. Our visual contrastive reasoning then eliminates the hallucinations and validates the gathered predictions, aiding in the accurate identification of the caterpillar.
A task-generic prompt is both coarse and potentially ambiguous, can result in poor segmentation when directly applied. To address this problem, existing methods [21]8] utilize the prior knowledge embedded in Multimodal Large Language Models (MLLMs) to infer more detailed, instance-specific prompts, such as bounding boxes or keywords, to guide the segmentation. However, these MLLMS
often generate hallucinations due to object co-occurrence priors 668 655 mistakenly predicting non non-existent elements based on the environment as instance-specific prompts (Fig. (a)). This can mislead segmentation and degrade model performance. While it is common to consider MLLM's hallucinations as detrimental and should be eradicated [52]. this phenomenon actually demonstrates a MLLM's significant capacity for contextual inference based prior training. We want to explore MLLM hallucinations as a valuable untapped knowledge resource for scene understanding, critical in complex segmentation scenarios. In practice, when task-related objects are not prominently visible, hallucinations can fill in missing information with plausible predictions based on learned patterns of association. Moreover, they can also extend beyond these familiar patterns, exploring and identifying new relationships within the data that were not explicitly taught during training. This dual ability to replicate and innovate makes hallucinations valuable asset for enhancing model performance in complex or new situations. This predictive reasoning capacity not only fills perceptual gaps but also enriches the model's understanding, as hallucinations utilize prior knowledge to replicate and discover new patterns, enhancing insight into the target domain (see Fig. ((b).. Despite the potential benefits, using hallucinations extract useful information from images to aid task remains unexplored.
In this work, instead of direct eliminating hallucinations, we utilize them as prior knowledge to mine extended task-related information from given test image, performing scene understanding on the image before segmentation, then systematically reduce irrelevant hallucinations iteratively by visual masking verification, optimizing jointly instance-specific prompts and masks. To this end, we introduce an iterative, training-free Prompt-Mask Cycle Generation method (ProMaC) that refines segmentation through cyclic interactions between a prompt and mask generator (see Fig. 22. The prompt generator uses a multi-scale chain-of-thought prompting mechanism, which utilizes hallucinations to hypothesize and visual masking to verify, thereby creating more accurate -nstance-specific prompts. We trigger the hallucinatory tendencies of MLLMs, the process starts by dividing the image into patches at different scales and positions. Such partial visibilities of objects facilitate MLLMs to hypothesize potential object semantic labels and visual locations based on its prior knowledge. For validating the correctness of these hypotheses, we formulate a visual contrastive reasoning mechanism to generate contrastive images that contain only the background without any potential task-related objects. This helps identify all possible co-occurrence hallucinations caused by the background. By comparing these contrastive images with the original images, the MI.IM effectively distinguishes between accurate hypotheses and those influenced by misleading prior knowledge, leading to more reliable prompts. Given the current promptable segmentation models' strength at mask prediction but struggle with label prediction, the mask generator uses mask semantic alignment to ensure that the produced masks align with the task semantics. These aligned masks not only serve as outputs but also guide the prompt generator in subsequent cycles, enhancing both prompt and mask quality continuously. Our contributions are three-folds:
1). We introduce a training-free Prompt-Mask Cycle Generation (ProMaC) to perform two tasks: Explore MLLM hallucinations as prior knowledge to enhance contextual scene understanding on each test image; systematically reduce irrelevant hallucinations to verify iteratively and optimize jointly both generated prompts and visual masking in object segmentation.

---

initial instance- specife prompt
mask generation
predicted mask
initial mask
prompt generation
predicted Instance- specifc prompt
rwweshoppes
spider
prompt
mask generation
generation
Improved Instance- specifc prompt
improved mask
Multi-scale Chain of Thought Prompting
Mask Semantic Alignment (2) Mask-to-Prompt Cycle Generation
(1) Prompt-to-Mask Cycle Generation
Figure 2: An overview of ProMac: Masks created iteratively by the mask generator guide the prompt
generator jointly improve instance-specific prompts and visual masking in segmentation.
2). We formulate an iterative optimization method including a prompt generator and mask generator To improve prompt relevance, the prompt generator utilizes a multi-scale chain of thought approach. first leverages hallucinations to expand task-related plausible prompts, then applies visual contrastive reasoning to validate and reduce irrelevant prompts. ProMaC's mask generator overcomes SAM's shortcomings in label prediction by creating masks that align semantically with generated prompts.
3). Comprehensive comparative evaluations on 5 different segmentation tasks with 12 diverse datasets against 22 existing models demonstrate the effectiveness of ProMaC.
2 Related Works
Promptable Segmentation refers to object segmentation with active interactions from user inputs. Interaction methods vary from points, boxes, to scribbles. SAM [29]. AV1 Ground- ingSAM (38) and SEEM [71] accept video, audio, and multimodal inputs. However, they often rely on manual prompts, which can be unclear and subjective. Even with these prompts, they typically excel only in specific tasks. To address this issue, GenSAM [2]] introduces a nanual-frre promptable segmentation setting, where only task-generic prompt is provided. This prompt can be applied to all images within the task for instance-specife segmentation without any additional manual prompting.
GenSAM primarily utilizes MLLM to infer the names of task-related objects in the images and uses them as instance-spectin prompts for SAM to guide segmentation. However, GenSAM lacks spatial information about objects and may lead inaccurate prompt predictions in complex scenes.
Hallucinations in MILMs refers to models generate content that does not exist in the input data (65) This issue often arises from the models leveraging extensive prior training rather than just the immediate input, leading to false predictions on fine teee-graied details. There are some efforts to mitigate this problem, including refining training processes [55] 44 and improving model architectures [4.
Other efforts focus on aligning model outputs more closely with actual data, employing feedback mechanisms for real-time adjustments [54]. While current works focus eliminating hallucinations to enhance performance [31] 64], our work explores how to utilize hallucinations to expand and reason plausible context and validate them iteratively to remove irrelevant generalizations.
Visual Marking for MILMS has been explored in recent research to prompt MILLMs through manipulation of visual inputs: (i) adding learnable soft tokens to visual inputs for efficient parameter tuning 28). (ii) using image sequences as demonstrations of a new task 12 99, and (iii) overlaying
visual markers like masks, boxes, and circles onto visual inputs to ground regions [6] 54) Our
work falls into the third category, employing visual guidance for reasoning. Yang et al. (59] propose set-of-mark (SoM) prompts, where images are segmented and numbered regions to improve GPT-4V 43 visual grounding. However, as detailed in Tab we confirm previous findings (5] that this visual marker approach struggles with open-source MLLMs like LLaVA. Instead of proprietary models 54 or fine-tuning [5] 8, 23), our training-fiee free ProMaC uses inpainting task-related regions and contrasting model output distributions prompt MILMs.
3 Methodology
We introduce ProMaC, cycle-generation method for segmenting unknown multiple classes of objects training-free with only a single task lask-generic prompt. Specifically, given an image X RR. #xxx from a test set, ProMaC employs a task-generic prompt P, across datasets in the same task to produce
3

---

Multi-scale Chain Thought Prompting
seahorse
0.95
MILM
Beahorse sea
A
0.73
VCR
10.3, 0.2. B 0.68, 0.76
Seg
CLIP
MLLM
snake
grass
0.18 X
A
Prompt Generator
Mask Generator
MLLM
inpainting
anaki crab seahorse
MLLM
Visual Contrastive Reasoning (VCR)
snakecrab wninhorae
Figure 3: ProMaC consists of prompt generator and mask generator for cyclical optimization. The prompt generator employs multi-scale chain-of-thought prompting. initially use hallucinations for exploring task-related information within image patches. It identifies task-relevant objects and their backgrounds (Aface Abus)) along with their locations (B*)). Subsequently, uses visual contrastive reasoning refine and finalize instance-specific prompts (AF, B,¹) by eliminating hallucinations. The mask generator then processes these prompts into the segmentation model ("Seg") ") producing mask aligned with task semantics. This mask further guides the visual contrastive reasoning process, which leverages an inpainting model to eliminate masked regions, creating contrastive images. These images enable the prompt generator further refine its prompts, enhancing segmentation accuracy.
a final segmentation mask M E Hx thereby removing the need for individual supervision for each image. The prompt generator leverages prior knowledge gained to reason and deduce instance- specific prompts, which then guide a mask generator to create masks aligned with task semantics. These masks act both the current segmentation outcome and as visual markers for generating subsequent prompts. Training-free ProMaC relies solely on test-time adaptation.
3.1 Prompt Generator
Prompt generator employs MLLMs to generate instance-specift prompts based image content and prior knowledge. transforms the general prompt Pg. into an instance-specific prompt for each individual instance, providing parameters 9 receives an image X and query P as inputs. provides contextual visual information to assist the model in generating relevant response y to the query P. The response y sampled auto-regressively from the probability distribution conditioned on P and X as follows:
detailed descriptions of task-relevant objects. MLLM with
Ve (o( Pct)) x exp(logit, X, P. P.ye)))
where ys denotes the token at time step f, and y < t represents the sequence of generated tokens up to the time step (t 1). In practice, predicted task-relevant objects can often blend into the background due to texture, color, size, or position, leading to inaccuracies in instance-specific prompts. To address this problem, explore MILM hallucinations as contextual prior knowledge from pretraining, rather than eliminate them. These hallucinations are particularly useful when direct visual cues are absent or ambiguous, helping the model fill information gaps and hypothesize potential task-related elements within the image that are not prominent. By revealing these often- fifen-overlooked subtle associations, hallucinations provide more comprehensive scene understanding of the image content. This deeper contextual understanding provide reasoning context for generating more accurate and relevant instance-specio specific prompts candidates. Thus, using hallucinations to uncover task-related knowledge helps overcome challenges from visual ambiguities and object camouflage in complex scenes. To this end, we propose a multi-scale chain-of-thoughh prompting strategy that stimulates hallucinations to leverage prior knowledge, fully extracts task-relevant information, and then uses this information to enhance the precision the generated instance-specifi prompts.
3.1.1 Multi-scale Chain of Thought Prompting
Multi-scale Chain of Thought Prompting consists of two processes: Gathering candidate knowledge and generating accurate instance-specific prompts. To efficiently collect task-relevant information from an image, shown in Fig. we divide the input image into patches various scales by cutting

---

horizontally, vertically, or by leaving it whole. These patches are then processed by the MLLM to gather preliminary instance-speeific prompts, The differing levels of task-relevant object visibility in each patch prompt the MI.LM to induce hallucinations. These hallucinations utilize prior knowledge to explore connections between the image data and the associated task, aiding in the detection of potential bounding boxes and object names. The process is computed by:
B = MLLM (x* C*, ,n),
Alsee Aback MLLM (x+, ,C*,
(2)
where C² is the caption generated by MLLM for the k-th image patch X Pg is task-generic prompt. For bounding box prediction, the prompt PB. which instructs "This image is from the Pg detection task, output the bounding box of the Pg. " This guides the MLLM to predict the bounding box B of the task-related objects within the patch. For predicting name, the prompt PA. stating "Ouiput the name of the Pg and its environment in one word." " is used, guiding the MLLM to predict the names of the relatee objects Airee and their backgrounds Afack from each patch. The preliminary data, including object names Ali.e and bounding boxes B* gathered from various patches, are compiled into candidate lists A, and B, Here, denotes the iteration in the iterative learning cycle. In this process, the hallucinations employed are essentially based on object co-occurrence priors, where objects commonly associated with background elements during pre-training are predicted to be task-relevant, even if they are not present in the current image. This prior knowledge is useful during the knowledge collection stage as it uncovers implicit relationships and details in the image. However, it can also reduce accuracy of the later fine-graine instance-specifty prompts generation. Therefore, is crucial to control these hallucinations in the latter stage to prevent incorrect predictions.
Visual Contrastive Reasoning. To mitigate hallucinations caused by object co-occurrence priors, recent research highlights particularly relevant regions of an image direct MI.LMs focus toward task-related elements, thereby minimizing background interference and enhancing model accuracy
[59 [5]. To achieve this, visual markers are employed to steer MLLM attention on task-relevant visual regions, thereby reducing hallucinations. While closed-source MLLMS like GPT-4V 443 can interpret these markers effectively, they are costly and large. In contrast, models like LLaVA 37 are open-source, but cannot process visual markers such as points or bounding boxes, and employing these markers might disrupt the original pixel data, degrading performance on LLa (see Tab. [5) Moreover, accurate pixel-level visual markers are unavailable in our setting. To solve this problem, we aim to enable LLaVA to focus on reskerelated regions without altering the original pixel data, therehy effectively minimizing hallucinations and enhancing the precision of instance-specific prompts.
Despite the absence of instance-level annotations, promptable segmentation models produce masks with detailed textures, which provide rich positional and textural information about interested regions. We use these masks as visual markers to guide a MLLM to focus on task-related areas during the generation of instance-specific prompts. Inspired by classifier-fre free guidance [20 47), we introduce visual contrastive reasoning (VCR), a training-fre free visual marking method help MLLM focus on specific regions, reducing hallucinations. The relevance of region is assessed by observing MLLM output changes when key areas are excluded. It guides the MILM to focus on areas with notable changes (bottom of 3). Based on Eq. (1. we derive a probability distribution by comparing original image X with a modified image, X process XX IM), where region IM is excluded.
(yv I..P.
Y/ x (o (,P.y
(ve process(XXIM) P,
softmax (1 + a)) logit, (ve X, P. - a logit, process( P.), (3)
where a is the level of focus on region IM. A higher increases emphasis that region. Following [54), we set a = in all tasks. It preserves the integrity of the original image pixels X, while constructing contrastive samples X' that encourage the model to focus on task-related regions. Ideally, X' should exclude task-related objects while maintaining a uniform appearance and overall context with the original image. But directly marking X' disrupts its pixels, making contrastive sample generation challenging.
Contrastive Sample Generation. To address it, we employ inpainting, where the mask M obtained from the previous iteration segmentation treated as the inpainting mask IM, to guide the creation of X'. We use a negative prompt P.: "Afor is a Pg", to ensure that the inpainted X' does not contain potentially tesk-related objects Afore Additionally, we use a positive prompt Pp: "Aheck high quality, detailed, blended to the original image.", ensure consistency between the generated
5

---

portion and the surrounding background A The corresponding inpainting is defined as:
X' = Fin Fx, Pp., Pn),
where Fin represents the inpainting module, and we choose Stable Diffusion to perform this operation. This method ensures the generated X' excludes task-related objects without disrupting the pixel continuity. In the first iteration, since IM, does not yet exist, we use bounding box predictions from various patches B; as an alternative. As X' contains only the background, comparing it with X eliminates co-occurrence hallucination caused by the background and highlights differences in task- related regions, subtly guiding the model focus on these areas. Finally, we use visual contrastive reasoning to identify accurate instance-spect speciifc prompt to guide segmentation as follows,
(5) where VCR represents our visual contrastive reasoning, and C' is the caption of the image. The collected knowledge, Ai and Bi, is integrated into the prompt Pa and PB- This process aids in identify the ultimate instance-specific names A and bounding boxes B.rh of the objects.
D VCR (X, X', C,,
A VCR XX, ,
3.2 Mask Generator
Until now, we described how to use SAM- generated masks as visual marker to guide the model to focus on task-relevant areas for generating accurate sestance-specifi prompts. But this method relies on an assumption that the mask accurately delineates task-related regions. However, SAM is trained on large-scale prompt-mask pairs without category labels, excels identifying masks based on image textures but lacks label prediction capabilities. Consequently, the SAM-generated mask may not always align with task semantics, yet such alignment is crucial for our method.
3.2.1 Mask Semantic Alignment
We need to utilize texture generalization capabilities of SAM to describe possible task-related objects within the prompt-targeted areas, while also ensuring that the generated masks align with the task semantics. To achieve this, we divide the input image into patches of varying scales using horizontal, vertical, and uncut divisions as outlined in the last section. these processed patches are then reintegrated onto the original image with surrounding areas blacked out, and fed into SAM to ensure it focuses exclusively on the patch. Finally, masks generated from different patches are aggregated based on their relevance to task semantics, providing an accurate representation of relatee objects. The masks for each patch is generated as follows,
m SAM(Spatial CLIP(AN, By, X
where mask m. is obtained by inputting the corresponding image patch X and associated prompts into SAM during the i-th iteration. Following 211. Spatial CLIP maps the text prompt A to regions in the image X; that correspond to the content of the prompt. The processed images, along with the generated spstafc text prompts A,, are then input into CLIP to assess semantic similarity.
smm)) = (L @ X;, A?).
(7)
the operation @ results in retaining only those parts of X; that are covered by the predicted mask. s(m²) represents the similarity between masked image and A calculated using CLIP. The similarity scores obtained from different patches are denoted as St [s(m)) ssm?) (..e) s(m )] After
normalizing the elements within S; the closer the normalized s(m)) 1, the more semantically aligned m is with the instance-speciff text prompt AV. Finally, compute the weighted sum of the normalized s(m{) and m. as follows.
K
M2 Σ (s(mn)) m)), A=1
(8)
M, is the output mask of the i-th iteration of X. The generated M, leverages SAM's mask prediction capabilities to create highly detailed masks. Simultaneously, through the mask semantic alignment process, it ensures that the output mask aligns with the task's semantics, thereby overcoming the limitation of SAM's mask prediction lacking semantic understanding. The mask is applied to the original image as a weight, to generate the next iteration image X, for segmentation. This excludes irrelevant regions to reduce interference during segmentation.
X = w (X, C M,) + (1 - w) X
where w is a hyperparameter, which we have assigned value of 0.3.

---

Table 1: Results Camouflaged Object Detection (COD) under different settings. Best are in bold.
Camoullaged Object Detection
Methods
CHAMELEON
50
CAMO
CODTOK
14 E
Venue
F5
S.
Fa
Ee
8.1
Pa 1
8.
Scribble Supervision Setting
WSSA
CVPR20
0.067
0.692
0.8GU
0.782
0.118
0.615
0.786
0.896
0.071
0.536
0.770
0.684
SCWS|60
AAAI21
0.053
0.758
0.881
0.792
0.102
0.658
0.795
0.713
0.055
0.602
0.805
0.710
TEL 62]
CVPR22
0.073
0.708
0.827
0.785
0.104
0.681
0.797
0.717
0.057
0.633
0.826
0.724
17
AAAI23
0.046
0.791
0.897
0.818
0.092
0.709
0.815
0.735
0.049
0.637
0.832
0.733
SAM-S 29
ICCV23
0.076
0.729
0.820
0.650
0.105
0.682
0.774
0.731
0.046
0.695
0.828
0.772
WS-SAM
NourIPS23
0.046
0.777
0.897
0.824
0,092
0.742
0.818
0.759
0.038
0.719
0.878
0.803
Point Supervision
Setting
WSSA 62
CVPR20
0.105
0.660
0.712
0.711
0.1488
0.607
0.652
0.649
0,087
0,509
0.733
0.642
SCWS
AAAI2I
0.097
0.684
0.739
0.714
0.142
0,624
0.672
0.687
0,082
0.593
0.777
0.738
TEL 62
CVPR22
0.194
0.712
0.751
0.746
0.133
0.662
0.674
0.645
0.063
0.623
0.803
0.727
SCOD T
AAAI23
0.092
0.688
0.746
0.725
0.137
0.629
0.688
0.663
0.060
0.607
0.802
0.711
SAM
ICCV23
0.207
0.595
0.647
0.635
0.160
0.597
0.639
0.643
0.093
0.673
0.737
0.730
SAM- P 29
ICCV23
0.101
0.696
0.745
0.697
0.123
0.649
0.693
0.677
0.069
0.694
0.796
0.765
16
NeurIPS23
0.056
0.767
0.868
0.805
0.102
0.703
0.757
0.718
0.039
0.698
0.856
0.790
Task- Task-Generic Prompt Setting
CLIP CLIP_Sungey SUA
Arxi233
0.147
0.60G
0.741
0.689
0.189
0.520
0.692
0.612
0.175
0.488
U.698
U.629
GPT4V+SAM 33
Arxiv33
0.180
0.557
0.710
0.637
0.206
0.466
0.666
0.573
0.187
0,448
0.672
0.601
LLAVAI.S+AAM 37 29
NeuriPS23
0.168
0.561
0.718
0.666
0.314
0.401
0.585
0.501
0.170
0.530
0.728
0.662
X-Decader
CVPR23
0.124
0.654
0.748
0.716
0.104
0.628
0.745
0.709
0.171
0.556
0.705
0.652
SEEM [1]
NeurlPS23
0.094
0.011
0.307
0.454
0.192
0.023
0.315
0.404
0.143
0.001
0.280
0.425
Grounding SAM 22 B8
ICCV23
0.122
0.662
0.776
0.744
0.157
0.656
0.753
0.707
0.085
0.670
0.813
0.764
GenSAM AM 21
AAAI24
0.696
0.774
0.106
0.798
0.729
0.058
0.783
ProMaC
Ours
0.044
0.790
0.899
0.833
0.090
0.725
0.846
0.767
0.042
0.716
0.876
0.805
3.3 Mask Prompt Cycle Generation
The mask generated from the last iteration will guide the prompt generator in the next iteration to focus on potential task-related regions, eliminating the erroneous effects of irrelevant hallucinations and generating more accurate instance-specific prompts. These prompts, in turn, help the mask generator produce better masks. Through iterative prompt generation and mask generation jointly, we yield both better -ppecncc prompts and visual masks. Finally, the masks from different iterations are averaged, and the mask closest to the mean is considered the final output.
1 arg min (
E,
(M M1)
(10)
'result
Here, I is the number of adaptation epoches and M.. is the corresponding final mask for image X.
Table 2: Results for Medical Image Segmentation (MIS) under task-generic prompt setting.
Image Seamentahon
Scementation
Methods
Venue
CVV-ColonDDD
51
Kvasir
ISTC
[T0
37
Fg 1
Eg
5.
A7
F5
Eg
8g
18
F.1
Eg 1
5,
GPI4V 29 29
Arsi223
0.378
0,051
0.246
0,242
0.614
0.128
0.236
0.253
0.514
0.387
0.366
0.334
LavVAI 54MMM
29
NemiP523
0.355
0.479
0.293
0.400
0.369
0.497
0.477
X. Deccdeer
69
CVPR23
0.462
0.095
0.327
0.331
0.489
0.202
0.371
0.384
0.338
0.315
0.127
0.407
SEEM
NeruIP233
0.570
0.085
0.280
0.284
0.520
0.215
0.339
0.367
0.362
0.250
0.002
0.280
GM
29 38
I0CV23
0.711
0.071
0.195
0.206
0.387
0.353
0.521
0.468
0.301
0.348
0.247
0.533
GenSAM
AAAI24
0.244
0.059
0.494
0.379
0.172
0.210
0.619
0.487
0.171
0.699
0.744
0.678
ProMaC
Ours
0.176
0.243
0.583
0.530
0.166
0.394
0.726
0.573
0.168
0.717
0.755
0.689
Table 3: Result on Transparent Object Segmentation and Open- Vocabulary Segmentation Tasks.
(a) Transparent Object Segmentation.
(b) Open-vocabulary Segmentation.
Mithois
M
GSD 34
Tram Jun [6]
Fa F.1
Mishads
Venue
Saz. Ansin
Image-Text goine
mloli 1 mmle)
VOC Cantesl
Olject mloo.
GPT4V
0,104
0.592
0.238
0.607
MasKCIFF 67
V2C
23.6
20.0
ALLVAI.54AAM 33
8.197
0.545
0:372
0.621 0.555
TCL 0
CC3M
34.3
30.4
DeDedda
8.19:
0.240 0.643 0,450 0.568 9.611 0.215 0,280
Group 57
CVP922
CC12M C
YFCCL4M 53
$2.3
22.4
SPFM [2]
0.184 0.224 0.57% a
p.557 0.501 0.013 0:256
ViewDn
4h
ICL223
C012M
H
YRO0
31
$3.4
23.0
23.5
GmundingSAM 26 38
n.168 0.572 0 R.436 0.415
0.947 0.424
SLyCLIP
30
ICML23
C000 35
CC 35
52.6
24.7
26.5
GenSAM
0.394 0.700 0.559 0.659 0.612 0.536
O'Segmentor
CVP423
CC12M
53.8
20.4
25.1
PreMat
25.
4 Experimental Setup
Baselines. To evaluate our approach across various scenarios, we first assess the performance of ProMaC on challenging segmentation tasks, including Camouflaged Object Detection (COD), Medical
7

---

Table 4: Ablation Study on COD and MIS Tasks
Table 5: VCR Result on SR task
Method's Variants
CHAMELEON
50
CVC.ColobNB
[S]
Model
Indiy. Pairs Set of
MCoT IVP ITP VCR MSA
Fs 1 Fg
Sa
M
Fa 1 F. S.
CLIP WTL 14 45
26.1
1.5
0.D
0.052
0.7G4 0.885
0816
0.187
0.214 0.570
0.513
CLIP RN50x64
45
26.2
2.0
0.0
0.080 0.720 0.833
0.257
0.260
0.123 0.466
0.425
[9
30.4
10.9
0.061
0.089 0.685
0.769
0.893
0.823
0.756
0.815
0.177
0.311
0.233
0.152
0.460
0.556
0.424
0.524
ViP-LLAVA-138
5
70.9
57.5
21.8
0.054
0.740
0.884
0.798
0.156
0.220
0.565
0.517
LLAVA-1.5-138
60.6
0.044
0.790
0.899
0.833
0.176
0.243
0.583
0.530
+VC (Ours)
75.4
63.6
36.7
Table 6: Parameter ablation study on CODIOK 14]
Ga) Number of ittntion
(b) Image preprocess strilgy
(c) Visual marker stralegy.
cos
1GL1
21 1
T
Agg
30
Scale
M
F3
Fg
8,
stntley
Fia
0.876
11:0
0.593
11361
0.3150
113001
115.25
0.683
0.859
0:818
11:755
Havel
0.009
0,075
0.335
0.579
0.750
0.775
0.682
0.689
None
Bbox
0.058
0.690
0.682
0.855
0.766
0.789
0.88
0,882
0.56
0.602
0.342
0.041
0.714
0.718
0,855
0.825
0.804
0.804
Drivenal-Flvvel
0.042
VCD
0.047
0.705
0.863
0.793
0.882
0.599
0.941
0.721
0.876
0.803
Originll -llavelQQurriess
0.019
0.867
0,714
0,875
0.804
Image Segmentation (MIS), and Transparent Object Detection (TOD). These tasks are areas where SAM struggle [26]. In the COD task, we compare ProMaC with weakly supervised segmentation methods [29 62 60 62
22 24 I8]. Two supervision levels are used for comparison: scribble supervision, where main structures for the foreground and background are sketched during training, and point supervision, where separate points are provided for both foreground and background. In the task-generic prompt setting, we introduce a challenging scenario by only providing task description as generic prompt for segmentation. ProMaC integrates VLa [77] with SAM [29]. We also experiment on the MIS and PIS tasks to demonstrate the effectiveness of our method using task-generic prompts compared to previous methods. We assess GPT4V+SAM and LLaVA A.MMMMMM in this setting to demonstrate that current MILM models cannot address it well. We also compare ProMaC against current SOTA promptable segmentation methods to showcase its effectiveness. Next, we evaluate ProMaC Open- Vocabulary Segmentation (OVS), and compare with leading methods [33 77 71 38 63 66 32 [9]. Our Visual Contrastive Reasoning (VCR) strategy is applicable to other tasks, especially those requiring complex spatial understanding. We used the What's Up spatial reasoning dataset [27 to evaluate how well VCR guides models to focus on task-relevant regions in Spatial Reasoning (SR) task. Our results are the average of three trials.
Metric. For evaluating the first three tasks, metries Mean Absolute Error (M), adaptive F-measure (Fa) 44), mean E-measure (Eş) [15, and structure measure (S,) 133 are used. Lower M or higher values for F3, Eeg, and S., reflect better performance. Mean Intersection over Union (mloU) and accuracy measure OVS and SR performance respectively, with higher values indicating better results.
PyTorch Implementation Details. For the MLLM models, we utilize LLaVA-1.5- for evaluation purposes. For CLIP, we adopt the CS-ViT-B/16 pretrained model. The inpainting model is stable- erffusionn2iippainting The task genkric prompts for the COD task is "camouflaged animal". The MIS task consists of two sub-tasks: polyp image segmentation and skin lesion segmentation, each with its own task-generic prompts. "polyp" and "skin lesion" respectively. For TOD task, prompt is fixed as "glass". All tasks are optimized using training frree dest-time adaptation, with each task iterating for four epochs, except for the polyp image segmentation task, which undergoes six epochs. Since the second epoch, VCR also operates on different patches, ensuring that while the non-inpainted parts still gather information through hallucinations, the inpainted parts eliminate hallucinations to generate accurate candidate prompts. The promptable segmentation methods is the VIT-H/I6 version of SAM. Our experiment is conducted on a single NVIDIA A100 GPU. More details are in appendix.
5 Results and Analysis
Results on COD Task. The COD tasks focus on finding animals that blend into their complex surroundings. We evaluated ProMaC on three representative datasets: CHAMELEON (50), CAMO [30. and CODIOK [14] As shown in Tab. [ we compared ProMaC with others that utilize varying levels of supervision. Overall, methods with scribble supervision generally perform better than those with point supervision. Importantly, ProMaC only uses single generic task prompt for the entire task and it stil outperforms all point-supervised methods. It also surpasses methods with scribble supervision on the CHAMELEON and CAMO datasets, and matches the top-performing ccribllessuperisce methods CODIOK. It demonstrates the superiority of ProMaC.
8

---

Camoutlaged Objeet Detection
Transparent Objjet Segmentation
Image Segmentation
GonNAM Groniling
Ours
Contrastiv Sample
Figure 4: Visualization of various segmentation methods among various segmentation tasks.
Results on MIS and TOD Task. The MIS task identifies pathological tissues in medical images. We used three datasets: ColonDB [51] and Kvasir (25] for polyp image segmentation, and ISIC [I0] for skin lesion segmentation. We compared our approach with others using task lask-generic prompt settings (see Tab. 2). While other models underperform in medical imaging due to limited generalization, ProMaC improves significantly over the baseline by iteratively mining task-related knowledge. For the TOD task, we evaluated ProMaC the GSD (44 and Trans 10K-hard (56) datasets (See Tab. 3(a)) Using the task-generic prompt setting, our method achieves the best results despite challenging scenarios. This demonstrates ProMaC's versatility and adaptability across complex visual tasks.
Results on OVS and SR Task. We evaluated ProMaC's effectiveness the OVS task for multi-class segmentation based on a list of candidate classes. Specifically, we tested it on the validation splits of PASCAL VOC (21 classes) [T. Pascal Context (59 classes) [42] and COCO-Objer (80 classes) 3. using LLaVA to identify and confirm the presence of candidate classes. After obtaining masks, we resolved overlaps using the argmax operation based on SAM probabilities. Tab. 3 shows how ProMaC compares other state-of-the-art OVS methods. Unlike some methods trained specifically on these datasets (risking knowledge leaking), ProMaC not. Yet, ProMaC still outperforms all others on PASCAL VOC and Pascal Context and is competitive COCO-Objer Additionally, as shown in Tab. 5 we integrated our VCR into LLaVA for enhanced spatial reasoning. This integration allows LLaVA to better focus critical areas, thereby boosting performance.
Module Analysis. As shown in Tab. we perform an ablation study on the COD and MIS tasks to assess the effects of different modules. "MCoT" is multi-scale chain of thought prompting. "TTP" and "TVP" refer using only instance-specific text prompts or visual prompts. "VCR" visual contrastive reasoning, and "MSA" is mask semantic alignment. The first row shows replacing MCoT with just one original image results in reduced performance, highlighting the importance of using hallucinations to extract task-relevant information. The second and third rows show that single modal prompts perform worse than multimodal prompts, highlighting the significance of multimodal prompting. Removing VCR causes significant drop performance, indicating that visual prompts are crucial for directing LLaVA's focus on relevant areas during inference. The comparison between the fifth and final rows emphasizes the importance of mask alignment with task semantics. The consistent positive results across tasks confirm the robustness and effectiveness our approach.
Parameter Analysis. Tab. a) examines how iterations influence performance. "cos" measures the cosine similarity between the predicted text prompt and the ground truth class through CLIP. "IoU" assesses the overlap between the predicted bounding box and the ground truth, comparing it against rectangular outline of the mask. Mask predictions improve and stabilize after the fourth epoch. Tab. 6/b) investigates the effects of various image processing techniques. "Original" uses modifications.

---

Image& GT
iteration
iteration
iteration
iteration
Prediction
Contrastive Image
Prediction
Contrastive Image
Prediction
Contrastive
Image
Figure 5: Visualization of the generated masks and contrastive samples over iterations.
"Halve" divides the image horizontally vertically into halves, and "Quarters" divides it into four quarter-sized patches. Testing shows that combining "Original" and "Halve" yields the best results by balancing global and local information without excessive fragmentation.
Visual Marker Strategy. Tab. ((c) assesses the impact of different visual marker strategies. "None" uses no visual prompts, while "Bbox" places bounding boxes directly on the image. "VCD" employs previous methods that introduce Gaussian noise into comparison images for contrastive reasoning. Results indicate that bounding boxes decrease performance, suggesting LLaVA struggles with this type of markers. Although VCD methods improve performance, they distort pixel data, making them less effective than our approach. Our VCR generates contrastive samples that focus task-relevant areas without altering the image, reducing hallucinations and enhancing performance.
Visualization. Fig. and Fig visually compares our ProMaC with other methods across 3 tasks and also shows the contrastive images we generated. GenSAM handles clear objects well but struggles with complex background. Although GenSAN AM performs well in complex backgrounds, but struggles with challenging tasks. ProMaC delivers solid segmentation results across different tasks, and our contrastive images remove task-related regions while maintaining semantic and pixel consistency.
Conclusion
In this work, we introduce an iterative ProMaC that uses MLLM hallucinations to guide automatic prompt generation, significantly improving segmentation without training. This iterative approach aligns masks with task semanties, enhancing model performance. Testing on multiple benchmarks has demonstrated ProMaC's effectiveness in a wide range of complex segmentation tasks.
10

---

References
[1] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274 2022.
[2] Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei Efros. Sequential modeling enables scalable learning for large vision models. arXiv preprint arXiv: 200077 2023.
[3] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209-1218, 2018.
[4] Deng Cai, Yan Wang, Huayang Li, Wai Lam, and Lemao Liu. Neural machine translation with monolingual translation memory. arXiv preprint arXiv: 20::2105./4 1269,, 2021.
[5] Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jac Lee. Making large multimodal models understand arbitrary visual prompts. arXiv preprint 2xXiv:33/2.00760 2023.
[6] Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate rext-grounded mask for open- open-world semantic segmentation from only image-text pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11165-11174, 2023.
[7] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Sorieut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558 -3688, 2021.
[8] Junjie Chen, Li Niu, Liu Liu, and Liqing Zhang. Weak-shot fine-grained classification via similarity transfer. Advances in Neural Information Processing Systems, 34:7306-7318 -318, 2021.
(9] Junjie Chen, Li Niu, Jianfu Zhang, Jianlou Si, Chen Qian, and Liqing Zhang. Amodal instance segmentation via prior-guided expansion. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 313-321, 2023.
[10] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv: /902.0999000000 036.033.8..... 2019.
[11] M Everingham, L Van Gool, CKI Williams, J Winn, and A Zisserman. The pascal visual object classes challenge 2012 (voc2012) results. 2012 http://www. pascal-network. org/challenges. In 20CC/oo22112/wwrkshee////o
himl, 2012.
[12] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-338 2010.
[13] Deng-Ping Fan, Ming-Ming Cheng. Yun Liu, Tao Li, and Ali Borji. Structure-meastte A new way to evaluate foreground maps. In Proceedings of the IEEE international conference on computer vision, pages 4548 4557, 2017.
[14] Deng-Ping Fan, Ge-Peng Ji. Ming-Mine Cheng, and Ling Shao. Concealed object detection. IEEE transactions on pattern analysis and machine intelligence, 44(10):60246604 6042, 2021.
[15] Deng-Ping Fan, Ge-Peng Ji, Xuebin Qin, and Ming-Ming Cheng. Cognitive vision inspired object segmentation metric and loss function. Scientia Sinica Informationis, 6 2021.
[16) Chunming He, Kai Li, Yachao Zhang, Guoxia Xu, Longxiang Tang, Yulun Zhang, Zhenhua Guo, and Xiu Li. Weakly-superve labeling and multi-scale feature grouping. arXiv preprint arXiv:2305.11000 2023.
concealed object segmentation with sam-based pseudo
[17] Ruozhen He, Qihua Dong, Jiaying Lin, and Rynson WH Lau. Weakly-supervised camouflaged object detection with scribble annotations. In Proceedings of the AAÁI Conference on Artificial Intelligence, volume 37, pages 781-789, 2023.

---

[18] He, Chunming and Li, Kai and Zhang, Yachao and Tang, Longxiang and Zhang, Yulun and Guo, Zhenhua and Li, Xiu. Camouflaged object detection with feature decomposition and edge reconstruction. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 22046-220555 2023
[19] He, Chunming and Li, Kai and Zhang, Yachao and Zhang, Yulun and Guo, Zhenhua and Li, Xiu and Danelljan, Martin and Yu, Fisher. Strategic preys make acute predators: Enhancing camou- flaged object detectors by generating camouflaged objects. In arXiv preprint arXiv:2308.00 0333008.3166, 2023.
[20] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207. /2598 2022.
arXiv preprint
[21] Jian Hu, Jiayi Lin, Shaogang Gong, and Weitong Cai. Relax image-specific prompt requirement in sam: A single generic prompt for segmenting camoutlaged objects. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1-511-, 2024.
[22] Jian Hu, Hongya Tuo, Chao Wang. Lingfeng Qiao, Haowen Zhong, and Zhongliang Jing. Multi-weight partial domain adaptation. In BMVC, page 5, 2019.
[23] Jian Hu, Hongya Tuo, Chao Wang, Lingfeng Qiao, Haowen Zhong, Junchi Yan, Zhongliang Jing, and Henry Leung. Discriminative partial domain adversarial network. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow. UK, August 23-28, 2020, Proceedings, Part XXVII 16, pages 632-648. Springer, 2020.
[24] Jian Hu, Haowen Zhong, Fei Yang, Shaogang Gong, Guile Wu, and Junchi Yan. Learning unbiased transferability for domain adaptation by uncertainty modeling. In European Conference on Computer Vision, pages 223-241. Springer, 2022.
[25] Debesh Jha, Pia H Smedsrud, Michael A Riegler, Päl Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D Johansen. Kvasir-seg: A segmented polyp dataset. In MultiMedia Modeling: 26th International Conference, MMM 2020, Daejeon, South Korea, January 5-8, 2020, Proceedings, Part Il 26, pages 451 462-462. Springer, 2020.
[26] Wei Ji, Jingjing Li, Qi Bi, Wenbo Li, and Li Cheng. Segment anything is not always perfect: An investigation of sam on different real-world applications. arXiv preprint arXiv:2304.0575: 2023.
[27] Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's" up" with vision-language models? investigating their struggle with spatial reasoning. arXiv preprint arXiv:2310. 2023.
[28) Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113-19122, 2023.
[29] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Wan-Yen Lo, al. Segment anything. arXiv preprint w:Xiv:2304.02644 2023.
[30] Trung-Nghia Le, Tam V Nguyen, Zhongliang Nie, Minh-Triet Tran, and Akihiro Sugimoto. Anabranch network for camouflaged object segmentation. Computer vision and image under- standing, 1 184:45 56, 2019.
[31] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision- languael models through visual contrastive decoding. arXiv preprint 2rXiv:2311.1692) 2023.
[32] Ang Li, Jian Hu, Ke Ding, Xiaolu Zhang, Jun Zhou, Yong He, and Xu Min. Uncertainty-based heterogeneous privileged knowledge distillation for recommendation system. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2471-2473 2023.
[33] Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open-vocabulaty tasks. arXiv preprint arX1v:2004005633, 2023.
12

---

[34] Jiaying Lin, Zebang He, and Rynson WH Lau. Rich context aggregation with reflection prior for glass surface detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13415-13424, 2021.
[35] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[36] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. 2023.
[37] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint 2304::2200, 2023.
[38] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint urXiv:2303. 054999 2023.
[39] Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. In International Conference on Machine Learning, pages 23033-23044. PMLR, 2023.
[40] Ran Margolin, Lihi Zelnik-Manor, and Ayellet Tal. How evaluate foreground maps? In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 248-255, 2014.
[41] Shentong Mo and Yapeng Tian. Av-sam: Segment anything model meets audio-visual localiza- tion and segmentation. arXiv:2305.01833 2023.
[42] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of IEEE conference computer vision and pattern recognition, pages 891-898, 2014.
[43] OpenAl. Gpt-4v: Enhancing gpt-4 for visual processing. 2024. Accessed: 2024-05-20.
[44] Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. Totto: A controlled table-to-text generation dataset. arXiv preprint 20Xhv:2004.13373 2020.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8766 8763. PMLR, 2021.
[46] Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiaodan Liang. Viewco: Discovering tupe-supe segmentation masks via multi-view semantic consistency. arXiv preprint arXiv:2302. 100. 2023.
[47] Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammana- manchi, and Stella Biderman. Stay on topic with classifier-free guidance. arXiv preprint 2336v:2306. / /7866, 2023.
[48] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2556-2565, 2018.
[49] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recogni- tion, pages 15638-15650 2022.
13

---

[50] Przemystaw Skurowski, Hassan Abdulameer, J Blaszczyk, Tomasz Depta, Adam Kornacki, and Koziel. Animal camouflage analysis: Chameleon database. Unpublished manuscript, 2(6):7.
2018.
[51] Nima Tajbakhsh, Suryakanth Gurudu, and Jianming Liang, Automated polyp detection in colonoscopy videos using shape and context information. IEEE transactions on medical imaging, 35(2):630 644, 2015.
[52] Lv Tang, Peng-Tar Jiang, Zhihao Shen, Hao Zhang, Jinwei Chen, and Bo Li. Generalization and hallucination of large vision-language models through a camouflaged lens. arXiv preprint arXiv:2311.1122 2023.
[53] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Commumications of the ACM, 59(2):64-73, 2016.
[54] David Wan, Jaemin Cho, Elias Stengel-Rskin, and Mohit Bansal. Contrastive region guid- ance: Improving grounding in vision-language models without training. arXiv preprint arXiv:2403.02325, 2024.
[55] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions. arXiv preprint arXiv:2212.10560, 2022.
[56] Enze Xie, Wenjia Wang, Wenhai Wang, Mingyu Ding, Chunhua Shen, and Ping Luo. Seg- menting transparent objects in the wild. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pages 696-711. Springer, 2020.
[57] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and pattern Recognition, pages 18134-18 18144, 2022.
[58] Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open- vocabulary semantic segmentation models from natural language supervision. In Proceedings of the IEEESCVF Conference on Computer Vision and Pattern Recognition, pages 2935-2944, 2023.
[59] Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mart mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310. 11441, 2023.
[60] Siyue Yu, Bingfeng Zhang, Jimin Xiao, and Eng Gee Lim. Structur--cosisterr weakly supervised salient object detection with local saliency coherence. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 3234 3242, 2021.
[61] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 01223334-23655 2021.
[62] Jing Zhang, Xin Yu, Aixuan Li, Peipei Song, Bowen Liu, and Yuchao Dai. Weakly-supervised salient object detection via scribble annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12546-12555, 2020.
[63] Shizhao Zhang, Hongya Tuo, Jian Hu, and Zhongliang Jing. Domain adaptive yolo for one-stage cross-domain detection. In Asian conference machine learning, pages 785-797. PMLR, 2021.
[64] Yi-Fan Zhang, Weichen Yu, Qingsong Wen, Xue Wang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Debiasing large visual language models. arXiv preprint 2xXxv:2403.05266 2024.
14

---

[65] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, al. Siren's song the ocean: survey on hallucination in large language models. arXiv preprint arXiv:2309.012199 2023.
[66] Haowen Zhong, Chao Wang, Hongya Tuo, Jian Hu, Lingfeng Qiao, and Zhongliang Jing. Transfer learning based on joint feature matching and adversarial networks. Journal of Shanghai Jiaotong University (Science), 24:699-705 2019.
[67] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pages 696-712. Springer, 2022.
[68] Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754, 2023.
[69] Xueyan Zou, Zi-Yi Dou, Jianwei Yang. Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang. Lu Yuan, al. Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15116-15127, 2023.
[70] Xucyan Zou Zi-Yi Yi Dou Jianwei Yang*, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Jianfeng Wang, Lu Yuan, Nanyun Peng, Lijuan Wang. Yong Jae Lee*, and Jianfeng Gao*. Generalized decoding for pixel, image and language. 2022.
[71] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all once. arXiv:2304.0671 067188.0066..... 2023.
15

---

A Appendix
A.1 More Explanation on Task-generie Promptable Segmentation
Task-generic promptable segmentation, first introduced by [2I], aims to solve a key challenge in promptable segmentation: the requirement for manual prompts, such as bounding boxes, scribbles, or points, for each sample within the same task. This method allows a model, like the Segment Anything Model (SAM), to segment samples based on these prompts. Task-generic promptable segmentation seeks to enable models to automatically infer the task-related objects in different images based general task description. This approach eliminates the need manually annotate every task-related object for different images under the same Compared to previous methods, this approach only requires single task description task-generic prompt for batch of five annotated samples within the same task, significantly reducing the annotation burden and better aligning with practical needs. Although this setting presents greater challenges than past methods, our ProMaC system achieves excellent performance across variety of tasks. It even surpasses the performance of state-of-thh-aar (SOTA) methods trained on weakly supervised datasets for camouflaged object detection, demonstrating the robustness and superior performance of our approach.
A.2
More Experiments on the Motivation
Instance-speeifle
between Ground and Predicted Tesk-Ralsted Obpects Acress Mathods
Knowledge Collection
Prompt Generation
chameleon
crab
0.79
caterpillar
Caiken Medarty
crab
chameleon
moth collect
prompts
57-
Original Image
Vertically Split Image
Hortzontully Split Image
Final Prediction
Methods
Figure 6: Left: In the bar chart, we analyze MILM predictions with two versions of an image: the original (blue) and another with task-related objects removed via inpainting (orange). We then compare their predictions to the ground truth using CLIP similarity on the COD 10K dataset. Despite missing key objects, the inpainted image's 's predictions still somewhat match the ground truth. When we select the higher similarity score from both images as the final score (green), it surpassed that of the original alone. It shows that prior knowledge from hallucinations can also provide useful information for generating prompts. Right: A example of using hallucinations to assist instance speecffcc prompt generation. Specifically, utilizing hallucination can leverage prior knowledge of image elements to better recognize and locate task-relater objects. Directly inputting the image into LLaVA results in the hidden chameleon being incorrectly predicted. Splitting the image results interested objects being incomplete or absent, prompting L. to induce hallucinations and utilize prior knowledge to uncover potential related knowledge within the image. This knowledge assists in final accurately identifying and locating the chameleon.
A.3 Further Demonstration on Implement Details
We conducted experiments across multiple datasets and compared our method with various ap- proaches across different tasks. Initially, we benchmarked against methods such as GPT4V+SAM and I.L.VA 5+SAM to demonstrate that even the current state-of-the-arr (SOTA) MILM methods combined with SAM cannot directly address this issue. Here, GPT4 utilized the 1ett4-visioo-previee model, and LLa used the LLaVA-1.55-3B model, with SAM employing the ViT-H/16 version. Both models were tested in single iteration setup, our experiments showed that multiple itera- tions resulted in performance degradation as iterations increased. Specifically, these MLLMs were tasked with inferring instance-specife prompts to guide SAM in segmentation. Similarly, in our setup, we fine-tuned the source code of representative promptable segmentation methods such as SEEM, GroundingSAM, and X-Decoder to adapt them to our setting. These models were directly fed task-generic prompts to segment each image, with performance evaluated based on single iteration of inference. This methodological adjustment allowed us to assess the efficacy of leveraging task-generic prompts in practical segmentation tasks.
16

---

Our ProMaC system employs LLaVA-1.5-13B and the ViT-H/16 version of SAM for inference, utilizing instance-specifie prompts generated by the prompt generator. These prompts include instance-specifer text prompts and instance-specifie bounding boxes. The instance-specift text prompts are processed by Spatial CLIP, mapped to corresponding image regions, and along with the instance-specific bounding boxes, are input into SAM to guide segmentation. All tasks, except for the PIS task, underwent four epochs of iteration, while the PIS task was iterated over six epochs. In the Camouflaged Object Detection task, the task-generic prompt used was "camouflaged animal". We also compared our results with past weakly supervised learning methods, which require at least one point per image in the training set supervision. Despite requiring less manual supervision with our task-generie promptable segmentation setting, our method achieved comparable or even better performance. Experiments were also conducted in Medical Image Segmentation and Transparent Object Segmentation tasks, with task-generic prompts "polyp" and "skin lesion" " respectively for each task. Notably, the datasets used for the transparent object detection task were exclusively glass, allowing the instance-specific text prompt "glass" " to be obtained directly without inference, requiring only the inference of bounding boxes through multi-scale chains of thought prompting. This streamlined approach emphasizes the efficiency and adaptability of our system across varied segmentation tasks.
Table 7: Results for Our Method under various task task-generic prompt settings.
Methods
Fg 1
CHAMELEON 5u
So
M
CAMO
S.
ProMaC
004400003
0.790=00003
0.8900000 0833+0001
0,090:0:002
00
0.00-0002
0.767
Methods
31
Fg
CODIUK [H]
8. 0.80100002
M
Pg
ColonDD
[S
ProMaC
0,0421 0,002 0.71620.002
0.7620.003
0.17620.002
0.24300002
0.83300003 0.500=0.002 ISIC [T)
Methods
Kvasir
M
Me
S.
M
Fa
S.
ProMat
0.166 10005
0.3941
0.002 0.766
0.00
0373
0.003 0.160 b.0ss 0.729 0.04 0.766 0000
0.703
Methods
GSD [34]
Trans P3
Trans (56)
Fg
Fe
No
Su
ProMaC
0.147 0.02
0.409
0.033 0.7251
0.00
0.569
0.003
0.251 0.003 0.309
0.654
0,02
0.357
0.002
A.4 Further Experiments Results Analysis and Visualization
In Tab. 7 we present the puriance-inclusio experimental results of our ProMaC framework across three major datasets, obtained under three different seeds, with detailed environment setup available in the code instructions provided in the supplementary materials. For the COD dataset, we used LLaVAI.5+SAM our baseline model to ensure fairness, with the comparative GenSAM model utilizing the same configuration. The three datasets utilized involve challenging scenarios with camouflaged animals or people, where some task-related targets are obscured or very small. Under such complex conditions, ProMaC achieves performance similar to or even better than weakly- supervised training methods with only brief task description like "camouflaged animal" through test-time adaptation, demonstrating our method's 's superiority. Similarly, on the MIS and TOD tasks, our method significantly outperforms the comparative promptable segmentation approaches. On the OVS task, we first use LLaVA1.5 .5 to identify which categories from a given multi-class list are actually present in the image. The categories identified are then further reasoned through our ProMaC to derive the final results. What's is a benchmark designed to evaluate the spatial understanding abilities of MLLMs. It comprises 820 images that depict clear spatial relationships between two household items, such as a chair and a bowl, each image exclusively features the two objects in one of four distinct spatial configurations. We regard ProMaC's VCR as a visual marker strategy integrated into LLaVAL 1.5 to guide spatial reasoning and compared with traditional methods. Compared to previous approaches and baselines, our method significantly enhances performance, also highlighting the efficacy of the VCR module.
Moreover, in 5 we analyze how masks evolve across multiple tasks with iterations. It is evident that as iterations increase, the segmentation results improve, the boundaries of task-related objects become clearer, and some targets initially undetected are progressively recognized. Additionally, in Fig we display masks for different tasks and samples of generated inpainting of task-related objects contrastive samples. These results further demonstrate the versatility and superiority of our method.
17

---

A.5 Limitation
compared ProMaC with the recently introduced GPT-40-SAM !? 29 approach on the CHAMELEON and CAMO datasets for the COD task, and the CVC-ColonDB dataset for the MIS task, with results displayed in the table below. It's evident that MLLMs based on GOT-4V 43] and L.. [37) excel in the COD task but experience significant performance drop in the MIS task. This suggests that LLaVA and GPT4V lack specialized data like polyp detection, which leads their underperformance in tasks requiring high specificity. This issue, stemming from the generalization limitations of MLLM datasets, also evident in our ProMaC, based on LLaVA. While ProMaC achieves impressive results the COD task due to LLaVA's general capabilities, its performance on the more specialized MIS dataset, though better than the baseline, falls short when compared the COD task. The newly proposed GPT-4o, trained GPT-4V across various fields and shows significant improvement on the MIS task, further highlighting the impact of the underlying MLLM's generalization capabilities on ProMaC. This points to a need for further exploration and research into the generalization potential of foundational MLLM models future work.
broader dataset, outperforms
Table 8: Comparison with present SOTA MLLM approaches.
Camoullared ObjocT Derection
Polyp Image Segmentation
Methods
Venue
CHAMELEON
[30]
CAMO
CVC-ColonDe [X]]
Fg
8.
M
Fs
Eg
M
Eg
0.180
0357
0.578
54SAM
37 79
NeulPS23
0.168
0.561
0.718
0.666
0.314
0.401
0.585
0.501
0.491
0.194
0.355
0.357
GPM
29
ArXiv41
0.073
0.638
0.779
0.706
0.116
0.582
0.727
0.659
0.067
0.340
0.655
0.575
ProMaC
0.790
0.833
0.090
0.725
0.846
0.767
0.176
0.583
0.530
Table 9: Comparison with present SOTA MILM approaches.
Polyp Image Segmentation
Methods
Venue
CVC-ColonDDB [S]
M
F'g f
Es
S,
GPT-40+SAM ? 29
ArXiv24
0.067
0.340
0.655
0.575
Owen- 4BM
ArXiv24
0.189
0,271
0.533
0.536
LLLVAA)
Ours
0.767
0.176
0.243
0.583
ProMaC(Qwen)
Ours
0.104
0.289
0.601
0.583
Original Image
GT
ProMaC
Contrastive Sample
Figure 7: Visualization of the generated mask and contrastive samples on CHAMELEON dataset.
A.6 Robustness of ProMaC
ProMaC uses MLLM to infer instance-specifee prompts that guide the promptable segmentation model. key question arises: how does the model ensure convergence to the accurate task-related region when initial instance-specife prompts from early iterations are imprecise? To address this, we utilize hallucinations to mine a potential list of instance-specific prompts from multiple scales and repeated image segmentations, ensuring comprehensive exploration of possible task-related objects.
18

---

Original Image
GT
ProMaC
Contrastive Sample
Figure Visualization of the generated masks and contrastive samples on CAMO dataset.
Original Image
GT
z
PreMaC
2
Contrustive
Original Image
GT
ProMaC
Contrustive Sample
Figure 10: Visualization of the generated masks and contrastive samples Trans10K dataset.
19

---

Original Image
GT
ProMaC
Contrastive Sample
Figure 11: Visualization of the generated masks and contrastive samples on GSD dataset.
Original Image
GT
ProMuC
Contrastive Sample
Figure 12: Visualization of the generated masks and contrastive samples medical dataset.
Subsequently, VCR climinates less accurate hallucinations to pinpoint the most precise instance- specific prompt from the candidate set. Additionally, instance-specifie prompts include both bounding boxes and keywords, enhancing fault tolerance through a multimodal setup. Experiments in Tab demonstrate the robustness of our multi-modal prompts. Even if the generated prompts do not perfectly match the ground truth, semantic similarity often accurately locates objects of interest within images, facilitating gradual fine-tuning through successive iterations. Even if the initial iteration produces inaccurate prompts, the input for the next iteration remains the original image, preserving correct object information for potential correction in finer-grained iterations. In summary, by providing wide range of candidates and employing a rigorous strategy for generating instance- specific prompts, along with multimodal prompt inputs and leveraging SAM's s generalizability. ProMaC consistently produces accurate instance-specific prompts across various tasks.
A.7
Social impact
Our work equivalent general works in computer vision field, which aims reducing the manual prompt dependency in promptable segmentation. Therefore, our work has similar potential societal impacts previous works (2
20

---

Algorithm Algorithm of our ProMaC
1: procedure PROMAC(X, S SAM, Pg. PA, Pn)
2: Input: Sample X E xxxWxx, Multimodal Large Language Model LLaVA1.5, promptable
segmentation model SAM, task-generic prompt pg. keyword prompt PA. bbox prompt PB-
3: Output: Segmentation result Mr..
4:
for iter 1 to do Split image into K multi--cale patches. if = 1 then
5:
6:
7:
for k = 1 to K do
8:
C +- GENERATECCAPTIONLLLAAALS x* Bk GENERATEBBBXX X* C* PB),
9:
10:
A fore A + GENERATEKEEWRRR((LLLAA,,, PA),
PA),
11:
end for
12:
else if j!
then
13:
for k = 1 to do
14:
C GENERATTCAAPTON(ILLAAL.. X*), B* + GENERATEBBOX(VCIR, X X** C* PB).
15:
16:
A fore: Aback + GENERATEEEYYORRD(YRR X* X CK,, PA).
end for end if
18:
A; < {A) AK J PA = PA}. Tare
PAJ.
B, {B' ...., BK PB (B,, PB).
21:
X' <- INPAINTIMAGE(X (X, IM; Pp. Pn),
AV B. <- FINALPROMPT GENERATION(X, X', PAs PB).
23:
M; < MASKGENERATIINN SAM, AY, B
24:
X + WEIGHTTEDIMAGE(XX M.). end for
25:
26:
M.- + SHLECTBESTMASX(MM))
27:
end procedure
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification; We have claimed our main contributions accurately in abstract and introduction clearly. We also provide further explanations in appendix A.2
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:We We have claimed limitation in the limitation section in appendix A.5
Guidelines:
21

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA].
Justification:
We do not have theoretical proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide our codes in the supplemental materials, we also provide implement details both main text and the appendix. All of our experiment results are averaged with three trials.
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have released our codes and the instruction on how to run the codes in supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines https://nnis..c/pubbic/
quidessCodeSubmissoooooeeee for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: hwtp///pwbbcguuuuss/oooeeeeeeeie
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
23

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: We have provide sufficient instructions on the settings and the details in the paper and appendix A.3 and A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes].
Justification:
We have illustrated it in experiment section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification: We gave clarified such things in implement details and instruction of the codes.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]:
Justification: We have conducted the paper conform to make sure it follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification:
We have discussed about the potential social impact in appendix A.7
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA].
Justification: We conduct experiments on public datasets, and our model is training-free apporaches, the model used are all open-sourced model from the community.
25

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: we have cited the data and models we used in our paper properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use package should be provided. For popular datasets, paperswithcode, com./aaaam/amttsttssssssssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
We do not introduce new datasets in the papers, and our codes will be released after publication.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
26

---

Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA].
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "5SEEWGp2N9", "reviewer": "Reviewer_E5Cz", "summary": "The paper introduces an iterative Prompt-Mask Cycle generation framework (ProMaC) with a prompt generator and a mask generator. The prompt generator uses a multi-scale chain of thought prompting, initially exploring hallucinations for extracting extended contextual knowledge on a test image. These hallucinations are then reduced to formulate precise instance-specific prompts, directing the mask generator to produce masks that are consistent with task semantics by mask semantic alignment. The generated masks iteratively induce the prompt generator to focus more on task-relevant image areas and reduce irrelevant hallucinations, resulting jointly in better prompts and masks. The results on several challenging datasets validate the effectiveness of it.", "questions": "See the weakness part.", "limitations": "The paper presents the limitation in the appendix. The authors suggest further exploration and research into the generalization potential of foundational MLLM models in future work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-A task-generic prompt is used to prompt the proposed ProMaC to perform segmentation. The results on several challenging datasets validate the effectiveness of it.\n\n-The authors use useful techniques, multi-scale chain of Thought Prompting, visual Contrastive Reasoning, and Contrastive Sampling Generation, to generate accurate instance-specific prompts.", "weaknesses": "-The authors mentions that the main motivation of this paper is to utilize hallucinations instead of eliminate it. However, the VCR module is used to “reducing” and “minimizing” it. This confuses me. The authors should clearly explain how this paper utilizes the hallucinations.\n\n- The iterative process and the integration of various techniques (e.g., MLLM, SAM, CLIP, and inpainting) add complexity, which may hinder the ease of implementation and understanding.\n\n-Compared with GenSAM [19]， What is the main innovation of the Mask Generator?\n\n- At the first iteration, since masks have not yet been generated, what are the visual markers in this process? \n\n-The core of this task is how to generate accurate visual prompts through MLLM using a general text prompt. From my perspective, this task should be evaluated more on general datasets, such as COCO, to demonstrate the method's generalizability rather than on specific domain tasks (e.g., camouflage). Some comparison methods in this paper, such as X-Decoder and SEEM, are all generalized methods. Comparing with them, I believe, is unreasonable. I hope the author can discuss this issue with me.\n\n-Minor: The authors should list the genetic task prompts used for different tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful reading and insightful comments. Following are the responses regarding your concerns.\n\n> *The authors should clearly explain how this paper utilizes the hallucinations.*\n\nAs we explained in Line 143-150, we utilize hallucination to bootstrap a scene-understanding of each test image. By dividing the input image into patches at various scales, the MLLM induces hallucinations based on varying object visibility in diverse patches. These hallucinations use prior knowledge to explore connections between the image data and the associated task, gathering task-relevant information before eliminating irrelevant semantics in our reasoning process. This is critical when visual cue is weak / ambiguous for segmentation. We then selectively reduce irrelevant semantics in order to remove irrelevant influence whilst benefit from relevant influence obtained from pre-trained knowledge (aka hallucination). This is in contrast to existing methods which all aim to remove hallucination blindly regardless its usefulness in helping scene-understanding in an image, therefore improving segmentation when visual cues are weak / ambiguous.\n\n>  *The iterative process and the integration of various techniques (e.g., MLLM, SAM, CLIP, and inpainting) add complexity, which may hinder the ease of implementation and understanding.*\n\n- Traditional methods often require multiple GPUs for extensive training, leading to higher computational and memory demands. In contrast, as we discussed in Line 281, our method performs iterative test-time adaptation on a single 40G A100 GPU training-free, meaning all module parameters are fixed from the start. This eliminates the need for training and parameter tuning, and therefore reducing implementation complexity. \n\n- We also provide implementation details in the PyTorch Implementation Details section (Line 271-281) of the main text and Line 551-581 of the appendix. Our code is available in the supplementary materials, ensuring our method is easy to understand and implement.\n\n> *Compared with GenSAM [19]， What is the main innovation of the Mask Generator?*\n\n- As we explained in Line 212-213, the key innovation of our mask generator is the proposed Mask Semantic Alignment (MSA) module, which ensures that the generated masks align with task semantics—an achievement that GenSAM cannot match. \n\n- In GenSAM, the mask generator uses spatial CLIP to convert instance-specific text prompts into positive/negative point prompts and directly feeds them into SAM to obtain masks. However, since SAM is trained without category labels, it lacks label prediction capabilities. This leads to masks that do not meet task requirements when the point prompts fed into SAM are not accurate. \n\n- Our MSA module addresses this limitation by using similarity-weighted masks aggregation to ensure that the output masks are aligned with the task semantics. The effectiveness of our MSA module is demonstrated in the last two rows of the ablation experiment in Tab. 4 in original paper.\n\n> *At the first iteration, since masks have not yet been generated, what are the visual markers in this process?*\n\nAs we explained in Line 199-201, in the first iteration, Multi-scale Chain of Thought Prompting (MCoT) infers predicted bounding boxes for multiple different patches. We use the union of these bounding boxes as the visual marker for the first iteration to guide the generation of the contrastive sample $X^{'}$.\n\n> *From my perspective, this task should be evaluated more on general datasets, such as COCO, to demonstrate the method's generalizability rather than on specific domain tasks (e.g., camouflage). Some comparison methods in this paper, such as X-Decoder and SEEM, are all generalized methods. Comparing with them, I believe, is unreasonable. I hope the author can discuss this issue with me.*\n\n- In Tab. 3(b), we have conducted experiments on three general datasets: COCO Object, PASCAL VOC, and Pascal Context, showing that our method achieves good performance on general segmentation tasks comparing to other unsupervised/weakly supervised methods. \n\n- As for generalized methods like X-Decoder and SEEM, it is ideal to fairly compare X-Decoder and SEEM on both general and specific tasks. However, these two methods are trained on general segmentation datasets with semantic labels, while our pre-trained model only trains with image mask pairs without semantic labels. Therefore, it is unfair to directly compare ProMaC with these methods on general segmentation tasks. Meanwhile, since neither these methods nor our ProMaC have been trained on specific domain datasets, comparing them on these tasks is fair.\n\n> *List the genetic task prompts used for different tasks.*\n\nAs we explained in Line 273-276, the text generic prompt $P_g$ for the camouflaged animal detection task is “camouflaged animal”. For the two sub-tasks in medical imaging, the text generic prompts are “polyp” for polyp image segmentation and “skin lesion” for skin lesion segmentation. For the transparent object detection task, the text generic prompt is “glass”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CcqDP4CvQT", "reviewer": "Reviewer_fu4N", "summary": "The paper introduces the Prompt-Mask Cycle generation framework (ProMaC), which innovatively uses hallucinations from Multimodal Large Language Models to refine segmentation prompts and masks. This method contrasts with traditional approaches by leveraging rather than eliminating hallucinations, enhancing task-related accuracy through an iterative prompting and masking process. ProMaC's efficacy is demonstrated across various benchmark.", "questions": "1)Dependence on Initial Prompts: While the iterative refinement process is beneficial, the initial quality of prompts still plays a crucial role. Poor initial prompts might lead to suboptimal starting points, impacting the overall effectiveness of the adaptation process. I would like to see some discussion on this.\n2)The authors utilize the LLaVA1.5 as the base MLLM for experiments, achieving results comparable to weakly-supervised and even supervised training in tasks such as Camouflaged Animal Detection, whereas performance in tasks like Medical Image Segmentation is more modest. The authors attribute this to the inherent generalization limitations of LLaVA. Recently developed methods (e.g., GPT4o) have shown better generalization capabilities in medical imaging. Could the authors conduct experiments using these more robust MLLMs on specific sub-tasks to substantiate this perspective?", "limitations": "The paper encounters a few limitations, such as the potential dependence on the initial quality of prompts and potential resource intensity due to the iterative and computationally demanding nature of the ProMaC method. Additionally, the performance disparity across different tasks highlights the limitations of the underlying MLLM's generalization capabilities. Despite these challenges, the overall contribution of the paper remains significant. The novel approach to leveraging hallucinations as a resource rather than a drawback, coupled with comprehensive experiments and detailed reproducibility, underscores the paper's value.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Unlike previous methods that considered hallucinations as negative, the authors view them as prior knowledge from the pre-trained model, first extracting task-relevant information and then validating it. This perspective is very insightful.\n- The article is well-structured and clearly written, making it easy to follow.\n- The experiments are comprehensive, conducted across various tasks and diverse datasets, demonstrating the method's effectiveness and robustness. Ablation studies are also thoroughly conducted on datasets from different tasks to better illustrate the contribution of each component.\n- The idea of combining SAM and inpainting to generate images without task-related objects to eliminate hallucinations is intriguing and could be adapted to more works aiming to reduce hallucinations.", "weaknesses": "- The authors demonstrate the outstanding performance of the proposed ProMaC method across various tasks. Could they provide comparison results between the predictions of the ProMaC method and the ground truth across iterations? This would offer a more visual representation of the method's performance.\n- The proposed method's computational and memory requirements aren't clearly discussed, which might be significantly higher due to the iterative nature of the Prompt-Mask Cycle and multiscale chain of thought prompting. This could limit its applicability in resource-constrained environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive evaluation of our work and for acknowledging the value and insight of the proposed method.\n\n> *Could they provide comparison results between the predictions of the ProMaC method and the ground truth across iterations? This would offer a more visual representation of the method's performance.*\n\nOn Page 16, in Fig. 6, we provide visualizations of segmentation results across different tasks as iterations progress. These results demonstrate that our method shows consistent performance improvement with each iteration across various tasks. Additionally, the quantitative experimental results in Tab. 6 further corroborate this finding, illustrating the incremental enhancement in performance as the iterations proceed.\n\n> *The proposed method's computational and memory requirements aren't clearly discussed, which might be significantly higher due to the iterative nature of the Prompt-Mask Cycle and multiscale chain of thought prompting. This could limit its applicability in resource-constrained environments.*\n\nTraditional methods often require multiple GPUs for extensive training, leading to higher computational and memory demands. In contrast, as we discussed in Line 281, our method performs iterative test-time adaptation on a single 40G A100 GPU without the need for training, which is computationally efficient. Compared to GenSAM, which also uses test-time adaptation, our method achieves better performance with fewer iterations. As shown in Tab. 6(a), ProMaC reaches higher performance by the second iteration than GenSAM does after six iterations, further demonstrating the efficiency of our approach.\n\n> *Dependence on Initial Prompts: While the iterative refinement process is beneficial, the initial quality of prompts still plays a crucial role. Poor initial prompts might lead to suboptimal starting points, impacting the overall effectiveness of the adaptation process. I would like to see some discussion on this.*\n\nThe format of the initial prompt is fixed, with the only modifiable part being the task-generic prompt $P_g$. However, $P_g$ has minimal impact on experimental results. Using the camouflaged animal detection task as an example, we selected \"camouflaged animal\" as $P_g$ because the task is named \"camouflaged animal detection\". To explore the impact of different $P_g$ prompts, we used ChatGPT to suggest two synonyms for \"camouflaged animal\": \"hidden animal\" and \"disguised animal\" as possible $P_g$ candidates. We evaluated the effect of different $P_g$ on the results, as shown in Tab. 2 of the attached PDF file. Although different $P_g$ prompts cause slight fluctuations in the outcomes, the overall performance remains stable. This demonstrates that as long as $P_g$ effectively describes the task semantics, our method can achieve relatively consistent segmentation performance.\n\n>*The authors attribute this to the inherent generalization limitations of LLaVA. Recently developed methods (e.g., GPT4o) have shown better generalization capabilities in medical imaging. Could the authors conduct experiments using these more robust MLLMs on specific sub-tasks to substantiate this perspective?*\n\nIn Tab. 8 on Page 16, we compare the performance of our method with different baseline methods using various MLLMs across different tasks. It is evident from the comparisons among the different baselines that the generalization capability of an MLLM directly impacts its performance on different tasks, which supports our explain. Furthermore, we evaluate an MLLM model based on LLaVA-Med [1] on the Polyp Image Segmentation task, as shown in Tab.1 in attached pdf file, which also support this claim.\n\n[1] Li C, Wong C, Zhang S, et al. Llava-med: Training a large language-and-vision assistant for biomedicine in one day[J]. Advances in Neural Information Processing Systems, 2024, 36.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "za7Fwh78Xh", "reviewer": "Reviewer_u8FA", "summary": "The paper focuses on promptable segmentation, aiming to minimize the need for manually designing prompts. Specifically, it explores the hallucination issue in MLLMs and finds that hallucinations can reveal valuable contextual information, which could be largely beneficial to promptable segmentation tasks, especially for complex tasks such as camouflaged object segmentation. To this end, the paper proposes a Prompt-Mask Cycle generation framework (ProMaC) that iteratively refines the generated prompts and masks. The proposed method leverages hallucinations to mine more context-related information while simultaneously reducing irrelevant hallucinations for better segmentation results.", "questions": "Overall, the idea of this paper is quite interesting. However, it lacks some necessary explanations about the motivation and the proposed method. For example, the motivation is primarily illustrated using examples from camouflaged object segmentation. More diverse examples could help illustrate its broader potential. Additionally, some terms are unclear, suggesting a need for clearer definitions and possibly a background section explaining the foundational concepts.", "limitations": "The author has discussed the limitations of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of this paper is quite interesting. Instead of merely mitigating hallucinations, the paper explores how to leverage them for better segmentation performance. The paper also provides detailed analysis to explain its motivation and illustrate how hallucinations can benefit the segmentation task.\n\n2. Extensive experiments over diverse segmentation benchmarks demonstrate the effectiveness of the proposed method in tackling various segmentation tasks, including very challenging camouflage segmentation and general segmentation tasks.", "weaknesses": "Motivation\n\n1. The author illustrates the motivation mainly using samples from the camouflaged object segmentation task (in Figure 1 and A.2). This makes it quite straightforward to understand the rationale behind the proposed method and how it can benefit the camouflaged object segmentation task. However, it would be better to involve more examples from different segmentation tasks to illustrate the motivation comprehensively, especially general segmentation like COCO and Pascal VOC. I am curious whether this phenomenon would still occur when the objects are more visible\n\n\nMethod\n\n1. Some terms are unclear and need explanation. For example, I’m not very clear about the meaning of the term ‘prompt’ used in the paper. Does it refer to text prompts or spatial prompts like boxes or points? From the method section, it seems that P_B and P_A are text prompts, and the instance-specific prompts refer to bounding boxes? Also, what does query P refer to? If its meaning differs from P_g, P_B, and P_A, it would be better to clarify this at the beginning of the method section. Since the method is based on prior work [19] and some terms are borrowed directly from [19], it is necessary to include a preliminary section to explain the background and terms. This would make the method section easier to follow.\n\n\n2. Question about the proposed Multi-scale Chain of Thought Prompting. From my understanding, chain-of-thought prompting aims to achieve complex reasoning capabilities through multiple intermediate reasoning steps. Could the author please explain how this idea is reflected in the proposed method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive evaluation of our work and for acknowledging the originality of the proposed method as well as its significance for future research. We are glad that the reviewer finds the results impressive and the idea of this paper is quite interesting.\n\n> *It would be better to involve more examples from different segmentation tasks to illustrate the motivation comprehensively, especially general segmentation like COCO and Pascal VOC. I am curious whether this phenomenon would still occur when the objects are more visible.*\n\nWe have also included additional samples for the polyp dataset and general object dataset in the attached PDF (Fig. 1) to illustrate our motivation. It is evident that our method, although specifically designed for segmentation tasks where visual cues are weak or ambiguous (e.g., hidden/camouflaged foreground objects in visually similar backgrounds), can also utilize hallucinations as prior knowledge to identify potential categories in complex images in general task, such as those in the VOC dataset (Fig. 1(b) in the attached PDF). However, when visual cues are strong, obvious, or distinct, the benefit of scene understanding through exploring hallucinations becomes less critical. In such situations, there is less likelihood of confusion between foreground and background, and less need for reasoning about all plausible backgrounds versus foregrounds, in such cases, our method may not provide significant improvements. (Fig. 1(c) in the PDF).\n\n> *Some terms are unclear and need explanation. For example, I’m not very clear about the meaning of the term ‘prompt’ used in the paper. Does it refer to text prompts or spatial prompts like boxes or points? From the method section, it seems that P_B and P_A are text prompts, and the instance-specific prompts refer to bounding boxes? Also, what does query P refer to? If its meaning differs from P_g, P_B, and P_A, it would be better to clarify this at the beginning of the method section. Since the method is based on prior work [19] and some terms are borrowed directly from [19], it is necessary to include a preliminary section to explain the background and terms. This would make the method section easier to follow.*\n\nThe term 'prompt' used in the paper includes both text prompts and spatial prompts (bounding boxes and points). \n\n- As explained in Line 152-155, in each task, the input text prompt consists of two parts: prompt for bounding box prediction ($P_B$) and prompt for class prediction ($P_A$):\n\n1. Prompt for bounding box prediction ($P_B$) instructs: \"This image is from the $P_g$ detection task, output the bounding box of the $P_g$.\"\n\n2. Prompt for class prediction ($P_A$) states: \"Output the name of the $P_g$ and its environment in one word.\"\n\nHere, $P_g$ is a test-generic prompt that is consistent across tasks. For example, for camouflaged animal detection task, $P_g$ is \"camouflaged animal\", for polyp detection task, $P_g$ is \"polyp\". \n- $P_B$ and $P_A$ are fed into MLLM to infer instance-specific spatial prompt bounding box and instance-specific text prompt class name. This class name is then mapped into anther instance-specific spatial point prompts using spatial CLIP. These instance-specific spatial prompts (both point prompts and bounding box prompts) are subsequently fed into SAM to guide the segmentation process.\n\n- We refer to the queries inputted into the MLLM as P. \n- The terms borrowed from [19] will be explained in more detail in the final version.\n\n> *Question about the proposed Multi-scale Chain of Thought Prompting. From my understanding, chain-of-thought prompting aims to achieve complex reasoning capabilities through multiple intermediate reasoning steps. Could the author please explain how this idea is reflected in the proposed method?*\n\nAs we explained in Line 143-190, our MCoT uses multiple intermediate reasoning steps on various chains to infer accurate instance-specific prompts from task-generic prompts.\n- First, as we mentioned in Line 146-156, each chain includes two intermediate reasoning steps: 1. MLLM first captions the image and 2. Based on this captions, MLLM infers the names and backgrounds of task-relevant objects. Without this initial image captioning step, the inferred instance-specific prompts would be inaccurate.\n- Second, The input image scales differ across various chains, leading to variations in the positions and completeness of task-relevant objects in each chain. This difference results in distinct intermediate reasoning paths, ensuring diverse information extraction across chains. This variation allows the MLLM to fully leverage prior knowledge from hallucinations to uncover potential task-relevant information.\n- Finally, the visual contrastive reasoning module aggregates effective information from different chains, eliminating the influence of task-irrelevant hallucinations. This multi-chain, multi-step reasoning process ultimately produces accurate instance-specific prompts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1g49Qlpo1u", "reviewer": "Reviewer_E6it", "summary": "The paper introduces an innovative approach called ProMaC (Prompt-Mask Cycle) to improve promptable segmentation. The primary goal of ProMaC is to reduce the dependency on instance-specific manual prompts by leveraging hallucinations from Multimodal Large Language Models (MLLMs) to generate more accurate and task-specific prompts.", "questions": "see the weakness", "limitations": "see the weakness", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Innovation in Utilizing Hallucinations: The paper's novel approach of leveraging rather than eliminating hallucinations to improve segmentation tasks is groundbreaking. It recognizes the potential of hallucinations to provide contextual insights, which can be valuable for enhancing model performance.\n\nReduction of Manual Effort: By introducing task-generic prompts and iteratively refining them, ProMaC significantly reduces the need for manual annotation, making it more feasible for large-scale applications.\n\nIterative Refinement Process: The iterative approach of ProMaC, which continuously improves prompts and masks, ensures higher accuracy and adaptability across various tasks and datasets.\n\nComprehensive Evaluation: The paper provides extensive evaluations on multiple benchmarks, demonstrating the robustness and effectiveness of ProMaC in diverse and challenging scenarios.\n\nAdaptability and Versatility: ProMaC's success across different segmentation tasks, including those in medical imaging and transparent object detection, highlights its versatility and potential for broad application.\n\nOpen-Source Contribution: The inclusion of code in the supplemental materials encourages further research and development, facilitating community engagement and collaboration.\n\nIn summary, the paper makes a significant contribution to the field of promptable segmentation by introducing a novel method that leverages MLLM hallucinations, reduces manual dependency, and demonstrates superior performance across various challenging tasks.", "weaknesses": "Reliance on Hallucinations:\n\nInconsistent Performance: The approach relies on hallucinations, which can be unpredictable and inconsistent. In scenarios where hallucinations are highly inaccurate, the performance of ProMaC could degrade significantly.\nDependence on MLLM Quality: The effectiveness of leveraging hallucinations is heavily dependent on the quality and training of the Multimodal Large Language Models (MLLMs). Variations in the MLLM's training data and methodology can lead to differing hallucination patterns, affecting ProMaC's reliability.\nComplexity of the Iterative Process:\n\nComputational Overhead: The iterative process of refining prompts and masks can be computationally intensive. This complexity might limit the scalability of ProMaC for real-time applications or large datasets.\nImplementation Challenges: The method's implementation is intricate, involving multiple stages of hallucination generation, contrastive reasoning, and mask refinement. This complexity can pose challenges for practitioners looking to adopt the method.\nGeneralization Concerns:\n\nTask-Specific Fine-Tuning: While ProMaC aims to reduce manual prompt dependency, the initial setup still requires careful selection of task-generic prompts. Ensuring that these prompts are effective across diverse tasks without additional fine-tuning could be challenging.\nTransferability: The approach might not generalize well to entirely new tasks or domains where the hallucinations generated by MLLMs do not align well with the actual task requirements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for valuable feedback and for appreciating the idea of our method, strong justification of each component of our framework and extensiveness of our experiments. Following are the responses regarding your concerns.\n\n> *Inconsistent Performance: In scenarios where hallucinations are highly inaccurate, the performance of ProMaC could degrade significantly.*\n\n- Hallucinations are used solely to gather task-relevant information from the image and are effectively filtered by the Visual Contrastive Reasoning module and the subsequent Mask Semantic Alignment model to eliminate the negative impact of task-irrelevant information. This ensures the generated instance-specific prompts are accurate and not influenced by irrelevant hallucinations, maintaining robust performance even when hallucinations are unreliable.\n\n- To elaborate, as stated in Lines 143-150, by dividing the input image into patches at various scales, MLLM induces hallucinations based on varying object visibility. These hallucinations use prior knowledge to explore connections between the image data and the associated task, gathering task-relevant information. Simultaneously, the original image is also fed directly into the MLLM to obtain task-relevant candidates without relying on hallucinations. This ensures that even when hallucinations are highly inaccurate, the collected information still includes relatively reliable content.\n\n- Visual Contrastive Reasoning module then selectively reduces irrelevant semantics to remove irrelevant influences while benefiting from relevant insights obtained from pre-trained knowledge (hallucinations). When hallucinations are completely inaccurate, the information derived from them is discarded. Additionally, the Mask Semantic Alignment module further ensures that the generated masks align with the task semantics.\n\n> *Dependence on MLLM Quality.*\n\nAs we explained in the limitations section, the quality of MLLMs affects our model's generalization ability, which is a potential direction for our future research.\n\n> *Computational overhead and implementation challenges.*\n\n- Computational Overhead: Traditional methods often require multiple GPUs for extensive training, leading to higher computational and memory demands. In contrast, as we discussed in Line 281, our method performs iterative test-time adaptation on a single 40G A100 GPU without the need for training, which is computationally efficient. Compared to GenSAM [19], which also uses test-time adaptation, our method achieves better performance with fewer iterations. As shown in Tab. 6(a), ProMaC reaches higher performance by the second iteration than GenSAM does after six iterations, further demonstrating the efficiency of our approach.\n\n- Implementation Challenges: Although our approach uses multiple modules, all module parameters are fixed and do not require training, making it easier to reproduce. In Tab. 7, we conducted three sets of experiments randomly and calculated the mean and variance. The results show that our training-free method exhibits very low variance, further proving its robustness. Additionally, we provide implementation details in the PyTorch Implementation Details section (Line 271-281) of the main text and Line 551-581 of the appendix. Our code is also available in the supplementary materials, ensuring that our method is easy to understand and implement.\n\n> *The initial setup still requires careful selection of task-generic prompts. Ensuring that these prompts are effective across diverse tasks without additional fine-tuning could be challenging.*\n\n- Task-generic prompt $P_g$ is decided by the task name rather than our manual selection, and has minimal impact on experimental results. Using the camouflaged animal detection task as an example, we selected \"camouflaged animal\" as $P_g$ because the task is named \"camouflaged animal detection\". Similarly, \"polyp\" is selected as $P_g$ because the corresponding task is named as \"polyp detection\". \n- To explore the impact of different $P_g$ prompts, we used ChatGPT to suggest two synonyms for \"camouflaged animal\": \"hidden animal\" and \"disguised animal\" as possible $P_g$ candidates. We evaluated the effect of different $P_g$, as shown in Tab. 2 of the attached PDF file. Although different $P_g$ prompts cause slight fluctuations in the outcomes, the overall performance remains stable. This demonstrates that as long as $P_g$ effectively describes the task semantics, our method can achieve relatively consistent segmentation performance.\n\n> *Transferability: The approach might not generalize well to entirely new tasks or domains where the hallucinations generated by MLLMs do not align well with the actual task requirements.*\n\n- Because our ProMaC can be easily applied to different MLLMs, we can ensure robust performance across various tasks and domains by selecting the most appropriate MLLM for each specific task. \n\n- PoMaC is designed to be flexible and can be adapted to different MLLMs that are more suitable for specific tasks or domains. By selecting an MLLM that has been pre-trained on data relevant to the target domain, we can improve alignment with task requirements and enhance performance.\n\n- For example, as shown in Tab. 1 in attached pdf, in medical imaging segmentation, ProMaC based on LLaVA performed modestly. However, when we used LLaVA-Med [1], which has stronger medical characteristics, as the base, ProMaC significantly outperformed both LLaVA-Med combined with SAM and LLaVA-based ProMaC. This demonstrates that by applying our ProMaC method to a more suitable base MLLM, we can achieve better performance, thereby ensuring transferability across various tasks.\n\n[1] Li C, Wong C, Zhang S, et al. Llava-med: Training a large language-and-vision assistant for biomedicine in one day[J]. Advances in Neural Information Processing Systems, 2024, 36.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their uniformly positive evaluations and valuable feedback. We appreciate fruitful suggestions of the reviewers that helped to improve the overall presentation of our work. We are encouraged by the positive comments from reviewers for the following: (i) The idea of this paper is quite interesting (Reviewers u8FA, fu4N, E6it, E5Cz) (ii) extensive experiments over diverse segmentation benchmarks demonstrate the effectiveness of the proposed method (Reviewers u8FA, fu4N, E6it) and (iii) the writing quality and the clarity of the presentation of the ideas (Reviewer fu4N). Additionally, as mentioned by reviewer E6it, we emphasize that another major strength of our algorithm is that it can reduction of manual effort and have good adaptability and versatility.

In the response to the reviewers’ feedback, we have conducted additional experiments and motivation visualizations shown in the attached pdf. The attached pdf includes:

- Fig. 1: More motivation visualization on medical image and general image tasks.
- Tab. 1: ProMaC performance on the Polyp Image Segmentation task using LLaVA-Med.
- Tab. 2: Performance comparison of ProMaC with different task-generic prompts $P_g$.

We provide more details and address reviewers’ comments in the individual response to reviewers. We hope that our detailed response and additional experiments will help to increase reviewers’ confidence.

Meta-review: Paper was reviewed by four expert reviewers and, post-rebuttal, received uniformly positive scores; mainly: 3 x Weak Accepts and 1 x Accept. While prior to rebuttal reviewers raised some concerns with the work, those concerns have been resolved in rebuttal and discussion. Specifically, [fu4N, E6it, u8FA] acknowledge that "all concerns have been well addressed", while [E5Cz] mentions that "most of  concerns have been addressed".  Overall, all reviewers agree that the approach is novel, interesting and the paper is well written. AC agrees with the reviewers and believes that the paper will be a welcome addition to NeurIPS. Therefore the decision is to Accept the paper. Authors are encouraged to incorporate elements of the rebuttal and discussion into the paper for the camera ready.
Decision: Accept (poster)