Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation

Authors: jian  hu
jiayi  lin
junchi  yan
shaogang  gong
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-27 23:49:51

Abstract: Promptable segmentation typically requires instance-specific manual prompts to guide the segmentation of each desired object. To minimize such a need, task-generic promptable segmentation has been introduced, which employs a single task-generic prompt to segment various images of different objects in the same task. Current methods use Multimodal Large Language Models (MLLMs) to reason detailed instance-specific prompts from a task-generic prompt for improving segmentation accuracy. The effectiveness of this segmentation heavily depends on the precision of these derived prompts. However, MLLMs often suffer hallucinations during reasoning, resulting in inaccurate prompting.  While existing methods focus on eliminating hallucinations to improve a model, we argue that MLLM hallucinations can reveal valuable contextual insights when leveraged correctly, as they represent pre-trained large-scale knowledge beyond individual images. In this paper, we first utilize hallucinations to mine task-related information from images and verify its accuracy to enhance precision of the generated prompts.  Specifically, we introduce an iterative \textbf{Pro}mpt-\textbf{Ma}sk \textbf{C}ycle generation framework (ProMaC) with a prompt generator and a mask generator.  The prompt generator uses a multi-scale chain of thought prompting, initially leveraging hallucinations to extract extended contextual prompts on a test image. These hallucinations are then minimized to formulate precise instance-specific prompts, directing the mask generator to produce masks that are consistent with task semantics by mask semantic alignment. Iteratively the generated masks induce the prompt generator to focus more on task-relevant image areas and reduce irrelevant hallucinations, resulting jointly in better prompts and masks. Experiments on 5 benchmarks demonstrate the effectiveness of ProMaC. Code is in https://lwpyh.github.io/ProMaC/.

Main paper: 
 introduction: 
 Current promptable segmentation methods rely on instance-specific manual prompts to guide segmentation, greatly limiting its large-scale application. Recently, a manual-free task-generic promptable segmentation approach was introduced [21] : only a single task-generic prompt is needed for all samples under the same task, e.g., "camouflaged animal" is a task-generic prompt for all images in a camouflaged object detection task. The model segments task-relevant objects in various images based on this generic prompt, significantly reducing the annotation workload. A task-generic prompt is both coarse and potentially ambiguous, can result in poor segmentation when directly applied. To address this problem, existing methods [21, 38] utilize the prior knowledge embedded in Multimodal Large Language Models (MLLMs) to infer more detailed, instance-specific prompts, such as bounding boxes or keywords, to guide the segmentation. However, these MLLMs often generate hallucinations due to object co-occurrence priors [68, 65] , mistakenly predicting non-existent elements based on the environment as instance-specific prompts (Fig. 1(a) ). This can mislead segmentation and degrade model performance. While it is common to consider MLLM's hallucinations as detrimental and should be eradicated [52] , this phenomenon actually demonstrates a MLLM's significant capacity for contextual inference based on prior training. We want to explore MLLM hallucinations as a valuable untapped knowledge resource for scene understanding, critical in complex segmentation scenarios. In practice, when task-related objects are not prominently visible, hallucinations can fill in missing information with plausible predictions based on learned patterns of association. Moreover, they can also extend beyond these familiar patterns, exploring and identifying new relationships within the data that were not explicitly taught during training. This dual ability to replicate and innovate makes hallucinations a valuable asset for enhancing model performance in complex or new situations. This predictive reasoning capacity not only fills perceptual gaps but also enriches the model's understanding, as hallucinations utilize prior knowledge to replicate and discover new patterns, enhancing insight into the target domain (see Fig. 1(b )). Despite the potential benefits, using hallucinations to extract useful information from images to aid task remains unexplored.In this work, instead of direct eliminating hallucinations, we utilize them as prior knowledge to mine extended task-related information from a given test image, performing scene understanding on the image before segmentation, then systematically reduce irrelevant hallucinations iteratively by visual masking verification, optimizing jointly instance-specific prompts and masks. To this end, we introduce an iterative, training-free Prompt-Mask Cycle Generation method (ProMaC) that refines segmentation through cyclic interactions between a prompt and mask generator (see Fig. 2 ). The prompt generator uses a multi-scale chain-of-thought prompting mechanism, which utilizes hallucinations to hypothesize and visual masking to verify, thereby creating more accurate instance-specific prompts. We trigger the hallucinatory tendencies of MLLMs, the process starts by dividing the image into patches at different scales and positions. Such partial visibilities of objects facilitate MLLMs to hypothesize potential object semantic labels and visual locations based on its prior knowledge. For validating the correctness of these hypotheses, we formulate a visual contrastive reasoning mechanism to generate contrastive images that contain only the background without any potential task-related objects. This helps identify all possible co-occurrence hallucinations caused by the background. By comparing these contrastive images with the original images, the MLLM effectively distinguishes between accurate hypotheses and those influenced by misleading prior knowledge, leading to more reliable prompts. Given the current promptable segmentation models' strength at mask prediction but struggle with label prediction, the mask generator uses mask semantic alignment to ensure that the produced masks align with the task semantics. These aligned masks not only serve as outputs but also guide the prompt generator in subsequent cycles, enhancing both prompt and mask quality continuously. Our contributions are three-folds: 1). We introduce a training-free Prompt-Mask Cycle Generation (ProMaC) to perform two tasks: Explore MLLM hallucinations as prior knowledge to enhance contextual scene understanding on each test image; systematically reduce irrelevant hallucinations to verify iteratively and optimize jointly both generated prompts and visual masking in object segmentation. 2). We formulate an iterative optimization method including a prompt generator and a mask generator. To improve prompt relevance, the prompt generator utilizes a multi-scale chain of thought approach. It first leverages hallucinations to expand task-related plausible prompts, then applies visual contrastive reasoning to validate and reduce irrelevant prompts. ProMaC's mask generator overcomes SAM's shortcomings in label prediction by creating masks that align semantically with generated prompts.3). Comprehensive comparative evaluations on 5 different segmentation tasks with 12 diverse datasets against 22 existing models demonstrate the effectiveness of ProMaC. 
related works: 
 Promptable Segmentation refers to object segmentation with active interactions from user inputs. Interaction methods vary from points, boxes, to scribbles. SAM [29] , AV-SAM [41] , Ground-ingSAM [38] and SEEM [71] accept video, audio, and multimodal inputs. However, they often rely on manual prompts, which can be unclear and subjective. Even with these prompts, they typically excel only in specific tasks. To address this issue, GenSAM [21] introduces a manual-free promptable segmentation setting, where only one task-generic prompt is provided. This prompt can be applied to all images within the task for instance-specific segmentation without any additional manual prompting. GenSAM primarily utilizes MLLM to infer the names of task-related objects in the images and uses them as instance-specific prompts for SAM to guide segmentation. However, GenSAM lacks spatial information about objects and may lead to inaccurate prompt predictions in complex scenes.Hallucinations in MLLMs refers to models generate content that does not exist in the input data [65] . This issue often arises from the models leveraging extensive prior training rather than just the immediate input, leading to false predictions on fine-grained details. There are some efforts to mitigate this problem, including refining training processes [55, 44] and improving model architectures [4] . Other efforts focus on aligning model outputs more closely with actual data, employing feedback mechanisms for real-time adjustments [54] . While current works focus on eliminating hallucinations to enhance performance [31, 64] , our work explores how to utilize hallucinations to expand and reason plausible context and validate them iteratively to remove irrelevant generalizations.Visual Marking for MLLMs has been explored in recent research to prompt MLLMs through manipulation of visual inputs: (i) adding learnable soft tokens to visual inputs for efficient parameter tuning [1, 28] , (ii) using image sequences as demonstrations of a new task [2, 9] , and (iii) overlaying visual markers like masks, boxes, and circles onto visual inputs to ground regions [61, 54] . Our work falls into the third category, employing visual guidance for reasoning. Yang et al. [59] propose set-of-mark (SoM) prompts, where images are segmented and numbered regions to improve GPT-4V [43] visual grounding. However, as detailed in Tab.5, we confirm previous findings [5] that this visual marker approach struggles with open-source MLLMs like LLaVA. Instead of proprietary models [54] or fine-tuning [5, 8, 23] , our training-free ProMaC uses inpainting task-related regions and contrasting model output distributions to prompt MLLMs. 
methodology: 
 We introduce ProMaC, a cycle-generation method for segmenting unknown multiple classes of objects training-free with only a single task-generic prompt. Specifically, given an image X ∈ R H×W ×3 from a test set, ProMaC employs a task-generic prompt P g across datasets in the same task to produce ). Subsequently, it uses visual contrastive reasoning to refine and finalize instance-specific prompts (A u i , B u i ) by eliminating hallucinations. The mask generator then processes these prompts into the segmentation model ("Seg"), producing a mask aligned with task semantics. This mask further guides the visual contrastive reasoning process, which leverages an inpainting model to eliminate masked regions, creating contrastive images. These images enable the prompt generator to further refine its prompts, enhancing segmentation accuracy. a final segmentation mask M ∈ R H×W , thereby removing the need for individual supervision for each image. The prompt generator leverages prior knowledge gained to reason and deduce instancespecific prompts, which then guide a mask generator to create masks aligned with task semantics. These masks act both as the current segmentation outcome and as visual markers for generating subsequent prompts. Training-free ProMaC relies solely on test-time adaptation.Prompt generator employs MLLMs to generate instance-specific prompts based on image content and prior knowledge. It transforms the general prompt P g , into an instance-specific prompt for each individual instance, providing more detailed descriptions of task-relevant objects. MLLM with parameters θ receives an image X and query P as inputs. X provides contextual visual information to assist the model in generating a relevant response y to the query P . The response y is sampled auto-regressively from the probability distribution conditioned on P and X as follows:EQUATIONwhere y t denotes the token at time step t, and y < t represents the sequence of generated tokens up to the time step (t -1). In practice, predicted task-relevant objects can often blend into the background due to texture, color, size, or position, leading to inaccuracies in instance-specific prompts. To address this problem, we explore MLLM hallucinations as contextual prior knowledge from pretraining, rather than eliminate them. These hallucinations are particularly useful when direct visual cues are absent or ambiguous, helping the model fill in information gaps and hypothesize potential task-related elements within the image that are not prominent. By revealing these often-overlooked subtle associations, hallucinations provide a more comprehensive scene understanding of the image content. This deeper contextual understanding provide a reasoning context for generating more accurate and relevant instance-specific prompts candidates.Thus, using hallucinations to uncover task-related knowledge helps overcome challenges from visual ambiguities and object camouflage in complex scenes. To this end, we propose a multi-scale chain-of-thought prompting strategy that stimulates hallucinations to leverage prior knowledge, fully extracts task-relevant information, and then uses this information to enhance the precision of the generated instance-specific prompts.Multi-scale Chain of Thought Prompting consists of two processes: Gathering candidate knowledge and generating accurate instance-specific prompts. To efficiently collect task-relevant information from an image, as shown in Fig. 3 , we divide the input image into patches at various scales by cutting horizontally, vertically, or by leaving it whole. These patches are then processed by the MLLM to gather preliminary instance-specific prompts. The differing levels of task-relevant object visibility in each patch prompt the MLLM to induce hallucinations. These hallucinations utilize prior knowledge to explore connections between the image data and the associated task, aiding in the detection of potential bounding boxes and object names. The process is computed by:EQUATIONwhere C k is the caption generated by MLLM for the k-th image patch X k . P g is task-generic prompt.For bounding box prediction, the prompt P B , which instructs "This image is from the P g detection task, output the bounding box of the P g .". This guides the MLLM to predict the bounding box B k of the task-related objects within the patch. For predicting name, the prompt P A , stating "Output the name of the P g and its environment in one word." is used, guiding the MLLM to predict the names of the task-related objects A k fore and their backgrounds A k back from each patch. The preliminary data, including object names A k fore and bounding boxes B k , gathered from various patches, are compiled into candidate lists A i and B i . Here, i denotes the iteration in the iterative learning cycle. In this process, the hallucinations employed are essentially based on object co-occurrence priors, where objects commonly associated with background elements during pre-training are predicted to be task-relevant, even if they are not present in the current image. This prior knowledge is useful during the knowledge collection stage as it uncovers implicit relationships and details in the image. However, it can also reduce accuracy of the later fine-grained instance-specific prompts generation. Therefore, it is crucial to control these hallucinations in the latter stage to prevent incorrect predictions.Visual Contrastive Reasoning. To mitigate hallucinations caused by object co-occurrence priors, recent research highlights particularly relevant regions of an image to direct MLLMs focus toward task-related elements, thereby minimizing background interference and enhancing model accuracy [59, 54] . To achieve this, visual markers are employed to steer MLLM attention on task-relevant visual regions, thereby reducing hallucinations. While closed-source MLLMs like GPT-4V [43] can interpret these markers effectively, they are costly and large. In contrast, models like LLaVA [37] are open-source, but cannot process visual markers such as points or bounding boxes, and employing these markers might disrupt the original pixel data, degrading performance on LLaVA (see Tab. 5). Moreover, accurate pixel-level visual markers are unavailable in our setting. To solve this problem, we aim to enable LLaVA to focus on task-related regions without altering the original pixel data, thereby effectively minimizing hallucinations and enhancing the precision of instance-specific prompts.Despite the absence of instance-level annotations, promptable segmentation models produce masks with detailed textures, which provide rich positional and textural information about interested regions. We use these masks as visual markers to guide a MLLM to focus on task-related areas during the generation of instance-specific prompts. Inspired by classifier-free guidance [20, 47] , we introduce visual contrastive reasoning (VCR), a training-free visual marking method to help MLLM focus on specific regions, reducing hallucinations. The relevance of a region is assessed by observing MLLM output changes when key areas are excluded. It guides the MLLM to focus on areas with notable changes (bottom of Fig. 3 ). Based on Eq. ( 1), we derive a probability distribution by comparing original image X with a modified image, X ′ = process(X, IM), where region IM is excluded.EQUATION)where α is the level of focus on region IM. A higher α increases emphasis on that region. Following [54] , we set α = 1 in all tasks. It preserves the integrity of the original image pixels X, while constructing contrastive samples X ′ that encourage the model to focus on task-related regions. Ideally, X ′ should exclude task-related objects while maintaining a uniform appearance and overall context with the original image. But directly marking X ′ disrupts its pixels, making contrastive sample generation challenging.Contrastive Sample Generation. To address it, we employ inpainting, where the mask M i-1 obtained from the previous iteration segmentation is treated as the inpainting mask IM i to guide the creation of X ′ . We use a negative prompt P n : "A fore i , is a P g ", to ensure that the inpainted X ′ does not contain potentially task-related objects A fore i . Additionally, we use a positive prompt P p : "A back i , high quality, detailed, blended to the original image.", to ensure consistency between the generated portion and the surrounding background A back i . The corresponding inpainting is defined as:EQUATIONwhere F in represents the inpainting module, and we choose Stable Diffusion to perform this operation. This method ensures the generated X ′ excludes task-related objects without disrupting the pixel continuity. In the first iteration, since IM i does not yet exist, we use bounding box predictions from various patches B i as an alternative. As X ′ contains only the background, comparing it with X eliminates co-occurrence hallucination caused by the background and highlights differences in taskrelated regions, subtly guiding the model to focus on these areas. Finally, we use visual contrastive reasoning to identify accurate instance-specific prompt to guide segmentation as follows,EQUATIONwhere VCR represents our visual contrastive reasoning, and C is the caption of the image. The collected knowledge, Ai and Bi, is integrated into the prompt P A and P B . This process aids in identify the ultimate instance-specific names A u i and bounding boxes B u i of the objects.Until now, we described how to use SAM-generated masks as a visual marker to guide the model to focus on task-relevant areas for generating accurate instance-specific prompts. But this method relies on an assumption that the mask accurately delineates task-related regions. However, SAM is trained on large-scale prompt-mask pairs without category labels, it excels at identifying masks based on image textures but lacks label prediction capabilities. Consequently, the SAM-generated mask may not always align with task semantics, yet such alignment is crucial for our method.We need to utilize texture generalization capabilities of SAM to describe possible task-related objects within the prompt-targeted areas, while also ensuring that the generated masks align with the task semantics. To achieve this, we divide the input image into patches of varying scales using horizontal, vertical, and uncut divisions as outlined in the last section. these processed patches are then reintegrated onto the original image with surrounding areas blacked out, and fed into SAM to ensure it focuses exclusively on the patch. Finally, masks generated from different patches are aggregated based on their relevance to task semantics, providing an accurate representation of task-related objects.The masks for each patch is generated as follows,EQUATIONwhere mask m k i is obtained by inputting the corresponding image patch X k i and associated prompts into SAM during the i-th iteration. Following [21] , Spatial CLIP maps the text prompt A u i to regions in the image X i that correspond to the content of the prompt. The processed images, along with the generated instance-specific text prompts A u i , are then input into CLIP to assess semantic similarity.EQUATIONthe operation ⊙ results in retaining only those parts of X i that are covered by the predicted mask. s(m k i ) represents the similarity between masked image and A u i , calculated using CLIP. The similarity scores obtained from different patches are denoted asS i = [s(m 1 i ), s(m 2 i ), . . . , s(m k i )]. After normalizing the elements within S i , the closer the normalized s(m k i ) is to 1, the more semantically aligned m k i is with the instance-specific text prompt A u i . Finally, we compute the weighted sum of the normalized s(m k i ) and m k i as follows.EQUATIONM i is the output mask of the i-th iteration of X. The generated M i leverages SAM's mask prediction capabilities to create highly detailed masks. Simultaneously, through the mask semantic alignment process, it ensures that the output mask aligns with the task's semantics, thereby overcoming the limitation of SAM's mask prediction lacking semantic understanding. The mask is applied to the original image as a weight, to generate the next iteration image X i for segmentation. This excludes irrelevant regions to reduce interference during segmentation.EQUATIONwhere w is a hyperparameter, which we have assigned a value of 0.3. M ↓ Fβ ↑ Eϕ ↑ Sα ↑ M ↓ Fβ ↑ Eϕ ↑ Sα ↑ M ↓ Fβ ↑ Eϕ ↑ Sα ↑Scribble Supervision Setting WSSA [62] CVPR20 0.067 0.692 0.860 0.782 0.118 0.615 0.786 0.696 0.071 0.536 0.770 0.684 SCWS [60] AAAI21 0.053 0.758 0.881 0.792 0.102 0.658 0.795 0.713 0.055 0.602 0.805 0.710 TEL [62] CVPR22 0.073 0.708 0.827 0.785 0.104 0.681 0.797 0.717 0.057 0.633 0.826 0.724 SCOD [17] AAAI23 0.046 0.791 0.897 0.818 0.092 0.709 0.815 0.735 0.049 0.637 0.832 0.733 SAM-S [29] ICCV23 0.076 0.729 0.820 0.650 0.105 0.682 0.774 0.731 0.046 0.695 0.828 0.772 WS-SAM [16] NeurlPS23 0.046 0.777 0.897 0.824 0.092 0.742 0.818 0.759 0.038 0.719 0.878 0.803 Point Supervision Setting WSSA [62] CVPR20 0.105 0.660 0.712 0.711 0.148 0.607 0.652 0.649 0.087 0.509 0.733 0.642 SCWS [60] AAAI21 0.097 0.684 0.739 0.714 0.142 0.624 0.672 0.687 0.082 0.593 0.777 0.738 TEL [62] CVPR22 0.094 0.712 0.751 0.746 0.133 0.662 0.674 0.645 0.063 0.623 0.803 0.727 SCOD [17] AAAI23 0.092 0.688 0.746 0.725 0.137 0.629 0.688 0.663 0.060 0.607 0.802 0.711 SAM [ The mask generated from the last iteration will guide the prompt generator in the next iteration to focus on potential task-related regions, eliminating the erroneous effects of irrelevant hallucinations and generating more accurate instance-specific prompts. These prompts, in turn, help the mask generator produce better masks. Through iterative prompt generation and mask generation jointly, we yield both better instance-specific prompts and visual masks. Finally, the masks from different iterations are averaged, and the mask closest to the mean is considered the final output.EQUATION)Here, I is the number of adaptation epoches and M i * is the corresponding final mask for image X. [51] Kvasir [25] ISIC [10] [43, 29] 0.312 0.104 0.392 0.363 0.288 0.199 0.607 0.512 LLaVA1.5+SAM [37, 29] Image Segmentation (MIS), and Transparent Object Detection (TOD). These tasks are areas where SAM struggle [26] . In the COD task, we compare ProMaC with weakly supervised segmentation methods [29, 62, 60, 62, 17, 17, 22, 24, 18] . Two supervision levels are used for comparison: scribble supervision, where main structures for the foreground and background are sketched during training, and point supervision, where separate points are provided for both foreground and background.M ↓ Fβ ↑ Eϕ ↑ Sα ↑ M ↓ Fβ ↑ Eϕ ↑ Sα ↑ M ↓ Fβ ↑ Eϕ ↑ Sα ↑ GPT4V+SAM [M ↓ Fβ ↑ Eϕ ↑ Sα ↑ M ↓ Fβ ↑ Eϕ ↑ Sα ↑ GPT4V+SAMM ↓ F β ↑ E ϕ ↑ Sα ↑ M ↓ F β ↑ E ϕ ↑ Sα ↑ ✓ ✓ ✓ ✓ 0.In the task-generic prompt setting, we introduce a challenging scenario by only providing a task description as a generic prompt for segmentation. ProMaC integrates LLaVA1.5 [37] with SAM [29] .We also experiment on the MIS and PIS tasks to demonstrate the effectiveness of our method using task-generic prompts compared to previous methods. We assess GPT4V+SAM and LLaVA1.5+SAM in this setting to demonstrate that current MLLM models cannot address it well. We also compare ProMaC against current SOTA promptable segmentation methods to showcase its effectiveness. Next, we evaluate ProMaC on Open-Vocabulary Segmentation (OVS), and compare with leading methods [33, 70, 71, 38, 63, 66, 32, 19] . Our Visual Contrastive Reasoning (VCR) strategy is applicable to other tasks, especially those requiring complex spatial understanding. We used the What's Up spatial reasoning dataset [27] to evaluate how well VCR guides models to focus on task-relevant regions in Spatial Reasoning (SR) task. Our results are the average of three trials.Metric. For evaluating the first three tasks, metrics Mean Absolute Error (M), adaptive F-measure (F β ) [40] , mean E-measure (E ϕ ) [15] , and structure measure (S α ) [13] are used. Lower M or higher values for F β , E ϕ , and S α reflect better performance. Mean Intersection over Union (mIoU) and accuracy measure OVS and SR performance respectively, with higher values indicating better results.PyTorch Implementation Details. For the MLLM models, we utilize LLaVA-1.5-13B for evaluation purposes. For CLIP, we adopt the CS-ViT-B/16 pretrained model. The inpainting model is stablediffusion-2-inpainting. The task-generic prompts for the COD task is "camouflaged animal". The MIS task consists of two sub-tasks: polyp image segmentation and skin lesion segmentation, each with its own task-generic prompts, "polyp" and "skin lesion" respectively. For TOD task, prompt is fixed as "glass". All tasks are optimized using training-free test-time adaptation, with each task iterating for four epochs, except for the polyp image segmentation task, which undergoes six epochs. Since the second epoch, VCR also operates on different patches, ensuring that while the non-inpainted parts still gather information through hallucinations, the inpainted parts eliminate hallucinations to generate accurate candidate prompts. The promptable segmentation methods is the ViT-H/16 version of SAM. Our experiment is conducted on a single NVIDIA A100 GPU. More details are in appendix.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] .Justification: We do not have theoretical proof.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental result reproducibility: 
 Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide our codes in the supplemental materials, we also provide implement details in both main text and the appendix. All of our experiment results are averaged with three trials. 
results and analysis: 
 Results on COD Task. The COD tasks focus on finding animals that blend into their complex surroundings. We evaluated ProMaC on three representative datasets: CHAMELEON [50] , CAMO [30] , and COD10K [14] . As shown in Tab. 1, we compared ProMaC with others that utilize varying levels of supervision. Overall, methods with scribble supervision generally perform better than those with point supervision. Importantly, ProMaC only uses a single generic task prompt for the entire task and it stil outperforms all point-supervised methods. It also surpasses methods with scribble supervision on the CHAMELEON and CAMO datasets, and matches the top-performing scribble-supervised methods on COD10K. It demonstrates the superiority of ProMaC. Visual Marker Strategy. Tab. 6(c) assesses the impact of different visual marker strategies. "None" uses no visual prompts, while "Bbox" places bounding boxes directly on the image. "VCD" employs previous methods that introduce Gaussian noise into comparison images for contrastive reasoning. Results indicate that bounding boxes decrease performance, suggesting LLaVA struggles with this type of markers. Although VCD methods improve performance, they distort pixel data, making them less effective than our approach. Our VCR generates contrastive samples that focus on task-relevant areas without altering the image, reducing hallucinations and enhancing performance.Visualization. Fig. 4 and Fig. 5 visually compares our ProMaC with other methods across 3 tasks and also shows the contrastive images we generated. GenSAM handles clear objects well but struggles with complex background. Although GenSAM performs well in complex backgrounds, but struggles with challenging tasks. ProMaC delivers solid segmentation results across different tasks, and our contrastive images remove task-related regions while maintaining semantic and pixel consistency.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We have released our codes and the instruction on how to run the codes in supplemental materials. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
conclusion: 
 In this work, we introduce an iterative ProMaC that uses MLLM hallucinations to guide automatic prompt generation, significantly improving segmentation without training. This iterative approach aligns masks with task semantics, enhancing model performance. Testing on multiple benchmarks has demonstrated ProMaC's effectiveness in a wide range of complex segmentation tasks. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] . 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] .Justification: We conduct experiments on public datasets, and our model is training-free apporaches, the model we used are all open-sourced model from the community.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] . Justification: we have cited the data and models we used in our paper properly. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] . Justification: We do not introduce new datasets in the papers, and our codes will be released after publication. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 More Explanation on Task-generic Promptable Segmentation Task-generic promptable segmentation, first introduced by [21] , aims to solve a key challenge in promptable segmentation: the requirement for manual prompts, such as bounding boxes, scribbles, or points, for each sample within the same task. This method allows a model, like the Segment Anything Model (SAM), to segment samples based on these prompts. Task-generic promptable segmentation seeks to enable models to automatically infer the task-related objects in different images based on a general task description. This approach eliminates the need to manually annotate every task-related object for different images under the same task.Compared to previous methods, this approach only requires a single task description as a task-generic prompt for a batch of five annotated samples within the same task, significantly reducing the annotation burden and better aligning with practical needs. Although this setting presents greater challenges than past methods, our ProMaC system achieves excellent performance across a variety of tasks. It even surpasses the performance of state-of-the-art (SOTA) methods trained on weakly supervised datasets for camouflaged object detection, demonstrating the robustness and superior performance of our approach.Vertically Split ImageFigure 6 : Left: In the bar chart, we analyze MLLM predictions with two versions of an image: the original (blue) and another with task-related objects removed via inpainting (orange). We then compare their predictions to the ground truth using CLIP similarity on the COD10K dataset. Despite missing key objects, the inpainted image's predictions still somewhat match the ground truth. When we select the higher similarity score from both images as the final score (green), it surpassed that of the original alone. It shows that prior knowledge from hallucinations can also provide useful information for generating prompts. Right: A example of using hallucinations to assist instance-specific prompt generation. Specifically, utilizing hallucination can leverage prior knowledge of image elements to better recognize and locate task-related objects. Directly inputting the image into LLaVA results in the hidden chameleon being incorrectly predicted. Splitting the image results in interested objects being incomplete or absent, prompting LLaVA to induce hallucinations and utilize prior knowledge to uncover potential task-related knowledge within the image. This knowledge assists in final accurately identifying and locating the chameleon.We conducted experiments across multiple datasets and compared our method with various approaches across different tasks. Initially, we benchmarked against methods such as GPT4V+SAM and LLaVA1.5+SAM to demonstrate that even the current state-of-the-art (SOTA) MLLM methods combined with SAM cannot directly address this issue. Here, GPT4 utilized the gpt-4-vision-preview model, and LLaVA1.5 used the LLaVA-1.5-13B model, with SAM employing the ViT-H/16 version. Both models were tested in a single iteration setup, as our experiments showed that multiple iterations resulted in performance degradation as iterations increased. Specifically, these MLLMs were tasked with inferring instance-specific prompts to guide SAM in segmentation. Similarly, in our setup, we fine-tuned the source code of representative promptable segmentation methods such as SEEM, GroundingSAM, and X-Decoder to adapt them to our setting. These models were directly fed task-generic prompts to segment each image, with performance evaluated based on a single iteration of inference. This methodological adjustment allowed us to assess the efficacy of leveraging task-generic prompts in practical segmentation tasks.Our ProMaC system employs LLaVA-1.5-13B and the ViT-H/16 version of SAM for inference, utilizing instance-specific prompts generated by the prompt generator. These prompts include instance-specific text prompts and instance-specific bounding boxes. The instance-specific text prompts are processed by Spatial CLIP, mapped to corresponding image regions, and along with the instance-specific bounding boxes, are input into SAM to guide segmentation. All tasks, except for the PIS task, underwent four epochs of iteration, while the PIS task was iterated over six epochs. In the Camouflaged Object Detection task, the task-generic prompt used was "camouflaged animal". We also compared our results with past weakly supervised learning methods, which require at least one point per image in the training set as supervision. Despite requiring less manual supervision with our task-generic promptable segmentation setting, our method achieved comparable or even better performance. Experiments were also conducted in Medical Image Segmentation and Transparent Object Segmentation tasks, with task-generic prompts "polyp" and "skin lesion" respectively for each task. Notably, the datasets used for the transparent object detection task were exclusively glass, allowing the instance-specific text prompt "glass" to be obtained directly without inference, requiring only the inference of bounding boxes through multi-scale chains of thought prompting. This streamlined approach emphasizes the efficiency and adaptability of our system across varied segmentation tasks. M ↓ F β ↑ E ϕ ↑ S α ↑ M ↓ F β ↑ E ϕ ↑S α ↑ ProMaC 0.044±0.003 0.790±0.003 0.899±0.003 0.833±0.01 0.090±0.002 0.725±0.002 0.846±0.002 0.767±0.002Methods COD10K [14] CVC-ColonDB [51] M ↓ F β ↑ E ϕ ↑ S α ↑ M ↓ F β ↑ E ϕ ↑S α ↑ ProMaC 0.042± 0.002 0.716±0.002 0.876±0.003 0.805±0.002 0.176±0.002 0.243±0.002 0.583±0.003 0.530±0.002Kvasir [25] ISIC[10] M ↓ F β ↑ E ϕ ↑ S α ↑ M ↓ F β ↑ E ϕ ↑S α ↑ ProMaC 0.166 ± 0.003 0.394± 0.002 0.726 ± 0.004 0.573 ± 0.003 0.160 ± 0.003 0.729 ± 0.004 0.766 ± 0.003 0.703 ± 0.004Methods GSD [34] Trans10K-hard [56] M ↓ F β ↑ E ϕ ↑ S α ↑ M ↓ F β ↑ E ϕ ↑S α ↑ ProMaC 0.147 ± 0.002 0.409 ± 0.003 0.723± 0.004 0.569 ± 0.003 0.251 ± 0.003 0.509 ± 0.004 0.654 ± 0.002 0.557 ± 0.002In Tab. 7, we present the variance-inclusive experimental results of our ProMaC framework across three major datasets, obtained under three different seeds, with detailed environment setup available in the code instructions provided in the supplementary materials. For the COD dataset, we used LLaVA1.5+SAM as our baseline model to ensure fairness, with the comparative GenSAM model utilizing the same configuration. The three datasets utilized involve challenging scenarios with camouflaged animals or people, where some task-related targets are obscured or very small. Under such complex conditions, ProMaC achieves performance similar to or even better than weaklysupervised training methods with only a brief task description like "camouflaged animal" through test-time adaptation, demonstrating our method's superiority. Similarly, on the MIS and TOD tasks, our method significantly outperforms the comparative promptable segmentation approaches. On the OVS task, we first use LLaVA1.5 to identify which categories from a given multi-class list are actually present in the image. The categories identified are then further reasoned through our ProMaC to derive the final results. What'sUp is a benchmark designed to evaluate the spatial understanding abilities of MLLMs. It comprises 820 images that depict clear spatial relationships between two household items, such as a chair and a bowl, each image exclusively features the two objects in one of four distinct spatial configurations. We regard ProMaC's VCR as a visual marker strategy integrated into LLaVA1.5 to guide spatial reasoning and compared with traditional methods. Compared to previous approaches and baselines, our method significantly enhances performance, also highlighting the efficacy of the VCR module.Moreover, in Fig. 5 , we analyze how masks evolve across multiple tasks with iterations. It is evident that as iterations increase, the segmentation results improve, the boundaries of task-related objects become clearer, and some targets initially undetected are progressively recognized. Additionally, in Fig. 7 -12, we display masks for different tasks and samples of generated inpainting of task-related objects in contrastive samples. These results further demonstrate the versatility and superiority of our method.We compared ProMaC with the recently introduced GPT-4o+SAM [? 29] approach on the CHAMELEON and CAMO datasets for the COD task, and on the CVC-ColonDB dataset for the MIS task, with results displayed in the table below. It's evident that MLLMs based on GOT-4V [43] and LLaVA1.5 [37] excel in the COD task but experience a significant performance drop in the MIS task. This suggests that LLaVA and GPT4V lack specialized data like polyp detection, which leads to their underperformance in tasks requiring high specificity. This issue, stemming from the generalization limitations of MLLM datasets, is also evident in our ProMaC, based on LLaVA. While ProMaC achieves impressive results on the COD task due to LLaVA's general capabilities, its performance on the more specialized MIS dataset, though better than the baseline, falls short when compared to the COD task. The newly proposed GPT-4o, trained on a broader dataset, outperforms GPT-4V across various fields and shows significant improvement on the MIS task, further highlighting the impact of the underlying MLLM's generalization capabilities on ProMaC. This points to a need for further exploration and research into the generalization potential of foundational MLLM models in future work. M ↓ Fβ ↑ Eϕ ↑ Sα ↑ M ↓ Fβ ↑ Eϕ ↑ Sα ↑ M ↓ Fβ ↑ Eϕ ↑ Sα ↑ GPT4V+SAM [ProMaC uses MLLM to infer instance-specific prompts that guide the promptable segmentation model. A key question arises: how does the model ensure convergence to the accurate task-related region when initial instance-specific prompts from early iterations are imprecise? To address this, we utilize hallucinations to mine a potential list of instance-specific prompts from multiple scales and repeated image segmentations, ensuring comprehensive exploration of possible task-related objects.• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.We have discussed about the potential social impact in appendix A.7.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: Paper was reviewed by four expert reviewers and, post-rebuttal, received uniformly positive scores; mainly: 3 x Weak Accepts and 1 x Accept. While prior to rebuttal reviewers raised some concerns with the work, those concerns have been resolved in rebuttal and discussion. Specifically, [fu4N, E6it, u8FA] acknowledge that "all concerns have been well addressed", while [E5Cz] mentions that "most of  concerns have been addressed".  Overall, all reviewers agree that the approach is novel, interesting and the paper is well written. AC agrees with the reviewers and believes that the paper will be a welcome addition to NeurIPS. Therefore the decision is to Accept the paper. Authors are encouraged to incorporate elements of the rebuttal and discussion into the paper for the camera ready.

Reviews: review: paper_summary: paper_summary: The paper introduces an iterative Prompt-Mask Cycle generation framework (ProMaC) with a prompt generator and a mask generator. The prompt generator uses a multi-scale chain of thought prompting, initially exploring hallucinations for extracting extended contextual knowledge on a test image. These hallucinations are then reduced to formulate precise instance-specific prompts, directing the mask generator to produce masks that are consistent with task semantics by mask semantic alignment. The generated masks iteratively induce the prompt generator to focus more on task-relevant image areas and reduce irrelevant hallucinations, resulting jointly in better prompts and masks. The results on several challenging datasets validate the effectiveness of it.
strength_weakness: strength_weakness: -The authors mentions that the main motivation of this paper is to utilize hallucinations instead of eliminate it. However, the VCR module is used to “reducing” and “minimizing” it. This confuses me. The authors should clearly explain how this paper utilizes the hallucinations.

- The iterative process and the integration of various techniques (e.g., MLLM, SAM, CLIP, and inpainting) add complexity, which may hinder the ease of implementation and understanding.

-Compared with GenSAM [19]， What is the main innovation of the Mask Generator?

- At the first iteration, since masks have not yet been generated, what are the visual markers in this process? 

-The core of this task is how to generate accurate visual prompts through MLLM using a general text prompt. From my perspective, this task should be evaluated more on general datasets, such as COCO, to demonstrate the method's generalizability rather than on specific domain tasks (e.g., camouflage). Some comparison methods in this paper, such as X-Decoder and SEEM, are all generalized methods. Comparing with them, I believe, is unreasonable. I hope the author can discuss this issue with me.

-Minor: The authors should list the genetic task prompts used for different tasks.
questions: questions: See the weakness part.
limitations: limitations: The paper presents the limitation in the appendix. The authors suggest further exploration and research into the generalization potential of foundational MLLM models in future work.
paper_summary: paper_summary: The paper introduces the Prompt-Mask Cycle generation framework (ProMaC), which innovatively uses hallucinations from Multimodal Large Language Models to refine segmentation prompts and masks. This method contrasts with traditional approaches by leveraging rather than eliminating hallucinations, enhancing task-related accuracy through an iterative prompting and masking process. ProMaC's efficacy is demonstrated across various benchmark.
strength_weakness: strength_weakness: - The authors demonstrate the outstanding performance of the proposed ProMaC method across various tasks. Could they provide comparison results between the predictions of the ProMaC method and the ground truth across iterations? This would offer a more visual representation of the method's performance.
- The proposed method's computational and memory requirements aren't clearly discussed, which might be significantly higher due to the iterative nature of the Prompt-Mask Cycle and multiscale chain of thought prompting. This could limit its applicability in resource-constrained environments.
questions: questions: 1)Dependence on Initial Prompts: While the iterative refinement process is beneficial, the initial quality of prompts still plays a crucial role. Poor initial prompts might lead to suboptimal starting points, impacting the overall effectiveness of the adaptation process. I would like to see some discussion on this.
2)The authors utilize the LLaVA1.5 as the base MLLM for experiments, achieving results comparable to weakly-supervised and even supervised training in tasks such as Camouflaged Animal Detection, whereas performance in tasks like Medical Image Segmentation is more modest. The authors attribute this to the inherent generalization limitations of LLaVA. Recently developed methods (e.g., GPT4o) have shown better generalization capabilities in medical imaging. Could the authors conduct experiments using these more robust MLLMs on specific sub-tasks to substantiate this perspective?
limitations: limitations: The paper encounters a few limitations, such as the potential dependence on the initial quality of prompts and potential resource intensity due to the iterative and computationally demanding nature of the ProMaC method. Additionally, the performance disparity across different tasks highlights the limitations of the underlying MLLM's generalization capabilities. Despite these challenges, the overall contribution of the paper remains significant. The novel approach to leveraging hallucinations as a resource rather than a drawback, coupled with comprehensive experiments and detailed reproducibility, underscores the paper's value.
paper_summary: paper_summary: The paper focuses on promptable segmentation, aiming to minimize the need for manually designing prompts. Specifically, it explores the hallucination issue in MLLMs and finds that hallucinations can reveal valuable contextual information, which could be largely beneficial to promptable segmentation tasks, especially for complex tasks such as camouflaged object segmentation. To this end, the paper proposes a Prompt-Mask Cycle generation framework (ProMaC) that iteratively refines the generated prompts and masks. The proposed method leverages hallucinations to mine more context-related information while simultaneously reducing irrelevant hallucinations for better segmentation results.
strength_weakness: strength_weakness: Motivation

1. The author illustrates the motivation mainly using samples from the camouflaged object segmentation task (in Figure 1 and A.2). This makes it quite straightforward to understand the rationale behind the proposed method and how it can benefit the camouflaged object segmentation task. However, it would be better to involve more examples from different segmentation tasks to illustrate the motivation comprehensively, especially general segmentation like COCO and Pascal VOC. I am curious whether this phenomenon would still occur when the objects are more visible


Method

1. Some terms are unclear and need explanation. For example, I’m not very clear about the meaning of the term ‘prompt’ used in the paper. Does it refer to text prompts or spatial prompts like boxes or points? From the method section, it seems that P_B and P_A are text prompts, and the instance-specific prompts refer to bounding boxes? Also, what does query P refer to? If its meaning differs from P_g, P_B, and P_A, it would be better to clarify this at the beginning of the method section. Since the method is based on prior work [19] and some terms are borrowed directly from [19], it is necessary to include a preliminary section to explain the background and terms. This would make the method section easier to follow.


2. Question about the proposed Multi-scale Chain of Thought Prompting. From my understanding, chain-of-thought prompting aims to achieve complex reasoning capabilities through multiple intermediate reasoning steps. Could the author please explain how this idea is reflected in the proposed method?
questions: questions: Overall, the idea of this paper is quite interesting. However, it lacks some necessary explanations about the motivation and the proposed method. For example, the motivation is primarily illustrated using examples from camouflaged object segmentation. More diverse examples could help illustrate its broader potential. Additionally, some terms are unclear, suggesting a need for clearer definitions and possibly a background section explaining the foundational concepts.
limitations: limitations: The author has discussed the limitations of the work.
paper_summary: paper_summary: The paper introduces an innovative approach called ProMaC (Prompt-Mask Cycle) to improve promptable segmentation. The primary goal of ProMaC is to reduce the dependency on instance-specific manual prompts by leveraging hallucinations from Multimodal Large Language Models (MLLMs) to generate more accurate and task-specific prompts.
strength_weakness: strength_weakness: Reliance on Hallucinations:

Inconsistent Performance: The approach relies on hallucinations, which can be unpredictable and inconsistent. In scenarios where hallucinations are highly inaccurate, the performance of ProMaC could degrade significantly.
Dependence on MLLM Quality: The effectiveness of leveraging hallucinations is heavily dependent on the quality and training of the Multimodal Large Language Models (MLLMs). Variations in the MLLM's training data and methodology can lead to differing hallucination patterns, affecting ProMaC's reliability.
Complexity of the Iterative Process:

Computational Overhead: The iterative process of refining prompts and masks can be computationally intensive. This complexity might limit the scalability of ProMaC for real-time applications or large datasets.
Implementation Challenges: The method's implementation is intricate, involving multiple stages of hallucination generation, contrastive reasoning, and mask refinement. This complexity can pose challenges for practitioners looking to adopt the method.
Generalization Concerns:

Task-Specific Fine-Tuning: While ProMaC aims to reduce manual prompt dependency, the initial setup still requires careful selection of task-generic prompts. Ensuring that these prompts are effective across diverse tasks without additional fine-tuning could be challenging.
Transferability: The approach might not generalize well to entirely new tasks or domains where the hallucinations generated by MLLMs do not align well with the actual task requirements.
questions: questions: see the weakness
limitations: limitations: see the weakness

Comments: comment: After reading the author's response, most of my concerns have been addressed. I've decided to raise my current score.
The authors addressed all my concerns. I suggest to accept this paper.
Thank the authors for the response. All my concerns have been well addressed. I'd like to raise my score to weak accept.
thanks the rebuttal from the authors. they solved all my concerns. I agreed with all the other reviewers that we should accept the paper.