PerSense: Personalized Instance Segmentation in Dense Images

Authors: Muhammad Ibraheem Siddiqui, Muhammad Umer Sheikh, Hassan Abid, Muhammad Haris Khan
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: dense image segmentation, personalized instance segmentation, training-free, one-shot segmentation

Abstract:
Leveraging large-scale pre-training, vision foundational models showcase notable performance benefits. Recent segmentation algorithms for natural scenes have advanced significantly. However, existing models still struggle to automatically segment personalized instances in dense and crowded scenarios, where severe occlusions, scale variations, and background clutter pose a challenge to accurately delineate densely packed instances of the target object. To address this, we propose **PerSense**, an end-to-end, training-free, and model-agnostic one-shot framework for **Per**sonalized instance **S**egmentation in d**ense** images. Towards developing this framework, we make the following core contributions. **(a)** We develop a new baseline capable of automatically generating instance-level point prompts via proposing a novel Instance Detection Module (IDM) that leverages density maps, encapsulating spatial distribution of objects in an image. **(b)** To mitigate false positives within generated point prompts, we design Point Prompt Selection Module (PPSM). Both IDM and PPSM transform density maps into personalized precise point prompts for instance-level segmentation and offer a seamless integration in our model-agnostic framework. **(c)** We introduce a feedback mechanism which enables PerSense to improve the accuracy of density maps by automating the exemplar selection process for density map generation. **(d)** To promote algorithmic advances and effective tools for this relatively underexplored task, we introduce PerSense-D, a diverse dataset exclusive to personalized instance segmentation in dense images. Our extensive experiments establish PerSense superiority in dense scenarios by achieving an mIoU of **71.61%** on PerSense-D, outperforming recent SOTA models by significant margins of **+47.16%**, **+42.27%**, **+8.83%**, and **+5.69%**. Additionally, our qualitative findings demonstrate the adaptability of our framework to images captured in-the-wild.

Decision: reject
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730434989640

thread_type:
official_review

<thread_object>
creator: reviewer_l45p

<content>
summary:
This paper proposes a training-free and model-agnostic one-shot framework for personalized instance segmentation in dense images. The proposed framework, PerSense, is featured with a Instance Detection Module (IDM), a Point Prompt Selection Module (PPSM), and feedback machanism to improve the accuracy of density maps. Besides the framework, a dataset PerSense-D is introduced for personalized segmentation in dense images and demonstrating the superiority of the proposed method by comparing it with the SOTA on this benchmark. According to the quantitative and qualitative results, the proposed method shows a performance improvement with a clear margin. Experiments are well analyzed and ablation study is well designed to show the effectiveness of proposed component.

strengths:
1. Personalized instance-level segmentation which this paper target to solve is an interesting and important scenario for automation. The training-free manner ensures the ease for deployment.
2. Good performance of the proposed method from both qualitative and quantitative view and well experimental analysis. 
3. Dataset contribution. PerSense-D is introduced as a dataset exclusive to personalized segmentation in dense images.
4. The whole paper is well organized and presented.

weaknesses:
1. This paper claim they provide a one-shot personalized segmentation framework in instance level. However, the main results reported in table 1 seems come from 4-shot setting according to table 2 (c). If so, the 1-shot result should be reported to support the claimed contribution.
2. The proposed framework is complex by combining lots of existing methods or modules. Although the inference time is evaluated, memory consumption is not reported, which is critical for real deployment. 
3. No result on standard segmentation datasets is reported. Results on datasets with fewer objects can provide a better understanding on the proposed framework.
4. It seems the feedback mechanism plays an important role to ensure a high performance. But better feedback may involve more local iteration to avoid false positive and instance missing, which leads to longer inference time.

questions:
1. Will the codes and PreSense-D dataset be released?
2. Why does PerSense fail to have a superior performance over other methods on standard segmentation datasets with few object instances? Could authors have more discussion on the drawbacks of the proposed framework?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730470610576

thread_type:
official_review

<thread_object>
creator: reviewer_gosf

<content>
summary:
This work focuses on personalized instance segmentation in dense and crowded scenarios. To this end, they propose PerSense, a training-free framework. Specifically, they use a class-label extractor (CLE) and a grounding detector to select the effective exemplars for a density map generator (DMG). Then, they propose an instance detection module (IDM) to generate point prompts from the density map and design a point prompt selection module (PPSM) to reduce false positive predictions. Last, the authors further introduce a feedback scheme to refine the exemplar selection and thus improve the performance of PerSense. Moreover, they propose a dataset, PerSense-D, for the evaluation of instance segmentation in dense images. However, I still have some questions about the experimental settings.

strengths:
1. The proposed PPSM effectively removes the false positive predictions.
2. The proposed feedback mechanism further improves the density maps of DMG and boosts the performance of personalized instance segmentation.
3. The authors propose a new data set with 717 images and 28,395 objects for personalized instance segmentation.

weaknesses:
1. The authors compare the proposed method with the SOTA methods on PerSense-D. However, I noticed that the authors conducted ablation studies on the same dataset to pursue the best performance. As there are no more results on other datasets, the superior performance of the proposed method needs to be verified.
2. In Table 2, it is unclear why the method achieves the best performance with a normalized factor equal to sqrt(2). More discussions on the insight of normalized factors are required.
3. Although the feedback mechanism effectively improves the performance of the proposed method, it seems that this mechanism violates the one-shot setting. In this sense, the comparison with Grounded-SAM seems not fair.
4. At line 379, page 8, “x 1.02” should be “$\times 1.02$”.
5. In Table 2(c), “No of Shots” should be “No. of Shots”.

questions:
I think at least the authors should split the proposed dataset for validation and testing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730628936577

thread_type:
official_review

<thread_object>
creator: reviewer_cscq

<content>
summary:
The paper proposes a training-free method to achieve personalized instance segmentation, which aims to segment what people want with refering images.  It develops a new baseline capable of automatically generating instance-level point prompts via proposing a novel Instance Detection Module (IDM) that leverages density maps, encapsulating spatial distribution of objects in an imageA dataset PerSense-D is proposed to boost this research area.

strengths:
1. This article's research on the task of refining segmentation is valuable. The proposed dataset is helpful for the development of this field.
2. This method can achieve the desired effect without training, and has higher application value in this regard.

weaknesses:
1. The writing and organization of the article need to be improved:
a. In Figure 1, "(a)" appears twice
b. The method section lacks a paragraph that links together how several modules operate. And the caption in Figure 2 is also very brief, requiring a description of the structure.
c. It is not recommended to write the abstract in the form of contribution (1), (2), (3)
2. In line 43, I did not get the difference between personalized instance segmentation and traditional instance segmentation. The traditional setting is also aimed at segmenting the specified categories in the image. Do I need to give a category name for input? After reading the article, my understanding is that by providing a template for referencing, other similar targets are required to be segmented. It does not mean segmenting the specified category, but segmenting the specified referencing. If I understand correctly, C3Det([1],CVPR2022) has a similar idea (even if it is a detection) and needs to be compared and discussed (its inspection results can be used for segmentation by SAM). SegGPT([2] ICCV2023) also uses some template images to segment the desired objects. Please compare and discuss.
3. SAPNet (CVPR2024) uses Point prompt combined with SAM to generate candidate masks, and selects masks that meet specific categories as outputs. This idea is similar to this article, please discuss it. (Even if the method uses point annotation, the point prompt predicted in the first stage of this article can be used as its input instead of point annotation.) In addition, methods such as Bestie (CVPR2022) have also predicted peak points as point prompts through affinity maps, density maps, and other methods. Please compare and discuss.
4. Although COCO and LVIS are not specifically designed for dense scenarios, they do not hinder the validation of our method in this paper. And the size of COCO and LVIS is still much larger than the dataset in this article. I acknowledge the contribution of the dataset, but I believe that validation experiments on COCO or LVIS are still necessary to demonstrate the effectiveness of the proposed method in this article. After verifying with more datasets, I will consider increasing the score.
5. Visualization can incorporate qualitative validation of some methods to demonstrate comparison with benchmark methods and show the effectiveness of the proposed modules.

[1] Interactive Multi-Class Tiny-Object Detection.

[2] SegGPT: Segmenting Everything In Context.

[3] Semantic-aware SAM for Point-Prompted Instance Segmentation

[4] Beyond Semantic to Instance Segmentation: Weakly-Supervised Instance Segmentation via Semantic Knowledge Transfer and Self-Refinement

questions:
See weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652843035

thread_type:
official_review

<thread_object>
creator: reviewer_pnzs

<content>
summary:
The paper introduces PerSense, a novel, training-free, and model-agnostic framework for personalized instance segmentation in dense images. The contributions of the paper are as follows:

1. New Baseline and Instance Detection Module (IDM): A new baseline is proposed, capable of automatically generating instance-level point prompts, featuring an Instance Detection Module (IDM) that utilizes density maps (DM) to provide candidate point prompts. A density map generator (DMG) highlights the spatial distribution of target objects. Automated Exemplar Selection: A class-label extractor (CLE) and grounding detector are used to automate the selection of effective exemplars, simplifying the DMG’s manual process. Point Prompt Selection Module (PPSM): A Point Prompt Selection Module (PPSM) is designed to reduce false positives within the candidate point prompts. IDM and PPSM are plug-and-play components that integrate seamlessly into the PerSense framework.

2. Feedback Mechanism: A feedback mechanism is introduced to automatically refine exemplar selection based on PerSense’s initial segmentation output, identifying multiple rich exemplars for DMG to improve segmentation accuracy.

3. PerSense-D Dataset: To facilitate personalized segmentation in dense images, the authors introduce PerSense-D, a dataset with 717 densely populated images across 28 object categories, providing a challenging benchmark for future studies.

strengths:
(1) Innovative Training-Free Approach: By using density maps, PerSense avoids the need for extensive training, making it computationally efficient and adaptable to different dense segmentation tasks.
(2) Model-Agnostic Design: The PerSense framework can be seamlessly integrated with various density map generators and grounding detectors, enhancing its flexibility and usability.
(3) Comprehensive Evaluation: Experimental results on the newly introduced PerSense-D dataset show that PerSense significantly outperforms state-of-the-art methods, demonstrating its robustness and high performance in densely packed scenarios.

weaknesses:
1. Methodological Innovation:  Although PerSense's training-free framework demonstrates some level of innovation, its reliance on density maps is not entirely novel, as similar approaches have been applied in traditional vision tasks. While the Instance Detection Module (IDM) and Point Prompt Selection Module (PPSM) bring some originality to the framework, they lack sufficient mathematical derivation or theoretical analysis to substantiate their uniqueness and theoretical advantage over existing methods. 
2. Insufficient Baseline Model Comparisons: The experimental comparisons of PerSense are limited to a few general segmentation models (e.g., PerSAM, Matcher) and lack comparisons with other classic indoor segmentation or dense scene methods. For instance, in dense scene segmentation tasks, other clustering-based or feature-matching models might also provide effective solutions. The absence of such baseline model comparisons weakens the demonstration of PerSense's advantage in specific tasks.
3.  Dataset diversity: Although the PerSense-D dataset focuses on dense scenes, it is relatively small in scale (717 images) with limited category coverage, failing to fully represent diverse dense scenes. Real-world applications in personalized segmentation for dense scenes often require larger and more varied samples (e.g., different resolutions, scene types).
4. Experimental Analysis : The ablation studies primarily highlight the incremental effects of IDM, PPSM, and the feedback mechanism, but do not thoroughly explore the synergy among components. For example, the impact of different density map generators (DMG) on model performance is not fully analyzed. Additionally, there is a lack of analysis on the specific contributions of multimodal features (e.g., visual and textual features) in personalized segmentation tasks, making it challenging for readers to understand the role of each feature in the overall performance.

questions:
1. Although the design of PerSense's modules is somewhat innovative, the theoretical support is insufficient. For instance, there is no mathematical proof of how IDM and PPSM improve segmentation accuracy, particularly regarding how they outperform other methods in generating point prompts in dense scenes. Further mathematical derivation would enhance the theoretical foundation of this approach.
2.The experimental validation of this feedback mechanism is limited, as it does not demonstrate the impact of different iteration counts on segmentation accuracy. For example, does varying the initial exemplar selection strategy or iteration count significantly affect model performance? .
3.In dense scenes, challenges like occlusion, lighting changes, and varying object sizes can affect the robustness of segmentation models. However, the paper does not conduct robustness experiments to address these factors. For instance, how does the model perform under low lighting, complex backgrounds, or heavy occlusion?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732298505038

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response and Summary of Updates to Manuscript

comment:
We thank the reviewers **(PNZS, CscQ, GosF, L45P)** for their positive and thoughtful feedback, and we appreciate the comments aimed at improving our work. We summarize the main points presented in our response and hope that we have satisfactorily addressed all the concerns raised by reviewers.  

- We provide additional mathematical insights for the proposed instance detection module (IDM) starting with composite contour detection via statistical thresholding to the centroid computation for candidate point prompts. We also provide a detailed theoretical rationale behind the formulation of the adaptive threshold in the point prompt selection module (PPSM) (see Appendix B in the updated pdf).
  
- We include additional comparisons of our PerSense with C3Det [1], Painter [2], SegGPT [3] in addition to Matcher [4], PerSAM [5], and Grounded-SAM [6] (see Appendix C in the updated pdf).  
- In addition to the PerSense-D dataset, we also evaluate our PerSense on standard segmentation datasets, specifically COCO-20$^i$ [7] and LVIS-92$^i$ [8], and conduct comparisons with a diverse range of approaches, including both in-domain training methods and training-free techniques (see Appendix C in the updated pdf).  

- We conduct additional ablation studies, including the impact of multiple iterations in the proposed feedback mechanism, a component-wise analysis of PerSense on the COCO dataset, and an evaluation of PerSense's memory consumption (see Appendix D in the updated pdf). 
 
- We enhanced Figure 7 to provide a detailed step-by-step component-wise qualitative analysis of PerSense, incorporating qualitative validation of benchmark methods to demonstrate the effectiveness of the proposed modules (see Appendix A.2 in the updated pdf).  

- We address the questions raised by reviewers and provide clarifications on the feedback resulting in the improvement of our work.  

Please do not hesitate to reach out if you need any further clarifications. Thank you.  

---

References:  

[1] Interactive multi-class tiny-object detection. CVPR 2022. 

[2] Images speak in images: A generalist painter for in-context visual learning. CVPR 2023.
  
[3] Seggpt: Segmenting everything in context. ICCV 2023.  

[4] Matcher: Segment anything with one shot using all-purpose feature matching. ICLR 2024. 
 
[5] Personalize Segment Anything Model with one shot. ICLR 2024. 
 
[6] Grounded sam: Assembling open-world models for diverse visual tasks. arXiv 2024. 
 
[7] Feature weighting and boosting for few-shot segmentation. ICCV 2019. 
 
[8] Dataset for large vocabulary instance segmentation. CVPR 2019.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission proposed a new method and dataset for personalized instance segmentation in dense images. The approach performs well using a model-agnostic framework, and the overall effectiveness is verified. The authors also highlight their contribution as the first to explicitly propose the concept of dense image segmentation. The weaknesses of the paper include the novelty issue of using the density map for segmentation, missing some experimental analysis, and the small scale and unclear evaluation scheme of the PerSense-D dataset. After rebuttal, the authors addressed some concerns, but the concerns about the dataset still exist. A small-scale dataset without an evaluation scheme design could result in overfitting results by heavy manual adjustment of the models or pipelines. Considering these concerns, the overall recommendation is reject.