RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything

Authors: Shilin Xu, Haobo Yuan, Qingyu Shi, Lu Qi, Jingbo Wang, Yibo Yang, Yining Li, Kai Chen, Yunhai Tong, Bernard Ghanem, Xiangtai Li, Ming-Hsuan Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: segment anything; real-time segmentation; multi-purpose model;

Abstract:
Recent segmentation methods, which adopt large-scale data training and transformer architecture, aim to create one foundation model that can perform multiple tasks.
    However, most of these methods rely on heavy encoder and decoder frameworks, hindering their performance in real-time scenarios.
    To explore real-time segmentation, recent advancements primarily focus on semantic segmentation within specific environments, such as autonomous driving. However, they often overlook the generalization ability of these models across diverse scenarios.
    Therefore, to fill this gap, this work explores a novel real-time segmentation setting called real-time multi-purpose segmentation.
    It contains three fundamental sub-tasks: interactive segmentation, panoptic segmentation, and video instance segmentation. 
    Unlike previous methods, which use a specific design for each task, we aim to use only a single end-to-end model to accomplish all these tasks in real-time.
    To meet real-time requirements and balance multi-task learning, we present a novel dynamic convolution-based method, Real-Time Multi-Purpose SAM (RMP-SAM). 
    It contains an efficient encoder and an efficient decoupled adapter to perform prompt-driven decoding. 
    Moreover, we further explore different training strategies and one new adapter design to boost co-training performance further. 
    We benchmark several strong baselines by extending existing works to support our multi-purpose segmentation.
    Extensive experiments demonstrate that RMP-SAM is effective and generalizes well on proposed benchmarks and other specific semantic tasks. 
    Our implementation of RMP-SAM achieves the optimal balance between accuracy and speed for these tasks. The code is released at 
    \url{https://github.com/xushilin1/RAP-SAM}

Decision: accept (oral)
Reviewer scores: 8, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1729520194736

thread_type:
official_review

<thread_object>
creator: reviewer_hzhq

<content>
summary:
The authors explore a novel real-time segmentation setting called real-time multi-purpose segmentation. It contains three fundamental sub-tasks: interactive segmentation, panoptic segmentation, and video instance segmentation. In contrast to previous methods that use a separate design for each task, the authors use only a single end-to-end model to handle all these tasks in real time. To fulfill the real-time requirements and balance multitask learning, a new dynamic convolution-based method, Real-Time Multi-Purpose SAM (RMP-SAM), is introduced. They benchmark several strong baselines by extending existing work to support multi-purpose segmentation.

strengths:
- Large models can perform many tasks, but are not real-time capable because of the large encoders, while real-time models are often specialized in only one task. The method presented here aims to combine the two things, i.e., "the first real-time multi-purpose segmentation model".
- Precise implementation details are given and the comparisons with the other methods appear to be fair.
- The method achieves good results in the trade-off between performance and speed across the various tasks and datasets.
- The ablation studies are useful and show interesting insights.

weaknesses:
- Many architectural elements were adopted from other works, it is not clear to me if there are already similar architectures as proposed here, or where exactly is the innovation (except the jointly training).
- In the related work section, many works are cited and also compared at the task level, but I also miss a comparison at the architectural level.
- The tables, especially table 3, are difficult to read because nothing is in bold print and you have to search for the trade-off here. A plot like Fig. 1b would be more useful.
- The references to the appendix could be a little more precise and there is no reference to Table 2.

questions:
- What does the dot size in Fig. 1b indicate?
- The abstract says "generalization ability of these models across diverse scenarios", a learnable classifier with CLIP text embeddings is also used and “segment anything” is in the title. Is there a connection to open-vocabulary?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730465674213

thread_type:
official_review

<thread_object>
creator: reviewer_nma5

<content>
summary:
This paper presents a real-time multi-purpose segmentation model called RMP-SAM. RMP-SAM handles various tasks such as interactive segmentation, panoptic segmentation, and video instance segmentation using a single model. To balance the accuracy and speed, RMP-SAM utilizes a lightweight encoder and a dynamic convolution-based decoder. RMP-SAM achieves fast inference while maintaining satisfactory performance.

strengths:
- RMP-SAM unifies interactive segmentation, panoptic segmentation, and video instance segmentation within a single model.
- RMP-SAM offers a good trade-off between speed and accuracy.
- Extensive experiments demonstrate the model's effectiveness.

weaknesses:
- The authors do not provide detailed information for joint training. Joint training for multiple tasks can be complex. How do the authors train RMP-SAM for some potential problems, such as avoiding the model being dominated by a single task and performance degradation by conflicts between different tasks?

- This paper ignores some related methods, making it difficult to assess the model's performance relative to existing SOTA approaches. For example, some universal methods[1,2,3] obtain better results than RMP-SAM using ResNet50. The authors should make a comprehensive comparison with other methods. 

[1] Tube-Link: A flexible cross tube framework for universal video segmentation. CVPR 2023.

[2] Dvis: Decoupled video instance segmentation framework. CVPR 2023.

[3] Univs: Unified and universal video segmentation with prompts as queries. CVPR 2024.

questions:
Please see above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730545918906

thread_type:
official_review

<thread_object>
creator: reviewer_mnka

<content>
summary:
This work addresses the need for real-time multi-purpose segmentation by introducing a novel setting that encompasses interactive, panoptic, and video instance segmentation, striving for a single end-to-end model capable of handling all tasks in real-time. The proposed Real-Time Multi-Purpose SAM (RMP-SAM) utilizes an efficient encoder and a decoupled adapter for prompt-driven decoding, along with innovative training strategies and adapter designs, demonstrating effectiveness and strong generalization across benchmarks and specific semantic tasks while achieving an optimal balance between accuracy and speed.

strengths:
1.Demonstrates impressive performance and inference speed.

2.Filling the gap in real-time multi-purpose segmentation.

3.The whole method is very simple and easy to understand.

4.Code is provided for easy reproduction by the reader.

weaknesses:
1.Based on existing technology development, the entire pipeline is not novel.

2.Differences with SAMv2 should be further clarified, especially in terms of claimed semantic labels?

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730751617756

thread_type:
official_review

<thread_object>
creator: reviewer_ycmt

<content>
summary:
The paper presents a real-time, versatile segmentation model capable of interactive segmentation, panoptic segmentation, and video instance segmentation. 
While retaining the SAM encoder-decoder structure, the model incorporates an efficient encoder and adapter to enhance performance.
In the decoder, RAP-SAM introduces a three-stage pipeline that leverages novel pooling-based dynamic convolutions to refine mask tokens. Following the decoder, two additional prompt adapters are implemented to improve interaction between visual prompts and segmentation tokens.
RAP-SAM demonstrates efficiency and generalizability across various segmentation benchmarks.

strengths:
1. The model achieves multi-purpose segmentation through an efficient structure and unified training approach.
2. The paper is well-written and easy to follow.
3. The experiments on panoptic segmentation, interactive segmentation, and video segmentation are solid, comprehensive, and persuasive, effectively demonstrating the model's contribution.

weaknesses:
1. The paper lacks a detailed comparison with other SAM-like methods. A single COCO instance segmentation comparison in Table 4 is insufficient to substantiate claims of superiority over SAM. The results presented in Table 4 are not particularly outstanding. Additional experiments, such as on the SegAny task, with detailed metrics (AP for small, medium, large objects) on COCO instance segmentation, and evaluations with different object detectors, would strengthen the case.

2. Efficiency benchmarks are insufficiently detailed. For a model promoting efficiency, there should be a more comprehensive evaluation across different GPU platforms, such as the 3090 and V100, testing throughput and latency. Additionally, plotting latency versus performance compared to other SAM-like methods would provide a clearer visualization of the model's efficiency.

questions:
1. Do you have the results for TopFormer in Table 3? Additionally, please bold the results in all comparison tables for clarity.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732282633419

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
### General Response 

We thank all reviewers for their valuable comments and constructive suggestions. 

#### General Questions


1, Detailed Comparison with recent efficient SAM-like models.

In the previous version, we only report our model results in Tab.4, with one detector as the box prompts. Following the reviewers' suggestion, we add a more detailed comparison with these efficient SAM-like models, including more detectors and detailed metrics results on COCO-instance segmentation and more devices for speed testing. Our RMP-SAM can achieve stronger results compared with those specialists. 


2, Contributions and Novelty. 


Our main contribution is introducing a **new setting** that supports various segmentation tasks within a single real-time model. To our knowledge, there are no previous works in this direction. Most efficient models[1]-[3] are working on a single task and verifying their method on a single dataset. 

Thus, we benchmark several existing real-time segmentation models (including Mask2Former), extending them to handle panoptic, video instance, and interactive segmentation within one unified framework. However, most works cannot achieve the best speed and accuracy trade-off on multiple tasks.

Next, we present our model, RMP-SAM, a dynamic convolution-based approach with an improved task adaptation design. Our key designs include a shared convolution-based decoder with mask pooling to accelerate the decoding process and a decoupled adapter to decode the semantic-aware masks (image masks, video tube masks) and visual prompt-aware masks (interactive masks). With these designs, our method, RMP-SAM, can achieve the speed and accuracy trade-off on our proposed benchmark.
We also verify the effectiveness of decoupled adapters on various methods and balanced results for COCO panoptic and COCO-SAM segmentation.

For the results, RMP-SAM achieves the best trade-off between performance, task versatility, and speed. 


3, comparison with stronger models, SAM-2, and other strong video segmentation methods.


Following the reviewers' suggestions, we have compared with recent video segmentation methods, including SAM-2, strong baselines (DVIS and Tube-Link).

Compared with these foundation models, our model is more **efficient, multi-purposes** (both image/video, interactive/instance/panoptic in one small/efficient model, see the Tab.1 and Tab.2 in our paper). The goal of our work is orthogonal to these works, and our work is fundamentally different from these works. We present a more detailed comparison of these methods.


4, Training setup. 

We adopt joint image and video co-training, rather than first pre-training on image and then fine-tuning on video. We present a more detailed implementation and data-balancing strategy in the appendix. We will opensource our codebase and model for the community. 


##### Summary of Changes 

We've made the revisions to the main paper according to all reviewers' comments. The main revisions are summarized as follows:

1. We have revised the structure of Table 2 and bolded some information for easy reading and comparison.

2. We have added the results of RMP-SAM using TopFormer as the backbone for a more detailed comparison.

3. We have compared our method with other SAM-like methods with more detailed metrics and stronger detectors, see the appendix of Tab.10-Tab.12.

4. We have compared our method with SAM-2 in the appendix. 

5. We have compared our method with recent strong video segmentation methods in the appendix. 

6. We have made the reference to the appendix more precise.


The details of the revisions and other supplemented experiment results are in the following official comments to each reviewer.



[1] Tube-Link: A flexible cross tube framework for universal video segmentation. ICCV 2023.

[2] Dvis: Decoupled video instance segmentation framework. ICCV 2023.

[3] Univs: Unified and universal video segmentation with prompts as queries. CVPR 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1732448021590

thread_type:
official_comment

<thread_object>
creator: area_chair_yv4z

<content>
comment:
Dear Reviewers,

The discussion with the authors will conclude soon. The authors have provided detailed rebuttals. If there are any points that you feel have not been adequately clarified or if there are misunderstandings in their responses, please take this opportunity to raise them now. Thank you for your contributions to this review process.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents RAP-SAM, a real-time, versatile segmentation model capable of handling interactive segmentation, panoptic segmentation, and video instance segmentation. The model retains the SAM encoder-decoder structure while incorporating an efficient encoder and adapter to enhance performance. In the decoder, RAP-SAM introduces a three-stage pipeline leveraging pooling-based dynamic convolutions to refine mask tokens. Additionally, two prompt adapters are implemented to improve interactions between visual prompts and segmentation tokens. RAP-SAM demonstrates strong efficiency and generalizability across various segmentation benchmarks, filling the gap in real-time multi-purpose segmentation.

The reviewers unanimously praised the paper for its innovative approach to multi-purpose segmentation, including:
1.The ability to perform interactive segmentation, panoptic segmentation, and video instance segmentation through a unified training approach.
2.A well-written and easy-to-follow presentation.
3.Comprehensive experiments on multiple segmentation benchmarks, demonstrating the model's contributions and performance.
4.Strong inference speed and efficiency, meeting real-time requirements.



Therefore, I recommend the acceptance based on the unanimous support from reviewers. Since this research fills the gap in real-time multi-purpose segmentation and achieved impressive performance and inference speed, I would like to recommend for an oral presentation.