RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything
Authors: Shilin Xu
Haobo Yuan
Qingyu Shi
Lu Qi
Jingbo Wang
Yibo Yang
Yining Li
Kai Chen
Yunhai Tong
Bernard Ghanem
Xiangtai Li
Ming-Hsuan Yang
Venue: ICLR 2025 Oral (2025)
Keywords: segment anything; real-time segmentation; multi-purpose model;
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent segmentation methods, which adopt large-scale data training and transformer architecture, aim to create one foundation model that can perform multiple tasks.
    However, most of these methods rely on heavy encoder and decoder frameworks, hindering their performance in real-time scenarios.
    To explore real-time segmentation, recent advancements primarily focus on semantic segmentation within specific environments, such as autonomous driving. However, they often overlook the generalization ability of these models across diverse scenarios.
    Therefore, to fill this gap, this work explores a novel real-time segmentation setting called real-time multi-purpose segmentation.
    It contains three fundamental sub-tasks: interactive segmentation, panoptic segmentation, and video instance segmentation. 
    Unlike previous methods, which use a specific design for each task, we aim to use only a single end-to-end model to accomplish all these tasks in real-time.
    To meet real-time requirements and balance multi-task learning, we present a novel dynamic convolution-based method, Real-Time Multi-Purpose SAM (RMP-SAM). 
    It contains an efficient encoder and an efficient decoupled adapter to perform prompt-driven decoding. 
    Moreover, we further explore different training strategies and one new adapter design to boost co-training performance further. 
    We benchmark several strong baselines by extending existing works to support our multi-purpose segmentation.
    Extensive experiments demonstrate that RMP-SAM is effective and generalizes well on proposed benchmarks and other specific semantic tasks. 
    Our implementation of RMP-SAM achieves the optimal balance between accuracy and speed for these tasks. The code is released at 
    \url{https://github.com/xushilin1/RAP-SAM}

Published as a conference paper at ICLR 2025
RMP-SAM: TOWARDS REAL-TIME MULTI-PURPOSH
SEGMENT ANYTHING
Shilin Xu 1u ; Yuan 2:3 Qingyu Shi', Lu Jingbo Wang", Yibo Yang², Yining Li", Kai Chen', Yunhai Tong', Bernard Ghanem", Xiangtai Li Ming-Hsuan Yang". 3.6 Peking University. *Nanyang Technology University, UC, Merced SShanghai AI Laboratory, *KAUST, "Google Research
xushilingstu, wwt..kku..... edu. cn, siangtal 94 gmai com
ABSTRACT
Recent segmentation methods, which adopt large-scale data training and trans- former architecture, aim to create one foundation model that can perform multiple tasks. However, most of these methods rely on heavy encoder and decoder frame- works, hindering their performance in real-time scenarios. To explore real-time segmentation, recent advancements primarily focus on semantic segmentation within specific environments, such as autonomous driving. However, they of- ten overlook the generalization ability of these models across diverse scenarios. Therefore, to fill this gap, this work explores a novel real-time segmentation set- ting called real-time multi-purposs purpose segmentation. It contains three fundamental sub-tasks: interactive segmentation, panoptic segmentation, and video instance segmentation. Unlike previous methods, which use a specific design for each task, we aim to use only single end-to-end model to accomplish all these tasks real-time. To meet real-time requirements and balance multi-task learning, we present a novel dynamic convolution-banl based method, Real-Time Multi-Purpose SAM (RMP-SAM). It contains an efficient encoder and an efficient decoupled adapter to perform prompt-driven decoding. Moreover, we further explore different training strategies and one new adapter design to boost co-training performance further. We benchmark several strong baselines by extending existing works to support our multi pulliee segmentation. Extensive experiments demonstrate that RMP-SAM is effective and generalizes well on proposed benchmarks and other specific semantic tasks. Our implementation of RMP-SAM achieves the optimal balance between accuracy and speed for these tasks. The code is released at https: /tte:://gttttu com/ wwshiliniii /RAP--APP SAM
INTRODUCTION
Recent advancements in computer vision, driven by transformer architectures (Dosovitskiy et al., 2021; Carion et al., 2020; Wang al., 2023b), focus creating single large-scale model versatile enough to handle different tasks like detection and segmentation with general-purpose design. Several studies utilizing the same architectures (Wang al., 2023a; Yan et al., 2023; Jain et al., 2023) have shown superior performance across various tasks. Enjoying the benefit of the large-scale model capacity and training data, the general-purpose models demonstrate strong performance for specific applications (Kirillov et al., 2023; Jia et al., 2021). For image and video segmentation at the semantic level, numerous studies (Cheng et al., 2022: Zhang et al., 2021: Li al., 2022d; 2023e; Jain et al., 2023) employing a universal design have surpassed previous models tailored for specific image and video segmentation tasks. The Segment Anything Model (SAM) (Kirillov al., 2023) is leading approach in interactive segmentation, offering unified model for various interactions such as point, bounding box, mask, and text input.
versatility
"Wor done during internship at Shanghai Al Lab
1 Equal contribution
Corresponning author

---

Published as a conference paper at ICLR 2025
Real-Time Multi- Purpi-e Segmentation
FV
IAP
Nack (240
YINKENI
AVat
40
RMP-SAM
Poropeic Segnerration
Iinance Segtioration
nioli
66
Madinesen EESC
Cerstiy
Video Supeesstin
SAMED
20
40
50
Readee the Inteructive Segmrettnnonhhrr
Roorhh for interatine sigreetatoogggoee
Prop Sepreeetiio
We present real-time multi-purpose segmentation to segment and recognize objects for image, video, and visual prompt inputs. In addition benchmarking, propose simple yet effective baseline, named RMP-SAM, which achieves the best performance and speed trade-off among three different tasks. The larger dot indicates more parameters.
Figure 1:
However, most of these studies face challenges in achieving real-time performance on devices with limited computing resources due to their heavy encoders or cascaded decoders. Despite their nigh-performanee capabilities and versatility, this limitation hinders the practical use of these advanced architectures in vision applications, such as editing tools on edge devices. In contrast, several studies (Zhao al., 2018; Hu et al., 2023) focus real-time segmentation using specific designs (e.g., efficient decoder fusion) but are often tailored to a specific application trained on a particular dataset (e.g., COCO) or restricted to similar tasks like video semantic segmentation. To our knowledge, no study investigates real-time, murlp--prpose segmentation, where a single model can perform universal segmentation tasks, including image, video, and SAM-like interactive segmentation. For example, given video input, we want to create tool to perform various segmentation tasks on the device, including interactive segmentation with visual cues, labeling the masks, and tracking the masks.
Inspired by the abovementioned challenges, we focus on real-time multi-purpose segmentation and efficiently investigate the novel issue of training versatile models. We pose critical inquiry: Given the limited computational resources and model capacity, how can we develop an efficient, multi- purpose segmentation model? This entails creating single model capable of segmenting, tracking, and classifying each pixel in real-time, similar to performing interactive segmentation similar to SAM. We present illustration figure in Fig. 1(a).
First, we explore this problem by benchmarking existing methods. Specifically, we use the COCO (Lin al., 2014) and YouTube-VIR (Yang et al., 2019) for joint co-training, employing identical hyperpa- rameters. Beyond the object queries used for image and video segmentation in prior research, our approach incorporates visual prompt queries for interactive segmentation. However, none of these works can achieve the best accuracy and speed trade-off on three tasks using one model. This is because: No previous works explore joint co-training for segmentation for different purposes. Thus, extensive experiments are needed to gain a better understanding of meta- aotta-architeture. 2) Interactive and semantic-level segmentation needs better strategies to balance each other. 3) Under the real-time setting, several previous architectures (Yan et al., 2023; Li et al., 2022c) for multi-task learning may not be practical.
To solve these issues, we present real-time multi- purpose segment anything model (RMP-SAM). It contains an efficient encoder and shared decoder with multiple queries as inputs. Unlike previous works, avoid cascaded transformers encoder layers and pixel-wise cross-attention decoder (Cheng et al., 2022). Instead, we propose mooling-based dynamic convolution to replace per-pixel cross- attention to achieve better accuracy and speed trade-off. We represent all three tasks using queries with shared dynamic convolution operation. According to different outputs, we can divide the segmentation tasks into two types: one for class-agnostic masks (SAM-like) and the other for class-
2

---

Published as a conference paper at ICLR 2025
Table 1: Real Time Segmentation Settings. Our multi framework.
multi-purpose purpose segmentation supports more tasks in one
Property Image Misks
Image Seg
Video Seg
SAM-Like
SAM-2-Like
Ours: Real-Time Multi-Purppee Seg
X
/
Video Mosks
Interactive
X
Semantic Lahels
X
Multitasks
X
Table 2: Comparison of Segmentation Methods. Our proposed RMP-SAM supports various segmentation tasks and performs them in real-time
Methods
VIS
Interactive
Multi Task One Model
Real Time
ICNer (Zhao et al., 2018)
X
Seg (Yu al., 2018)
X
YOSO al., 2023)
v
x
X
X
Mobilie-V VIS (Zhang al. 2024)
1
SAM (Kinillv al., 2023)
Mask2Fommer (Cheng 2022)
x
X
Video K-Net
X
OndFomer Ohin al., 2023)
RMP-SAM (Ours)
specific masks (contains one mask and one class label). Two types have different goals and will conflict during the multi-task learning process. Thus, further explore four different architectures in the case of various decoder designs. In particular, we find a single decoder with multiple queries for different tasks can achieve good enough results and best speed and accuracy trade-off. Moreover, we present an effective dual adapter design that better adapts shared knowledge of the same decoder. The dual adapter contains an asymmetric design to better transfer shared features. To the best of our knowledge, RMP-SAM is the first real-time multi- purpose segmentation model that can achieve the best speed and accuracy trade-off on four different tasks, shown in Fig. 1(b). Moreover, due the semantic level and interactive level segmentation ability, our model can create more new applications, such as interactive video segmentation. In addition, we also verify our model in the SAM-like setting by co-training with SAM-data, it also achieves comparable results. Extensive experiments show the effectiveness of our design. The main contributions of this work are:
We introduce real-time multi nulli-purpose segmentation, novel multi-tas segmentation aiming to segment objects for real-time image, video, and interactive inputs.
We benchmark several real-time transformer-batt segmentation approaches for the new settings.
We present simple yet fast baseline named RMP-SAM. It contains lightweight feature extractor, a unified decoder, and two asymmetric adapters.
Extensive experiments demonstrate that RMPP-SM achieves the best speed and accuracy trade-off in the proposed benchmark and regular real-time semantic and panoptic segmenta- tion benchmarks. We also show scalability across datasets and application demos.
2 REAL-TIME MULTI-PURPOSE SEGMENTATION
Problem Settings. Our proposed real-time multi-purpose segmentation contains three visual inputs: image, video. and visual prompts (boxes and points). predicts the corresponding masks, labels, and instance IDs for the content within the given input. Basically, we adopt the definition of panoptic segmentation Kirillov al. (2019) for image inputs since covers all concepts in the scene. For video inputs, we segment and track each foreground instance (video instance segmentation). Given visual prompts, we follow the SAM-like Zhang al. (2023a); Zhou et al. (2023); Shu et al. (2023); Xiong et al. (2023) setting for interactive segmentation, performing class-agnostic segmentation according to the user's 's inputs. Although recent advancements, such as SAM-2 (Ravi et al., 2024), have attempted to unify the interactive segmentation task for both image and video inputs, emphasizes the importance of object semantics. incorporates multi-task task learning in a single shot. Combined with interactive segmentation and video segmentation,
framework
also achieve interactive
3

---

Published as a conference paper at ICLR 2025
video segmentation video object segmentation. As shown in Tab. 1 and Tab. 2, we compare previous segmentation methods with and indicate that our method unifies different segmentation tasks within a single framework. This approach enables more flexible usage across various segmentation tasks, particularly regarding input modalities and predictions.
the well-known COCO dataset (Lin al., 2014) for panoptic and interactive segmentation. During the training of interactive segmentation, the boxes derived from the ground truth, along with points randomly sampled from the masks, serve as visual prompts. For testing, we obtain the center point of the mask as a visual prompt. For video segmentation, we adopt the widely used YouTube- VIS 2019 dataset (Yang al., 2019) for training. We do not use the SAM data for our benchmarking. There are several reasons: (1) SAM data is not as well annotated as the COCO dataset. (2) We only focus on object-level and scene-level segmentation, while SAM data contain more granularity. (3) SAM data has no semantics, while our goal is to predict semantic labels for real applications. To fairly compare with SAM-like models, we adopt the SA.M-data training for specific setting, shown in Sec. 4.1.
Datasets. For benchmarking,
PROPOSED METHOD
First, we revise panoptic segmentation, video instance segmentation, and interactive segmentation under the unified formulation. When combined with a query-based segmentation model, object queries can represent all entities within the scene, encompassing image-level objects, vidco-level tracked objects, and prompt-based specific objects. For panoptic segmentation (Kirillov et al., 2019), given an input image E TR. # x3 the objective is to generate a group of masks } = {(m, cs) N where e denotes the class label of the binary mask m; and N is the number of queries, H x W indicates the spatial dimensions. For video instance segmentation (Yang et al., 2019), given a video where T denotes the number of frames, the process aims to generate an instance mask {yr J = {(m, C, d where N represents the number of the tube masks m, E 10, 1 x each instance. For SAM-like segmentation (Kirillov et al., 2023), both the image I and visual prompts P Rx*2 (point prompts are used here) are taken as inputs. These prompts, which include points and boxes, lead to the outputs of corresponding binary image masks {y: } {m E {0, 1) x } where K denotes the number of visual prompts. Each visual prompt is encoded into one object query, which naturally can be the input of the decoder.
clip input V R
C; indicates the class label of the tube m,, and d, identifies the instance ID for
3.1 RMMP-SAM ARCHITECTURE
Overall Architecture. As shown in Fig. 2, our RMP-SAM is built upon simple encoder and decoder architecture that can input images, videos, and visual prompts. Following SAM, we utilize the prompt encoder to convert visual prompts into queries. These visual prompts and object queries share the same decoder for the efficiency of computation and parameter size. However, the goals of the two types of queries are different: the former pays more attention to the local details of the visual prompts, while the latter considers the scene and temporal features. To better balance the results for interactive segmentation and image/video segmentation, we design a prompt adapter and an object adapter at the end of the decoder.
Lite Feature Extractor. Due to computation cost limitations, we avoid using previous meth- ods (Cheng et al., 2022; Li et al., 2024b) that involve large backbones and heavier encoders. As required by real-time constraint, we investigate lightweight backbones such as ResNet18 (He et al., 2016) and TopFormer (Zhang et al., 2022). We incorporate a feature pyramid network with de- formable convolution to enhance representation alignment and fuse sclti-scale features in real- time. When dealing with video input, we extract the temporal features. For simplicity, we use Fing& E P
/ x W xd for image inputs and Foid E Tx W xd for video inputs.
Unified Dynamic Convolution Decoder. The goal of the decoder is to refine the object query. However, many of these methods (Cheng et al., 2021b; 2022), are not feasible for real-time settings due to their reliance on heavily cascaded layers and pixel-wise cross-attention mechanisms. In contrast, method uses pooling-based dynamic convolution framework (Zhang al., 2021; Li et al., 2022d) to enhance the efficiency of the decoder. Given object query Q E N xd and prompt query P E R R xd the masks M, obtained by dot product with F Fing or Foid- Here, i represents the decoder stage
4

---

Published as a conference paper at ICLR 2025
object/ ogjec//poepp quades
Visual Prompls
Thre-Stage Decodir
prompt quates
pronge-quartes
lol:
add & nomm
Primpt Brande
d CiErr ae doco
d G drantie-cmollle conoluttoon
Strrpe Ackato
Mask
ITN
Pheling lave
Paeling heiinggheen
Odpect
Mask
sdF setertion
Image or Video
di ect quadies
object queres
Class
add & rarm
Linese parject
Dynanic comselution
Bocihoon
Lite Noda
Inner product
Image Teeture Fang C x Vidio Fistur Nap Fusst Tx C. W
Proegg quay I
Object quey
festure map object/proept quetee
Figure 2: RMP--AM overview. Our method contains three visual inputs: image, video, and visual prompts. Utilizing positional encoding, we generate prompt queries from these visual prompts. The learnable object queries, prompt queries, and the feature map F are directed the multi-stage decoder. This process generates multi-stage predictions and refined queries. These refined queries engage in cross-attention with F, resulting in the final prediction.
index, N is the total number of object queries, and K is the number of visual prompts. According to the inputs, the masks can be image masks for panoptic segmentation M E R x 4 and interactive segmentation Miter E R 4 th or tube masks Mfube E SRNxTx # Then, we can obtain the corresponding query features XPaa E Nxd, Xiter E KRd XLube E Nxd via mask pooling:
W Σ
XPaa
Σ
M (u, u) Fing(u, v).
For video tasks, group the spatial-temports features follows:
T
Xtube -ΣΣ
M Mf (t, Foialt,
(2)
where u, e are the indices of spatial location, t is index of input frame. Since the dynamic convolution performs similarly for each task, we use panoptic segmentation XP as an illustration. For interactive segmentation, we adopt the same equation (Equ. 1). In our implementation, the learned query is shared for both Quen and Quaee
In particular, we adopt the design of previous methods (Zhang et al., 2021; Sun et al., 2021; Li et al., 2022c) to refine input queries Quaan with features X which are grouped from their masks,
Quan DynamicConv(X! (X.an QP27).
where the dynamic convolution the query features X,Th to generate parameters to weight input queries Q.an- Specifically, DynamicConv uses input query features X.Pon to generate gating parameters via a multilayer perceptron (MLP) and multiply back to the original query input Q We adopt the same design (Zhang et al., 2021; Sun et al., 2021) by learning gating functions to update the refined queries. The DynamicConv operation based on:
QREn = Gate, G (X!"") + Gate, (XPan) QQ,er
(4)
where Gate is implemented with fully connected layer followed by Layer Norm and a sigmoid layer. We adopt two different gate functions, including Gate, and Gateg. The former is to weigh the query features, while the latter to weigh corresponding queries. Next, we adopt one self-attention layer with feed-forward layers (Vaswani et al., 2017; Wang et al., 2021) learn the correspondence among each query and update them accordingly. This operation leads to the full correlation among queries, shown follows:
Q₂ (FN Q.).
where MHSA means Multi Head Self Attention, FFN is the Feed Forward Network commonly used in the current version of Transformers (Carion et al., 2020; Dosovitskiy et al., 2021). We adopt several feed-forward layers Q.ee for mask classification and directly output the class scores. Then,
5

---

Published as a conference paper at ICLR 2025
perform the inner product between the learned queries Q, and image features Foid and Fang- The only difference for the video segmentation task is that we predict the tube mask rather than only a single image mask. The entire process (Equ. Equ. 2, Equ. 3 and Equ. 5) is repeated three times in total.
Moreover, also explore various decoupled designs, shown in Fig. 3. In particular, as previous works suggest, we also explore decoupled decoders and their corresponding adapters. Via our detailed experiments, we do not find extra gains in the cases of decoupled decoders that introduce more computation and parameter costs. Please refer to Sec. 4.2 for the detailed decoder design.
Light- Weight Decoupled Adapter. After the shared decoder, also add two lightweight adapters, Aob and Aprompt to adapt the shared decoder's knowledge better. In particular, we use the asymmet- ric design. Aobj uses the same dynamic convolution design to refine the object query further, while A Aprompt uses pixel-wise cross-altention design. Our key insights are: (1) The interaction between the whole scene feature and object queries is necessary for panoptic and video segmentation but not for interactive segmentation. The visual prompts provide strong positional information, and they pay more attention to local details features. (2) Despite the decoder being shared, the details are still missing for interactive segmentation due to the pooling effect in Equ. I, where we even find inferior results after adopting a pooling- based adapter. More detailed adapter design can be found in Sec. 4.2.
3.2 TRAINING, INFERENCE, AND EXTENSION
Joint Image and Video Segmentation Co-training. Our goal is to train all segmentation tasks only once jointly. All training targets are one entity label and mask for all three cases. The entity can be thing, stuff, class-agnostic masks, and corresponding labels. Note that the instance masks with the same ID) d form the tube masks. During training, we apply Hungarian matching between the predicted and ground-truth entity masks assign object queries to video/image entities and then supervise their predicted masks and classification. To unify label taxonomies across datasets, we replace the learnable classifier with CLIP text embeddings following ViLD (Gu et al., 2021).
Following previous works (Cheng et al., 2022; Li et al., 2023e). the final loss function is given as = Aeta Lim_el + DxceLm_o + Adice L Here, Lm is the Pross-Entropy (CE) loss for mask classification, and Lin_ce and Lm_dice are mask Cross-Entropy (CE) loss and Dice loss (Milletari et al., 2016; Wang et al., 2020) for segmentation, respectively, By default, we set Acta = 2, Ace 5, Adice 5.
Inference. We follow the same inference procedure of Mask2Former (Cheng et al., 2022) for image segmentation. In panoptic segmentation, merge the prediction of things and stuff according to the sorted scores. For video instance segmentation, we use query matching rather than introducing extra tracking components to generate instance ID, following previous work (Li et al., 2022d; Huang et al.. 2022: Li et al., 2023e). We adopt a near-online method for inference. We follow the original SAM (Kirillov et al., 2023) for interactive segmentation tasks by providing box and point prompts and obtaining the binary masks. All the parameters are shared across three different tasks.
Extension and Application. Since our model can perform various segmentation tasks, Our method can be extended to more datasets and tasks, including semantic segmentation on ADE20k (Zhou al., 2017) or video panoptic segmentation on VIPSeg (Miao et al., 2022). Moreover, our method can support prompt-driven video object segmentation by combining interactive segmentation with a video instance segmentation framework. We put the details of the extension and application for RMP-SAM in the appendix (Sec. 7.1).
4 EXPERIMENTS
Datasets. Our benchmark uses COCO (Lin et al., 2014) and YouTube-Vi VIS 2019 (Yang al., 2019) datasets for benchmarking. COCO-SAM is constructed using ground truth boxes and points randomly sampled from ground truth masks visual prompt inputs. In addition, to verify the effectiveness and generality of -MP-SAM, we also use other datasets, including ADE-20K (Zhou et al., 2017) and VIP-Seg dataset (Miao et al., 2022) in Sec. 4.1.
Evaluation Metries and Devices. For image segmentation, we adopt panoptic quality (PQ), which can be further decomposed into the segmentation quality (SQ) and the recognition quality (RQ). For

---

Published as a conference paper at ICLR 2025
Table Real-Time Multi-Purposs. Segmentation benchmark. Our proposed RMP-SAM achieves the best accuracy and speed trade-off on three tasks.
Methed
Backhone
COCO-Panoptic PQ...
PQ
COCO-SAM mloU
YouTube- VIS 2019
FL.PPs
Parameters
FPS
Mask2Former
R18
35.6
77.5
39.0
30.3
$4.7
38.6
89.8G
18.6
31.2
MaskFormer
R18
31.0
76.0
33.2
27.7
55.6
34,1
79.7G
18.5M
38,0
Daeepaa
18.7M
YOSO
R18
31.6
76.8
36.4
24.4
45.7
31.4
57.3G
18.7M
41.0
RMP-SAN (MMPS))M(Ors)
R18
39.9
78.6
43.3
34.8
52.7
36.7
60.5G
22.8M
40.3
2armerrrmer
79.8
35.6
1836
26.6
MaskFormer
R50
37.4
77.3
41.2
31.8
58.8
40.0
143.0G
45.2M
34,3
KMuX-Deepl.ab
28.4
20.1
YOSO
R50
37.3
76.9
42.7
29.2
49.2
40.7
119.0G
45.1M
36.3
RMP-SAM(Ours)
R50
46.9
80.8
51.6
39.8
$7.9
46.2
123.0G
47.2M
35.1
MaskFormer
TopFormer
29,9
YOSO
TopFormer
31.0
75.9
33.0
26.9
45.1
27.9
36.8G
12.9M
31.1
RMP-SAM(Ours)
TopFormer
34.6
77.0
37.8
29.8
53.3
41.7
40.1G
15.0M
30.7
the VIS task, we use mAP as our primary metric. We further report the semantic segmentation results (mloU) and video panoptic segmentation results (STQ (Weber et al., 2021) and VPQ (Kim et al., 2020)). Speed testing is conducted fairly on one A100 GPU for all models.
Re-implemented Baselines. Since previous works explore joint co- co-training for image, video, and interactive segmentation in one model. We re-implement several representative baselines using the same codebase, including non-real-time models (Mask2Former (Cheng al., 2022), kMaX- Dae (Yu et al., 2022)) for reference. We benchmark multiple different methods, including different backbones and decoders.
Implementation Details. We implement our models and all other baselines in PyTTrch (Paszke al., 2019). We use the distributed training framework with 16 A100 GPUs. Each mini-batch has two images per GPU, and each batch contains one data type. In particular, we adopt pseudo video training on COCO by moving image masks with random directions. All the models are trained with 12 epochs. For data augmentation, we adopt large-scale jitter as previous works (Cheng et al., 2022) build strong baselines. For all models, we adopt the same training steps and optimizers. Refer to the appendix (Sec. 7.1) for more details.
4.1
MAIN RESULTS
Our Benchmark Results. We list our benchmark results in Tab. We benchmark recent state-of- the-art methods using the same training and test settings. From the table, our proposed RMM-SSA achieves the best speed and accuracy trade-off on all three visual segmentation tasks under the various backbones. Mask2Former (Cheng et al., 2022) achieves similar our method. However, their speed is still limited, and they are 7-8 FPS slower than our methods. YOSO (Hu al., 2023) runs as fast as our method. However, the performance gap still significant. Thus, our proposed RMP-SAM is simple yet effective baseline for real-time and Multi-Purpose settings.
partially stronger results than
Compare with SAM-like Methods. Tab. 4 provides detailed comparison between our models and previous SAM-like models. We report the results of mask AP of the COCO validation set, the parameter count, and FLOPs of each model. We first generate bounding boxes for each image using Mask R-CNN with ResNet50. Then, we send these bounding boxes to SAM-like models as box prompts. Our method is co-trained using SAM data and demonstrates even better results than most previous efficient SAM-like methods. has strength in computational efficiency, with an additional function for video instance segmentation and panoptic segmentation.
Comparison with Specific Design Models on VIP-Seg. Tab. 5(a), we verify the effectiveness of RMP-SAM more challenging video segmentation task, video panoptic segmentation (VPS) the VIP-Seg dataset. Compared with recent works (Li et al., 2023e; Miao al., 2022), RMP-SAM also archives the best speed and accuracy trade-off, despite RMP-SAM is not specifically designed for VPS.
Comparison with Specific Design Models ADE-20k. In Tab. 5(b), we further transfer SMP-SAM on ADE-20k datasets. Since our method is a real-time model, we only list several representative works for reference. Our method still achieves stronger results than recent work (Hu et al., 2023).

---

Published as a conference paper at ICLR 2025
Table 4: Comparison with SAM-like methods on COCO instance segmentation dataset. We first generate bounding boxes using Faster R-CNN, which are used as visual prompts for segmentation. We report the results of mask AP of the COCO validation set, the parameters, and the FLOPs.
Method
mAP
#aramm.
FLOPs(G)
VIS
Interactive
SAM-H (Kirillov et al., 2023)
35.6
641M
3000
X
X
TinySMM (Shu et al., 2023)
32.5
10M
X
MobileSAM (Zhang
2023a)
32.1
10M
42
X
EdgeSAM (Zhou
2023)
31.9
10M
22
X
X
EfficientSA (Xiong al., 2023)
32.4
10M
X
X
FastSAM (Zhao al., 2023)
34.3
68M
344
X
X
RMP-SAM
33.2
47M
35
(b)
Oppergenntss
(c)
Maa dipecquation Adapio
Mad
Mussc
Adla
Mid
Appattdd quatics
Cless
Doooder
Uxdan
Deooder
Alpoelquatio
Civss
Deroder
Pistan
Docouptdd
Decoglee
Musi
Doodls
Mask
Masi
Docedar
Mask
Clas
Cless
Class
Pronpt quates
Uydndiqaaiio
Uxiand quatics
Pompp quates
paurpe spasim
Figure 3: Meta-architecture exploration. (a), Simple shared decoder design. (b), Decoupled decoder design with two heads. (c), Shared decoder with decoupled adapter. (d), Decoupled decoder with the decoupled adapters. Best viewed in color and zoom in.
Table Compared with specifically designed segmentation models.
(a) Comparison with video segmentation method on (b) Comparison with panoptic segmentation method on ADE-20k validation set.
VIP-Seg validation set.
Method
backbone
VPQ
STQ
FPS
Method
backbone
PQ
Parameters
FLOPs
Cii-PanoFCNN
R50
22.9
31.5
8
PanSegFormer
R50
36.4
5IM
214G
Video K-Net
R50
33.1
10
MaskFormer
R50
34.7
45.2M
181G
Tube-Link
STDCvl
30.6
32.0
14
Mask2Former
R50
39.7
45.2M
226G
Tube-Link
STDCv2
31.4
32.8
12
YOSO
R50
38.0
45.
176G
Ours
R18
32.5
33.7
30
RMP-SAM
R50
47.2M
179G
4.2 ABLATION STUDY AND VISUAL ANALYSIS
Ablation Meta-Architecture Design. In Tab. 6(a), further explore the meta-architecture design, as shown in Fig. 3. We use R50 as the backbone and keep the adapter unchanged. The table shows that a shared decoder architecture achieves the best parameter and performance trade- off. These findings may conflict with recent multi-task work (Li et al., 2022c) since the decoupled decoders may have works and the representation power is far from the heavier backbone such as Swin (Liu et al., 2021) or ViT-large (Dosovitskiy al., 2021), adding more capacity in the decoder may not boost the final performance.
capacity. We argue that since our feature extractors
weaker than those
Effectiveness of Joint Co-training. We further explore the impact of each dataset in Tab. 6(b). We find joint co-training with image and video data leads to better performance for video instance segmentation but reduces the performance of panoptic segmentation. Adding COCO-SAM training leads to few performance drops. This is because the learning targets are different for object queries and prompt queries.
Ablation Shared Decoder Design. Tab. 6(c) shows using simple pooling-basee dynamic convolu- tion performs well under real-time settings, where we adopt ResNet18 as the backbone and keep the adapter unchanged. In particular, there is difference in panoptic segmentation results and slight gap for interactive segmentation. However, per-pixel cross-attention introduces extra computation costs.
Ablation on Adapter Design. Then, we explore the adapter design in Tab. 6(d). In particular, we pre-trained model without adapters for initialization to the effect of different adapters.
8

---

Published as a conference paper at ICLR 2025
Figure 4: The visualization results of YouTube- VIS 2019 and COCO datascts. The first two rows visualize five frames of inputs. The same instances are in the same color. The third row shows the interactive segmentation results with single-point prompt (green color). The last row shows the panoptic segmentation results.
Table 6: Ablation study on RMP-SAM's component design.
(a) Ablation on meta-architecture.
(b) Ablation joint co-training. COCO--CCCOppp
b: YT-VIS 2019. COCO-SAM.
Setting
COCO-Panopiic
COCO-SAM
Parameters
Fig. (a)
43.8
54,2
46.3M
Setting
COcO-Panoptic
YT-VIS 2019
COCO-SAM
Fig.
44.0
55.2
53.6M
36.6
Fig. 3 (c)
44.6
56.7
47.3M
b
Fig.
a+b
36.2
36.0
50.7
+ bbec
35.7
35.3
$0.9
(c) Ablation on shared decoder design. DC: dynamic (d) Ablation on adapter design. CA: cross-attention. convolution. DCG: dynamic convolution with gating. DC: dynamic convolution.
Settings
COCO-Panoptic
COCO-SAM
Auti
Aprompe
COCO-Panoptic
COCO-SAM
Per-Pixel Cross Attention
55.3
Pooling DC
43.8
55.7
CA
CA
42.6
$4.3
Pooling + DCG
44.6
56.7
DC
DC
44.7
52.1
DC
CA
44.6
56.7
CA
DC
44.5
56.2
In particular, we find that using asymmetric adapters works well for balanced results for object queries and prompt queries since the goals of the two queries are different. The former needs the spatio-temporal and scene-level context, while the latter only cares about the region context with the input position the guidance. Our adapter can also work with other architecture, shown in the appendix (Sec. 7.2).
Result Visualization. In Fig. 4, give the three different visualization results the YouTube- VIS 2019 and COCO datasets. We generate these visualization results using only single model thanks to the joint co-training strategy. The first three rows Fig. show the video instance segmentation. The fourth row shows the interactive segmentation results with a single-point prompt (shown in green color), and the last row shows the results of panoptic segmentation.
RELATED WORK
Universal Segmentation. Prior research has focused on designing segmentation models tailored for specific tasks. Recent developments in this field (Dosovitskiy et al., 2021; Liu al., 2021; Carion et al., 2020; Zhang et al., 2021; Cheng et al., 2022; Yu et al., 2022; Li et al., 2021b; 2023c; Gu al., 2023; Yuan et al., 2022: Li et al., 2022c; 2023d; 2024b; Yuan al., 2024; Yang et al., 2021b;

---

Published as a conference paper at ICLR 2025
2024) have shifted towards employing mask classification architectures, coupled with an end-to- end set prediction objective, to facilitate universal segmentation. This approach has outperformed specialized models (Chen et al., 2019; Kirillov et al., 2019; He al., 2017; Li et al., 202 Huang et al., 2019; Yuan et al., 2020; Zhao et al., 2017; Chen et al., 2018; Li et al., 2022b; 2020a; Chen al., 2017) across various segmentation tasks, including those involving images, videos, and point clouds (Kim et al., 2022; Li et al.. 2022d: 2023e; Cheng ct al., 2021a; Xu et al., 2022). Specifically, Mask2Former (Cheng et al., 2022) uses masked-attention mechanism and surpasses the performance of prior specific image segmentation models. Similarly, Tube- (Li et al., 2023e) outperforms earlier models (Li et al., 2022d:b; Liu et al., 2020; Shin et al., 2023; Goel al., 2021) tailored for specific video segmentation tasks. Semantic-SAM (Li et al., 2023a), built on Mask-DINO (Li et al., 2023b), employs a hybrid query design to facilitate panoptic and interactive segmentation within single framework. However, despite the strong performance of these works, their complexity, stemming from the heavier encoder or the cascaded decoder designs, poses significant challenges for integration into actual products. Slow processing speeds and substantial parameter overhead primarily hinder these.
Efficient Model Design. This research direction primarily concentrates on the development of efficient CNNs (Howard et al., 2017; Sandler et al., 2018; Howard et al., 2019; landola et al., 2016; Zhang al., 2018; Ma et al., 2018; Han et al., 2020), transformers (Maaz et al., 2022; Pan et al., 2022; Mehta & Rastegari, 2022), and hybrid architectures (Zhang et al., 2023b; Ma et al., 2022; Chen et al., 2022b; Li et al., 2022a; Zhou et al., 2022), aimed at advancing visual representation learning. Most research investigates backbone design within the constraints of real-time processing and parameter efficiency. These studies are orthogonal to our work, which primarily explores the impact of data, task, and decoder design. Additionally, our work employs efficient models as encoders and presents detailed benchmarks and analyses of the encoder's effects.
Efficient Segmentation. Previous studies (Zhao et al., 2018; Li et al., 2020b; Yu et al., 2018; Hu et al., 2023; Hong et al., 2021; Wan et al., 2023; Yu et al., 2021; Mehta et al., 2018) on efficient segmentation have predominantly concentrated on closed-set and specific domains. In particular, a significant portion of this research (Li et al., 2020b; Hu et al., 2023; Mchta et al., 2019) is dedicated to driving scenarios. Recently, various studies (Zhang et al., 2022; Wan et al., 2023; Zhou et al., 2023) have developed efficient segmentation techniques that facilitate model execution on mobile devices. Mobile SAM (Zhang et al., 2023a) introduces streamlined encoder distillation method. Fast SAM (Zhao et al., 2023) employs a single-stage instance segmentation framework that directly decodes llss-agnosttic masks. Recently, multiple studies have been conducted on efficient panoptic segmentation (Sun et al., 2023; Hu et al., 2023; Mohan & Valada, 2021) and rapid video instance segmentation (Yang et al., 2021a; Zhang et al., 2024). However, these real-time segmentation methods are limited to specific tasks. We contend that a multi-purpose model capable of real-time performance would have wide-ranging applications in editing, tracking, and segmentation functionalities within various products.
Interactive Segmentation. Interactive segmentation (Kirillov et al., 2023; Chen et al., 2022a; Sofiiuk et al., 2022: Lin et al., 2020; 2022) distinguishes objects by actively incorporating user inputs. Recently, SAM (Kirillov et al., 2023) has shown remarkable segmentation outcomes by leveraging multiple user interaction inputs and training on SA-1B data. Inspired by SAM,, several studies (Zhang et al., 2023a; Zhou et al., 2023) have concentrated on employing a lighter image encoder to adapt SAM for real-time scenarios.
6 CONCLUSION
In this work, we explore a new challenging setting, real-time multi-purpose segmentation. to explore multi-task segmentation in one efficient model. To achieve this, we first benchmark existing image segmentation approaches by extending their query for video and interactive segmentation. However, no work can achieve the best speed and accuracy trade-off on all three sub-tasks. To solve this problem, we introduce RMP-SAM, a real-time segmentation model. has a simple shared decoder with two key designs: shared dynamic convolution and asymmetric adapters. These designs lead to the best trade-off between speed and accuracy on three segmentation tasks. The proposed RMP-SAM model is versatile and effective for various tasks, including image segmentation, video segmentation, and iterative segmentation. To our knowledge, no real-time model can achieve this goal. The proposed benchmark and RMP-SAM baseline can facilitate future research in this new setting.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by the National Key Research and Development Program of China No.2003YFF38877003) This work is also supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) contract number 140D0423C0074, The U.S. Government is authorized reproduce and distribute reprints for Gov- ermmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, or the U.S. Government.
REFERENCES
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 1,5,9 9
Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Hybrid task cascade for instance segmentation. In CVPR, 2019. 10
Liang- Chig- Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. WxXiv:776.00588/ 2017. 10
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018. 10
Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, and Hengshuang Zhao. Focalclick: Towards practical interactive image segmentation. CVPR, 2022a. 10
Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengehen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. In CVPR, 2022b. 10
Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G Schwing. Mask2former for video instance segmentation. arXiv:2112.10764, 2021a. 10
Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS, 2021b.
Bowen Cheng. Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022. 1, 2, 3, 6, 7, 9, 10, 17, 18
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021. 1, 5.
Golnaz Ghiasi, Yin Cui, A. Srinivas, Rui Qian, Tsung-Yi Yi Lin, Ekin Dogus Cubuk, Quoc V. Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In CVPR, 2020. 17
Vidit Goel, Jiachen Li, Shubhika Garg, Harsh Maheshwari, and Humphrey Shi. Msn: Efficient online mask selection network for video instance segmentation. In CVPR, 2021. 10
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. ICLR, 2021.
Xiuye Gu, Yin Cui, Jonathan Huang, Abdullah Rashwan, Xuan Yang. Xingyi Zhou, Golnaz Ghiasi, Weicheng Kuo, Huizhong Chen, Liang- -Chnh Chen, et al. Dataseg: Taming universal multi- dataset multi-rask segmentation model. arXiv:2306.01736 2023.
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In CVPR, 2020. 10
11

---

Published as a conference paper at ICLR 2025
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017. 10
Weixiang Hong, Qingpei Guo, Wei Zhang, Jingdong Chen, and Wei Chu. Lpsnet: A lightweight solution for fast panoptic segmentation. In CVPR, 2021. 10
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In ICCV, 2019. 10
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv: 1704.0486, 2017. 10
Jie Hu, Linyan Huang, Tianhe Ren, Shengchuan Zhang, Rongrong Ji, and Liujuan Cao. You only segment once: Towards real-time panoptic segmentation. CVPR, 2023. 2, 3, 7, 10
De-An Huang, Zhiding Yu, and Anima Anandkumar. Minvis: A minimal video instance segmentation framework without video-based training. In NewrIPS, 2022. 6, 17
Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In ICCV, 2019. 10
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv: 100,00660, 2016. 10
Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. OneFormer:
One Transformer to Rule Universal Image Segmentation. In CVPR, 2023. 1,3
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021. 1
Dahun Kim, Sanghyun Woo, Joon- Joon-Young Lee, and In So Kweon. Video panoptic segmentation. In CVPR, 2020.
Dahun Kim, Jun Xie, Huiyu Wang. Siyuan Qiao, Qihang Yu, Hong-Seok Kim, Hartwig Adam, In So Kweon, and Liang-Chich Chen. Tubeformee-deeplaab. Vidco mask transformer. In CVPR, 2022. 10
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In CVPR, 2019. 3, 4, 10
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, 2023. 1, 3, 4, 6, 8, 10, 18
Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything at any granularity. In ECCV, 2023a. 10, 18
Feng Li, Hao Zhang, Huaizhe xu, Shilong Liu, Lei Zhang, Lionel M. Ni, and Heung- Yeung Shum. Mask DINO: Towards a unified transformer-base framework for object detection and segmentation. In CVPR, 2023b. 10
Jiashi Li, Xin Xia, Wei Li, Huixia Li, Xing Wang, Xuefeng Xiao, Rui Wang, Min Zheng, and Xin Pan. Next-vit: Next generation vision transformer for efficient deployment in realistic industrial scenarios. arXiv:2207.05501, 2022a. 10
Minghan Li. Shuai Li, Xindong Zhang, and Lei Zhang. Univs: Unified and universal video segmenta- tion with prompts queries. In CVPR, 2024a. 18
12

---

Published as a conference paper at ICLR 2025
Xiangtai Li, Zhao Houlong, Han Lei, Tong Yunhai, and Yang Kuiyuan. Gff: Gated fully fusion for semantic segmentation. In AAAI, 2020a. 10
Xiangtai Li, Ansheng You, Zeping Zhu, Houlong Zhao, Maoke Yang, Kuiyuan Yang, and Yunhai Tong. Semantic flow for fast and accurate scene parsing. In ECCV, 2020b. 10
Xiangtai Li. Hao He, Yibo Yang, Henghui Ding, Kuiyuan Yang, Guangliang Cheng, Yunhai Tong, and Dacheng Tao. Improving video instance segmentation via temporal pyramid routing. T-PAMI, 2022b. 10
Xiangtai Li. Shilin Xu, Yibo Yang, Guangliang Cheng, Yunhai Tong, and Dacheng Tao. Panoptic- partformer: Learning a unified model for panoptic part segmentation. In ECCV, 2022c. 2, 5, 8,
Xiangtai Li, Wenwei Zhang, Jiangmiao Pang, Kai Chen, Guangliang Cheng, Yunhai Tong, and Chen Change Loy. Video k-net: simple, strong, and unified baseline for video segmentation. In CVPR, 2022d. 3, 4, 6, 10, 17
Xiangtai Li, Henghui Ding, Wenwei Zhang, Haobo Yuan, Guangliang Cheng, Pang Jiangmiao, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer-basee based visual segmentation: A survey. TPAMI, 2023c. 9
Xiangtai Li, Shilin Xu, Yibo Yang, Haobo Yuan, Guangliang Cheng, Yunhai Tong, Zhouchen Lin, Ming-Hsuan Yang, and Dacheng Tao. Panopticpantforerr+:: A unified and decoupled view for panoptic part segmentation. TPAMI, 2023d. 9
Xiangtai Li, Haobo Yuan, Wenwei Zhang, Guangliang Cheng, Jiangmiao Pang, and Chen Change Loy. Tube-link: A flexible cross tube baseline for universal video segmentation. In ICCV, 2023e.
6, 7, 10, 18
Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. Omg-seg: Is one model good enough for all segmentation? In CVPR, 2024b. 4,
Yanwei Li, Hengshuang Zhao, Xiaojuan Qi. Liwei Wang, Zeming Li, Jian Sun, and Jíaya Jia. Fully convolutional networks for panoptic segmentation. In CVPR, 2021a. 10
Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, Tong Lu, and Ping Luo. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. arXiv:2109.03814 2021b. 9
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 4, 6
Zheng Lin, Zhao Zhang, Lin-Zhuo Chen, Ming-Ming Cheng, and Shao-Ping Lu. Interactive image segmentation with first click attention. CVPR, 2020. 10
Zheng Lin, Zhao Zhang, Ling-Hao Han, and Shao-Ping Lu. Multi-mode interactive image segmenta- tion. In ACM MM, 2022. 10
Haotian Liu, Rafael A. Rivera Soto, Fanyi Xiao, and Yong Jae Lee. Yolactedge: Real-time instance segmentation on the edge (jetson agx xavier: 30 fps, rtx 2080 ti: 170 fps). arXiv:2012.122553 2020. 10
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 8.9
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s. In CVPR, 2022. 17
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 17
Hailong Ma, Xin Xia, Xing Wang. Xuefeng Xiao, Jiashi Li, and Min Zheng, Mocovit: Mobile convolutional vision transformer. arXiv:2205. /2635, 2022. 10
13

---

Published as a conference paper at ICLR 2025
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In ECCV, 2018. 10
Muhammad Maaz, Abdelrahman Shaker, Hisham Cholakkal, Salman Khan, Syed Waqas Zamir, Rao Muhammad Anwer, and Fahad Shahbaz Khan. Edgenext: efficiently amalgamated cnn- transformer architecture for mobile vision applications. In ECCVW, 2022. 10, 17
Sachin Mchta and Mohammad Rastegari. Mobilevit: Light- general- purpose, and mobile-
friendly vision transformer. In ICLR, 2022. 10, 17
Sachin Mehta, Mohammad Rastegari, Anat Caspi. Linda Shapiro, and Hannaneh Hajishirzi. Espnet: Efficient spatial pyramid dilated convolutions for semantic segmentation. In ECCV, 2018. 10
Sachin Mehta, Mohammad Rastegari, Linda Shapiro, and Hannaneh Hajishirzi. Espnetv2: A light- weight, power efficient, and general purpose convolutional neural network. In CVPR, 2019. 10
Jiaxu Miao, Xiaohan Wang, Yu Wu, Wei Li, Xu Zhang, Yunchao Wei, and Yi Yang. Large-scale video panoptic segmentation in the wild: A benchmark. In CVPR, 2022. 6,7
Fausto Milletari. Nassir Navab, and Seyed-Ahmal Ahmadi. V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV, 2016.
Rohit Mohan and Abhinav Valada. Efficientps: Efficient panoptic segmentation. 1JCV, 2021. 10
Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tzimiropoulos, and Brais Martinez. Edgevits: Competing light-weight cnns on mobile devices with vision transformers. In ECCV, 2022. 10
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: an imperative style, high-perfone regh-performance deep learning library. In NeurIPS, 2019.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv:2408.0077 2024. 3, 17
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. 10
Inkyu Shin, Dahun Kim, Qihang Yu, Jun Xie, Hong-Seok Seog-Seok&aaa Kim, Bradley Green, In-So Kweon, Kuk-Jin Yoon, and Liang-Chieh Chen. Video-kmax A simple unified approach for online and near-online video panoptic segmentation. WACV, 2023. 10
Han Shu, Wenshuo Li, Ychui Tang, Yiman Zhang, Yihao Chen, Houqiang Li, Yunhe Wang, and Xing- hao Chen. Tinysam: Pushing the envelope for efficient segment anything model. 1r21 2023. 3,8 8
Konstantin Sofiiuk, Ilya A. Petrov, and Anton Konushin. Reviving iterative training with mask guidance for interactive segmentation. In ICIP, 2022. 10
Peize Sun, Rufeng Zhang. Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka. Lei Li, Zehuan Yuan, Changhu Wang, and Ping Luo. SparseR-CNN, End-to-end object detection with learnable proposals. In CVPR, 2021. 5
Shuyang Sun, Weijun Wang, Qihang Yu, Andrew Howard, Philip Torr, and Liang- -hinh Chen. Remax: Relaxing for better training on efficient panoptic segmentation. arXiv:2306.17319, 2023. 10
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurlPS, 2017.
Qiang Wan, Zilong Huang. Jiachen Lu, Gang Yu, and Li Zhang. Seaformer: Squeeze-enhanced axial transformer for mobile semantic segmentation. In ICLR, 2023. 10
14

---

Published as a conference paper at ICLR 2025
Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chiel Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. CVPR, 2021.
Xinlong Wang, Tao Kong. Chunhua Shen, Yuning Jiang, and Lei Li. Solo: Segmenting objects by locations. In ECCV, 2020. 6
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In CVPR, 2023a.
Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. In ICCV, 2023b.
M. Weber, Xie, M. Collins, Yukun Zhu, P. Voigtlaender, H. Adam, B. Green, A. Geiger, B. Leibe, D. Cremers, Aljosa Osep, L. Leal-Taixé, and Liang-Chich Chen. Step: Segmenting and tracking every pixel. In NeurlPS, 2021. 7
Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xiang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest N. landola, Raghuraman Krishnamoorthi, and Vikas Chandra. Efficientsam: Leveraged masked image pretraining for efficient segment anything. In CVPR, 2023.
3.8
Shilin Xu, Xiangtai Li, Jingbo Wang. Guangliang Cheng, Yunhai Tong, and Dacheng Tao. Fashion- former: A simple, effective and unified baseline for human fashion segmentation and recognition. In ECCV, 2022. 10
Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Zehuan Yuan, Ping Luo, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In CVPR, 2023. 1,2
Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV, 2019. 2, 2.4, 6
Shusheng Yang, Yuxin Fang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Crossover learning for fast online video instance segmentation. In ICCV, 2021a. 10
Zongxin Yang, Yunchao Wei, and Yi Yang. Collaborative video object segmentation by multi-scale foreground-backgroundd integration. TPAMI, 2021b.
Zongxin Yang, Jiaxu Miao, Yunchao Wei, Wenguan Wang, Xiaohan Wang, and Yi Yang. Scalable video object segmentation with identification mechanism. TPAMI, 2024. 10
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In ECCV, 2018. 3, 10
Changqian Yu, Changxin Gao, Jingbo Wang, Gang Yu, Chunhua Shen, and Nong Sang. Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation. IJCV, 2021. 10
Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chich Chen. k-means mask transformer. In ECCV, 2022. 7, 9, 17
Haobo Yuan, Xiangtai Li, Yibo Yang, Guangliang Cheng, Jing Zhang, Yunhai Tong, Lefei Zhang, and Dacheng Tao. Polyphonicformer: Unified query learning for depth-aware video panoptic segmentation. ECCV, 2022. 9
Haobo Yuan, Xiangtai Li, Chong Zhou, Yining Li, Kai Chen, and Chen Change Loy. Open-vocabulan sam: Segment and recognize twenty-thousand classes interactively. ECCV, 2024.
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. In ECCV, 2020. 10
Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung- Ho Bae, Seungkyu Lee, and Choong Scon Hong. Faster segment anything: Towards lightweight sam for mobile applications. arXiv:2306.144 14289, 2023a. 3, 8, 10, 17
Jiangning Zhang, Xiangtai Li, Jian Li. Liang Liu, Zhucun Xue, Boshen Zhang, Zhengkai Jiang, Tianxin Huang. Yabiao Wang, and Chengjie Wang. Rethinking mobile block for efficient attention- based models. In ICCV, 2023b. 10
15

---

Published as a conference paper at ICLR 2025
Renhong Zhang, Tianheng Cheng, Shusheng Yang, Haoyi Jiang, Shuai Zhang, Jiancheng Lyu, Xin Li, Xiaowen Ying. Dashan Gao, Wenyu Liu, et al. Mobileinst: Video instance segmentation on the mobile. In AAAI, 2024. 3, 10
Tao Zhang, Xingye Tian, Yu Wu, Shunping Ji, Xuebo Wang, Yuan Zhang, and Pengfei Wan. Dvis: Decoupled video instance segmentation framework. ICCV, 2023c. 18
Wenqiang Zhang, Zilong Huang, Guozhong Luo, Tao Chen, Xinggang Wang, Wenyu Liu, Gang Yu, and Chunhua Shen. Topformer: Token pyramid transformer for mobile semantic segmentation. In CVPR, 2022. 4, 10
Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. In NewrIPS, 2021. 1, 4, 17
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018. 10
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017. 10
Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Icnet for real-time semantic segmentation on high-resolution images. In ECCV, 2018. 2, 3, 10
Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything. arXiv:2306.12135 12156, 2023. 8, 10
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. In CVPR, 2017. 6
Chong Zhou, Xiangtai Li, Chen Change Loy, and Bo Dai. Edgesam: Prompt-in in- ln-t distillation for on-device deployment of sam. arXiv:2312.06660, 2023. 3, 8, 10
Qianyu Zhou, Xiangtai Li, Lu He, Yibo Yang. Guangliang Cheng, Yunhai Tong, Lizhuang Ma, and Dacheng Tao. Transvod: End-to-end video object detection with spatial-temporal transformers. TPAMI, 2022. 10
16

---

Published as a conference paper at ICLR 2025
7 APPENDIX
In this appendix, we add deeper discussion different views in Sec. 7.1. Next, we present more detailed results to support the benchmark and our proposed RMP-SAM in Sec. 7.2. Then, in Sec. 7.3, present more detailed visual comparison using our model. Finally, in Sec. 7.4, we discuss several failure cases and potential future work. Moreover, we have appended short video to introduce work briefly and append the code for reference.
RMP-SAM with existing unified methods from
7.1 MORE METHOD DETAILS
More Implementation Details. We implement our proposed RMP-SAM and benchmark methods using PyTorch and train them in the same environment. For KMaX-DeepLab (Yu et al., 2022), we excluded the auxiliary semantic segmentation loss, instance discrimination loss, and preserving PQ-style loss alignment with Mask2former (Cheng et al., 2022) for fair comparison. We utilize the AdamW W (Loshchilov & Hutter, 2019) optimizer with a weight decay of 0.05, and the learning rate is set to le-4 for all methods. We warm up the learning rate in the first 500 iterations using linearly increased strategy and decay the learning rate at 8 and 11 epochs by factor of 10. For data augmentation, we use the large-scale jittering (LSJ) augmentation (Ghiasi al., 2020) with random scale sampled from the range of 0.1 to 2.0 and followed by a fixed size crop to 1024x 1024. For joint dataset training, we sample clip images with ratio of 25: for COCO- POCCO-panptic, Youtube-2019, and SAM datasets. For fair comparison with efficient SAM models, we adopt the SAM dataset for fine-tuning. In particular, we adopt the co-trained checkpoint and fine our visual prompt segmentation branch using 10% SAM dataset. The training follows the previous works (Zhang et al., 2023a).
More Inference Details. For video instance segmentation, we adopt simple object query matching (Li et al., 2022d; Huang et al., 2022) for all methods to keep the fair comparison to assign instance ID. We mainly use point prompts for interactive segmentation, which are more challenging than box prompts. For panoptic segmentation, all hyperparameters are the same. The FPS is obtained one 40GB A100 GPU for the main paper and supplementary.
Detailed Comparison with SAM-2. We present more detailed comparison with SAM-2 (Ravi et al., 2024) here. (1) For functionality, SAM-2 mainly focuses on video object segmentation and interactive segmentation. Compared with SAM, adds mask tracking ability given visual prompts, while our method mainly explores multi multi-purpose and multi-task real-time segmentation. We unify panoptic segmentation, video instance segmentation, and interactive segmentation one model with the requirements in real time. (2) For data scale and diversity, SAM-2 builds a large scale dataset and mainly has one purpose: Video Object Segmentation. Our RMP-SAM only involves small set of public data sources and has multiple purposes: segmentation, mask labeling, mask tracking, and panoptic segmentation. (3) For goals, SAM-2 aims at the production level with large-scale datasets (including internal datasets) co-training. Our RMP-SAM aims at efficient model design and performs well under real-time constraints. (4) Last, our work concurrent with SAM-2 since our work also inspired by pioncering SAM.
7.2 MORE EXPERIMENT RESULTS
Benchmark More Methods. In Tab. 7, we benchmark more models and backbones (Zhang et al., 2021; Maaz et al., 2022; Mehta & Rastegari, 2022) as the supplementary to main paper. Again, for various backbones and different heads, our model can achieve better comparable results across three different tasks while running in real-time. based method, K-Net, our approach achieves better performance on the R18 and R50 backbones and comparable results on the lightweight backbones (EdgeNexT).
particular, compared with the dynamic kernel-
Scale Up Joint Co-training. In the bottom of Tab. 7, we further scale up our model on a large backbone, ConvNeXt large (Liu et al., 2022). From the table, we find there are significant gains in all settings while still running with 8 FPS. This means proposed multi-purpose segmentation still has room to balance the accuracy and inference speed when adopting the large model. This also verifies the scalability of SMM.
17

---

Published as a conference paper at ICLR 2025
Method
Rackhone
PQ
COCO-Pamppic
SQ
PQn
PQL
COCO-SAM
mloU
YouTub--VIS 2019
mAP
FL.P%
Parameters
FPS
YOSO
EdgeNexT
31.8
74.5
35.6
26.1
47.4
35.4
403
15.3M
28.6
K-Net
R18
33.1
75.7
36.8
27.4
53.6
34,8
124G
25.6M
30.4
R50
K-Net
EdgoNexTT
38.0
76.7
42.5
31.4
55.7
44,0
1080
19.5M
30.5
Mask Furmer
EdgeNexT
39.8
78.8
43.6
34.0
54.2
73G
12.0M
25.6
RMPSAM
RT8
199
78.6
43.3
34.8
52.7
38.7
60.50
22XM
40.3
RMP-SAM
RS0
46.9
80.8
51.6
39.8
57.9
46.2
123.00
47.2M
35.1
RMP-SAM
EdgeNexT
38.0
42.1
31.9
46.3
RMPSAM
Co
52.3
82.5
58.6
42.8
61.1
55.6
7000
239M
8.0
Table 7: Benchmark More Models on Real-Time multi-purpose Segmentation. The GFLOPs are
obtained with 1333 800 imputs.
Table Explore adapter Design on More Methods. We apply our dual adapter on the Mask2Former architecture and find it works well.
Method Mask2Former
Backbone
PQ
mloU
R50
43.2
57.0
Mask2Former adapter
R50
43.0
58.1
Table 9: Compared with real-time based interactive segmentation model.
Method
I-loU
FPS
FLOPs
SAM(L)
55.7
1.2
1315.0
Semuntic-SAMLL)
57.0
2.3
745.8G
RMP-SAM (ours)
379
33.1
123.00
Explore Dual Adapter Design on Other Methods. We explore the effectiveness of our dual adapter on Mask2Former (Cheng al., 2022) and add the adapter module the last decoder layer of Mask2Former. As shown in Tab, 8, Mask2Former with adapter will improve 1.1 mIoU score compared with origin Mask2Former for interactive segmentation and only drop 0.3 PQ for panoptic segmentation. This result indicates that the adapter module applies to our method and can be generalized other methods.
Single-granularity Interactive Segmentation. In Tab. 9, we evaluate the 1-click mloU (denoted as 1-IoU) for SAM (Kirillov et al., 2023), Semantic-SAM (Li et al., 2023a) and our model on COCO Va12017. Our model surpasses SAM and Semantic-SAM in 1-IoU performance while also being faster and requiring less computation.
Comparison with Recent Efficient SAM Models. As shown in Tab. 4, compare method
with SAM-like methods for coco instance segmentation. Tab. 10, Tab. 11, Tab. 12. Our method achieve the stronger results than these counterparts, indicating the effectiveness of model design. We also provide visualization in Fig. 5 for easier and direct comparison.
Comparison with Strong Video Segmentation Models. Several works explore stronger video segmentation designs. We argue the scope of our research is orthogonality to the works. Firstly, since method is not designed for specific models and method is trained multiple purposes tasks, directly comparing our method with previous works (Li et al., 2023e; Zhang et al., 2023c) may not be fair. As our core goal is unifying multiple tasks in real-time scenarios and adopting multi-task co-training strategy, our method may not perform the best on a single task. Secondly, The work (Li al., 2024a) also designs a unified model for video segmentation tasks. However, it cannot perform SAM-like segmentation in real time. We adopt joint image-video data co-training rather on image or video data. Thus, we can keep SAM-like segmentation, panoptic segmentation, and video instance segmentation in one model and run in real time. Thirdly, all these methods focus on achieving strong results rather than real-time design. The work (Zhang et al., 2023c) uses extra transformer encoders after heavy Fosk2rrrmer architectures, bringing more computation costs. We follow the work (Li et al., 2023e; Zhang et al., 2023c), with the modification on replacing pre- pre-training on COCO with co-training COCO, COCO-SAM, and YTVIS-2011 Then, follow the works (Li et al., 2023e; Zhang et al., 2023c) and finetune the pre-trained model on YTVIS-2019, As shown in Tab. 14, nevertheless, achieved mAP of 47.2 when using the ResNet-50 backbone. To provide more
18

---

Published as a conference paper at ICLR 2025
Table 10: Compared with SAM-like methods on COCO instance segmentation. We adopt the Mask R-CNN with ResNet50 as detector. The bounding box obtained from Mask R-CNN is adopted as a box prrmpt for SAk-like methods. We report average precision at various thresholds. The AP,. APm m. and AP, mean the mean average precision for small, medium, and large objects.
Method
mAP
AP
AP7s
AP,
AP
AP:
#param.
FLOPx(G)
SAM-H
35.6
54.9
38.4
17.2
39.1
51.4
641M
3000
TinySAM
32.7
53.6
34.0
15.5
35.3
48.3
10M
42
MobileSAM
32.3
53.0
33.6
14.7
25.3
48.6
10M
42
EdgeSAM
32.2
53.4
33.3
14.8
35.0
47.0
10M
22
EfficientSAM
32.6
53.3
33.9
16.2
35.2
47.1
10M
RMP-SAM(OUrs)
33.5
54.6
35.4
14.3
36.3
50.8
47M
35
Table 11: Compared with SAM-like methods COCO instance segmentation and adopt Mask R-CNN with ResNet101 as a detector.
Method
mAP
AP 30
AP7s
AP,
APm m
AP:
#puram.
FLOPs(G)
SAM-H
38.4
59.4
41.3
24.1
46.9
56.9
641M
3000
TinySAM
35,4
57.8
36.9
17.6
39.2
51.8
10M
42
MobileSAM
34.9
57.2
36.2
17.0
38,9
52.1
10M
42
EdgeSAM
34.9
57.6
36.2
17.3
38.9
50.3
10M
22
EfficienSAA
35.2
57.6
36.5
18.2
39.0
51.0
10M
RMP-SAM((Ors))
35.6
58.0
36.9
17.4
40.2
53.3
47M
35
comparison, ConyNext-L. backbone and train model based the joint co-training described in the paper, achieving result of 62.2. The results indicate our method can still achieve comparable results with these expert models.
Testing the efficiency across different GPU platforms. We have evaluated several methods from Table across multiple GPU platforms, including A100-40G, A10-22G, and 3090-24G. Unfortunately, we currently do not have access to a V 1100 GPU and are unable to provide its corresponding results.
All model results for each specific GPU are generated on the same machine, The FPS and GFlops values are calculated using a 1333 x 800 pixels resolution image. We report these results with the ResNet- backbone.
7.3 VISUAL COMPARISON
Attention Map Visualization. As shown in Fig. 6, we generate the heatmap to illustrate our integrated multi-tasks, which include panoptic segmentation, interactive segmentation, and video instance segmentation. We sample four continuous frames from the YouTube-VVL 2019 validation set and visualize randomly selected object. The heatmap the attention score between object/prompt query and image features. The results indicate that despite using a shared decoder across different
FAISAM
40
TaySaM
RMP-AM
EladAM
300
3000
FLOPs(G)
Figure 5: The visualization results of SAM-like methods on COCO instance segmentation. We adopt the ViTDet with ViT-b the detector. The larger dot means more parameters.
19

---

Published as a conference paper at ICLR 2025
Table 12: Compared with SAM-like methods on COCO instance segmentation and adopt ViTDet with ViT-b as a detector.
Method
mAP
AP50
AP75
AP;
AP
APt
#param.
FLOPs(G)
SAM-H
45.7
69.4
49.8
26.9
49.0
64.1
641M
3000
TinySAM
39.2
40.4
23.7
42.3
MobileSAM
38.3
63.5
39.3
22.3
41.7
55.3
10M
42
EdgeSAM
39.0
64.8
40.1
23.8
42.3
54.5
10M
22
EfficientSAM
39.4
64.9
40.9
23.8
42.3
55.3
10M
RMP-SAM
41.0
65.9
42.5
24.2
44.7
57.3
47M
35
Table 13: Testing the efficiency across different GPU platforms.
Method
GPU
FLOPs
Parameters
FPS
Mask2Former
A100-40G
89.8G
18.6M
31.2
KMaX-DeepLat
A100-40G
87.1G
18.7M
15.0
YOSO
A100-40G
57.3G
18.7M
41.0
RMP-SAM
A100-40G
60.5G
22.8M
40.3
Mask2Former
A10-22G
89.8G
18.6M
10.1
kMaX-Deepl.ab
A10-22G
87.1G
18.7M
4.3
YOSO
A10-22G
57.3G
18.7M
13.6
RMP-SAM
A10-22G
60.5G
22.8M
14.2
Fask2Former
3090-24G
89.8G
18.6M
25.6
KMaX-DeepLab
3090-24G
87.1G
18.7M
9.0
YOSO
3090-24G
57.3G
18.7M
31.4
RMP-SAM
3090-24G
60.5G
22.8M
32.0
Table 14: Comparison with stronger video instance segmentation methods. Our RMP-SAM can achieve comparable results with more funetionalities on image panoptic segmentation and interactive segmentation.
Method
Backbone
Youtube- VIS 2019
COCO-SAM
COCO-Panoptic
UniVS
R50
47.4
DVIS
R50
51.2
Tube-Link
R50
52.8
RMP-SAM
R50
47.2
UniVS
Swin-L
60.0
DVIS
Swin-L
63,9
Tube-Link
Swin-L
64.6
RMP-SAM
ConvNeXt-L.
62.2
60.8
52.0
tasks, each task can effectively learn. This indicates that our design is effective and can significantly reduce the computational complexity of the model.
Comparison on Youtube- VIS dataset. In Fig. 7, we compare our RMP-SAM (right) with a strong baseline Mask2Former (left) with the same ResNet50 backbone. Our methods achieve more consistent tracking and segmentation results these examples.
Comparison COCO Panoptic Segmentation dataset. In Fig. 8, We demonstrate that RMP-SAM model surpasses K-Net panoptic segmentation tasks. RMP-SAM exhibits enhanced accuracy in processing complex scenes and maintaining fine details, especially in segmenting object edges and managing overlapping objects.
More Interactive Segmentation Results. In Fig. 9, present further visualizations of interactive segmentation, showcasing segmentation outcomes for 'things' and 'stuff'. In these visualizations, green dots signify manually specified prompts. Our model exhibits precise segmentation abilities
20

---

Published as a conference paper at ICLR 2025
Image
PS
IS
VIS
Figure 6: The heatmap shows the results of three distinct tasks: panoptic, interactive, and video instance segmentation, represented by PS, IS, and VIS, respectively. The four test images sampled from the YouTube- VIS 2019 validation set, and the heatmap is the attention score between object/prompt query and image features. The red point in is the point prompt.
RAP-SAM
Mask2Former
Figure 7: When compared to the Mask2Former(riht)) on the YouTube-VIS 2019 dataset, our RMP- SAM (left) demonstrates superior performance in recognizing and segmenting objects in certain scenarios, and it also holds advantage in detailing edges.
21

---

Published as a conference paper at ICLR 2025
Figure 8: When compared with K-Net (bottom) on the COCO dataset, our RMP-SAM (top) shows significant advantage, achieving more accurate segmentation of foreground and background, with smoother edges.
22

---

Published as a conference paper at ICLR 2025
Things
M
Stuff
Figure 9: Here are additional results of interactive segmentation. For point prompts box prompts, RMP-SAM segments out the corresponding object based on the spatial location of the prompt. We have provided both segmentation results for 'things' and 'stuff", demonstrating that RMP-SAM can segment both while maintaining strong performance.
Figure 10: Failure modes of RMP-SAM in image/video segmentation.
for "things' and 'stuff'. Notably, our model accurately identifies and segments the object even when prompt points are situated on object boundaries.
7.4 LIMITATIONS AND FUTURE WORK
Failure Cases Analysis. We also analyze failure cases in video/image segmentation and identify two typical failure modes of RMP-SAM. Firstly, illustrated in Fig. 10, when faced with multiple objects with high overlap, RMP-SAM struggles to fully recognize them, resulting in the omission of several masks. Secondly, in crowded scenarios, RMP-SAM faces challenges recognizing and segmenting all instances with a limited number of instance kernels.
Future Work Discussion. There several directions explore for real-time multi-purposs segmentation. The first direction is to achieve better-balanced performance on image, video, and interactive segmentation (since there is still lot of room for performance, as shown in the last of Tab. 7). The second direction is to speed up the model and deploy the model on the edge device, such as the iPhone. The third direction to explore different knowledge distillation approaches transfer the vision foundation model in real-time multi-purpose models. These will be our future work.
23


Reviews:
[{"review_id": "GDrfMprzci", "reviewer": "Reviewer_ycmt", "summary": "The paper presents a real-time, versatile segmentation model capable of interactive segmentation, panoptic segmentation, and video instance segmentation. \nWhile retaining the SAM encoder-decoder structure, the model incorporates an efficient encoder and adapter to enhance performance.\nIn the decoder, RAP-SAM introduces a three-stage pipeline that leverages novel pooling-based dynamic convolutions to refine mask tokens. Following the decoder, two additional prompt adapters are implemented to improve interaction between visual prompts and segmentation tokens.\nRAP-SAM demonstrates efficiency and generalizability across various segmentation benchmarks.", "questions": "1. Do you have the results for TopFormer in Table 3? Additionally, please bold the results in all comparison tables for clarity.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The model achieves multi-purpose segmentation through an efficient structure and unified training approach.\n2. The paper is well-written and easy to follow.\n3. The experiments on panoptic segmentation, interactive segmentation, and video segmentation are solid, comprehensive, and persuasive, effectively demonstrating the model's contribution.", "weaknesses": "1. The paper lacks a detailed comparison with other SAM-like methods. A single COCO instance segmentation comparison in Table 4 is insufficient to substantiate claims of superiority over SAM. The results presented in Table 4 are not particularly outstanding. Additional experiments, such as on the SegAny task, with detailed metrics (AP for small, medium, large objects) on COCO instance segmentation, and evaluations with different object detectors, would strengthen the case.\n\n2. Efficiency benchmarks are insufficiently detailed. For a model promoting efficiency, there should be a more comprehensive evaluation across different GPU platforms, such as the 3090 and V100, testing throughput and latency. Additionally, plotting latency versus performance compared to other SAM-like methods would provide a clearer visualization of the model's efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6oYsYcqJZX", "reviewer": "Reviewer_mNKa", "summary": "This work addresses the need for real-time multi-purpose segmentation by introducing a novel setting that encompasses interactive, panoptic, and video instance segmentation, striving for a single end-to-end model capable of handling all tasks in real-time. The proposed Real-Time Multi-Purpose SAM (RMP-SAM) utilizes an efficient encoder and a decoupled adapter for prompt-driven decoding, along with innovative training strategies and adapter designs, demonstrating effectiveness and strong generalization across benchmarks and specific semantic tasks while achieving an optimal balance between accuracy and speed.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Demonstrates impressive performance and inference speed.\n\n2.Filling the gap in real-time multi-purpose segmentation.\n\n3.The whole method is very simple and easy to understand.\n\n4.Code is provided for easy reproduction by the reader.", "weaknesses": "1.Based on existing technology development, the entire pipeline is not novel.\n\n2.Differences with SAMv2 should be further clarified, especially in terms of claimed semantic labels?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5r0EtWaIyt", "reviewer": "Reviewer_NMA5", "summary": "This paper presents a real-time multi-purpose segmentation model called RMP-SAM. RMP-SAM handles various tasks such as interactive segmentation, panoptic segmentation, and video instance segmentation using a single model. To balance the accuracy and speed, RMP-SAM utilizes a lightweight encoder and a dynamic convolution-based decoder. RMP-SAM achieves fast inference while maintaining satisfactory performance.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- RMP-SAM unifies interactive segmentation, panoptic segmentation, and video instance segmentation within a single model.\n- RMP-SAM offers a good trade-off between speed and accuracy.\n- Extensive experiments demonstrate the model's effectiveness.", "weaknesses": "- The authors do not provide detailed information for joint training. Joint training for multiple tasks can be complex. How do the authors train RMP-SAM for some potential problems, such as avoiding the model being dominated by a single task and performance degradation by conflicts between different tasks?\n\n- This paper ignores some related methods, making it difficult to assess the model's performance relative to existing SOTA approaches. For example, some universal methods[1,2,3] obtain better results than RMP-SAM using ResNet50. The authors should make a comprehensive comparison with other methods. \n\n[1] Tube-Link: A flexible cross tube framework for universal video segmentation. CVPR 2023.\n\n[2] Dvis: Decoupled video instance segmentation framework. CVPR 2023.\n\n[3] Univs: Unified and universal video segmentation with prompts as queries. CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TQNnAYlfhn", "reviewer": "Reviewer_HZhq", "summary": "The authors explore a novel real-time segmentation setting called real-time multi-purpose segmentation. It contains three fundamental sub-tasks: interactive segmentation, panoptic segmentation, and video instance segmentation. In contrast to previous methods that use a separate design for each task, the authors use only a single end-to-end model to handle all these tasks in real time. To fulfill the real-time requirements and balance multitask learning, a new dynamic convolution-based method, Real-Time Multi-Purpose SAM (RMP-SAM), is introduced. They benchmark several strong baselines by extending existing work to support multi-purpose segmentation.", "questions": "- What does the dot size in Fig. 1b indicate?\n- The abstract says \"generalization ability of these models across diverse scenarios\", a learnable classifier with CLIP text embeddings is also used and “segment anything” is in the title. Is there a connection to open-vocabulary?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Large models can perform many tasks, but are not real-time capable because of the large encoders, while real-time models are often specialized in only one task. The method presented here aims to combine the two things, i.e., \"the first real-time multi-purpose segmentation model\".\n- Precise implementation details are given and the comparisons with the other methods appear to be fair.\n- The method achieves good results in the trade-off between performance and speed across the various tasks and datasets.\n- The ablation studies are useful and show interesting insights.", "weaknesses": "- Many architectural elements were adopted from other works, it is not clear to me if there are already similar architectures as proposed here, or where exactly is the innovation (except the jointly training).\n- In the related work section, many works are cited and also compared at the task level, but I also miss a comparison at the architectural level.\n- The tables, especially table 3, are difficult to read because nothing is in bold print and you have to search for the trade-off here. A plot like Fig. 1b would be more useful.\n- The references to the appendix could be a little more precise and there is no reference to Table 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)