X-Prompt: Multi-modal Visual Prompt for Video Object Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-27 08:39:55

Abstract: Multi-modal Video Object Segmentation (VOS), including RGB-Thermal, RGB-Depth, and RGB-Event, has garnered attention due to its capability to address challenging scenarios where traditional VOS methods struggle, such as extreme illumination, rapid motion, and background distraction. Existing approaches often involve designing specific additional branches and performing full-parameter fine-tuning for fusion in each task. However, this approach not only duplicates research efforts and hardware costs but also risks model collapse with the limited multi-modal annotated data. In this paper, we propose a universal framework named X-Prompt for all multi-modal video object segmentation tasks, designated as RGB+X. The X-Prompt framework first pre-trains a video object segmentation foundation model using RGB data, and then utilize the additional modality of the prompt to adapt it to downstream multi-modal tasks with limited data. Within the X-Prompt framework, we introduce the Multi-modal Visual Prompter (MVP), which allows prompting foundation model the with various modalities to segment objects precisely. We further propose the Multi-modal Adaptation Expert (MAEs) to adapt the foundation model with pluggable modality-specific knowledge without compromising the generalization capacity. To evaluate the effectiveness of the X-Prompt framework, we conduct extensive experiments on 3 tasks across 4 benchmarks. The proposed universal X-Prompt framework consistently outperforms the full fine-tuning paradigm and achieves state-of-the-art performance. Codes will be available.

Main paper: 
 introduction: 
 Video Object Segmentation (VOS) [4, 5, 21, 22, 54, 57, 76, 90] is a pivotal task designed to delineate target objects throughout a video sequence. This paper primarily delves into semi-supervised VOS, a foundational endeavor that aims to segment a specified object across an entire video sequence, leveraging the object mask provided in the initial frame. This task is instrumental for a myriad of applications, such as video editing [42, 50, 67] , surveillance [1, 43, 52, 89] , robotics [20] , and autonomous driving [33, 83, 86] . Recent studies, particularly those employing matching-based methods [7, 29, 63, 78, 81, 92] even enhanced by memory mechanisms [16, 23, 25, 51, [58] [59] [60] , have made significant progress and achieved advanced results. These methods assess similarities between previous and current frames through semantic matching at the pixel level. However, despite these efforts and advancements, contemporary state-ofthe-art methodologies still struggle in challenging scenarios like extreme illumination, obscured scenes, rapid motion, occlusions, and background distraction.Alternative modalities offer a vital solution to these limitations, by serving as a crucial complement to RGB data in challenging scenarios. For example, thermal [35, 62] and depth [46] information remains unaffected by variations in illumination and visibility conditions. Similarly, event data [18] , which records changes in pixel intensity at high frequencies, does not blur during rapid motion. Consequently, Multi-modal Video Object Segmentation is garnering increased interest for its capacity to facilitate more resilient tracking through the exploitation of inter-modal complementarities, including combinations like RGB-Thermal (RGB-T) [73, 85] , RGB-Depth (RGB-D) [74, 88] , and RGB-Event (RGB-E) [36, 66] . Current multi-modal VOS approaches predominantly augment an existing VOS model's RGB branch with an additional modality branch and a fusion module. These enhancements are then subjected to either full training or fine-tuning on downstream tasks, as depicted in Fig. 1 (a), (b), and (c). However, on one hand, while all these modifications aim to improve VOS task performance in complex scenarios by introducing additional modal information, the branches and fusion mechanisms are tailored separately for each task, necessitating different structures for each downstream modality. This paradigm not only duplicates research efforts but also escalates the computational and hardware costs associated with developing specialized architectures for every task. On the other hand, full-parameters training or fine-tuning these newly designed architectures with limited data can easily lead to overfitting, alongside the risk of catastrophic forgetting, thus lose the generalized capabilities of the original pretrained model. Moreover, the collection of multimodal paired data and pixel-level annotations remains challenging and costly, resulting in a scarcity of such data, with each modality having approximately only a hundred or so annotated sequences. Training with such limited data restricts the scale of the model, making it difficult to effectively train the new modality branch and the fusion module without degrading or collapsing the performance and the generalizability of the pretrained model, thereby failing to leverage prior knowledge.To address these challenges, we argue that these multi-modal tasks of RGB-T, RGB-D, and RGB-E can be unified into a single effort, RGB-X, where X represents any additional modality. And they can share the common segmentation capability and benefit from the robust generalization of a video object segmentation foundation model, which can segment target objects in the current frame based on reference frames and their segmentation masks in various videos. Therefore, we propose the X-Prompt framework, a universal solution for multi-modal video object segmentation. This framework utilizes numerous RGB video sequences to pre-train a transformer-based foundation model with general object segmentation capacity. Subsequently, additional modality information is utilized to prompt and adapt this foundation model to downstream multi-modal tasks with limited data while retaining the model's generalization ability.Within the X-Prompt framework, we further propose two key components: the Multi-modal Visual Prompter and the Multi-modal Adaptation Experts. The Multi-modal Visual Prompter (MVP) encodes cross-modality information into visual prompts that used to prompt the frozen foundation model. The spatial-channel attended complementary prompt are integrated with the patch embeddings of the foundation model to guide segmentation of downstream modality. Additionally, multi-scale multi-modal prompts are obtained to prompt the segmentation mask decoder to incorporate diverse cross-modality information, enhancing the precision of object delineation in this dense prediction tasks. The design described above enables the foundation model to leverage cross-modality prompts to achieve RGB-X VOS tasks, but it does not allow the foundation model to learn new modality-specific knowledge without any tuning. However, full-parameter fine-tuning poses a risk of degrading or collapsing the foundation model, especially with limited data. Therefore, we introduce the Multi-modal Adaptation Experts (MAEs) for plugging modality-specific knowledge without forgetting the generalized prior knowledge embedded in the frozen foundation model. Within each transformer layer of the frozen foundation model, we employ low-rank adaptations to serve as the modality experts. These experts' outputs are combined via a router to facilitate multi-modal collaboration. This approach ensures that the prior knowledge of the foundation model is retained while integrating additional expertise required for new tasks, such as extracting new modality patterns including feature extraction and matching, as well as facilitating collaborative usage of RGB with other modalities. The proposed universal X-Prompt Framework achieves state-of-the-art performance on 3 multi-modal video object segmentation tasks including RGB-T, RGB-D, and RGB-E, surpassing existing methods significantly. In summary, our contributions are as follows:• We propose X-Prompt, the first universal framework for multi-modal video object segmentation, including RGB-T, RGB-D, and RGB-E tasks. It utilizes the X-modality as the visual prompt to adapt a foundation model to various downstream tasks.  
related work 2.1 video object segmentation: 
 Semi-supervised Video Object Segmentation (VOS) accurately segments objects in a video sequence using the provided mask in initial frame. Model-based methods [2, 3, 48, 69] start with offline pre-training on static images to capture segmentation features, then refining them using mask from the first frame of the test video. However, the time-consuming fine-tuning step limits their suitability for real-world applications. Then, Propagation-based methods [10, 28, 31, 50, 53, 65] iteratively propagate masks with temporal correlations. However, these methods heavily rely on previous segmentation masks, making them susceptible to target drift and error accumulation. To address these drawbacks, Matching-based methods [7, 29, 63, 78, 81, 92] calculate correspondence pixel maps between the current and reference frames. Among them, FEELVOS [63] CFBI [78] and CFBI+ [81] improve it by integrating foregroundbackground features and employing local-global matching. More recently, memory-based methods [16, 23, 25, 51, [58] [59] [60] propose an external memory bank to store history features, thus addressing contextual limitations. Techniques such as XMem [8] extend memory to long-term VOS and AOT [77] , DeAOT [82] enhancing performance through Transformer. Despite advancements in existing methods, challenges persist particularly in scenarios with background clutter, illumination variation, and motion blur. This paper tackles these challenges by emphasizing multi-modal VOS, utilizing diverse data sources beyond RGB inputs.In video object tracking, researchers have explored using multiple modalities to improve accuracy. For instance, [44] and [49] use depth maps for handling occlusions. DAL [56] introduces embedded depth-aware convolution into RGB tracking frameworks to bolster target estimation robustness. Besides, [40] combines RGB and event flows to predict Regions of Interest (ROIs) in complex scenarios. To leverage dual modalities, MDNet [66] merge visible frames and event flows onto RGB tracking frameworks to construct a series of baseline trackers. Similarly, JMMAC [84] integrates RGB and thermal infrared modalities to capture reliable appearance and motion cues, with event flows significantly contributing to robust tracking. More recently, to tackle severe data deficiencies, some studies have proposed data-efficient methods. For instance, Pro-Track [75] transfers multi-modal inputs to a single modality, while ViPT [91] designs modality-complementary prompts. As a more challenging task, multi-modal VOS becomes increasingly important as it provides precise pixel-level masks, critical for applications like surveillance in adverse weather and nighttime autonomous driving. VTiNet [73] tackles Visible-Thermal VOS by integrating cross-modal and cross-frame features in modality and temporal domains, alongside a modality-sensitive memory bank. [36] improves low-light VOS with event assistance through the adaptive cross-modal fusion and event-guided memory matching modules. Notely, FusedCDNet [74] proposes a weakly-supervised RGB-D VOS method, utilizing bounding box level supervision in both training and testing phases, mitigating challenges of expensive data collection and annotation. To address the issue of model generalization degradation due to limited data, as well as the redundant research effort and computational deployment costs resulting from designs tailored for each specific modal task, this paper proposes a universal framework for all multi-modal VOS tasks. It utilizes the auxiliary modality as the prompt to adapt a foundational model.The advancement of foundation models has prompted exploration into their adaptation for downstream tasks. Recently, freezing the pre-trained model and only fine-tune a few additional parameters to attain strong performance has emerged as a notable approach, which is particularly valuable when data of downstream task is limited. Among them, prompt tuning-based methods [34] fine-tune models by introducing learnable prompt tokens. For instance, VPT [30] enhances transformer encoders with learnable parameters, outperforming full fine-tuning on 20 downstream recognition tasks. AdaptFormer [6] investigates efficient fine-tuning in video action recognition by integrating lightweight modules into a ViT, surpassing the performance of existing fully fine-tuned models. ProTrack [75] and ViPT [91] utilize prompt tuning in multimodal tracking to address challenging scenarios. Additionally, adapter-based methods [26] incorporate trainable adapters into pre-trained models. For example, LoRA [27] injects trainable low-rank matrices into transformer layers to approximate the weight updates and (𝐼𝐴) 3 [41] scales activations by learned vectors to attain stronger performance with a relatively tiny amount of new parameters. This paper not only uses multi-modal visual prompts to enable the foundation model to handle various multi-modal tasks, but also employs modality adaptation experts to introduce knowledge about new modalities to the foundation model. 
method: 
 In this section, we begin by formulating of multi-modal video object segmentation tasks under a unified formulation termed RGB-X (Sec. 3.1.1). We then introduce the video object segmentation foundational model with generalization capabilities (Sec 3.1.2). Following this, we propose the universal X-Prompt framework, which then leverages the additional modality as the prompt to adapt the foundation model to various multi-modal tasks (Sec. 3.2). Within the X-Prompt framework, we further introduce the Multi-modal Visual Prompter that enables prompting the foundation model to segment objects precisely with various modalities (Sec. 3.3) and the Multi-modal Adaptation Experts to adapt the foundational model with pluggable modality-specific knowledge without forgetting generalized prior knowledge (Sec. 3.4).Multi-modal video object segmentation tasks, such as RGB-Thermal, RGB-Depth, and RGB-Event, can be abstracted into a unified task, termed RGB-X, which involves leveraging an auxiliary modality in addition to RGB to segment target objects in each frame. To formulate, given a video with 𝑇 frames denoted as {𝐼 𝑡 , 𝑋 𝑡 } 𝑇 𝑡 =1 , along with the object masks in the first frame 𝑀 1 ∈ O 𝐻 ×𝑊 that identify 𝑂 targets of interest, the goal is to predict the object masks in all frames, represented as{𝑀 𝑡 ∈ O 𝐻 ×𝑊 } 𝑇 𝑡 =1 . 3.1.2 Foundation Model.The models required for multi-modal video object segmentation tasks all hinge on a core capability: computing the correlation between the current frame and the first frame with the known object mask. Therefore, we argue that RGB-X tasks can share the segmentation capability and the generalization of a foundational model for RGB video object segmentation. This model should be capable of segmenting target objects in the current frames by referencing frames and their segmentation masks when dealing with diverse videos.Specifically, we built our foundation model based on the Vision Transformer [15, 19] within an All-in-One Transformer architecture following [37] . To begin, we generate the non-overlapping patch embeddings with Patch Embedding layer for the reference and current frame P (𝐼 𝑟 ) and P (𝐼 𝑡 ), and concatenate them along the spatial dimension as the initial input of transformer:EQUATION)where P is the Patch Embedding layer. Supposing 𝑝 is the stride of the patch embedding, thus 𝑁 = 𝐻𝑊 /𝑝 2 . Then, this input along with the mask of reference frame are fed into Foundation Model:EQUATIONwhere F is a Visual Transformer with 𝐿 layers And in each layer F (𝑙 ) , 𝑙 ∈ {1, 2, • • • , 𝐿}, the forward process of each layer 𝑧 𝑙 = F (𝑙 ) 𝑧 𝑙 -1 , 𝑚 𝑟 can be written as:EQUATIONwhere LN denotes the Layer Norm and FFN represents the Feed-Forward Network. MSA can use the naive multi-head self attention or the specifically designed uni-hybrid attention in [37] . For integration with mask information, the query, key, and value are:EQUATIONwhere 𝑊 𝑙 𝑞 , 𝑊 𝑙 𝑘 , and 𝑊 𝑙 𝑣 are projection weights for query, key, and value transformations. 𝑚 𝑟 ∈ R 𝑁 𝑟 ×𝐷 is the mask-embedding generated through a mask embedding layer through a single convolutional layer. Finally, the output 𝑧 𝐿 of the foundation model, is input to the lightweight mask decoder D and decoded into final multi-object masks 𝑀 𝑡 = D 𝑧 𝐿 . We have included only one reference frame here for brevity. Actually, multiple reference frames are dynamically updated and stored in memory. We refer readers to [37] for more design details about our video object segmentation foundation model.We train the foundation model with both synthetic video sequences generated from static RGB image datasets [11, 17, 24, 39, 61] and various RGB VOS datasets [14, 55, 70] . This pretraining on largescale datasets equips our video object segmentation foundation model with robust temporal matching capabilities and facilitates effective transferability to downstream multi-modal RGB+X VOS tasks.Multi-modal video object segmentation tasks involve an additional auxiliary X-modality flow that is temporally synchronized and spatially aligned with the RGB flow. We propose the X-Prompt framework, which first trains a robust RGB video object segmentation foundation model with strong segmentation capabilities and generalization. Subsequently, we utilize the auxiliary modality flow as the prompt to adapt the model to downstream multi-modal tasks with limited data. As illustrated in Fig. 2 , in the X-Prompt framework, the RGB images 𝐼 and the X-modality maps 𝑋 are feed into their respective patch embedding layers P 𝑖 and P 𝑥 to obtain patch tokens:EQUATIONwhere 𝑁 𝑡 and 𝑁 𝑟 are the number of patch tokens of the current frame and the reference frames. Then the X-modality tokens are utilized to supplement the image tokens, resulting in complementary combined multi-modal prompts embedding through the Multimodal Visual Prompter (detailed in Sec. 3.3):EQUATIONwhere 𝑧 0 is the spatial-channel attended multi-modal prompts and Φ indicates the proposed Multi-modal Visual Prompter. This visual prompt to the foundation model operates with a stride of 16. Additionally, for the pixel-level dense prediction task, we also design the multi-scale multi-modal prompts to mask decoder to accurately segment object masks. Subsequently, the multi-modal prompt 𝑧 0 is fed to the foundation model, enabling multi-modal video object segmentation without requiring any modifications to the RGB foundation model:EQUATIONwhere 𝑧 𝐿 𝑡 then can be decoded to the object segmentation mask 𝑀 𝑡 with the mask decoder D.Moreover, to adapt the foundation model with modality-specific knowledge in a pluggable manner without compromising its generalization capacity, we introduce Multi-modal Adaptation Experts (detailed in Sec. 3.4) to the frozen foundation model:EQUATION)where ẑ𝑙 is the output of the 𝑙-th block adapted with modality experts and Ψ represents the proposed adaptation experts in Sec. 3.3. It's worth noting that within the X-Prompt framework, only the newly introduced prompter, modality experts, and the patch embedding layer for the X-modality are trainable. All other network parameters, including the transformer encoder layers in the foundation model, the mask decoder, and the patch embedding layer for the RGB image, remain frozen from pretraining.To leverage the powerful segmentation capabilities and generalization of foundation models for multi-modal video object tasks, we propose the Multi-modal Visual Prompter (MVP) to encode RGB-X information into visual prompts. To accommodate dense prediction tasks, we employ convolutional patch embedding layers with progressive downsampling to embed patch tokens. This progress yields multi-scale RGB image patch tokens 𝑧 4× rgb , 𝑧 8× rgb , and 𝑧 16× rgb , as well as X-modality patch tokens 𝑧 4× 𝑥 , 𝑧 8× 𝑥 , and 𝑧 16× 𝑥 ,. We use 𝑧 16×16 rgb and 𝑧 16×to encode primary prompts that enable the foundation model to integrate information from both modalities for object tracking and segmentation. The other patch tokens are efficiently encoded into multi-scale multi-modal prompts to guide the mask decoder for more precise object segmentation masks.For the visual prompt for the foundation model, given 𝑧 16× rgb and 𝑧 16× 𝑥 , we first concatenate them into a single embedding along the specified dimension, followed by a Linear layer that reduces the dimensionality from 2𝐷 to 𝐷 to obtain 𝑧 16× fuse . Subsequently, we obtain spatial attention 𝐴 s ∈ R 𝐻 16 × 𝑊 16 ×1 and channel attention 𝐴 c ∈ R 1×1×𝐷 separately, then merge them into spatial-channel attention through broadcast multiplication:EQUATIONEQUATION)EQUATIONwhere 𝜎 denotes the sigmoid function. Conv and MLP represents the convolution layer and the multi-layer perceptron, respectively. Finally, this spatial-channel attention weight is applied to the image path token embedding to produce a multi-modal prompt that is spatial-channel attended, enhancing and complementing the original tokens: 𝑧 0 = 𝐴 sc × 𝑧 16× fuse . For the visual prompt for the mask decoder, we employ a simple, efficient, yet effective Linear Adapter to encode 𝑧 4× rgb , 𝑧 4× x , and 𝑧 8× rgb , 𝑧 8×x into multi-scale multi-modal prompts {𝑧 4× , 𝑧 8× } for the mask decoder through residual connection. To be more efficient, we find that the linear layer preceding the upsampling operation in each block of the mask decoder can play the role of the adapter, thus eliminating the need for additional parameters for this step.Although the proposed MVP effectively encodes multi-modal prompt embeddings for the frozen foundation model, enabling it to handle various multi-modal tasks without degrading the foundation model's performance, the foundation model itself does not learn any new knowledge about X-modality in this process. As a result, the model's performance on each downstream task is sub-optimal. To infuse new knowledge of this modality into the foundation model using limited data, while avoiding the risks of degrading or collapsing the foundation model that full-parameter fine-tuning might entail, we introduce the Multi-modal Adaptation Experts (MAEs). This approach offers a pluggable knowledge adaptation mechanism that works with the frozen foundation model. Specifically, within each transformer layer of the frozen foundation model, we employ 𝐾 low-rank adaptations to act as modality adaptation experts, assisting linear layers of both MSA and FFN in parallel. Here we denote the input tokens to the linear layer of MSA or FFN within the transformer layer, which also are inputs to the experts, as ℎ ∈ R (𝑁 𝑡 +𝑁 𝑟 ) ×𝐷 , to describe how these modality adaptation experts function:EQUATIONwhere △ℎ 𝑖 ∈ R (𝑁 𝑡 +𝑁 𝑟 ) ×𝐷 is the adaptation of each expert. 𝑊 0 ∈ R 𝐷 ×𝐷 denotes the original parameters of the foundation model. △𝑊 𝑖 ∈ R 𝐷 ×𝐷 represents the learnable parameters for modality adaptation, implemented efficiently by △𝑊 𝑖 = 𝐵 𝑖 𝐴 𝑖 according to the LoRA [27] technique for LLMs. These experts' outputs are combined via a router to facilitate multi-modal collaboration:EQUATIONwhere 𝑤 𝑅 𝑖 ∈ R 𝐷 ×𝐾 is the learnable parameter of the Router and is implemented through a linear layer.This pluggable knowledge adaptation mechanism ensures that general capabilities of the foundation model are retained while integrating additional expertise required for new tasks, such as extracting new modality patterns including feature extraction and matching, as well as facilitating collaborative usage of RGB with X-modality. 
experiment 4.1 implementation details: 
 Networks. In our foundation model, following the OneVOS [37] , we utilize an All-in-One Transformer architecture. The entire network operates within a ConvMAE [19] structure, where patch embedding layers produce patch tokens at 1/4, 1/8, and 1/16 resolutions through progressively downsampled convolutional layers, effectively preserving multi-scale information. The transformer comprises ten transformer encoder layers that concurrently perform feature extraction and object matching, with each dimension set to 768. The mask decoder is structured as an FPN [38] , progressively increasing feature resolution while decreasing the channel dimension. The prompter's convolutional layers utilize a 7 × 7 kernel size, and the MLP layer includes one hidden layer with a dimension that is one-sixteenth of the input dimension. The modality adaptation experts are configured with a low-rank of 8, and both the low-rank adaptation and routing are implemented with linear layers. According to our experiments, we typically employ two experts for each multi-modal task.Training. The Foundation model is initialized with ImageNet1k [13] pretrained weights, and is trained on synthetic video sequences generated from still image datasets [11, 17, 24, 39, 61] for 200,000 iterations with a batch size of 4 and a learning rate of 2e-4. Subsequently, it undergoes main training on collected standard VOS datasets including DAVIS [55] , YouTube [70] , and MOSE [14] for another 200,000 iterations with a learning rate of 1e-4 on four 3090 GPUs. After pre-training, the Foundation model is adapted to multimodal downstream tasks with all pre-trained parameters frozen, while only the newly introduced prompter, adaptation experts, and the patch embedding layer for the X-modality are trainable. Depending on the number of annotated videos available for each task, the training durations vary from 20,000 to 60,000 iterations. Throughout all stages of training, a consistent 0.5:0.5 combination of bootstrapped cross-entropy loss and soft Jaccard loss is used.X-Prompt accomplishes the consolidation of multiple downstream multi-modal video object segmentation tasks. In this study, we select 4 benchmarks [66, 73, 85, 88] of 3 challenging tasks to assess the effectiveness and generalization of the proposed framework. We conduct modality prompt and adaptation on these downstream tasks without specific modulation. Evaluation metrics include J , F , and J &F [55] , where J represents the Intersection over Union score between the prediction and the ground truth mask, F denotes the boundary similarity measure between the prediction boundary and the ground truth, and J &F represents the average score of both J and F . RGB-Thermal Task. To evaluate the performance of the proposed framework on the RGB-Thermal task, we selected the VisT300 [73] and VT-UAV [85] benchmarks. VisT300, a general dataset for RGB-T VOS, includes a variety of scenes, with 250 training sequences and 50 testing sequences. The VT-UAV dataset, designed for the challenging Unmanned Aerial Vehicle (UAV) scenarios, with 50 sequences for training and another 50 for testing. We adapt the training data from these two datasets for the thermal modality task. As Tab. 1 indicates, our proposed X-Prompt outperforms all previous SOTA methods, achieving the highest scores of 84.2% and 87.3% J and F on these VisT300 and VT-UAV respectively, especially on the challenging VT-UAV dataset, where it surpasses others by 10.6%. RGB-Depth Task. For the RGB-D task, we use the ARKitTrack to evaluate the performance. ARKitTrack [88] utilizes LiDAR to collect and annotate 300 RGB-D sequences in both static and dynamic scenes, comprising 245 training videos and 55 testing videos. As shown in Tab. 2, after pre-training the foundation model with RGB data and adapting it for the Depth modality using ARKitTrack, our X-Prompt achieves an impressive J &F score of 82.1%, surpassing previous models whether trained exclusively on RGB-D datasets or pre-trained on RGB data.RGB-Event Task. For the RGB-E task, we use VisEvent [66] dataset, the largest visible-event benchmark dataset collected from realworld scenarios for evaluation. However, the VisEvent dataset only provides object annotations at the box level. Therefore, we employed the powerful HQ-SAM [32] method to generate mask annotations based on these box annotations. Through HQ-SAM predictions and our filtering, we obtained a VisEvent-VOS subset suitable for evaluating RGB-E video object segmentation. Since there were previously no models capable of RGB-E video object segmentation, we compared our approach with current state-of-the-art VOS models and a fine-tuned AOT-L-Swin model with concatenated RGB and event inputs (row-4). Our method significantly outperformed them, as demonstrated in Table 3 .X-Prompt Framework. To verify the effectiveness of the X-Prompt framework, including foundation model, multi-modal visual prompter, and multi-modal adaptation experts, Tab. 4 presents a series of tests: the foundation model, using only RGB information, achieves basic object segmentation (row-1); with the addition of the Multi-modal Visual Prompter, we effectively encode X-modality into the prompt, thereby enhancing the intrinsic capabilities of the foundation model (row-2); the best results are achieved when modality experts are involved to adapt the foundation model for specific modality downstream tasks (row-4). We also independently assessed the contribution of the multi-modal adaptation experts (row-3) by simply concatenating RGB and X-modality data, instead of using the proposed prompter.Multi-modal Visual Prompter. We employed different visual prompts for the foundation model to analyze the roles of various modalities and demonstrate the effectiveness of the proposed multimodal visual prompter. As shown in Tab. 5, RGB contains primary information, and achieving accurate segmentation solely with auxiliary X-modalities is difficult. While thermal modality, in particular, inherently contains relatively more information for object segmentation. The proposed MVP effectively encodes RGB images and Multi-modal Adaptation Experts. First, we implemented various approaches for adapting the foundation model to downstream X-modality tasks, as illustrated in Tab. 6. "Frozen" refers to training only the multi-modal visual prompter (MVP) while keeping the entire foundation model frozen (row-1). Full parameter fine-tuning leads to model degradation and catastrophic forgetting issues (row-2). Adapter and LoRA techniques allow the foundation model to adapt without degradation (row-3), demonstrating the feasibility and effectiveness of the foundation model -prompt -adaptation paradigm (rows 3 and 4). The proposed Multi-modal Adaptation Experts achieve the best performance by introducing additional independent experts during training to learn multi-modal knowledge. Next, as shown in Tab. 7, we experimented with the number of experts employed and found that introducing two experts achieves the best modality adaptation for most RGB-X tasks, except for the RGB-D tasks in the ARKitTrack scenario, where three experts yield better performance. 
conclusion: 
 In this work, we introduce X-Prompt, the first universal framework for multi-modal RGB+X video object segmentation tasks, to address the issue of model generalization degradation due to limited multi-modal data and reduces redundant research effort and computational deployment costs associated with task-specific designs. Following the pre-training of an RGB VOS foundation model with robust segmentation capabilities and generalization, X-Prompt utilizes the X-modality to prompt and adapt the foundation model for various downstream multi-modal tasks, employing our proposed Multi-modal Visual Prompter and Multi-modal Adaptation Experts. Extensive experiments on 3 tasks across 4 benchmarks demonstrate the effectiveness and generalization of X-Prompt.  
 
 Appendix:  


Decision: The authors did a good rebuttal. The reviewers unanimously recommend acceptance in the final rating. After checking the rebuttal, the review, and the paper, the AC agrees with this assessment.

Reviews: review: paper_summary: paper_summary: The paper introduces a universal framework named X-Prompt designed for multi-modal video object segmentation tasks, such as RGB-Thermal, RGB-Depth, and RGB-Event. This framework aims to address challenges faced by traditional video object segmentation (VOS) methods in scenarios with extreme illumination, rapid motion, and background distractions.
strength_weakness: strength_weakness: 1.The paper introduces a novel framework, X-Prompt, which is a universal solution for multi-modal video object segmentation tasks. This approach is designed to handle various challenging scenarios like extreme illumination, rapid motion, and background distraction.
2. X-Prompt leverages a foundation model pre-trained on RGB data and adapts it to multi-modal tasks using additional modalities with limited data, which can be more efficient than training from scratch and helps maintain the model's generalization ability.
3. X-Prompt achieves state-of-the-art performance on multiple benchmarks for RGB-Thermal, RGB-Depth, and RGB-Event tasks, demonstrating its effectiveness.
limitations: limitations: 1. X-Prompt might be more complex compared to traditional VOS methods, which could potentially make it harder to implement and maintain. And there's an inherent risk of overfitting when fine-tuning with limited data, especially in complex real-world scenarios. And the ability of X-Prompt to generalize to completely new and unseen scenarios still require further validation.
2. Give more details about the computational cost, X-Prompt introduces additional modalities, which might lead to higher computational costs compared to simpler models.
3. A thorough check for grammatical,spelling errors and figures is recommended to polish the paper.
paper_summary: paper_summary: This paper introduces X-Prompt, a universal framework for multi-modal video object segmentation (VOS) that supports RGB-Thermal, RGB-Depth, and RGB-Event modalities. X-Prompt pre-trains a foundation model on RGB data and adapts it to multi-modal tasks using the Multi-modal Visual Prompter (MVP) and Multi-modal Adaptation Experts (MAEs) to effectively handle limited data scenarios. Extensive experiments across multiple benchmarks demonstrate that X-Prompt consistently outperforms traditional fine-tuning methods, achieving state-of-the-art performance in multi-modal VOS tasks.
strength_weakness: strength_weakness: 1. The proposed X prompt is novel and has good practical usage. Arbitrary modality fusion in video object segmentation is also very important and less explored in the past. I think the motivation is significant to the community,

2. MVP and MAEs are well designed to tackle the integration and adaptation of the multi-modal data effectively.

3. The proposed approach is verified to be effective among different datasets. Comprehensive ablation studies are done to illustrate the efficay of each designed component.
limitations: limitations: 1. Lack of comparison with arbitrary modality fusion approaches, e.g., [1].
[1] Zhang, J., Liu, R., Shi, H., Yang, K., Reiß, S., Peng, K., ... & Stiefelhagen, R. (2023). Delivering arbitrary-modal semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1136-1147).

2. The RGB-T performance are verified on two datasets while the RGB-D and RGB-E performances are only verified on one dataset. It lacks of justification of the generalizability on  RGB-D and RGB-E tasks.

3. Can the model be adapted into image based object segmentation task? Will it deliver comparable performances?

4. Lack of the analyses of the limitations and the failure cases. The authors are encouraged to enrich these analyses in their main paper.
paper_summary: paper_summary: This paper presents X-Prompt, a new multi-modal video object segmentation (VOS) framework for RGB-Thermal, RGB-Depth, and RGB-Event scenarios. First, X-Prompt is used for traditional VOS tasks by training a foundation model. Immediately after that, the proposed method constructs the Multi-modal Visual Prompter and introduces the Multi-modal Adaptation Experts for adapting the information of each modality. The manuscript shows extensive experiments and ablations. X-Prompt is superior to existing VOS methods.
strength_weakness: strength_weakness: 1. The manuscript is well organized.

2. The motivation for multi-modal VOS is clear and well-illustrated in the manuscript.

3. This work is solid, and the experimental results are promising.
limitations: limitations: 1. This paper uses ConvMAE as the backbone but does not specify what version. Small, Base, or Large? Also, ConvMAE uses pre-training of MAE and then fine-tuning on IN1K. How well would the proposed method work if it used traditional ViT?

2. Section 2.1, titled Video Object Segmentation, only surveys the SVOS methods. The authors should discuss and describe the unsupervised VOS methods or change the title to Semi-supervised VOS.

3. The best performance for the ARKitTrack dataset in Table 7 is an Expert Number of 3, which gives the learnable parameters of 3.2M instead of 2.3M in Table 6.

4. The foundation model's video sequence in Figure 3 of the supplementary material is incorrect.

5. Typos.
1) In L26, ‘Multi-modal Adaptation Expert’ should be ‘Multi-modal Adaptation Experts’.
2) In L228, ‘And’ should be ‘and’.
3) In L635, ‘The Foundation model’ should be ‘The foundation model’.
paper_summary: paper_summary: The paper proposes a unified framework (X-Prompt) for semi-supervised multi-modal video object segmentation tasks, including RGB-Thermal, RGB-Depth and RGB-Event. Meanwhile, it proposes the Multi-modal Visual Prompter (MVP) for early multi-modal fusion and introduces the Multi-modal Adaptation Expert (MAEs) for parameter-efficient transfer learning. It conducts extensive experiments on 3 tasks across 4 benchmarks and achieves impressive performance.
strength_weakness: strength_weakness: - Novel paradigm: pre-training for single modal (RGB) and parameter-efficient transfer learning for multi-modal (RGB+X).
- Extensive experiments on 3 tasks across 4 benchmarks and impressive performance on all the benchmarks.
- Comprehensive and valuable ablation experiments.
limitations: limitations: **METHOD**
- The novelty of MVP is limited, just following general multi-modal fusion strategies. And why only enhance RGB feature by Spatial-Channel
Attention Weights instead of both RGB feature and X-modal feature?
- Confusing symbols. (1) What are the $z_t^0$ and $z_r^0$ in Eq.(2) meaning?  (2) $m_r$ appears at Eq.(3), but is defined at Line-390. I don't know the differences between $m_r$ and $M_r$ until I read the definition in Line-390. (3) Does the $z_t^L = F(z_t^0, z_r^0, M_r)$ mean the feature of target frame from L-th layer? But it seems that it outputs features of both reference frame and target frame for each layer ($z^l = F^{(l)}(z^{l-1}, m_r)$).
- The detailed architecture of Downsample Patch Embedding is several cascaded convolution layers? Can Mask Decoder really benefit from features extracted by shallow convolutional layers?


**Ablation Study:**

- In Table 4, row-2 has 0M learnable parameters, since the results is same as row-1 in Table.6. Thus, I guess the row-1 in Table 4 also has 0M learnable parameters. Meanwhile, the row-1 and row-2 in Table 5 don’t have learnable parameters.

Table 5 row-1: Frozen Foundation Model + RGB

Table 5 row-2: Frozen Foundation Model + X
Table 5 row-3: Frozen Foundation Model + MAEs + Concat (RGB & X) (same as row-3 in Table 4)

Table 5 row-4: Frozen Foundation Model + MAEs + MVP(RGB & X)

In Table 5, row-1 and row-2 cannot be fairly compared to the results from row-3 and row-4. I would like to know the results of:

Frozen Foundation Model + MAEs + RGB

Frozen Foundation Model + MAEs + X

**Minor Errors:**

- Page 5, Line 518: extra “,”  and wrong symbol $z_{rgb}^{16 \times 16}$;

- For Figure 2, I don't understand how to use the reference mask;

- For Figure 3, the lines extending down from the RGB and X encoders are hard to understand.