Title: MixPrompt: Efficient Mixed Prompting for Multimodal Semantic Segmentation
Venue: NeurIPS 2025
Accepted: yes
Keywords: semantic segmentation

multimodal fusion

prompt tuning

low-rank space
Abstract: Recent advances in multimodal semantic segmentation show that incorporating auxiliary inputs—such as depth or thermal images—can significantly improve performance over single-modality (RGB-only) approaches. However, most existing solutions rely on parallel backbone networks and complex fusion modules, greatly increasing model size and computational demands. Inspired by prompt tuning in large language models, we introduce \textbf{MixPrompt}: a prompting-based framework that integrates auxiliary modalities into a pretrained RGB segmentation model without modifying its architecture. MixPrompt uses a lightweight prompting module to extract and fuse information from auxiliary inputs into the main RGB backbone. This module is initialized using the early layers of a pretrained RGB feature extractor, ensuring a strong starting point. At each backbone layer, MixPrompt aligns RGB and auxiliary features in multiple low-rank subspaces, maximizing information use with minimal parameter overhead. An information mixing scheme enables cross-subspace interaction for further performance gains. During training, only the prompting module and segmentation head are updated, keeping the RGB backbone frozen for parameter efficiency. Experiments across NYU Depth V2, SUN-RGBD, MFNet, and DELIVER datasets show that MixPrompt achieves improvements of 4.3, 1.1, 0.4, and 1.1 mIoU, respectively, over two-branch baselines, while using nearly half the parameters. MixPrompt also outperforms recent prompting-based methods under similar compute budgets.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a framework to incorporate auxiliary modalities, such as depth or lidar, into a pretrained RGB segmentation model to enhance segmentation quality. To achieve better results, it introduces several techniques: initializing the integration module with early layers of a pretrained CNN backbone, and aligning features across multiple low-rank subspaces. Overall, the approach achieves SOTA results on several benchmark datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The paper provides well done rationalization for every modification of proposed module, supported by thorough ablation studies.\n2. The MixPrompt achieves good performance on proposed benchmark datasets.\n3. The authros evaluate the framework across many modalities (e.g. depth, lidar, thermal, event), showing the generalizability of it over modalities.\n4. The paper is well-wrtitten and easy to follow. \n\n**Weaknesses:**\n1. Incremental novelty of the proposed framework and initialization strategy for auxiliary modalities contributions.\n\nQUESTIONS:\n1. **Concerns about Paper’s Novelty**\n\n1.1\nThe concept of a framework that extracts features from depth and then integrates them using a prompt adapter has already been explored in prior work [1].\n\n1.2\nSimilar initialization techniques have been employed in previous studies to obtain depth features(e.g. making depth \"colorful\" then using frozen RGB backbone early layers [2]). While the paper conducts more proper research, its novelty appears to be incremental. Could you clarify what is fundamentally new in your method?\n\n2. **Missing Comparison**\n\nWhy is there no comparison with OmniVec 2 [3]? Please explain the rationale behind excluding this relevant work.\n\n3. **Parameter Count in Table 1**\n\nConsider adding a column in Table 1 that indicates the number of trainable parameters, especially since your method freezes the backbone and only trains a specific module to demonstrate your efficiency (I saw it later but it will be still valuable in the main table).\n\n4. **RGB-Only Baselines in Table 1**\n\nPlease consider including results from RGB-only models in Table 1 to highlight the performance gain of RGB-D fusion. While some modality comparisons are shown later, RGB-D results are missing.\n\n5. **Dataset Selection Justification**\n\nThe evaluation focuses on NYU Depth V2 and SUN RGB-D, but widely used datasets such as KITTI and ScanNet are not included. Please explain the rationale behind excluding these datasets from evaluation.\n\n**References**\n\n[1] Dong, Shaohua, Yunhe Feng, Qing Yang, Yan Huang, Dongfang Liu, and Heng Fan. \"Efficient multimodal semantic segmentation via dual-prompt learning.\" In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 14196-14203. IEEE, 2024.\n\n[2] Gopalapillai, Radhakrishnan, Deepa Gupta, Mohammed Zakariah, and Yousef Ajami Alotaibi. 2021. \"Convolution-Based Encoding of Depth Images for Transfer Learning in RGB-D Scene Classification\" Sensors 21, no. 23: 7950.\n\n[3] S. Srivastava and G. Sharma, \"OmniVec2 - A Novel Transformer Based Network for Large Scale Multimodal and Multitask Learning,\" 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2024, pp. 27402-27414\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a MixPrompt framework to multi-modal semantic segmentation task, which adopts prompt tuning to integrate information from other modalities into the processing of RGB. To further improve the performance and efficiency, the authors propose Multi-subspace prompting method to project RGB features into multiple subspaces and mix them with the prompt features from the auxiliary branch. The authors conduct extensive experiments on NYU Depth V2, SUN-RGBD, MFNet, and DELIVER datasets to validate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed framework seems reasonable. The prompt learning method based on the pre-trained model has indeed greatly improved the performance and training efficiency on many tasks.\n2. The experimental results are good on several benchmarks.\n\nWeaknesses:\n1. The motivation for proposing Multi-subspace prompting is unclear. The authors did not use theoretical derivation or experimental observation to derive the reason why RGB or other modality features need to be projected to multiple subspaces, which requires a deeper analysis.\n2. Regarding the method, the method proposed in this paper is more like an incremental improvement of the prompt tuning method proposed by DPLNet, which constructs a multi subspace mapping layer based on DPLNet. Therefore, the contribution is somewhat insufficient.\n3. For the experiments, the method in this paper is based on the backbone pre-trained on ADE20K, which is relatively unfair to compare with the previous method pre-trained on ImageNet, like DFormer. I understand that different method paradigms have different starting points, but the author should clearly indicate it in the comparison table, which is more friendly to other readers.\n\nQUESTIONS:\n1. Providing more visual observations of the multi-subspace distributions would better illustrate the effect of the proposed method.\n2. It would be better to clearly display and distinguish different experimental settings in the table.\n3. Can the authors demonstrate more experiments based on the ImageNet pre-trained backbone to verify the benefits of pre-training on ADE20K, even unfreeze all parameters?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MixPrompt, a prompting-based framework for multimodal semantic segmentation that integrates auxiliary modalities into a pretrained RGB segmentation model. The method introduces a lightweight prompting module and a multi-subspace alignment strategy to effectively adapt the model to multimodal inputs without retraining the entire backbone. The proposed approach is evaluated on multiple benchmark datasets, demonstrating competitive performance in multimodal segmentation tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper proposes a simple yet effective prompting-based approach that can be efficiently integrated into existing pretrained RGB segmentation models.\n- The method demonstrates strong performance across multiple benchmark datasets for multimodal semantic segmentation, validating its general applicability.\n- The framework offers a computationally efficient solution that avoids retraining large backbones, which is especially appealing for real-world deployment scenarios.\n\n**Weaknesses**\n- The core concept of using a pretrained RGB model while injecting auxiliary modalities through prompts was already adopted in prior work such as [14]. As a result, the originality of MixPrompt is limited to the specific design of its prompting module.\n- Multimodal semantic segmentation, particularly in RGB-Thermal or RGB-Depth settings, is a well-established field with strong existing baselines. The paper does not clearly differentiate MixPrompt from prior methods in terms of either methodology or practical impact.\n- The paper does not sufficiently clarify why the proposed method works. For instance, the effectiveness of multi-subspace prompting is attributed to improved information exploitation through multiple down- and up-projection pairs, but this claim lacks theoretical justification.\n- In the ablation study, key design choices, such as setting the rank downscale ratio and the number of subspaces for prompt mixing to four, are not well motivated. It remains unclear whether these hyperparameters are dataset-specific or reflect more general design principles.\n- In several experiments, the performance improvements over state-of-the-art methods are relatively modest (e.g., 0.5–1.0% mIoU), raising concerns about the practical significance of the proposed approach. Additionally, the qualitative results presented in Figure 5 do not clearly demonstrate a noticeable visual improvement, making it difficult to assess the perceptual advantage of MixPrompt.\n\nQUESTIONS:\n- Given that multimodal semantic segmentation (e.g., RGB-Thermal, RGB-Depth) is a mature area, what do you see as the practical advantage of MixPrompt over prior methods? Does it offer clear benefits in efficiency, robustness, or generalizability that would motivate its adoption?\n- The paper claims that multi-subspace prompting enhances information utilization. Can you provide empirical or theoretical support for this claim? \n- You performed ablation studies on a single dataset to select the rank downscale ratio and the number of subspaces for prompt mixing. Can you clarify whether the selected values (e.g., 4) generalize well across other datasets, or whether they require dataset-specific tuning? This would help assess the robustness and usability of your method in broader settings.\n- The reported performance gains are relatively small (e.g., 0.5–1.0% mIoU in some cases), and the visual results in Figure 5 do not clearly indicate a qualitative advantage. Could you provide further evidence to justify the practical impact of MixPrompt?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces MixPrompt, an efficient method for integrating auxiliary modalities with RGB images for semantic segmentation through prompt tuning. MixPrompt utilizes a frozen backbone that extracts features exclusively from the RGB input. At each stage of the backbone, a lightweight prompter module is employed to effectively fuse RGB features with those from the auxiliary modality.\nA key insight of the paper lies in the initialization of the prompt derived from the auxiliary modality. This initial prompt is merged with the RGB features *before* the first backbone block. To ensure a consistent alignment, the prompt is generated by processing the auxiliary input through the early layers of a pretrained RGB model.\nThe proposed method is evaluated on four datasets: NYU Depth V2, SUN-RGBD, MFNet, and DELIVER. MixPrompt consistently outperforms prior approaches while maintaining a significantly smaller parameter count.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The paper introduces a **novel and efficient fusion strategy** for combining RGB images with auxiliary modalities (e.g., depth, thermal, event, LiDAR), requiring only a minimal number of learnable parameters.\n\n* The design of the **initial prompt generation** mechanism offers an elegant and principled approach to the early fusion of modalities, ensuring meaningful feature alignment from the start of the network.\n\n* The proposed method achieves **competitive or superior performance** on multiple benchmarks while maintaining a very small parameter footprint.\n\n* The paper is **well written, clear, and easy to follow**, which facilitates understanding of the core ideas and implementation details.\n\n---\n\n**Weaknesses**\n\n* **Related Works Section.** While the related work is included in the supplementary material, placing it outside the main paper is unfair in comparison to other submissions that incorporate it within the main text, often sacrificing space for methodology and experiments. I recommend including at least a condensed version of the related work section in the main paper to ensure a fair and comprehensive presentation.\n\n* **Merging Strategy in the Mixed Prompting Module.** The two prompts ($h\\_{i-1}$ and $e\\_{i-1}$) are currently merged via simple summation. Have alternative merging strategies, such as concatenation, learnable weighted summation, attention-based fusion, or other parameter-efficient mechanisms, been explored? Investigating or at least discussing these alternatives could strengthen the contribution and potential generalizability of the proposed module.\n\nQUESTIONS:\nSee the questions in the Strengths And Weaknesses section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}