MixPrompt: Efficient Mixed Prompting for Multimodal Semantic Segmentation
Authors: Zhiwei Hao
Zhongyu Xiao
Jianyuan Guo
Li Shen
Yong Luo
Han Hu
Dan Zeng
Venue: NeurIPS 2025 poster ()
Keywords: semantic segmentation
multimodal fusion
prompt tuning
low-rank space
Primary area: deep_learning
TL;DR: We present a prompt based multimodal semantic segmentation on the basis of pertained single-modality RGB model

Abstract: Recent advances in multimodal semantic segmentation show that incorporating auxiliary inputs—such as depth or thermal images—can significantly improve performance over single-modality (RGB-only) approaches. However, most existing solutions rely on parallel backbone networks and complex fusion modules, greatly increasing model size and computational demands. Inspired by prompt tuning in large language models, we introduce \textbf{MixPrompt}: a prompting-based framework that integrates auxiliary modalities into a pretrained RGB segmentation model without modifying its architecture. MixPrompt uses a lightweight prompting module to extract and fuse information from auxiliary inputs into the main RGB backbone. This module is initialized using the early layers of a pretrained RGB feature extractor, ensuring a strong starting point. At each backbone layer, MixPrompt aligns RGB and auxiliary features in multiple low-rank subspaces, maximizing information use with minimal parameter overhead. An information mixing scheme enables cross-subspace interaction for further performance gains. During training, only the prompting module and segmentation head are updated, keeping the RGB backbone frozen for parameter efficiency. Experiments across NYU Depth V2, SUN-RGBD, MFNet, and DELIVER datasets show that MixPrompt achieves improvements of 4.3, 1.1, 0.4, and 1.1 mIoU, respectively, over two-branch baselines, while using nearly half the parameters. MixPrompt also outperforms recent prompting-based methods under similar compute budgets.

MixPrompt: Efficient Mixed Prompting for Multimodal Semantic Segmentation
Zhiwei Hao'; Zhongyu Xiao Jianyuan Guo? Shen", Yong Luo", Han Hu Dan Zeng
School of information and Electronics, Beijing Institute of Technology.
2 Department of Computer Science, City University Hong Kong. Schooo of Cyber Science and Technology, Sun Yat-sen University. "School of Computer Science, Wuhan University. i "School of Communication and Information Engineering, Shanghai University. (haozhw, zhongyu.xian hhu] edu. en, jianyguo@cityu. edu hk, mathshenl @gmaill com, luoyongōshu. boyyon@wwhu.edu.cr edu. en, dzengūshu edu. cn
Abstract
Recent advances in multimodal semantic segmentation show that incorporating auxiliary inputs-such depth or thermal images-can significantly improve performance over single-modality (RGB-only) approaches. However, most exist- ing solutions rely on parallel backbone networks and complex fusion modules, greatly increasing model size and computational demands. Inspired by prompt tuning large language models, we introduce MixPrompt: a prompting- based framework that integrates auxiliary modalities into pretrained RGB segmentation model without modifying ing module to extract and fuse information from auxiliary inputs into the main RGB backbone. This module is initialized using the carly layers of a pretrained RGB feature extractor, ensuring strong starting point. At each backbone layer, MixPrompt aligns RGB and auxiliary features in multiple low-rank subspaces, maximizing information use with minimal parameter overhead. An information mixing scheme enables cross-subspace interaction for further performance gains. During training, only the prompting module and segmentation head are updated, keeping the RGB backbone frozen for parameter efficiency. Experiments across NYU Depth V2, SUN-RGBD, MFNet, and DELIVER datasets show that Mix- Prompt achieves improvements of 4.3, 1.1, 0.4, and 1.1 mloU, respectively, over two-branch baselines, while using nearly half the parameters. MixPrompt also outperforms recent prompting-bsse methods under similar compute budgets. The code is available at https//ggghub..co//iaahhhttiiet
architecture. MixPrompt uses a lightweight prompt-
Introduction
Semantic segmentation assigns a label to each pixel in an image and is core task in computer vision. Progress in this area has largely been driven by large-scale RGB datasets such as Cityscapes [ and ADE20K 2 However, models trained on RGB data alone often struggle in challenging environments, such as low-light or poor weather, where visual cues are weak or missing. To address these limitations, multimodal segmentation approaches integrate data from additional sensors
[ 6
10
12 13
For example, RGB-D segmentation pairs RGB images with depth information, improving
'These authors contributed equally this work. Cooresppondig to Han Hu and Jianyuan Guo.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

RGB Backleene
Head
nnemen
XOR Ixput
modillo
Input
Prompier
Aus. Inpui
Aus. Inpuz
Thiiable
Two-branch methods
(b) Existing prompting method
MixPrompt method
Figure 1: Comparison between (a) conventional multimodal semantic segmentation framework, (b) prompting-bar -bseed framework, and (c) our mixed prompting framework. Our framework excels by offering superior parameter efficiency while providing a more effective prompting module design.
9 As result, multimodal methods have surpassed RGB-only models in both accuracy and robustness.
object separation and spatial understanding 3
5
Despite these benefits, main challenges remain for multimodal segmentation. First, model size increases significantly, adding more modalities usually means extra backbone branches and fusion modules [14], which raise computational costs and make deployment difficult on eesuuree-lmiite platforms. Second. multimodal datasets are often much smaller than RGB ones, as collecting aligmed data from auxiliary sensors is expensive and labor-intensive. For instance, NYU Depth V2 [T5] contains only 1,449 RGB-D pairs, while ADE20K has over 25,000 RGB images. This data scarcity limits both training and generalization.
Recognizing these challenges, we draw inspiration from the vision-language domain [16 18]. where prompt tuning allows pretrained language model process visual inputs by introducing visual features as prompts. This method efficiently adapts large models to new tasks or modalities with minimal data and parameters. We extend this idea semantic segmentation by using pretrained RGB model as the base, and introducing auxiliary modalities as prompts. Prior work sharing the similar idea [14] simply embeds data from the additional modality directly into the feature space and integrates it with the RGB features through simple addition, treating the modalities as basic addends rather than leveraging their complementary strengths. This method may not fully exploit the potential of the auxiliary modality, potentially leading to suboptimal performance.
To bridge this gap, we propose MixPrompt, mixed prompting framework for efficient multimodal semantic segmentation. MixPrompt embeds auxiliary data using the initial layers of pretrained RGB extractor, then aligns and fuses features from both modalities multiple subspaces throughout the backbone. This approach maximizes information transfer with minimal parameter increase. As shown Figure [] MixPrompt achieves the efficiency of prompting-basd based methods while delivering stronger performance. We evaluate MixPrompt on four benchmark datasets: NYU Depth V2 [15]. SUN-RGBD [9), MFNet [IO], and DELIVER { MixPrompt consistently outperforms strong two-branch and prompting based baselines, achieving higher accuracy with fewer parameters.
The contributions of paper be summarized follows:
We introduce MixPrompt, an efficient prompting framework that integrates auxiliary modali- ties into pretrained RGB segmentation model, ensuring both data and parameter efficiency. We show that reusing early layers of a pretrained RGB backbone is an effective embedding strategy for auxiliary modalities.
We propose a nulti-subspace alignment and prompting strategy that fully leverages auxiliary information, resulting in improved performance over existing baselines.
2
Related works
2.1 Semantic segmentation
Semantic segmentation faces limitations in complex scenarios when relying solely on RGB images, such as low-light conditions and occluded objects. To address these challenges, researchers have introduced multimodal inputs like depth and thermal images. Existing approaches primarily focus on two directions: cross-modal alignment and fusion strategies at various network levels [ 6 20 g
21
22
10
and developing specialized feature extraction architectures multimodal However, these methods often suffer from increased model complexity due to
data 24 5
25 26

---

soddality-speciff networks and are constrained by the scarcity of large-scale multimodal datasets for pretraining.
2.2 Multimodal prompting
Multimodal prompting has emerged as an effective technique to enhance cross-modal reasoning in vision-language models. Visual prompts, complementing textual prompts, enable pixel-level instructions that help mitigate challenges like visual hallucinations and linguistic biases. Models like CLIP [27] and LLa VA [T6] demonstrate how visual prompts combined with large language models can achieve strong performance in multimodal tasks. However, early applications in semantic segmentation, such Dong et al. [1]. often fuse multimodal features through simple operations like summation without adequately addressing inter-modal differences, which may interfere with RGB feature distributions and limit visual information capture. The complete literature review, including extensive analysis of prior research and additional references, can be found in the supplementary material (Section A.I.
Rethinking Multimodal Prompted Segmentation
Problem setup. Prompt tuning is technique originally designed to improve the performance of pretrained language model on a target task without modifying its internal architecture. It involves providing the model with spec-se context through properly constructed prompts. In the field of computer vision, similar ideas have emerged to adapt pretrained models to downstream tasks 22 general workflow for this procedure be formulated as follows:
= P((hs-1,6t-1,, h; = L,(h:-1.e;),
head (hx),
where E (1, is the layer index. P. and L, denote the i-th prompting module and model layer, respectively. h, represents the output hidden states of layer L,, and hu is the embedded original input. e denotes the prompt for layer Li, while en is the initial prompt at the input stage, which can be either learnable task-specific parameters. Under this framework, multimodal prompted segmentation can be achieved by using a pretrained single-modality model to extract hidden states from an RGB image, where the prompt e; is incorporated into the input space each layer Li. The initial prompt en is obtained based on the auxiliary modality input, and the intermediate prompt e; is generated by the prompting module P.. which takes both the previous hidden state h;-1 and prompt e--1 as inputs. Finally, the result of the segmentation y is obtained by processing the hidden state N using the prediction head.
N
RGB
Depth
Figure 2: Feature maps extracted from the first stage pretrained RGB ResNet50 for RGB and depth inputs, showing that RGB pretrained model can extract meaningful features from depth images.
Rethinking prompt initialization. In the multimodal prompted segmentation framework described above, the way in which auxiliary modality information is introduced into pretrained model significantly impacts segmentation performance, specifically the initial prompt ep, as serves as the entry point for incorporating the auxiliary modality into the model. We reconsider the typical approach to prompt initialization, which often relies on training a new feature extractor specific the auxiliary modality using randomly initialized prompts. This random initialization may negatively
3

---

affect the final performance of the prompted model. In contrast, we hypothesize that early layers of a pretrained RGB model could provide a more effective initialization.
To validate this, we conducted experiments to explore the viability of using pretrained RGB models for initializing the prompt en- Specifically, analyze the feature maps obtained from the first stage of pretrained ResNet50 model, originally trained on RGB data, using both RGB and depth modality inputs. For this analysis, we use the first sample from the test of the NYU Depth V2 dataset, which consists of an RGB-depth image pair.
We begin by comparing the feature maps obtained from the model when fed with both RGB and depth inputs. Figure 2 presents a subset of the extracted feature maps for both modalities. These maps show that both RGB and depth inputs exhibit distinet but rich structural patterns, with each modality focusing on different aspects of the scene. For instance, the feature maps from the RGB input capture fine edge and texture details, while the depth modality emphasizes depth-related information, such as surface contours and object shapes. Despite the differences in data characteristics, the pretrained RGB model is still capable of extracting meaningful features from the depth modality.
4 Method
4.1
MixPrompt framework
We introduce the MixPrompt framework, a method designed to integrate auxiliary modality infor- mation into a pretrained RGB model through efficient and lightweight prompt tuning. As illustrated in Figure 3 our framework utilizes a pretrained RGB model as the backbone for processing RGB images. The auxiliary modality is incorporated via lightweight prompting module that generates the initial prompt en based on features extracted from the auxiliary input. At each layer i, the backbone layer L; and the prompting module P. iteratively fuse and refine features from the hidden state h4-1 and the prompt 6;-1. The final output is then passed through a segmentation foud--nitialized with a pretrained RGB segmentation model-to to generate the segmentation mask. During training, the backbone remains frozen, as it accounts for the majority of the model parameters. Therefore, only small subset of parameters trainable in the MixPrompt framework, ensuring efficient optimization.
The design of prompt initialization stems from our rethinking of how the initial prompt co should be generated. Instead of training an additional modality-specifie feature extractor, we propose leveraging the early layers of the pretrained RGB model. These layers are effective for initializing eo. as they capture general, low-level features that can transfer across different modalities. In the following sections, we will detail the prompting process after obtaining the initial prompt Eq.
4.2 Multi-subspace prompting
We delve into the design of the prompting module in this subsection. In our MixPrompt framework. the prompting module serves as the core, fusing useful information from the auxiliary modality into the backbone. This integration ensures that the final prediction takes the auxiliary modality into account, leading to superior results. However, the features of the main modality in the backbone and those of the auxiliary modality in the prompting branch exist in different feature spaces. Consequently, the prompting module must first align these mismatched feature spaces. Once aligned, the features are fused and fed back into the backbone. This alignment and fusion process relies on two key design principles of our prompting module. The first principle is parameter-efffcieet feature alignment, which ensures that the prompting branch remains lightweight. The second principle is informatiioneexploiaaiionn efficiency which ensures that we fully utilize the auxiliary information available. The designed prompting module, based on these two principles, is presented at the bottom right of Figure 3
To achieve parameter-efficien feature alignment, is essential that the alignment module remains uncomplicated. Therefore, we employ a straightforward linear projection to align the features between the RGB and auxiliary modalities within a low-rank subspace. This approach not only simplifies the alignment process but also enhances computational efficiency by reducing the complexity of the module. Furthermore, the fusion of these aligned features is implemented by simply adding them together, which further promotes efficient computation. This process is formulated as follows:
e, = P((h
( (Wishh + Wse-1),
(2)

---

Backbone
Patch Embedding
Block 1
Block 2
Block N
Head
RGB Input
Feature Extractor
Prompter 1
2rompter2
Prompter N
A ux. Imput
Prediction
Mixed Prompter
RGB Pretzinee Model
Ci-1 Down
6i-1
Down
Down
Frozen module
Mixing Layer
Trainable module
point- wrint-wise add
Down
Up
Prompt Initialization
Mixed Prompting Module
Concateration
Figure 3: Overall framework of the proposed MixPrompt for multimodal semantic segmentation. The RGB input is processed by an RGB- BGB-prreraiied segmentation model, while the auxiliary input is processed by lightweight prompting module. In the prompting module, the initial prompt is derived from the early blocks of the pretrained RGB model (Section 3 At each stage, auxiliary features are projected into multiple subspaces and subsequently mixed align with the RGB features by mixed prompting modules (Section 4.2). The fused features are used as prompt and is fed back into the main branch. Finally, the segmentation mask is generated by prediction head based on the output features of the main branch. During training, only the lightweight prompting module and the prediction head are trainable, ensuring the parameter efficiency of the proposed framework.
where W W. E Rr are down projections and Wap up original feature space, and r < is the dimension of the low-rank alignment subspace. By adopting small value for r, the prompting module can be made parameter-efficeent
Rdxr is up projection. Here, d is the
To enhance information-epplotattio efficiency, we propose introducing multiple pairs of down and up projections. Each pair aligns the RGB and auxiliary modalities within a distinct subspace, thereby boosting the utilization of useful information. However, this approach increases the number of parameters significantly. Drawing inspiration from the LoRA finetuning method for LLMs 30, we observe that employing an extremely low rank can still yield promising performance. This because the pretrained LLM operates within low intrinsic dimension. We make a similar assumption: our pretrained RGB segmentation model also resides in a low intrinsic dimension. Consequently, we reduce the rank of each subspace. This reduction ensures that the overall number of parameters remains unchanged, thereby maintaining parameter efficiency. Taking the RGB features as an example, the projection into multiple subspaces can be represented as:
( wgg 1h4 Wigh,sh .... Wigh.nh--1]
where 1 2i.., W y E =x division of the original rank r by ensures that the introduction of multiple subspaces does not bring additional parameters. To keep concise, we denote W hi- as hi Then, to further improve the representation ability, we introduce a mixing matrix M Boxn to exchange information between each subspace. The mixed feature h;-1 becomes:
xd__ n indicates the number adopted subspaces, where the
h - 13 hr
M Σ E M,, ............. Ej Mi.nh(i.)
where Mr, indicates value of the according element in the mixing matrix M. Similarly, we obtain the low-rank representation of the prompt 6 as é; The information from the RGB modality is then injected into the low-rank prompt by addition and back-projected into the original feature space via the linear projection Wip E d to yield the new prompt e, Once the prompt e; is obtained, the backbone layer L, directly fuses the RGB feature with the prompt via addition. The whole process can be formulated as:
(5) The mixed prompting module performs information mixing only between different subspaces of the RGB modality, based on the rationale that mixing information across subspaces on both modalities
h4 = L, + W (h. + ê;-1)).
5

---

Table 1: Comparison various multi-modes fusion methods on RGB-D segmentation datasets. Results are obtained through multiscale testing. '-' indicates that the corresponding results are not provided in the original paper. denotes the result from the the single-modal version of the method using only RGB input.
NYU Depth
SUN-RGBD
Model
Backbone
Params
Input size
FLOPs
mloU
Input size
FLOPs
mloU
SegFormer
MiT-B4
62.4M
440x640
74.7G
52.3
530x
96.3G
49.1
ACNet 3
ResNet50
116.6M
440
126.7G
48.3
530x730
163.9G
48.1
SGNet 4
ResNet101
64.7M
440
108.5G
51.1
530x730
151.5G
48.6
SA-Gate
ResNet101
110.9M
440
193.7G
52.4
530x730
250.1G
49.4
GEN 6
ResNet101
118.2M
480x640
118.2G
51.7
530x
790.3G
50.2
ShapeConv
241
ResNext-101
86.8M
640
124.6G
51.3
530x730
161.8G
48.6
ESANet 02
ResNet34
31.2M
480x
31.2G
50.3
480x640
34.9G
48.2
TokenFusion
21
MiT-B3
45.9M
640
94.4G
54.2
TransID-Fusion
20
Swin-B
84.0M
x40
55.5
530x
51.9
Omnivore
Swin-B
95.7M
x40
95.7G
54.0
CMX 7
MíT-B2
66.6M
640
67.6G
54.4
530x730
86.3G
49.7
CMX 7
MiT-B4
139.9M
640
134.3G
56.3
530x
173,8G
52.1
CMX 7
MiT-B5
181.1M
480x640
167.8G
56.9
530x730
217.6G
52.4
CMNext
MiT-B4
119.6M
440
131.9G
56.9
DFormer 26
DFomer-L
39.0M
440
65.7G
57.2
530x
83.3G
52.5
DPLNet
14
MiT-B5
88.58M
480
105.0G
59.3
530x730
132.9G
52.8
Ours
MiT-B5
87.2M
640
109.0G
61.2
530x730
137.9G
53.5
could hinder alignment and lead suboptimal performance. The overall workflow of is presented in the supplementary material (Algorithm D
Analysis. The proposed prompting method adopts multiple subspaces for feature fusion, which acts somewhat like the adaptation of mixture of experts in LLM design [31 32). where matrix M is the router assign different importance to each subspace by mixing them with different weights. When or when all elements in M are equal to 1/n, the prompting module downgrades to the base architecture in Equation 2 More quantitative analysis in the supplementary material (Section C.5) further validates this design, showing that moderate numbers of subspaces yield optimal performance by enhancing inter-subspace diversity. Another impact of the mixed prompting module comes from reparameterization B3 While the introduction of multiple subspaces does not involve any non-linear bialti-bannch architecture. This modification of the
operations, it allows the model to leverage gradient flow helps the model achieve better performance by facilitating more effective information propagation. By aligning features in low-rank subspace, we adhere to the design principle of parameter-efficien feature alignment. Additionally, the introduction of multi-subspaac prompting and the information mixing scheme ensures the principle of -xxplittteeeliiiaation efficiency, enabling the final model to achieve effective and efficient multimodal fusion.
Experiments
5.1 Experimental Setup
To validate the effectiveness of MixPrompt, we conduct experiments on multiple datasets with RGB and auxiliary modalities, including RGB-Depth, RGB-Thermal, RGB-Exent, and RGB-Lidar. NYU Depth V2 [15] dataset contains 1.449 RGB-D samples across 40 categories. SUN-RGBD [19 includes 10.335 RGB-D images with 38 classes. MFNet (10) provides 1.569
pairs
from 9 classes. DELIVER [ comprises 3,983 training and 2,005 testing samples with RGB, Depth, Event, and Lidar modalities across 25 categories.
We use the mean Intersection over Union (mloU) metric for evaluation, with multiscale testing on NYU Depth V2 and SUN-RGBD, and single-scale testing on other datasets, aligning with the evaluation practices of prior works. For the backbone network, we utilize the Mix Transformer encoder (MiT) 34 pretrained on the ADE20K dataset 2 The auxiliary modalities are projected into the feature space using modules placed before the second stage of a ResNet50 (35) network, which initializes the prompt information. This prompt is then fused across multiple scales using four mixed

---

Table 2: Comparison of RGB-T segmentation performance MFNet. Results marked with underline show the second-best performance each category, while those in bold indicate the highest score for that class. UL: Unlabeled, PS: Person, CT: Car top, GD: Guardrail, CC: Color cone.
Method
Backbone
Params FLOPs UL Car PS Bike Curve CT
GD
CC
Bump mloU
MPNet
[00
96.9 65.9 58.9 42.9
29.9
9.9
0.0
25.2
27.7
39.7
RTFNet
ResNet152
245.7M
185.2G 98.5 87.4 70.3
62.7
45.3
29.8
0.0
29.1
55.7
53.2
PSTNet
ResNet18
105.8M
123.4G
97.0 76.8 52.6 55.3
29.6
25.1
15.1
39.4
45.0
48.4
FuseSeg 13
DenseNet161
141.5M
193.4G
97.6 87.9 71.7 64.6
44.8
22.7
6.4
46.9
47.9
54.5
U2Fusion
36
VGG16
97.7 82.8 64.8 61.0
32.3
20.9
45.2
50.2
50.8
AFNet
FV
ResNet50
98.0 86.0 67.4 62.0
43.0
28.9
4.6
44.9
56.6
54.6
ABMDRNct
ResNet50
64.6M
194.3G
98.6 84.8 69.6 60.3
45.1
33.1
5.1
47.4
50.0
54.8
FEANet
39
ResNet152
337.1M
255.2G
98.3 87.8 71.1 61.1
46.5
22.1
55.3
48.9
55.3
GMNet
40
ResNet50
149.8M
153.0G
97.5 86.5 73.1 61.7
44.0
42.3
14.5
48.7
47.4
57.3
TarDAL
41
297M
97.6 80.7 67.1 60.1
34.9
10.5
38.7
45.5
48.6
EAEFNet
42
ResNet152
200.4M
147.3G
87.6 72.6 63.8
48.6
35.0
14.2
52.4
58.3
58.9
CACFNet
4
ConvNect-B 198.6M
101.4G
89.2 69.5 63.3
46.6
32.4
54.9
58.3
57.8
PAIF
260M
88.1
72.4 48.1 60.8
56.0
57.2
56.5
CENet
23
ResNet50
98.1 87.8 71.4 63.2
47.5
31.1
48.9
50.3
56.1
SegMir
45
MiT-B3
98.1 87.8 71.4 63.2
47.5
31.1
48.9
50.3
56.1
CMX 7
MiT-B2
66.6M
67.6G
98.3
89.4 74.8 64.7
47.3
30.1
8.1
52.4
59.4
58.2
CMX
MiT-B4
139.9M
134.3G
98.3
90.1
75.2
64.5
50.2
60.6
59.7
CMNeXt
McT-B4
119.6M
131.9G 98.4
91.5
75.3
67.6
50,5
40.1
9.3
53.4
52.8
59.9
DPLNet
14
MiT-B5
88.58M
105.0G
59.3
Ours
MiT-B5
87.2M
109.0G 98.3 90.2 74.5 65.2
50.1
48.3
10.5
51.7
52.0
60.1
prompting modules, effectively integrating the auxiliary information into the RGB backbone various levels. Additional optimization details are provided in the supplementary material (Section C.1
5.2
Segmentation results
5.2.1
RGB-D segmentation
We first analyze the performance of the proposed MixPrompt framework for RGB-D (depth) seg- mentation, comparing it with several state-of-the-art multimodal fusion methods. The results summarized in Table [ which reports the performance on the NYU Depth V2 and SUN-RGBD
datasets.
On the NYU Depth V2 dataset, MixPrompt achieves the highest mloU score of 61.2 with 87.2M parameters and 109.0G FLOOPs computational cost, demonstrating the efficiency of our approach. In comparison, the second-best method, DPLNet, achieves an mloU of 59.3 with slightly more parameters and lower FLOPs. Methods like ACNet and SGNet, with mloUs of 48.3 and 51.1, respectively, perform significantly worse, highlighting the advantages of our MixPrompt.
On the SUN-RGBD dataset, MixPrompt again achieves the highest mloU of 53.5, surpassing other methods by notable margin. DPLNet ranks second with an mloU of 52.8, while CMX- MiT-B5 follows closely with 52.4. Other methods, such as ACNet and SGNet, continue to show lower performance, with mloUs of 48.1 and 48.6, respectively. The results on SUN-RGBI further reinforce the strength of MixPrompt. A key advantage of MixPrompt is its computational efficiency. With 87.2M parameters and 109.0G FLOPs on NYU Depth V2 dataset, MixPrompt achieves competitive accuracy while being more lightweight compared to other hiff-pceformance models, such as CMX (MiT-B5). The efficient design of the MixPrompt framework, using a lightweight prompting module integrated with pretrained RGB backbone, avoids the need for parallel networks or complex fusion architectures that typically increase both model size and computational cost. Another notable method is DFormer [26), which has significantly fewer parameters compared to other recent approaches. While its design of using a shared branch for both RGB and auxiliary modalities contributes parameter-efficieet architecture, this shared parameterization also limits flexibility of the model. As a result, DForer significantly underperforms in comparison to our framework.
To ensure the fairness of our experimental setup, we additionally examined the impact of different backbone pretraining settings. Specifically, we re-trained several representative multi-modal methods (c.g., CMX and CMNeXt) using the same ADE20K -0E20K--retrained weights ours to provide equitable
7

---

comparison. The results consistently show that while these models benefit slightly from stronger initialization, our method still achieves clear advantage (around 2.9-3.1 mloU higher on NYUDepth) with significantly fewer trainable parameters. Detailed experimental results and analysis are provided the supplementary material (Section
5.2.2 RGB-T segmentation
We then analyze the performance of MixPrompt on the RGB-T (Thermal) segmentation dataset MENet. The results, shown in Table 2 report both class-wise and overall mloU.
On the overall mloU metric, MixPrompt achieves the highest score of 60.1, outperforming all other models in the comparison. The second-bess method, CMNeXt, achieves mloU of 59.9. These top performers are followed by other notable methods such as CMX (MiT-B4), which scored 59.7. In contrast, models like MFNet and RTFNeet achieved significantly lower mloUs of 39.7 and 53.2, respectively, showcasing the advantages of the MixPrompt framework. Focusing on the class-wise performance, our model performs exceptionally well on classes such "Car" and "Bike", with scores of 90.2 and 74.5, respectively. These scores are higher than those achieved by the best-performing models, such CMNeXt, which scored 91.5 and 75.3 for the same classes.
Notably, our model excels in the Car Top (CT) class, achieving an outstanding performance of 48.3, which surpasses all other models in this category.
Despite the competitive performance in many classes, our model continues demonstrate an overall advantage, achieving a balance between class-wise accuracy and computational efficiency. With 87.2M parameters and 109.0G FLOOPs, MixPrompt outperforms others with comparable or even fewer parameters, while maintaining strong class performance.
5.2.3 Other auxiliary modalities
The results 3 different methods on the RGB-E (Event) and RGB-L (Lidar) seg- mentation tasks from the DELIVER dataset. For the RGB-E task. our model achieves notable mloU score of 58.0, surpassing all other RGB-E models, including the second-best CMNeXt and CMX, demonstrating the efficiency and effective- ness of our approach in handling multi-modas RGB-E data.
Table 3: Comparison of RGB-Event and RGB- Lidar segmentation performance on the DE- LIVER dataset. The highest mloU for each condi- tion is highlighted bold, while the second-best score is underlined.
Method
Modal
Backbone Params mloU
HRFuser CMNeXt
46
RGB HRFormer-T
29.9M
48.0
8
RGB
MiT-B2
25.8M
57.2
In the RGB-L task, our model further establishes its superiority by achieving the highest mloU of 59.1. It outperforms the next best method by a considerable margin. Despite having only 29, 19.9M parameters, our model delivers top-tier perfor- mance, demonstrating both efficiency and strong segmentation capabilities in the RGB-L setting.
HRFuser 46
RGB-E HRFormer-T T 30.5M
42.2
TokenFus. 21
RGB-E
MiT-B2
26.0M
45.6
CMX 7
RGB-E
MiT-B2
66.6M
56.5
CMNeXt
RGB-E
MiT-B2
58.7M
57.5
Ours
RGB-E
MiT-B2
29.9M
58.0
FRFeur
16
RGB-L HRFormer-T
30.5M
43.1
TokenFus.
21
RGB-L
MíT-B2
26.0M
53.0
Overall, our model consistently performs the top across both RGB-E and RGB-L modalities, showing a robust ability integrate and process multimodal data for segmentation tasks. The rela-
CMX 7
RGB-L
MiT-B2
66.6M
56.4
CMNeXt
RGB-L
MiT-B2
58.7M
58,0
RGB-L
MiT-B2
29.9M
59.1
tively low number of parameters required further emphasizes the efficiency of our approach compared to other beggh-erforming models.
5.3 Ablation study
To comprehensively evaluate our approach, we conduct an ablation study on the NYU Depth V2 dataset using MiT-B5 as the default model. The ablations cover three main aspects: the effectiveness of each module, the detailed design choices within each module, and the impact of key hyperparameters.

---

5.3.1 Effectiveness of each module
In order to evaluate the contributions of the key components in our proposed method, we conduct an ablation study focusing on two essential modules: the pretrained prompt extractor and the mixed prompting module. The results are summarized in Table
When both modules are enabled, the best performance, with an mloU 60. 10,, is achieved. However, when the pretrained prompt extractor is disabled, where we employ simple patch embedding layer to integrate the auxiliary modality, the model achieves an mloU of 58.8%. Similarly, the mixed prompting module is removed, modality fusion is achieved through direct addition of the features without any alignment, and the mIoU score decreases slightly to 59.5%. From these experiments, the combination of the pretrained prompt extractor and the mixed prompting module provides the best results, highlighting that both modules in our proposed framework play an important role in enhancing performance.
Table 4: Ablation results for the pretrained prompt extractor and the mixed prompting module.
Prompt extractor
Mixed Prompting
Trainable params
FLOPs
mloU (%)
5.7M
109.0G
60.1
5.5M
104.7G
58.8
x
5.4M
108.7G
59.5
5.3.2 Prompt extractor
We further investigate the design of the pretrained prompt extrac- beginning with analysis of whether initializing the extractor with pretrained weights affects performance. As shown in Table using randomly initialized ResNet50 prompt extractor results in an mloU of 59.5%, whereas initializing it with pretrained weights improves performance 60.1%. This highlights the effective- ness of leveraging pretrained RGB models for extracting initial prompts. The pretrained backbone enables the model to extract more meaningful representations from the auxiliary modality, ultimately leading to better segmenta- tion results.
Table 5: Impact of initializing the ResNet50 prompt extractor with pretrained weights.
Initialization type mloU (%
Random
59.5
Pretrained
60.1
Next, we assess the impact of different prompt ex- tractor architectures, with the results presented in Table Given that the backbone follows the MiT architecture, we first evaluate various MiT variants as the prompt extractor. The results from MiT-B1 show that deeper extractors lead lower mloU, while progressively reducing the depth improves performance, with the best result achieved using only the first stage. Notably, ResNet50 outperforms MIT-BI, highlighting the effectiveness of convolu- tional models in extracting initial prompts. Addi- tionally, increasing the size of MiT extractors offers no further improvements, reinforcing the advantage of convolutional extractors.
Table 6: Impact of different prompt extractor architectures and the number of layers used. A shallower convolutional extractor focusing on carly-stage features achieves the best results.
Prompt extractor
Layers
mloU (%)
mit-hl
stage (1,2,3,4)
mit-hl
stage (1,2,3)
59.5
mit-bl
stage (1,2)
59.6
mit-bl
stage
59.7
mit-b2
stage (1)
59.6
mit-b4
stage (1)
59,2
mit-b5
stage (1}
59,3
ResNet50
stage [1,2,3,4]
58.9
ResNet50
stage (1.2.3)
59.5
These findings suggest that pretrained convolu- tional prompt extractor focusing on carly-stage fea- tures is more beneficial for prompted segmentation. The first stage of pretrained ResNet50 provides the best performance. Therefore, we adopt this configuration as our final design.
ResNet50
stage (1,2)
59,9
ResNet50
stage (1)
60.1
5.3.3 Prompt mixing
We further investigate the impact of different prompt mixing configurations, particularly the effect of multi-subspace prompt mixing when applied to either the RGB auxiliary modality. The results are summarized in Z

---

From the results, the model achieves an mloU of 59.3% without multi-subspace mixing. When multi- subspace mixing is introduced to the RGB modal- ity, performance improves significantly demonstrating the benefit of refining RGB prompts through multi-subspacc alignment. Conversely, ap- plying multi-subspace mixing only the auxiliary modality yields a smaller improvement, reaching 59.7%. Interestingly, enabling multi-subspace mix- ing for both modalities does not lead to further gains and instead results in slightly lower mIoU of 59.8% compared to the RGB-only setting.
Table Ablation study on multi-subsspace prompt mixing. The best result is achieved when multi- subspace mixing applied only to RGB prompts.
60. %,
RGB mixing
Auxiliary mixing
mloU (%)
x
x
59.3
V
x
60.1
X
/
59.7
59.8
These results suggest that enhancing the RGB modality with multi-subspace mixing plays more crucial role in improving segmentation performance. In contrast, applying the same strategy to the auxiliary modality provides more limited benefits, and simultaneous application to both may introduce redundant or conflicting information. Thus, our final design prioritizes multi-subssaac mixing for RGB prompts to achieve optimal results.
5.3.4 Hyperparameters
We further perform an ablation study on key hyperparameters. First, we analyze the effect of the rank downscale ratio d. which controls the intermediate feature dimension and effectively determines the rank of the low-rank subspace. A larger ratio corresponds to a smaller rank and fewer trainable parameters. As shown in Table 8 increasing the downscale ratio from I to improves the mIoU from 59.4% to 60.1%, indicating that moderate reduction in rank enhances feature efficiency. However, when the ratio is further increased to 8, performance drops slightly to 59.6%, suggesting that an excessively small rank may lead to information loss and hinder performance. The optimal trade-off is achieved at ratio of 4.
Next, we investigate the impact of the number of subspaces n. used for prompt mixing. Unlike the rank downscale ratio, this parameter does not affect parameter efficiency the model. As reported in Table 9 increasing the number of subspaces from 1 to 4 progressively enhances performance, reaching a peak mloU of 60.1%. However, further increasing the number of subspaces 8 results in performance decline, suggesting that overly complex prompt mixing may introduce unnecessary redundancy.
Furthermore, we validated our hyperparameter configuration on multiple multimodal datasets, in- cluding -he-malrmal, RGB-LiDAR, and RGB-Event. The results indicate that the selected values strike an effective balance among model capacity, generalization, and computational efficiency. Due to space limitations, more detailed results are provided in the supplementary material (Section
Table 8: Ablation study on the rank downscale ratio.
Table 9: Ablation study on the number of sub- spaces for prompt mixing.
Rank down scale ratio
Num. of Subspace n
mloU (%)
59.4 59.8 60.1 59.6
mIoU (%)
59.3 59.8 60.1 59.3
6
Conclusion
In this paper, we introduced MixPrompt, novel framework for multimodal semantic segmentation that efficiently integrates auxiliary modalities into pretrained RGB models through prompt tuning. By leveraging lightweight prompting module and multi multi-subspace alignment, MixPrompt successfully enhances model performance while maintaining parameter efficiency. Our method addresses the challenges of increased model complexity and data scarcity commonly associated with multimodal segmentation tasks. Through comprehensive experiments on multiple datasets, including NYU Depth V2, SUN-RGBD, MFNet, and DELIVER, we demonstrated that Mix outperforms existing hual-backbooe approaches with fewer parameters, establishing it as a highly effective and scalable solution.
10

---

Acknowledgment
This work was supported by the Joint Funds of the National Natural Science Foundation of China (NSFC) (No. U2336211), I), the Major Research Plan of the NSFC (No. 92467206), and the NSFC Grant (No. 62576364).
References
[1] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban secne understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[2] Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through adc20k datasct. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-6411 2017.
[3] Xinxin Hu, Kailun Yang. Lei Fei, and Kaiwei Wang. Acnet: Attention based network to exploit com- plementary features for rgbd semantic segmentation. In 2019 IEEE International conference on image processing (ICIP), pages
IEEE, 2019.
[4] Lin-Zhuo Chen, Zheng Lin, Ziqin Wang. Yong-Liang Yang, and Ming-Ming Cheng. Spatial information guided convolution for real-time rgbd semantic segmentation. IEEF Transactions on Image Processing. 50:2313-2324, 2021.
[5] Xiaokang Chen, Kwan- Yee Lin, Jingbo Wang, Wayne Wu, Chen Qian, Hongsheng Li, and Gang Zeng. Bi-directional cross-modality feature propagation with separation-amllaggreaaiioo gate for rgb-d semantic segmentation. In European conference on computer vision, pages 561-577. Springer, 2020.
[6] Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, and Junzhou Huang. Deep multimodal fusion by channel exchanging. Advances in neural information processing systems, 33:4835-44 4845, 2020.
(7) Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, and Rainer Stiefelhagen. Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers. IEEE Transactions on Intelligent Transportation Systems, 2023.
[8] Jiaming Zhang, Ruiping Liu, Hao Shi, Kailun Yang, Simon Reiß, Kunyu Peng, Haodong Fu, Kaiwei Wang, and Rainer Stiefelhagen. Delivering arbitrary-modal semantic segmentation. In Proceedings of the IEEF/VV Conference Computer Vision and Pattern Recognition, pages 1 1136-1147, 2023.
[9] Zhiwei Hao, Zhongyu Xiao, Yong Luo, Jianyuan Guo, Jing Wang, Li Shen, and Han Hu. Primkd: Primary modality guided multimodal fusion for rgb-d semantic segmentation. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1943-1951, 2024.
[10] Qishen Ha, Kohei Watanabe, Takumi Karasawa, Yoshitaka Ushiku, and Taisuya Harada. Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5108-5115. IEEE, 2017.
[11] Yuxiang Sun, Weixun Zuo, and Ming Liu. Rtfnet: Rgb-thermal fusion network for semantic segmentation of urban scenes. IEEE Robotics and Automation Letiers, 4(3):2576-2588 2019.
[12] Shreyas S Shivakumar, Neil Rodrigues. Alex Zhou, Ian D Miller, Vijay Kumar, and Camillo J Taylor. Ps1900: Rgh-thermal calibration, dataset and segmentation network. In 2020 IEEE international conference on robotics and automation (ICRA), pages 9441-9447. IEFF, 2020.
[13] Yuxiang Sun, Weixun Zuo, Peng Yun, Hengli Wang, and Ming Liu. Fuseseg: Semantic segmentation of urban scenes based on rgb and thermal data fusion. IEEE Transactions on Automation Science and Engineering, 1803):1000 1011. 2020.
[14] Shaohua Dong, Yunhe Feng, Qing Yang, Yan Huang, Dongfang Liu, and Heng Fan, Efficient multimodal semantic segmentation via dual-prompt learning. arXiv preprint arXiv:2312.00360 2023.
[15] Nathan Silherman, Derek Hoiem, Pushmeet Kohli, and Roh Fergus. Indoor segmentation and support inference from rghd images. In Computer Vision-ECCV 2012: 12th European Conference Computer Vision, Florence, Italy. October 7-13, 2012, Proceedings. Part V 12, pages 746-760. Springer, 2012.
[16] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
11

---

[17] Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun. and Rongrong Ji. Cheap and quick: Efficient vision-language instruction luning for large language models. Advances in Neural Information Processing Systems, 36, 2024.
[18] Zhiwei Hao, Jianyuan Guo, Li Shen, Yong Luo, Han Hu, and Yonggang Wen. Adem-vl: Adaptive and embedded fusion for efficient vision-language tuning. arXiv preprint arXiv:2410.17779 2024.
[19] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgh-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 667-576,32 2015.
[20] Zongwei WU, Zhuyun Zhou, Guillaume Allibert, Christophe Stolz, Cédric Demonccaux, and Chao Ma. Transformer fusion for indoor rgb-d semantic segmentation. Available at SSRN 4251286, 2022.
[21] Yikai Wang, Xinghao Chen, Lele Cao, Wenhing Huang, Fuchun Sun, and Yunhe Wang. Multimodal token fusion for vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, pages 12186-122 111955, 2022.
[22] Daniel Seichter, Mona Köhler, Benjamin Lewandowski. Tim Wengefeld, and Horst-Michad Gross. Effi- cient rgb-d semantic segmentation indoor scene analysis. In 2021 IEEE international conference on robotics and automation (ICRA), pages 13525-13531. IEFF, 2021.
[23] Zhen Feng, Yanning Guo, and Yuxiang Sun. Cekd: Cross-modal edge-privileged knowledge distillation for semantic scene understanding using only thermal images. IEEE Robotics and Automation Letters, 8(4):2205-2212, 2023.
[24] Jinming Cao, Hanchao Leng, Dani Lischinski, Daniel Cohen-Or, Changhe Tu, and Yangyan Li. Shapeconv: Shape-aware convolutional layer for indoor rgb-d semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7088-7097, 2021.
[25] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens Van Der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16102-16112, 2022.
[26] Bowen Yin, Xuying Zhang, Zhongyu Li, Li Liu, Ming-Ming Cheng, and Qibin Hou. Dformer: Rethinking rgbd representation learning for semantic segmentation. arXiv preprint 2XXi:23309009666 2023.
[27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabricl Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[28] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint ar 2rXiv:2203.17274 2022.
[29] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongic, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
[30] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1xXi:2106.006855 2021.
[31] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang. Zhiheng Xi, Xiaoran Fan, al. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1932-1945, 2024.
[32] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The sparsely- gated mixture-of-expert layer. arXiv preprint Sxtv:/701.0653 2017.
[33] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF conference on computer vision and puttern recugnition, pages 13733-13742, 2021.
[34] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 14:17777-12090 2021.
12

---

[35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[36] Han Xu, Jiayi Ma, Junjun Jiang, Xiaojie Guo, and Haibin Ling. U2fusion: A unified unsupervised image fusion network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):502- 518, 2020,
[37] Jiangtao Xu. Kaige Lu, and Han Wang. Attention fusion network for multi-specttal semantic segmentation. Pattern Recognition Letters, 146: 146:179-184, 2021.
[38] Qiang Zhang, Shenlu Zhao, Yongjiang Luo, Dingwen Zhang, Nianchang Huang, and Jungong Han. Abmdmet: Adeptive-weighted bi-directional modality difference reduction network for rgb-1 semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 2633 2642, 2021.
[39] Fuqin Deng, Hua Feng, Mingjian Liang, Hongmin Wang. Yong Yang, Yuan Gao, Junfeng Chen, Junjie Hu, Xiyue Guo, and Tin Lun Lam. Feanet: Feature-enhanced attention network for rgb-thermal real-time semantic segmentation. In 202/ JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 4467 4473. IEEE, 2021.
[40] Wujie Zhou, Jinfu Liu, Jingsheng Lei. Lu Yu, and Jenq-Neng Hwang. Gmnet: Graded-feature multilabel- learning network for rgb-thermal urban scene semantic segmentation. IEEE Transactions on Image Processing, 30:7790-7802, 2021.
[41] Jinyuan Liu, Xin Fan, Zhanbo Huang, Guanyao Wu, Risheng Liu, Wei Zhong, and Zhongxuan Luo.. Target-aware dual adversarial learning and multi-scenario multi-modalig benchmark to fuse infrared and visible for object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5802-5811, 2022.
[42] Mingjian Liang, Junjie Hu, Chenyu Bao, Hua Feng, Fuqin Deng, and Tin Lun Lam. Explicit attention- enhanced fusion for rgb-thermal perception tasks. IEFE Robotics and Automation Letters, 8(7):406044067, 2023.
[43] Wujie Zhou, Shaohua Dong, Mcixin Fang, and Lu Yu. Cacfnet: Cross-modal attention cascaded fusion network for rgb-t urban scene parsing. IEEE Transactions Intelligent Vehicles, 2023.
[44] Zhu Liu, Jinyuan Liu, Benzhuang Zhang, Long Ma, Xin Fan, and Risheng Liu. Paif: Perception-aware infrared-visihle image fusion for attack-tolerant semantic segmentation. In Proceedings of the 31 ACM International Conference on Multimedia, pages 3706-3714, 2023.
[45] Jinyuan Liu, Zhu Liu, Guanyao Wu, Long Ma, Risheng Liu, Wei Zhong, Zhongxuan Luo, and Xin Fan. Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8115-8124, 2023.
[46] Tim Broedermann, Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Hrfuser: multi-resolution sensor fusion architecture for object detection. In 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), pages 4159-4166. IEEE, 2023.
[47] Nameirakpam Dhanachandra, Khumanthem Manglem, and Yambem Jina Chanu. Image segmentation using k-means clustering algorithm and subtractive clustering algorithm. Procedia Computer Science, 54:764-774 2015.
[48] Ziwei Liu, Xiaoxiao Li, Ping Luo, Chen Change Loy, and Xiaoou Tang. Deep learning markov random field semantic segmentation. IEEE transactions on pattern analysis and machine intelligence, 1814):1814 1828, 2017.
[49] Raviteja Vemulapalli, Oncel Tuzel, Ming-Yu Liu, and Rama Chellapa. Gaussian conditional random field network for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 3224 3233, 2016.
[50] Jonathan Long, Evan Shelhamer. and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 3431-3440. 2015.
[51] Di Lin, Yuanfeng Ji. Dani Lischinski, Daniel Cohen -r. and Hui Huang. Multi-scale context intertwining for semantic segmentation. In Proceedings of the European Conference on Computer Vision (ECCV). pages 603-619, 2018.
13

---

[52] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3146-3154, 2019.
[53] Yunchao Wei, Huaxin Xiao, Honghui Shi, Zequn Jie. Jiashi Feng, and Thomas Huang. Revisiting dilated convolution: A simple approach weakly- and supervised semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2018.
[54] Wen Huang. Hongbin Liu, Minxin Guo, and Neil Zhengiang Gong. Visual hallucinations of large language models. arXiv preprint 24t.:2402.146833 2024.
[55] Dianmo Sheng. Dongdong Chen, Zhentao Tan, Qiankun Liu. Qi Chu, Jianmin Bao, Tao Gong, Bin Liu. Shengwei Xu, and Nenghai Yu. Towards more unified in-context visual understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13362-13372, 2024.
[56] Léon Bottou. Stochastic gradient descent tricks. In Neural Networks: Tricks of the Trade: Second Edition, pages 421-4334 Springer, 2012.
[57] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint M4Xiv:1412.69000 2014.
[58] Radhakrishnan Gopalapillai. Deepa Gupta, Mohammed Zakariah, and Yousef Ajami Alotaibi. Convolution- based encoding of depth images for transfer learning in rgb-d scene classification. Sensors, 21(23)77950, 2021.
[59] Tobias Pohlen, Alexander Hermans, Markus Mathias, and Bastian Leibe. Full-resolution residual networks for semantic segmentation in street scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4151-4160 2017.
[60] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Learning discrimina- tive feature network for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1857- 1866, 2018.
[61] Changqian Jingbo Wang. Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 325-341, 2018.
[62] Caner Hazirbas. Lingni Ma. Csaba Domokos, and Daniel Cremers. Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture. In Computer Vision-ACCV 2016: 13th Asian Conference on Computer Vision, Tripei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part I 13, pages 213-228. Springer, 2017.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly summarize the three main contributions this article.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations of work in the supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have disclosed the information required for the main experimental results of the paper in the experimental section
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We will open source the code after the draft is finalized
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeeee 7/nips.com htc/:/ubbb///gdddee///eeeeeeisss ssiniiiiccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We have detailed experimental instructions the experimental section and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
All experimental results meet the requirements
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have relevant explanations the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research fully complies with the required ethical standards.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the relevant impacts in the supplementary materials.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no such risk in this paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper meets the relevant requirements.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs does not affect methods, scientific rigor, or originality of research
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neurii: /ttp://p...///ooeeeerr/////////0 for what should or should not be described.
21

---

A Related works
A.1 Semantic segmentation.
The task of semantic segmentation not only requires identifying objects within an image but also de- mands precise outlining of their boundaries, rendering them considerably more challenging compared to image classification tasks. The development of traditional methods can be divided into two major stages. The first stage revolves around classic approaches based on handcrafted features, ranging from k-means clustering 447 to Markov Random Fields [48] and Conditional Random Fields [49]. The second stage marks the shift to early deep learning- based methods, such as Fully Convolutional Networks [50]. which laid the foundation for modern segmentation tasks. Building on this foundation, techniques like multi-scale pyramids [5T], attention mechanisms [52], and dilated convolutions [53]
have been introduced, significantly improving segmentation accuracy by capturing richer contextual information and enhancing feature representation.
Although these methods have achieved commendable performance, they typically rely on RGB images for predictions. However, RGB images alone often lack sufficient information for precise semantic segmentation, especially in complex scenarios such as low-light conditions, blurred textures, or occluded target objects. To address these challenges, researchers are exploring the integration of other data modalities, such as depth maps [15] [9] and Thermal images [IO], leveraging the complementary advantages of multimodal data to enhance the accuracy and robustness of semantic segmentation across diverse environments.
Existing methods primarily focus on two aspects. The first aspect emphasizes the alignment and fusion between different modalities 3
8 21 22 TO 1 23 Researchers explored diverse strategies for alignment and fusion across multiple levels, encompassing the input, intermediate feature extraction layers, and output. For instance, Cao et al. [24] incorporates geometric information from auxiliary modalities into convolutional weights, establishing link between the weights and the underlying spatial relationships of corresponding pixels to better capture the spatial structure of scenes. Similarly, Hu et al. []] introduces an additional attention based auxiliary module to fuse features from different modalities, further balancing feature distributions and enabling the network to focus more effectively on the most relevant regions of the image. Zhang et al. 7 proposes innovative cross-modal feature calibration and fusion modules, aligning and calibrating feature differences in spatial and channel dimensions across modalities at multiple scales of the model.The second aspect
6 20,
focuses on more effective feature extraction 24 5
25 26 For example, Girdhar et al. [25]
introduces a novel Transformer architecture that is jointly pre-trained on images, videos, and single- view 3D data, equipping the model with cross-modal semantic feature extraction capabilities and making it suitable for downstream tasks across different modalities. Meanwhile, Zhang et al. [26] constructs new large-scale RGB-Ding image dataset for pre-training, enhancing the ability of the model to encode both RGB and depth images.
Nevertheless, the introduction of additional modality-specific feature extraction networks significantly increases model complexity, resulting in an overwhelming training overhead. Moreover, the scarcity of existing multimodal datasets makes it challenging to support large-scale pretraining tasks, which hinders the model ability to be quickly fine-tuned for various downstream tasks across different modalities.
A.2 Multimodal Prompting.
Vision-Language Models (VLMs) are deep learning models that integrate both visual and textual information, with visual capabilities that enable understanding and reasoning across complex mul- timodal tasks. The introduction of visual prompts, complementing textual prompts, enables more granular, pixel-level instructions on multimodal inputs, helping to mitigate challenges in traditional multimodal language models, such as visual hallucinations [54] and linguistic biases (35). For
instance, CLIP [27] leverages contrastive learning to align shared semantic spaces between images and text. LLaVA [6), by combining visual prompts with large-scale pretrained language models, has achieved remarkable results in image-text reasoning and generation tasks. This approach not only taps into the powerful foundational capabilities of large-scale pretrained models but also allows visual prompts to adaptively establish connections between different modalities in various forms tailored to specific tasks. An initial attempt to apply prompt tuning to the multimodal semantic segmentation task was made by Dong et al. [14]. However, their approach utilizes the same summation operation
22

---

to fuse features from different modalities without considering the differences between the additional modality and the RGB modality, which may interfere with the feature distribution of the primary RGB modality and affect ability of the model to capture important visual information.
B Methodology Details
B.1 Details of mixed prompting
The detailed algorithmic procedure for the mixed prompting module is presented Algorithm
Algorithm 1 Mixed Prompting Module
Require: hidden state f; prompt 6;-1. low-rank matrices Wigh Rxd,, W, E Rexd Wup E Rdxr, number of subspaces mixing matrix M E IR*x", model layer L; Ensure: hidden state h;, prompt e;
1: êi- <- Wyei-
2: 4 1
3: 14-1 f- reshape(h;-10) (-1,
4: h4-1 t- M
Mixing
5: h t Batten(h:-)
6: e +- (mn(é + h1-1)
new prompt
7: h4 + Li(h;-1 ee)
b new hidden state
C
Additional Experiment Details.
C.1 Optimization and Schedule.
We provide detailed configurations for training on datasets including NYU Depth v2 15). SUN-
RGBD [99, MFNet [0], and DELIVER datasets. The specific hyperparameters are summarized
in Table 10
1 and an initial learning rate of 0.04. The model is trained for 500 epochs with batch size of 8. For the SUN-RGBD
For the NYU Depth V2 dataset, use SGD [56] with a weight decay
dataset, we adopt the AdamW optimizer [57] with 100 epochs and an initial learning rate 0.005. For the MFNet dataset, we train for 500 epochs with the Adam W optimizer, a learning rate of 6 x 10 and a batch size of 4. For the DELIVER dataset, we train for 200 epochs with batch size of using a learning rate of 6 10-5 and weight decay factor of 0.01. All experiments are conducted on NVIDIA GeForce RTX 3090 GPUs. Data augmentation techniques, including random flipping, random cropping, and multiscale inference with scales (0.5, 0.75, 1.0, 1 1.25, 1.5, 1.75), are applied during training for all datasets.
Table 10: Training configurations for different datasets. LR: Learning Rate, WD: Weight Decay,
Mom: Momentum.
Dataset
Input Size
Batch Size
Epochs
Optimizer
LR
WD
Mom
NYUD-v2
480x640
500
SGD
4e-2
0.0005
0.9
SUN-RGBD
530x730
4
100
Adam W
5e-3
0.01
(0.9, 0.999)
MENet
480x640
500
Adam
6e-4
0.01
(0.9, 0.999)
DELIVER
1024x1024
2
200
AdamW
6e-5
0.01
(0.9, 0.999)
23

---

100
100
50
0
-50
-50
-100
-100
-50
50
100
-100
-50
30
100
(a) RGB.
(b) Depth.
Figure 4: t-SNE visualization of feature embeddings for RGB and depth inputs, indicating that the feature from depth images results in similar clustering levels to those from RGB images.
C.2 Analysis for Prompt Initialization
To further validate the feasibility using pre-trained RGB models for prompt initialization, we conduct t-SNE analysis to visualize the distribution of features extracted from both modalities.
In this analysis, each pixel is treated as data point, and the corresponding ground truth serves as the label. The results, presented in Figure ! demonstrate that the feature representations for both RGB and depth inputs form clusters a similar level in the lower-dimensional space. For the RGB modality, the points are spread out with multiple clusters, reflecting the rich variety of information captured from the RGB image. Similarly, in the depth modality, the points also form well- defined clusters. This suggests that the features extracted from the depth modality, although distinet from RGB features in some aspects, are sufficiently informative, indicating that pretrained RGB models allows MixPrompt to efficiently integrate auxiliary modalities without requiring additional, modality-specine pretrained extractors.
C.3 Analysis for Different Backbone Pretraining Settings
This note addresses the concerns raised regarding the faimess of the experimental setup. To ensure cquitable comparison, have conducted additional experiments where we initialized several prior multi-modal approaches (c.g., CMX and CMNeXt) with ADE20K pretrained weights, followed by training on the NYUDepth dataset. The results are summarized below.
ADE20K-pretrained
Table 11: Comparison of different method with different backbone pretraining settings.
Method
Backbone Pretraining
Trainable Params
mloU (%)
Segformer
MiT-B5 (ADE20K)
82.7M
54.7
CMX
MiT-B5 (ImageNet)
181.1M
56.9
CMX
MiT-B5 (ADE20K)
181.1M
58.1
CMNeXt
MrT-B4 (ImageNet)
119.6M
56,9
CMNeXt
MiT-B4 (ADE20K)
119.6M
58.3
Ours
MiT-B5 (ADE20K)
5.7M
61.2
As shown in Table II we observe that CMX and CMNeXt do benefit from using the same strong baseline ours, resulting in limited performance gains. However, the gap between their results and ours remains substantial (e.g., a 2.9-3.1 mloU difference on the NYUDepth dataset). This comparison under the pretrained setting demonstrates the effectiveness of our method over other molti-modal fusion approaches. Notably, our model achieves this with only 5.7M trainable parameters--sinifiianty fewer than the alternatives.
Additionally, we conducted experiments where the single-modality RGB backbone (SegFormer) was also initialized with ADE20K-pretrainn weights. The results further indicate that the performance

---

gains achieved by our method go well beyond what can be attributed to pretraining alone, highlighting the strength of our fusion strategy.
Overall, these experiments provide meaningful evidence for our fairness of the experimental setup.
C.4 Experiments on the Effectiveness of the Encoding Strategy
To evaluate the contribution of reusing the early layers of a pretrained RGB backbone as an embedding strategy for auxiliary modalities, we conducted additional experiments by replacing our pretrained RGB backbone with the convolution- based encoder proposed [58). Experiments were performed on both RGB-D and RGB-T datasets, and the results are summarized in Table 12
Table 12: Comparison of different encoding methods in the Mixed Prompting Module.
Encoder Method
NYUDepth (RGB-D)
MFNet (RGB-T)
Convolution Based Encoding [58] Ours
58.7
56.8
61.2
60.1
As shown in Table 12 the previous convolution- based depth encoder does not adapt well to the prompting framework. It likely provides suboptimal prompt initialization, leading to an approximate performance drop of 2.5% mloU compared to our trainable backbone on RGB-D datasets. Moreover, results on the RGB-T dataset reveal that it fails to generalize effectively to arbitrary modality encoders, limiting its scalability.
C.5 Quantitative Analysis of multi-subspace distributions.
To quantitatively characterize the diversity among the learned subspaces, we conducted additional analysis. Following the hierarchical encoder architecture of SegFormer, our backbone extracts four- stage multi-scale features with progressively lower spatial resolutions, denoted as dl-d4 (from highest to lowest resolution). For each stage, we compute the average pairwise cosine similarity between projected RGB subspace features, where values range from -1 (completely dissimilar) to (identical). Lower (more negative) similarity values indicate greater diversity across subspaces. The "Avg.Sim (overall)" column summarizes the mean similarity across all four stages, providing an overall measure of inter-subspace diversity.
Table 13: Average pairwise cosine similarity (Sim) between RGB subspace features across different numbers of subspaces (n) and corresponding segmentation performance.
Sim (d1)
Sim (d2)
Sim (d3)
Sim (d4)
Sim (overall)
mloU (%)
N/A
N/A
N/A
N/A
N/A
59.3
-0,0338
-0,0355
-0.0508
-0,0853
-0.0514
59.8
-0.1434
-0.1715
-0.2031
-0,2506
-0.1922
60.1
-0.0135
-0.0046
-0.0175
-0.0321
-0.0169
59.6
As reported in Table 13 we evaluated how varying the number of subspaces (n) affects feature diversity using RGB-D inputs from the NYU Depth V2 test set. Our analysis reveals that within the same n, similarity values become increasingly negative from high-resolution (d1) to low-resolution (d4) features. This occurs because high-resolution features predominantly capture modality- local patterns (e.g., edges and textures) that remain relatively consistent across subspaces, whereas lower-resolution features encode more abstract, global, and tross-modal semantics. Consequently, multi-subspaae prompt mixing more effectively disentangles these higher-leved representations, resulting in greater divergence at coarser scales.
Furthermore, increasing from 1 to 4 substantially reduces overall similarity (from -0.0514 to -9.12,,), reflecting enhanced structural diversity that correlates with optimal segmentation perfor- mance (mloU of 60.1%). However, further increasing to 8 causes similarity magnitude to decrease (from -0. 1922 to -0.0169), indicating over fregffragmnttiionn that diminishes subspace diversity and aligns with the observed performance degradation.
25

---

These quantitative results validate the core motivation behind our multi--ubsppcee design: by introduc- ing structural diversity and encouraging disentangled prompt composition, it effectively improves segmentation outcomes. While comprehensive visualizations of subspace distributions would provide additional intuitive insights, we believe this detailed statistical characterization offers compelling evidence for the effectiveness of our approach.
C.6 Analysis for Merging Strategy in the Mixed Prompting Module
We adopt simple summation of the two prompt embeddings (hş-1 and 6;-1), primarily due to its parameter-efficinncy and empirical effectiveness in our preliminary experiments. While our current approach is effective, fully recognize the potential of alternative fusion strategies atrategie-suuhh as concatenation followed by a projection layer, learnable weighted summation via gating, or attention- based fusion-to enable more expressive cross-modal interactions, as the reviewer rightly pointed out. However, these methods typically introduce additional parameters and computational overhead, potentially undermining the objective of preserving a compact and efficient prompting module, while also increasing the complexity of training.
To futher investigate the effectiveness of these fusion methods, conducted ablation studies on different prompt fusion strategies using the NYUDepth dataset. The quantitative results are shown in the table below, and the implementation details of each variant are described as Table [4]
Concatenation. Concatenate h and 6j-1 along the module dimension, and apply linear transfor- mation to fuse the combined representation back into the original embedding space.
Learnable Weighted Summation. Introduce learnable gating vector g E RC and compute the fused prompt as (g) h + (1 (g)) es-1, where denotes the sigmoid activation function. The gate g initialized as a trainable parameter and applied channel-wise to adaptively modulate the contribution of each modality.
Attention Fusion. The fused prompt is obtained by computing cross-modal attention between 6:-1 and h4-1 (Qt K, V) = softmax QKT
va V.
Overall, our results on NYUDepth indicate that simple summation achieves the best performance among the evaluated strategies, highlighting its effectiveness as a lightweight and robust fusion mech- anism. Although concatenation and learnable weighted summation offer more flexible interactions between the two prompt streams, they do not yield noticeable performance improvements in our setting. One possible explanation that the two prompts already provide sufficiently complementary information, and more complex merging may introduce redundancy disrupt this balance. In contrast, attention fusion not only degrades performance, but also incurs substantial memory overhead. This primarily due to the pairwise similarity computation between all token positions, which generates large N x N attention map, where N corresponds to the number of spatial positions the feature map. For high-resolution inputs, N can be large, making this step particularly memory-intensive.
These findings collectively support our use of simple summation as practical and efficient fusion strategy in the mixed prompting module. We believe these findings provide a useful reference for future work exploring more adaptive fusion mechanisms in multimodal prompting.
Table 14: Comparison of different prompt fusion strategies in the Mixed Prompting Module.
Fusion Method
Trainable params
FLOPs
mloU (%)
Summation
5.74M
109.01G
60.1
Concatcnation
5.83M
109.09G
59.4
Learable Weighted Sum
5.74M
109.01G
59.5
Attention Fusion
5.84M
109.09G
57.0
C.7 Further Validation of Hyperparameter Selection Generalizability
To evaluate the generalizability of our selected settings (e.g., a rank downscale ratio of 4 and subspaces), we conducted additional ablation studies on multiple datasets beyond the NYU Depth v2 dataset originally reported. Specifically, we evaluated the effect of these parameters on MFNet and DELIVER datasets, which encompass GGe-mherall, RGB-LiDAR, and RGB-Event modalities.
26

---

These datasets represent diverse sensing conditions and semantic distributions, offering a broader testing ground for robustness.
Across the diverse datasets evaluated, our selected parameter configuration consistently demonstrated competitive performance and stable optimization behavior. These results indicate that chosen values strike favorable balance among model capacity, generalization, and computational efficiency, making them practical default for wide range of multimodal segmentation scenarios.
While slight tuning may still benefit extremely domain-shifted settings, our results demonstrate that the chosen configuration is robust and transferable, reducing the burden of per-dataset hyperparameter adjustment practice.
Table 15: Ablation study on the Ratio (rank downscale ratio).
Dataset
Modal
Ratio=1
Ratio=2
Ratio=4
Ratio=8
NYU Depth v2
RGB-D
59.4
59.8
60.1
59.6
MFNet
RGB-T
58.8
59.2
60.1
59.7
DELIVER
RGB-L
57.6
58.0
59.1
58.4
DELIVER
RGB-E
57.2
57.8
58.0
57.8
Table 16: Ablation study the Num (number of subspaces for prompt mixing)
Dataset
Modal
Num=1
Num=2
Num=4
Num=8
NYU Depth v2
RGB-D
59.3
59.8
60.1
59.3
MFNet
RGB-T
58.4
58.9
60.1
59.5
DELIVER
RGB-L
58.3
58.5
59.1
58,8
DELIVER
RGB-E
57.1
57.6
58.0
57.5
C.8 Experiments for Different Illumination Conditions
To validate the robustness of our method under varying illumination conditions, report the performance on the RGB-T MFNet dataset under both daytime and nighttime conditions in Table [7
On the daytime mloU, our method achieves mloU score of 51.8, ranking second after the best-performing model, CMX (MiT-B4), which scores 52.5. In the nighttime mloU, standout performance is achieved, where model achieves the highest score of 61.0, surpassing all other methods. This demonstrates superior ability of our method to segment objects under nighttime conditions, where challenges such poor lighting and low visibility most pronounced.
Table 17: Comparison of RGB-T segmentation performance the MFNet dataset across daytime and nighttime conditions. The highest mloU for each condition is highlighted in bold, while the second-best score is underlined.
Method
Modal
Day
Night
FRRN
RGB
40.0
37.3
DFN
60
RGB
38.0
42.3
BiSeNet
61
RGB
44.8
47.7
SegFormer-B2
RGB
48.6
49.2
SegFomme-
B4
RGB
49.4
52.4
MFNet
10
RGB-T
36.1
36.8
FuseNet
62
RGB-T
41.0
43.9
RTFNet 11
RGB-T
45.8
54.8
FuseSeg
13
RGB-T
47.8
54.6
GMNet
RGB-T
57.7
CMX(MIT-B2
7
RGB-T
$1.3
57.8
CMX
RGB-T
52.5
59.4
CMNeXt 8
RGB-T
50.5
59.8
Ours
RGB-T
51.8
61.0
27

---

(RG eepth)
(RGB-Event)
N/A
(RGB-Lidar)
NA
600
CTA
CTOP
Auxiliary
CMX
CMNeXt
DPLNet
Ours
Figure 5: Visualization of results on the DELIVER dataset under depth, thermal, Event, and Lidar auxiliary modalities. DPLNet is not implemented for RGB-Event and RGB-Lidar data in the original paper.
C.9
Visualization
To provide a qualitative comparison between MixPrompt method and existing approaches, present visualizations of segmentation results from a sample image in each dataset used above. These visualizations shown in Figure
The comparison illustrates the ability of our model to produce more accurate and refined segmentation boundaries across variety of auxiliary modalities, including depth, thermal, Event, and Lidar. Overall, the visual comparisons demonstrate the effectiveness our multimodal fusion framework in generating high- nigh-quality segmentation outputs.
D Others
D.1
Social Impact Analysis
Our proposed multimodal fusion method deploy artificial intelligence widely in resource constrained environments such as agricultural robots or low-power edge devices, potentially reducing computational costs. However, the dependence the pre training RGB backbone may inherit the deviation of training data, and the simplified architecture needs strict security verification before being deployed to key applications such as autonomous vehicle. We encourage responsible use and conduct additional fairness and robustness testing.
D.2 Limitation
Our current work is confined to the image multimodal domain and requires strictly aligned RGB- auxiliary data pairs. The exploration into other multimodal scenarios, such as weakly supervised or unpaired learning tasks is still limited. Addressing these limitations will become the basis of future research.
28


Reviews:
[{"review_id": "MB6X6f57hY", "reviewer": "Reviewer_DbRo", "summary": "The paper introduces MixPrompt, an efficient method for integrating auxiliary modalities with RGB images for semantic segmentation through prompt tuning. MixPrompt utilizes a frozen backbone that extracts features exclusively from the RGB input. At each stage of the backbone, a lightweight prompter module is employed to effectively fuse RGB features with those from the auxiliary modality.\nA key insight of the paper lies in the initialization of the prompt derived from the auxiliary modality. This initial prompt is merged with the RGB features *before* the first backbone block. To ensure a consistent alignment, the prompt is generated by processing the auxiliary input through the early layers of a pretrained RGB model.\nThe proposed method is evaluated on four datasets: NYU Depth V2, SUN-RGBD, MFNet, and DELIVER. MixPrompt consistently outperforms prior approaches while maintaining a significantly smaller parameter count.", "questions": "See the questions in the Strengths And Weaknesses section.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* The paper introduces a **novel and efficient fusion strategy** for combining RGB images with auxiliary modalities (e.g., depth, thermal, event, LiDAR), requiring only a minimal number of learnable parameters.\n\n* The design of the **initial prompt generation** mechanism offers an elegant and principled approach to the early fusion of modalities, ensuring meaningful feature alignment from the start of the network.\n\n* The proposed method achieves **competitive or superior performance** on multiple benchmarks while maintaining a very small parameter footprint.\n\n* The paper is **well written, clear, and easy to follow**, which facilitates understanding of the core ideas and implementation details.\n\n---\n\n**Weaknesses**\n\n* **Related Works Section.** While the related work is included in the supplementary material, placing it outside the main paper is unfair in comparison to other submissions that incorporate it within the main text, often sacrificing space for methodology and experiments. I recommend including at least a condensed version of the related work section in the main paper to ensure a fair and comprehensive presentation.\n\n* **Merging Strategy in the Mixed Prompting Module.** The two prompts ($h\\_{i-1}$ and $e\\_{i-1}$) are currently merged via simple summation. Have alternative merging strategies, such as concatenation, learnable weighted summation, attention-based fusion, or other parameter-efficient mechanisms, been explored? Investigating or at least discussing these alternatives could strengthen the contribution and potential generalizability of the proposed module.", "was_revised": true, "final_justification": "After the rebuttal, all of my original concerns were successfully addressed, so I decided to increase my rating.", "rebuttal": "**We sincerely appreciate reviewer DbRo recognition of the novelty and efficiency of our proposed fusion strategy, elegant prompt generation design, and strong performance with minimal parameters**. We provide our feedbacks as follows regarding your valuable suggestions.\n\n------\n\n> **Q1: Related Works Section. While the related work is included in the supplementary material, placing it outside the main paper is unfair in comparison to other submissions that incorporate it within the main text, often sacrificing space for methodology and experiments. I recommend including at least a condensed version of the related work section in the main paper to ensure a fair and comprehensive presentation.**\n\n- We agree with your point regarding fairness and presentation. **To address this concern, we promise to include a condensed version of the related works section in the main body of the paper in the camera-ready version, while retaining the full, detailed version in the supplementary material**. This revision will not affect other parts of the main text and help ensure a fair and comprehensive presentation of our work.\n- Specifically, condensed version will briefly cover two parts.\n  - **Semantic segmentation**. It summarizes the motivation for introducing multimodal inputs (e.g., depth, thermal) to address the limitations of RGB-only methods, reviews the two main existing research directions aboat multimodal semantic segmentation, and concludes the current challenges in this field.\n  - **Multimodal prompting**. It outlines how visual prompts, together with textual prompts, enhance cross-modal reasoning in vision-language models, and notes the limitations of early applications in semantic segmentation.\n\n------\n\n> **Q2: Merging Strategy in the Mixed Prompting Module. The two prompts ($h\\_{i-1}$ and $e\\_{i-1}$) are currently merged via simple summation. Have alternative merging strategies, such as concatenation, learnable weighted summation, attention-based fusion, or other parameter-efficient mechanisms, been explored? Investigating or at least discussing these alternatives could strengthen the contribution and potential generalizability of the proposed module.**\n- We adopt a simple summation of the two prompt embeddings ($h\\_{i-1}$ and $e\\_{i-1}$), primarily due to its parameter-efficiency and empirical effectiveness in our preliminary experiments. While our current approach is effective, we fully recognize the potential of alternative fusion strategies—such as concatenation followed by a projection layer, learnable weighted summation via gating, or attention-based fusion—to enable more expressive cross-modal interactions, as the reviewer rightly pointed out. However, these methods typically introduce additional parameters and computational overhead, potentially undermining the objective of preserving a compact and efficient prompting module, while also increasing the complexity of training.\n- To futher investigate the effectiveness of these fusion methods, we conducted ablation studies on different prompt fusion strategies using the NYUDepth dataset. The quantitative results are shown in the table below, and the implementation details of each variant are described as follows.\n  - **Concatenation.** Concatenate $h\\_{i-1}$ and $e\\_{i-1}$ along the module dimension, and apply a linear transformation to fuse the combined representation back into the original embedding space.\n  - **Learnable Weighted Summation.** Introduce a learnable gating vector $g \\in \\mathbb{R}^C$ and compute the fused prompt as $\\sigma(g) \\cdot h\\_{i-1} + (1 - \\sigma(g)) \\cdot e\\_{i-1}$, where $\\sigma$, where $\\sigma$ denotes the sigmoid activation function. The gate $g$ is initialized as a trainable parameter and applied channel-wise to adaptively modulate the contribution of each modality.\n  - **Attention Fusion.** The fused prompt is obtained by computing cross-modal attention between $e\\_{i-1}$ and $h\\_{i-1}$: $\\mathrm{Attn}(Q,K,V)=\\mathrm{softmax}\\left(\\frac{QK\\^{\\intercal}}{\\sqrt{d}}\\right)V$.\n\n  *Table 1: Comparison of different prompt fusion strategies in the Mixed Prompting Module*\n\n  |Fusion Method|Trainable Params|FLOPs|mIoU (%)|\n  |:-:|:-:|:-:|:-:|\n  |Summation|5.74M| 109.01G |**60.1**|\n  |Concatenation|5.83M| 109.09G|59.4|\n  |Learnable Weighted Sum|5.74M|109.01G|59.5|\n  |Attention Fusion|5.84M|109.09G|57.0|\n\n- Overall, our results on NYUDepth indicate that simple summation achieves the best performance among the evaluated strategies, highlighting its effectiveness as a lightweight and robust fusion mechanism. Although concatenation and learnable weighted summation offer more flexible interactions between the two prompt streams, they do not yield noticeable performance improvements in our setting. One possible explanation is that the two prompts already provide sufficiently complementary information, and more complex merging may introduce redundancy or disrupt this balance. In contrast, attention fusion not only degrades performance, but also incurs substantial memory overhead. This is primarily due to the pairwise similarity computation between all token positions, which generates a large $N \\times N$ attention map, where $N$ corresponds to the number of spatial positions in the feature map. For high-resolution inputs, $N$ can be large, making this step particularly memory-intensive.\n- These findings collectively support our use of simple summation as a practical and efficient fusion strategy in the mixed prompting module. We appreciate your suggestion and believe these findings provide a useful reference for future work exploring more adaptive fusion mechanisms in multimodal prompting."}, {"review_id": "RJWUppnZtt", "reviewer": "Reviewer_cD6A", "summary": "This paper presents MixPrompt, a prompting-based framework for multimodal semantic segmentation that integrates auxiliary modalities into a pretrained RGB segmentation model. The method introduces a lightweight prompting module and a multi-subspace alignment strategy to effectively adapt the model to multimodal inputs without retraining the entire backbone. The proposed approach is evaluated on multiple benchmark datasets, demonstrating competitive performance in multimodal segmentation tasks.", "questions": "- Given that multimodal semantic segmentation (e.g., RGB-Thermal, RGB-Depth) is a mature area, what do you see as the practical advantage of MixPrompt over prior methods? Does it offer clear benefits in efficiency, robustness, or generalizability that would motivate its adoption?\n- The paper claims that multi-subspace prompting enhances information utilization. Can you provide empirical or theoretical support for this claim? \n- You performed ablation studies on a single dataset to select the rank downscale ratio and the number of subspaces for prompt mixing. Can you clarify whether the selected values (e.g., 4) generalize well across other datasets, or whether they require dataset-specific tuning? This would help assess the robustness and usability of your method in broader settings.\n- The reported performance gains are relatively small (e.g., 0.5–1.0% mIoU in some cases), and the visual results in Figure 5 do not clearly indicate a qualitative advantage. Could you provide further evidence to justify the practical impact of MixPrompt?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper proposes a simple yet effective prompting-based approach that can be efficiently integrated into existing pretrained RGB segmentation models.\n- The method demonstrates strong performance across multiple benchmark datasets for multimodal semantic segmentation, validating its general applicability.\n- The framework offers a computationally efficient solution that avoids retraining large backbones, which is especially appealing for real-world deployment scenarios.\n\n**Weaknesses**\n- The core concept of using a pretrained RGB model while injecting auxiliary modalities through prompts was already adopted in prior work such as [14]. As a result, the originality of MixPrompt is limited to the specific design of its prompting module.\n- Multimodal semantic segmentation, particularly in RGB-Thermal or RGB-Depth settings, is a well-established field with strong existing baselines. The paper does not clearly differentiate MixPrompt from prior methods in terms of either methodology or practical impact.\n- The paper does not sufficiently clarify why the proposed method works. For instance, the effectiveness of multi-subspace prompting is attributed to improved information exploitation through multiple down- and up-projection pairs, but this claim lacks theoretical justification.\n- In the ablation study, key design choices, such as setting the rank downscale ratio and the number of subspaces for prompt mixing to four, are not well motivated. It remains unclear whether these hyperparameters are dataset-specific or reflect more general design principles.\n- In several experiments, the performance improvements over state-of-the-art methods are relatively modest (e.g., 0.5–1.0% mIoU), raising concerns about the practical significance of the proposed approach. Additionally, the qualitative results presented in Figure 5 do not clearly demonstrate a noticeable visual improvement, making it difficult to assess the perceptual advantage of MixPrompt.", "was_revised": true, "final_justification": "The rebuttal provided additional experiments and clarifications, addressing my questions on efficiency, robustness, and generalizability. I recognize the practical value of the lightweight, flexible design. However, my concern about the limited conceptual novelty, especially the incremental differences from prior prompting-based methods such as DPLNet, remains. Given the technical soundness and broad applicability, I maintain a borderline rating.", "rebuttal": "**We are pleased that the reviewer cD6A found our method to be simple yet effective, easy to integrate, and highly generalizable, which makes it particularly appealing for real-world deployment scenarios**. Most of the concerns raised seem to stem from misunderstandings of our work, which we address point-by-point below.\n\n------\n\n>**Q1: Given that multimodal semantic segmentation (e.g., RGB-Thermal, RGB-Depth) is a mature area, what do you see as the practical advantage of MixPrompt over prior methods? Does it offer clear benefits in efficiency, robustness, or generalizability that would motivate its adoption?**\n- As noted in the *Introduction* and *Related Works*, compared to existing multimodal segmentation methods that often rely on modality-specific branches and heavy fusion modules, MixPrompt introduces clear advantages in terms of efficiency, robustness, and generalizability.\n  - **Efficiency.**  MixPrompt achieves superior efficiency with only **5.7M trainable parameters** required to adapt to downstream tasks, which is significantly lower than prior methods such as CMX-B5 (181.1M), DFormer (39.0M), and DPLNet (7.15M). Despite its lightweight nature, MixPrompt outperforms all of these methods in segmentation accuracy, demonstrating that our prompt-based design achieves a better trade-off between performance and parameter overhead.\n  - **Robustness.**  The DELIVER dataset includes diverse challenging conditions such as cloudy, foggy, nighttime, and rainy environments, as well as sensor degradations including Motion Blur (MB), Over-Exposure (OE), LiDAR-Jitter (LJ), Event Low-resolution (EL) and Under-Exposure (UE). As shown in **Table 3 of our paper**, under these adverse conditions and corner cases, MixPrompt consistently outperforms previous methods such as CMX and CMNeXt. In addition, on the MFNet dataset, which contains both daytime and nighttime scenes, DPLNet fails to surpass earlier baselines like CMX and CMNeXt under relatively mild environmental disturbances. In contrast, MixPrompt achieves state-of-the-art performance, highlighting its robustness in real-world multimodal scenarios.\n  - **(3) Generalizability.**  **As kindly noted in your review, “the method demonstrates state-of-the-art results across multiple benchmark datasets for multimodal semantic segmentation, validating its general applicability.”** Unlike prior methods (DPLNet) that are typically restricted to RGB-D or RGB-T settings, MixPrompt generalizes well to additional modalities such as radar and event data without requiring architectural modifications or retraining, thanks to its modular and flexible prompt design.\n- In summary, MixPrompt achieves better performance than prior work such as DPLNet in terms of efficiency, generality, and robustness, while being theoretically motivated and empirically validated under more challenging multimodal segmentation scenarios.\n\n------\n\n>**Q2: The paper claims that multi-subspace prompting enhances information utilization. Can you provide empirical or theoretical support for this claim?**\n- As illustrated in **Algorithm 1 of our supplementary material**, our approach decomposes the high-dimensional features into *n* subspaces and performs linear mixing independently within each. This subspace-wise interaction scheme encourages diverse and disentangled feature combinations, which is beneficial for multi-modal fusion. **From the perspective of information theory, such factorized subspace interactions can be interpreted as promoting conditional independence and reducing redundancy, thereby potentially increasing the mutual information between the resulting prompt and the fused input modalities**. \n- In addition, we present additional quantitative analyses with detailed tabular statistics to assess the diversity of the learned subspaces. Our backbone outputs four-stage multi‑scale features (d1–d4, high→low resolution). For each stage, we compute the average pairwise cosine similarity between projected RGB subspace features (−1 to 1, lower means greater diversity). \n- As reported in *Table 1*, increasing $n$ from 1 to 4 markedly lowers overall similarity (−0.0514→−0.1922), boosting structural diversity and yielding the best mIoU (60.1%). Further raising \\(n\\) to 8 reverses this trend (−0.1922→−0.0169), suggesting over‑fragmentation that reduces diversity and degrades performance. **These quantitative results validate the core motivation behind our multi-subspace design: by introducing structural diversity and encouraging disentangled prompt composition, it effectively improves segmentation outcomes**. \n\n  *Table 1: Average pairwise cosine similarity between subspace features across different numbers of subspaces ($n$) .*\n  |n|Avg.Sim (d1)|Avg.Sim (d2)|Avg.Sim (d3)|Avg.Sim (d4)|Avg.Sim (overall)|mIoU (%)|\n  |:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n  |1|N/A|N/A|N/A|N/A|N/A|59.3|\n  |2|-0.0338|-0.0355|-0.0508|-0.0853|-0.0514|59.8|\n  |4|-0.1434|-0.1715|-0.2031|-0.2506|-0.1922|60.1|\n  |8|-0.0135|-0.0046|-0.0175|-0.0321|-0.0169|59.6|\n\n- **We commit to including the above experiments along with more comprehensive visualizations in the final version of the paper to better support our method**.\n\n------\n\n>**Q3: You performed ablation studies on a single dataset to select the rank downscale ratio and the number of subspaces for prompt mixing. Can you clarify whether the selected values (e.g., 4) generalize well across other datasets, or whether they require dataset-specific tuning? This would help assess the robustness and usability of your method in broader settings.**\n- To evaluate the generalizability of our selected settings (e.g., a rank downscale ratio of 4 and 4 subspaces), we conducted additional ablation studies on multiple datasets beyond the NYU Depth v2 dataset originally reported. Specifically, we evaluated the effect of these parameters on MFNet and DELIVER datasets, which encompass *RGB-Thermal*, *RGB-LiDAR*, and *RGB-Event* modalities. These datasets represent diverse sensing conditions and semantic distributions, offering a broader testing ground for robustness.\n- Across the diverse datasets evaluated, our selected parameter configuration consistently demonstrated competitive performance and stable optimization behavior. These results indicate that our chosen values strike a favorable balance among *model capacity*, *generalization*, and *computational efficiency*, making them a practical default for a wide range of multimodal segmentation scenarios.\n\n  *Table 2: Ablation study on the Ratio (rank downscale ratio).*\n  |Dataset|Modal|Ratio=1|Ratio=2|Ratio=4|Ratio=8|\n  |:-:|:-:|:-:|:-:|:-:|:-:|\n  |NYU Depth v2|RGB-D|59.4|59.8|**60.1**|59.6|\n  |MFNet|RGB-T|58.8|59.2|**60.1**|59.7|\n  |DELIVER|RGB-L|57.6|58.0|**59.1**|58.4|\n  |DELIVER|RGB-E|57.2|57.8|**58.0**|57.8|\n\n  *Table 3:  Ablation study on the Num (number of subspaces for prompt mixing)*\n  |Dataset|Modal|Num=1|Num=2|Num=4|Num=8|\n  |:-:|:-:|:-:|:-:|:-:|:-:|\n  |NYU Depth v2|RGB-D|59.3|59.8|**60.1**|59.3|\n  |MFNet|RGB-T|58.4|58.9|**60.1**|59.5|\n  |DELIVER|RGB-L|58.3|58.5|**59.1**|58.8|\n  |DELIVER|RGB-E|57.1|57.6|**58.0**|57.5|\n\n- While slight tuning may still benefit extremely domain-shifted settings, our results demonstrate that the chosen configuration is **robust and transferable**, reducing the burden of per-dataset hyperparameter adjustment in practice.\n------\n\n>**Q4: The reported performance gains are relatively small (e.g., 0.5–1.0% mIoU in some cases), and the visual results in Figure 5 do not clearly indicate a qualitative advantage. Could you provide further evidence to justify the practical impact of MixPrompt?**\n\n- We would like to highlight that **MixPrompt is designed with parameter efficiency as a core motivation**: during training, *only the prompting module and the segmentation head are updated*, while the RGB backbone remains frozen. This significantly reduces the number of trainable parameters and computational cost. Despite this constraint, MixPrompt consistently improves performance across diverse datasets\n\n- While the improvements may appear moderate in absolute terms on some datasets, they are nevertheless significant when compared to prior works. As shown in the *Table 4*, CMNeXt improves over the RGB-only baseline by only 0.3 and 0.8 mIoU on RGB-Event and RGB-Lidar, respectively, while CMX even underperforms the unimodal baseline by 0.7 and 0.8 mIoU. These results highlight the inherent challenges in effective multimodal fusion. In contrast, our method achieves consistent gains of 0.8 and 1.9 mIoU over the RGB-only baseline, outperforming CMNeXt despite using **only 5.7M trainable parameters—less than 9.7% of those used in previous methods**. These results are especially encouraging, suggesting that our approach offers a more efficient and effective solution for resource-constrained scenarios.\n\n  *Table 4: Comparison of RGB-Event and RGB-Lidar segmentation performance on the DELIVER dataset.*\n  |Method|Modal|Trainable Params|mIoU|\n  |:-:|:-:|:-:|:-:|\n  |Segformer|RGB|25.8M|57.2|\n  |CMX|RGB-E|66.6M|56.5 (-0.7)|\n  |CMNeXt|RGB-E|58.7M|57.5 (+0.3)|\n  |**Ours**|**RGB-E**|**5.7M**|**58.0 (+0.8)**|\n  |CMX|RGB-L|66.6M|56.4 (-0.8)|\n  |CMNeXt|RGB-L|58.7M|58.0 (+0.8)|\n  |**Ours**|**RGB-L**|**5.7M**|**59.1 (+1.9)**|\n\n- Regarding Figure 5, we acknowledge that the overall scene-level outputs may appear similar at first glance. However, a closer inspection of the highlighted regions—already annotated in our submission version—reveals clear qualitative improvements over existing multimodal methods such as CMX and CMNeXt. Specifically, our method shows more accurate predictions in distant regions and for small, fine-grained structures, where prior approaches often produce fragmented or overly smoothed outputs.\n- Overall, we believe these quantitative and qualitative results jointly support the practical utility of MixPrompt, especially when considering the favorable trade-off between performance and parameter efficiency."}, {"review_id": "bipg7peuPu", "reviewer": "Reviewer_LKhh", "summary": "This paper proposes a MixPrompt framework to multi-modal semantic segmentation task, which adopts prompt tuning to integrate information from other modalities into the processing of RGB. To further improve the performance and efficiency, the authors propose Multi-subspace prompting method to project RGB features into multiple subspaces and mix them with the prompt features from the auxiliary branch. The authors conduct extensive experiments on NYU Depth V2, SUN-RGBD, MFNet, and DELIVER datasets to validate the effectiveness of the proposed method.", "questions": "1. Providing more visual observations of the multi-subspace distributions would better illustrate the effect of the proposed method.\n2. It would be better to clearly display and distinguish different experimental settings in the table.\n3. Can the authors demonstrate more experiments based on the ImageNet pre-trained backbone to verify the benefits of pre-training on ADE20K, even unfreeze all parameters?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed framework seems reasonable. The prompt learning method based on the pre-trained model has indeed greatly improved the performance and training efficiency on many tasks.\n2. The experimental results are good on several benchmarks.\n\nWeaknesses:\n1. The motivation for proposing Multi-subspace prompting is unclear. The authors did not use theoretical derivation or experimental observation to derive the reason why RGB or other modality features need to be projected to multiple subspaces, which requires a deeper analysis.\n2. Regarding the method, the method proposed in this paper is more like an incremental improvement of the prompt tuning method proposed by DPLNet, which constructs a multi subspace mapping layer based on DPLNet. Therefore, the contribution is somewhat insufficient.\n3. For the experiments, the method in this paper is based on the backbone pre-trained on ADE20K, which is relatively unfair to compare with the previous method pre-trained on ImageNet, like DFormer. I understand that different method paradigms have different starting points, but the author should clearly indicate it in the comparison table, which is more friendly to other readers.", "was_revised": true, "final_justification": "The author's response addressed most of the questions, and the method is effective. However, I am still concerned about the consistency comparison of the methods, and the incremental innovation. I would like to raise the rating to 'Borderline', but there is no such level, I raised it to 'Borderline accept'.", "rebuttal": "**We greatly appreciate reviewer LKhh acknowledgement of our method’s strong performance, efficient training, and comprehensive experimental validation**. At the same time, we highly appreciate the thoughtful suggestions offered and present comprehensive explanations and clarifications for each concern below.\n\n>**Q1: Providing more visual observations of the multi-subspace distributions would better illustrate the effect of the proposed method.**\n\n- While the rebuttal format restricts the inclusion of visualizations, **we conducted additional quantitative analysis to characterize the diversity among the learned subspaces through detailed tabular statistics**. Following the hierarchical encoder architecture of SegFormer, our backbone extracts four-stage multi-scale features with progressively lower spatial resolutions, denoted as d1–d4 (from highest to lowest resolution). For each stage, we compute the average pairwise cosine similarity between projected RGB subspace features, where cosine similarity values range from -1 (completely dissimilar) to 1 (identical). Lower (more negative) similarity values indicate greater diversity across subspaces. The Avg.Sim (overall) column summarizes the mean similarity across all four stages, providing an overall measure of inter-subspace diversity.\n- As reported in *Table 1*, we use the RGB-D inputs from the NYU Depth V2 test set to evaluate how varying the number of subspaces (n) affects the diversity of learned subspace features. Our analysis reveals that, within the same number of subspaces (n), similarity values become increasingly negative from high-resolution (d1) to low-resolution (d4) features. This is because high-resolution features predominantly capture modality-agnostic local patterns (such as edges and textures) which remain relatively consistent across subspaces, whereas lower-resolution features encode more abstract, global, and cross-modal semantics. Consequently, our multi-subspace prompt mixing more effectively disentangles these higher-level representations, resulting in greater divergence at coarser scales.\n- In addition, increasing n from 1 to 4 substantially reduces overall similarity (from -0.0514 to -0.1922), reflecting enhanced structural diversity and correlating with the best segmentation performance (mIoU of 60.1\\%). However, further increasing n to 8 causes the similarity magnitude to shrink (from -0.1922 to -0.0169), indicating over-fragmentation that diminishes subspace diversity and aligns with the observed performance degradation.\n\n  *Table 1: Average pairwise cosine similarity between RGB subspace features across different numbers of subspaces ($n$) and corresponding segmentation performance.*\n\n  |n|Avg.Sim (d1)|Avg.Sim (d2)|Avg.Sim (d3)|Avg.Sim (d4)|Avg.Sim (overall)|mIoU (%)|\n  |:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n  |1|N/A|N/A|N/A|N/A|N/A|59.3|\n  |2|-0.0338|-0.0355|-0.0508|-0.0853|-0.0514|59.8|\n  |4|-0.1434|-0.1715|-0.2031|-0.2506|-0.1922|60.1|\n  |8|-0.0135|-0.0046|-0.0175|-0.0321|-0.0169|59.6|\n\n- These quantitative results indirectly validate the core motivation behind our multi-subspace design: by introducing structural diversity and encouraging disentangled prompt composition, it effectively improves segmentation outcomes. We acknowledge that providing more visual observations of the multi-subspace distributions would offer an even clearer and more intuitive demonstration of the proposed method’s effectiveness. **Due to rebuttal format constraints, we hope this detailed statistical characterization serves as a compelling alternative and we plan to include comprehensive visualizations in the final version of the paper**.\n\n------\n\n>**Q2: It would be better to clearly display and distinguish different experimental settings in the table.**\n\n- Thank you for your valuable suggestion. Although we have provided detailed configuration information in the “Additional Experiment Details” section of the supplementary materials, we agree that it is not very intuitive for readers to quickly grasp. Therefore, we will provide a detailed and comprehensive listing of all relevant configurations. Due to space limitations, we present here a simplified example table as *Table 2*. **We commit to including in the final version a comprehensive and clearly organized table summarizing all experimental settings across datasets**. \n\n  *Table 2: Training settings sample table.*\n\n  |Dataset|input size|optimizer|learning rate|weight decay|batch size|epochs|optimizer momentum|warmup epochs|warmup schedule|learning rate schedule|\n  |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n  |SUN-RGBD|530×730|AdamW|5e-3|0.01|4|100|β1, β2=0.9, 0.999|5|linear|polynomial decay|\n\n------\n\n>**Q3: Can the authors demonstrate more experiments based on the ImageNet pre-trained backbone to verify the benefits of pre-training on ADE20K, even unfreeze all parameters?**\n- We agree that our method uses a backbone pre-trained on ADE20K, whereas some prior works such as DFormer are based on ImageNet pre-training. **To clarify, DFormer does not simply use ImageNet pre-trained weights. It begins by creating RGB-Depth image pairs derived from the ImageNet dataset through a detailed and involved preprocessing step, and then conducts an additional pre-training stage on these pairs, adding unnecessary complexity to the workflow**. Different methods have their own variations in how pretrained weights are loaded and utilized, leading to discrepancies in model initialization. We commit to fully considering these differences and clearly indicating them in the comparison tables and the final version of the paper for a more reader-friendly and transparent evaluation.\n- Following your suggestion, we conducted additional experiments using the ImageNet-pretrained backbone under both frozen and unfrozen settings. The results are summarized in the table below.\n\n  *Table 3: Comparison of different backbone pretraining settings.*\n\n  | Backbone Pretraining | Backbone Frozen | Trainable Params |   mIoU   |\n  |:-:|:-:|:-:|:-:|\n  |MiT-B5 (ImageNet)|✓|5.7M|50.4|\n  |MiT-B5 (ImageNet)|✗|87.2M|56.3|\n  |MiT-B5 (ADE20K)|✓|5.7M|**61.2**|\n\n- As observed, even when fully fine-tuning the ImageNet-pretrained backbone, the performance is still noticeably lower than using the ADE20K-pretrained parameter with a frozen backbone. This is likely because ImageNet pre-training focuses on image classification, which does not equip the model with rich segmentation-related semantic knowledge. Consequently, the prompt learning module struggles to extract effective task-specific information from such backbones. In contrast, the ADE20K-pretrained backbone already encodes strong segmentation features, enabling the prompt module to leverage more informative representations and thus achieve better performance.\n- We will include these experimental details and analyses explicitly in the final manuscript to address the reviewer’s concerns and ensure a fair and comprehensive comparison.\n\n------\n\n>**Additional Response about DPLNet**\n\n- We are grateful for the reviewer’s insightful comment on the relationship between our method and DPLNet. While both approaches are indeed inspired by prompt learning, they differ fundamentally in their design and capabilities across several key dimensions.\n  - **Prompt adaptation mechanism.** DPLNet adapts the query features in a frozen backbone by inserting a set of learnable tokens into the attention computation. However, such token-based adaptation has inherent limitations: (i) the adaptation happens indirectly via attention weights, which may weaken the prompt’s ability to inject explicit task- or modality-specific cues; (ii) the fixed set of tokens cannot fully capture the diverse and complementary information required across different modalities.  In contrast, our multi-subspace prompt mixing approach directly projects prompts into multiple learnable subspaces and fuses them without modifying the attention queries inside the backbone. This design better aligns with the original motivation of prompt-based architectures—decoupling modality adaptation from backbone internals—while enabling richer and more flexible cross-modal alignment.\n  - **Generalization capability.** DPLNet’s applicability is restricted to RGB-D and RGB-T settings, and its generalization is limited. Remarkably, on the MFNet dataset (RGB-T), DPLNet performs worse than CMNeXt (-0.6 mIoU), suggesting that its learned prompts may be overfitted to specific modality combinations rather than capturing modality-agnostic patterns.  In contrast, MixPrompt achieves state-of-the-art results across five different modality settings, including RGB-D, RGB-T, RGB-Radar, RGB-Event, and RGB-LiDAR, without requiring architectural modifications or retraining. This strong cross-modal performance indicates that our design is a step toward a more general and scalable multimodal segmentation framework.\n- In summary, the differences in prompt adaptation strategy and generalization capability make MixPrompt a distinct and more versatile formulation compared to DPLNet, addressing the limitations of token-based adaptation while demonstrating superior applicability across diverse multimodal scenarios."}, {"review_id": "tBiaaaPprI", "reviewer": "Reviewer_sm6J", "summary": "The paper proposes a framework to incorporate auxiliary modalities, such as depth or lidar, into a pretrained RGB segmentation model to enhance segmentation quality. To achieve better results, it introduces several techniques: initializing the integration module with early layers of a pretrained CNN backbone, and aligning features across multiple low-rank subspaces. Overall, the approach achieves SOTA results on several benchmark datasets.", "questions": "1. **Concerns about Paper’s Novelty**\n\n1.1\nThe concept of a framework that extracts features from depth and then integrates them using a prompt adapter has already been explored in prior work [1].\n\n1.2\nSimilar initialization techniques have been employed in previous studies to obtain depth features(e.g. making depth \"colorful\" then using frozen RGB backbone early layers [2]). While the paper conducts more proper research, its novelty appears to be incremental. Could you clarify what is fundamentally new in your method?\n\n2. **Missing Comparison**\n\nWhy is there no comparison with OmniVec 2 [3]? Please explain the rationale behind excluding this relevant work.\n\n3. **Parameter Count in Table 1**\n\nConsider adding a column in Table 1 that indicates the number of trainable parameters, especially since your method freezes the backbone and only trains a specific module to demonstrate your efficiency (I saw it later but it will be still valuable in the main table).\n\n4. **RGB-Only Baselines in Table 1**\n\nPlease consider including results from RGB-only models in Table 1 to highlight the performance gain of RGB-D fusion. While some modality comparisons are shown later, RGB-D results are missing.\n\n5. **Dataset Selection Justification**\n\nThe evaluation focuses on NYU Depth V2 and SUN RGB-D, but widely used datasets such as KITTI and ScanNet are not included. Please explain the rationale behind excluding these datasets from evaluation.\n\n**References**\n\n[1] Dong, Shaohua, Yunhe Feng, Qing Yang, Yan Huang, Dongfang Liu, and Heng Fan. \"Efficient multimodal semantic segmentation via dual-prompt learning.\" In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 14196-14203. IEEE, 2024.\n\n[2] Gopalapillai, Radhakrishnan, Deepa Gupta, Mohammed Zakariah, and Yousef Ajami Alotaibi. 2021. \"Convolution-Based Encoding of Depth Images for Transfer Learning in RGB-D Scene Classification\" Sensors 21, no. 23: 7950.\n\n[3] S. Srivastava and G. Sharma, \"OmniVec2 - A Novel Transformer Based Network for Large Scale Multimodal and Multitask Learning,\" 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2024, pp. 27402-27414", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1. The paper provides well done rationalization for every modification of proposed module, supported by thorough ablation studies.\n2. The MixPrompt achieves good performance on proposed benchmark datasets.\n3. The authros evaluate the framework across many modalities (e.g. depth, lidar, thermal, event), showing the generalizability of it over modalities.\n4. The paper is well-wrtitten and easy to follow. \n\n**Weaknesses:**\n1. Incremental novelty of the proposed framework and initialization strategy for auxiliary modalities contributions.", "was_revised": true, "final_justification": "The authors attempted to address my concerns, but they succeeded only partially.\nIn the first part, they demonstrated that they generalize previous prompt‑adapter approaches to other modalities.\nIn the second part, they have not convinced me that their contribution is non‑incremental.\nTherefore, I would have preferred to assign a score of 3.5, but since I cannot, I am raising it to 4.", "rebuttal": "**We appreciate reviewer sm6J recognition of our method’s strong performance and broad generalizability as well as the robustness of our experimental evaluation**. Meanwhile, we greatly value the constructive feedback provided and provide detailed clarifications and responses to each point below.\n>**Q1: Concerns about Paper’s Novelty.**\n\n> **Q1.1: The concept of a framework that extracts features from depth and then integrates them using a prompt adapter has already been explored in prior work [1].**\n- We acknowledge that the concept of extracting features from the depth modality and subsequently integrating them using a prompt-based adapter has indeed been explored in prior work, such as DPLNet. However, we note several important limitations in DPLNet design and applicability.\n- DPLNet has primarily been applied to RGB-D and RGB-T settings. While it demonstrates certain improvements on RGB-D datasets, its performance on the RGB-T dataset MFNet is suboptimal, even falling behind the previous state-of-the-art CMNeXt by 0.6 mIoU. This suggests that, **as you mentioned, “it extracts features from depth and then integrates them using a prompt adapter”**, **but such a design may not generalize well, especially when the auxiliary modality differs from depth**. This limitation indicates that DPLNet’s integration strategy is somewhat modality-specific—effective for depth but less adaptable to other auxiliary modalities. In contrast, our proposed method is modality-agnostic: it does not rely on depth-specific feature extraction but instead adopts a generalized prompt-guided fusion mechanism that has been validated across multiple modality combinations, including RGB-Depth, RGB-Thermal, RGB-LiDAR, and RGB-Event.\n\n> **Q1.2: Similar initialization techniques have been employed in previous studies to obtain depth features(e.g. making depth \"colorful\" then using frozen RGB backbone early layers [2]). While the paper conducts more proper research, its novelty appears to be incremental. Could you clarify what is fundamentally new in your method?**\n- We would like to clarify that the cited Convolution-base Encoding(CBE) method is specifically designed for a two-stream architecture with separate RGB and depth backbones, both of which are fully trainable. This encoding scheme is depth-specific and designed as a replacement for the more complex HHA encoding.\n- In contrast, our approach differs fundamentally in terms of architecture, training strategy, supported modalities, and extensibility. The key distinctions are summarized in *Table 1*.\n  - **Architecture.** We employ a single-stream frozen RGB backbone shared across all modalities, rather than separate RGB and depth streams. Modality-specific adaptation is achieved by injecting learnable prompts into the frozen backbone.\n  - **Training strategy.** As stated in Section 5.2 of the CBE paper, “**all experiments with RGBD CNN were carried out with all layers trainable and randomly initialized weights for the dense layers**.” In contrast, our backbone is completely frozen; only the prompt modules are trained, resulting in substantially higher parameter efficiency.\n  - **Generality.** CBE is depth-specific and cannot be directly extended to other modalities. Our method is modality-agnostic, supporting diverse modalities without redesigning the encoding scheme.\n\n  *Table 1: Comparison between the Convolution-Based Encoding(CBE) method and our MixPrompt.*\n\n  ||**CBE**|**Our MixPrompt**|\n  |:-:|:-:|:-:|\n  |**Task**|Multimodal Classification|Multimodal Semantic Segmentation|\n  |**Architecture**|Two-stream (RGB + depth backbone)| Single-stream (RGB backbone + prompt) |\n  |**Training strategy**|Full parameter training|Backbone freezing|\n  |**Modalities**|Depth-specific only|Depth, Thermal, Event, LiDAR|\n  |**Extensibility**|New modality requires new redesign|New modality requires no redesign|\n------\n\n>**Q2: Missing Comparison. Why is there no comparison with OmniVec 2 [3]? Please explain the rationale behind excluding this relevant work.**\n- We would like to clarify that OmniVec 2 is fundamentally designed as a general large-scale multimodal and multitask learning framework, whereas our work specifically targets lightweight multimodal semantic segmentation. \n- **OmniVec 2 requires substantial pre-requisite work, including the construction of a large-scale multimodal pretraining dataset and a multi-stage pretraining pipeline** to obtain a strong base model.  In contrast, our MixPrompt does not rely on such extensive pretraining; instead, it achieves competitive results through prompt-based adaptation with a frozen backbone, making it significantly more efficient in both computation and parameter usage.  For clarity, we summarize the major conceptual and practical differences in *Table 2*.  \n\n  *Table 2: Comparison between the reviewer-cited OmniVec 2 method and our MixPrompt.*\n  \n  ||**OmniVec 2**|**Our MixPrompt**|\n  |:-:|:-:|:-:|\n  |**Task**|General multimodal multitask|Multimodal Semantic Segmentation|\n  |**Architecture**|Two-stream (base BERT) + Transformer|Single-stream (RGB backbone + prompt)|\n  |**Training strategy**|Three-stage pretrain + finetune|Finetune with frozen backbone|\n  |**Trainable Parameters**|Large (value not reported)|small (5.7M)|\n  |**Computation cost**|Large (value not reported)|small (109.0G)|\n- Despite the above differences, we still report a quantitative comparison with OmniVec 2 and its predecessor OmniVec *[4]* on the NYU dataset.  Results show that our method, with significantly fewer parameters and much lower computational cost, surpasses OmniVec's finetuned performance on NYU.  While OmniVec 2 still achieves higher accuracy, it comes at the cost of extremely large-scale pretraining and computational demands, which are not aligned with the scope and design philosophy of our work. \n\n  *Table 3: Quantitative comparison on the NYU dataset. Pre: pretrained only; FT: fine‑tuned on NYU.*\n\n  |Method|Re‑Pretrain|mIoU (%)|\n  |:-:|:-:|:-:|\n  |OmniVec (Pre)|✔|58.6|\n  |OmniVec (FT)|✔|60.8|\n  |OmniVec 2 (FT)|✔|63.6|\n  |**MixPrompt (Ours)**|✘|61.2|\n------\n\n>**Q3: Parameter Count in Table 1. Consider adding a column in Table 1 that indicates the number of trainable parameters, especially since your method freezes the backbone and only trains a specific module to demonstrate your efficiency (I saw it later but it will be still valuable in the main table).**\n\n- We agree that explicitly reporting the number of trainable parameters in Table 1 would provide a clearer illustration of the efficiency of our method and we provide a sample table as follows due to space limitations. **We commit to revising Table 1 in the final version to include trainable parameter counts for all methods**.\n\n  *Table 4: Sample table.*\n\n  |Model|Backbone|Params(Trainable)|NYU: Input/FLOPs/mIoU|SUN: Input/FLOPs/mIoU|\n  |:-:|:-:|:-:|:-:|:-:|\n  |ACNet|ResNet50|116.6M(116.6M)|480×640/126.7G/48.3%|530×730/163.9G/48.1%|\n------\n>**Q4: RGB-Only Baselines in Table 1. Please consider including results from RGB-only models in Table 1 to highlight the performance gain of RGB-D fusion. While some modality comparisons are shown later, RGB-D results are missing.**\n\n- Following your advice, we have conducted additional experiments using RGB-only baselines. The results are summarized in *Table 5*.  **We will include these RGB-only results in Table 1 of the final version**.\n\n  *Table 5: RGB-only models on the RGB-D dataset.*\n\n  |Model|Backbone|Params|NYU: Input/FLOPs/mIoU|SUN: Input/FLOPs/mIoU|\n  |:-:|:-:|:-:|:-:|:-:|\n  |SegFormer|MiT-B4|62.44M|480×640/74.7G/52.3%|530×730/96.3G/49.1%|\n------\n\n>**Q5: Dataset Selection Justification. The evaluation focuses on NYU Depth V2 and SUN RGB-D, but widely used datasets such as KITTI and ScanNet are not included. Please explain the rationale behind excluding these datasets from evaluation.**\n\n- We would like to clarify that NYU Depth V2 and SUN RGB-D are the two most widely used benchmarks for RGB-D semantic segmentation, and **we follow prior works *[1,2,3,4,5,6,8]* in conducting evaluation on these datasets**.\n- We would like to further elaborate on our considerations and analysis regarding the datasets mentioned by the reviewer:\n  - **KITTI.**  KITTI Vision Benchmark *[7]* is primarily an object detection dataset. Although the official KITTI Semantic Segmentation Evaluation *[8]* (containing 200 training and 200 testing RGB images) was later released, the lack of auxiliary modalities has limited its use in multimodal semantic segmentation research.\n  - **ScanNet.** ScanNet *[9]* is a large-scale RGB-D video dataset containing over 2.5 million views, annotated with 3D camera poses, surface reconstructions, and instance-level semantic segmentations.  We acknowledge ScanNet as a valuable indoor scene benchmark and have submitted the required agreement form but have not yet received access approval. We are committed to including ScanNet evaluation results in the final paper.\n\n**Reference**\n\n*[1] Efficient multimodal semantic segmentation via dual-prompt learning. IROS, 2024.*\n\n*[2] Convolution-Based Encoding of Depth Images for Transfer Learning in RGB-D Scene Classification. Sensors, 2021*\n\n*[3] OmniVec2 - A Novel Transformer Based Network for Large Scale Multimodal and Multitask Learning. CVPR, 2024*\n\n*[4] Omnivec: Learning robust representations with cross modal sharing. WACV, 2024.*\n\n*[5] Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers. IEEE T-ITS, 2023*\n\n*[6] Delivering arbitrary-modal semantic segmentation. CVPR, 2023*\n\n*[7] Are we ready for autonomous driving? the kitti vision benchmark suite. CVPR, 2012*\n\n*[8] Augmented reality meets computer vision: Efficient data generation for urban driving scenes. IJCV, 2018*\n\n*[9] Scannet: Richly-annotated 3d reconstructions of indoor scenes. CVPR, 2017*"}]

Author rebuttal:

Meta-review: This paper proposes a novel efficient mixed prompting method for multimodal semantic segmentation. The authors conduct experiments on multiple benchmarks to validate the effectiveness of the proposed method. Initially, the reviewers are concerned on the motivation, method details, and experiments of this work. The authors provide an effective rebuttal that addresses most of the questions. The AC agrees with most reviewers that this work provides some interesting insights and design for efficient multimodal semantic segmentation. Considering all factors, the AC recommends accepting this paper. The authors should integrate their responses in the final version.
Decision: Accept (poster)