PrimKD: Primary Modality Guided Multimodal Fusion for RGB-D Semantic Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 09:57:11

Abstract: The recent advancements in cross-modal transformers have demonstrated their superior performance in RGB-D segmentation tasks by effectively integrating information from both RGB and depth modalities. However, existing methods often overlook the varying levels of informative content present in each modality, treating them equally and using models of the same architecture. This oversight can potentially hinder segmentation performance, especially considering that RGB images typically contain significantly more information than depth images. To address this issue, we propose PrimKD, a knowledge distillation based approach that focuses on guided multimodal fusion, with an emphasis on leveraging the primary RGB modality. In our approach, we utilize a model trained exclusively on the RGB modality as the teacher, guiding the learning process of a student model that fuses both RGB and depth modalities.
To prioritize information from the primary RGB modality while leveraging the depth modality, we incorporate primary focused feature reconstruction and a selective alignment scheme. This integration enhances the overall freature fusion, resulting in improved segmentation results.
We evaluate our proposed method on the NYU Depth V2 and SUN-RGBD datasets, and the experimental results demonstrate the effectiveness of PrimKD. Specifically, our approach achieves mIoU scores of 57.8 and 52.5 on these two datasets, respectively, surpassing existing counterparts by 1.5 and 0.4 mIoU.

Main paper: 
 introduction: 
 Semantic segmentation has been a long-standing research topic in the field of computer vision due to its wide application in autonomous driving and intelligent transportation systems. Recent advancements in modular sensors have paved the way for collecting multimodal data, presenting a promising opportunity to achieve significantly improved segmentation accuracy. For instance, RGB-D data, which combines RGB color channel information with depth information, has seen widespread use in semantic segmentation. By Comparison between single modal based methods [26] , an existing multimodal fusion approach CMX [30] , and our proposed method on two datasets. Our primary modality guided fusion yields significant improvements in segmentation accuracy.integrating this additional depth information, RGB-D segmentation demonstrates significant improvements in distinguishing instances and context. While the introduction of an additional modality does indeed offer more information, effectively harnessing and leveraging this additional information still poses a significant challenge. Since data from different modalities depict the scene in distinct manners, they cannot be processed separately as with a single modality. Therefore, achieving better multimodal fusion is significant for attaining remarkable performance.Within contemporary research, dominant paradigms for information fusion in the context of multimodal segmentation can be categorized into two classes: static fusion approaches and interactive fusion approaches. In early static fusion methods, multimodal inputs are fused, after which a single network is employed to extract features from the fused inputs [3] . For example, Cao et al. [2] decomposes depth features into shape and base components, applies learnable weights to balance their importance, and performs convolution on the re-weighted combination to enhance segmentation accuracy. Alternatively, two backbones may perform feature extraction separately from the RGB and depth modalities, followed by fusion of these features for segmentation [4] . ACNet [9] combines RGB and depth features using attention complementary modules and a multibranch architecture, allowing for selective feature gathering and fusion while preserving original RGB-D features for improved semantic segmentation performance. While these static fusion methods have achieved satisfactory performance, their straightforward approach often lacks inter-modal interactions, potentially resulting in inferior performance compared to single-modal methods [21] .Interactive fusion approaches have been proposed to enhance multimodal fusion. These approaches involve the integration of interactive fusion modules that link the RGB and depth features extracted by two parallel pretrained backbones, resulting in improved performance. For instance, TokenFusion [22] and HiDANet [24] dynamically fuse representations from RGB and depth encoders, aggregating them in the decoder, thereby pushing the performance boundaries in RGB-D scene parsing applications. Zhang et al. [30] also introduced a unified fusion framework, CMX, for RGB-X semantic segmentation. This framework incorporates a cross-modal feature rectification module for feature calibration and a feature fusion module for long-range context exchange and feature mixing, achieving superior segmentation performance. However, most existing interactive fusion approaches utilize identical backbone architectures to extract features from both RGB and depth data for fusion, overlooking potential differences between these two modalities. As the two modalities contain information of varying degrees, they may contribute differently to the final prediction. The use of identical backbones may suppress the effect of the primary modality, which contributes more to segmentation, while exaggerating the influence of the secondary one. Therefore, specialized designs are necessary to balance the participation of RGB and depth information for improved segmentation results.To tackle this challenge, we introduce PrimKD, a knowledge distillation (KD)-based methodology, aimed at guiding multimodal fusion in RGB-D semantic segmentation, with a focus on leveraging the primary modality. Under the assumption of the varying importance between RGB and depth modalities in segmentation tasks, we employ a teacher model trained exclusively on data from the primary modality to guide the learning process of the fusion model. To enhance primary modality guided fusion, we employ both prediction-level and feature-level guidance strategies to attain superior results. Specifically, in addition to aligning final predictions, we explore direct reconstruction of intermediate features, integrating with an adaptive guidance scheme. In our experimental analyses, we empirically observe that employing an RGB data-trained teacher model leads to a notable enhancement in performance, thereby confirming the RGB data as the primary modality in RGB-D semantic segmentation. Extensive experiments conducted on two widely recognized datasets, namely NYU Depth V2 [19] and SUN-RGBD [20] , validate the effectiveness of our proposed method. Specifically, our PrimKD achieves mIoU scores of 57.8 and 52.5 on these two datasets, respectively, outperforming the counterpart by a significant margin. Our contributions can be summarized as follows:• We identify the discrepancy in the contributions of the RGB and depth modalities in RGB-D segmentation and propose a KD-based approach to enhance multimodal fusion, guided by the primary RGB modality, without altering the inference model architecture. • We consider both prediction-level and feature-level guidance mechanisms for primary modality-guided fusion and validate an effective configuration that yields satisfactory multimodal fusion results. • We conduct extensive experiments on two prominent RGB-D segmentation datasets to showcase the superiority of our proposed method, with ablation study demonstrating the effectiveness of each module within our framework. 
related works 2.1 rgb-d semantic segmentation.: 
 Semantic segmentation involves categorizing each pixel in an image into a specific class or object. Traditional approaches typically rely solely on RGB images for prediction. However, with the proliferation of depth sensors in recent years, the availability of depth data has increased significantly. This surge in data availability has led to a growing interest among researchers and practitioners in exploiting depth data to enhance RGB semantic segmentation. This emerging field is known as RGB-D semantic segmentation. RGB images primarily capture color and texture details, whereas depth images predominantly convey spatial positional information. Integrating these modalities can enrich the semantic information of the original RGB images, leading to enhanced task performance to a certain extent. However, the disparity in information between these modalities presents a significant challenge in achieving effective fusion. To bridge this gap, various methods have been proposed, which can be categorized into two directions: developing dedicated feature extraction architectures tailored for RGB and depth image data [1-4, 6, 17, 28] , and introducing innovative methods for feature alignment fusion [9, 18, 22, 23, 25, 31, 34, 35] .The first category of methods focuses on designing novel network architectures to better extract feature information from different modalities. For instance, Girdhar et al. [6] introduce a new Transformer architecture capable of jointly pre-training classification tasks across various modalities such as images, videos, and single-view 3D data, thereby achieving cross-modal semantic feature extraction. Zhang et al. [28] suggest incorporating depth images during pre-training to facilitate improved extraction of depth image features during fine-tuning.The second category of methods involves designing innovative feature fusion approaches to align features from different modalities, thereby improving the semantic information of the RGB modality. For instance, Wu et al. [25] introduces a transformer-based fusion strategy to model context more effectively at the level of long-range information dependencies. Zhang et al. [30] propose cross-modal feature calibration modules and feature fusion modules to calibrate features of the current modality in both spatial and channel dimensions. They also utilize a cross-attention mechanism to globally enhance features from both modalities.While existing methods have achieved notable performance enhancements through multimodal fusion, they often process information from different modalities using the same architecture. This can inadvertently lead to the suppression of the more informative modality, which we refer to as the primary modality, leading to a decrease in performance.Knowledge distillation (KD) is a technique used to transfer knowledge from one or more pretrained teacher models to a more compact student model. Depending on the type of information utilized for transferring, existing KD approaches can be broadly categorized into two groups: those that leverage the final predictions of the teacher and those that utilize intermediate features for distillation.The vanilla KD method, as introduced by Hinton et al. [8] , is the first to adopt the final predictions for KD. In this method, the output distribution of the student classifier is required to mimic that of the teacher classifier by minimizing the KL divergence between the two distributions. Following the prior work, several enhancements to prediction-based KD have been proposed. Zhao et al. [32] introduced to decouple the original KL divergence into two separate parts and adjust the impact of these parts using hyperparameters. Huang et al. [10] focused on scenarios with large differences in model capacity between the teacher and student models and designed a Pearson correlation-based loss function. These improvements have demonstrated significant performance enhancements over the vanilla KD method.In addition to using final predictions, intermediate features can also be leveraged for knowledge transfer. Romero et al. [15] proposed a method to directly align features of the teacher and the student using an additional convolutional layer for channel alignment. Apart from this straightforward design, various feature distillation approaches have been explored. For instance, attention maps [29] , solution flow [5] , and sample-level feature manifold representations [13, 14] are also viable knowledge formats. More recently, inspired by the success of masked image modeling [7] , approaches introducing this idea into the KD field have emerged [27] . Moreover, integrating intermediate feature-based methods with final prediction-based approaches enables a more comprehensive amalgamation of information, thereby augmenting model performance.Many existing KD approaches are tailored for model compression, prioritizing enhancement of the performance of lightweight student models. However, in this paper, we deviate from this norm by employing the teacher model as an external regularization factor to refine the multimodal fusion process of the student model. 
method: 
 In this section, we provide an overview of existing RGB-D segmentation approaches, noting that many of them utilize identical architectures to process both modalities, inadvertently suppressing the primary modality containing more useful information. To address this limitation, we propose a KD-based approach. Our method leverages the primary modality to guide the multimodal fusion process, aiming to enhance segmentation performance.Semantic segmentation plays a crucial role in computer vision by assigning each pixel in an image to its corresponding semantic category. While models trained on RGB information excel in differentiating between colors and textures, they often struggle to capture geometric information, making it challenging to distinguish instances and contexts with similar visual characteristics [33] . To address this limitation, several studies have introduced depth images to enhance model performance [30, 31] .Existing methods commonly utilize two separate backbone models to extract features from both RGB and depth modalities. Given an input pair of RGB image 𝒙 rgb ∈ R 3×𝐻 ×𝑊 and depth image 𝒙 depth ∈ R 𝐻 ×𝑊 , where 𝐻 and 𝑊 represent the height and width of the images, their features are extracted as follows:EQUATIONwhere M is the backbone model, with 𝜽 rgb and 𝜽 depth representing parameters of the corresponding backbones used for extracting features from the RGB and depth modalities, respectively. Subsequently, these features are fused together by a modal fusion module, which can be either statically or adaptively:EQUATIONFinally, a decoder module is employed to obtain the final prediction by taking 𝒇 rgbd as input:EQUATIONwhere D is the decoder module, x ∈ R 𝐻 ×𝑊 represents the corresponding segmentation result of 𝒙 rgb .Integrating representations from both RGB and depth modalities has been shown to enhance model performance. As depicted in Equation 1, many current methods employ uniform architectures for feature extraction from each modality independently, differing only in parameters.Since models with identical architectures share the same capacity, utilizing identical backbones implies treating both RGB and depth modalities uniformly. However, due to potential differences in information density between modalities, this approach may inadvertently suppress the primary modality containing richer information while exaggerating the secondary modality with less information, thus leading to suboptimal performance.To address this disparity, specialized designs tailored to each modality are necessary. However, customizing model architecture for each modality can be cumbersome. Therefore, we aim to explore a simpler approach to calibrate the two modalities without modifying the model architecture, ensuring compatibility with existing RGB-D segmentation frameworks.KD has been proven to be an effective approach for enhancing model performance by transferring knowledge from a pretrained teacher model to a student model. Drawing inspiration from its success, we aim to leverage a teacher model trained on the primary modality to guide the learning process of a fusion model incorporating both RGB and depth modalities. This approach allows us to achieve calibrated multimodal fusion.Figure 2 presents an overview of our proposed primary modality guided multimodal fusion framework, termed PrimKD. Compared with existing multimodal fusion approaches that involve only an individual model learning from RGB-D data, our PrimKD framework incorporates an additional teacher model trained on the primary modality, which inherently contains more useful information for the segmentation task compared to the other modality. Building upon existing approaches that fuse multimodal information at the feature level, we further enhance the learning of the student model by employing KD guided by the teacher model. This is motivated by the assumption that the fused modality should resemble the primary modality more closely, thereby leading to improved performance. Our experiments demonstrate that RGB serves as the primary modality in RGB-D semantic segmentation. Although the teacher and the student are trained with different data, they share the same output space. Therefore, we initially consider performing KD at the prediction level.The teacher model is trained on the primary modality, its prediction process can be formulated as:EQUATIONwhere subscript𝑇 is used to indicate the teacher model. Additionally, we adopt x𝑆 to represent the predictions of the student model, which is obtained following the process depicted in Section 3.1.Given that semantic segmentation involves classifying each pixel in the image to determine its corresponding class, we can apply KD at the prediction level, similar to traditional classification tasks. Specifically, the distillation loss is defined as following:EQUATIONwhere H KL denotes the KL divergence, and 𝜓 : R 𝐻 ×𝑊 → R 𝐻𝑊 represents an operation that flattens the segmentation result into a one-dimension vector.Since the teacher model is trained exclusively using the primary modality, its predictions rely entirely on information from this modality. By training the student model to mimic these predictions, the fusion process is encouraged to prioritize information from the primary modality, resulting in improved fusion results.Feature-level guidance. In addition to the final prediction, intermediate features have also been widely utilized in existing KD approaches. Therefore, we incorporate feature-level guidance to transfer information from the primary modality. However, unlike in many existing studies where the teacher and student share the same model architecture, in our proposed PrimKD framework, the teacher model follows a single-stream architecture, while the student model adopts a dual-stream architecture. Considering our aim to impart more knowledge to the student model regarding the primary modality, it is reasonable to infer that the primary feature can be derived from the fused feature. Consequently, KD is performed using intermediate features of the teacher model and fused features of the student model. A lightweight module is employed to utilize fused features for reconstructing primary features, with the reconstruction error serving as the loss function. To explore the most suitable design, we consider several reconstruction schemes as follows:Direct reconstruction. The simplest way to reconstruct the primary feature is to directly adopt a learnable module to take the fused features as input and output reconstructed features. This process can be formulated as:EQUATION)where R is the learnable reconstruction module, which can consist of only a single layer or a sequence of layers. We evaluate different designs of this module in our experiments.Adaptive guidance. Feature-level guidance involves comparing intermediate features from both the teacher and the student model. For semantic segmentation tasks, backbones typically consist of four stages, with each stage producing a feature map. These four feature maps of different scales are then utilized in the decoder module for prediction. Typically, reconstruction and comparison entail the utilization of four feature maps. However, in our PrimKD scenario, the teacher and the student are trained using disparate datasets, resulting in divergence within their respective feature maps. Consequently, directly applying feature-level guidance across all four feature maps may present challenges.To facilitate the knowledge transfer process, we explore two schemes. The first involves conducting feature-level guidance consistently on one of the four feature maps. The second scheme involves adaptive selection of features for reconstruction and comparison. Generally, this process can be formulated as follows: select rule({L feat,𝑖 |𝑖 ∈ {1, 2, 3, 4}}), (7) where L feat,𝑖 represents the feature-level guidance loss associated with the 𝑖-th feature map. In our experiments, we will assess the impact of varying fixed map choices and employing different adaptive selection rules, including maximum-and minimum-based selection.Discussion. Primary modality guided fusion mainly aims to utilize the primary modality trained teacher model as an external regularization to enhance the effectiveness of student learning of multimodal fusion. Particularly in feature-level guidance, the fused feature must encompass sufficient information from the primary modality for accurate reconstruction. Additionally, the segmentation target encourages the student to extract useful information from the secondary modality simultaneously. Ultimately, the student model is expected to achieve enhanced performance through a more balanced multimodal fusion.The training of our PrimKD framework involves two stages. In the first stage, we train a teacher model with a single-stream architecture on the primary modality, identified as the RGB modality in our experiments. Subsequently, in the second stage, the student model is trained with the guidance of the frozen teacher. The total loss function during training is defined as:EQUATIONwhere L seg represents the original RGB-D semantic segmentation loss, and 𝛼 and 𝛽 are hyperparameters used to balance the two distillation losses. The student model learns multimodal fusion strategy that taking more primary information into consideration. At inference time, the primary teacher model becomes unnecessary, resulting in no additional inference cost associated with our method. 
experiment: 
 To assess the effectiveness of our proposed PrimKD method, we conduct experiments using two popular RGB-D segmentation datasets. Additionally, we conduct ablation studies to verify the effectiveness of each module within our framework. To initiate, we present our experimental setup.4.1.1 Datasets. In our experiments, we utilize two commonly referenced RGB-D segmentation datasets: NYU Depth V2 [19] and SUN-RGBD [20] . These datasets are frequently employed in existing literature [28, 30] , and their details are outlined below: In both datasets, the depth images are converted into HHA format following [30] . The HHA format represents images with horizontal disparity, height above ground, and angle information.4.1.2 Models. Given our primary goal of enhancing the multimodal fusion capability of existing dual-stream RGB-D semantic segmentation architectures, we opt not to introduce a novel model architecture. Instead, we adopt the dual-stream architecture proposed by Zhang et al. [30] , which incorporates two MiT backbones [26] . In our experiments, we evaluate two different backbones: MiT-B2 and MiT-B4, both pre-trained on ImageNet-1K [16] .Our training process consists of two stages. In the first stage, we train the teacher model using only the primary modality. Subsequently, in the second stage, we train the student model using both modalities with KD. Both stages share the same optimization configurations.For the NYU Depth V2 dataset, we train models for 500 epochs with a batch size of 8. The initial learning rate is set to 6 × 10 -5 with a polynomial decay scheme. We employ AdamW [11] as the optimizer with a weight decay of 0.01. Data augmentation techniques including random flipping and random cropping are utilized. Experiments on the SUN-RGBD dataset follow a similar setting, with the only differences being the number of epochs, batch size, and initial learning rate, which are set to 300, 128, and 8 × 10 -5 , respectively. The hyperparameters 𝛼 and 𝛽 are set to 1 and 0.05, respectively, by default. All experiments are conducted using NVIDIA Tesla V100 GPUs. For the NYU Depth V2 dataset, we utilized 2 GPUs, while for the SUN-RGBD dataset, we employed 8 GPUs.For evaluation, we employ the mean Intersection over Union (mIoU) metric. While many existing works typically report results obtained under multiscale testing, we employ this technique solely for comparison with these works. Our ablation experiments, in contrast, are conducted without multiscale testing. [26] and the two-stream method CMX [30] , which do not utilize primary modality guidance.In our analysis, we hypothesize that there exists a discrepancy in the significance of the RGB and depth modalities for the segmentation task, with one modality potentially providing more crucial information and thus serving as the primary modality. To ascertain the primary modality, we conduct experiments using teacher models trained on either modality to guide the learning of the student. We then compare these results with a non-KD baseline.The comparison results are presented in Table 1 . Without adopting any teacher, the student model achieves a test mIoU of 56.1 on the NYU Depth V2 dataset. When the model pretrained solely on the depth modality is adopted as the teacher, the student shows only a marginal improvement in performance, with a 0.1 mIoU increase. In contrast, when the RGB modality is used for teacher training, the student achieves a mIoU of 57.5, significantly outperforming its counterparts. These results demonstrate that constraining the fused features to contain more information from RGB data is beneficial. As a result, RGB is identified as the primary modality, and it will serve as the primary modality in all our subsequent experiments.To evaluate the effectiveness of our proposed method, we compare it with several recently proposed approaches on the NYU Depth v2 and SUN-RGBD datasets. Baseline results are obtained from [28] . Since our method primarily focuses on enhancing multimodal fusion performance for two-stream architectures, we do not include single-stream methods for comparison.Table 2 presents the comparison results. On the NYU Depth v2 dataset, our PrimKD achieved an mIoU of 54.7 with the MiT-B2 backbone and 57.8 with the MiT-B4 backbone, surpassing the baseline CMX [30] without primary modality-guided multimodal fusion by 0.4 and 1.5 mIoU points, respectively. Similar performance improvements are observed on the SUN-RGBD dataset, with increases of 0.9 and 0.4 mIoU points using the MiT-B2 and MiT-B4 backbones, respectively. For clearer illustration of these improvements, we provide a bar plot in Figure 1 . Furthermore, our method only incorporates the primary modality-trained teacher model during training, without modifying the architecture of the student model. This indicates that the performance improvements are achieved without sacrificing inference efficiency.Visualization. To provide a qualitative comparison between our primary modality guided multimodal fusion method and existing approaches, we present visualization of several samples from NYU Depth v2 in Figure 3 . From the results, model trained using our method is able to achieve more accurate segmentation results, further demonstrating the effectiveness of our design.To further investigate the effectiveness of each module in our PrimKD framework, we conduct extensive ablation experiments. All ablation experiments are performed using the MiT-B4 backbone on the NYU Depth v2 dataset, with results reported without the use of multiscale testing. 4.4.1 Prediction-level guidance and feature-level guidance. We employ both prediction-level and feature-level guidance to direct the multimodal fusion process of the student model. To assess the effectiveness of each guidance method, we compare all possible combinations and present the corresponding results in Table 3 . From the results, the introduction of either prediction-level or feature-level guidance improves performance, with enhancements of 0.5 and 0.4 mIoU, respectively. Furthermore, combining both forms of guidance yields the highest mIoU of 57.5, demonstrating the effectiveness of both approaches. 4.4.4 Hyperparameters. In our framework, there are two hyperparameters corresponding to the loss weight of prediction-level and feature-level guidance, respectively. We compare different configurations of these two hyperparameters and present the results in Table 6 and Table 7 . From the results, the optimal combination of these two hyperparameters is 𝛼 equal to 1 and 𝛽 equal to 0.05. To delve deeper into the impact of our proposed primary modality guided fusion, we analyze features of the trained fusion model. Specifically, we compare feature similarity using centered kernel alignment (CKA) [12] and feature value distribution through histogram analysis.CKA analysis. To examine the feature similarity between dual modality trained models and single modality trained models, we utilize CKA as the measurement. CKA takes two feature maps as input and returns their similarity, ranging from 0 to 1, where a larger value indicates greater similarity.We adopt the MiT-B4 as the backbone model and evaluate using the test set of NYU Depth V2. Table 8 presents the results. When guided fusion is not used, the features of the model trained using CMX achieve similar similarity compared to the single-stream model trained by either RGB or depth modality. Additionally, as the model goes deeper, the similarity also increases. After incorporating our primary modality guided fusion, the RGB modality trained teacher model is used to modulate the fusion of the student. Although the similarity to RGB modalities remains similar, there is a noticeable decrease in similarity of the trained model compared to the depth model, especially at the early stages of the model. This demonstrates that modulating the beginning stages is more significant, which is consistent with the result in Table 4 .Feature value distribution analysis. We further compare the distribution of intermediate feature values and present the histogram plot in Figure 4 . The teacher model trained with the RGB modality extracts features with more concentrated values, while the CMX trained model without guidance presents scattered feature values. Equipped with our proposed method, the distribution of feature values shifts toward that of the teacher model, demonstrating that more information from the RGB modality is considered.  
conclusion: 
 Due to the recent advancements in depth sensors, the availability of depth images has significantly increased. Consequently, integrating these images with RGB images to enhance segmentation results has become increasingly appealing. Current methods often utilize two parallel backbones of the same architecture to extract features from each modality, followed by fusion modules for prediction. However, employing identical backbones overlooks the differences between these modalities, potentially resulting in suboptimal performance.To address this limitation, we introduce PrimKD, which employs a teacher model pretrained on the more informative modality to guide the multimodal fusion process of the student model under the KD framework. Both prediction-level and feature-level guidance are utilized in this approach. To enhance feature-level guidance, we further introduce adaptive guidance design. In our experiments, we identify that the RGB modality provides more information for the segmentation task, thus using it to train the teacher model. With our primary modality-guided multimodal fusion, the trained student model achieves an mIoU of 57.8 on the NYU Depth V2 dataset and 52.5 on the SUN-RGBD dataset, respectively, surpassing its counterparts by a significant margin while incurring no additional inference cost.  
 
 Appendix:  


Decision: The paper introduces an approach using knowledge distillation to enhance multimodal fusion in RGB-D semantic segmentation tasks. The core idea revolves around using a teacher model trained solely on RGB data to guide the fusion of RGB and depth data in a student model. This process aims to improve segmentation performance by ensuring that the fusion process prioritizes information from the RGB modality, which is identified as the primary modality containing more informative content for segmentation tasks.

After the rebuttal, the paper receives three borderline accept and one borderline reject. The authors are encouraged to pay further attention to the limitation improvement mentioned by reviewer sRSQ.

Reviews: review: paper_summary: paper_summary: The authors propose a new PrimKD method, which aims to leverage the knowledge learned from a model trained on the primary modality, to guide the learning process of a student model that fuses multiple modalities.
strength_weakness: strength_weakness: The authors discover that using identical backbones to extract information from multiply modalities will suppress the contribution of primary modality when multimodal fusion, and propose a KD-based method for enhencing the feature presentation of primary modality.
limitations: limitations: 1.	The determination of the primary modal requires experimental verification in advance.  Besides, the autor consider the RGB is the primary modal in all experiments. How to consider different RGB-D application scenarios where RGB modal maybe not the primary modal?
2.	The improvement of the proposed method to CMX with MiT-B2 is 0.3 instead of 0.4 in 4.3.
3.	The visualization results between proposed and comparison methods shown in Figure 3 need to be eleborated in detail.
4.	Ablation experiments with two parameters in 4.4.4 need to explain in detail how one parameter is set when adjusting the other parameter.
paper_summary: paper_summary: The authors utilize a model trained exclusively on the RGB modality as the teacher, guiding the learning process of a student model that fuses both RGB and depth modalities.  The experiments are conducted on on the NYU Depth V2 and SUN-RGBD datasets, and the experimental results show the effectiveness.
strength_weakness: strength_weakness: It is  a KD-based way to enhance multimodal fusion, guided by the primary RGB modality. It considers both prediction-level and feature-level guidance mechanisms. The struture is simple, but achieves good multimodal fusion results.
limitations: limitations: (1) The fusion module in the manuscript is not clearly described. 

(2) It is neccesary to discuss whether the fusion way can influence the effect of the KD learning.

(3) To be strict,  the mentioned adaptive guidance is not adaptive.
paper_summary: paper_summary: This paper proposes a knowledge distillation method, called PrimKD, for improving the performance of RGB-D semantic segmentation. PrimKD focuses on leveraging a primary modality (RGB) during knowledge transfer to guide the learning of a student model that fuses both RGB and depth data. The authors consider both prediction-level and feature-level guidance mechanisms for primary modality-guided fusion.
strength_weakness: strength_weakness: The paper explores a KD-based method to improve the performance of multimodal semantic segmentation. 

The experiment is conducted on two RGB-D datasets. The experiments show good results.
limitations: limitations: A key question is the technical novelty of PrimKD's knowledge distillation method. The paper needs to clearly explain how PrimKD's approach differs from existing work and what specific contributions it makes.

PrimKD utilizes an RGB-only model as the teacher. Exploring an alternative approach where a richer RGB-D model acts as the teacher could potentially improve student model performance.

The paper lacks details on the teacher and student model architectures. Additionally, discrepancies exist between PrimKD's parameter count and baseline methods like CMX. A comprehensive explanation of these architectural choices is necessary, particularly regarding their impact on both efficiency and performance.

The reported mIoU improvements achieved by PrimKD are modest compared to the baseline method without knowledge distillation. This raises concerns about PrimKD's overall effectiveness.

The evaluation focuses solely on segmentation performance. Including comparisons between different knowledge distillation methods would provide a broader context for PrimKD's effectiveness.
paper_summary: paper_summary: The paper introduces a novel approach leveraging knowledge distillation to enhance multimodal fusion in RGB-D semantic segmentation tasks. The core idea revolves around using a teacher model trained solely on RGB data to guide the fusion of RGB and depth data in a student model. This process aims to improve segmentation performance by ensuring that the fusion process prioritizes information from the RGB modality, which is identified as the primary modality containing more informative content for segmentation tasks.
strength_weakness: strength_weakness: - Novel Approach: The concept of using a knowledge distillation framework to prioritize the primary RGB modality in the fusion process is innovative. It addresses the common issue in existing fusion methods where equal weight is given to both RGB and depth modalities, which can dilute the more informative RGB signals.  
&nbsp;
- Empirical Validation: The authors provide extensive experimental results to validate the effectiveness of the PrimKD method. The improvements are demonstrated on two standard datasets, NYU Depth V2 and SUN-RGBD, with the proposed method outperforming existing approaches in terms of mIoU scores.  
&nbsp;
- Detailed Analysis: The paper includes a thorough ablation study that explores different aspects of the proposed method, such as the impact of prediction-level and feature-level guidance, the effectiveness of adaptive guidance, and the architecture of the alignment module. This comprehensive analysis strengthens the credibility of the results.  
&nbsp;
- No Additional Inference Cost: It is advantageous that the primary modality-guided fusion introduced by PrimKD does not require the teacher model at inference time, which means there are no additional computational costs during the deployment phase.
limitations: limitations: - Complexity in Training: While the inference phase does not incur additional costs, the training phase could be significantly more complex and resource-intensive due to the dual-model setup and the need for knowledge distillation.  
&nbsp;
- Lack of Broader Validation: The evaluation primarily focuses on RGB-D segmentation with specific datasets. Additional experiments across different modalities or segmentation tasks could help generalize the findings and prove the method’s applicability in other contexts.  
&nbsp;
- Potential Overfitting: The method’s reliance on the primary RGB modality, while beneficial as shown, might also lead to overfitting to features predominant in RGB data, potentially ignoring useful cues from the depth data in some scenarios.  
&nbsp;
- Limited Discussion on Failure Cases: The paper could benefit from a more detailed discussion on scenarios where PrimKD does not perform as expected, which would provide a deeper understanding of the method's limitations and scope.