# DBCAN: DFormer-Based Cross-Attention Network for RGB Depth Semantic Segmentation
###### Abstract  
Existing RGB-depth semantic segmentation methods primarily rely on symmetric two-stream Convolutional Neural Networks (CNNs) to extract RGB and spatial features separately. However, these architectures have limitations in incorporating spatial features and efficiently fusing RGB and depth information. In this study, we propose a novel architecture called the DFormer-Based Cross-Attention Network (DBCAN), which utilizes DFormer as an encoder for feature extraction and integrates several modifications to address these challenges. While DFormer is leveraged for its strong feature extraction capabilities, our modifications in the decoder focus on improving cross-modal fusion and spatial feature incorporation. We introduce three modules in the decoding process: the Object-Region Generated Module (ORGM), the Feature-Region Relation Module (FRRM), and the Spatial-Semantic Fusion Module (SSFM), which enhance feature interaction and segmentation accuracy. Experimental results on the NYUDepthv2 and SUN-RGBD datasets demonstrate that DBCAN achieves state-of-the-art performance, highlighting the effectiveness of our architectural enhancements in overcoming the limitations of existing models. deep learning; attention mechanism; RGB-D semantic segmentation +
Footnote †: journal:  
To ensure that the model can also obtain sufficient spatial features during the decoding process and achieve accurate pixel-level boundary segmentation, we propose an RGB-D semantic segmentation model named DBCAN, which uses the encoder of DFormer to extract features, but with an innovative decoder. Although DFormer effectively fuses and separates RGB and depth features during the encoding process, the lightweight ham-head decoder [11], used for preliminary RGB feature decoding, is shallow and neglects depth information, limiting its ability to capture spatial features. To address this, we designed a series of modules that work together to ensure proper integration of RGB and depth features, improve spatial feature representation, and achieve more accurate segmentation results. First, we introduce the Object-Region Generated Module (ORGM), which processes only RGB features and performs coarse classification through a deep supervision mechanism. ORGM enhances the semantic representation of the RGB features, allowing the model to better utilize the RGB information in subsequent stages for further processing and fusion. Next, we propose the Feature-Region Relation Module (FRRM), which separately fuses RGB and depth features with classification features using attention mechanisms. This enables the parallel processing of both modalities, allowing the model to focus on regions that are more effectively classified. By enhancing the expressive power of both RGB and depth features, FRRM improves the model's ability to capture important scene details. Finally, the Spatial-Semantic Fusion Module (SSFM) refines the integration of RGB and depth features by adjusting their channel weights. SSFM distinguishes the importance of different features, selectively enhancing relevant ones while suppressing less important information, leading to a unified and optimized feature representation. Extensive experiments on the NYUDepthv2 [12] and SUN-RGBD [13] datasets demonstrate that DBCAN achieves state-of-the-art performance in terms of mIoU and prediction accuracy. In summary, our main contributions can be summarized as follows:  
* A novel DFormer-based network is proposed for RGB-D semantic segmentation tasks. It leverages the strengths of CNNs in capturing local features and the attention mechanisms for modeling global dependencies, enabling the network to effectively fuse and separate rich features from the original RGB-D images. * The Feature-Region Relation Module is designed to enhance the attention of multi-modal features (RGB and depth). This method can improve the accuracy of semantic and spatial information. * The Spatial-Semantic Fusion Module is designed to adjust the weights of each feature point and to fuse RGB and depth features.