# Encoder-Decoder Structure Fusing Depth Information for Outdoor Semantic Segmentation
## 1 Introduction
[19] believed that the depth information obtained from sensors is not always reliable, so they proposed the uncertain aware self-attention mechanism to achieve control from unreliable depth information to reliable depth information flow to solve the problem of RGB-D semantic segmentation. [PERSON] et al. [20] proposed a new semantic segmentation solution named LDFNet by fusing brightness, depth and color information. However, current state-of-the-art methods [16; 17; 18; 19; 20] still have many problems: (a) relying on expensive depth sensors, such as Microsoft Kinect or 3D LiDAR, to obtain high-quality ground truth depth maps for RGB-D semantic segmentation [21]; (b) simply using depth maps as the fourth channel of RGB images without fully utilizing the complementarity of RGB and depth information (due to the significant differences in features between RGB images and depth maps, it is necessary to fuse depth maps encoded with RGB images [22]); and (c) there are problems such as losing multi-scale information features, which have a great impact on the segmentation accuracy of small objects. If small receptive fields are used to extract the features of small objects, it is difficult to extract global semantic information, and if larger receptive fields are used to extract the background information of images, the features of small targets will be lost. The deep learning method based on multiple scales can combine the deep semantic information and shallow representation information of small objects to solve the problem of feature and position information loss caused by the increase in network layers in small object semantic segmentation [23]. To this end, inspiration from multi-modal joint training can ensure the robustness of semantic segmentation models based on single-modal feature learning [24]. Thus, we improved the semantic segmentation method from the following perspective: a neural network model for image semantic segmentation fused with predicted depth information, which utilizes the multi-modal fusion of RGB and depth images to achieve image semantic segmentation to enhance scene understanding. Specifically, our method is divided into two stages. First, depth estimation of the input RGB images was performed using our previous related work [25; 26]. Second, after fusing the predicted depth information with the RGB images, FCNs with encoding and decoding were further used to implement semantic labeling of the outdoor scenes. The overall contribution of our research is summarized below:  
(1) The raw RGB image and the predicted depth image were fused into a four-channel RGB-D image using self-calibrating fusion architecture and then an encoding-decoding model was established to output the predicted semantic image;  
(2) We proposed a new feature pyramid attention structure, which integrates the fused information at multiple scales to obtain features with rich semantic information;  
(3) Our method was evaluated using the publicly available Cityscapes dataset [27] and achieved comparable performance to the ground truth depth map, performing even better than some competitive methods. Furthermore, our method was generalized according to two different forest scenes to demonstrate its effectiveness. The rest of this paper is arranged as follows: the image semantic segmentation network model and detailed implementation process are outlined in Section 2; the experimental comparison and analysis are shown in Section 3; and finally, the conclusions and recommendations for future works are given in Section 4.