# Encoder-Decoder Structure Fusing Depth Information for Outdoor Semantic Segmentation
## 2 Materials and Methods
### Semantic Segmentation Model
#### 2.1.1 The Encoder Module  
The backbone network of the encoder module was mainly composed of improved residual blocks, which improved the raw residual block and the preactivated residual block [29], as shown in Figure 2.  
Conv1\(\times\)1 and Conv3\(\times\)3 were the convolution operations with kernel sizes of 1 \(\times\) 1 and 3 \(\times\) 3, respectively. BN represents batch normalization (BN), and the rectified linear unit (ReLU) was the activation function. The raw residual block was added a ReLU function to the main path to return the negative signal to zero. However, in the early stage of convolutional neural network (CNN) training, there may have been more negative signals that may have had a negative impact on information propagation. To solve this problem, a preactivation residual block was proposed. The BN and ReLU functions were placed before the convolution operator, and the feature map was directly output after Conv1\(\times\)1, which led to a lack of nonlinear characteristics between the modules and limited the learning ability of the CNN. In addition, the raw residual and preactivation residual blocks were not added to be the BN function on the main path, which also means that the feature maps added channel-by-channel were not normalized, increasing the difficulty of CNN training. We proposed an improved residual network structure as the backbone network for semantic segmentation, which is mainly divided into three parts: the start block, middle block and end block, as shown in Figure 3.  
Figure 1: The overall structure of RGB–D image semantic segmentation neural networks. BN* indicates that the BN was not added to the first middle block because the BN function is already included in the auxiliary path of the start block. Maxpool (3, 2) is a 3 \(\times\) 3 max pooling layer with a stride of 2. For the nonlinear activation function, we used the parametric rectified linear unit (PReLU) [30] to replace ReLU. ResNet was the first choice for the semantic segmentation tasks [31; 32; 33; 34]. In the process of training ResNet, stochastic dropping of hidden nodes or connection layers (the most  
Figure 3: The residual block in the IResNet network. (**a**) Start block, (**b**) middle block and (**c**) end block. Figure 2: The raw residual block and preactivated residual block. (**a**) Raw residual block and (**b**) preactivated residual block. common is the dropout method [35]) did not affect the convergence of the algorithm, showing that ResNet had good redundancy. We used an improved ResNet, called IResNet, which contains the start, main and end stages. There are four main stages, including three, four, six and three blocks, respectively. Each main stage includes a start block and an end block, and the rest are middle blocks. The entire IResNet network only contains the activation function PReLU [30] in the main path of the end block in the main stage, which reduces the adverse effects of the nonlinear function in information propagation and makes use of the advantages of nonlinear mapping. At the same time, the network adds a regularization term to the main path of the end block to regularize the output features added channel-by-channel. The end block of each main stage is connected with the start block of the next stage. The auxiliary path of the start block plus the regularization term and the main path parameters are added channel-by-channel to normalize all the features. Adding regularization items and activation functions to the main path of the end block can stabilize the signal entering the next stage, which is in preparation for the next stage of feature processing. The network can effectively control the flow of information without increasing the network parameters and only changing the layout. When the scales of the input and output features in the start block do not match, a down-sampling operation needs to be added to the main path to maintain the same scale as adding by the channel. ResNet-50 [29] uses a 1 \(\times\) 1 convolution with a stride of 2 to adjust the scale of the feature map.