# Detail Guided Multilateral Segmentation Network for Real-Time Semantic Segmentation
## 3 Detail Guided Multilateral Segmentation Network
### Network Structure
#### 3.2.2 Detail Information Guidance Path  
There are multi-scale segmented objects in the image. With the deepening of the network layers, the detail information declines with the deepening of the model layers. Compared with the larger segmented objects, the loss of detail information of small objects is more severe with the increase of the model layers. In order to improve the segmentation accuracy of small and thin objects and produce fine segmentation results near the boundary, we introduce the detail information guidance path.  
In the Figure 3, the image gradient represents the image edge detail information extracted via the Gaussian Laplace operator, and a feature map of size 1/2 is obtained. The edge information is used to further guide the lightweight backbone network to perform feature learning on the detailed information of the segmented object, to further improve the segmentation accuracy of fine or small objects. Image Gradient gets a feature map of size 1/4 after the operation of a convolution layer with step size 2. After the feature map and the feature map of the same size of the lightweight backbone network are input to the detail feature supplement module (DFSM) at the same time, a feature containing rich bottom-level detail information is obtained, and then, the information is fused with the input of the lightweight backbone network to supplement the detail information lost when the network layers are deepened. Similar operations are also performed after the Conv_3 module and Conv_4 module to obtain feature maps with richer edge details, so as to improve the segmentation accuracy of the network for thin and small objects in the input image. The FAM module is formulated as follows:  
\[\left\{\begin{aligned} X_{c}^{h}(h)&=\frac{1}{W} \sum_{0\leq i<W}x_{c}(h,i)+Max\big{(}x_{c}(h)\big{)},\\ \\ X_{c}^{w}(h)&=\frac{1}{W}\sum_{0\leq j<H}x_{c}(j,w)+ Max\big{(}x_{c}(w)\big{)}\\ \\ f&=\delta\bigg{(}F_{1}\Big{(}\Big{[}X^{h},X^{w} \Big{]}\Big{)}\bigg{)}\end{aligned}\right. \tag{1}\]  
\[\left\{\begin{aligned} & f=\delta\bigg{(}F_{1}\Big{(}\Big{[}X^{h},X^{ w}\Big{]}\Big{)}\bigg{)}\\ \\ & f=\delta\bigg{(}F_{1}\Big{(}\Big{[}X^{h},X^{w}\Big{]}\Big{)} \bigg{)}\end{aligned}\right. \tag{2}\]\[\left\{\begin{array}{l}F^{h}=\sigma\Big{(}F_{h}\big{(}f^{h}\big{)}\Big{)},\\ F^{w}=\sigma\Big{(}F_{w}\big{(}f^{w}\big{)}\Big{)},\\ F^{C}=\sigma\big{(}X_{\mathbf{max}}^{\mathbf{c}}+X_{\mathbf{avg}}^{\mathbf{c}} \big{)}\end{array}\right. \tag{3}\]  
where \(\frac{1}{W}\sum\limits_{0\leq i<W}x_{c}(h,i)\) denotes the average pooling of the h-th row of the c-th channel of the input feature \(X\in\mathbb{R}^{C\times W\times H}\); \(\frac{1}{W}\sum\limits_{0\leq j<H}x_{c}(j,w)\) denotes the average pooling of the w-th column of the c-th channel. \(Max\big{(}x_{c}(h)\big{)}\) represents the max-pooling of the h-th row of the c-th channel of the input feature; \(Max\big{(}x_{c}(w)\big{)}\) refers to the max-pooling of the w-th column of the c-th channel of the input feature. Where \([\cdot,\cdot]\) represents the join operation of \(X_{c}^{h}(h)\in\mathbb{R}^{C\times W\times 1}\) and \(X_{c}^{w}(h)\in\mathbb{R}^{C\times 1\times H}\) that will be generated along the spatial dimension, \(\delta\) is a non-linear activation function, and \(F_{1}(\cdot)\), \(F_{h}(\cdot)\) and \(F_{w}(\cdot)\) are \(1\times 1\) convolution. \(X_{\mathbf{max}}^{\mathbf{c}}\) and \(X_{\mathbf{avg}}^{\mathbf{c}}\) represent maximum pooling and average pooling of input features, respectively. \(f^{h}\in\mathbb{R}^{C/r\times H\times 1}\) and \(f^{w}\in\mathbb{R}^{C/r\times 1\times W}\) divide \(f\) into two independent tensors along with the two-dimensional directions. The outputs \(F^{h}\), \(F^{w}\) and \(R^{C}\) are attention feature maps. The output of the FAM module is as follows:  
\[y_{c}(i,j)=F_{c}^{h}(i)\times F_{c}^{w}(j)\times x_{c}(i,j) \tag{4}\]  
\[Y=y_{c}\times F^{C} \tag{5}\]