## 3 System and Methodology
### Semantic Segmentation  
As a significant component of the proposed visual system, the semantic segmentation algorithm is a system comprising multiple networks. Figure 4 illustrates this framework. The segmentation system distinguishes detected objects from the background at the pixel level to facilitate subsequent pose estimation. The black plate with three round marks in addition to the object is a calibration plate used to generate the SIAT datasets and works in the same way in all figures in the following text.  
Figure 3: The process of object detection. The original image is cut into several 300 \(\times\) 300 sub-images, and then the SSD algorithm is applied to each sub-image, respectively. Finally, these bounding boxes are mapped to the original image.  
During tracking, the RGB-D camera acted as a depth sensor to measure the distance from the UAV to the server. Semantic segmentation began when the UAV approached the target within 1 m. We introduced a novel segmentation algorithm based on an encoder-decoder structure to obtain a precise segmentation mask. Our semantic segmentation network was inspired by the classical segmentation network U-Net [28] and consisted of an encoder network and a decoder network. The convolutional and max-pooling layers compose the encoder network to extract features at different scales, whereas the decoder layer applies convolution and upsampling to output the pixel-level classification results. In addition, because the features of the encoder and decoder layers contain complementary information from different layers in the convolutional network, a skip network is applied to fuse this information to improve the network performance.  
Compared with the prior network, the innovation of our network is reflected in the following two points:  
* To better utilize the geometric information of the environment, in addition to the RGB images, we input the depth images into the segmentation network. Therefore, our encoder network consists of two branches, with one branch extracting color features and the other extracting geometric features. Next, the color and geometric features of the Max-Pooling layer will be concatenated at each downsampling stage to reinforce the expressiveness of the features.
* We introduced the PPM [29] to aggregate contextual information at different scales to reduce computation. In addition, to strengthen the recognition ability of each stage, we added RRB [48] to refine the feature map, the details of which are shown in Figure 4.  
In the training stage, cross-entropy is used as the loss function of the segmentation network and is expressed as follows:  
\[L=-\sum_{i=1}^{H}\sum_{j=1}^{W}y_{i,j}^{{}^{\prime}}\log y_{i,j}, \tag{1}\]  
where \(y_{i,j}\in\{1,2,\dots,C\}\) is the ground truth of each pixel and \(y_{i,j}^{{}^{\prime}}\) is the prediction. \(H\) and \(W\) denote the height and width of the image, respectively.  
In summary, the proposed semantic segmentation network can achieve pixel-level classification by fusing color and depth information, which significantly improves the accuracy with a slight increase in the computational cost. However, the segmentation  
Figure 4: The overall framework of the segmentation network. The encoder network uses convolutional layers and MaxPooling layers to extract features of different scales from RGB and depth images. The decoder layer uses convolution and Upsampling to output pixel-level classification results.  
network performs unsatisfactorily on objects with similar or the same textures but diverse geometries. Therefore, we leveraged a classification network to address this problem.