# Deep Learning-Based Automated Background Removal for Structural Exterior Image Stitching
## 2 Structural Exterior Image Stitching through Automated Background Removal
### Automated Background Removal Using Deep Learning-Based Depth Estimation  
Figure 2 shows the deep learning-based automated background removal process. The key idea of this step is under assumption that structural ROI, which is obtained through the UAV's close-up scanning, is much closer than the background objects from the UAV-embedded digital camera within FOV. Thus, a deep learning-based depth estimation network, called Monodepth2, which is based on a U-net architecture, is employed in this study [18]. Monodepth2 is trained in a self-supervised manner by exploiting spatial geometry constraints. Monodepth2 utilizes a full-resolution multi-scale sampling method for reducing visual artifacts and an auto masking loss to ignore training pixels that violate camera motion assumption. This network can rapidly estimate depth value using only monocular RGB images, thus it is suitable for high-resolution image processing. The effectiveness of Monodepth2 was validated by comparing with 28 other depth estimation models using measurement metrics [18]. The employed model was implemented in PyTorch and trained for 20 epochs using an Adam optimizer with a batch size of 12. The learning rate of \(10^{-4}\) is used for the first 15 epochs which is then dropped to \(10^{-5}\) for the remainder. The smoothness term is set to 0.001. The KITTI dataset is used for pre-training, and 10% of the dataset is used as a validation set.  
Once the depth values were estimated with respect to each pixel on the background, including FOV images acquired from the edge of the target structure, the depth image was obtained as shown in Figure 2. Subsequently, the ROI boundary was extracted by using depth difference. However, precise ROI boundary extraction is often difficult due to undesired noise components on the depth images. Thus, post image processing was necessary. On the depth images in Figure 2, brighter pixels indicate the closer working  
Figure 1: Overview of structural exterior map establishment through deep learning-based automated background removal and optimal image selection: FOV and ROI denote the field of view and region of interest, respectively.  
distance from the digital camera. The ROI boundaries within the depth image can be extracted by using the magnitude of depth gradients (\(G\)), which is given by:  
\[G = \sqrt{\left(\frac{\partial I^{\prime}}{\partial x}\right)^{2}+\left( \frac{\partial I^{\prime}}{\partial y}\right)^{2}} \tag{1}\]  
where \(I^{\prime}\) is the depth image corresponding to the FOV image, and \(\frac{\partial I^{\prime}}{\partial x}\) and \(\frac{\partial I^{\prime}}{\partial y}\) are the depth gradients of \(I^{\prime}\) along the \(x\) and \(y\) directions, respectively. To precisely extract the ROI boundaries, the \(G\) image was binarized by an Otsu's method, so that each pixel had a value 1 or 0 [19]. Here, dotted pepper noises were removed on the boundary image, resulting in clear a ROI boundary consisting of consecutive pixel sets as depicted in Figure 2. Then, ROI was extracted by retaining the region, which has smaller mean depth values on the depth image across the ROI boundary. Finally, the background region was automatically removed by overlapping the extracted ROI region on the FOV image.