# Integration of the 3D Environment for UAV Onboard Visual Object Tracking
## 3 Single Visual Object Tracking Pipeline for UAV
### Point Cloud Reconstruction of the Environment  
We employ Structure-from-Motion (SfM) for achieving the mapping between the 2D image space and the 3D reference system, i.e., scene space. Owing to our modular design, the mapping between the 2D image space and the 3D scene space can be replaced with an alternative photogrammetric technique. For this paper, we create a point cloud representation of the observed environment, which is sufficient for the demonstration of our approach. We leverage COLMAP (in our experiments we used COLMAP 3.6, available publicly under [[https://colmap.github.io/](https://colmap.github.io/)]([https://colmap.github.io/](https://colmap.github.io/)) (accessed on 14 October 2020)) [40; 59], which is an established approach for SfM. For a given image set that contains overlapping images of the same scene and taken from different viewpoints, COLMAP automatically extracts the corresponding camera poses and reconstructs the scene structures in 3D. To achieve this, the library follows common SfM steps. (1) A correspondence  
Figure 5: Block diagram of a visual object tracking pipeline—i.e., ATOM and DiMP—without the update step. During the tracking cycle, salient features from the search area are extracted through a feature extractor. Based on the extracted features, a similarity score map is inferred, used in estimating an initial bounding box of the object. Afterward, a refined bounding box is estimated trough the bounding box regressor, i.e., IoU-Net. search, where salient features from an image set are extracted and matched across the images, and incorrectly matched features are filtered out. (2) The scene is reconstructed as a point cloud by performing image registration, feature triangulation, and bundle adjustments. Figure 6a presents a point cloud reconstruction leveraged by COLMAP on an image sequence of AU-AIR-Track. The initial reconstruction of the scene contains noise near the camera frame and statistical outliers. Points near the camera frame are falsely triangulated and outliers are points correctly positioned but that are too sparse for reliably representing structures of the scene. To reduce the number of incorrect points in the reconstruction, we filter out points close to the camera frame and statistical outliers. We discard near-camera points from the point cloud by comparing their Euclidean distance with a threshold. For statistical outliers, we proceed as follows: (1) We define a neighborhood of 10 neighbors. The average distance \(d_{i}\) of a given point \(i\) to its neighbors is calculated using the Euclidean distance. (2) A standard deviation threshold \(\sigma_{lim}\) is defined and the overall average for all \(d_{i}\) is computed as \(d_{avg}\). Points with an average distance \(d_{i}\
ot\in[d_{avg}-\sigma_{lim},d_{avg}+\sigma_{lim}]\) are identified as outliers and discarded from the point cloud. Figure 6b,c display the removal of points close to the camera frame and statistical outliers. Moreover, a plane ground \((\mathbf{x}_{g},\mathbf{y}_{g})\) is estimated by using the Random sample consensus (RANSAC) [60] method (see Figure 7). Points below the ground plane \((\mathbf{x}_{g},\mathbf{y}_{g})\) are discarded. We compute depth map approximations based on the new scene reconstructed, i.e., the filtered point cloud with the estimated ground plane. The depth map approximations encode, for every pixel, the distance between the positions of the UAV and the visible points in the scene for the corresponding viewpoint, i.e., camera pose. For every frame in the AU-AIR-Track dataset, a corresponding depth map approximation is constructed by relying on the pinhole camera model; thus, enabling the mapping between the 2D image space and the 3D scene space. Figure 8 displays a depth map approximation example with the corresponding frame. Figure 6: (a) A point cloud reconstruction using COLMAP. (b) Highlighted noise near the frame of reference in red. (c) Highlighted outliers in red after removing near-frame reference noise.