# Integration of the 3D Environment for UAV Onboard Visual Object Tracking
## 2 Related Work  
In recent years, great progress regarding single visual object tracking has been made, owing to the abundance of benchmarks available [2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13]. Most of them are designed towards evaluating tracking algorithms on ground-level perspectives, resulting in state-of-the-art visual object trackers following the tracking-by-detection paradigm. Currently, three tracking designs prevail on those benchmarks: the discriminative correlation filters [14; 15; 16; 17; 18; 19; 20]; the Siamese-based approach [21; 22; 23; 24; 25; 26]; and recently, trackers inspired by correlation filters that employ a small convolutional neural network for learning the appearance model of the object [27; 28]. In all three design choices, the main difference lies in how they learn an appearance model of the object. The latter style is explored in this paper. These tracking algorithms are tailored to scenarios presenting ground-level perspectives, but onboard UAV perspectives present particular challenges. For instance, the object occupies a relatively small portion of the image space, resulting in a less-accurate learned appearance model. This leads to lower discrimination capabilities when similar objects to the tracked object are encountered. In particular, tracking an object with an oblique view on the scene from a UAV can present  
Figure 3: Qualitative results presenting the trajectory of the object in image space (upper figure) and 3D scene space (lower figure). The red points in both images indicate the 2D and 3D trajectory extracted during tracking. Yellow points (2D) and blue points (3D) indicate the occluded trajectory reconstructed by leveraging a state estimator. Figure 2: Qualitative comparison between an original tracking algorithm, its 2D variant, and its 3D variant for UAV onboard visual object tracking. A closer look of the scenario is shown on the lower part of the figure, based on a region delimited by a red dash–dot rectangle in the corresponding upper image. Only the 3D variant is able to overcome the occurring occlusion. multiple occlusion situations compared to a top-down view. To analyze the performances of tracking algorithms in UAV scenarios, several benchmarks have been introduced [3; 29; 30; 31], ranging from low to high altitudes, and propose either an oblique view from the UAV or a top-down view. Most participating trackers in the SOT UAV benchmarks use state-of-the-art single visual object trackers presented in ground-level perspective benchmarks without or only with minor adaptations. However, none of the adapted trackers participating in the SOT UAV benchmarks attempt to utilize 3D information. This can be explained by the lack of such information being provided in the datasets. In contrast, the AU-AIR dataset [32] introduced recently is oriented toward object detection from a UAV viewpoint. It offers sequences that capture typical traffic on a roundabout, as would a surveillance drone for traffic monitoring, and contains sequences that are suitable for reconstructing the observed scene in 3D--sufficient translation movements, not flying around excessively, and enough structures in the environment. In contrast to the SOT UAV benchmark approaches mentioned previously, there are different application domains such as autonomous driving, where objects are tracked in a 3D system of reference through a detection-by-tracking paradigm. Typically, the 2D image detections generated serve as measurements and are mapped from image space in the ego-motion-compensated reference system of the car. An example of applications for traffic monitoring is presented in [33], where the authors propose a pipeline including Multiple Object Tracking (MOT), stereo cues, visual odometry, optical scene flow, and a Kalman filter [34] to enhance tracking performance on the KITTI benchmark [35; 36]. A follow up to this study was [37], which reconstructed the static scene and the object in 3D, allowing the shift from tracking in the 2D image space toward tracking in 3D scene space.