DOI: 10.25549/usctheses-c40-336736
title: 3D mediated detection and tracking in wide area aerial surveillance

description: With recent breakthrough technologies in unmanned aerial vehicles (UAVs) and digital photography, monitoring a wide area of view is now possible. This new type of imagery, which is called wide area motion imagery (WAMI), contains tens of megapixels to gigapixels per image. It is extremely laborious for human operators to monitor such large amount of data frame by frame. An automatic system for detection and tracking in WAMI is, therefore, essential. ❧ Our goal is to detect and track moving objects (vehicles) in wide area aerial surveillance (WAAS). Specifically, we aim to achieve high recall and precision as well as low ID-switches and track breaks. Despite detection and tracking problems have been investigated in computer vision for many years, unique characteristics in WAMI introduces new challenges to these problems including weak appearance targets and low frame rate videos. We propose to attack the problem from different aspects including using information in 3D, appearance, motion, and context. ❧ State-of-the-art trackers in WAAS detect moving targets in each frame and attempt to associate them into tracks. Therefore, the performance in motion detection has a significant impact on the performance in tracking. Unlike popular 2D stabilization which introduces misalignment around 3D structures in the scene and further produces many false alarms in motion detection results, we propose that using 3-D information significantly improves the performance in motion detection. ❧ To further improve detection performance, we introduce 3D stabilization to compensate camera motion effect as if the scene had been captured by a static camera. We propose a novel approach which generates accurate long-term dense mapping and handles occlusion robustly. Our evaluation on real-world aerial sequences indicates that the proposed method outperforms state-of-the-art 2D and 3D stabilization approaches. Moreover, we demonstrate that using 3D stabilized imagery as an input, the number of false detection significantly reduces in a simple moving object detection method. ❧ In real world scenarios, non-linear motion such as sharp turns after slowing down or U-shape trajectories occur. To tackle various motion patterns in tracking, we propose an efficient association based tracker that relaxes linear motion assumption in extracting short tracks from motion detections in a low frame rate video. It avoids track breaks in complex motion trajectories including sharp turns and U-shape turns. Merged detections and short-term missing detections are handled by generating additional hypotheses to be considered in the optimization problem. The evaluation on WAMI sequences shows that the proposed approach improves tracking performance compared with existing approaches. ❧ To address the limitation of association based trackers in long-term missing detection from slow or stopped vehicles, we propose a novel hybrid tracking system. We couple the association based tracker with a local context tracker, which explores spatial relation with targets in a neighborhood and tracks them without using detection responses. It alleviates the dependency on weak appearance model of small targets and therefore is more robust against model drifting. The experimental results show further improvement in recovering missing detections and the proposed method outperforms state-of-the-art detection and tracking methods in all common axes of evaluation metrics.