# Hierarchical Active Tracking Control for UAVs via Deep Reinforcement Learning
###### Abstract  
Active tracking control is essential for UAVs to perform autonomous operations in GPS-denied environments. In the active tracking task, UAVs take high-dimensional raw images as input and execute motor actions to actively follow the dynamic target. Most research focuses on three-stage methods, which entail perception first, followed by high-level decision-making based on extracted spatial information of the dynamic target, and then UAV movement control, using a low-level dynamic controller. Perception methods based on deep neural networks are powerful but require considerable effort for manual ground truth labeling. Instead, we unify the perception and decision-making stages using a high-level controller and then leverage deep reinforcement learning to learn the mapping from raw images to the high-level action commands in the V-REP-based environment, where simulation data are infinite and inexpensive. This end-to-end method also has the advantages of a small parameter size and reduced effort requirements for parameter turning in the decision-making stage. The high-level controller, which has a novel architecture, explicitly encodes the spatial and temporal features of the dynamic target. Auxiliary segmentation and motion-in-depth losses are introduced to generate denser training signals for the high-level controller's fast and stable training. The high-level controller and a conventional low-level PID controller constitute our hierarchical active tracking control framework for the UAVs' active tracking task. Simulation experiments show that our controller trained with several augmentation techniques sufficiently generalizes dynamic targets with random appearances and velocities, and achieves significantly better performance, compared with three-stage methods. unmanned aerial vehicle; deep reinforcement learning; visual active tracking +
Footnote †: journal: Medical  
features, such as SIFT [5] and SURF [6], to detect more random objects. Recent research combines a convolutional neural network (CNN)-based object detector [7; 8; 9; 10] and a passive tracker [11; 12; 13; 14; 15] to obtain the bounding box of the target of interest. In the decision-making stage, control methods, such as proportional integral derivative (PID), calculate high-level action commands based on the coordinates of the bounding box's center and relative changes in the bounding box's height. PID is also usually used to control UAVs' low-level dynamics. However, training the CNN-based visual perception module requires considerable effort for ground truth labeling, and the parameter tuning process for the high-level decision-making module may cause unexpected damage to UAVs. Reinforcement learning (RL) holds promise for automated learning perception and control simultaneously. RL has several applications in the UAV community. [PERSON] et al. used RL to learn an aggressive trajectory tracking controller for UAVs [16]. [PERSON] et al. applied RL to teach UAV vision-based guidance tasks [17]. [PERSON] et al. achieved autonomous landing on the deck of an unmanned surface vehicle, using RL [18]. Through trial and error, model-free deep RL can learn complicated high-level control policies for UAVs, which maps visual observations to action commands directly. This end-to-end solution trains perception and control concurrently for UAVs' active tracking task in the virtual environment and overcomes the problems encountered by three-stage methods. [PERSON] et al. [19] used the ConvNet-LSTM network as the high-level policy network, which performs well in different active tracking scenarios but does not focus on the UAV domain and cannot transfer directly without considering the dynamic properties of the UAV. [PERSON] et al. [20] proposed a hierarchical control system that uses the policy network trained by deep RL as the high-level decision-making layer and the PID controller as the low-level command execution layer.