# Hierarchical Active Tracking Control for UAVs via Deep Reinforcement Learning
## 3 Methodology
### Simulator Set-Up and Augmentation  
Due to the data inefficiency issue of model-free RL, extensive interaction data with the environment are necessary to improve the high-level RL controller through trial and error. We cannot afford to train our RL controller for quadrotors in the real world. Instead, we set up a simulation environment where the quadrotors can gain infinite and high-fidelity training data.  
Based on the virtual robot experimentation platform (V-REP) [21], we build an RL training environment for the quadrotor's active tracking task, as shown in Figure 2. In this environment, three main entities exist: a quadrotor equipped with a camera, an IMU and an altitude ranging sensor, and a dynamic person (the target) walking along a path from the beginning to the end. The camera is tilted down 30 degrees relative to the horizontal plane. The quadrotor uses sequential images from the camera as the observation and decides the desired speed commands using the RL controller to follow the dynamic target. Then, the quadrotor uses the IMU and altitude ranging sensor to obtain the current velocity and height information and applies the desired motor commands to track the desired speed commands while maintaining its altitude via the PID controller. Next, the environment returns the reward function and the next observation. At the initialization of each training episode, the target people will be in front of the quadrotor.  
To enhance the generalizability of our trained RL controller to various domains, we adopt several environmental augmentation methods as follows.  
1. Visual randomization: We divide the appearance of the target person into five parts--hair, skin, shirt, trousers, and shoes. Then, we change the color of each part at every timestep, which is helpful to learn the dynamic target's more essential features rather than simply memorizing the colors.
2. Speed randomization: To learn an RL controller that is less sensitive to the target's velocity, we change the walking speed of the target person at every timestep in the interval \([0,V_{target}]\) uniformly.
3. Path randomization: To increase the robustness of the RL controller to the gestures and relative position of the target, we randomly sample \(n\) control points between the beginning and the end of the path and then generate a smooth trajectory using a B-spline for every training episode. The target person will follow the random trajectories under control.  
The simulation speed is the main concern when training with data-driven model-free RL methods. However, the original Python remote API of V-REP is not sufficiently fast. To accelerate the training process, we leverage PyRep [22] to break this data generation bottleneck. PyRep is built on top of the V-REP and can provide a flexible API and significant  
Figure 2: V-REP based virtual training environment for the quadrotor’s active tracking task.  
acceleration. Moreover, the simulation environments are easy to parallelize, implying that several workers can be used to interact with different environments at the same time and accelerate the training even further.