# Hierarchical Active Tracking Control for UAVs via Deep Reinforcement Learning
## 4 Experiments
### Training
#### 4.1.2 Implementation Details
The mapping is almost linear in the interval \(\delta\varphi\in[-0.1,0.1]\) rad and becomes saturated when beyond this interval. We note some slow-down commands in the interval \(\delta\varphi\in[0.1,0.45]\cup[0.6,0.9]\) rad because of the dynamic target's rapid position change. To illustrate the effectiveness of our controller's temporal representation ability, we visualize the sequences of desired speed commands in the test episode. The desired normalized linear velocity \(\bar{V}_{fb}\) and the ground truth distance error \(\delta\rho\) over time are presented in Figure 10a. When the distance error \(\delta\rho\) increases, the output \(\bar{V}_{fb}\) also increases. Similarly, \(\bar{V}_{fb}\) decreases when \(\delta\rho\) begins to drop. Moreover, the change in \(\bar{V}_{fb}\) can adapt to different variation scales of \(\delta\rho\). These results indicate that the RL controller can obtain not only the spatial features, but also the temporal feature of the dynamic target. We can draw a similar conclusion from Figure 10b, which shows the sequences of ground-truth yaw angle error \(\delta\varphi\) and the desired normalized angular velocity \(\bar{V}_{yaw}\). By comparing the change in \(\delta\varphi\) and \(\bar{V}_{yaw}\) in the range \([0,350]\) and the \([350,600]\) timestep, we observe that the correlation between the change rates of \(\delta\varphi\) and those of \(\bar{V}_{yaw}\) is positive. To better understand the proposed RL controller, we sample three sequences with a length of six timesteps during the test episode and then visualize the layers in the RL controller and the corresponding action commands in Figures 11-13. The dynamic target changes its appearance, position, and speed at every timestep, as shown in Figures 11a, 12a and 13a. Given the sequence of raw observations, the RL controller's position perception layer can precisely segment the dynamic target from the background, as presented in Figures 11b, 12b and 13b. The spatial information of the dynamic target is extracted for further temporal sensing. The corresponding outputs of

Figure 8: Comparison of different high-level controller for quadrotor’s active tracking tasks. All controllers use additional PID for quadrotor’s dynamic control. velocity perception layers \(\mathbf{W}_{t}\) in Figures 11c, 12c and 13c are highly abstract and fuse the spatial and temporal features of the dynamic targets. We observe that the RL controller concentrates on the dynamic target when it vanishes in the quadrotor's view; otherwise, it concentrates on the overall information in the observation. From timesteps \(t=640\) to \(t=645\), the dynamic target is to the left of the quadrotor, and the RL controller yields the maximum yaw-left command to pull the target back to the center of the view. From timesteps \(t=720\) to \(t=725\), the dynamic target is in front of the quadrotor and accelerates, and the RL controller adjusts the yaw command \(\bar{V}_{yaw}\) slightly to \(0\) and increases the linear speed command \(\bar{V}_{fb}\) to keep up with the dynamic target. From timesteps \(t=880\) to \(t=885\), the target accelerates and moves to the quadrotor's right; then, the RL controller outputs the maximum \(\bar{V}_{yaw}\) and commands \(\bar{V}_{fb}\) to follow the dynamic target. Figure 9: Visualization of the mapping from the state errors to the corresponding actions. (**a**) Mapping from the ground truth depth error to the output normalized linear velocity. (**b**) Mapping from the ground truth yaw angle error to the output normalized angular velocity. Figure 11: _Cont_. Figure 10: Visualization of the sequences of the state errors and the corresponding actions. (**a**) Comparison of sequences for the ground truth distance error and the output normalized linear velocity. (**b**) Comparison of sequences for the ground truth yaw angle error and the output normalized angular velocity. Figure 11: Visualization of the layers and output commands in the RL controller given a sequence of raw observations from timesteps \(t=640\) to \(t=645\).