# Hierarchical Active Tracking Control for UAVs via Deep Reinforcement Learning
## 4 Experiments
### Training
#### 4.1.2 Implementation Details
**(a)** Sequence of raw observation: \(\mathbf{O}_{640}\) to \(\mathbf{O}_{645}\). **(b)** The corresponding output of the position perception layer: \(\mathbf{Z}_{640}\) to \(\mathbf{Z}_{645}\). **(c)** The corresponding output of the velocity perception layer: \(\mathbf{W}_{640}\) to \(\mathbf{W}_{645}\). **(d)** The final commands of the RL controller \(\tilde{V}_{fb}\), \(\tilde{V}_{gav}\). Figure 5:0 _Cont._

The sequence of action commands is shown in Figures 11d, 12d and 13d, and the corresponding observations indicate that the RL controller for the active tracking task is robust to a dynamic target with different appearances, positions, and speeds. Figure 5:1 Visualization of layers and output commands in the RL controller given a sequence of raw observations from timesteps \(t=720\) to \(t=725\). (**a**) Sequence of raw observation: \(\mathbf{O}_{720}\) to \(\mathbf{O}_{725}\). (**b**) The corresponding output of the position perception layer: \(\mathbf{Z}_{720}\) to \(\mathbf{Z}_{725}\). (**c**) The corresponding output of the velocity perception layer: \(\mathbf{W}_{720}\) to \(\mathbf{W}_{725}\). (**d**) The final commands of the RL controller \(\bar{V}_{fb}\), \(\bar{V}_{gaw}\). Figure 5:2 Visualization of layers and output commands in the RL controller, given a sequence of raw observations from timesteps \(t=880\) to \(t=885\). (**a**) Sequence of raw observation: \(\mathbf{O}_{880}\) to \(\mathbf{O}_{885}\). (**b**) The corresponding output of the position perception layer: \(\mathbf{Z}_{880}\) to \(\mathbf{Z}_{885}\). (**c**) The corresponding output of the velocity perception layer: \(\mathbf{W}_{880}\) to \(\mathbf{W}_{885}\). (**d**) The final commands of the RL controller \(\bar{V}_{fb}\), \(\bar{V}_{gaw}\). ## 5 Conclusions

In this paper, we propose a hierarchical control framework for UAVs' active tracking task. This framework combines the PID-based low-level controller with the high-level RL controller. The RL controller consists of a novel perception layer and a standard actor-critic layer, enabling end-to-end perception and high-level control with high-dimensional raw images as input. The perception layer encodes the spatial and temporal features of the dynamic target by the convolutional block attention module and spatial feature difference, respectively. The auxiliary segmentation loss and motion-in-depth loss introduced are essential for our RL controller's fast and stable learning. No ground-truth labeling is required in the learning process. Simulation experiments show that our method is effective in the active tracking task and robust to a dynamic target's appearance and velocity, compared with conventional three-stage methods. Further research should be devoted to the smooth RL controller, which outputs energy-saving actions. More augmentation techniques should be introduced to achieve stable tracking for various dynamic targets in complex environments and mitigate the sim-to-real problem. Conceptualization, W.Z. and Z.M. ; methodology, W.Z. and K.W. ; software, validation, formal analysis, W.Z. and J.Z. ; writing--original draft preparation, W.Z. ; writing--review and editing, W.Z., S.L. and K.W. ; visualization, K.W. and S.L. ; supervision, project administration, funding acquisition, Z.M. All authors have read and agreed to the published version of the manuscript. This work was supported by the National Natural Science Foundation (NSF) of China (No. 61976014). Not applicable. The data presented in this paper are available on request from the first author. The authors declare no conflict of interest. ## Appendix A Architecture of the Network

The network architecture of the RL controller is detailed in Table 1, where Conv2a + ReLU and Conv2b are shared in the channel attention module of the position perception layer, and Plus2a, Plus2b and Plus2c are the outputs of the shared position perception layer given the observations \(\mathbf{O}_{t},\mathbf{O}_{t-1},\mathbf{O}_{t-2}\), respectively. The parameters kernel_size, stride, and padding are used for the height and width dimensions.