# Hierarchical Active Tracking Control for UAVs via Deep Reinforcement Learning
## 4 Experiments
### Training
#### 4.1.2 Implementation Details  
We use the model-free algorithm PPO with auxiliary losses to train our active tracking RL controller. The modified PPO algorithm is summarized in Algorithm 1. [PERSON] [31] is used for optimization of the RL controller. The hyperparameters for the reward function, modified PPO algorithm, and optimizer are listed in Table 1. Our algorithm is implemented by [PERSON] [32]. ```
1 foriteration=1,2,...do
2 foractor=1,2,...,Ndo
3 Run policy \(\pi_{\theta_{old}}\) in the environment for \(T\) timesteps;
4 Compute advantage estimates \(\hat{A}_{1},\ldots,\hat{A}_{T}\) using Equation (6);
5 Calculate the total loss \(\mathcal{L}^{\text{Total}}\) using Equation (20);
6 Optimize \(\mathcal{L}^{\text{Total}}\) wrt \(\theta\), with \(K\) epochs and minibatch size \(M\leq NT\);
7 Update policy \(\theta_{old}\leftarrow\theta\) ;

#### 4.1.3 Ablation Studies in the Learning Process

We train our RL controller for the active tracking task through trial and error in the customized V-REP-based environment. We also perform ablation studies to evaluate the importance of the proposed auxiliary losses and the hierarchical control framework. Two groups of experiments are compared:

1. The proposed hierarchical active tracking controller framework, which uses the RL controller for perception and high-level decision-making, and the PID controller is used for low-level dynamic control. 2. The end-to-end active tracking controller framework, which maps raw image observations to a UAV's motor commands directly. In each group, four additional experiments are also compared:

1. Training with auxiliary segmentation and motion-in-depth loss. 2. Training with auxiliary segmentation but without motion-in-depth loss. 3. Training without auxiliary segmentation but with motion-in-depth loss. 4. Training without auxiliary segmentation and motion-in-depth loss. The learning curves of all experiments are presented in Figure 5. All experiments in the end-to-end active tracking control learning group, as shown in Figure 5a, cannot achieve notable progress during the learning process, indicating that learning the direct mapping from high-dimensional observations to low-level motor commands is extremely complicated. With the hierarchical control framework, the learning of an active tracking controller is much more efficient, as presented in Figure 5b. We can observe that training with auxiliary segmentation and motion-in-depth loss can converge quickly and achieve good asymptotic performance. The performance of training with only one of the two auxiliary losses suffers from the problem of weak stability. Training without any auxiliary losses is relatively data inefficient and cannot achieve good performance within \(10^{5}\) training episodes. Therefore, the proposed hierarchical control framework and auxiliary losses can improve the data efficiency of model-free RL and must be introduced in the quadrotor's active tracking task to achieve good and stable performance. ### Simulation Results

#### 4.2.1 Comparison with Baselines

To highlight the effectiveness of our hierarchical active tracking controller, we compare our controller and two baseline controllers in the unseen scenario, as illustrated in Figure 6. \begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Value** \\ \hline Number of actors N & 2 \\ Horizon T & 2048 \\ Adam stepsize & \(5\times 10^{-5}\) \\ Num. epochs & 2 \\ Minibatch size & 640 \\ Discount \(\gamma\) & 0.99 \\ GAE parameter \(\lambda\) & 0.98 \\ Clipping \(\epsilon\) & 0.2 \\ Auxiliary loss coefficient \(C_{Seg}\) & 1.0 \\ Auxiliary loss coefficient \(C_{Dep}\) & 0.05 \\ Reward coefficient \(\alpha_{1}\) & 0.5 \\ Reward coefficient \(\alpha_{2}\) & 0.5 \\ Reward coefficient \(\beta_{1}\) & 0.5 \\ Reward coefficient \(\beta_{2}\) & 0.5 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Hyperparameters in our implementation.