# Hierarchical Active Tracking Control for UAVs via Deep Reinforcement Learning
## 3 Methodology
### Auxiliary Segmentation and Motion-in-Depth Loss
#### 3.5.2 Auxiliary Motion-in-Depth Loss  
The temporal feature encoder in the RL controller is designed to encode the temporal information of the dynamic target. The temporal information should contain the change in the dynamic target's position that is parallel to the quadrotor's camera view plane and  
Figure 4: An aerial view of the quadrotor and dynamic target in the training environment.  
the relative depth change of the dynamic target that is vertical to the quadrotor's camera view plane. Among these, the depth change feature is more difficult to extract since the input resolution is set to be small for faster training, and the occupied resolution of the dynamic target is even smaller. To learn the dynamic target's position change vertical to the quadrotor's camera view plane, we need to introduce some auxiliary training signals at the end of the velocity perception layer.  
For a dynamic target, suppose that the length projected on the image plane and the depth from the camera center at timestep \(t\) are \(l_{t}\) and \(d_{t}\), respectively. We can denote the optical expansion \(s\) between two timesteps as \(t_{i},t_{j}\), which indicates the relative scale change of the dynamic target as follows:  
\[s_{i}^{j}=\frac{l_{j}}{l_{i}} \tag{15}\]  
We can also denote the motion-in-depth \(\tau\) between two timesteps \(t_{i},t_{j}\), which indicates the relative depth change of the dynamic target as follows:  
\[\tau_{i}^{j}=\frac{d_{j}}{d_{i}} \tag{16}\]  
Interestingly, the motion-in-depth is the reciprocal of the optical expansion [30], indicating that they are unified and can both represent the vertical position change information of the dynamic target. Because the motion-in-depth is easier to calculate, we choose to adopt auxiliary motion-in-depth loss as the additional training signal for the velocity perception layer, as shown in Figure 3.  
In [30], the motion-in-depth \(\tau\) is calculated as the ratio between the depth of corresponding points over two frames as follows:  
\[\tau_{i}^{j}=\frac{d\left(\mathbf{x_{i}}\right)}{d\left(\mathbf{x_{i}}\right)} \tag{17}\]  
where \(\mathbf{x_{i}}\) represents the occupied pixels of the dynamic target at the first frame and \(\mathbf{x_{j}}\) represents the correspondence of \(\mathbf{x_{i}}\) at the second frame.  
We further simplify the calculation of motion-in-depth by ignoring the pixel matching over two frames for fast training:  
\[\tau_{i}^{j}=\frac{d\left(\mathbf{y_{j}}\right)}{\min d\left(\mathbf{y_{i}} \right)} \tag{18}\]  
where \(\mathbf{y_{i}}\) and \(\mathbf{y_{j}}\) are the occupied pixels of the dynamic target over two frames at timesteps \(t_{i}\) and \(t_{j}\), respectively. Due to the condition for training episode termination in our setting (listed in Section 4.1.1), the dynamic target is always in view of the quadrotor at training time, and \(\left[d\left(\mathbf{y_{i}}\right),d\left(\mathbf{y_{i}}\right)\right]\) can be obtained directly in V-REP; thus, \(\tau_{i}^{j}\) is always available during training.  
The output map of the velocity perception layer is denoted as \(\mathbf{W}\), and the motion-in-depth map over timestep \(t-2\) and \(t\) is denoted as \(\tau_{t-2}^{t}\). We resize \(\tau_{t-2}^{t}\) as \(\mathbf{M}\) to be the same size as \(\mathbf{W}\) and then calculate the auxiliary motion-in-depth loss as follows:  
\[\mathcal{L}^{\text{Dep}}=\frac{1}{\left|\mathbf{W}\right|}\sum_{(i,j)\in \mathbf{W}}\left\|\log m_{i,j}-w_{ij}\right\|_{2}^{2} \tag{19}\]  
where \(m_{i,j}\) and \(w_{i,j}\) are the pixels of \(\mathbf{M}\) and \(\mathbf{W}\), respectively.  
Note that we skip frame \(t-1\) because the motion-in-depth over two contiguous frames can be too small to guide the learning process effectively.  
With both auxiliary segmentation and motion-in-depth losses, the total loss for the updating policy is the following:  
\[\mathcal{L}^{\text{Total}}=\mathcal{L}^{\text{CLip}}+C_{Seg}\mathcal{L}^{\text{ Seg}}+C_{Dep}\mathcal{L}^{\text{Dep}} \tag{20}\]  
where \(C_{Seg}>0\) and \(C_{Dep}>0\) are tunable hyperparameters.