# UAV Dynamic Object Tracking with Lightweight Deep Vision Reinforcement Learning
## 4 Research Methodology
### Reward Function  
The reward function is crucial for the learning process; a good reward function is able to balance exploration and exploitation. In our approach, we neither make the reward function too specific nor too simple. This reward function allows effective learning while maintaining exploration. Firstly, we extract the bounding box coordinates from our object detection module because it holds two useful pieces of information: the distance from the  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Interger** & **Action** \\ \hline
0 & MoveRight \\
1 & MoveLeft \\
2 & TurnRight \\
3 & TurnLeft \\
4 & MoveUp \\
5 & MoveDown \\
6 & MoveForward \\
7 & Hover \\ \hline \hline \end{tabular}
\end{table}
Table 1: Mapping from the DRL algorithm’s output to actions.  
UAV to the target, and the relative position of the target. There are two types of rewards in our reward function design: positive and negative. Positive rewards are given when the bounding box lies entirely within the middle 50% of the image width. We chose 50% because after trying with different thresholds, 50% showed the most satisfying results, as shown in Table 2. Because all images are 128 \(\times\) 72 pixels, we can tell the bounding box is inside the middle 50% if the top-left x-coordinate of the bounding box is greater than 128 \(\cdot\frac{1}{4}=32\) and its bottom-right x-coordinate is less than \(128\cdot\frac{3}{4}=96\). The amount of this positive reward is defined as the distance between the bounding box center and the frame center. The closer the distance is, the higher reward the UAV can achieve. That is because we want to encourage the UAV to keep the target as close to the center as possible.  
The agent gets negative rewards (penalties) when the bounding box is within the left 25% or the right 25% of the image width, as the middle 50% is allocated for positive rewards. This penalty discourages the agent from keeping the target object in the left or right 25% of the image width because there is a high probability that the target will move out of the current view from those regions. The agent also gets penalties when it makes a collision with an object. Furthermore, to prevent the agent from approaching too close to the target, we also apply a penalty if the bounding box height occupies more than 50% of the image height. On the other hand, to prevent the agent from positioning too far away from the target, the penalty is also applied if the bounding box height takes less than 10% of the image height. Because all images are 128 \(\times\) 72 pixels, 50% of its height equivalents to \(\frac{72}{2}=36\) pixels and 10% of its height is equivalent to \(\frac{72}{10}=7.2\) pixels. We choose these thresholds as a baseline to run our experiment. Other thresholds and their performances will be investigated further in future works. Our complete reward function is defined as:  
\[reward=\begin{cases}-1&\text{if collides}\\ \frac{1}{distance}&\text{if }x_{1}>32\text{ and }x_{2}<96\\ -distance&\text{if }x_{1}\leq 32\text{ or }x_{2}\geq 96\\ -\frac{boundingBoxHeight}{imageHeight}&\text{if }boundingBoxHeight>36\text{ pixels}\\ -\frac{0.1\cdot imageHeight}{boundingBoxHeight}&\text{if }boundingBoxHeight<7.2\text{ pixels}\\ 0&\text{otherwise}\end{cases} \tag{8}\]  
where \(distance\) is the Euclidean Distance between the bounding box center and the frame center, \(x_{1}\) is the bounding box's top-left x-coordinate, and \(x_{2}\) is the bounding box's bottom-right x-coordinate.