# UAV Dynamic Object Tracking with Lightweight Deep Vision Reinforcement Learning
## 4 Research Methodology
### Deep Deterministic Policy Gradient (DDPG)  
A DDPG is a typical example of the Actor-Critic method. It has two distinct networks: the Actor (denoted by \(\mu\)), and the Critic (denoted by \(Q\)). In order to improve the stability of the learning, two more target networks are used: one is for the Actor, and the other is for the Critic. Therefore, the DDPG algorithm has four networks: one Actor, one Critic, one Target Actor, and one Target Critic. The target networks are updated via soft update. The role of the Actor is to decide which action is the best to pick based on the current state. On the other hand, the Critic evaluates state and action pairs, giving the Actor feedback on whether the taken actions are good in that state. The DDPG uses a deterministic policy, which produces the action values, but not the probabilities. The deterministic policy has a problem with the explore-exploit dilemma. This issue can be reduced by adding some  
Figure 6: The CNN architecture used in the DQN and DDPG algorithms in this paper. Figure 7: The complete state representation. extra noise into the output of the Actor Network (e.g., the Gaussian noise). The update rule of the Actor Network is:  
\[ \begin{split}\
abla_{\beta^{\mu}}J&\approx\mathbb{E}_{ S_{t}\sim\rho^{\beta}}\bigg{[}\
abla_{\beta^{\mu}}Q\Big{(}s,a\mid\theta^{Q} \Big{)}\mid\\ s&=S_{t},a=\mu(S_{t}\mid\theta^{\mu})\bigg{]}\end{split}  \tag{4}\]  
where \(\theta^{\mu}\) is the parameter of the Actor Network \(\mu\), \(\theta^{Q}\) is the parameter of the Critic Network \(Q\), \(\mu(S_{t}\mid\theta^{\mu})\) is the action chosen by the Actor Network \(\mu\) at state \(S_{t}\), and \(Q(s,a\mid\theta^{Q})\) is the state-action value when taking action \(a\) in state \(s\). The objective of the Critic Network is to shift the estimated state-action value \(Q(s,a)\) as close as possible to the target value. The target value is defined as:  
\[y_{i}=r_{i}+\gamma Q^{\prime}(S_{i+1},\mu^{\prime}(S_{i+1}\mid\theta^{\mu^{ \prime}})\mid\theta^{Q^{\prime}}) \tag{5}\]  
where \(r_{i}\) is the current reward, \(S_{i+1}\) is the next state, \(Q^{\prime}(\cdot)\) is the state-action value estimated by the target Critic Network, and \(\mu^{\prime}(\cdot)\) is the action chosen by the target Actor Network. The objective of the Critic Network can be achieved by minimizing the following loss function:  
\[L=\frac{1}{N}\sum_{i}\Big{(}y_{i}-Q\Big{(}S_{i},A_{i}\mid\theta^{Q}\Big{)} \Big{)}^{2} \tag{6}\]  
where \(y_{i}\) is the target value defined in Equation (5). At the beginning, the weights \(\theta^{\mu}\) and \(\theta^{Q}\) of the Actor Network \(\mu(s\mid\theta^{\mu})\) and the Critic Network \(Q(s,a\mid\theta^{Q})\) are randomly initialized. After that, those parameters are copied exactly into the corresponding target network, and this is the only time the exact hard copy is executed. Since then, the \"soft update\" rule has been adopted and it only makes partial updates from the original networks. The soft update rule is described below:  
\[ \begin{split}\theta^{\mu^{\prime}}&\leftarrow \tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}}\\ \theta^{Q^{\prime}}&\leftarrow\tau\theta^{Q}+(1- \tau)\theta^{Q^{\prime}}\end{split}  \tag{7}\]  
where \(\theta^{\mu^{\prime}}\) is the weights of the target Actor Network, \(\theta^{Q^{\prime}}\) is the weights of the target Critic Network, and \(\tau\) is a small number that is close to 1 (e.g., 0.99). The implementation detail of this algorithm in our approach can be viewed in Algorithm 2. Similar to the DQN, we need to initialize the essential objects in the first step, which includes the Airsim UAV, the target object's segmentation ID, the Environment, the Experience Replay, the Actor Network, the Critic Network, the Target Actor Network, the Target Critic Network, and the DDPG agent. Then, we initialize a random policy that selects the \(\epsilon\)-greedy action to fill the Experience Replay partially. After that, the DDPG agent can use the data in the Experience Replay to train.