# End-to-End Deep Reinforcement Learning for Image-Based UAV Autonomous Control
## 4 Proposed Method
### Algorithm Implementation  
To train the RL-agent, the image-based RL framework is developed with the DDPG algorithm, which is a kind of DRL, as an extension of deep Q-network algorithms. In contrast to deep Q-network algorithm, DDPG is based on the actor-critic framework, in which the critic is a parameterized function for estimating the Q-value, and the actor uses the parameterized function to approximate the optimal policy. Additionally, it can operate over continuous action spaces. On the one hand, DDPG algorithm introduces a deep neural network to approximate the critic function and the actor function, respectively, as a value network and a policy network, which makes processing high-dimensional and complex representations possible. On the other hand, it refers to the techniques of the deep Q-network algorithm named experience replay and target network in order to improve the stability of learning. Therefore, DDPG algorithm is proper for the image-based RL problem because it can resolve the image feature and generate the continuous control commands. With the designed network architecture and the reward function, the DDPG algorithm can be implemented for the image-based autonomous landing task, and its diagram is shown in Figure 5. In the figure, arrows with different colors represent the data flow during different phases of the algorithm. In the beginning, the parameters of the networks are randomly initialized. The UAV starts from the designated initial position with certain velocity and attitude. In each episode, the agent starts with a random exploration phase. The DDPG agent receives the processed image from the UAV and gives the action based on the current actor. Then an instant reward can be calculated and a new image is captured. This process is a step of interaction with the simulated environment, which collects a new piece of experience \((s_{t},a_{t},r_{t+1},s_{t+1})\) and stores it in the experience replay buffer. To enhance the exploration in the continuous action space, noise is added during the random exploration before producing the action, as shown in Equation (2). In this paper, the policy is deterministic and parameterized by \(\theta\). Thus, it is denoted as \(\mu(s_{t}|\theta)\). In addition, the exploration noise \(n_{t}\) is sampled from an Ornstein-Uhlenbeck process for exploration efficiency. \[a_{t}=\mu(s_{t}|\theta)+n_{t} \tag{2}\]  
Then, the DDPG agent is trained. A mini-batch of data is sampled from the experience replay buffer and fed to the actor and the critic, respectively. The idea of the target network is introduced here. The behavior value network parameterized by \(\mathbf{\omega}\) is considered to be \(Q(s,a;\mathbf{\omega})\) and the target value network parameterized by \(\mathbf{\omega}^{\prime}\) is \(Q^{\prime}(s,a;\mathbf{\omega}^{\prime})\). Similarly, the behavior policy network is \(\mu(s;\theta)\) and the target policy network is \(\mu^{\prime}(s;\theta^{\prime})\). The loss function for the value network is depicted as in Equation (3) in the form of mean square error (MSE):  
\[L(Q)=\frac{1}{N_{b}}\sum_{i=1}^{N_{b}}\left(y_{i}-Q(s_{i},a_{i};\mathbf{\omega}) \right)^{2} \tag{3}\]  
where \(N_{b}\) is the size of the mini-batch, \(y_{i}\) is the estimated return of the \(i\)-th experience according to Equation (4):  
\[y_{i}=r_{i+1}+\gamma Q^{\prime}(s_{i+1},\mu^{\prime}(s_{i+1};\theta^{\prime}) ;\mathbf{\omega}^{\prime}) \tag{4}\]  
For the actor, the policy network is updated by calculated the policy gradient concerning the actor parameter \(\theta\) according to Equation (5):  
\[\
abla_{\theta}L(\mu)=E_{s\sim\rho^{\mu}}[\
abla_{a}Q(s,a;\mathbf{\omega})\Big{|} _{a=\mu(s;\theta)}\cdot\
abla_{\theta}\mu(s;\theta)] \tag{5}\]  
where \(\rho^{\mu}\) is the distribution of states under the current policy \(\mu(s;\theta)\), and \(\
abla\) represents the gradient operator. Thus, the parameters of the behavior value and policy network are updated according to Equation (6), where \(\alpha_{Q}\) and \(\alpha_{\mu}\) are the learning rates of critic and actor, respectively.