# End-to-End Deep Reinforcement Learning for Image-Based UAV Autonomous Control
## 4 Proposed Method
### Algorithm Implementation
In our implementation, both the actor and the critic are updated by Adam optimizer. \[\left\{ \begin{array}{l}\mathbf{\omega}\leftarrow\mathbf{\omega}+\alpha_{Q}\
abla_{ \omega}L(Q)\\ \theta\leftarrow\theta+\alpha_{\mu}\
abla_{\theta}L(\mu)\end{array} \right. \tag{6}\]  
Figure 5: The diagram of the algorithm implementation. The target network is a copy of the behavior network and is only updated after certain updates of the behavior network according to Equation (7), where \(\tau\) is the soft update rate. \[\left\{ \begin{array}{l}\mathbf{\omega}^{\prime}\leftarrow\tau\mathbf{\omega}+(1-\tau) \mathbf{\omega}^{\prime}\\ \mathbf{\theta}^{\prime}\leftarrow\tau\mathbf{\theta}+(1-\tau)\mathbf{\theta}^{\prime} \end{array} \right. \tag{7}\]  
An episode terminates with the UAV successfully landing, going beyond the normal states, or attaining the maximum number of steps per episode. For a new episode, the UAV is reset to the initial state and continues trial-and-error. The DDPG-based training algorithm is summarized as Algorithm 1. When it comes to testing, the trained networks can be directly loaded to provide optimal action output according to the image input without further updating. ```
1. Randomly initialize the behavior network ( \(\mathbf{\theta},\mathbf{\omega}\));
2. Initialize the target network ( \(\mathbf{\theta}^{\prime},\mathbf{\omega}^{\prime}\)) by copying the behavior network;
3. Initialize the OU process for action exploration noise;
4. Initialize the replay buffer;
5. For each episode: (1) Reset the UAV with initial states; (2) Receive initial observation of the image state \(l_{0}\); (3) For each time step \(t=0\),\(1\),\(2\).........................................



### Environment Setup

The proposed end-to-end control method is validated by simulation experiments, using the autonomous landing of a fixed-wing UAV as an example. The overall system was developed on ROS Kinetic with the Ubuntu16.04 operating system. The DDPG agent part was implemented with Python and the environment part was implemented with C++. The neural network was established using the Tensorflow framework so that it could run on both CPU and GPU to accelerate computation. Both the training and testing phases were conducted in the simulated environment established with Gazebo. As shown in Figure 7, the simulated environment includes the runway on the ground and the UAV model. The centerline of the runway starts at the origin point (0, 0, 0) m and overlaps with the \(x\)-axis of the global coordinate system. The UAV model is the Techpod model provided by the Rotor5 simulator, including the inner loop controllers and basic sensors. Additionally, a downwards camera is attached to the model by editing the corresponding file to obtain the images. The overall system contains three main nodes: the DDPG agent node, the environment node, and the Gazebo node. The communication among the nodes is realized by publishing or subscribing to topics and services. The computing graph of the program that describes the nodes' relationship is shown in Figure 8.

Figure 8: The computing graph of the simulation program. Figure 6: The relationship of the UAV model and the proposed controller. Figure 7: The simulation environment and setup. As for the parameters setting, the deflection range of the UAV actuators is \([-0.35,0.35]\) rad. The learning rate for the actor-network is \(1\times 10^{-4}\). In addition, the learning rate for the critic-network is \(1\times 10^{-3}\). The soft update rate is \(\tau=0.001\). The downward inclination angle of the UAV's onboard camera is \(45^{\circ}\). The range of the pitch angle is \([-60^{\circ},60^{\circ}]\). The range of the roll and the yaw angle is \([-15^{\circ},15^{\circ}]\). Additionally, the border of \(z\)-axis is [0,15]. In addition, the border of the \(x\)-axis and \(y\)-axis depends on the different conditions of the simulation experiments.