# End-to-End Deep Reinforcement Learning for Image-Based UAV Autonomous Control
## 1 Introduction  
In recent years, Unmanned Aerial Vehicles (UAVs) have been widely applied in various military and civil situations [1; 2; 3]. In particular, increasing attention has been paid to the autonomous control of UAVs [4]. UAVs are expected to fly safely on their own and make appropriate decisions without human intervention, completing their mission with higher efficiency and lower cost using entirely onboard sensing and computing. In missions of increasing complexity and uncertainty, it is becoming increasingly challenging for UAVs to react appropriately to dynamic environments in time. The typical paradigm for achieving autonomous control is perception-planning-control. In this paradigm, the UAV first needs to observe the environment and determine its states relative to the environment with multiple sensors, including the exteroceptive information from the Global Positioning System (GPS) or the motion capture system, as well as the proprioceptive information from the Inertial Navigation System (INS) or visual sensors [5]. After perception, the UAV plans its trajectory or motion according to the optimization goal within various constraints [6]. Furthermore, low-level control commands are then generated for the inner loop controller in order to resolve commands hierarchically into signals for the actuator. Schemes based on the perception-planning-control paradigm often consist of well-defined separate modules with the interfaces among them requiring appropriate handcrafted designs. However, these specifically designed interfaces may cause the intermediate representation to be restrictive and less adaptive to uncertainty. In addition, the unidirectional paradigm may lead to the accumulation of errors from different modules. Additionally, it is always necessary to build a prior model. On the one hand, the models are an idealized approximation and many significant effects are often omitted or cannot be accurately modeled [4]. On the other hand, it is difficult for model-based algorithms to deal with unknown system dynamics. When it comes to implementation, complicated algorithms such as mapping [7], estimation and online optimization [8] have a high computation cost, which is challenging for onboard micro-computers to meet. Machine learning techniques have been widely used to provide robust and adaptive performance, while simultaneously reducing the required development and deployment time [9]. In particular, deep learning (DL) plays an essential role in various fields of study due to its capacity for extracting representative features of the input data by learning. It gives a possible solution for replacing handcrafted features with deep-learned features. Furthermore, the combination of DL with another branch of machine learning named reinforcement learning (RL), which depends on interaction with the environment, has demonstrated superiority in tasks with respect to both perception and decision making. Typical applications of deep reinforcement learning (DRL) methods include playing games [10; 11], natural language processing [12; 13], autonomous driving [14; 15], robotics manipulation [16; 17], and so on. Inspired by the idea of DRL, which integrates the feature representation ability of DL and the intelligent decision ability of RL, this paper proposes a UAV autonomous control framework based on end-to-end DRL with the image as the only input.. Unlike several previous works that produce discrete action [18] or reference velocity [19], a single deep neural network is established to map the input image directly onto the continuous actuator control command, and is trained with a reinforcement learning algorithm.