# Obstacle Avoidance Drone by Deep Reinforcement Learning and Its Racing with Human Pilot
## 1 Introduction  
The drone is an unmanned aerial vehicle that has been around for a long time, and yet it has become a major research field recently. It seems that the recent success of the drone may come from the stable control of rotors and a bird's eye view provided by a camera installed in front of it. With the growing number of possible applications for the drone, i.e., disaster management [1] and agriculture [2, 3], the demand for expert drone pilots is growing up nowadays. To become an expert level drone pilot, one has to spend extended time training. Dexterity in maneuvering a fast-flying drone has become an essential asset for professional drone racers, as a survey indicates that drone racing is one of the fastest growing e-sports, among many others [4]. Indeed, one of the major research fields for the drone has been autonomous drone navigation. One of the critical requirements for the successful application of drones is the ability to navigate through buildings and trees. To achieve this, obstacle avoidance capability is essential in particular. Controlling a drone with such dexterity involves solving many challenges in perception as well as in action using lightweight and yet high performing sensors. Among many, simultaneous localization and mapping (SLAM) has been a major representative approach to solve such challenges by utilizing the stereo camera, Lidar, and other sensors. Although SLAM can cover a range of challenges with the localization problem [5], it also shows some limitations, for example, when the target area has visually changed. Recently, an alternative by utilizing deep learning has arisen, as its ability to deal with vision data is promising. These approaches are mainly divided into two machine learning paradigms: (1) supervised/ unsupervised learning and (2) reinforcement learning (RL). In supervised learning, one has to gather a large set of data for a specific environment where the drone will operate before the training. This schemecould be effective when the size of the dataset is large enough with high-quality labels. However, collecting massive amounts of data whenever the given environment changes could become a problem. Moreover, labeling all these data not only imposes a time-consuming job but also makes it hard to guarantee whether or not hand-crafted labeling for the massive number of data is optimal for the task. Due to this issue, unsupervised learning-based methods are adopted for the automatic labeling of the dataset in several studies. On the other hand, the RL based approach aims to deal with this issue by letting the drone-agent itself learn a simulated environment in a trial and error manner. In RL based approaches, the agent tries to learn in a virtual environment that is similar to the real environment and then the trained model is transferred to the real drone for testing. In this study, we experiment with obstacle avoidance drones within two control spaces: one is discrete space and the other is continuous space. For the former, Deep Q-Network (DQN) [6] will be adopted for controlling the drone while utilizing two visual inputs such as RGB and depth map from the simulated environment constructed with Airsim [7]. There are four deep RL algorithms such as DQN, Double DQN [8], Dueling DQN [9], and Double Dueling DQN (DD-DQN). The present study aims to test the possibility of whether a drone can be trained using deep RL algorithms to find a 3D path through various 3D obstacles within the Airsim environment. The first one is the woodland, called the Modular package, in which trees are randomly distributed along the pathway. The second one is a custom-made block world in which differently shaped 3D objects, such as cone, cube, and sphere, are distributed, and two wall-style obstacles are installed along the potential paths as shown in Figure 1. The third is an arena world wherein the drone needs to navigate the various obstacles used for the block world.