# Obstacle Avoidance Drone by Deep Reinforcement Learning and Its Racing with Human Pilot
## 4 Method  
### Methods for Discrete Action Space  
Airsim is a new simulation platform from Microsoft for autonomous vehicles such as drones and autonomous cars. For the graphics, it uses the Unreal Engine that provides rich repertoires of shader and other drawing tools with which third party suppliers or research users can make the realistic landscapes. One of the early releases is called as \"modular package\", mainly consisting of the urban landscape and woodland areas. In particular, the present study utilizes the woodland area as it provides an ideal woodland landscape for testing and training an autonomous drone, as shown in the top of Figure 1. In this case, the drone supposes to avoid the trees and find a 3D path to the goal. It is found in the early stage that the small branches and leaves of the trees were not recognized as the obstacles, unlike the main branch, because the default option was set for the simple collision case presumably because the processing time for detail graphics is to save within Unreal Engine [48]. Therefore, it required to bond the small brunches and leaves to the main branch of a tree as one object and to work in the complex collision mode. Though the woodland provides a stimulating environment for an agent, it does not require much moving up or down direction simply because most of the trees are planted on a plain ground. Therefore, we have designed more challenging landscapes, called the block world and the arena, consisting of diverse solid objects, such as a cube, sphere, cone and other two wall-style objects such as half-wall and arch bridge as shown in Figure 1. Given that most of the drones that use SLAM for their autonomous navigation typically adopt either stereovision or Lidar, we utilize RGB and depth sensors for our experiment. The sensed images are given to two CNNs as input to the deep RL network. The size of the memory buffer is set to 1000, which means 1000 pairs of depth and RGB images are stored and sampled for training. DQN, Double DQN, Dueling DQN, DD-DQN algorithms are used for training and evaluating, respectively, in the environment of two different worlds: the woodland and the block world. For both cases, an agent learns to act with 5 actions, consisting of \(forward\), \(left\), \(right\), \(up\), and \(down\). After choosing an action, we use Euler angle \(pitch_{action}\), \(roll_{action}\), and \(throttle_{action}\) values to control the drone. Therefore, \(Pitch_{drone}\), \(Roll_{drone}\), \(Throttle_{drone}\) commands sent to set the drone's attitude are  
\[ \begin{split} roll_{drone}&=roll_{action}*max_{angle} \\ pitch_{drone}&=pitch_{action}*max_{angle}\\ throttle_{drone}&=throttle_{drone}+ throttle_{action}\end{split} . \tag{21}\]\(pitch_{action}\) is set to \(+1\) for \(forward\) and \(0\) for other \(4\) actions. \(roll_{action}\) is set to \(+1\) for \(right\) and \(-1\) for \(left\). \(throttle_{action}\) is set to \(0.2\) for \(up\), and \(-0.2\) for \(down\), respectively. The above \(roll_{drone}\), and \(pitch_{drone}\) are converted to Quaternions and then sent to the drone to match the pose, whereas \(throttle_{action}\) is sent to the drone directly. ROS was used for transmission from the RL agent to Airsim environment. Although we empirically chose \(max_{angle}\) and \(throttle_{action}\) for our environments, one can manually set those values depending on environments. For the woodland, reward \(r\) is set to \(0.1\) only when the agent chooses to go forward and \(-10\) when the agent collides with any obstacle recognized as an object. For example, every tree, leaves, and rocks are recognized as objects. For the block world case, \(r\) is set to \(0.1\) when the agent chooses to go forward, \(-10\) on a collision, and \(0.08\) when it makes an action such as left, right, up and down, respectively. Because the block world has a more complicated environment than the woodland case, the agent has to choose various actions to find a path and to prevent a deadlock: a reward is given to each action, i.e., promotion of exploration.