# Obstacle Avoidance Drone by Deep Reinforcement Learning and Its Racing with Human Pilot
## 5 Experiment
### Network Training for Discrete and Continuous Action Spaces
#### 5.2.2 Training for Continuous Action Space  
This experiment was made to see if actor-critic networks described in Section 4 for continuous action space can maximize the reward. As the reward is accumulated larger as an agent avoids obstacles, such a reward can indicate that a drone has an obstacle avoidance capability. For this purpose, starting points for the training were set as identical locations as training for discrete action spaces in 3 environments, respectively. After the setting, processes for training actor-critic networks and U-Net followed the typical RL process. Since U-Net already plays a role in semantic segmentation, and its role is in a way similar to convert RGB to the depth map, only RGB image was used for the learning in continuous action spaces. With the same specification of U-Net, 3 actor-critic algorithms, such as TRPO, PPO, ACKTR, were adopted to compare their performance for obstacle avoidance drones. Figure 8 shows how our U-Net segmentation model produces the output as the training proceeds. Decrement of U-Net loss, as well as increment of reward, indicates that the agent's performance improves. As a result, it can make a successful path even in a newly configured environment. Note that when the drone saw a different scene for the first time, the segmentation map produced by U-Net was blurry on the given scene. However, by repeating trial and error made by actor-critic networks, the U-Net loss decreases as the networks learn the surrounding area. After the loss became stable by learning many scenes indicated by training steps around 55 and 78 in Figure 8 (left), the agent obtained a high reward score. It shows that our agent became familiar with these obstacles through trials and errors. The segmentation model learned by knowing where the agent had to go with the given optical flow and the feedback from the critic net, whose role was to evaluate how good the action was given the state. The obtained rewards by the actor network also increased as the U-Net loss decreased, indicating that the segmentation model and actor-critic networks cooperated to avoid  
Figure 6: A drone pilot navigates through 3D objects within the block world where an Radio Controller(RC) and a PX4 are connected to Airsim, playing a Hardware-In-The-Loop (HITL) mode. Note that the subject sees the first person view of the block world and performance of human is measured during the navigation task. obstacles and made a successful path. Figure 9 shows how the actor network made flight trajectory given segmentation outputs. The second experiment was to determine which actor-critic based algorithm performed better on the given task. It was conducted by replacing the actor-critic algorithm one by one, whereas the U-net and reward scheme was identical. As shown in Figure 10, 3 recent and high-performing  
Figure 8: U-net training and its corresponding outputs. As the training makes a progress, the U-net loss decreases and the reward obtained by an RL (Kronecker-Factored Trust Region (ACKTR)) agent increases (**Left**). Three different points during the training period are selected to illustrate how the segmented outputs from U-Net become clearer as training makes a progress (**Right**). The three locations, A, B, and C, selected along with a drone flight within the maze, are also shown in Figure 12a. Figure 7: Illustration of episode rewards for four deep RL algorithms with RGB (**left**), depth (**middle**), and RGB + depth (**right**), respectively, in 3 environments such as the woodland (**top**), the block world (**middle**), and the area world (**bottom**). The reward was the biggest when both RGB and depth input were combined with DD-DQN, as shown in (c) of the woodland and the block world. However, since these algorithms could not complete their races in the arena world, the differences between them were not significant. Implementations for these 4 algorithms were based on OpenAI [45]. actor-critic algorithms, TRPO [10], PPO [11], and ACKTR [43] were evaluated.