# AidA*-RL: Anytime Graph-Based Path Planning with Deep Reinforcement Learning for an Autonomous UAV
## 2 Proposed iADA*-RL Planning Algorithm
### Learning Phase  
The proposed autonomous mission planning system for an UAV combines a global path planner based on an iADA* graph with local planning based on learning between the nodes of the global path planner. The global planner and the local behaviors acquired through learning must be prepared in advance, and the system provides initial learning results in order to acquire these local behaviors. In this section, the training flow is summarized and the result of the training network is also presented. For the training phase, a series of processes from action selection to learning rates are repeated. This series of proposes is called _'step'_. An 'episode' refers to a series of UAV steps from the initial position to the destination position, or a position at which the UAV hits obstacles. In the training processes, episodes including several steps are looped. The integration of the initial start and goal positions of the UAV are randomly selected at the beginning of each episode. In the training process, the environment also including both static and dynamic moving obstacles. All the dynamic moving obstacles are moving with different speeds and directions randomly. In the action output selection, the index number of actions \(i\) is obtained on \(\epsilon\)-greedy algorithm. An \(\epsilon\)-greedy selects an action that takes the maximum Q-value in the current state \(s_{t}\) or a random action with a higher probability at the beginning of the learning process. As a result of action \(a_{t}(i)\), reward \(r_{t}\) and judgment \(d_{t}\) and the next UAV's state \(s_{t=1}\) are obtained. Then, these are added to experience memory. Learning is performed by retrieving randomly from the experience memory and updating the unit weights of the neural network by the error back propagation algorithm. From these processes, the Q-value of the action which should take in the \(s_{t}\) is updated to become higher. In this process, the UAV learns the behaviors needed to move towards the destination by avoiding obstacles along the path. The destinations for the learning stages were set randomly. The training process in the experiment took around four days to finish the training. Figures 8 and 9 show the accumulated rewards and losses applied during the training of the DQN network, and it can be seen that the rewards change from \(-150\) to \(120\) after 200,000 steps. This is a way to check that the agents effectively get better. In the current training result, the accumulated reward of the Q-learning does not finish as a convergence when the experiment stops the training, and we stopped the training after we received our goal reward value \(100\). The goal for the reward value is decided when the mission success cases achieve more than \(70\%\) of running the experiments test cases in the simulation. This means that the UAV can learn how to arrive at the destination and to avoid dynamic obstacles within a low number of training steps. In Figure 8, the orange line represents the moving average trend-line of the DQN model.