# AidA*-RL: Anytime Graph-Based Path Planning with Deep Reinforcement Learning for an Autonomous UAV
## 2 Proposed iADA*-RL Planning Algorithm
### RL Agent Training  
The network architecture for the DRL agent, for both DQN and DDPG, used in this experiment is shown in Figure 6. The network takes a \(30\times 100\) state representation, current position, and velocity as inputs. The first layer convolves 32 kernels of \(8\times 8\) with stride three with the input image, with an Exponential Linear Unit (ELU) value of 0.8, and uses a Rectified Linear Unit (ReLU) activation function. The second convolutional layer obtains 64 kernels of \(4\times 4\) of stride two and an ELU value of 0.8, and is again followed by a rectifier. The third convolutional layer convolves 64 kernels of \(3\times 3\) of stride one, followed by a rectifier, while the output from the fourth layer is flattened and concatenated with the position vector. The combined inputs are fed to the two fully connected layers, where the first layer includes 256 hidden units and the second 128 hidden units. The action space determines the number of hidden units in the last fully connected layer. In our experiment, we used eight action spaces. Action space: The action space is a set of all actions that the agent can act out in a certain environment. It consists of eight discrete actions that can be taken to avoid obstacles. One involves moving in the direction of the current heading, with a speed of 1, 2, 3, or 4 m/s, and another involves moving backward along the current direction with a speed of 1 m/s. A further two actions involve rotating to the right and to the left directions by 30 degrees. The remaining two actions involve moving to up and down positions, respectively, with a speed of 2 m/s. At each time step, the policy is used to choose the observation space as the input, and the output is one of these eight actions. These high-level actions are mapped to the low-level flight command using the flight controller provided by AirSim. State space: The state space is a set of all the states that the agent can transition to and use in the training and learning process of the agent. The state of the input used for the  
Figure 5: Environment used to test for dynamic obstacle avoidance [30]. Figure 6: Network architecture for the DQN and DDPG agents. input layer is configured using (i) depth image data; (ii) relative goal position, and current velocity as follows. 1. Depth image data: The state representation must provide the information needed by the agent to avoid real-time obstacles. To avoid these obstacles, the agent receives a pre-processed depth image from the drone's front camera, and follows and returns to the planned path and waypoints provided by the iADA* global path planner. The agent also receives the current position and state of the drone. The depth image shown in Figure 7 shows the distances to the surfaces of the obstacles seen by the camera, mapped to grey-scale, in an image of size \(256\times 144\) pixels. In this experiment, we applied a basic image processing step to reduce the dimensionality of the input, in order to reduce the computation and memory requirements. A drone is typically moving along a fixed plane, and the middle section of the image is therefore most relevant. The size of the pre-processed image used is \(100\times 30\) pixel sizes. 2. Relative goal position and current velocity: The relative global position is a 3D vector that represents the goal in polar coordinates (distance and goal) with respect to the current position of the UAV vehicle. The distance \(s_{\text{distance}}\) and the direction to the destination are calculated using Equation (2): \[s_{\text{distance}}=\left\|\rho\right\|=\left\|P_{\text{current}}\right.-P_{ \text{goal}}\left\|\right.\] (2) Finally, the reward used in the learning process is set using Equation (3), following the approach used in [2]. When the UAV reaches the destination, an arrival reward is given. In this study, the condition \((s_{\text{distance}}<0.5)\) is regarded as representing arrival at the destination. When colliding with obstacles such as walls, other UAVs or cars, a collision reward is given.