# Evaluation of Reinforcement and Deep Learning Algorithms in Controlling Unmanned Aerial Vehicles
## 6 Evaluation
### Result and Discussion  
Here, we discuss the results obtained from all three learning algorithms discussed above. For DQN, we use the RMSProp optimizer [12]. Figure 5 shows the moving average of the rewards that the agent received after completing an episode. The Euclidean distance reward function is used during this run. All three learning algorithms gained rewards as the number of episodes increased, in all directions. However, DQN showed significant improvement and had no negative moving average reward in any direction. Compared to SARSA, Q-learning has better performance in X-DDP and Y-DDP (Figure 5a,b, respectively). Nevertheless, both algorithms have a negative reward in Z-DDP (Figure 5c). This indicates that DQN has no problem in flying horizontally or vertically, while SARSA and Q-learning are able to fly horizontally but not vertically. As can be seen from the results, the change in reward after the 500 th episode is small. Hence, the rest of the evaluations are tested for 500 episodes only, whereas the reward function remains the Euclidean distance. \begin{table}
\begin{tabular}{l l l l} \hline \hline  & **Coordinates** & & \\ \hline \hline
**X** & **Y** & **Z** & **Description** \\ \hline
0.0 & 0.0 & 0.0 & IDP \\
9.0 & 0.0 & 1.0 & X-DDP \\
0.0 & 9.0 & 1.0 & Y-DDP \\
0.0 & 0.0 & 10.0 & Z-DDP \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Initial and desired positions. \begin{table}
\begin{tabular}{l l l} \hline \hline
**Parameter** & **Q-Learning and SARSA** & **DQN** \\ \hline Episode & 500–1000 & 500–1000 \\ Steps & 50–100 & 50–100 \\ \(\alpha\) & 0.1 & 0.00025 \\ \(\gamma\) & 0.9 & 0.99 \\ \(\epsilon\) & 0.9 & 1 \\ Memory Size & - & 1,000,000 \\ Network Input & - & 3 \\ Network Output & - & 8 \\ Network Structure & - & 300,300 \\ Update Target Network & - & 10,000 \\ Mini Batch Size & - & 128 \\ Learn Start & - & 128 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Simulation hyperparameters. Since DQN with RMSProp optimizer has better rewards, we analyzed the performance of DQN under other optimizers such as Adadelta [11], SGD [13], and ADAM [14]. Here, the agent is expected to reach the desired position in only 500 steps. In X-DDP and Z-DDP, the Adadelta optimizer has a better reward (Figure 6a,c). Even though the SGD optimizer showed a good reward in X-DDP, the result in Figure 6b,c shows that SGD has a negative reward. When the agent uses the RMSProp optimizer, it never outperforms other optimizers in any of the directions, whereas the agent shows good performance with the ADAM optimizer in all directions (Figure 6). In Figure 5, there are six results that obtained negative rewards; by dropping Q-learning and SARSA and adopting other optimizers, we reduced that number to three (Figure 6). Then, we replaced the reward function with MSE, which further reduced the negative reward results to two (Figure 7). Although all optimizers show a good sign in flying the quadrotor autonomously, Adam and Adadelta are the best optimizers in both horizontal and vertical desired positions. In addition, we can see from Figures 6 and 7 that the agent is always improving towards a positive reward when with the Adadelta optimizer. This shows the significant role that the optimizers and the reward functions play in flying the drone autonomously. Therefore, using the Adadelta optimizer and MSE reward function, we evaluated the performance of the DQN agent under different steps, that is, the maximum number of steps an agent can take between the initial and desired position. The results in Figure 8 indicate that by limiting the number of steps, the reward increases, which means that the agent performs better. This is due to the fact that the agent is not taking unnecessary actions that might lead to negative rewards. However, reducing the number of steps just to improve reward does not result in better performance, as there are small differences between 50 and 75 steps. In addition, realistic scenarios are not as simple as the simulation cases which can be reached in a few steps.