# Deep Reinforcement Learning Algorithms for Path Planning Domain in Grid-like Environment
## 4 Results
### Performance Comparison  
Finally, in order to compare the performance of the algorithms, the algorithms were tested by running 10,000 iterations of the simulation. During a gathering of those data, the algorithms were no longer learning--which can be observed in the increase of their performance compared with the learning processes, as the lack of policy modification during the run increased the stability of the decision-making process. In order to ensure that the random parts of the environments remain exact, the map generation function was fixed by the usage of a preset list of 10,000 different seed values--ensuring that the generation of the quest locations would differ between the iterations yet remain the same while testing different algorithms. Since the A2C implementation learning process was longer, this implementation was tested twice--after 1 mln learning episodes (the same number as PPO) and after 1.5 mln learning episodes (total number of the learning episodes). The results are shown in Table 3 and described in greater detail in the following sections. The values presented in the table are obtained by averaging 10,000 samples to ensure that they present the trend and are robust against the variation of individual samples. As mentioned, the results presented above are visibly better than the results presented during the learning process--especially in the case of PPO. This results from a lack of  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Parameter** & **Nr of Iteration** & **PPO** & **A2C** \\ \hline Rewards & 1 mln & 2.03 & 0.46 \\  & 1.5 mln & - & 0.73 \\ Certainty & 1 mln & 44.49\% & 84.49\% \\  & 1.5 mln & - & 87.75\% \\ Actions & 1 mln & 56.47 & 37.76 \\  & 1.5 mln & - & 48.3  
variation in the policy resulting from online learning--as the policy is updated through the episode, it creates variation during a single playthrough, thus decreasing performance. Nevertheless, it is vital to keep in mind that both algorithms are still far from converging to the optimum--much larger learning would be required to obtain the desired solution. Those results, however, are enough to observe that the algorithms, especially PPO, are relatively quick in increasing their performance, and with sufficient learning time, they would be a suitable solution to the problem presented in this paper. The results presented above clearly visualise the assumed benefits of the PPO algorithm. It can be observed that the average reward obtained during an iteration (representing the number of completed tasks) is more than four times better after the same number of learning iterations. An additional 500,000 iteration does improve the A2C performance, yet it is still far from the one achieved by the PPO algorithm. The main reason for that is visible in the second row of the table, describing the mean probability of taking the actions--the agent certainty through the game. It can be observed that the A2C agents have much higher certainty. This can be correlated to previously presented Figure 6a where it is shown, how quickly the certainty reaches high values--around 90% in the first part of the learning. It is important to note that we should not expect the high values of the certainty at this part of the learning--as clearly visible from the rewards and number of the iteration, the algorithms are yet very far from converging. This high certainty of the A2C algorithm is actually a problem rather than a virtue, leading to a decrease of the exploration and increasing the variance of the learning process (as higher certainties produce higher loss values). On the other hand, the certainty of the PPO implementation does variates but oscillates on much lower levels (Figure 9a) which is beneficial to a learning process and caused by the diminishment of the loss values due to clipping.