# Deep Reinforcement Learning Algorithms for Path Planning Domain in Grid-like Environment
## 4 Results
### Learning Process
#### 4.1.2 Ppo  
The learning process of the agent with the usage of PPO was significantly shorter. However, it can be observed that the results are much more promising. The following subsections present those results and highlight the most important differences. Same as in the previous section, the graphs presented here are smoothed with the usage of the sliding window technique to ensure readability. **Performance**  
Figure 7 presents the reward values obtained during the PPO learning process. The first noticeable thing is the high spike right in the middle of the graph. A remarkable shift in most of the other learning parameters is noticeable around the same iteration, most due to exploding gradients, as both critic and actor present some drastic variations at that moment. Apart from that one shift in the performance, the graph's curve clearly shows another hill-climbing process--slowly yet steadily increasing the performance over time. Even after that sudden and abrupt drop, the agent is slowly coming back to his previous performance. Compared to the reward curve of A2C (Figure 4), this process is much faster and characterised by smaller variations. The final values obtained by the actor are higher than the ones from A2C implementation, despite the fact that the agent was trained less. The graphs (Figure 8a,b shows the values of the losses used for learning through the entire process. From the very first glance, it is visible how different they are from their counterparts (Figure 5a,b). The reason for that lies in as the priority memory buffer implementation guarantees that two high-value chunks--which are much more likely to generate high loss values--will be present in the learning batch. Due to that, dozens of very steep up-and-down movements are visible. Since the high-value samples could be completing a quest (0.9), starting a quest (0.1) and losing (\(-0.1\)), the uneven distribution of those causes formulation of such spikes on the graph; since with the priority batch of size two an even distribution is impossible to be achieved, such situation repeats itself a few times during the learning. It appears to give much higher variation in the PPO  
Figure 6: (**a**) The mean value of the probability of chosen action—describes the certainty of action taking decisions of the actor. (**b**) The percentage of invalid moves in the episode. The slope decrease clearly visualise steady progress in the decision-making process while still remaining slightly high due to non-deterministic policy. implementation; this is caused by the fact that absolute values of the critic losses are around four times smaller--and thus, they appear bigger. This could be prevented with the usage of a more sophisticated memory buffer and sampling method, yet the computational cost required to form and expand such data structure would cause more pain than gain. On the other hand, Figure 8b is characterised by the same spike in the middle, visible previously in Figure 7. As the performance of the actor dropped, the high reward samples became less likely to appear in the priority buffer, thus reducing the total loss values. **Movement during the Learning Process**  
As in the previous case, the data about the agent's behaviour during the learning process was gathered. The last two figures, Figure 9a,b, visibly differ from their counterparts (Figure 6a,b). The certainty, as previously, starts around 50%, yet it decreases rather than increases during the learning. This does not necessarily mean that the actor performs worse--as there always exist multiple viable paths, decreased certainty shows the increased level of spatial awareness of the actor, who can see the resemblance between paths and juggle between them, rather than follow a path from memory. Keeping the probabilities smaller than 50%, yet presenting the increasing trend, shows that the agent learning process is more optimal than before. Lastly, the percentage of the invalid actions is presented in Figure 9b.