# Deep Reinforcement Learning Algorithms for Path Planning Domain in Grid-like Environment
## 4 Results
### Learning Process
#### 4.1.1 A2c  
As an initial implementation, the A2C implementation was trained significantly longer to ensure its progress. The final results of that are presented below, in Figure 4. In order to achieve readability in Figure 4, each point of the line is created by averaging using the sliding windows technique--taking an average from multiple points behind and after the current one. In that figure, as well as in all of the figures from the A2C section, the window size is set to 10,000, meaning that each point is an average from 5000 previous and 5000 next episodes. \begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Hyperparameter** & **A2C Value** & **PPO Value** & **Description** \\ \hline Actor learning rate & \(1\times 10^{-4}\) & \(4\times 10^{-5}\) & The learning rate used in actor learning process \\ Critic learning rate & \(1\times 10^{-5}\) & \(1\times 10^{-4}\) & The learning rate used in the critic learning process \\ Batch size & 30 & 30 & The number of memory chunks used for a single iteration of a learning process \\ Priority batch size & 2 & 2 & The number of high reward chunks in the batch \\ Priority batch ISWeight & 0.03 & 0.03 & The parameter decreasing the magnitude of the loss from priority samples. \\ Discount & 0.97 & 0.97 & The factor responsible for setting the ratio between importance of current vs future reward \\ Target network update frequency & 50 & 50 & The number of learning episodes before resetting the critic target network parameters \\ Actor previous policy update frequency & - & 50 & The actor network responsible for evaluating ’old’ probabilities \\ Memory size & \(2^{19}\) & \(2^{19}\) & The size of memory buffer \\ Priority memory size & \(10^{5}\) & \(10^{5}\) & The size of priority memory buffer \\ Clip ratio parameter & - & 0.2 & The value of clip parameters (\(\epsilon\)) used in Equation (4) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Hyperparameters used during the learning. **Performance**  
The first interesting take-away from the graph is the fact that the first 10,000 episodes performed outstandingly well compared to the numerous episodes ahead. This is the result of PyTorch model construction, which initialises new models in such a fashion that regardless of the inputs, the outputs tend to be almost identical, creating a nearly uniform random exploration policy at the very beginning of learning. As the parameters get slightly pushed into the direction of any action, the performance drops drastically, as the random exploration gets reduced to the part of the map, to which leads the most probable action currently; thus reducing the number of fields that are likely to be visited from the entire map, to some part of the map towards the preferred direction. This creates the illusion of errors in the learning process; however, it is just a misunderstanding, as the initial performance was not good at all; it just could lead to decent results due to environment mechanics and luck. Despite the initial drop of the score, the graph clearly represents the mountain climbing process--small yet persistent steps toward better results and self-improvement. There are visible up and downs, especially in the middle part of the training. However, the trend is persistently increasing. Such behaviour is very much expected when it comes to reinforcement learning, which often requires some significant amount of time to achieve even decent results. Another set of data gathered during the experiment is the total loss of the batch sampled for the learning process, both for critic and actor. As can be clearly seen in Figure 5a,b, curves presented by those do not differ much. However, it is important to recognise that those similar curves represent different behaviour. The decrease of the critic losses means approaching the optimum--the initially random policy is performing better and better at guessing the rewards yielded from state transformations.