# Deep Reinforcement Learning Algorithms for Path Planning Domain in Grid-like Environment
## 3 Materials and Methods
### Advantage Actor-Critic and Proximal Policy Optimisation
The Actor-Critic methods utilise two function approximators; therefore, there exist two solutions for that situation--a classic single-headed neural network for each of the modules, or more novel, a single two-headed neural network in which actor parameters are more imperative. The latter promises a faster learning process due to the smaller number of parameters, and thus, a lesser amount of computation is required with the learning cycle, yet they couple the critic and actor results together, making them partially dependent [10]. Due to that, in this article, the classical single-headed approach is used. The parameters used for the neural networks are presented in Table 1. Both of the networks are visualised in Figure 1. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Value** & **Actor Network** & **Critic Network** \\ \hline Input layer & & \\ Size & 102 & 102 \\ Activation & LeakyReLU & ReLU \\ Hidden layer I & & \\ Size & 512 & 512 \\ Activation & LeakyReLU & ReLU \\ Hidden layer II & & \\ Size & 64 & 32 \\ Activation & LeakyReLU & ReLU \\ Output layer & & \\ Size & 4 & 1 \\ Activation & LogSoftmax & Linear \\ Optimizer & Adam & Adam \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Neural networks parameters. Figure 1: The layout of the networks used in this paper. Both of the networks consist of two hidden layers, with a similar number of nodes. The critic network uses half of the nodes in the second layer, due to the singular output and linear activation. In the actor network, the activation function of the layers was chosen to be a leaky ReLU, in order to allow for some small negative values to be passed on to allow discouraging of some actions. In both of the implementations, the neural networks used for the function evaluations consisted of two hidden layers. The first is of significant size to ensure that the number of parameters for processing the above-average large-state vector would be sufficient. The second hidden layer has different sizes for actor and critic networks due to the linear activation of critics output, which usually works better with a smaller number of parameters due to fewer variations. However, the main difference between the actor and critic is utilising the LeakyReLU instead of a ReLU activation function in actor networks. In the actor's case, the LeakyReLU was more suitable since due to negative reinforcements of some actions, the negative values are more likely to occur, while most of the critic values are somewhat positive due to the minor death penalty. Nevertheless, both of those are commonly used in deep reinforcement learning [52; 53]. The actor's output layer is utilising Softmax activation in order to provide the outputs in the form of a stochastic probability distribution rather than a deterministic decision. The networks and their gradients were implemented and calculated with the usage of the PyTorch library, as its functionality suited better the reinforcement learning approach. The ability to manipulate the losses before the backpropagation aligned with the gradient descent of the Advantage Actor-Critic optimisation step (Equation (1)) in which the loss from the policy is multiplied by the expected advantage provided by the Critic network. The optimiser chosen for both networks is Adam, which, due to its robustness, is one of the most commonly used optimisers. As powerful as it is, deep learning comes with some drawbacks. Most importantly, in terms of reinforcement learning, the data's high sequentiality impedes the learning process drastically. Since the data used in reinforcement learning are strictly sequential--one state derives from another with minor changes--additional measures have to be implemented. To cope with that, the idea of replay memory has been introduced [54; 55; 56; 57]. Instead of instantly analysing the samples, every action is recorded and stored in the memory buffer, from which randomly shuffled batches of samples are retrieved whenever the learning takes place.