# Deep Reinforcement Learning Algorithms for Path Planning Domain in Grid-like Environment
## 3 Materials and Methods
### Advantage Actor-Critic and Proximal Policy Optimisation  
The Actor-Critic is a family of three similar yet somehow distinctive approaches, considered currently to be the most effective reinforcement learning approaches [8; 10; 43]. Those are Actor-Critic (AC), Advantage Actor-Critic (A2C) and Asynchronous Advantage Actor-Critic (A3C). The names are derived from the fact that the agent consists of two value functions, one designed to evaluate the policy and the other for grading the partaken actions. Such separation of the functionality allows the agent to learn stochastic policies, which tends to work better than the deterministic value-based approach [8; 43]. While actor function learns how to act in the environment, the critic learns to evaluate their actions. They both support each other in their learning processes, as improvements of the critic lead to better decision making, which leads to the discovery of new states as situations not known previously to the critics, thus improving his perception and evaluation. To improve that process even further, the Advantage Actor-Critic approach learns the actor by evaluating the advantages--changes of \"quality\" of state transformation--rather than the actions themselves. This increases the stability of the learning by the introduction of a baseline and simplifying the working principle of critic from the evaluation of the actions with respect to the state into the evaluation of the state itself. Lastly, the A3C approach improves the data gathering with asynchronicity--while the actor and critic are learned, the data is gathered with the usage of old parameters, which are updated periodically. This increases the speed, especially in the scenarios in which the data gathering process is time-consuming yet might hinder the process, as the actor is acting based on the outdated parameters most of the time. The A2C approach was chosen over the A3C (Asynchronous Advantage Actor-Critic) since the main improvements provided by the A3C revolve around increasing the speed and efficiency of data gathering, which was not a problem in this particular scenario, and the possible drawbacks might overthrow the gains. The formula for the parameters update is described by Equation (1) [8; 10; 44]:  
\[\theta^{\prime}\leftarrow\theta+\alpha\bigtriangledown_{\theta}log(\pi_{ \theta}(S_{t}|a))*A(S_{t},S_{t+1}) \tag{1}\]  
where:  
\[A(S_{t},S_{t+1})=V(S_{t+1})-V(S_{t}), \tag{2}\]  
and:  
\(\theta\)--parameters of the policy network,  
\(\alpha\)--actor learning rate,  
\(\bigtriangledown\)--gradient with respect to the parameters,  
\(log(\pi_{\theta}(S_{t}|a))\)--probability of taking action \(a\) in state \(S_{t}\) according to the current policy \(\pi_{\theta}\),  
\(V(S_{t})\)--value of a state \(S\) at time \(t\),  
\(A_{t}\)--estimated advantage at time \(t\). The actor uses gradient ascent in order to minimise the inverse of the reward function--hence making the designed positively rewarded actions more probable. To ensure that the learning process is robust versus the possible randomness of the rewards, the value function's yield is multiplied by the probability of taking such an action. If the action is considered to be impaired by the actor--and thus, has a small probability of being chosen--the loss is significantly decreased in order to ensure that the step taken in this unlikely direction is cautious. If, however, the action turns out to be desired, with the increase of certainty for choosing this action, the update step increases as well. That formula ensures that even if a destructive action yields good results--due to sheer luck or as a result of previously taken actions, for example--it will not destabilise the learning process as much as it might. On the other hand, the critic learning process usually follows the temporal difference algorithms.