## 3 Methodological Approach
### Algorithms
#### 3.1.1 Ppo  
PPO [29] follows the actor-critic architecture and it is based on the trust-region policy optimisation (TRPO) [30] algorithm. This algorithm aims to learn a stochastic policy \(\pi_{\theta}(a_{t}|s_{t})\) that maps states with Gaussian distributions over actions. In addition, the critic is a value function \(V_{w}(s_{t})\) that outputs the mean expected reward in state \(s_{t}\). This algorithm has the benefits of TRPO and in general of trust region based methods but it is much simpler to implement it. The intuition behind trust-region based algorithms is that, at each parameter update of the policy, the output distribution cannot diverge too much from the original distribution.  
To update the actor's parameters, a clipped surrogate objective is used. Although another loss function is also proposed, using a Kullback-Leibler (KL) divergence [31] penalty on the loss function instead of the clipped surrogate objective, the experimental results obtained are not as good as with the clipped one. Let \(r_{t}(\theta)\) denote the probability ratio defined in Equation (1), so that \(r_{t}(\theta_{old})=1\).  
\[r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{t })} \tag{1}\]  
\(\theta_{old}\) is the actor's parameter vector before the update. The objective of TRPO is to maximise the objective function \(L(\theta)\) defined in Equation (2). Here, \(\mathbb{E}[...]\) indicates the average over a finite batch of samples. The usage of the advantage \(\hat{A}_{t}\) in policy gradient algorithms was popularised by [PERSON] et al. [32] and indicates how good the performed action is with respect to the average actions performed in each state. To compute the advantages, the algorithm executes a trajectory of \(T\) actions and computes them as defined in Equation (3). Here, \(t\) denotes the time index \([0,T]\) in the trajectory of length \(T\) and \(\gamma\) is the discount factor.  
\[L(\theta)=\mathbb{E}\bigg{[}\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old} }(a_{t}|s_{t})}\hat{A}_{t}\bigg{]} \tag{2}\]  
\[\hat{A}_{t}=-V_{w}(s_{t})+r_{t}+\gamma r_{t+1}+...+\gamma^{T-t+1}r_{T-1}+ \gamma^{T-t}V_{w}(s_{T}) \tag{3}\]At each policy update, if the advantage has a positive value, the policy gradient is pushed in that direction because it means that the action performed is better than the average. Otherwise, the gradient is pushed in the opposite direction.  
Without any constraint, the maximisation of the loss function \(L(\theta)\) would lead to big changes in the policy at each training step. PPO modifies the objective function so that penalises big changes in the policy that move \(r_{t}(\theta)\) away from 1. Maintaining \(r_{t}(\theta)\) near to 1 ensures that, at each policy update, the new distribution does not diverge to much from the old one. The objective function is defined in Equation (4).  
\[L^{CLIP}(\theta)=\mathbb{E}[min(r_{t}(\theta)\hat{A}_{t},clip(r_{t}(\theta),1 -\epsilon,1+\epsilon))\hat{A}_{t}] \tag{4}\]  
Here, \(\epsilon\) is a hyper-parameter that changes the clip range.  
To update the value function \(V_{w}(s)\) (the critic), the squared-error loss function is used (Equation (5)) between the current state value and a target value. The target value is defined in Equation (6).  
\[J(w)=(V_{w}(s_{t})-V_{t}^{targ})^{2} \tag{5}\]  
\[V_{t}^{targ}=\hat{A}_{t}+V_{w}(s_{t}) \tag{6}\]  
The PPO algorithm is detailed in Algorithm 1. Although this algorithm is designed to be able to have multiple parallel actors getting experience, only one actor is being used.  
```
1:for\(e\in\)episodesdo
2:for\(a\in\)actorsdo
3: Run policy \(\pi_{\theta_{old}}\) in environment for T time-steps
4: Compute advantage estimates \(\hat{A}_{1}\)...\(\hat{A}_{T}\)
5:endfor
6: Optimise actor's loss function \(L^{CLIP}\) with regard to \(\theta\), with \(K\) epochs and minibatch size N \(\leq T\cdot\)actors
7:\(\theta_{old}\leftarrow\theta\)
8:endfor
```  
**Algorithm 1** Proximal Policy Optimisation (PPO).