## 5 Reinforcement Learning Algorithms
### Proximal Policy Optimization  
Proximal policy optimization is a reinforcement learning algorithm that directly optimizes a policy without making significant changes to parameters due to high advantage estimates that could induce unstable behavior. In addition, it maintains a good balance between exploration and exploitation phases. For this purpose, the basic formulation of PPO includes an objective function to optimize, including clipping operations to ensure that no drastic changes are made. The so-called surrogate objective function of PPO can be expressed as  
\[L^{CLIP}(\theta)=\mathbb{E}_{t}\big{[}\min\big{(}r_{t}(\theta)\hat{A}_{t}, \text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\big{)}\big{]} \tag{10}\]  
where \(L^{CLIP}(\theta)\) is the objective function. The parameters in the vector \(\theta\) are adjusted to maximize this objective while preventing large updates through clipping. This function takes into account the ratio between the new policy and the old policy, aiming to improve the current policy without making drastic changes. \(\mathbb{E}_{t}\) is the expectation over time \(t\), i.e., an average over the various time steps considered in the training process; \(r_{t}(\theta)\) is the probability ratio (importance ratio) between the new policy \(\pi_{\theta}\) and the old policy \(\pi_{\theta_{\text{old}}}\):  
\[r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\text{old}}}(a_{ t}|s_{t})} \tag{11}\]  
This ratio indicates how much the new policy differs from the old policy for a given action \(a_{t}\) in a state \(s_{t}\). \(\hat{A}_{t}\) is the estimated advantage at time \(t\), representing the difference between the actual return and the estimated value of the state. It indicates how good the action was compared to the average; \(\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\) is a clipping function that limits the ratio \(r_{t}(\theta)\), to keep policy changes under control. The interval \([1-\epsilon,1+\epsilon]\) limits how much the new policy can deviate from the old policy; \(\epsilon\) is a hyperparameter that determines the width of the clipping. Typical values are small, such as 0.1 or 0.2.  
The loss of the critical value function can be expressed as  
\[L^{VF}(\theta)=\frac{1}{2}\mathbb{E}_{t}\bigg{[}\big{(}V_{\theta}(s_{t})-V_{ t}^{\text{target}}\big{)}^{2}\bigg{]} \tag{12}\]  
This function measures how much the value estimate of the state by the model \(V_{\theta}(s_{t})\) differs from the target value \(V_{t}^{\text{target}}\). The latter is the estimate of the value of state \(s_{t}\) given by the value function parameterized by \(\theta\); \(\frac{1}{2}\) is a scaling factor for the quadratic loss used to compensate with the derivative coefficient.  
The total combined loss that is minimized during training by its empirical average is  
\[L(\theta)=L^{CLIP}(\theta)-c_{1}L^{VF}(\theta)+c_{2}S[\pi_{\theta}](s_{t}) \tag{13}\]  
where \(c_{1}\) and \(c_{2}\) are coefficients that balance the three loss terms, and  
\[S[\pi_{\theta}](s)=-\sum_{\theta}\pi_{\theta}(a|s)\log\pi_{\theta}(a|s) \tag{14}\]is the entropy term that encourages additional exploration. Entropy measures the uncertainty or randomness of the policy, encouraging exploration. Increasing entropy can help avoid overfitting and promote the discovery of new strategies. These parameters and terms are designed in order to gradually improve the policy while maintaining training stability and efficiency. The actual values used in our experiments were the default values of the library Stable-Baselines3, reported in Table 1: