## 4 Selection of Computational Intelligence and Reinforcement Learning Solutions
### Trust Region Policy Optimization and Proximal Policy Optimization  
Different from the off-policy nature that characterizes the DQN, Trust Region Policy Optimization (TRPO) is an on-policy RL algorithm that aims to identify the optimal step size in policy gradient descent for convergence speed and robustness purposes [45]. Specifically, TRPO searches for the best way to improve the policy by satisfying a constraint (called the Trust Region), which defines the highest accepted distance between the updated policy and the old one, thus tackling the problems of performance collapse and sample inefficiency typical of policy gradient RL algorithms. To do so, it defines the following surrogate objective:  
\[\max_{\theta}\mathbb{E}_{t}[\frac{\pi_{\theta}(A_{t}|S_{t})}{\pi_{\theta_{old }}(A_{t}|A_{t})}\hat{A}^{\pi_{\theta_{old}}}(S_{t},A_{t})] \tag{6}\]  
where \(\theta_{old}\) represents the vector of policy parameters before the update and \(\hat{A}^{\pi_{\theta_{old}}}\) is an estimator of the advantage function from the older policy \(\pi_{\theta_{old}}\). It is possible to prove that applying specific constraints to this equation, i.e., bounding the Kullback-Leibler divergence between \(\pi_{\theta}\) and \(\pi_{\theta_{old}}\), guarantees a monotonic policy improvement and allows the reuse of off-policy data in the training process, making TRPO more-stable and sample-efficient than previous policy-gradient-based RL algorithms [65].  
Unfortunately, since TRPO requires a constrained optimization at every update, it could become too complex and computationally expensive (Theoretically, it is possible to transform the constrained optimization step into an unconstrained optimization one through a penalty-based approach. However, in turn, this raises the need to identify a proper penalty coefficient \(\beta\) to consider--which is very difficult in practice). These drawbacks call for simpler and more-effective methods for policy gradient descent. Towards that goal, [PERSON] et al. in [66] introduce the Proximal Policy Optimization (PPO) algorithm. There are two main variants of PPO: PPO-Penalty and PPO-Clip [66; 67; 68]. PPO-Penalty optimizes a regularized version of Equation (6), introducing an adaptive regularization parameter \(\lambda\) that depends on \(\pi_{\theta}\). On the other hand, PPO-Clip calculates a clipped version of the term:  
\[\frac{\pi_{\theta}(A_{t}|S_{t})}{\pi_{\theta_{old}}(A_{t}|S_{t})}\]  
and considers as a learning objective the minimum between the clipped and the unclipped versions. This ensures that the update from \(\pi_{\theta_{old}}\) to \(\pi_{\theta}\) remains controllable, preventing excessively large parameter updates, which could cause massive changes to the current policy, resulting in a performance collapse. More specifically, PPO-Clip leverages a modified version of the surrogate function in Equation (6) as follows:  
\[\max_{\theta}\mathbb{E}_{t}[\min(\frac{\pi_{\theta}(A_{t}|S_{t})}{\pi_{\theta_{ old}}(A_{t}|A_{t})}\hat{A}^{\pi_{\theta_{old}}}(S_{t},A_{t}),clip(\frac{\pi_{ \theta}(A_{t}|S_{t})}{\pi_{\theta_{old}}(A_{t}|A_{t})},1-\epsilon,1+\epsilon) \hat{A}^{\pi_{\theta_{old}}}(S_{t},A_{t}))] \tag{7}\]  
where function \(clip(x,1-\epsilon,1+\epsilon)\) clips \(x\) within the interval \([1-\epsilon,1+\epsilon]\), with \(\epsilon\) being a hyperparameter that defines the clipping neighborhood.  
Thanks to a more-elegant and -computationally efficient behavior than TRPO, PPO is a particularly interesting solution for deep RL applications. PPO-Clip is arguably the most-interesting variant of PPO: it has proven to be remarkably simple and stable and to work consistently well in a wide range of scenarios, outperforming other algorithms, such as Advantage Actor-Critic [66]. It, thus, represents a very solid candidate to consider in our investigation. For simplicity, in the rest of the manuscript, we use the term PPO to refer to the PPO-Clip algorithm.