## 2 Materials and Methods
### Background
It is an essential element of the PPO because the algorithm uses this collected data for multiple learning updates before discarding them and collecting a new trajectory. 3. _PPO components: actor and critic models_--The PPO block contains the two core learning models. * _Actor model (or policy model)_: It can be considered as the agent's decision-maker. It receives information about the current state \(s_{t}\) and decides which action \(a_{t}\) to take. Its goal is to develop a stochastic policy \(\pi_{\theta}(a_{t}|s_{t})\), which is essentially a set of rules or probabilities that dictate the best action for any given state. Indeed, given the state \(s_{t}\) as input, it produces the probabilities of taking each different action \(a_{t}\). The parameter \(\theta\) represents the knowledge it has learned so far. * _Critic model (or value model)_: This acts as the agent's evaluator. It assesses how good a particular state \(s_{t}\) is, or how good a particular action \(a_{t}\) taken in a state \(s_{t}\) is. It learns to estimate the expected future reward (\(V_{\mu}(s_{t})\)) achievable from the current state \(s_{t}\). The parameter \(\mu\) represents the learned evaluation knowledge. 4. _Data preparation for learning_--Once an iteration has been completed, the two models can be updated. This update exploits the data collected inside the _rollout storage_ component and requires the computation of the following parameters: * _Policy ratio_\(r_{t}(\theta)\): For each action \(a_{t}\), the algorithm calculates the probability of taking that action under the current policy \(\pi_{\theta}(a_{t}|s_{t})\) and compares it to the probability under the old policy \(\pi_{\theta_{old}}(a_{t}|S_{t})\). This comparison produces the _importance sampling ratio (\(r_{t}(\theta)\))_, defined as follows: \[r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{t})}\] (1) This ratio is a key factor in PPO stability, ensuring that updates do not deviate too far from the old policy. * _Temporal Difference (TD) error_\(\delta_{t}\): The estimates provided by the critic network are used to calculate the TD error, which measures the difference between the expected return and the actual value observed. It is typically calculated as: \[\delta_{t}=r_{t}+\gamma\cdot V_{\mu}(s_{t+1})-V_{\mu}(s_{t})\] (2) where \(\gamma\) is the discount factor used to modulate the role of future rewards with respect to the current one. * _Advantage estimation_\(\hat{A}_{t}\): Using the TD errors, the _Generalized Advantage Estimation (GAE)_ is calculated. This provides a more accurate measure of how much better a chosen action was compared to the average expected outcome from that state. It is a crucial input for updating the actor model. 5. _Model updates (optimization)_--The prepared data is then used to update both the actor and critic models. A batch of samples, referred to as _mini-batch samples (length: U)_, is drawn from the _rollout storage_ for these updates. * _Critic model update_: The objective of the critic model is to minimize its _value loss_, denoted as \(L^{V}(\mu)\). This loss measures the squared difference between the predicted values (\(V_{h}(s_{t})\)) and the actual returns. The parameter \(\mu\) characterizing this network is then updated using its gradient, \(\
abla L^{V}(\mu)\), typically through an optimization algorithm like Stochastic Gradient Descent (SGD). * _Actor model update_: The objective of the actor model is to minimize its _clipped policy loss_, denoted as \(L^{CLIP}(\theta)\). This is the core of PPO. It uses the _importance sampling ratio_\(r_{t}(\theta)\) and the _advantage estimation_\(\hat{A}_{t}\) to construct a loss function that encourages beneficial actions while preventing large, destabilizing changes to the policy. In particular, the clipping mechanism within this loss computation ensures that the ratio \(r_{t}(\theta)\) always stays within a certain range.