## 2 Materials and Methods
### Background  
RL is a machine learning technique in which an _agent_ interacts with its _environment_ by making observations, taking _actions_ based on those observations, and receiving _rewards_ that reflect the resulting _state_ changes. The sequence of actions chosen by the agent represents the so-called _policy_, and the final goal of the agent is to find the best policy, namely the one that maximizes the _cumulative reward_ obtained through the sequence of actions taken in various states. Indeed, while the reward associated with each action \(a\) performed in a given state \(s\) represents the goodness of \(a\) in the short term, the cumulative reward represents the goodness of the entire strategy in the long term. In this case, the learning process is based on the continuous interaction of the agent with the environment. Reinforcement learning (RL) has seen remarkable progress in recent years, especially with the advent of deep reinforcement learning (DRL), in which neural networks are used to approximate the functions needed to perform the training and identify the best policy. Several variants of DRL have been proposed in the literature, differing in how they model agent-environment interactions and compute the optimal policy efficiently. In this paper, we use a method known as the Proximal Policy Optimization (PPO) algorithm [43]. PPO works by iteratively improving the policy through a trial-and-error approach through which it trains a stochastic policy \(\pi_{\theta}\), allowing the agent to explore different actions and observe their outcomes. During each training cycle, the agent collects fresh trajectories (i.e., sequences of states, actions, and rewards) using the current version of the policy. These recent trajectories are then used to update the policy in a way that maximizes the expected reward. As the number of interactions increases, the agent's actions become progressively less random and more accurate. PPO is a first-order optimization method for RL that balances simplicity, stability, and performance. Unlike standard policy gradient methods, which suffer from high variance and instability, like the Deep Q-Learning (DQL) method [18] or Trust Region Policy Optimization (TRPO), which requires complex constrained optimization, PPO avoids large, destabilizing policy updates by introducing a clipping mechanism in the objective function. In particular, a clipping region is used to restrict the difference between the new and old policy to a small, controlled range. This _proximal_ update ensures that learning remains stable and does not diverge, which is especially beneficial in complex environments or in the presence of limited training data. PPO is implemented with an actor-critic architecture. The actor network (or policy network) selects actions to take, while the critic network (or value network) estimates state values to compute the goodness of the choices taken. Figure 4 shows the architecture of the PPO algorithm using an actor-critic approach. As illustrated in the figure, the system operates in a continuous loop with the aim of refining both the actor's decisions and the critic's evaluations. It includes the following main components:  
1. _Environment interaction and experience collection_--Given the modeled environment \(E\), the action model takes an action \(a_{t}\) in \(E\), obtaining in response from \(E\) the following _experience tuple_: \(\langle s_{t},a_{t},r_{t},s_{t+1}\rangle\), where \(s_{t}\) is the current state, \(a_{t}\) is the action taken,\(r_{t}\) is the reward received by performing \(a_{t}\) in \(s_{t}\), and \(s_{t+1}\) is the reached state. This interaction continues for a fixed number of steps, generating a sequence of experiences, also called a trajectory of experiences. 2. _Rollout storage (trajectory memory)_--The sequence of collected _experience tuples_ are temporarily stored in a component called _rollout storage_. This component acts as a buffer that holds the sequence of experiences generated by the agent's current policy \(\pi_{\theta}\).