## 3 Methodology
### Proximal Policy Optimization  
As mentioned previously, Proximal Policy Optimization (PPO) [36; 37] is an algorithm based on the actor-critic framework. An actor neural network is utilized to generate actions, and a critic is used to evaluate the performance of the actor. Reinforcement learning algorithms are divided into on-policy and off-policy. On-policy means that the agent used to take samples is the same as the training agent. Off-policy is different, which has changed from \"learning by doing by yourself\" to learning through the experience of others so that the samples can be reused. As an off-policy algorithm, PPO first uses an actor for data collection to generate sample sets. After that, the actor and critic are updated multiple times in a loop. The samples used for each update are randomly sampled again from the sample set. After multiple updates are completed, a new round of data collection through the latest actor begins. PPO implements off-policy through importance sampling. We can calculate the expected value of a distribution of the function \(p(x)\) that is difficult to take samples, by sampling another arbitrary distribution \(q(x)\). Importance sampling uses important weight \(\frac{p(x)}{q(x)}\) as a coefficient for each datum to correct the difference between the two distributions. Finally, the expectation function of \(q(x)\) is substituted by the following formula:  
\[E_{x\sim p}[f(x)]=E_{x\sim q}[f(x)\frac{p(x)}{q(x)}] \tag{13}\]Theoretically, \(q(x)\) can be any distribution. However, if the difference from the original distribution is too large, the variance of the sample will be too large, even though the expected value is the same. If the sample is not sufficient, the deviation may be unacceptably large. The nature of importance sampling dictates that, if the difference between two distributions is too large, more samples are needed to ensure the reliability of the results. Therefore, the algorithm needs to tightly control the difference between the two distributions, i.e., the gradient of each update of the neural network. KL scatter is one of the most common methods used to measure the difference of the distributions. Therefore, the most common method is to add a KL constraint to the objective function, and then use the Lagrangian pairing method to pass the constraint to the objective function, which is the core idea of PPO. Since the PG algorithm is on-policy, the actor can only be updated once after each sampling. This is because the objective function needs to be substituted for the latest actor sample for calculation. Through importance sampling, PPO uses samples of actors that are not the latest to replace the samples of the latest actor, allowing multiple updates after each sampling. The nature of importance sampling determines that if the difference between the two distributions is excessively large, more samples are needed to ensure the reliability of the results. Therefore, it is necessary to strictly control the difference between the two distributions, that is, the gradient of each update of the neural network. The KL divergence is one of the most common methods used to measure the difference in distributions. Therefore, the most frequent method is adding KL constraints to the objective function. After adding some other simplification techniques, this becomes the Trust Region Policy Optimization (TRPO) algorithm. Nevertheless, even if it has been simplified, TRPO still has a huge amount of calculations. Because the calculation of adding constraints is very complicated, a common method is to use the Lagrangian pairing method to pass the constraints to the objective function, which is the core idea of PPO. PPO is divided into two ways, as follows:  
* when using the dual Lagrangian method, a dynamically changing \(\beta\) value is used to constrain the update speed. The objective function is as follows: \[