## 3 Methodology
### Proximal Policy Optimization
\begin{split} L^{KL}(\theta)=\hat{E}_{t}[\frac{\pi_{\theta}(a_{t }|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{t})}\hat{A}_{t}-\\ \beta KL[\pi_{\theta_{old}}(\cdot|s_{t}),\pi_{\theta}(\cdot|s_{t })]]\end{split} \] (14) where \(\pi_{\theta}\) means the newest actor stochastic, \(\pi_{\theta_{old}}\) means the newest strategy when taking samples, and \(\hat{A}_{t}\) is the estimator of the advantage function of time \(t\) based on the current critic value function \(V_{\varphi_{k}}\). During the training process, the value of \(\beta\) is changed depending on the expected value of the KL divergence. The formula is as follows: \[d=\hat{E}_{t}[KL[\pi_{\theta_{old}}(\cdot|s_{t}),\pi_{\theta}(\cdot|s_{t})]]\] (15) Let \(d_{targ}\) present the target value. If \(d\) is less than \(d_{targ}/1.5\), then \(\beta\) is reduced to a half. If d is greater than \(d_{targ}\times 1.5\), then \(\beta\) is doubled. Other conditions remain unchanged. * Use the clip function in Figure 3 to directly limit the difference between the output actions of the latest actor and the older actor. The probability ratio is as follows: \[r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{t})}\] (16) where \(\theta\) denotes the current parameters of the actor neural network; \(\theta_{o}ld\) denotes the parameters at the time of sampling; and \(\pi_{\theta}(a_{t}|s_{t})\) denotes the action probability of the actor output after inputting the observed states \(s_{t}\) in which the parameters are. It is worth mentioning that due to the use of the LSTM neural network, the current actor output is influenced by the previously inputted observation states in the same input sequence, corresponding to the learning environment as the learning trajectory and feedback from all previous times during the complete learning process of a course. Therefore, the closer \(r_{t}\) is to 1, the better. The reconstruction objective function is as follows:  
\[L^{CLIP}(\theta)=\hat{E}_{t}[min(r_{t}(\theta)\hat{A}_{t}, \tag{17}\] \[clip(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t})]\]  
where \(\epsilon\) is a custom constant, and the \(clip\) function limits the upper and lower bounds of the input. Of the two methods, the training process is the same. Only some functions have changed. The training process is shown in Figure 4. The calculation method of the advantage value is the same. We previously mentioned that \(\hat{A}\) is obtained based on the value function. The value function is defined by the critic neural network. By putting the time \(t\) and state \(s_{t}\) into the critic, we can obtain the value \(V(s_{t})\) at the current moment. The formula for calculating the advantage value is as follows:  
\[\hat{A}_{t}=\xi+(\gamma\lambda)\xi_{t+1}+\cdots+\cdots+(\gamma\lambda)^{T-t+1} \xi_{T-1} \tag{18}\]  
where \(\gamma\) means the discount factor, \(T\) means the total of time, \(\lambda\) means the GAE parameter, and \(\xi\) is calculated by the following formula:  
\[\xi=r_{t}+\gamma V(s_{t+1})-V(s_{t}) \tag{19}\]  
where \(r_{t}\) means the reward value given by the environment for the \(t\)-th action. \(v(s_{t})\) represents the evaluation value obtained when the input state is entered into the observed state \(s_{t}\) similar to the actor, and the current output is influenced by the previous input. Figure 4: PPO: (I) Sampling by actor; (II) the collected data are sampled again and passed to the actor and critic; and (III) updating through the two different loss functions. Figure 3: Clip function.