## 3 Semi-Active Suspension Control Based on the PPO Algorithm
### PPO Algorithm Network Model  
The proximal policy optimization (PPO) algorithm is an actor-critic-based reinforcement learning algorithm, which combines a value-based approach and a policy-based approach [27]. The architecture of the algorithm consists of two parts: the actor network part is based on the policy approach, which is used to optimize the policy model \(\pi(s,a)\) by generating actions and interacting with the environment; the critic network part is based on the value approach, which is used to determine the merit of an action and select the next action accordingly to optimize the value function \(Q(s,a)\). The network structure of the PPO algorithm is shown in Figure 4.  
It contains four neural networks [28], which are the policy network, the target policy network, the value network, and the target value network. The network architectures of the policy network and the value network are the same as those of their corresponding target networks, and there are only differences in the network parameters. In the architecture of this control algorithm, the actor network obtains the control data based on the output state information of the suspension system, so as to obtain the action information damping force to optimize the entire algorithm. By adjusting different speeds and road bumps, the car's body acceleration and suspension deflection at the current moment are affected, resulting in the next new state.  
After a period of interaction between the agent and the environment, a series of trajectory data is generated, and the network parameters are updated by sampling the data stored in the memory network using a policy gradient. In the policy gradient (PG) [29] method, the goal is to learn a parameterized policy that determines a unique action or obtains a corresponding action probability density function directly from a selection in the state space. In the learning process, the strategy is optimized by calculating an estimate of  
Figure 3: The changing of the damping force of the semi-active suspension.  
the policy gradient, and the methods used are gradient up or gradient down depending on the expectation, respectively. The policy gradient function is shown in Equation (5):  
\[\
abla\hat{R}_{\theta}=E_{(s_{t},a_{t})\sim\pi_{\theta}}[R(s_{t},a_{t})\
abla \text{log}_{\pi_{\theta}}(s_{t},a_{t})], \tag{5}\]  
\[\begin{array}{ll}A^{\pi,\gamma}(s_{t},a_{t})&=E_{s_{t+1}}[r_{t}+\gamma V^{ \pi,\gamma}(s_{t+1})-V^{\pi,\gamma}(s_{t})]\\ &=E_{s_{t+1}}[Q^{\pi,\gamma}(s_{t},a_{t})-V^{\pi,\gamma}(s_{t})],\end{array} \tag{6}\]  
where \(\pi_{\theta}\) is a stochastic policy function and \(R(s_{t},a_{t})\) is the reward value of the current state action pair. \(V^{\pi,\gamma}(s_{t})\) is the state value function of the strategy \(\pi\) at the current moment. \(Q^{\pi,\gamma}(s_{t},a_{t})\) is the action value function of the strategy \(\pi\) at the current moment. \(A^{\pi,\gamma}(s_{t},a_{t})\) is an estimate of the advantage function at the moment \(t\)[30], containing a hyperparameter \(\gamma\), which is usually obtained by estimating the advantage function based on the value of the reward function, and then, the policy gradient is estimated according to the parameter \(\theta\) so as to optimize the policy with the policy gradient. If \(A^{\pi,\gamma}(s_{t},a_{t})>0\), this means that the current state action pair \((s_{t},a_{t})\) is able to obtain a higher reward than the baseline, and conversely, a lower reward is obtained.