# Reinforcement Learning for Semi-Active Vertical Dynamics Control with Real-World Tests
### Related Work
Both used a linear QVM to train the controller and assess the trained agent in simulation. The contributions of [14; 15] applied the proximal policy optimization (PPO) DRL algorithm to vehicle vertical dynamics control. Both contributions trained the agent on a linear QVM that was extended by a friction term resulting in a nonlinear model. In ref. [15], the authors trained the agent on a single road bump excitation, while it seems in [14] the agent was trained on artificial road excitations. The work of [14] compared the trained agent in a simulation against a passive- and a fuzzy-based control strategy and was able to show some improvements for different road types and vehicle loading conditions. A simulative evaluation in [15] was conducted on a bump excitation and showed that the PPO agent was able to minimize the momentum of the unsprung wheel mass but concurrently increased the momentum of the body mass. In ref. [16], the ability of RL to handle an uncertain delay of the input was investigated. The authors applied the twin delayed DDPG (TD3) algorithm to a linear QVM extended by an input delay. Even though an input delay violates the Markov assumption that is assumed in the theoretical analysis of RL algorithms, the authors were able to show an advantage of their agent compared to a pure passive suspension setup in a simulative assessment. Compared to a DDPG agent, the TD3 agent proposed in the contribution outperformed the DDPG agent in some simulative scenarios. The comparative study in [17] benchmarked different DRL algorithms with different state-of-the-art vertical dynamics controllers. To train the agents, the authors used a nonlinear two-mass oscillator with the damping coefficient as input and a first-order actuator dynamics. The authors found that the use of trust region policy optimization (TRPO) with generalized advantage estimation (GAE) yielded a close-to-optimal policy and was advantageous in comparison with other RL algorithms. The simulative assessment on the QVM was supported by further investigation of the selected agent on a nonlinear full-vehicle model. In ref. [18], different DRL algorithms were trained on a nonlinear full-vehicle model excited by synthetical road profiles. An evaluation of different DRL algorithms showed that the soft actor-critic (SAC) algorithm performed best in their setting. The authors proposed a dual approach in which one agent is trained to cope with the road excitation according to [19] and another agent is trained for impulse excitations. Additionally, an impulse detector algorithm was developed, based on which one of the two agents was selected for application. The proposed approach was evaluated in simulation and showed an improvement in the comfort criterion compared to an SH controller, two different MPC versions, and a passive setup. Additionally, the RL-based agent that was trained for handling the road excitations was tested on a real test circuit. In these real-world tests, the SAC agent was able to outperform the SH controller and a passive setup on the RMS of the vertical acceleration. In our work, we describe the derivation of the training model in great detail. Additionally, instead of choosing to test the agent in a real-road setting, we evaluated our trained agent on a four-post test rig. This way, we were able to not only measure the chassis vertical acceleration as a basis for the comfort criterion but were also able to measure the wheel loads and could derive the road-holding criterion from these measurements.