# Reinforcement Learning for Semi-Active Vertical Dynamics Control with Real-World Tests
### Related Work  
An early application of RL to the vehicle suspension control is documented in [8]. In this contribution, the authors used a stochastic continuous action RL automata algorithm to obtain the parameters of a simple linear control law. The training was performed in a real-world setting on a four-post test rig with the aim of minimizing the root mean square (RMS) of the chassis acceleration. In the approach presented in the paper, the RMS of the chassis acceleration could be minimized in comparison with a fixed damper characteristic. In contrast to our approach, the policy utilized in the abovementioned paper was a pure linear control law and the RL algorithm could only adjust three parameters per vehicle corner. We utilized an ANN as a policy that enabled the RL algorithm to obtain a much more sophisticated control law. In ref. [9], the authors applied a Batch RL algorithm to the semi-active damping of a vehicle. In their contribution, the RL algorithm was trained in a simulation on a QVM. The applied RL algorithm used tree-based regression methods as function approximators. Additionally, the authors selected a discrete action space, i.e., the RL algorithm could only choose between minimum and maximum damping. The goal of the approach was to maximize ride comfort. To achieve this, the authors compared three different reward functions: one was based on the vertical chassis acceleration, one was based on the vertical chassis velocity, and one was based on the vertical chassis displacement. The authors found that the velocity-based reward function performed best. Simulative assessments showed that the trained controller was able to outperform their benchmark controller, a combined SH- and acceleration-driven damping controller (SH-ADD controller, c.f. [4]), in low frequencies, but performed slightly worse on higher frequencies. In contrast to [9], we used a deep RL algorithm and utilized a more sophisticated training model. Additionally, we validated our trained controller in real-world tests, which was not performed in [9]. An explicit model predictive control (MPC) approach was presented in [10]. In this contribution, the authors designed an MPC for a nonlinear QVM incorporating nonlinear axle kinematics, damper friction, and nonlinear damper characteristics. The MPC was then solved offline for various sampled points from the state space and the result was stored. After that, an ANN was trained such that a damper input could be obtained by evaluating the ANN with measurement inputs. Experiments on a quarter vehicle test rig and in a full vehicle simulation showed the benefit of the proposed approach compared to a combined SH/GH controller. In general, the explicit MPC approach was tractable as long as the state space was small. The computational effort grew exponentially with the number of states and, thus, became computationally intractable very fast. Additionally, it was not straightforward to generate the sample pattern for the state space. On the one hand, the samples should cover all regions that might be encountered during operation. On the other hand, oversampling increases the computational cost and, thus, should be kept at a minimum. In the RL approach presented in the contribution at hand, no sample grid had to be selected manually. The authors of [11] proposed a sequential learning algorithm to iteratively optimize the parameters of a predefined policy. In contrast to standard DRL, the proposed algorithm used a policy that is parametrized as a quadratic function of all available measurements. The authors showed that their approach was superior to a linearized variant of the skyhook control algorithm in real-world tests. In addition to the non-standard DRL approaches to tackle the vertical dynamics control problem via the learning-based approaches listed above, many applications of standard DRL algorithms can be found in the literature. In refs. [12; 13], the deep deterministic policy gradient (DDPG) RL algorithm was applied to the vertical dynamics control of a QVM.