# Reinforcement Learning for Semi-Active Vertical Dynamics Control with Real-World Tests
###### Abstract  
In vertical vehicle dynamics control, semi-active dampers are used to enhance ride comfort and road-holding with only minor additional energy expenses. However, a complex control problem arises from the combined effects of (1) the constrained semi-active damper characteristic, (2) the opposing control objectives of improving ride comfort and road-holding, and (3) the additionally coupled vertical dynamic system. This work presents the application of Reinforcement Learning to the vertical dynamics control problem of a real street vehicle to address these issues. We discuss the entire Reinforcement Learning-based controller design process, which started with deriving a sufficiently accurate training model representing the vehicle behavior. The obtained model was then used to train a Reinforcement Learning agent, which offered improved vehicle ride qualities. After that, we verified the trained agent in a full-vehicle simulation setup before the agent was deployed in the real vehicle. Quantitative and qualitative real-world tests highlight the increased performance of the trained agent in comparison to a benchmark controller. Tests on a real-world four-post test rig showed that the trained RL-based controller was able to outperform an offline-optimized benchmark controller on road-like excitations, improving the comfort criterion by about 2.5% and the road-holding criterion by about 2.0% on average. reinforcement learning; vertical dynamics control; semi-active damping; FMI; Modelica 4.0.0000-0001-5002-0000]J.J.J.J., T.B., T.  
Despite the complex system dynamics, many vertical dynamics controllers are based on linear two-mass oscillators and incorporate other simplifications. The widely used skyhook controller (SH) introduced in [1] assumes a virtual control force acting on the vehicle body and neglects the induced force acting on the wheel. The groundhook control law (GH) introduced in [2] extends this concept by a virtual force acting on the wheel to control wheel oscillations. In ref. [3], the authors compare several semi-active suspension control methods on a linear two-mass oscillator vehicle model. Even though the incorporation of a first-order input dynamics results in a nonlinear quarter-vehicle model (QVM), a linear damper characteristic is assumed. In ref. [4], a simplified damper model is used during the controller design process and a more sophisticated damper model is used during simulative assessment. In contrast to analytically derived control laws, data-driven methods such as Deep Reinforcement Learning (DRL) have been increasingly used for solving complex real-world control tasks in recent years, e.g., for quadruped robots [5]. The increasing popularity of applying Reinforcement Learning (RL) methods to control problems is based on multiple advantages: Multi-physical simulation models can be directly used within the training process. Many analytically derived control laws rely on specific simplified model structures, e.g., linear models or state space models. Moreover, deriving analytical control laws is often a tedious and error-prone task that requires expert knowledge. RL methods are rather generic and can solve a wide range of control problems based on a repeated interaction with the system or a simulation model of the system. In contrast to first-generation RL algorithms, the increased popularity of RL is based on the use of artificial neural networks (ANN) as function approximators within DRL algorithms. The use of ANNs leveraged the performance of RL and enabled the applicability of RL algorithms to a wide range of sequential decision-making problems across different domains and research fields [6]. In this work, we applied RL to semi-active suspension systems for vertical vehicle dynamics control in the German Aerospace Center's (DLR's) research vehicle AI For Mobility (AFM) [7]. Since the training in simulation is fast, scalable, and safe, our approach relies on a simulation-based training strategy.