# Reinforcement Learning for Semi-Active Vertical Dynamics Control with Real-World Tests
## 2 The Vertical Dynamics RL Controller Design Process  
There are two main options for training an RL controller: training in simulation or training directly on the real plant, i.e., the test vehicle. The benefit of training in the real world is that no simulation model is needed and that no model-reality mismatch (sim-to-real gap, see [21]) can occur. Nevertheless, ensuring safety can pose a challenge. Additionally, the training is restricted to real time. In contrast to that, training in simulation is fast, scalable, and safe. Simulation models often can be simulated faster than real time and the training can be parallelized on high-performance computing systems, which provides an additional speed-up. Because of these advantages, training in simulation is often preferred. In some applications, the agent trained in simulation is afterwards trained on the real system to overcome the sim-to-real gap. In order to train a vertical dynamics controller directly on the vehicle, it is necessary to excite the vehicle with defined vertical excitations. Since this is not possible in normal road traffic, a so-called four-post test rig, as depicted in Figure 1, is required for real-world training. However, the experiments on the four-post test rig and equipping the vehicle on the test rig present a considerable effort. Starting directly with real-world training is unfavorable for the vertical dynamics control problem, since the training process may have to be repeated several times to find a robust training setup. Due to these limitations, we chose to train the controller in simulation. To narrow the sim-to-real gap, the training model structure as well as the parametrization were optimized with respect to extensive measurement data. The controllers were developed for and tested on the AI For Mobility (AFM) research platform [7], a test vehicle at the German Aerospace Center that enables the investigation of AI-based control methods. A hybrid production vehicle served as a test platform. To provide the data required for AI methods, the vehicle was equipped with an extensive range of sensors that could record numerous quantities. A complete by-wire control system enabled automated driving and the reproducible testing of controllers. Additionally, a rapid control  
Figure 1: The DLR’s test vehicle AFM on a four-post test rig (adopted from [20]). prototyping (RCP) system allowed control algorithms to be tested directly on the AFM. Furthermore, the AFM was equipped with a custom-made semi-active damper system, which is described in more detail in Section 3.2. Despite the modifications, the vehicle was approved for use on public roads and fulfilled the driving dynamics characteristics of a normal production vehicle. In Figure 2, the RL-based controller design process for vertical dynamics control applied in this work is depicted. The process applied in this work started by generating measurement data for the whole vehicle. Additional measurement data were collected for the semi-active hydraulic damper and additional components. In a second step, component models and several different QVM structures were generated. The obtained measurement data were then used to parametrize the models through optimization. Since the damper was the central component that converted the control signal into forces, special attention was paid to the measurement and modeling of the damper. Deriving an accurate training model is a crucial task for the RL training process. Since the controller adapts its behavior from interaction with the simulation model, the performance of the resulting controller in the real-world application relies on the accuracy of the model. For the modeling, we used the multi-domain modeling language Modelica [22] together with the integrated development environment Dymola. The most accurate QVM structures were then selected and exported as functional mock-up units (FMU) [23].