# Reinforcement Learning for Semi-Active Vertical Dynamics Control with Real-World Tests
## 7 Summary and Outlook  
In this work, we covered the whole RL controller design process for the semi-active vertical dynamics control problem. We derived a QVM-based enhanced training model, whose structure and parameters were obtained by optimization based on measurement data. We showed that our modeling approach approximates the real measurement data better than a standard QVM approach. Additionally, we derived a damper model and parametrized it by optimization on measurement data.  
The obtained models were then utilized to train an RL-based control policy. We proposed one way of incorporating different objectives into the reward function, which included the consideration of penalizing big jumps in the damper force. This consideration was based on previous real-world observations in which high damper force jumps induced undesirable bump sounds. After the training, all obtained controller variants were compared in an agent performance assessment, and the best was selected for further evaluation.  
The selected agent was then validated and tested in a high-fidelity FVM simulation setup. After the assessment proved the integrity of the trained controller, the agent was subject to quantitative and qualitative real-world tests. To ensure a fair comparison, a benchmark controller was optimized in simulation. The quantitative evaluation on a real-world four-post test rig revealed that the RL-based controller was able to outperform the offline-optimized benchmark controller on road-like excitations, improving the comfort criterion by about 2.5% and the road-holding criterion by about 2.0% on average. An additional qualitative driving assessment on real roads showed that the RL-based controller, together with a feed-forward module responsible for handling the driver, induced disturbances and ensured safe body control and safe driving stability even when extreme road excitations occurred. Future work will address the improvement of the RL-based controller on rough road excitations.  
**Author Contributions:** Conceptualization, J.U. and J.B.; methodology, J.U., J.R., A.P., J.B., T.K. and J.T.; software, J.U., T.K., A.P., J.R., J.T. and J.B.; validation, J.U., J.B., J.T., T.K., J.R. and A.P.; data curation, J.U., T.K. and J.R.; writing--original draft preparation, [PERSON], [PERSON], A.P., [PERSON], T.K. and [PERSON]; writing--review and editing, [PERSON], [PERSON], A.P., J.T., T.K. and J.B.; project administration, [PERSON], J.B. and J.U.; funding acquisition, [PERSON], A.P. and J.B. All authors have read and agreed to the published version of the manuscript.  
**Funding:** This research was partially funded by German Federal Ministry of Education and Research (BMBF, project _KIFAHR: KI-basierte Fahruverksregelung_, grant number 01 IS20010A). Additionally, the authors received DLR basic funding.  
**Data Availability Statement:** The original contributions presented in the study are included in the article, further inquiries can be directed to the corresponding author.  
**Acknowledgments:** The authors would like to thank KW automotive GmbH for excellent cooperation during the project. We want to highlight the support provided in the experiments with the test vehicle on the four-post testbench as well as the measurements of the individual components. Additionally, we want to thank KW for equipping the vehicle and their general expertise on vertical dynamics. We especially want to thank [PERSON], [PERSON], [PERSON], [PERSON], and [PERSON]. Our thanks also go to our colleague [PERSON] for supporting the preparation and execution of the real-world experiments. Additionally, we want to thank our former colleagues [PERSON], [PERSON], and [PERSON], who contributed to the project with their preliminary work.  
**Conflicts of Interest:** The funders had no role in the design of the study, in the collection, analyses, or interpretation of data, in the writing of the manuscript, or in the decision to publish the results.