# Reinforcement Learning for Semi-Active Vertical Dynamics Control with Real-World Tests
## 4 Training the Controller
### Application to the Vertical Dynamics Problem
#### 4.2.1 The Training Setup  
The QVM models developed and analyzed in Section 3 serve as a basis for the simulation-based training environment. We utilized Modelica/Dymola to develop and optimize the models, but most RL libraries interact with the environment via the OpenAI gym [34] interface, which is implemented in Python. It has to be mentioned that the _gym_ interface has now further evolved to _gymnasium_[35], but the main features remain unchanged or are only altered slightly. Since the training can most easily be conducted in a Python environment, the Modelica models were integrated as an FMU into Python. In this work, we used an improved version of the FMU-Python toolchain previously applied in [36; 37] to accomplish this integration. In addition, we utilized the RL baselines Zoo [38] to facilitate the saving of the agent and to track the hyperparameters used during each training.  
The aim of the training was to train _one_ agent to handle road-induced disturbances, which would then be applied to _each_ vehicle corner. Previous investigations showed that training separate agents specifically for each vehicle corner induced undesired pitch and roll oscillations for symmetric pure vertical excitations on the four-post test rig. Even though all posts performed the exact same motion in these tests, minor differences in the policy resulted in different damper forces at each wheel and, therefore, undesired pitch and roll motions occurred. In order to enable the agent to perform well on the different vehicle sides, we trained the controller sequentially on the three QVMs described in Section 3. In addition to changing the vehicle model during training, we exposed the agent to different road excitations. Based on previous experience and to cover a wide range of excitation profiles, we trained on sweep, bump, and wave excitations. The sweep excitations were chosen for the training to ensure that the eigenfrequencies of the wheel and the chassis were adequately excited during training. The bump and wave excitations posed challenging events, which frequently occur in the real world and also excite different modes of the vehicle.  
In this work, we used a SAC implementation from the stable-baselines3 library [33]. The SAC is an actor-critic off-policy RL algorithm that augments the standard expected sum of rewards optimization objective with an information theoretical entropy term of the policy [31]. This entropy term favors stochastic policy behavior and is weighted against the expected sum of rewards objective by a so-called temperature parameter. Thus, the exploration vs. exploitation dilemma of RL is directly addressed in the objective function. The SAC is designed for continuous action spaces and has successfully been applied to a wide range of control problems, e.g., [37; 39]. Additionally, a comparison of different DRL algorithms on the vertical dynamics control problem conducted in [18] showed that the SAC algorithm performed best in their setting. Therefore, we chose this algorithm for the vertical dynamics application.  
The environment was implemented with a sample time of 1 ms and trained with various hyperparameters. The training with different hyperparameter sets and reward function designs was performed parallelly on a DRL in-house computing cluster, where one training with 70 million timesteps took around 24 h. Since only small neural network sizes were used within the RL algorithm, the training was solely performed on the CPU, as the training on the GPU did not increase training speed. During each training, the agent was saved in regular time intervals. This way, we were able to restore performant agents, even in cases where catastrophic forgetting [40] occurred. As described in Section 4.2.4, each stored agent was benchmarked after the training and the most performant agents were selected for evaluation in the real-world experiment. The hyperparameters used to train the most performant agent are summarized in Appendix A.3 in Table A.3.