# Reinforcement Learning for Semi-Active Vertical Dynamics Control with Real-World Tests
## 3 Modeling and Parameter Optimization of the Training Model
### Quarter-Vehicle Modeling
In our approach, we selected a few parameter splits and some scenario subsets for the five models in Table 2. We selected all the parameters that were not measurable as free parameters within the optimization. In contrast to that, we kept all the parameters constant that were either measurable with a high certainty or resulted from previous investigations, e.g., the damper parameters obtained in Section 3.2. Measured parameters with a low certainty were kept constant in one of the first splits and selected as free parameters for the optimization in other splits. Since a large scenario subset means simulating the model many times with different input excitations in each evaluation of the optimization cost function, it was important to select only the most relevant excitations for the parameter identification. In our application, the sine sweeps showed to have the most information included. Thus, these excitations were preferred to define the optimization problems. We used the Optimization Library [26] and additionally extensive scripting in Dymola, an integrated development environment for Modelica, to set up and solve the problems automatically. The computations were executed on an in-house computing cluster using the parallelization features of the Optimization Library. After solving the optimization problems, we finally obtained a list of identified parameter sets \((p^{m,k}_{I},\hat{p}^{m,k})\) with a split \(k\) for the models \(m\) and excitation scenario subsets \(I\). To evaluate each of the solutions, we computed separate error metrics for each of the components \(y^{m}_{j}\) for the whole scenario set \(I_{\mathrm{c}}\). This meant simulating the models parametrized by the identified parameters with all the test scenarios. Based on qualitative and quantitative comparisons of these results, we selected the best model structure with its identified parameters (cf. Table A1 in the Appendix A). Since the body acceleration was only measured at the front left (FL), front right (FR), and the rear left (RL) but not at the rear right (RR) corner of the vehicle, we performed the optimizations only for the three vehicle corners where the full measurement data were available. The optimal model structures revealed by the optimization were the \"Engine QVM\" variant for FL and FR and the \"Topmount QVM\" for the RL side of the vehicle (cf. Table 2). This aligned very well with the structure of the vehicle: The AFM was equipped with uniball topmounts, which acted as an almost rigid link between the damper and the body. Additionally, the vehicle's engine was located in the front of the vehicle. This made the model variant \"Engine QVM\", which included an engine mass but no elastic topmount, a plausible choice for the front axle. In contrast to that, a standard elastic topmount was installed on the rear axle. Additionally, no big oscillating mass, like the engine in the front, was present in the rear. Therefore, the model variant \"Topmount QVM\" seemed an adequate choice for the rear left side of the vehicle. To validate the obtained parametrized QVMs, we excited the optimized models with one selected road profile (exponential sweep, 1-30 Hz, 100 mm/s zero-crossing velocity). This excitation was also applied during the measurements of the vehicle performed on the four-post test rig. In addition, a standard linear two-mass quarter-vehicle model equipped with the nonlinear damper model described in Section 3.2 with an optimized parameter set was also simulated (compare simple QVM from Table 2).