# Reinforcement Learning for Semi-Active Vertical Dynamics Control with Real-World Tests
## 3 Modeling and Parameter Optimization of the Training Model
### Quarter-Vehicle Modeling
We assumed with respect to \(l_{\text{bw}}\) linear varying transmission ratio  
\[i_{j}=i_{\text{a},j}+i_{\text{b},j}\,l_{\text{bw}}, \tag{3}\]  
which yielded  
\[l_{j}=i_{\text{a},j}\,l_{\text{bw}}+\frac{1}{2}i_{\text{b},j}l_{\text{bw}}^{2 }+l_{\text{bw},0}. \tag{4}\] \begin{table}
\begin{tabular}{c c c c} \hline
**QVM Structure Name** & **Nonlinear Spring/Damper Transmission** & **Topmount Bushing as Linear** & **Engine Mass with Linear** \\  & **Transmission** & **Spring/Damper Element** & **Spring/Damper Bearing** \\ \hline Simple QVM & ✗ & ✗ & ✗ \\ \hline Transmission QVM & ✓ & ✗ & ✗ \\ \hline Topmount QVM & ✓ & ✓ & ✗ \\ \hline Engine QVM & ✓ & ✗ & ✓ \\ \hline Topmount engine QVM & ✓ & ✓ & ✓ \\ \hline \end{tabular}
\end{table} 
Table 2: Properties of the different QVM model structures considered for comparison. All model variants from Table 2, even the more complex ones, are simplifications of the real-world system. Therefore, selecting the model's parameters as free parameters for the optimization-based parameter estimation offered the possibility to compensate for model mismatches. The optimization algorithm can achieve this by altering the physical parameters of the QVMs to match the measured real-world dynamics. Limiting the parameters within specific appropriate ranges by means of constraints prevented the optimization of non-physical solutions. Since it was not clear which excitations were best suited for the parameter optimization process, different subsets were used. In the following, we formalized the approach to identify the parameters of different model structures. We assumed to have \(n_{\mathrm{c}}>0\) measured excitation scenarios with input vectors \(u^{i}\) (in our case, post position and damper current) and the corresponding output vectors \(y^{\mathrm{ref},i}\in\mathbb{R}^{n_{\mathrm{y}}}\) (\(i=1,\ldots,n_{\mathrm{c}}\)): in our case, the dynamic wheel load, the vertical wheel acceleration, and the vertical vehicle body acceleration. The time duration of each scenario is represented by \(T_{i}\). For each model type \(m=1,\ldots,n_{\mathrm{M}}\) (\(n_{\mathrm{M}}>0\)), we could simulate model outputs \(y^{m}\in\mathbb{R}^{n_{\mathrm{y}}}\) depending on a vector of model parameters \(p^{m}\), the input signals \(u^{i}\), and the time \(t\). We split the components of the parameter vector \(p^{m}\) into two groups, namely, \(p^{m,k}\) and \(\hat{p}^{m,k}\) (\(k=1,\ldots,n_{S^{m}}\)). Here, \(n_{S^{m}}\) is the positive number of different parameter splits for the model \(m\). The first part of the split represents the free parameters to be identified by optimization; the second part collects the parameters that have directly assigned values from other sources. For a model \(m\) with a parameter split \(k\), we select a non-empty index subset \(I\subset I_{\mathrm{c}}:=\{1,\ldots,n_{\mathrm{c}}\}\) to define which set of scenarios is used for the identification of parameters \(p^{m,k}\). The following optimization problem focuses on the minimization of the time domain error between measured signals and simulated ones:  
\[\min_{p^{m,k}\in B^{m,k},i}\sum_{\in I}\sum_{j=1}^{n_{\mathrm{y}}}w_{ij}\int \limits_{0}^{T_{j}}\left(y^{m}_{j}\left(p^{m,k},\hat{p}^{m,k},u^{i}(t),t\right) -y^{\mathrm{ref},i}_{j}(t)\right)^{2}dt. \tag{5}\]  
The vector of free parameters \(p^{m,k}\) is constrained by a box \(B^{m,k}:=\left\{p^{m,k}\middle|\underline{p}^{m,k}\leq p^{m,k}\leq\overline{p} ^{m,k}\right.\right\}\) for fixed lower limits \(\underline{p}^{m,k}\) and upper limits \(\overline{p}^{m,k}\). The parameter subset \(\hat{p}^{m,k}\) is kept constant. The deviations between model outputs and measured outputs are summed up using positive weights \(w_{ij}\) (\(i=1,\ldots,n_{\mathrm{c}};\ j=1,\ldots,n_{\mathrm{y}}\)) to reflect the different scaling of physical variables. The numerical solutions of each optimization problem are named \(p^{m,k}_{I}\).