# Reinforcement Learning for Semi-Active Vertical Dynamics Control with Real-World Tests
## 4 Training the Controller
### Application to the Vertical Dynamics Problem
#### 4.2.3 Reward Function Design
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Reward Weights** & \multicolumn{2}{c}{**Reward Terms**} \\ \hline - & - & \(r_{\text{f}}\) & Force jump reward term \\ \hline \(k_{\text{cm}}\) & Chassis motion weight & \(r_{\text{cm}}\) & Chassis motion reward term \\ \(k_{\Delta u}\) & Control signal jump weight & \(r_{\Delta u}\) & Control signal jump reward term \\ \hline \(k_{\text{a}}\) & Control signal weight & \(r_{\text{a}}\) & Control signal reward term \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Weights and terms used in the reward function. with the parameters \(k_{\rm\tilde{f}}\), \(\theta_{v_{\rm d}}\), and \(\theta_{Au}\) and the function \(\text{clip}(\ldots,0.0,1.0)\), which limits the value range to [0.0, 1.0]. The force jump reward term \(r_{\rm\tilde{f}}\) takes the damper velocity \(v_{\rm d}\) and the demanded current jump \(\Delta u\) as input. It has to be noted that \(r_{\rm\tilde{f}}\) can take values between 0 and 1. This means that the whole reward in Equation (7) is 0 if \(r_{\rm\tilde{f}}=0\). If \(r_{\rm\tilde{f}}=1\), then the other terms of the reward function come into effect. This way, the force jump reward term is prioritized over the other terms, since it is able to scale the other terms. This kind of hierarchical reward function design was adopted from our previous works [36, 37]. Even though the main objective was to minimize the chassis motion, it was necessary to give the force jump reward term such a prominent position within the reward function. Adding the force jump reward term to the weighted sum in Equation (7) did not have a significant effect. The shape of the force jump reward term is depicted in Figure 8. It can be seen that the function value becomes 0 if a large current jump is commanded at high damper speeds. From the damper characteristics in Figure 3, it is evident that a big jump in the current yields a high jump in the damper force if applied at high damper velocities. We encouraged the agent to avoid high force jumps by shaping the force jump reward term as proposed in Equation (9) and depicted in Figure 8.  
We make use of the Gaussian-like function \(g_{\sigma}(x)\) in the following, which we define as  
\[g_{\sigma}(x)=e^{\frac{-x^{2}}{2\sigma}} \tag{10}\]  
As observed in our previous works on applying RL, we found this function to be helpful within the design of the reward function. The function has two helpful properties, which we make use of: First of all, its values are limited: \(0<g_{\sigma}(x)\leq 1\). This way the contribution of each term to the reward function is normalized and the different terms can be weighted by additional multipliers. The second advantage is that in addition to \(x=0\), it has a non-zero gradient, which can guide the training. The chassis motion term \(r_{\text{cm}}\) is designed to reward a minimized vertical chassis motion. Therefore, we designed the chassis motion term as  
\[r_{\text{cm}}(v_{\text{c}})=0.8\cdot g_{v_{\text{c}_{1}}}(v_{\text{c}})+0.2\cdot g _{v_{\text{c}_{2}}}(v_{\text{c}}) \tag{11}\]  
with the two chassis motion reward term parameters \(v_{\text{c}_{1}}\) and \(v_{\text{c}_{2}}\). It should be noted that the two terms in Equation (11) are weighted such that \(r_{\text{cm}}(v_{\text{c}})\) is limited between 0 and 1. The combination of the two Gaussian-like functions in that way essentially broadens the peak compared to a single Gaussian-like function. In line with the results in [9], we found that designing the comfort reward term based on the chassis vertical velocity yielded better results in contrast to using the chassis vertical acceleration. The next term in the reward function, i.e., the control signal jump reward term \(r_{\Delta u}(\Delta u)\), encourages a smooth action signal. To achieve this, we rewarded a small deviation between the commanded current setpoint and measured current via the Gaussian-like function defined in Equation (10). \[r_{\Delta u}(\Delta u)=g_{\sigma_{\Delta u}}(\Delta u), \tag{12}\]  
with the parameter \(\sigma_{\Delta u}\).