# Reinforcement Learning for Semi-Active Vertical Dynamics Control with Real-World Tests
## 4 Training the Controller
### Application to the Vertical Dynamics Problem
#### 4.2.3 Reward Function Design  
The design of the reward function is a crucial degree of freedom in the application of RL. In general, RL algorithms are designed to maximize the expected discounted sum of future rewards. This implies that the reward should quantify desirable behavior, where favorable behavior leads to a higher reward compared to a smaller reward for undesirable behavior. The main objective in vehicle vertical dynamics control is usually to optimize ride comfort as well as road-holding [41]. However, these two objectives are, to some extent, opposing each other. This means that at a certain point enhancing one metric yields a corresponding deterioration in the other. Previous experience has shown that it is much harder for the RL algorithm to optimize for comfort than for road-holding. Therefore, we designed the reward function to include ride comfort but neglect road-holding in the reward function to simplify the problem for the RL algorithm. Nevertheless, we integrated a safety module into the whole vertical dynamics control system to ensure driving safety during real-world road tests. To enable the optimization of these objectives, a numerical quantification of the objective was necessary. In general, it is not trivial to measure ride comfort, since it describes a subjective perception. Different metrics to quantify ride comfort are compared in [42]. In this work, we assumed to enhance ride comfort by minimizing the vertical chassis motion. Even though ride comfort is usually calculated from chassis vertical accelerations, we found that including the chassis vertical velocity in the reward function leads to better results compared to using the chassis acceleration. In previous experiments, we noticed several undesired agent behaviors: First, the agent actions tended to be very jitter, meaning that the action signal was very noisy from one timestep to another. As a second issue, we observed that the policy induced high current steps during high damper velocities. Given the damper characteristics depicted in Figure 3, this led to a sudden change in the damper force, which resulted in undesirable noises and might have harmed the damper due to increased mechanical stress. Additionally, it is desirable to keep a low damping characteristic as a default. This way, the damper can absorb high-frequency road disturbances, e.g., those induced by driving over a cobblestone road. We designed the reward function such that (1) chassis motion was minimized, (2) high jumps in the damper force were avoided, (3) the action signal was smooth, and (4) the default damper current was low. To achieve this, we constructed the reward function out of different terms, each accounting for a different objective weighted by an accompanying weighting factor. The used terms and weights are listed in Table 3.  
The individual terms are assembled as shown in the following equation:  
\[r=r_{\text{f}}(v_{\text{d}},\Delta u)\cdot(k_{\text{cm}}\cdot r_{\text{cm}}(v_{ \text{c}})+k_{\Delta u}\cdot r_{\Delta u}(\Delta u)+k_{\text{a}}\cdot r_{\text {a}}(a)) \tag{7}\]  
Herein, the damper velocity is denoted as \(v_{\text{d}}\), the chassis velocity as \(v_{\text{c}}\), and the action as \(a\). \(\Delta u\) describes the demanded current jump \(\Delta u\), defined as  
\[\Delta u=a-i_{\text{d}}. \tag{8}\]  
Herein, the current jump \(\Delta u\) represents the difference between the commanded current \(a\) and the measured current \(i_{\text{d}}\). The reward function \(r\) consists of two factors: The force jump reward term \(r_{\text{f}}\) is multiplied with the weighted sum of all other reward terms listed in Table 3. Herein, the force jump reward term is defined as  
\[r_{\text{f}}(v_{\text{d}},\Delta u)= \begin{cases}&1.0,\quad(v_{\text{d}}<| \theta_{v_{\text{d}}}|)\text{ or }(\Delta u<\mid\theta_{\Delta u}\mid)\\ clip\Big{(}1.0-k_{\text{f}}(|v_{\text{d}}|-\theta_{v_{\text{d}}})\cdot(| \Delta u|-\theta_{\Delta u}),0.0,\quad 1.0\Big{)},\quad\text{else}\end{cases}  \tag{9}\]