# Reinforcement Learning for Semi-Active Vertical Dynamics Control with Real-World Tests
## 4 Training the Controller
### Application to the Vertical Dynamics Problem
#### 4.2.2 Environment Interface  
In RL, the agent adapts its policy based on observations and rewards returned from the environment. The theory behind RL often assumes that the whole state of the environment can be observed by the agent. However, in real-world application, it is not always feasible to measure or estimate the whole state of the system and, therefore, it is not always possibleto provide the full state measurement to the agent, resulting in a Partially Observable Markov Decision Process (POMDP). Although many of the RL algorithms are developed based on the assumption that the whole state is available, in many applications RL delivers satisfying results even though not the whole state is available to the agent. In addition to the availability of measurement signals, the assembly of the observation vector can guide the training. Since it is not always obvious which signals provide valuable information to the agent, we performed various trainings with different observation sets. The model structure \"Engine QVM\" described in Section 3.3 and utilized to model the front left and front right sides of the vehicle was essentially a three-mass oscillator which yielded six states, two for each mass. An additional state resulted from the approximation of the input-to-force dynamics by a first-order system, as described in Section 3.2. In contrast, the model structure \"Topmount QVM\" used to approximate the rear left side of the vehicle did not include the engine mass and, in consequence, also the two states used to describe the engine motion. However, introducing the spring/damper element to model the topmount again resulted in additional states. These two states, together with the additional state for the first-order input-to-force approximation, added up to seven states for all the model variants. Nevertheless, in real-world applications, it is infeasible to measure all seven states on each corner of the vehicle. Even though the AFM test vehicle was equipped with a multitude of sensors [7], this is usually not the case for production vehicles. Therefore, we only utilized the following sensor signals for this work:  
* acceleration sensors at all four-wheel carriers;
* acceleration sensors at the front left, front right, and rear left chassis;
* displacement sensors between the chassis and wheel carrier at all four wheels;
* current sensor for each damper. These sensor signals are used to compute the vertical chassis velocity \(v_{\mathrm{c},j}\), the vertical wheel velocity \(v_{\mathrm{w},j}\), and the damper velocity \(v_{\mathrm{d},j}\) for each side of the vehicle, i.e., \(j\in\{\mathrm{fl},\,\mathrm{fr},\,\mathrm{rl},\,\mathrm{rr}\}\). In the end, the agent that performed best in the real-world tests included the following quantities in the observation vector  
\[o_{j}=\left[v_{\mathrm{c},j},v_{\mathrm{w},j},v_{\mathrm{d},j},i_{j}\right], \tag{6}\]  
with \(i_{j}\) representing the measured actual damper current and \(j\in\{\mathrm{fl},\mathrm{fr},\mathrm{rl},\mathrm{rr}\}\). This selection seems reasonable, given that the ride comfort is calculated based on the vertical chassis acceleration \(a_{\mathrm{c},j}\) and the road-holding is closely linked to the vertical wheel acceleration \(a_{\mathrm{w},j}\). The damper velocity \(v_{\mathrm{d},j}\) is an important value because, in combination with \(i_{j}\), it determines the damper force, as depicted in Figure 3. It should be noted that the quantities \(v_{\mathrm{c},j},v_{\mathrm{w},j},\) and \(v_{\mathrm{d},j}\) were used as inputs for an SH/GH controller implementation. In contrast to choosing the quantities for the observation vector, the assembly of the action vector was less complex. The force induced by the damper was controlled by the position of an electromagnetic valve, which itself was controlled by the current through its coil. On the damper control unit (DCU), the current flow could be controlled by adjusting the duty cycle of a pulse-width modulation (PWM).