# Reinforcement Learning for Semi-Active Vertical Dynamics Control with Real-World Tests
## 4 Training the Controller
### Application to the Vertical Dynamics Problem
#### 4.2.4 Agent Performance Assessment  
Most of the time, training a performant RL agent is an iterative process. This process consists of training an agent, evaluating the trained agent, changing the training setup, and then starting a new training. Different metrics are of interest during the evaluation of the agents. Since the reward compresses multiple objectives into one scalar measure, this reward may not be sufficient to assess the multiple objectives.  
Therefore, we implemented a performant multi-objective agent assessment pipeline to evaluate the trained agents. This pipeline was composed out of different steps that were performed for each trained agent. In the first step, every trained agent was evaluated on predefined road excitations and all signals of interest were temporarily stored. Then, predefined metrics were calculated based on these signals and the resulting metrics themselves were also stored in a database. In addition to the metrics, all hyperparameters and other information about the environment and the training were stored. Keeping track of all the information allowed an in-depth analysis of the promising hyperparameters, reward functions, or environment implementations. Additionally, this pipeline enabled traceability and reproducible training results.  
To rate the performance of the learned controller, we needed to relate the calculated metrics to a state-of-the-art controller. To achieve a fair comparison, we optimized an SH and a GH controller based on the optimal QVMs, which were also used during training. After that, the metrics of interest were calculated for the optimized SH controller, the optimized GH controller, and all trained RL-based controllers. To ensure a balanced assessment of the controllers, we evaluated each controller version on a set of different excitations including sweeps, wave shapes, and bump-like excitations. Following the calculation, we compared each trained agent to the better one from SH and GH and took the mean over all excitations. This resulted in a set of metrics that allowed us to compare the trained controllers to state-of-the art controllers.  
In the controller assessment, we were especially interested in the comfort criterion, the road-holding criterion, as well as the smoothness of the actions. During the assessment, we applied a simplified version of the comfort criterion \(J_{\text{c}}\) from [43], i.e.,  
\[J_{\text{c}}=\sqrt{\frac{1}{N}\sum_{k=1}^{N}a_{\text{c,k}}^{2}}, \tag{14}\]  
with the vertical chassis acceleration \(a_{\text{c}}\). To assess the road-holding \(J_{\text{rh}}\), we calculated the root mean square (rms) of the dynamic wheel load \(F_{\text{z,dyn}}\)  
\[J_{\text{rh}}=\sqrt{\frac{1}{N}\sum_{k=1}^{N}F_{\text{z,dyn},k}^{2}}. \tag{15}\]  
Finally, we measured the smoothness of the action signal \(J_{\text{s}}\) via a mean of the absolute change from one timestep \((k-1)\) to the next timestep \(k\):  
\[J_{\text{s}}=\frac{1}{N}\sum_{k=1}^{N}|a_{k}-a_{k-1}|. \tag{16}\]  
In order to select agents for the evaluation on the real vehicle, we filtered all agents in which the comfort criterion was a maximum of 5% worse compared to the reference controller. The remaining agents were then sorted for smoothness in the action signal. Finally, we chose the smoothest four agents for evaluation on the real vehicle.