## Appendix B Vehicle Dynamics of the ROboMObil  
Ideally, reinforcement learning is conducted on the real-world system to avoid the reality gap between the simulation setup and the real world. Often, however, training on the real-world system might raise major safety concerns since in safety-critical applications, such as autonomous driving, the system or surrounding humans can be endangered. Therefore, simulation-based reinforcement learning is preferred. For this, a training model needs to be provided that represents the behavior of the system. In this work, agents are trained to control the ROboMObil [12], which is a robotic research vehicle at the German Aerospace Center (DLR). Since certain dynamics parameters of the learning model are actively changed during the training processes, the vehicle model is introduced in detail. This section closely follows the work in [12]. The interested reader is pointed to the aforementioned publication for more detail. The vehicle configuration of the extended nonlinear single-track model of the ROboMObil is shown in Figure 19. The state vector of the model \(\mathbf{x}\) is given by  
\[\mathbf{x}=\left[\beta_{\rm C}^{\rm C},v_{\rm C}^{\rm C},\dot{\psi}_{\rm C}^{\rm C },\psi_{\rm C}^{\rm L},x_{\rm C}^{\rm L},y_{\rm C}^{\rm L}\right]^{\rm T} \tag{13}\]  
with \(\beta_{\rm C}^{\rm C}\) being the vehicle side slip angle, \(v_{\rm C}^{\rm C}\) the absolute value of the velocity vector and \(\dot{\psi}_{\rm C}^{\rm C}\) the yaw rate of the vehicle in the car coordinate system. Furthermore, \(\psi_{\rm C}^{\rm L}\) represents the yaw angle and \(x_{\rm C}^{\rm L}\) and \(y_{\rm C}^{\rm L}\) denote the position of the ROboMObil in a fixed inertial coordinate system. The control input vector of the model is set to  
\[\mathbf{u}=\left[\tau_{\rm f},\tau_{\rm r},\eta_{\rm f},\eta_{\rm r}\right]^{\rm T} \tag{14}\]  
where \(\tau_{\rm f}\) and \(\tau_{\rm r}\) denote the torque set-points to the front and rear in-wheel motors. Furthermore, \(\eta_{\rm f}\) and \(\eta_{\rm r}\) denote the steering rates for the front and rear vehicles axles. The differential equations of the vehicle states and the steering angles \(\delta_{\rm f}\) and \(\delta_{\rm r}\) are provided by \[ \begin{split}\frac{\mathrm{d}\theta_{\mathrm{C}}^{\mathrm{C}}}{ \mathrm{d}t}&=\frac{-\sin\left(\beta_{\mathrm{C}}^{\mathrm{C}}\right)F _{\mathrm{s}}^{\mathrm{C}}+\cos\left(\beta_{\mathrm{C}}^{\mathrm{C}}\right)F _{\mathrm{y}}^{\mathrm{C}}}{m\upsilon_{\mathrm{mod}}^{\mathrm{C}}}-\dot{ \psi}_{\mathrm{C}}^{\mathrm{C}}\\ \frac{\mathrm{d}\upsilon_{\mathrm{C}}^{\mathrm{C}}}{\mathrm{d}t}& =\frac{\cos\left(\beta_{\mathrm{C}}^{\mathrm{C}}\right)F_{\mathrm{s}}^{ \mathrm{C}}+\sin\left(\beta_{\mathrm{C}}^{\mathrm{C}}\right)F_{\mathrm{y}}^{ \mathrm{C}}}{m}\\ \frac{\mathrm{d}\vartheta_{\mathrm{C}}^{\mathrm{C}}}{\mathrm{d}t}& =\frac{M_{\mathrm{s}}^{\mathrm{C}}}{f_{\mathrm{s}}^{\mathrm{C}}}\\ \frac{\mathrm{d}\vartheta_{\mathrm{C}}^{\mathrm{C}}}{\mathrm{d}t}& =\dot{\psi}_{\mathrm{C}}^{\mathrm{C}}\\ \frac{\mathrm{d}\lambda_{\mathrm{C}}^{\mathrm{C}}}{\mathrm{d}t}& =\
u_{\mathrm{C}}^{\mathrm{C}}\cos\left(\psi_{\mathrm{C}}^{\mathrm{L}}+\beta_{ \mathrm{C}}^{\mathrm{C}}\right)\\ \frac{\mathrm{d}\vartheta_{\mathrm{C}}^{\mathrm{L}}}{\mathrm{d}t}& =\
u_{\mathrm{C}}^{\mathrm{C}}\sin\left(\psi_{\mathrm{C}}^{\mathrm{L}}+\beta_{ \mathrm{C}}^{\mathrm{C}}\right)\\ \frac{\mathrm{d}\lambda_{\mathrm{C}}^{\mathrm{L}}}{\mathrm{d}t}& =\
u_{\mathrm{f}}\\ \frac{\mathrm{d}\lambda_{\mathrm{C}}^{\mathrm{L}}}{\mathrm{d}t}& =\
u_{\mathrm{c}}\end{split}  \tag{12}\]  
where \(m\) denotes the vehicle mass and \(J_{z}^{\mathrm{C}}\) the yaw inertia. Here, it should be noted that both parameters, namely the mass \(m\) and the yaw inertia \(J_{z}^{\mathrm{C}}\), are being randomized during the reinforcement learning training process.