## 4 Training Setup
### Training Procedure  
The agents are trained on the path depicted in Figure 6, which represents a federal highway called \"Kesselberg\", located in the German Alps, and is parameterized by the arc length \(s\). The corresponding desired velocity profile \(v_{\mathrm{P}}^{\mathrm{I}}(s)\) is shown in Figure 7, calculated by taking the path curvature and the vehicle's acceleration limits into account [26]. This particular path is chosen for training since it consists of road sections with different characteristics, which are beneficial for learning the RL-based PFC. In Figures 6 and 7, for example, it can be seen that the path demands tight turns between the arc lengths \(s=300\) m and \(s=600\) m (cf. Figure 6) with a rather slow velocity around \(v_{\mathrm{P}}^{\mathrm{I}}(s)=10\) m/s (cf. Figure 7). On the other hand, the path section between \(s=900\) m and \(s=1200\) m represents an almost straight road, where the vehicle needs to accelerate quickly to successfully track the velocity demanded. All agents are trained for a total of 300.000 time steps with a step size of \(\Delta t=0.05\) s, and the reward function introduced in Equation (5). In this work, an episode consists of 300 time steps. During training, we apply the state-of-the-art Soft Actor-Critic (SAC) [27] learning algorithm from the Stable-Baselines 2 library [21]. The SAC algorithm is briefly discussed in Appendix C. The training with the SAC method is conducted with the hyperparameters given in Appendix D. The parameters of the reward function introduced in Equation (5) are set to:  
\[\theta_{y}=[1,\ 0.05],\theta_{\psi}=\Big{[}1,\ \sqrt{0.005}\Big{]},\ \theta_{\psi}=\Big{[}1,\ \sqrt{0.1}\Big{]}. \tag{15}\]  
Furthermore, the weighting parameters \(c_{\mathrm{f}}\) and \(c_{\mathrm{r}}\) in Equation (7) are both set to 1 to equally penalize large changes in both the front and rear steering angles.  
As introduced in [14], it is beneficial to randomly initialize the system with an offset to the path at the beginning of each training episode. This supports the exploration of the observation space since the agents must repeatedly try to successfully follow the path starting from different initial configurations. More specifically, the offset is applied to the initial position error \(e_{y,\mathrm{start}}^{\mathrm{p}}\), the initial orientation error \(e_{\psi,\mathrm{start}}\), and the initial longitudinal velocity error \(e_{v_{\mathrm{z},\mathrm{start}}}^{\mathrm{p}}\). At the beginning of each episode, these initial errors to the path are randomly sampled from three different uniform distributions within the following bounds:  
\[-0.8\ \mathrm{m} \leq e_{y,\mathrm{start}}^{\mathrm{p}}\leq 0.8\ \mathrm{m} \tag{16}\] \[-8.6^{\circ} \leq e_{\psi,\mathrm{start}}\leq 8.6^{\circ}\] \[-1.0\ \tfrac{\mathrm{m}}{\mathrm{s}} \leq e_{v,\mathrm{start}}^{\mathrm{p}}\leq 1.0\ \tfrac{\mathrm{m}}{\mathrm{s}}.\]  
To further encourage the agents to only explore important parts of the observation space, several training abortion criteria are introduced, as described in [14]. If one or more of these errors do not remain within their respective pre-defined thresholds, then the training episode is terminated early. In this work, a termination is triggered in the following cases:\[\left|e_{y}^{\mathrm{P}}\right|>2\;\mathrm{m},\;\left|e_{y}\right|>70^{\circ},\; \left|e_{v_{x}}^{\mathrm{P}}\right|>5\;\frac{\mathrm{m}}{\mathrm{s}},\;\left|e_ {v_{y}}^{\mathrm{P}}\right|>5\;\frac{\mathrm{m}}{\mathrm{s}}. \tag{17}\]  
Every time an episode is terminated, the negative terminal reward \(r_{\mathrm{T}}=-10\) is provoked. Therefore, the agents need to learn to stay within these error thresholds, since a negative reward contradicts the primary reinforcement learning goal of maximizing the expected sum of rewards, also called the return \(G\) (cf. Equation (A17) in Appendix C). When a new episode starts, the vehicle is reinitialized where the previous episode ended, with an initial offset according to Equation (16).