## 3 Learning-Based Path following Control with Parametric Uncertainties
### Design of the Reward Function  
The design of the reward function provides a crucial degree of freedom in reinforcement learning. As mentioned above, the agent should be rewarded positively when it approaches the control goal, i.e., when it has small or no errors to the path. For the path-following control task, the agent should learn to control the vehicle such that the lateral offset \(e_{y}^{\mathrm{P}}\), the orientation error \(e_{y}\) and the velocity errors \(e_{v_{x}}^{\mathrm{P}}\) are minimized. However, as mentioned in [14], the agent's primary control goal should be to minimize the lateral position error \(e_{y}^{\mathrm{P}}\), since a large lateral offset could negatively influence safety and possibly cause collisions. After minimizing the lateral position error, the agent should learn to control the vehicle such that it achieves the commanded orientations and velocities and both \(e_{\psi}^{\mathrm{P}}\) and \(e_{v_{x}}^{\mathrm{P}}\) approach zero. Furthermore, smooth steering behavior should be favored. Therefore, a hierarchical structure for the reward function is chosen, as in [14]. More specifically, the reward function is set to  
\[r_{\mathrm{PFC}}\left(e_{y}^{\mathrm{P}},e_{\psi}^{\mathrm{P}},e_{v_{x}}^{ \mathrm{P}},\Delta\delta_{\mathrm{f}},\Delta\delta_{\mathrm{f}}\right)=g_{ \Theta_{y}}\left(e_{y}^{\mathrm{P}}\right)\left(1+r_{\varepsilon}\left(e_{ \psi}^{\mathrm{P}},e_{v_{x}}^{\mathrm{P}}\right)\left(1+r_{\Delta S}\left( \Delta\delta_{\mathrm{f}},\Delta\delta_{\mathrm{f}}\right)\right)\right) \tag{5}\]  
with \(r_{\varepsilon}\left(e_{\psi}^{\mathrm{P}},e_{v_{x}}^{\mathrm{P}}\right)\) being  
\[r_{\varepsilon}\left(e_{\psi}^{\mathrm{P}},e_{v_{x}}^{\mathrm{P}}\right)=g_{ \Theta_{y}}\left(e_{\psi}^{\mathrm{P}}\right)+g_{\Theta_{x_{x}}}\left(e_{v_{ x}}^{\mathrm{P}}\right) \tag{6}\]  
and \(r_{\Delta S}\left(\Delta\delta_{\mathrm{f}},\Delta\delta_{\mathrm{f}}\right)\) being  
\[r_{\Delta S}\left(\Delta\delta_{\mathrm{f}},\Delta\delta_{\mathrm{f}}\right)= \frac{1}{1+c_{\mathrm{f}}|\Delta\delta_{\mathrm{f}}|+c_{\mathrm{f}}|\Delta \delta_{\mathrm{r}}|}. \tag{7}\]  
Here, the expressions \(\Delta\delta_{\mathrm{f}}\) and \(\Delta\delta_{\mathrm{f}}\) denote the changes of the front and rear steering angles between the two subsequent time steps \(k\) and \(k-1\) which are given by  
\[\Delta\delta_{\mathrm{f}}=\delta_{\mathrm{f}k}-\delta_{\mathrm{f}k-1} \tag{8}\]  
Furthermore, \(c_{\mathrm{f}}\) and \(c_{\mathrm{r}}\) represent their weighting parameters in (7) and are set manually. In Equations (5) and (6), the functions \(g_{\Theta}(x)\) are Gaussian-like functions  
\[g_{\Theta}(x)=\theta_{1}e^{\frac{\pi x^{2}}{22}} \tag{9}\]  
with the properties  
\[0<g_{\Theta}(x)\leq\theta_{1},\forall x\in\mathbb{R},\forall\theta=[\theta_{1 },\theta_{2}]\in\mathbb{R}_{+}^{2}. \tag{10}\]  
For the reward \(r_{\mathrm{PFC}}\left(e_{y}^{\mathrm{P}},e_{\psi}^{\mathrm{P}},e_{v_{x}}^{ \mathrm{P}},\Delta\delta_{\mathrm{f}},\Delta\delta_{\mathrm{f}}\right)\) in Equation (5), the function \(g_{\Theta_{y}}\left(e_{y}^{\mathrm{P}}\right)\) approaches zero for large lateral position errors \(e_{y}^{\mathrm{P}}\) and approaches \(\theta_{y,1}>0\) for small \(e_{y}^{\mathrm{P}}\). Hence, the agent is rewarded for small lateral position errors. This term dominates the overall reward, since it is the only term multiplied by one (cf. Equation (5)), which is in-line with the hierarchical reward structure of prioritizing the minimization of the lateral position error first [14]. Furthermore, the value of \(g_{\varphi}\left(e_{y}^{\mathrm{P}}\right)\) is multiplied by the reward term \(r_{e}\left(e_{\varphi}^{\mathrm{P}},e_{v_{x}}^{\mathrm{P}}\right)\) consisting of the Gaussian-like functions \(g_{\varphi_{\varphi}}\left(e_{\varphi}^{\mathrm{P}}\right)\) and \(g_{\varphi_{x}}\left(e_{v_{x}}^{\mathrm{P}}\right)\) (cf.