## 3 Path-Following Control Strategy with Deep Deterministic Policy Gradient
### Deep Deterministic Policy Gradient for Path-Following
#### 3.2.1 Observation Space and Action Space  
As the MDP under consideration is partially observable, we use the term \"observation\" instead of \"state\" to denote the information the agent relies on when taking actions. The agent performs some actions in the environment and observes the resulting changes in the environment's state. This interaction between action and observation is commonly known as a time step. The observation \(s\), which serves as the input to the path-following controller, is selected in an intuitive and low-dimensional manner:  
\[s=\{e_{p},\psi,\delta\} \tag{20}\]  
where \(e_{p}\) is the cross-track error in Equation (4), \(\psi\in[-\pi,\pi]\) is the orientation error between the path and the vehicle, and \(\delta\) is the steering angle of the front tire, as graphically presented in Figure 2. We calculate the cross-track error and heading error based on the C.G. of the vehicle, as described in Section 2.1. The steering angle \(\delta\) reflects the level of effort exerted by the vehicle. More details are presented in Table 3. It is worth noting that, for the purpose of constraining the sparsity of the reward function, the absolute value of the cross-track error is limited to below 2.0 m. This constraint helps reduce the size of the observation space. Additionally, any deviation exceeding 2.0 m is considered an unacceptable error and the vehicle should be forcefully brought to a stop.  
We choose to use the rate of steering angle \(\dot{\delta}\) over the steering angle \(\delta\) itself as the action, \(a_{t}=\mu(s_{t}|\theta^{\mu})\), to avoid undesired fast angle changes. Adopting an incremental control input to the vehicle makes it easier to achieve smooth vehicle motions. Furthermore, constraining the steering angle rate within a specified range, i.e., \(\dot{\delta}\in[\dot{\delta}_{min},\dot{\delta}_{max}]\), provides a better stability. The steering angle command \(\delta_{t}^{*}\) at each time step during training is computed using Equation (21), which incorporates the sampled noise signal \(n_{t}\) from an OU process \(\mathcal{N}\). However, the noise is excluded when it is not the training phase. The clipping operation is used to bound values within their ranges, typically to prevent saturation. The constraints on the action and OU process parameters are illustrated in Tables 4 and 5.  
\[\begin{split} n_{t}&=n_{t-1}+v_{n}(\mu_{n}-n_{t-1}) \Delta t+\zeta_{n}dW_{t}\\ \delta_{t}^{*}&=\delta_{t}+\text{clip}(\mu(s_{t}| \theta^{\mu})+n_{t},\dot{\delta}_{min},\dot{\delta}_{max})\Delta t\\ \delta_{t}^{*}&=\text{clip}(\delta_{t}^{*},\dot{ \delta}_{min},\dot{\delta}_{max})\end{split} \tag{21}\]  
where \(\
u_{n}\) determines the speed of mean reversion, the drift term \(\mu_{n}\) affects the asymptotic mean and the standard Wiener process scaled by the volatility \(\zeta_{n}\) is denoted by \(dW_{t}\).  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Symbol** & **Description** & **Min** & **Max** \\ \hline \(\dot{\delta}\) & Front tire angle rate & \(-1.5708\) rad/s & \(1.5708\) rad/s \\ \hline \hline \end{tabular}
\end{table}
Table 4: Action space.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Symbol** & **Description** & **Min** & **Max** \\ \hline \(e_{p}\) & Cross-track error & \(-2.0\) m & \(2.0\) m \\ \(\psi\) & Orientation error & \(-\pi\) rad & \(\pi\) rad \\ \(\delta\) & Front tire angle & \(-0.5236\) rad & \(0.5236\) rad \\ \hline \hline \end{tabular}
\end{table}
Table 3: Observation space.