## 3 Path-Following Control Strategy with Deep Deterministic Policy Gradient
### Preliminaries of Reinforcement Learning  
RL involves an agent interacting with the environment to learn the optimal actions that maximize cumulative reward. Here, we consider a standard RL architecture using a Markov Decision Process (MDP), where based on a given state \(s_{t}\in\mathcal{S}\) at time \(t\) the agent takes an action \(a_{t}\in\mathcal{A}\), receives a corresponding reward \(r_{t}\) and transitions to the next state \(s_{t+1}\) with a probability \(p(s_{t+1}|s_{t},a_{t})\)[27; 28; 29]. The MDP models the decision-making process of the agent as it interacts with the environment. Actions taken by the agent are specified by a policy, in general, which is stochastic and denoted by \(\pi(a|s)\). The policy \(\pi:\mathcal{S}\rightarrow\mathcal{P}(\mathcal{A})\) maps the states to a probability distribution over actions, specifying the likelihood of taking each action given a state. The return, which it is the objective of the agent to maximize, is the total discounted reward from time-step \(t\) onwards, as defined in Equation (13). \[R_{t}=r_{t}+\gamma r_{t+1}+\gamma^{2}r_{t+2}+...=\sum_{i=t}^{\infty}\gamma^{i- t}r(s_{i},a_{i}) \tag{13}\]  
where \(\gamma\in(0,1)\) is a discount rate that is used to prevent divergence of the return to infinity. Let \(\rho^{\pi}\) denote the discounted state distribution for a policy \(\pi\); the action-value function, also commonly referred to as Q-function is defined to be the expected total discounted reward, \(Q^{\pi}(s_{t},a_{t})=\mathbb{E}_{s_{t}\sim\rho^{\pi},\ a_{t}\sim\pi}[R_{t}|s_{ t},a_{t}]\). The aim of the agent is to acquire a policy that maximizes the cumulative discounted reward from the initial state, which is expressed as an expectation in Equation (14). \[ \begin{split} J(\pi)&=\sum_{\mathcal{S}}\rho^{\pi} (s)\sum_{\mathcal{A}}\pi(s,a)r(s,a)\\ &=\mathbb{E}_{s_{t}\sim\rho^{\pi},\ a_{t}\sim\pi}\left[r(s,a) \right]\end{split}  \tag{14}\]  
The Bellman equation is a fundamental recursive relationship that is widely employed in the field of RL [14; 30]. Specifically, the Q-function under a stochastic policy is expressed as Equation (15), whereas the same function under a deterministic policy \(\mu:\mathcal{S}\leftarrow\mathcal{P}(\mathcal{A})\) can be rewritten as Equation (16) [14; 15]. \[Q^{\pi}(s_{t},a_{t})=\mathbb{E}_{s_{t}\sim\rho^{\pi},\ a_{t}\sim\pi}[r_{t}+ \gamma\mathbb{E}_{a\sim\pi}[Q^{\pi}(s_{t+1},a_{t+1})] \tag{15}\]  
Figure 3: Structure of separated longitudinal and latitudinal controls. We focus on the path-following algorithm, which is solely used to determine the steering angle command \(\delta^{*}\). \[Q^{\mu}(s_{t},a_{t})=\mathbb{E}_{s_{t+1}\sim\rho^{\pi}}[r_{t}+\gamma Q^{\mu}(s_{t+1 },\mu(s_{t+1}))] \tag{16}\]  
With a deterministic policy, the expectation is solely dependent on the environment, allowing for the learning of \(Q^{\mu}\) off-policy by utilizing transitions generated from a distinct stochastic policy \(\alpha\). The actor-critic architecture is a widely utilized off-policy framework [31; 32], consisting of two eponymous components. Considering function approximators parameterized by \(\theta\in\mathbb{R}^{n}\), which refers to a vector of \(n\) parameters, an actor updates the parameters \(\theta^{\mu}\) of the actor function \(\mu\) using policy gradients, while a critic updates the parameters \(\theta^{Q}\) and estimates the unknown true action-value function \(Q^{\mu}\) by using a policy evaluation algorithm such as temporal-difference (TD) learning. By minimizing the mean squared loss given by Equation (17), we can optimize the parameters of the critic. \[L(\theta^{Q})=\mathbb{E}_{s_{t}\sim\rho^{\mu},\ a_{t}\sim\alpha}[(y_{t}-Q(s_{t },a_{t}|\theta^{Q}))^{2}] \tag{17}\]  
where \(y_{t}\) is the TD target that the Q-function is updating towards. \[y_{t}=r(s_{t},a_{t})+\gamma Q(s_{t+1},\mu(s_{t+1})|\theta^{Q}) \tag{18}\]  
By applying the chain rule to the actor parameters \(\theta^{\mu}\), the way to update the actor using the expected return from the starting \(J\) is shown in Equation (19). \[