# Vehicle Lateral Control Based on Augmented Lagrangian DDPG Algorithm
## 3 Problem Formulation
### Constrained Reinforcement Learning-Based Tracking Control Framework  
To address the control problem in the vehicle trajectory tracking system, this paper employs deep reinforcement learning methods, modeling the vehicle trajectory tracking system as a constrained Markov decision process. The vehicle is treated as the environment in the RL problem, while the four-wheel steering controller serves as the agent. Leveraging the periodicity of steering control, each control cycle's steering actions and resulting vehicle state changes are considered as an agent-environment interaction.  
When approaching curved road sections, insufficient lane boundary information acquisition may cause lateral control response delays, leading to overshooting control targets and reduced stability. To ensure timely control updates before entering curvature-changing zones, we select lateral tracking errors \(y_{\mathrm{L}}\), heading deviation \(\varepsilon_{\mathrm{L}}\), and trajectory curvature \(\rho\) as state space components. The state space \(\mathcal{S}\) is defined as: \(\mathcal{S}=\{y_{\mathrm{L}},\varepsilon_{\mathrm{L}},\rho\}\).  
The study focuses on four-wheel independent steering vehicles, thus defining the action space as the steering angles of the four wheels: \(\phi_{lf}\), \(\phi_{lr}\), \(\phi_{rf}\), \(\phi_{rf}\). The action space  
Figure 2: Vehicle Tracking Model with Preview Distance.  
\(\mathcal{A}\) is defined as: \(\mathcal{A}=\{\phi_{I},\phi_{Ir},\phi_{rf},\phi_{rr}\}\). During deep reinforcement learning training, excessively large action ranges in the control policy output degrade training performance, necessitating action range constraints through linear mapping of policy network outputs.  
To reduce the occurrence of dangerous behaviors such as overstepping and collisions during training, preventing numerous ineffective training cycles, this paper introduces lane boundary constraints as state constraints. Ensuring that the vehicle trajectory tracking error strictly satisfies the safety boundary constraints, \(|y_{\mathrm{L}}|<3\quad\forall t\), essentially requires the control policy to avoid entering dangerous state regions at any time.  
First, define the constraint cost function \(c:\mathcal{S}\rightarrow\{0,1\}\) to characterize the safety attributes in the state space:  
\[c(s_{t})=\mathbb{I}(|y_{\mathrm{L}}|\geq 3)=\begin{cases}1&|y_{\mathrm{L}}|\geq 3 \\ 0&\text{others.}\end{cases}\]  
Based on this, the constraints of the CMDP can be expressed as an upper bound on the expected cumulative constraint cost:  
\[\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma_{c}^{t}c(s_{t})\right]\leq\epsilon\]  
where \(\gamma_{c}\in[0,1]\) is the constraint discount factor. For strict hard constraint problems, set \(\gamma_{c}=1\) and \(\epsilon=0\), where the constraint condition is equivalent to requiring \(c(s_{t})=0\) at all times. Based on Definition 1, the vehicle trajectory tracking problem under safety constraints can be modeled as a CMDP as follows:  
\[\begin{split}\max_{\pi}J_{r}(\pi)=&\mathbb{E}_{\pi} \left[\sum_{t=0}^{T}\gamma^{t}r(s_{t},a_{t})|\pi\right]\\ \text{s.t.}&\mathbb{E}_{\pi}\left[\sum_{t=0}^{T}c(s_{t}) \right]\leq 0\end{split} \tag{6}\]  
where \(J_{r}(\pi)\) is the expected reward, \(r(s_{t},a_{t})\) is the immediate reward, \(c(s_{t})\) is the constraint cost, \(\gamma\) is the discount factor, and \(\pi\) is the policy. The objective of this optimization problem is to find the optimal policy \(\pi^{*}\) that maximizes the expected cumulative reward while satisfying the constraints.