# Vehicle Lateral Control Based on Augmented Lagrangian DDPG Algorithm
## 4 Main Results  
### Augmented Lagrangian Method for Safe RL  
The instantaneous constraints in safe reinforcement learning are typically solved using the Lagrange multiplier method, which converts the constrained optimization problem into an unconstrained optimization problem using the Lagrange function, adding the weighted constraints to the objective function to obtain the Lagrange function: \(L(\pi,\lambda)=J_{r}(\pi)-\lambda(J_{c}(\pi)-b)\). The Lagrange function \(L(\pi,\lambda)\) is used to convert the constrained optimization problem into an unconstrained optimization problem. \[\min_{\lambda\geq 0}\ \max_{\pi}\ L(\pi,\lambda)=\min_{\lambda\geq 0}\max_{ \pi}\left[J_{r}(\pi)-\lambda(J_{c}(\pi)-\alpha)\right].\]  
The existing methods exhibit dual theoretical limitations in alternately optimizing policy parameters \(\pi\) and Lagrange multipliers \(\lambda\). First, the multiplier update mechanism employs a fixed learning rate for gradient ascent, lacking adaptive capability to optimize surface curvature, which leads to unstable convergence and oscillatory divergence with improper learning rates. Second, zero initialization renders multipliers ineffective during early optimization, weakening the constraint violation penalty in the initial training phase, thereby reducing penalty strength during early exploration of safety boundaries. In traditional constrained optimization methods, there is another approach known as the Augmented Lagrangian Method (ALM). This method combines the concepts of the penalty function method and the Lagrange multiplier method to construct an augmented Lagrangian function that includes a quadratic penalty term. For the aforementioned constrained optimization problem, the augmented Lagrangian function can be defined as follows:  
\[L(\pi,\lambda,\rho)=J_{r}(\pi)-\lambda(J_{c}(\pi)-\alpha)-\frac{\rho}{2}(J_{c} (\pi)-\alpha)^{2}.\]  
where \(\rho>0\) is the penalty factor for the quadratic penalty term \((J_{c}(\pi)-\alpha)^{2}\). **Definition 2**. : _(Augmented Lagrangian MDP Optimization Problem) Using the framework defined in Definition 1, combined with the augmented Lagrangian multiplier method, formalize the constrained optimization problem of the vehicle trajectory tracking problem as the following unconstrained optimization problem:_  
\[ \begin{split}\max_{\pi}&\min_{\lambda\geq 0}\ \mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}\Big{(}r(s_{t},a_{t})-\lambda_{t}g (s_{t})+\frac{\rho}{2}\max{(0,g(s_{t},a_{t}))^{2}}\Big{)}\bigg{|}\pi\right]\\ s.t.s_{t+1}&\sim p_{s}(\cdot|s_{t},a_{t}),\forall t \in\{0,1,\ldots\},\\ a_{t}&\sim\pi(s_{t}),\forall t\in\{0,1,\ldots\},\\ s_{0}&\sim p_{0}\end{split}  \tag{7}\]  
The constraint function requires that the vehicle's desired position at each moment does not exceed the lane boundary. Compared with the traditional Lagrange multiplier method, the augmented Lagrangian multiplier method has two main advantages. First, during the iterative optimization process using the augmented Lagrangian function \(L(\pi,\lambda,\rho)\), the step size for updating \(\lambda\) is no longer fixed but is adaptively adjusted by the variable penalty factor \(\rho\), which increases progressively during optimization. This mechanism effectively adjusts the learning rate dynamically, where the synergistic effect of the multiplier term and quadratic penalty term enhances algorithmic convergence rate and stability. Second, initializing the penalty factor \(\rho\) to a positive value enforces strict \"warm-start\" penalization of safety constraint violations from the outset of training. This design prioritizes safety considerations during initial learning phases, thereby mitigating the risk of converging to unsafe local optimal solutions. Transforming the constrained Markov decision process problem into an unconstrained optimization problem using the augmented Lagrangian function enables the solution of constrained optimization problems through the augmented Lagrangian multiplier method.