# Vehicle Lateral Control Based on Augmented Lagrangian DDPG Algorithm
## 4 Main Results
### Vehicle Tracking Control Algorithm Based on Safety Constraints
The reward Critic network is updated through iteration (9),  
\[L_{C}=\frac{1}{N}\sum_{i}{(z_{i}-Q_{C}(s_{i},a_{i}|\theta_{C}^{Q}))^{2}} \tag{10}\]  
while the loss Critic network is updated through iteration (10). \[\Delta_{\theta^{\mu}}=\frac{1}{N}\sum_{i}{\Delta_{\theta^{\mu}}(Q_{R}(s,\mu(s| \theta^{\mu})|\theta_{R}^{Q})-\lambda Q_{C}(s,\mu(s|\theta^{\mu})|\theta_{C}^{ Q}))|_{s=s_{i}}} \tag{11}\]  
The sampled policy gradient (11) is used to update the action network, and the Lagrange multiplier is updated using the loss evaluation network. The pseudocode for the safe reinforcement learning algorithm based on the augmented Lagrangian function method is Algorithm 1. ```
1:Randomly initialize the network Reward Critic Q-network \(Q_{R}(s,a|\theta_{R}^{Q})\), Cost Critic Q-network \(Q_{C}(s,a|\theta_{C}^{Q})\)and Actor network \(\mu(s|\theta^{\mu})\)
2:Initialize the target network \(\theta_{R}^{Q^{\prime}}\leftarrow\theta_{R}^{Q}\theta_{C}^{Q^{\prime}} \leftarrow\theta_{C}^{Q}\theta^{\mu^{\prime}}\leftarrow\theta^{\mu}\)
3:Initialize parameters Reply buffer \(\mathcal{R}\) and Augmented Lagrangian Method \(\lambda\)
4:for episode \(k=0,1,\ldots,\)do
5: Initialize a random process \(\mathcal{N}\)for action exploration
6: Get the initial state \(s_{0}\sim p_{0}\)
7:for t = 1,...,T do
8: Select an action \(a_{t}=\mu(s_{t}|\theta^{\mu})\)
9: Execute the action \(a_{t}\)Observe \(r_{t},c_{t},s_{t+1}\)
10: Store the tuple \((s_{t},a_{t},r_{t},c_{t},s_{t+1})\) in the replay buffer \(\mathcal{R}\)
11: Randomly sample \(N\) tuples \(\{(s_{t},a_{t},r_{t},c_{t},s_{t+1})\}_{i=1}^{N}\) from the replay buffer \(\mathcal{R}\)
12: Define \(y_{i}=r_{i}+\gamma Q_{R}^{\prime}(s_{i+1},\mu^{\prime}(s_{i+1}|\theta^{\mu^{ \prime}})|\theta^{Q^{\prime}},z_{i}=c_{i}+\gamma Q_{C}^{\prime}(s_{i+1},\mu^{ \prime}(s_{i+1}|\theta^{\mu^{\prime}})|\theta_{C}^{Q^{\prime}})\)
13: Update the reward evaluation network by minimizing \(L_{R}=\frac{1}{N}\sum_{i}(y_{i}-Q_{R}(s_{i},a_{i}|\theta_{R}^{Q}))^{2}\), and update the loss evaluation network by minimizing \(L_{C}=\frac{1}{N}\sum_{i}(z_{i}-Q_{C}(s_{i},a_{i}|\theta_{C}^{Q}))^{2}\). 14: Update the action network using the sampled policy gradient: \(\Delta_{\theta^{\mu}}=\frac{1}{N}\sum\limits_{i}{\Delta_{\theta^{\mu}}(Q_{R}(s,\mu(s|\theta^{\mu})|\theta_{R}^{Q})-\lambda Q_{C}(s,\mu(s|\theta^{\mu})|\theta _{C}^{Q}))|_{s=s_{i}}}\)
15: Update the Augmented Lagrangian multiplier: \(\lambda\): \(\lambda_{\lambda}=\frac{1}{N}\sum\limits_{i}[Q_{C}(s_{i},\mu(s_{i}|\theta^{ \mu})-d]\)
16: Update the target network: \(\theta_{K}^{Q^{\prime}}\leftarrow\tau\theta_{K}^{Q}+(1-\tau)\theta_{K}^{Q^{ \prime}}\) \(\theta_{C}^{Q^{\prime}}\leftarrow\tau\theta_{C}^{Q}+(1-\tau)\theta_{C}^{Q^{ \prime}}\)
17:\(\theta^{\mu^{\prime}}\leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}}\)
18:endfor
```  
**Algorithm 1** Based on the Augmented Lagrangian Method for Safe DDPG Algorithms (DDPGALM). The algorithm based on the Augmented Lagrangian Method for Safe DDPG Algorithms addresses the safety constraints in trajectory tracking for four-wheel steering vehicles. Specifically, it aims to mitigate the issue of premature termination during model training caused by collisions with lane edges. By incorporating the Augmented Lagrangian Method, the algorithm ensures that safety constraints are rigorously enforced, thereby enhancing the robustness and reliability of the training process. This approach not only improves the vehicle's ability to track trajectories accurately but also significantly reduces the risk of unsafe behaviors, such as lane boundary violations, during both training and deployment.