# Vehicle Lateral Control Based on Augmented Lagrangian DDPG Algorithm
## 4 Main Results
### Vehicle Tracking Control Algorithm Based on Safety Constraints  
This study addresses the vehicle trajectory tracking problem, where the vehicle controller regulates the steering angles of all four wheels based on available lane trajectory information (including lane boundaries and trajectory curvature) and vehicle state parameters (such as velocity and acceleration). The objective is to ensure precise and safe lane-following performance. The framework of the proposed trajectory tracking decision control algorithm is illustrated in Figure 3.  
The reinforcement learning problem can be solved using the Actor-Critic framework, which combines the advantages of policy gradient methods and temporal difference methods, including an Actor network and a Critic network. The structure of the Actor network consists of four fully connected layers: an input layer, two hidden layers, and an output layer. The input layer of the Critic network is divided into two parts: one part is the state information of the vehicle simulation environment, and the other part is the action values output by the Actor network. The simulation environment information is processed by one hidden layer, then combined with the action values and input into the second hidden layer. Subsequently, the state information is processed by the output neurons to obtain the state-action reward value. To enhance the learning efficiency of reinforcement learning, we introduce a prioritized experience replay mechanism and a dual experience pool mechanism within the framework of constrained reinforcement learning algorithms. The prioritized experience replay mechanism allows the agent to preferentially select experience samples with higher importance or value for training. By focusing on more representative situations or critical learning moments, the agent can learn and optimize its strategy more effectively. By prioritizing the replay of key experiences, the agent can learn faster from past experiences, thereby accelerating the learning process. This is particularly useful in situations where rapid adaptation to environmental changes is required. The actor network generates a set of action values, i.e., the steering angles of the four wheels, and adds random noise before inputting them into the MATLAB (R2024b) simulation software. MATLAB then inputs the next state \(s^{\prime}\) into the reward function module based on the actions, storing it in the buffer along with the current state \(s\), reward value \(r\), and action value \(a\). To utilize high-quality data for neural network training and improve training stability, the buffer is divided into a standard error buffer \(R_{stand-e}\) and a high-error buffer \(R_{high-e}\). The experience buffer is separated by calculating the TD error \(TDE_{t}=y_{t}-\bar{Q}\theta(S_{t},a_{t})\), with the sampling probability of \(R_{high-e}\) increasing as model training progresses. A certain number of samples are proportionally sampled from the experience replay buffer, with each sample containing \(s,a,r,s^{\prime}\). Then, \(s\) and \(s^{\prime}\) are passed to the Actor network, and \(s,a,r,s^{\prime}\) are passed to the Critic network for iterative updates. For the Actor network, it accepts \(s\) and \(s^{\prime}\), then outputs \(a\) along with random noise to MATLAB. Simultaneously, after outputting the next action \(a^{\prime}\) to the Critic network, it accepts the state-action value gradient \(Q(s,a)\) to \(a^{\prime}\) to update the network. For the Critic network, it accepts \(s^{\prime}\) and \(a^{\prime}\) to calculate \(Q_{R}(s^{\prime},a^{\prime})\). Next, it combines \(Q_{R}(s^{\prime},a^{\prime})\) with \(r\) to compute the labels used for network iterative updates. Additionally, \(s\) and \(a\) are input into the Critic network, and the mean squared error between the output and the labels is used as the loss function. \[L_{R}=\frac{1}{N}\sum_{i}{(y_{i}-Q_{R}(s_{i},a_{i}|\theta_{R}^{Q}))^{2}} \tag{9}\]  
Figure 3: Track tracking decision control algorithm framework.