# Vehicle Lateral Control Based on Augmented Lagrangian DDPG Algorithm
## 4 Main Results
### Reward Function Design  
In reinforcement learning algorithms, network initialization is random. Consequently, the agent can only interact with the environment based on guidance from the reward function. The reward function is typically a scalar value, where positive values represent rewards and negative values represent penalties. Reinforcement learning algorithms optimize towards maximizing the reward value. The scenario focused on in this paper is lane-keeping for vehicles. This requires that, during driving, the vehicle should stay as close as possible to the center of the lane, and the vehicle's forward direction should be aligned with the road axis. Additionally, on this basis, the vehicle should aim to increase its speed as much as possible.  
Aggressive steering during turns may cause stability loss and uncomfortable lateral acceleration fluctuations. To prevent local optimal solutions, we design an exponential reward function that accelerates reward growth near optimal solutions. The proposed reward function integrates tracking accuracy and comfort. This reward structure systematically integrates trajectory tracking accuracy with steering stability constraints through the following formulation for each timestep t:  
\[r_{t}(s_{t},a_{t})=\begin{cases}r_{t}(s_{t},a_{t})=\underbrace{\delta_{1}\cdot e ^{-|a_{\text{lat}}|}}_{\text{lateral stability}}+\underbrace{\delta_{2}\cdot e^{- \varphi_{\text{steering efficiency}}}}_{\text{S steering efficiency}}+\underbrace{\delta_{3}\cdot\mathcal{P}_{t}}_{ \text{Oscillation penalty}},&\text{normal driving;}\\ -C,&\text{hit boundary}\end{cases} \tag{8}\]  
where:  
\[\text{Lateral Stability Term }(\delta_{1}\cdot e^{-|a_{\text{lat}}|})\text{: Penalizes excessive lateral acceleration }a_{\text{lat}}\text{ through exponential decay, where }\delta_{1}\text{ determines the penalty intensity. This term ensures vehicle stability during aggressive maneuvers.  
Steering Efficiency Term \((\delta_{2}\cdot e^{-\varphi_{\text{steer}}})\): Encourages minimal steering effort by penalizing the cumulative steering angle \(\varphi_{\text{steer}}=\sum_{i=1}^{4}|\phi_{i}|\), where \(\phi_{i}\) denotes the steering angle of each wheel. The exponential form prioritizes smooth steering operations.  
Oscillation Penalty Term \((\delta_{3}\cdot\mathcal{P}_{t})\): Suppresses steering oscillations through a sliding-window mechanism:  
\[\mathcal{P}_{t}=-\sum_{i=1}^{4}\eta_{i}\Big{[}\lambda_{1}^{(i)}\Delta_{\text{ mean}}^{(i)}(t)+\lambda_{2}^{(i)}\sigma^{(i)}(t)+\lambda_{3}^{(i)}R^{(i)}(t) \Big{]}\]  
where:  
\(\Delta_{\text{mean}}^{(i)}\): Mean absolute difference of steering channel \(i\) over window \(\mathcal{W}_{t}\)  
\(\sigma^{(i)}\): Variance of steering actions in channel \(i\)  
\(R^{(i)}\): Range (max-min) of steering actions in channel \(i\)  
\(\eta_{i}\): Channel-specific weights (\(\sum\eta_{i}=1\))  
\(\lambda_{1,3}^{(i)}\): Penalty coefficients for temporal variation characteristics  
The sliding window \(\mathcal{W}_{t}=\{a_{t-N+1},...,a_{t}\}\) maintains the latest \(N\) steering actions (\(a_{t}\in\mathbb{R}^{4}\)) using FIFO updating. This multi-timescale penalty simultaneously suppresses high-frequency jitter (via \(\Delta_{\text{mean}}\)), amplitude fluctuations (via \(\sigma\)), and abrupt changes (via \(R\)).