# Vehicle Lateral Control Based on Augmented Lagrangian DDPG Algorithm
## 2 Preliminaries  
To address the controller design issues encountered in vehicle trajectory tracking, this study restructures the problem as a Markov decision process and applies deep reinforcement learning techniques to solve it. Training vehicle trajectory tracking controllers using deep reinforcement learning methods can lead to scenarios where the vehicle collides with lane boundaries, prematurely ending the reinforcement learning training process. To mitigate this challenge, we introduce constrained reinforcement learning within lane boundary limitations, formally establishing a Safe Reinforcement Learning (Safe RL) framework with collision-avoidance guarantees. A Constrained Markov Decision Process (CMDP) \(\mathcal{M}_{\mathcal{C}}\) is defined by the tuple \((\mathcal{S},\mathcal{A},\mathbb{P},R,C,\rho_{0},\gamma)\), where \(\mathcal{S}\) and \(\mathcal{A}\) are the state and action spaces, respectively; \(\mathbb{P}:\mathcal{S}\times\mathcal{A}\to P(s)\) represents the transition probability \(\mathbb{P}(s^{\prime}|s,a)\) from state \(s\) to \(s^{\prime}\) under action \(a\); \(R:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to\mathbb{R}\) is the reward function, yielding the immediate reward \(r(s,a,s^{\prime})\) when transitioning from \(s\) to \(s^{\prime}\) via action \(a\); \(C\) is a set of safety constraints, where each \(c_{i}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}\) is a safety cost function with an associated threshold \(b_{i}\) for \(i=1,\ldots,m\); \(\rho_{0}:\mathcal{S}\to[0,1]\) is the initial state distribution; and \(\gamma\) is the discount factor for future rewards. A stationary policy \(\pi_{\theta}\) is a probability distribution over actions given states, with \(\pi_{\theta}(a|s)\) denoting the probability of taking action \(a\) in state \(s\). The set of all stationary policies is \(\Pi_{\theta}=\{\pi_{\theta}:\theta\in\mathbb{R}^{p}\}\). The state-value function for \(\pi_{\theta}\) is defined as \(V_{\pi_{\theta}}(s)=\mathbb{E}_{\pi_{\theta}}\big{[}\sum_{t=0}^{\infty}\gamma ^{t}r_{t+1}\mid s_{0}=s\big{]}\), where \(\mathbb{E}_{\pi_{\theta}}[\cdot|\cdot|]\) denotes the expectation over actions selected by \(\pi_{\theta}\). The objective of reinforcement learning is to maximize \(J_{r}(\pi_{\theta})=\mathbb{E}_{s\sim p_{0}(\cdot)}[V_{\pi_{\theta}}(s)]\). In CMDPs, the expected discounted cost for policy \(\pi_{\theta}\) is \(J_{C_{i}}(\pi_{\theta})=\mathbb{E}\big{[}\sum_{t=0}^{\infty}\gamma^{t}c_{i}( s_{t},a_{t})\big{]}\). The feasible policy set is defined as \(\Pi_{C}=\bigcap_{i=1}^{m}\{\pi_{\theta}\in\Pi_{\theta}\mid J_{C_{i}}(\pi_{ \theta})\leq b_{i}\}\). The goal of CMDP is to find the optimal policy \(\pi^{*}=\arg\max_{\pi_{\theta}\in\Pi_{C}}J_{r}(\pi_{\theta})\). Definition 1: (Optimization Problem of MDP with Instantaneous Constraints) Consider a Markov Decision Process defined by state transition probabilities \(s_{t+1}\sim p_{t}(\cdot\mid s_{t},a_{t})\), initial state distribution \(s_{0}\sim p_{0}\), a bounded reward function \(r(s,a)\), and a bounded constraint function \(g(s,a)\). The objective is to find the optimal policy \(\pi^{*}\) that maximizes the expected discounted cumulative reward while satisfying the given constraints. The optimization problem can be formulated as follows:  
\[ \begin{split}\max_{\pi}&\;J_{r}(\pi)=\mathbb{E} \Bigg{[}\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\mid\pi\Bigg{]}\\ s.t.&\;s_{t+1}\sim p(\cdot\mid s_{t},a_{t}),\forall t \in\{0,1,\ldots\},\\ &\;a_{t}\sim\pi(s_{t}),\forall t\in\{0,1,\ldots\},\\ &\;s_{0}\sim p_{0},\\ &\;\mathbb{E}[g(s_{t},a_{t})\mid\pi]\leq 0,\forall t\in\{0,1, \ldots\}.\end{split}