# Dynamic Path Planning for Vehicles Based on Causal State-Masking Deep Reinforcement Learning
## 3 Problem Formulation
### Formalization of the Markov Decision Process  
Based on the definitions in Section 3.1, we formalize the vehicle's actual control and dynamic path-planning optimization problem as an MDP to enable effective control decision making and optimization using DRL algorithms [38]. An MDP is a mathematical framework widely used for modeling decision-making problems, especially in the presence of uncertainty and dynamic environments. Within this framework, as represented in Figure 2, the agent's actions are defined to be the same as the vehicle's control variables, i.e., \(\mathbf{a}=\mathbf{u}\). The state is defined to include both the vehicle's current state and the environmental state, specifically represented as  
\[\mathbf{s}= \begin{bmatrix}\mathbf{s}_{p}&d_{\text{min}}&\mathbf{s}_{g}\end{bmatrix}  = \begin{bmatrix}x&y&\psi&v_{x}&v_{y}&\omega&d_{\text{min}}&x_{g}&y_{g}\end{bmatrix}  \tag{9}\]  
To ensure that the agent's actions are executable and can maintain the vehicle's stability, we constrain the vehicle's control inputs within predefined ranges. These constraints not only reflect the vehicle's physical limitations but also help prevent excessive control inputs that could lead to vehicle instability or loss of control. The specific constraints are as follows:  
\[\left\{ \begin{array}{l}F_{x,\text{min}}\leq F_{x}(t)\leq F_{x,\text{max}}\\ F_{y,\text{min}}\leq F_{y}(t)\leq F_{y,\text{max}}\\ M_{z,\text{min}}\leq M_{z}(t)\leq M_{z,\text{max}}\end{array} \right. \tag{10}\]  
These limitations ensure that the control inputs remain within a safe and executable range for the vehicle, avoiding unnecessary risks caused by excessively large or small forces and moments. For the vehicle to avoid collisions with any obstacles throughout the path-planning process, we introduce the concept of safety distance, in which the distance \(d_{i}\) between the vehicle and all obstacles must be greater than a preset safety distance \(d_{s}\). This constraint can be expressed as  
\[d_{i}(t)=\sqrt{(x(t)-x_{i}(t))^{2}+(y(t)-y_{i}(t))^{2}}\geq d_{s} \tag{11}\]  
where \(i\) denotes the number of obstacles and \(x_{i}(t)\) and \(y_{i}(t)\) represent the longitudinal and lateral positions, respectively, of the \(i\)th obstacle in the global coordinate system at time \(t\). By mandating that the vehicle maintain a distance of at least \(d_{s}\) from all obstacles, we can effectively avoid potential collision risks. We define the condition for the vehicle to reach the endpoint as the distance between the vehicle's current position, and the endpoint as less than a predefined distance \(d_{g}\):  
\[d(t)=\sqrt{(x(t)-x_{g})^{2}+(y(t)-y_{g})^{2}}\leq d_{g} \tag{12}\]  
Figure 2: Agent and environment interactions according to the Markov model. This condition ensures that the vehicle can reach the target position during the path-planning process while allowing a certain tolerance to accommodate uncertainties and dynamic changes in the actual environment. In summary, the optimization problem for dynamic path planning can be formalized as  
\[ \begin{split}\mathbf{a}^{*}=&\arg\min_{\pi_{\theta}} \sum_{t=0}^{T}\Delta t\big{[}\alpha C_{\text{pat}}(t)+\beta C_{\text{smo}}(t)+ \lambda C_{\text{col}}(t)\big{]}\\ \text{s.t.}&\quad F_{x,\text{min}}\leq F_{x}(t) \leq F_{x,\text{max}}\\ &\quad F_{y,\text{min}}\leq F_{y}(t)\leq F_{y,\text{max}}\\ &\quad M_{z,\text{min}}\leq M_{z}(t)\leq M_{z,\text{max}}\\ &\quad d_{i}(t)\geq d_{s},\quad\forall i,\forall t\\ &\quad d(t)\leq d_{g}\end{split}  \tag{13}\]  
The objective of this optimization problem is to minimize the cumulative cost from the start point to the target point by designing an appropriate control policy \(\pi_{\theta}\) while satisfying dynamic and environmental constraints. Solving this problem provides both theoretical foundations and practical guidance for the autonomous path planning of intelligent vehicles in complex dynamic environments. To guide the agent's action decisions and optimize the planned path toward the predefined target, we design the following reward function:  
\[r=