# Dynamic Path Planning for Vehicles Based on Causal State-Masking Deep Reinforcement Learning
## 4 Causal State-Masking TD3 Algorithm
### Actor Networks  
The actor network is responsible for generating optimal actions based on the current state. The design of the policy network directly affects the agent's decision-making capability. To enhance the model's focus on key causal features and improve convergence and generalization capabilities in path-planning tasks, we introduce state masks and intervention mechanisms into the actor network. Specifically, a trainable state-mask vector \(\mathbf{m}=[m_{1},m_{2},\dots,m_{n}]\), where \(m_{i}\in[0,1]\), is added before the input layer of the actor network. The original state \(\mathbf{s}_{t}\) is processed through the mask to obtain the masked state \(\mathbf{\hat{s}}_{t}\). The state-mask vector \(\mathbf{m}\) highlights causal features directly related to path-planning decisions while suppressing non-causal features and noise interference [43]. In complex dynamic environments, the state space typically contains a large number of features, only a subset of which directly influences path-planning decisions. By introducing state masks, the model can automatically identify and reinforce these key features while suppressing irrelevant or interfering features, enhancing the efficiency and stability of the policy. During the training process of the agent, the mask vector is treated as a trainable parameter and optimized through backpropagation. This means that the optimization of the mask vector is synchronized with the optimization of the policy network parameters, allowing the mask to adaptively adjust to accommodate different environments and task requirements. To enable the actor network to better learn the true relationship between causal features and path-planning decisions, we introduce intervention mechanisms and causal consistency loss during the training process. The specific operations are as follows:  
1. Intervention feature selection. For each training sample, randomly select a causal feature \(s_{t,k}\). This process simulates possible causal disturbances in real environments, forcing the model to learn robust policies under different causal contexts. 2. Intervention value generation. Apply perturbations to the selected causal feature to generate an intervention value \(s_{t,k}\): \[\hat{s}_{t,k}=s_{t,k}+\delta,\quad\delta\sim\mathcal{N}(0,\sigma_{\text{int}} ^{2})\] (22) where \(\delta\) follows a Gaussian distribution with a mean of zero and variance \(\sigma_{\text{int}}^{2}\). This perturbation simulates unpredictable changes or noise in the environment, helping the model learn robustness when causal features change. 3. Intervention state construction. Generate the intervened state \(\mathbf{s}_{t,\text{do}}\): \[\mathbf{s}_{t,\text{do}}=[s_{t,1},\dots,s_{t,k-1},\hat{s}_{t,k},s_{t,k+1}, \dots,s_{t,n}]\] (23)Replacing the selected causal feature with the intervention value constructs a new state, reflecting the environmental state when the feature is disturbed. 4. Intervention state masking. Apply mask processing to the intervened state to obtain \(\mathbf{\hat{s}}_{t,\mathrm{do}}=\mathbf{s}_{t,\mathrm{do}}\odot\mathbf{m}\). Through state masks, adjust the intervened state features, allowing the model to focus on key causal features under intervention conditions. 5. Policy network output calculation. Employ the policy network to generate actions \(\mathbf{a}_{t}\) and \(\mathbf{a}_{t,\mathrm{do}}\) on \(\mathbf{\hat{s}}_{t}\) and \(\mathbf{\hat{s}}_{t,\mathrm{do}}\), respectively. By comparing the actions generated under original and intervened states, the model can learn the true impact of causal feature changes on decisions. 6. Causal consistency loss calculation. Compute the causal consistency loss \(L_{c}(\theta)\). This loss term encourages the policy network to maintain consistency in action outputs when facing causal feature interventions, thereby reducing dependence on non-causal features and enhancing the model's causal inference capabilities.