## 3 Invalid Action Masking
### Masking Changes Policy Gradient  
This section focuses on the impact of invalid action masking on policy updates. [PERSON] et al. [23] designed a numerical example to illustrate that the gradient generated by invalid action masking differs from the naive policy gradient. As a supplement, we further investigate this finding and reveal the difference between the naive policy gradient and the invalid action masking policy gradient. In regular policy-based algorithms, the original policy \(\pi_{\theta}\) is employed to collect samples and calculate the policy gradient using the policy gradient theorem [25]. To avoid invalid actions, we apply the invalid action masking to the policy \(\pi_{\theta}\), resulting in a masked policy denoted as \(\pi_{\theta}^{\rm\omega}\). The straightforward approach to policy updating is the direct utilization of the original policy gradient, called _naive policy gradient_:  
\[g_{\rm naive}=\mathbb{E}_{\tau\sim\pi_{\theta}^{\rm\omega}}\left[\sum_{t=0}^ {H}\
abla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\hat{A}_{t}\right]. \tag{16}\]  
The naive policy gradient is intuitively improper because the policy gradient algorithm is an _on-policy_ algorithm, but the target policy (\(\pi_{\theta}\)) and behavior policy (\(\pi_{\theta}^{\rm\omega}\)) are different. [PERSON] et al. [23] proposed the _invalid action policy gradient_, which is the policy gradient for the masked policy \(\pi_{\theta}^{\rm\omega}\)  
\[g_{\rm mask}=\mathbb{E}_{\tau\sim\pi_{\theta}^{\rm\omega}}\left[\sum_{t=0}^{H }\
abla_{\theta}\log\pi_{\theta}^{\rm\omega}(a_{t}|s_{t})\hat{A}_{t}\right]. \tag{17}\]The distinction between the two policy gradients lies in different target policies for log probability calculation. Next, we provide a theoretical analysis to reveal the difference between the naive policy gradient (16) and invalid action policy gradient (17). **Theorem 1**. : _Consider a masked policy \(\pi_{\theta}^{\varDelta}\) produced by applying invalid action masking to the original policy \(\pi_{\theta}\). The difference between the naive policy gradient (16) and invalid action policy gradient (17) is a state-dependent gradient_  
\[g_{\text{mask}}-g_{\text{naive}}=\mathbb{E}_{\tau\sim\pi_{\theta}^{\varDelta }}\left[\sum\limits_{t=0}^{H}\frac{\hat{A}_{t}}{\mathcal{I}(s_{t};\theta)}\sum \limits_{a_{i}\in\mathcal{A}_{\text{inv}}(s_{t})}\
abla_{\theta}\pi_{\theta}(a_ {i}|s_{t})\right] \tag{18}\]  
_where \(\mathcal{I}(s_{t};\theta)=\sum_{a_{j}\in\mathcal{A}_{\text{v}}(s_{t})}\pi_{ \theta}(a_{j}|s_{t})\)._  
Proof of Theorem 1.: According to (8), the gradient item in \(g_{\text{mask}}\) can be written as:  
\[\
abla_{\theta}\log\pi_{\theta}^{\varDelta}(a_{t}|s_{t})=\
abla_{\theta}\log \frac{\pi_{\theta}(a_{t}|s_{t})\varDelta(a_{t}|s_{t})}{\sum_{a_{i}\in\mathcal{ A}}\pi_{\theta}(a_{i}|s_{t})\varDelta(a_{i}|s_{t})}. \tag{19}\]  
Since \(a_{t}\) is sampled from the masked policy \(\pi_{\theta}^{\varDelta}\), \(a_{t}\) must be a valid action, and \(\varDelta(a_{t}|s_{t})=1\). Then, the gradient item (19) can be written as:  
\[ \begin{split}\
abla_{\theta}\log\pi_{\theta}^{\varDelta}(a_{t}| s_{t})&=\
abla_{\theta}\log\frac{\pi_{\theta}(a_{t}|s_{t})}{\sum \limits_{a_{i}\in\mathcal{A}}\pi_{\theta}(a_{i}|s_{t})\varDelta(a_{i}|s_{t})} \\ &=\
abla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})-\
abla_{\theta} \log\sum\limits_{a_{i}\in\mathcal{A}}\pi_{\theta}(a_{i}|s_{t})\varDelta(a_{i} |s_{t})\\ &=\
abla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})-\frac{\
abla_{ \theta}\sum\limits_{a_{i}\in\mathcal{A}}\pi_{\theta}(a_{i}|s_{t})\varDelta(a_ {i}|s_{t})}{\sum\limits_{a_{j}\in\mathcal{A}}\pi_{\theta}(a_{j}|s_{t})\varDelta (a_{j}|s_{t})}\end{split}