## 3 Invalid Action Masking
### Equivalence Verification  
This section will be concerned with verifying the equivalence of logit-level masking and action-level masking, based on which the conclusions derived by [PERSON] et al. [23] are also applicable to action-level masking. The equivalence holds if two types of invalid action masking satisfy that:  
\[\pi_{\theta}^{m}(a|s)=\pi_{\theta}^{\varpi}(a|s),\forall s\in\mathcal{S}, \forall a\in\mathcal{A}. \tag{9}\]  
For any state \(s\in\mathcal{S}\), the action space can be divided into two parts, valid action space \(\mathcal{A}_{\text{v}}(s)\) and invalid action space \(\mathcal{A}_{\text{inv}}(s)\). We have:  
\[\mathcal{A}=\mathcal{A}_{\text{v}}(s)\cup\mathcal{A}_{\text{inv}}(s),\ \mathcal{A}_{\text{v}}(s)\cap\mathcal{A}_{\text{inv}}(s)=\emptyset. \tag{10}\]  
According to (5), for any valid action \(a_{\text{v}}\in\mathcal{A}_{\text{v}}(s)\), the action probability of the logit-level masked policy is:  
\[\pi_{\theta}^{m}(a_{\text{v}}|s) =\frac{\exp(m(l_{\theta}(a_{\text{v}}|s)))}{\sum_{a_{i}\in\mathcal{ A}}\exp(m(l_{\theta}(a_{i}|s)))} \tag{11}\] \[=\frac{\exp(m(l_{\theta}(a_{\text{v}}|s)))}{\sum\limits_{a_{j}\in \mathcal{A}_{\text{inv}}}\exp(m(l_{\theta}(a_{i}|s)))+\sum\limits_{a_{j}\in \mathcal{A}_{\text{inv}}}\exp(m(l_{\theta}(a_{j}|s)))}\] \[=\frac{\exp(l_{\theta}(a_{\text{v}}|s))}{\sum\
olimits_{a_{j}\in \mathcal{A}_{\text{v}}}\exp(l_{\theta}(a_{i}|s))}\]According to (8), for any valid action \(a_{\rm v}\in\mathcal{A}_{\rm v}(s)\), the action probability of the action-level masking policy is:  
\[\pi_{\theta}^{\rm\omega}(a_{\rm v}|s) =\frac{\pi_{\theta}(a_{\rm v}|s)\omega(a_{\rm v}|s)}{\sum_{a_{i} \in\mathcal{A}}\pi_{\theta}(a_{i}|s)\omega(a_{i}|s)}=\frac{\pi_{\theta}(a_{\rm v }|s)}{\sum_{a_{i}\in\mathcal{A}_{\rm v}(s)}\pi_{\theta}(a_{i}|s)}\] \[=\frac{\exp(l_{\theta}(a_{\rm v}|s))}{\sum_{a_{j}\in\mathcal{A}} \exp(l_{\theta}(a_{j}|s))}\Bigg{/}\sum_{a_{i}\in\mathcal{A}_{\rm v}(s)}\frac{ \exp(l_{\theta}(a_{i}|s))}{\sum_{a_{j}\in\mathcal{A}}\exp(l_{\theta}(a_{j}|s) )} \tag{12}\] \[=\frac{\exp(l_{\theta}(a_{\rm v}|s))}{\sum_{a_{i}\in\mathcal{A}_{ \rm v}(s)}\exp(l_{\theta}(a_{i}|s))}\]  
In light of (11) and (12), one has:  
\[\pi_{\theta}^{m}(a_{\rm v}|s)=\pi_{\theta}^{\rm\omega}(a_{\rm v}|s),\forall s \in\mathcal{S},\forall a_{\rm v}\in\mathcal{A}_{\rm v}(s). \tag{13}\]  
Obviously, the probabilities of invalid actions of both policies are equal to zero:  
\[\pi_{\theta}^{m}(a_{\rm inv}|s)=\pi_{\theta}^{\rm\omega}(a_{\rm inv}|s)=0, \forall s\in\mathcal{S},\forall a_{\rm inv}\in\mathcal{A}_{\rm inv}(s). \tag{14}\]  
Combining (13) with (14), we obtain that:  
\[\pi_{\theta}^{m}(a|s)=\pi_{\theta}^{\rm\omega}(a|s),\forall s\in\mathcal{S}, \forall a\in\mathcal{A}. \tag{15}\]  
Thus, logit-level masking and action-level masking are equivalent. This is a significant result because it shows that the conclusions of logit-level masking [23] also apply to action-level masking. It is important to note that the substitution of negative infinity with a significantly negative value may introduce minor rounding errors. However, these errors do not impact the overall equivalence. While both action-level masking and logit-level masking are practically equivalent, action-level masking enables the explicit derivation of policy gradients, whereas logit-level masking does not. In the following, action-level masking is used to derive the policy gradient algorithm for invalid action masking.