# Learning State-Specific Action Masks for Reinforcement Learning
## 4 Approach
### Bisimulation Metric on Actions  
Bisimulation, as defined by [PERSON] [34], is a widely used concept applied to the state space, signifying behavioral equivalence among states. To establish a reasonable metric for the action mask, we introduce a concept akin to bisimulation, aiming to indicate behavioral equivalence among actions. **Definition 2** (Bisimulation Relations on Actions (BRA)). : _In task \(\mathcal{M}\) with discrete action space \(\mathcal{A}\), for \(a^{i}\in\mathcal{A}\) and \(a^{j}\in\mathcal{A}\), define \(a^{i}\) and \(a^{j}\) is equivalent under Bisimulation Relation \(B\) if:_  
\[r(s,a^{i}) =r(s,a^{i})\quad\forall s\in\mathcal{S} \tag{3}\] \[p(s^{\prime}|s,a^{i}) =p(s^{\prime}|s,a^{j})\quad\forall s\in\mathcal{S} \tag{4}\]  
_and we define \(\mathcal{A}_{B}\) as a partition of action space \(\mathcal{A}\) that \(a^{i}=a^{j}\) under bisimulation relation \(B\) are separated in the same set, and \(A_{B}\in\mathcal{A}_{B}\) is an action group._  
In task \(\mathcal{M}\), it is obvious that if \(a_{i}\) and \(a_{j}\) belong to the same bisimilar action set \(A_{B}\), then \(a_{i}\) and \(a_{j}\) have the same behavioral consequences. In other words, removing \(a_{j}\) does not affect the MDP. While a global definition of behavioral equivalence on actions is reasonable, in most tasks, each action group \(A_{B}\) typically contains only one element, implying that not all actions can be removed. For example, in a Maze task with four actions stands for four directions, no two actions yield the same behavioral consequences across all states. However, when the agent moves to a corner, there may be more than one action that leads to staying in the same place and receiving no reward, which causes identical behavioral consequences. Therefore, we adopt a variant of BRA that defines the equivalence relation of actions based on states. **Definition 3** (Bisimulation Relations on Actions by States (BRAS)). : _In task \(\mathcal{M}\) with discrete action space \(\mathcal{A}\), given state \(s\), for \(a^{i}\in\mathcal{A}\) and \(a^{j}\in\mathcal{A}\), define \(a^{i}\) and \(a^{j}\) is equivalent under Bisimulation Relation B if:_  
\[r(s,a^{i}) =r(s,a^{j}) \tag{5}\] \[p(s^{\prime}|s,a^{j}) =p(s^{\prime}|s,a^{j})\]  
_and we define \(\mathcal{A}_{B_{s}}\) as the partition of action space \(\mathcal{A}\) that \(a^{i}=a^{j}\) under bisimulation relation \(B_{s}\) are separated in the same set, and \(A_{B_{s}}\in\mathcal{A}_{B_{s}}\) is an action group._  
We now present our proposition, asserting that equivalent actions under BRAS yield the same behavioral consequence in the MDP, signifying that actions within the set are interchangeable. **Proposition 1**. : _In task \(\mathcal{M}\) with discrete action space \(\mathcal{A}\), given state \(s\), if \(a^{i},a^{j}\in A_{B_{s}}\), then the optimal policy of task \(\mathcal{M}=(\mathcal{S},\mathcal{A},p,r,p_{0},\gamma,T)\) and task \(\mathcal{M}^{\prime}=(\mathcal{S},\mathcal{A}^{\prime},p,r,p_{0},\gamma,T)\) are same, where:_  
\[\mathcal{A}^{\prime}=\mathcal{A}\backslash(a^{j}|s) \tag{6}\]  
The proof of Proposition 1 can be found in Appendix A.  
Filtering actions strictly according to BRAS is impractical, since BRAS is sensitive to the next state and the reward. Therefore, we propose to measure the distance among actions to soften the BRAS constraints based on the smooth transition assumption that in task \(\mathcal{M}\), if \(s_{i}\approx s_{j}\), then \(\forall a\in\mathcal{A}\), \(p(\cdot|s_{i},a)\approx p(\cdot|s_{j},a)\). **Definition 4** (Bisimulation Distance on Actions by States). : _Given discrete action \(a^{i}\) and \(a^{j}\) and task \(\mathcal{M}\), define bisimulation distance \(d(a^{i},a^{j})\) as the \"behavioral distance\" between \(a^{i}\) and \(a^{j}\) in task \(\mathcal{M}\):_  
\[d(a^{i},a^{j}|s)=|r(s,a^{i})-r(s,a^{j})|+|p(\cdot|s,a^{i})-p(\cdot|s,a^{j})| \tag{7}\]  
When \(d(a^{i},a^{j}|s)=0\), it is evident that \(a^{i}\) and \(a^{j}\) belong to the same bisimulation group \(A_{B_{s}}\).