# Learning State-Specific Action Masks for Reinforcement Learning
## 4 Approach
### Learning the Action Mask Supervised  
We train the action mask with supervised learning, and the labels are generated through the DBSCAN clustering algorithm [35], as outlined in Algorithm 1. The objective of the mask model is to minimize the Hamming Distance between \(I_{t}\) and label \(L_{t}\), and the loss function of the mask model is:  
\[\mathcal{L}_{(}\phi)=\log mask_{\phi}(I_{t}|s_{t})d_{Hamming}(mask_{\phi}(I_{ t}|s_{t}),L_{t}) \tag{10}\]  
To achieve (9), a natural method is to aggregate all vectors \([s_{t+1},r_{t}]\), grouping actions with small \(d(a^{i},a^{j})\) as defined in (7) into \(\hat{A}_{B_{s}}\). Additionally, we construct \([s_{t},0]\) as an extra vector lead by the invalid action \(a^{|\mathcal{A}|+1}\) and aggregate it with other \([s_{t+1},r_{t}]\) vectors. Then, those actions that lead to \([s_{t+1},r_{t}]\) in the same group with \([s_{t},0]\) are also invalid actions whose masks should be set to 0. Figure 1: The categorical mask model. The squares represent vector values, and in grayscale, colors represent numerical values, where white corresponds to 0, and black corresponds to 1. The encoder layer produces \(2\times|\mathcal{A}|\) probabilities, forming the logits of \(|\mathcal{A}|\) groups, each containing 2 mask categories. For each dimension \(i\), the mask \(I^{i}\) is sampled from categorical \(i\). ```
0: batch data \(B\), clustering parameter \(\epsilon\) and \(minPts=1\) for\(s_{t}\in B\)do for\(a^{i}\in\mathcal{A}\)do  Get the next states: \(s^{i}_{t+1}\sim\hat{p}(s_{t},a^{i})\)  Get the reward: \(r^{i}_{t}\sim\hat{r}(s_{t},a^{i})\) endfor  Build the behavioral vectors: \(V_{t}=[[s^{i}_{t+1},r^{i}_{t+1}]]_{i=0,\dots,|\mathcal{A}|}]\)  Add the extra feature: \(V_{t}=V_{t}\bigcup[s_{t},0]\)  Aggregate \(V_{t}\) using DBSCAN with parameter \(\epsilon\) and \(minPts\): \(C_{t}=[c^{0}_{t},\dots,c^{|\mathcal{A}|+1}_{t}]\)  Build mask label \(L_{t}\) to achieve (8) and (9)  Update the mask model with Hamming Distance error: \(\mathcal{L}(\phi)\) Equation (10) endfor
```  
**Algorithm 1** Training an action mask model  
At each step, we update the mask model on a sampled batch of states. For each state \(s_{t}\) in the batch, we estimate \(s^{i}_{t+1}\) and \(r^{i}_{t}\) for each action \(a^{i}\in\mathcal{A}\) and concatenate them as the behavioral vectors \(v^{0},\dots,v^{|\mathcal{A}|}\) of action \(a^{i}\). Then, we construct \(v^{|\mathcal{A}|+1}=[s_{t},0]\) as an extra vector indicating the unchanged situation, and aggregate \([v^{0},\dots,v^{|\mathcal{A}|},v^{|\mathcal{A}|+1}]\) as clusters \(C\), where \(c^{i}\) is the cluster index number for feature \(v^{i}\). In each cluster, we appoint the action with the lowest index as a kernel whose mask label is 1 and others as alternatives whose mask labels are 0. In particular, all actions cause \([s^{i}_{t+1},r^{i}_{t}]\) clustered with the extra vector \([s_{t},0]\) are seen as invalid actions, and thus, the action masks of them are all 0. The mask model training is plugged into the vanilla RL training procedure to get a training loop, shown in Algorithm 2. The RL policy \(\pi_{\theta}(a_{t}|s_{t})\), the transition model \(\hat{p}(s_{t+1}|s_{t},a_{t})\), the reward model \(\hat{r}(r_{t+1}|s_{t},a_{t})\) and the mask model \(mask_{\theta}(l_{t}|s_{t})\) are trained in turn. The blue lines are individualized for the RL algorithms and we combine our method with two popular RL algorithms, DQN and PPO, as examples (see implementation details in Appendix B). When employing the untrained mask model, we eliminate lines 6 and 7 from the Algorithm 2.