## 6 Conclusions  
In this study, we compared the performance of three invalid action masking algorithms, On-PIAM, Off-PIAM, and CO-IAM. These algorithms optimize the masked policy, the original policy, and both policies, respectively. Our main contribution lies in the successful performance of the proposed Off-PIAM and CO-IAM algorithms when removing masking after training, while CO-IAM fails to work. Through our work, we have discovered several key findings. Firstly, while On-PIAM solves all tasks within fewer time steps, it fails to achieve the expected return when masking is removed. Secondly, we found that the naive policy gradient is equivalent to the CO-IAM algorithm, which optimizes the masked policy and the original policy in parallel. Thirdly, CO-IAM demonstrates superior performance compared to On-PIAM and scales well on complex tasks. Lastly, Off-PIAM outperforms other algorithms, regardless of whether masking is present or not. Based on our empirical findings, we recommend utilizing the Off-PIAM algorithm for addressing most tasks, as it consistently delivers strong performance. Additionally, the CO-IAM algorithm should be employed for handling more complex tasks. One limitation of this study is its focus on the effects of invalid action masking on RL agents in RTS games. Therefore, the findings cannot be generalized to all solutions, such as rule-based approaches or optimization-based \begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Algorithms** & **Map** & \(r_{\text{episode}}\) & \(a_{\text{null}}\) & \(a_{\text{busy}}\) & \(a_{\text{owner}}\) & \(\mathbf{f_{solve}}\) & \(\mathbf{f_{first}}\) \\ \hline CO-IAM & \(4\times 4\) & **38.50** & – & – & – & 4.17\% & 0.03\% \\  & \(10\times 10\) & **39.30** & – & – & – & 5.46\% & 0.04\% \\  & \(16\times 16\) & **39.30** & – & – & – & 6.21\% & 0.06\% \\  & \(24\times 24\) & **37.20** & – & – & – & 10.10\% & 0.05\% \\ \hline Off-PIAM & \(4\times 4\) & **39.98** & – & – & – & 4.57\% & 0.04\% \\  & \(10\times 10\) & **39.67** & – & – & – & 9.66\% & 0.02\% \\  & \(16\times 16\) & **39.55** & – & – & – & 9.16\% & 0.04\% \\  & \(24\times 24\) & **35.20** & – & – & – & 16.15\% & 0.04\% \\ \hline On-PIAM [23] & \(4\times 4\) & 36.55 & – & – & – & **3.49\%** & 0.04\% \\  & \(10\times 10\) & 35.40 & – & – & – & **4.50\%** & 0.02\% \\  & \(16\times 16\) & 33.58 & – & – & – & **5.40\%** & 0.04\% \\  & \(24\times 24\) & 31.85 & – & – & – & **7.09\%** & 0.04\% \\ \hline Penalty-0.01 & \(4\times 4\) & **39.33** & 59.62 & 1.70 & 19.38 & 6.01\% & 0.07\% \\  & \(10\times 10\) & 32.00 & 108.82 & 0.12 & 4.62 & 13.98\% & 1.34\% \\  & \(16\times 16\) & 8.89 & 30.68 & 0.00 & 0.60 & – & 1.10\% \\  & \(24\times 24\) & 11.44 & 163.05 & 0.30 & 1.45 & – & 0.93\% \\ \hline Penalty-0.1 & \(4\times 4\) & 29.97 & 18.55 & 0.78 & 5.80 & 34.48\% & 0.07\% \\  & \(10\times 10\) & 0.00 & 0.00 & 0.00 & 0.00 & – & 3.74\% \\  & \(16\times 16\) & 7.97 & 27.72 & 0.28 & 0.75 & – & 1.29\% \\  & \(24\times 24\) & 0.64 & 94.70 & 0.22 & 0.88 & – & 0.57\% \\ \hline CO-IAM-MR & \(4\times 4\) & 21.37 & **0.27** & **0.00** & 152.59 & – & 0.03\% \\  & \(10\times 10\) & 33.00 & 131.89 & **0.00** & **0.01** & – & 0.04\% \\  & \(16\times 16\) & 32.25 & 133.34 & **0.00** & **0.00** & – & 0.06\% \\  & \(24\times 24\) & 25.10 & 149.11 & **0.00** & **0.15** & – & 0.05\% \\ \hline Off-PIAM-MR & \(4\times 4\) & **39.88** & 9.53 & 17.81 & **1.31** & **7.22\%** & 0.04\% \\  & \(10\times 10\) & **39.33** & **9.86** & 4.56 & **1.59** & **34.92\%** & 0.02\% \\  & \(16\times 16\) & **38.65** & **10.19** & 1.95 & **1.01** & **25.91\%** & 0.04\% \\  & \(24\times 24\) & **35.23** & **15.90** & 5.97 & **0.18** & – & 0.04\% \\ \hline On-PIAM-MR & \(4\times 4\) & 7.13 & 49.33 & **0.00** & 133.20 & – & 0.04\% \\  & \(10\times 10\) & 0.23 & 188.20 & **0.00** & 10.89 & – & 0.02\% \\  & \(16\times 16\) & 0.15 & 195.66 & **0.00** & 3.90 & – & 0.04\% \\  & \(24\times 24\) & 0.48 & 197.64 & **0.00** & 1.37 & – & 0.04\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Results averaged over four random seeds. approaches.