## 5 Empirical Analysis
### Results  
The experimental results are presented in Figures 4 and 5 and Table 2. Here, we provide some important observations.  
Invalid action penalty is not a robust way to prevent the agent from taking invalid actions.  
In Figure 4, we observe that Penalty-0.01 achieves more than 30 rewards in \(4\times 4\) and \(10\times 10\) maps. It even surpasses On-PIAM in the \(4\times 4\) map but can hardly extend to other maps. Further, the performance of Penalty-0.1 is obviously worse than Penalty-0.01. However, Table 2 shows that \(a_{\text{null}}\), \(a_{\text{busy}}\), and \(a_{\text{owner}}\) of Penalty-0.1 is better than Penalty-0.01. It demonstrates that the larger penalty causes lower probabilities of invalid actions, but it fails to obtain more positive rewards. Thus, we have to make more efforts to tune the value of the negative reward when we adopt the invalid action penalty.  
On-PIAM solves all tasks within fewer time steps than Off-PIAM and CO-IAM, but it fails to achieve expected asymptotic performance. As shown in Table 2, On-PIAM's \(t_{\text{solve}}\) is roughly 3% to 7% and very similar across different map sizes. However, we can observe that the episode return of On-PIAM achieves 34 on average, but Off-PIAM and CO-IAM achieve almost 40. The green curves in Figure 4 verify that the episode return curves of On-PIAM are not stable, and its asymptotic performance is worse than Off-PIAM and CO-IAM. Furthermore, Figure 5 shows that once the masking is removed, On-PIAM-MR only sightly works in \(4\times 4\) map and cannot obtain rewards on other maps. In Table 2, we also notice that On-PIAM-MR has larger probabilities of choosing \(a_{\text{null}}\) and \(a_{\text{owner}}\). This is expected since On-PIAM aims to optimize the masked policy rather than the original policy.  
CO-IAM outperforms On-PIAM and scales well on complex tasks. As shown in Figure 5 and Table 2, CO-IAM is better than On-PIAM with or without masking. When the masking is removed, the episode return of CO-IAM-MR still maintains roughly 30. It verifies our idea that although \(g_{\text{naive}}\) seems to be an improper policy gradient, it indeed optimizes the masked policy and the original policy concurrently. CO-IAM's \(t_{\text{solve}}\) is better than Off-PIAM. In general, the performance of CO-IAM shows that CO-IAM seems to be a combination of On-PIAM and Off-PIAM. In the \(24\times 24\) map, CO-IAM obtains 37.2 rewardson average and outperforms Off-PIAM and On-PIAM. It demonstrates that CO-IAM has a better exploration ability to deal with complex tasks.  
Off-PIAM achieves the best performance when the masking is removed. In Figure 5, and Table 2, we observe that Off-PIAM-MR obtains the better asymptotic performance than On-PIAM-MR and CO-IAM-MR. In addition, Off-PIAM also realizes the great performance during the training. Figure 4 shows that Off-PIAM surpasses other algorithms in \(10\times 10\) and \(16\times 16\) maps, and it achieves similar asymptotic performance to CO-IAM in other maps. It demonstrates that the masked policy of Off-PIAM can work well even if it focuses on optimizing the original policy. Moreover, the second row of Figure 4 shows the Kullback-Leibler (KL) divergence between the target policy and the current policy. The target policy of Off-PIAM is \(\pi_{\theta}\), and the behavior policy is \(\pi_{\theta}^{\phi}\). We can observe that the KL divergence of Off-PIAM is large at the beginning and gradually decreases during the training. That is, \(\pi_{\theta}\) and \(