Exploring the Use of Invalid Action Masking in Reinforcement Learning: A Comparative Study of On-Policy and Off-Policy Algorithms in Real-Time Strategy Games  
[PERSON]  
1 Air Traffic Control and Navigation School, Air Force Engineering University, Xi'an 710051, China;  
[EMAIL_ADDRESS] (Y.H. ); [EMAIL_ADDRESS] (X.L. ); [EMAIL_ADDRESS] (A.Y. ); [EMAIL_ADDRESS] (N.W.) 2 Shaanxi Key Laboratory of Meta-Synthesis for Electronic and Information System, Air Force Engineering University, Xi'an 710051, China  
3 Xi'an Research Institute of High-Technology, Xi'an 710051, China; [EMAIL_ADDRESS]  
[PERSON]  
1 Air Traffic Control and Navigation School, Air Force Engineering University, Xi'an 710051, China;  
[EMAIL_ADDRESS] (Y.H. ); [EMAIL_ADDRESS] (X.L. ); [EMAIL_ADDRESS] (A.Y. ); [EMAIL_ADDRESS] (N.W.) [PERSON]  
1 Air Traffic Control and Navigation School, Air Force Engineering University, Xi'an 710051, China;  
[EMAIL_ADDRESS] (Y.H. ); [EMAIL_ADDRESS] (X.L. ); [EMAIL_ADDRESS] (A.Y. ); [EMAIL_ADDRESS] (N.W.) [PERSON]  
3 Xi'an Research Institute of High-Technology, Xi'an 710051, China; [EMAIL_ADDRESS]  
[PERSON]  
1 Air Traffic Control and Navigation School, Air Force Engineering University, Xi'an 710051, China;  
[EMAIL_ADDRESS] (Y.H. ); [EMAIL_ADDRESS] (X.L. ); [EMAIL_ADDRESS] (A.Y. ); [EMAIL_ADDRESS] (N.W.) [PERSON]  
1 Air Traffic Control and Navigation School, Air Force Engineering University, Xi'an 710051, China;  
[EMAIL_ADDRESS] (Y.H. ); [EMAIL_ADDRESS] (X.L. ); [EMAIL_ADDRESS] (A.Y. ); [EMAIL_ADDRESS] (N.W.) [PERSON]  
1 Air Traffic Control and Navigation School, Air Force Engineering University, Xi'an 710051, China;  
[EMAIL_ADDRESS] (Y.H. ); [EMAIL_ADDRESS] (X.L. ); [EMAIL_ADDRESS] (A.Y. ); [EMAIL_ADDRESS] (N.W.) [PERSON]  
3 Xi'an Research Institute of High-Technology, Xi'an 710051, China;  
[EMAIL_ADDRESS] (Y. Another popular technique is invalid action masking, which filters invalid actions by setting the probability of all invalid actions to near zero [7; 8; 9]. This technique ensures the agent can avoid selecting impossible or unavailable actions during training. In addition, it reduces the dimension of exploration space and improves exploration efficiency. Invalid action masking has been widely applied to many domains besides RTS or MOBA games, such as traffic signal control [21], unmanned vehicle [22], adaptive voltage control [12], etc. [PERSON] et al. [23] provides a closer look at invalid action masking for the first time, categorizing it into two types, invalid action policy gradient (utilizing the masked policy for gradient calculation) and naive invalid action policy gradient (utilizing the original policy for gradient calculation). Through a numerical example, it demonstrates that invalid action masking alters the policy gradient and that the invalid action policy gradient corresponds to a valid policy gradient. In addition, it is noted that employing the naive invalid action policy gradient may lead to instability. However, empirical results in [23] show that naive invalid action masking can outperform the invalid action policy gradient when the masking is removed. This finding challenges the existing theoretical analysis in [23], which has inspired our interest in understanding the mechanism behind naive invalid action masking. Inspired by aforementioned analysis, this study provides a theoretical analysis and reveals why naive invalid action masking is an effective way to train the agent to avoid invalid actions explicitly. Specifically, the regular invalid action masking is to replace the logits of invalid actions with a large negative number [23], which we call logit-level masking. To derive the policy gradient, we present a novel masking called action-level masking. We prove that two types of masking are equivalent so that the conclusions derived by one also apply to another. We utilize the action-level masking to verify that invalid action masking indeed changes the policy gradient.