SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} | \eta (\tilde {\pi} _ {\theta}) - \eta (\bar {\pi}) | \leq | \eta (\tilde {\pi} _ {\theta_ {o l d}}) - \eta (\bar {\pi}) | \\ \Longleftrightarrow \eta (\tilde {\pi} _ {\theta}) ^ {2} - 2 \eta (\tilde {\pi} _ {\theta}) \eta (\bar {\pi}) + \eta (\bar {\pi}) ^ {2} \leq \eta (\tilde {\pi} _ {\theta_ {o l d}}) ^ {2} - 2 \eta (\tilde {\pi} _ {\theta_ {o l d}}) \eta (\bar {\pi}) + \eta (\bar {\pi}) ^ {2} \tag {1} \\ \Longleftrightarrow \left(\eta (\tilde {\pi} _ {\theta}) + \eta (\tilde {\pi} _ {\theta_ {o l d}})\right) \left(\eta (\tilde {\pi} _ {\theta}) - \eta (\tilde {\pi} _ {\theta_ {o l d}})\right) \leq 2 \eta (\bar {\pi}) \left(\eta (\tilde {\pi} _ {\theta}) - \eta (\tilde {\pi} _ {\theta_ {o l d}})\right). \\ \end{array}
$$
Therefore, if  $\eta (\tilde{\pi}_{\theta_{old}})\leq \eta (\tilde{\pi}_{\theta})$  and  $\eta (\tilde{\pi}_{\theta}) + \eta (\tilde{\pi}_{\theta_{old}})\leq 2\eta (\bar{\pi})$  , or in other words,  $\eta (\tilde{\pi}_{\theta_{old}})\leq \eta (\tilde{\pi}_{\theta})\leq$ $2\eta (\bar{\pi}) - \eta (\tilde{\pi}_{\theta_{old}})$  , we have  $|\eta (\tilde{\pi}_\theta) - \eta (\bar{\pi})|\leq |\eta (\tilde{\pi}_{\theta_{old}}) - \eta (\bar{\pi})|$
Recall that Lemma 1 in Section 3.2 shows the upper bound and the lower bound of  $\eta (\tilde{\pi}_{\theta})$ . An immediate implication of Lemma 1 enables us to obtain the inequalities to satisfy Lemma 2.
From Lemma 1, we can easily derive the lower bound of  $\eta (\tilde{\pi}_{\theta})$  as  $\eta (\tilde{\pi}_{\theta})\geq L_{\tilde{\pi}_{\theta_{old}}}\left(\tilde{\pi}_{\theta}\right) - C\mathbb{KL}^{\max}(\tilde{\pi}_{\theta_{old}}(\cdot | s)\| \tilde{\pi}_{\theta}(\cdot | s))$ . If the lower bound is greater than  $\eta (\tilde{\pi}_{\theta_{old}})$ , then we can guarantee that  $\eta (\tilde{\pi}_{\theta})\geq \eta (\tilde{\pi}_{\theta_{old}})$ . Similarly, we can obtain the upper bound of  $\eta (\tilde{\pi}_{\theta})$  as  $\eta (\tilde{\pi}_{\theta})\leq L_{\tilde{\pi}_{\theta_{old}}}\left(\tilde{\pi}_{\theta}\right) + C\mathbb{KL}^{\max}(\tilde{\pi}_{\theta_{old}}(\cdot | s)\| \tilde{\pi}_{\theta}(\cdot | s))$ . If the upper bound is lower than  $2\eta (\bar{\pi}) - \eta (\tilde{\pi}_{\theta_{old}})$ , we can guarantee that  $\eta (\tilde{\pi}_{\theta})\leq 2\eta (\bar{\pi}) - \eta (\tilde{\pi}_{\theta_{old}})$ .
Putting all these inequalities together, the following inequalities are the sufficient conditions for satisfying  $|\eta(\tilde{\pi}_{\theta}) - \eta(\bar{\pi})| < |\eta(\tilde{\pi}_{old}) - \eta(\bar{\pi})|$ :
$$
L _ {\tilde {\pi} _ {\theta_ {o l d}}} (\tilde {\pi} _ {\theta}) \geq \eta (\tilde {\pi} _ {\theta_ {o l d}}) + C \mathbb {K L} ^ {\max } (\tilde {\pi} _ {\theta_ {o l d}} (\cdot | s) \| \tilde {\pi} _ {\theta} (\cdot | s)) \tag {2}
$$
$$
L _ {\tilde {\pi} _ {\theta_ {o l d}}} \big (\tilde {\pi} _ {\theta} \big) + C \mathbb {K L} ^ {\max} \big (\tilde {\pi} _ {\theta_ {o l d}} (\cdot | s) \| \tilde {\pi} _ {\theta} (\cdot | s) \big) \leq 2 \eta (\tilde {\pi}) - \eta (\tilde {\pi} _ {\theta_ {o l d}}).
$$
S2 Derivation of Eqn. (5)
First, let  $M_{\tilde{\pi}_{\theta_{old}}}\left(\tilde{\pi}_{\theta}\right) = L_{\tilde{\pi}_{\theta_{old}}}\left(\tilde{\pi}_{\theta}\right) - C\mathbb{K}\mathbb{L}^{\max}(\tilde{\pi}_{\theta_{old}}(\cdot |s)\| \tilde{\pi}_{\theta}(\cdot |s))$  . We can get  $M_{\tilde{\pi}_{\theta_{old}}}\left(\tilde{\pi}_{\theta_{old}}\right) = \eta (\tilde{\pi}_{\theta_{old}})$  By maximizing the  $M_{\tilde{\pi}_{\theta_{old}}}\bigl (\tilde{\pi}_{\theta}\bigr)$  , we can ensure  $M_{\tilde{\pi}_{\theta_{old}}}\bigl (\tilde{\pi}_{\theta}\bigr)\geq M_{\tilde{\pi}_{\theta_{old}}}\bigl (\tilde{\pi}_{\theta_{old}}\bigr) = \eta (\tilde{\pi}_{\theta_{old}})$  . Given that  $\eta (\tilde{\pi}_\theta)\geq M_{\tilde{\pi}_{\theta_{old}}}\left(\tilde{\pi}_\theta\right)$  , we could satisfy the first inequality of Eqn. (4).
By constraining the upper bound of  $\tilde{\pi}_{\theta_{old}}$  to be smaller than  $2\eta (\bar{\pi}) - \eta (\tilde{\pi}_{\theta_{old}})$ , we could guarantee the second inequality of Eqn. (4) holds.
Therefore, combining all these constraints together, we can obtain the following optimization objective:
$$
\begin{array}{l} \max  _ {\theta} L _ {\tilde {\pi} _ {\theta_ {o l d}}} \left(\tilde {\pi} _ {\theta}\right) - C \mathbb {K L} ^ {\max } \left(\tilde {\pi} _ {\theta_ {o l d}} (\cdot | s) \| \tilde {\pi} _ {\theta} (\cdot | s)\right) \tag {3} \\ \mathrm {s . t .} L _ {\tilde {\pi} _ {\theta_ {o l d}}} (\tilde {\pi} _ {\theta}) + C \mathbb {K L} ^ {\max} \left(\tilde {\pi} _ {\theta_ {o l d}} (\cdot | s) \| \tilde {\pi} _ {\theta} (\cdot | s)\right) \leq 2 \eta (\bar {\pi}) - \eta (\tilde {\pi} _ {\theta_ {o l d}}). \\ \end{array}
$$
In practice, Schulman et al. [11] pointed out that utilizing the penalty coefficient  $C$  may lead to a very small step size. Therefore, we replace the penalty on the maximal KL-divergence with a trust region constraint on an average KL-divergence (i.e.,  $\mathbb{E}_{s\sim \rho_{\tilde{\pi}_{\theta_{old}}}}[\mathbb{KL}(\tilde{\pi}_{\theta_{old}}(\cdot | s)\| \tilde{\pi}_{\theta}(\cdot | s))]\leq \delta$ ) as suggested in [11]. We can further obtain the following objective
$$
\begin{array}{l} \max _ {\theta} L _ {\tilde {\pi} _ {\theta_ {o l d}}} (\tilde {\pi} _ {\theta}) \\ \text {s . t .} L _ {\tilde {\pi} _ {\theta_ {o l d}}} \left(\tilde {\pi} _ {\theta}\right) \leq 2 \eta (\bar {\pi}) - \eta \left(\tilde {\pi} _ {\theta_ {o l d}}\right) - \delta , \tag {4} \\ \mathbb {E} _ {s \sim \rho_ {\tilde {\pi} _ {\theta_ {o l d}}}} \left[ \mathbb {K L} (\tilde {\pi} _ {\theta_ {o l d}} (\cdot | s) | | \tilde {\pi} _ {\theta} (\cdot | s)) \right] \leq \delta . \\ \end{array}
$$
Additionally, we add a sparsity constraint which will only tighten the problem without affecting the monotonicity property and obtain the following optimization objective
$$
\begin{array}{l} \max _ {\theta} L _ {\tilde {\pi} _ {\theta_ {o l d}}} (\tilde {\pi} _ {\theta}) \\ s. t. \quad ① \quad \mathbb {E} _ {a ^ {e} \sim \tilde {\pi} _ {\theta}} [ a ^ {e} ] \geq c, \tag {5} \\ ② \quad L _ {\tilde {\pi} _ {\theta_ {o l d}}} (\tilde {\pi} _ {\theta}) \leq 2 \eta (\bar {\pi}) - \eta (\tilde {\pi} _ {\theta_ {o l d}}) - \delta , \\ \text {③} \quad \mathbb {E} _ {s \sim \rho_ {\tilde {\pi} _ {\theta_ {o l d}}}} \left[ \mathbb {K L} \left(\tilde {\pi} _ {\theta_ {o l d}} (\cdot | s) \| \tilde {\pi} _ {\theta} (\cdot | s)\right) \right] \leq \delta , \\ \end{array}
$$
where  $\mathbb{E}_{a^e}\sim \tilde{\pi}_\theta [a^e ]\geq c$  constrains the lower bound sparsity for the output of  $\tilde{\pi}_{\theta}$  across all states.
By maximizing the local approximation with constraint ③, we could guarantee that  $L_{\tilde{\pi}_{\theta_{old}}}\left(\tilde{\pi}_{\theta}\right) \geq L_{\tilde{\pi}_{\theta_{old}}}\left(\tilde{\pi}_{\theta_{old}}\right) = \eta \left(\tilde{\pi}_{\theta_{old}}\right)$ . Since  $L_{\tilde{\pi}_{\theta_{old}}}\left(\tilde{\pi}_{\theta}\right)$  is the local approximation of  $\eta \left(\tilde{\pi}_{\theta}\right)$ , we could further guarantee that  $\eta \left(\tilde{\pi}_{\theta}\right) \geq \eta \left(\tilde{\pi}_{\theta_{old}}\right)$ . Moreover, the constraint  $L_{\tilde{\pi}_{\theta_{old}}}\left(\tilde{\pi}_{\theta}\right) \leq 2\eta (\bar{\pi}) - \eta \left(\tilde{\pi}_{\theta_{old}}\right) - \delta$  could ensure that the inequality  $\eta \left(\tilde{\pi}_{\theta}\right) \leq 2\eta (\bar{\pi}) - \eta \left(\tilde{\pi}_{\theta_{old}}\right)$  is satisfied. Therefore, a policy  $\tilde{\pi}_{\theta}$  solved from the optimization objective above satisfies Lemma 2 and thus enables the desired monotonicity.
S3 Derivative from Eqn. (6) to of Eqn. (7)
With respect to  $\theta$ , Eqn. (6) becomes
$$
\max  _ {\theta} (1 - \lambda) L _ {\tilde {\pi} _ {\theta_ {o l d}}} (\tilde {\pi} _ {\theta}) + C _ {1} \quad \text {s . t .} \mathbb {E} _ {s \sim \rho_ {\tilde {\pi} _ {\theta_ {o l d}}}} \left[ \mathbb {K L} \left(\tilde {\pi} _ {\theta_ {o l d}} (\cdot | s) \| \tilde {\pi} _ {\theta} (\cdot | s)\right) \right] \leq \delta , \quad \mathbb {E} _ {a ^ {e} \sim \tilde {\pi} _ {\theta}} \left[ a ^ {e} \right] \geq c, \tag {6}
$$
where  $C_1 = \lambda (2\eta (\bar{\pi}) - \eta (\tilde{\pi}_{\theta_{old}}) - \delta)$  is a constant that can be eliminated during the optimization of  $\theta$ . The KKT condition [1] implies that there exists Lagrange multiplier  $\xi$  (dual solution) such that the set of solutions of the constrained optimization problem Eqn. (6) are equivalent to those of the following relaxed objective:
$$
\max  _ {\theta} (1 - \lambda) L _ {\tilde {\pi} _ {\theta_ {o l d}}} (\tilde {\pi} _ {\theta}) + \xi \mathbb {E} _ {a ^ {e} \sim \tilde {\pi} _ {\theta}} [ a ^ {e} ] \quad \text {s . t .} \mathbb {E} _ {s \sim \rho_ {\tilde {\pi} _ {\theta_ {o l d}}}} [ \mathbb {K L} (\tilde {\pi} _ {\theta_ {o l d}} (\cdot | s) \| \tilde {\pi} _ {\theta} (\cdot | s)) ] \leq \delta . \tag {7}
$$
If  $\lambda < 1$ , the optimization problem is equivalent to
$$
\max  _ {\theta} L _ {\tilde {\pi} _ {\theta_ {o l d}}} \left(\tilde {\pi} _ {\theta}\right) + w \mathbb {E} _ {a ^ {e} \sim \tilde {\pi} _ {\theta}} \left[ a ^ {e} \right] \quad \text {s . t .} \mathbb {E} _ {s \sim \rho_ {\tilde {\pi} _ {\theta_ {o l d}}}} \left[ \mathbb {K L} \left(\tilde {\pi} _ {\theta_ {o l d}} (\cdot | s) \| \tilde {\pi} _ {\theta} (\cdot | s)\right) \right] \leq \delta , \tag {8}
$$
where  $w = \xi /(1 - \lambda)$ .
The optimization problem can further be transformed as
$$
\begin{array}{l} \max  _ {\theta} \sum_ {s} \rho_ {\tilde {\pi} _ {\theta_ {o l d}}} (s) \sum_ {a ^ {e}} \tilde {\pi} _ {\theta} \left(a ^ {e} \mid s\right) A _ {\tilde {\pi} _ {\theta_ {o l d}}} \left(s, a ^ {e}\right) + w \mathbb {E} _ {a ^ {e} \sim \tilde {\pi} _ {\theta}} \left[ a ^ {e} \right] \tag {9} \\ \begin{array}{r l} \text {s . t .} & \mathbb {E} _ {s \sim \rho_ {\tilde {\pi} _ {\theta_ {o l d}}}} \left[ \mathbb {K L} \left(\tilde {\pi} _ {\theta_ {o l d}} (\cdot | s) \| \tilde {\pi} _ {\theta} (\cdot | s)\right) \right] \leq \delta . \end{array} \\ \end{array}
$$
Due to the summation of the new policy  $\tilde{\pi}_{\theta}$ , optimization is still challenging to implement even after transformation. To resolve this issue, we follow the idea from [10]. Specifically, we use importance sampling and substitute it with the summation of the old policy through the Monte Carlo method [14]. Then, in order to reach our ultimate optimization function, we further substitute the trust region constraints with the clipped ratio operation proposed by the Proximal Policy Optimization algorithm (PPO) [11]. Therefore, if  $\lambda < 1$ , we could obtain the final objective to optimize  $\theta$  as
$$
\max  _ {\theta} \mathbb {E} \left[ \min  \left(\frac {\tilde {\pi} _ {\theta} \left(a _ {t} ^ {e} \mid s _ {t}\right)}{\tilde {\pi} _ {\theta_ {\mathrm {o l d}}} \left(a _ {t} ^ {e} \mid s _ {t}\right)} A _ {\tilde {\pi} _ {\theta_ {o l d}}}, \operatorname {c l i p} \left(\frac {\tilde {\pi} _ {\theta} \left(a _ {t} ^ {e} \mid s _ {t}\right)}{\tilde {\pi} _ {\theta_ {\mathrm {o l d}}} \left(a _ {t} ^ {e} \mid s _ {t}\right)}, 1 - \epsilon , 1 + \epsilon\right) A _ {\tilde {\pi} _ {\theta_ {o l d}}}\right) + w L _ {t} ^ {\text {M A S K}} \right], \tag {10}
$$
where  $L_{t}^{\mathrm{MASK}} = \mathbb{E}_{a_{t}^{e}\sim \tilde{\pi}_{\theta}}[a_{t}^{e}]$  is added to ensure the sparsity of critical steps.
If  $\lambda > 1$ , the optimization problem is equivalent to
$$
\min _ {\theta} L _ {\tilde {\pi} _ {\theta_ {o l d}}} \left(\tilde {\pi} _ {\theta}\right) - w \mathbb {E} _ {a ^ {e} \sim \tilde {\pi} _ {\theta}} \left[ a ^ {e} \right] \quad \text {s . t .} \quad \mathbb {E} _ {s \sim \rho_ {\tilde {\pi} _ {\theta_ {o l d}}}} \left[ \mathbb {K L} \left(\tilde {\pi} _ {\theta_ {o l d}} (\cdot | s) \| \tilde {\pi} _ {\theta} (\cdot | s)\right) \right] \leq \delta . \tag {11}
$$
Similar to the analysis above, if  $\lambda > 1$ , the final objective for updating  $\theta$  becomes
$$
\max _ {\theta} \mathbb {E} \left[ - \min  \left(\frac {\tilde {\pi} _ {\theta} \left(a _ {t} ^ {e} \mid s _ {t}\right)}{\tilde {\pi} _ {\theta_ {\mathrm {o l d}}} \left(a _ {t} ^ {e} \mid s _ {t}\right)} A _ {\tilde {\pi} _ {\theta_ {o l d}}}, \operatorname {c l i p} \left(\frac {\tilde {\pi} _ {\theta} \left(a _ {t} ^ {e} \mid s _ {t}\right)}{\tilde {\pi} _ {\theta_ {\mathrm {o l d}}} \left(a _ {t} ^ {e} \mid s _ {t}\right)}, 1 - \epsilon , 1 + \epsilon\right) A _ {\tilde {\pi} _ {\theta_ {o l d}}}\right) + w L _ {t} ^ {\text {M A S K}} \right], \tag {12}
$$
Putting all these objectives together, we get the final objective for updating  $\theta$  as
$$
\max  _ {\theta} \mathbb {E} \left[ \operatorname {s g n} (1 - \lambda) \min  \left(\frac {\tilde {\pi} _ {\theta} \left(a _ {t} ^ {e} \mid s _ {t}\right)}{\tilde {\pi} _ {\theta_ {\mathrm {o l d}}} \left(a _ {t} ^ {e} \mid s _ {t}\right)} A _ {\tilde {\pi} _ {\theta_ {o l d}}}, \operatorname {c l i p} \left(\frac {\tilde {\pi} _ {\theta} \left(a _ {t} ^ {e} \mid s _ {t}\right)}{\tilde {\pi} _ {\theta_ {\mathrm {o l d}}} \left(a _ {t} ^ {e} \mid s _ {t}\right)}, 1 - \epsilon , 1 + \epsilon\right) A _ {\tilde {\pi} _ {\theta_ {o l d}}}\right) + w L _ {t} ^ {\mathrm {M A S K}} \right]. \tag {13}
$$
S4 Our Training Algorithm
S4.1 Algorithm Detail
Optimizing  $\theta$ . Recall that the final objective function of  $\theta$  is
$$
\max _ {\theta} \mathbb {E} \left[ \operatorname {s g n} (1 - \lambda) \min  \left(\frac {\tilde {\pi} _ {\theta} \left(a _ {t} ^ {e} \mid s _ {t}\right)}{\tilde {\pi} _ {\theta_ {\mathrm {o l d}}} \left(a _ {t} ^ {e} \mid s _ {t}\right)} A _ {\tilde {\pi} _ {\theta_ {o l d}}}, \operatorname {c l i p} \left(\frac {\tilde {\pi} _ {\theta} \left(a _ {t} ^ {e} \mid s _ {t}\right)}{\tilde {\pi} _ {\theta_ {\mathrm {o l d}}} \left(a _ {t} ^ {e} \mid s _ {t}\right)}, 1 - \epsilon , 1 + \epsilon\right) A _ {\tilde {\pi} _ {\theta_ {o l d}}}\right) + w L _ {t} ^ {\mathrm {M A S K}} \right], \tag {14}
$$
where  $A_{\tilde{\pi}_{\theta_{old}}} = A_{\tilde{\pi}_{\theta_{old}}}(s_t,a_t^e)$  and  $L_{t}^{\mathrm{MASK}}(\theta) = \mathbb{E}_{a_{t}^{e}}\sim \tilde{\pi}_{\theta}[a_{t}^{e}]$ .
Note that the output of StateMask  $a_{t}^{e}$  is sampled from a categorical distribution over all actions (i.e., 0 or 1). However, as stated in [4], the categorical variable  $a_{t}^{e}$  is not capable of backpropagating through samples. Note that  $\mathbb{E}_{a_t^e\sim \tilde{\pi}_\theta}[a_t^e ] = \mathbb{E}_{s_t\sim \rho_{\tilde{\pi}_\theta}}\left[\mathbb{E}_{a_t^e\sim \tilde{\pi}_\theta(\cdot |s)}[a_t^e ]\right] = \mathbb{E}_{s_t\sim \rho_{\tilde{\pi}_\theta}}\left[Pr(a_t^e = 1)\right]$  and  $Pr(a_{t}^{e} = 1)$  is differentiable. Therefore, we replace  $\mathbb{E}_{a_t^e\sim \tilde{\pi}_\theta}[a_t^e ]$  with  $\mathbb{E}_{s_t\sim \rho_{\tilde{\pi}_\theta}}\left[Pr(a_t^e = 1)\right]$  in practice.
When estimating  $A_{\tilde{\pi}_{\theta_{old}}}$ , we adopt the same estimation formula as PPO[11]
$$
\hat {A} _ {\pi_ {\theta_ {o l d}}} = \delta_ {t} + (\gamma \zeta) \delta_ {t + 1} + \dots + \dots + (\gamma \zeta) ^ {T - t + 1} \delta_ {T - 1} \tag {15}
$$
$$
\text {w h e r e} \quad \delta_ {t} = r _ {t} + \gamma V (s _ {t + 1}) - V (s _ {t})
$$
Therefore, during the training process, in addition to updating the actor by solving  $\theta$  in Eqn. (14), we also have to update the state-value function  $V(s)$  by the Temporal-Difference (TD) method [13] for better estimating the advantage.
Optimizing  $\lambda$ . And recall that the objective function of  $\lambda$  is
$$
\min  _ {\lambda \geq 0} \lambda \left(2 \eta (\bar {\pi}) - \eta \left(\tilde {\pi} _ {\theta_ {o l d}}\right) - \delta - L _ {\tilde {\pi} _ {\theta_ {o l d}}} \left(\tilde {\pi} _ {\theta}\right)\right). \tag {16}
$$
where  $L_{\tilde{\pi}_{\theta_{old}}}\left(\tilde{\pi}_{\theta}\right) = \sum_{s}\rho_{\tilde{\pi}_{\theta_{old}}}(s)\sum_{a^{e}}\tilde{\pi}_{\theta}(a^{e} \mid s)A_{\tilde{\pi}_{\theta_{old}}}(s,a^{e})$
In practice,  $\eta(\tilde{\pi}_{\theta_{old}})$  and  $\eta(\bar{\pi})$  can be approximated via Monte Carlo method [7].  $\eta(\bar{\pi})$  can be estimated in advance, i.e., running the game with the agent's policy  $\bar{\pi}$  500 times and calculating the average expected discounted reward as follows
$$
\eta (\bar {\pi}) = \mathbb {E} _ {s _ {0}, a _ {0}, \dots \sim \bar {\pi}} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r \left(s _ {t}\right) \right], \tag {17}
$$
$\eta (\pi_{\theta_{old}})$  can be estimated using a similar formula in parallel with the collecting trajectories of old policy  $\pi_{\theta_{old}}$  while solving  $\theta$  with our training algorithm.
Nevertheless, the optimization objective remains challenging to solve because of the summation in  $L_{\tilde{\pi}_{\theta_{old}}}\left(\tilde{\pi}_{\theta}\right)$  involving the new policy  $\tilde{\pi}_{\theta}$ . To address this issue, we utilize importance sampling and substitute the summation over the new policy  $\tilde{\pi}_{\theta}$  with a summation over the old policy  $\tilde{\pi}_{\theta_{old}}$ , which can be computed using the Monte Carlo method [7]. Mathematically,  $L_{\tilde{\pi}_{\theta_{old}}}\left(\tilde{\pi}_{\theta}\right)$  can be estimated by  $\frac{\tilde{\pi}_{\theta}(a^{e}|s)}{\tilde{\pi}_{\theta_{old}}(a^{e}|s)} A_{\tilde{\pi}_{\theta_{old}}} (s,a^{e})$ , which has already been computed in Eqn. (14) when updating  $\theta$ .
Algorithm 1 The training algorithm of mask net.
Input: Target agent's policy  $\bar{\pi}$    
Output: Mask net policy  $\tilde{\pi}$    
Initialization: Initialize the weights  $\theta$  for the mask net  $\tilde{\pi}$  , Lagrange multiplier  $\lambda = 0$  Estimate the  $\eta (\bar{\pi})$  in advance by Eqn. (17)   
for iteration  $= 1,2,\ldots$  do   
 $s\gets s_0$ $\mathcal{D}\gets \emptyset$    
for  $t = 1,2,\dots,T$  do   
 $a_{t}^{e}\gets$  sample from  $\tilde{\pi}_{\theta_{old}}(\cdot |s)$ $a_{t}\gets$  sample from  $\tilde{\pi} (\cdot |s)$    
Compute actual taken action  $a\gets a_t\odot a_t^e$    
Obtain current step reward  $r_t$  , next state  $s^\prime$  from the environment given action  $a$ $\mathcal{D}\gets \mathcal{D}\cup (s,a_t^e,r_t,s')$ $s\gets s'$    
end for   
Compute estimates for  $A_{\tilde{\pi}_{\theta_{old}}}(s_t,a_t^e)$  using samples in  $\mathcal{D}$  by Eqn. (15)   
Solve  $\theta$  using the trajectory samples in  $\mathcal{D}$  by Eqn. (14) and update  $\theta_{old}\gets \theta$  Estimate the  $\eta (\tilde{\pi})$  using the trajectory samples in  $\mathcal{D}$  by Eqn. (17) Solve  $\lambda$  by Eqn. (16) and update  $\lambda_{old}\gets \lambda$    
end for
Extensive-form Games. Regarding extensive-form games, we train a state mask for the target agent while keeping all the agents' policies fixed. In this way, we could transform the extensive-form games into single-player normal-form games (i.e., we treat all of the agents' actions as part of the environment). However, the game may not end with the target agent, and the target agent will not receive the game reward in this case. To address this problem, we automatically overwrite the target agent's final reward with the final game result.
Based on what we have discussed above, we present the full algorithm in Algorithm 1.
S4.2 Implementation and Hyperparameters
We implement our training algorithm using PyTorch [9] and additionally discuss the hyperparameter setting of our training algorithm. The common hyperparameter setting is as follows. Regarding the clipping parameter  $\epsilon$  in Eqn. (14), we set it as 0.2. As for the advantage estimation in Eqn. (15), we set  $\gamma = 0.99$  and  $\zeta = 0.95$ . Moreover, we provide the game-specific hyperparameter choices in Table 1. Regarding the policy network of StateMask, we utilize a widely used network structure, commonly employed to train Deep Reinforcement Learning (DRL) agents for each game. Specifically, we use Convolutional Neural Network (CNN) for the Pong game and Residual Network (ResNet) for the Connect 4 game since these two games' input observation contains spatial information and are relatively complicated. For other games, we use either Long Short-Term Memory (LSTM) or Multi-Layer Perceptron (MLP) network, which depends on the input type of the observation and the game's complexity. We use two separate networks with the same backbone architecture for the policy and value networks. The output dimension of the policy network and value network are 2 and 1, respectively. As for the learning rate setting, we refer to other work of training a reinforcement learning agent in these games [6, 2]. Moreover, our method introduces an extra hyperparameter  $-w$  (To encourage the mask network to blind the target agent at some states, we add a sparsity constraint on the number of mask actions). We search the  $w$  in  $\{0, 1e-5, 1e-4, 1e-3, 1e-2\}$  and report the sensitivity experiment results in Supplement S5.3.
Table 1: Training hyperparameter settings of our method. The numbers in the bracket after "CNN" represent the number of kernels in each layer. The numbers in the bracket after "MLP" are the hidden dimensions.  $512 \times 10$  represents 10-layer MLP with hidden size 512.
Games	w	Learning rate	Optimizer	StateMask backbone	Batch Size	Num. of Training Steps
Pong	1e-4	1e-5	Adam	CNN(16, 32)	32	2e+4
You-Shall-Not-Pass	0	3e-4	Adam	MLP(256, 256)	2048	2e+7
Kick-And-Defend	0	3e-4	Adam	LSTM(512, 256)	2048	2e+7
CartPole	0	1e-3	Adam	MLP(256, 256)	5	8e+2
Pendulum	0	1e-3	Adam	MLP(256, 256)	5	8e+2
StarCraft II	0	1e-6	Adam	MLP(128,128,128)	32	8e+7
Connect 4	1e-3	1e-3	Adam	ResNet-128	1024	4e+5
Tic-Tac-Toe	1e-5	1e-2	Adam	MLP(128, 128)	128	2e+4
Breakthrough	1e-3	1e-3	Adam	MLP(512 × 10)	1024	2e+6
DouDizhu	1e-3	3e-4	Adam	MLP(512 × 10)	42	8e+5
S5 Details of Evaluation
S5.1 Other Implementation Details
Baseline Implementations. Regarding baseline approaches, we use the code released by the authors or implement our own version if the authors don't release the code. Specifically, as for EDGE, we use their released open-sourced code from https://github.com/Henrygwb/edge. For the Value-max and the LazyMDP method, we assume that we have access to the target agent's value function and implement them based on the description of the original method<sup>1</sup>.
Fidelity Tests and Metrics. As mentioned in Section 4.2, we adopt the experiment design in EDGE [3] to access the accuracy of the explanation. Specifically, the fidelity test first gathers  $N$  (i.e.,  $N = 500$  in our setting) trajectories. Each explanation method generates the importance score of each state accordingly without altering the agent's action. For each trajectory, we identify and record the length  $l$  of the longest continuous sequence within the top-  $K$  most critical time steps. Following this, we replay the sequence of actions from the original trajectory up to the start of this identified sequence. From this point onwards, we replace the original actions with randomly selected actions for the next  $l$  steps. Once the action substitution is completed, we present the newly reached state to the target agent at the end of the selected time steps. From this point forward, the agent operates according to the actions suggested by its policy network until the conclusion of the game. The reward difference before and after action replacement is denoted as  $d$ . The fidelity score is defined as  $\log(p_d) - \log(p_l)$  where  $p_d = |d| / d_{max}$  is the absolute reward difference divided by the maximum possible reward difference and  $p_l = l / L$  is the length of the longest sequence normalized by the original trajectory length  $L$ . Note that in the original definition of the fidelity score [3], a lower score implies a higher level of fidelity. However, for ease of presentation, we have introduced a negative sign to the fidelity score, such that a higher score indicates superior fidelity.
S5.2 Experiment Setup
Game Introduction and Target Agents. To assist in determining whether our explanation aligns with human perception, we briefly introduce each game below.