SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{r l} & {\mathbb {E} _ {\theta} ^ {e} \sim_ {\tilde {\pi} _ {\theta}} [ a ^ {e} ] \geq c, \quad \mathbb {E} _ {s \sim \rho_ {\tilde {\pi} _ {\theta_ {o l d}}}} [ \mathbb {K L} (\tilde {\pi} _ {\theta_ {o l d}} \| \tilde {\pi} _ {\theta}) ] \leq \delta .} \end{array} \tag {6}
$$
Below we further discuss our techniques for solving Eqn. (6). In particular, we first follow the PPO algorithm [44] and transform the Eqn. (6) into the following objective function for  $\theta$  (See Supplement S3 for more details)
$$
\max _ {\theta} \mathbb {E} \left[ \operatorname {s g n} (1 - \lambda) \min  \left(\frac {\tilde {\pi} _ {\theta} \left(a _ {t} ^ {e} \mid s _ {t}\right)}{\tilde {\pi} _ {\theta_ {\mathrm {o l d}}} \left(a _ {t} ^ {e} \mid s _ {t}\right)} A _ {\tilde {\pi} _ {\theta_ {o l d}}}, \operatorname {c l i p} \left(\frac {\tilde {\pi} _ {\theta} \left(a _ {t} ^ {e} \mid s _ {t}\right)}{\tilde {\pi} _ {\theta_ {\mathrm {o l d}}} \left(a _ {t} ^ {e} \mid s _ {t}\right)}, 1 - \epsilon , 1 + \epsilon\right) A _ {\tilde {\pi} _ {\theta_ {o l d}}}\right) + w L _ {t} ^ {\text {M A S K}} \right]. \tag {7}
$$
Here,  $A_{\tilde{\pi}_{\theta_{old}}} = A_{\tilde{\pi}_{\theta_{old}}}(s_t,a_t^e)$  and  $L_{t}^{\mathrm{MASK}} = \mathbb{E}_{a_{t}^{e}\sim \tilde{\pi}_{\theta}}[a_{t}^{e}]$ . sgn is the sign function which returns 1 or -1 based on whether the input is positive or negative.  $w$  is a hyperparameter controlling the weight of the corresponding term during optimization. The outer expectation is taken over a finite batch of state and action, sampled by running the current policy  $\tilde{\pi}_{\theta_{old}}$ .
Similarly, we can also transform Eqn. (6) into the following objective function with respect to  $\lambda$
$$
\min  _ {\lambda \geq 0} \lambda (2 \eta (\bar {\pi}) - \eta (\tilde {\pi} _ {\theta_ {o l d}}) - \delta - L _ {\tilde {\pi} _ {\theta_ {o l d}}} (\tilde {\pi} _ {\theta})). \tag {8}
$$
We follow the primal-dual method [46, 47] and iteratively update  $\theta$  and  $\lambda$  by solving Eqn. (7) and Eqn. (8) using the gradient-based optimization method. Algorithm 1 briefly presents our learning process. Supplement S4 shows a more detailed algorithm including how to estimate  $\eta(\bar{\pi})$  and  $A_{\tilde{\pi}_{\theta_{old}}}$ .
As mentioned above, as an explanation method, we fix the policy of the target agent in the environment and only train our state mask. For a multi-agent environment, we can learn an individual state mask for each target agent. This ensures the generalizability of our method to multi-agent setups. As specified in Supplement S4, with minor implementation modifications, we can also generalize our method from normal-form games, where the agents observe the same state and take action simultaneously, to extensive-form agents, where the agents make decisions in sequential order.
In the following, we start with our experiment setup and design, followed by experiment results and analysis. To ensure the reproducibility of our results, we specify the evaluation details in Supplement S5.
Environment Selection. We select 10 representative environments to demonstrate the effectiveness of StateMask across four types of environments: simple normal-form game (CartPole, Pendulum, and Pong [48]), sophisticated normal-form game (You-Shall-Not-Pass [49], Kick-And-Defend [49], and StarCraft II [50]), perfect-information (simple) extensive-form game (Connect 4, Tic-Tac-Toe and Breakthrough [51]), and imperfect-information (sophisticated) extensive-form game (DouDizhu [52]).
Baseline Selection. Sharing the same explanation goal, as discussed in Section 2, two existing methods explore the state-reward relationship – the value-function-based method and EDGE [18]. For the value-function-based method, we implement two methods - Value-max method [19, 20] and LazyMDP [36]. The value-max method associates the state importance with its value function  $V(s)$ . LazyMDP pinpoints critical steps according to the lazy gap, which is defined as  $\max_{a} Q(s, a) - \mathbb{E}_a[Q(s, a)]$ . Here,  $Q(s, a)$  is the action-value function. For EDGE, we implement it based on their code [18]. We take an absolute value for the important scores drawn by these methods to align their meaning with our methods.
4.2 Experiment Design
Experiment I: Expected Reward Preservation. Recall the state mask blinds the agent and forces it to take random actions at some least important time steps. If the blinding is accurate, the agent's performance should not be affected, and the agent's final reward should not vary significantly. As a result, one of our evaluations is to examine whether the masked agent could preserve its expected total reward. To do so, for a target agent from a game, we compute its discounted total reward across 500 rounds. Then, we treat the averaged computation result as the estimation of the agent's expected total reward  $\eta(\bar{\pi})$ . Following the computation of the expected reward, we further train our StateMask in the corresponding game.
As mentioned above, the combination of the state mask and the target agent's policy is the perturbed policy. Therefore, after having the state mask in hand, we further compute the discounted total reward of the perturbed policy across 500 rounds. We treat the averaged computation result as the approximation of  $\eta (\tilde{\pi}_{\theta})$ , and then use it to compute the relative value variation  $\frac{|\eta(\tilde{\pi}) - \eta(\tilde{\pi}_{\theta})|}{|\eta(\tilde{\pi})|}$ . As is shown from this equation, the relative value variation represents the change of the reward. If this measure is low, it indicates that the state mask has a minimal impact upon the target agent's performance. The state mask accurately pinpoints the time step least important to the final reward.
Experiment II: Explanation Fidelity. Apart from evaluating our explanation method based on the measure of expected reward preservation, we also employ a fidelity test proposed in [18] to assess its effectiveness. The fidelity test involves using the corresponding explanation method to identify and rank the most crucial time steps in a given trajectory. Subsequently, a continuous sequence of time steps is selected based on the rank of the time steps. The fidelity test evaluation algorithm proposed in [18] ensures that the selected sequence includes the maximum number of critical steps, indicating the most crucial continuous time steps leading to the final reward of the agent. Additionally, the length of the sequence can be specified as an input to the fidelity test evaluation algorithm. For instance, one can limit the sequence to only the top  $K$  time steps. For more information about the fidelity test, readers can refer to Supplement S5.
Following the rest of the fidelity test introduced in [18], for a given trajectory and the most critical sequence in it, we let the agent fast-forward to the beginning of the critical sequence. Then, we force the target agent to take random actions until the end of the critical sequence. Later onwards, we use the target agent's policy to complete the game. By doing so, we simulate a situation where we force the agent not to follow its policy at the most critical time steps. If the time steps are truly critical to the final reward, we expect the action replacement at critical time steps could introduce the highest reward variation compared with replacing any other continuous sequence containing  $K$  time steps. In [18], Guo et al. introduce a fidelity score to measure the change of the reward after the replacement of the actions at a continuous sequence of time steps. The higher the measure is, the higher the explanation fidelity is.
In this work, we use StateMask (the state masks trained above) and baseline methods to explain 500 trajectories of each target agent and obtain the time step importance for each trajectory. Then, for each trajectory, we select their top  $K$  important steps based on the importance ranking given by each
Figure 3: Fidelity scores of explanations generated by selected methods in 5 games. The x-axis represents the choice of  $K$ . "VALUE" stands for Value-max method, and "LAZY" stands for LazyMDP. The black line refers to the standard deviation of 3 runs. The bar refers to the mean and the line on the bar represents the standard deviation.
Figure 4: Visualization of the identified critical time steps in Pong game. The DRL agent controls the yellow paddle and a non-RL rule-based opponent controls the white paddle.
method and compute the fidelity score. We compute the average score across 500 trajectories as the fidelity score for that explanation method. We set  $K = 10\%$ ,  $20\%$ ,  $30\%$ ,  $40\%$  and report the fidelity of the selected methods under each setup. We repeat each experiment 3 times with different random seeds and report the mean and standard deviation.
4.3 Experiment Result
Due to the space limit, we show the results of five games: Pong, You-Shall-Not-Pass, Connect 4, DouDizhu, and StarCraft II. Supplement S5 shows the other games' results, which are aligned with those discussed in this section.
Relative Value Errors. The maximum/minimum/mean relative value errors across all 10 games are  $14.98\% / 0.00\% / 5.17\%$  (The exact values are in Supplement S5). These results verify the effectiveness of our design objective function (Eqn (6)) in preserving the target agent's performance. In addition, we also show the trends and variations in  $|\eta(\bar{\pi}) - \eta(\tilde{\pi}_{\theta})|$  during the training process for each game. The figures in Supplement S5 demonstrate that our design could indeed achieve the monotonical property, and further improve the stability of the training process.
Fidelity Scores. Figure 3 shows the fidelity scores of StateMask against the three baseline approaches in five games while the results of the remaining five games could be found in Supplement S5. First, we can observe that our method has the highest fidelity scores across all games in all settings, indicating that StateMask provides more faithful explanations for the target agents than the baseline approaches. Second, we discover that in two complex normal-form and extensive-form games (DouDizhu and StarCraft II), StateMask has a significant advantage over EDGE. This result validates our argument in Section 2 that EDGE's approximation fails to handle long trajectories. Besides, we also discover that StateMask outperforms value-based methods in those two games. We believe that the reason is that the value function is trained to facilitate policy learning and typically tends to output similar values for a sequence of continuous states [36], causing the time step importance to be less distinguishable. While StateMask specifically quantifies the importance of each time step via action randomization, giving more specific and distinguishable important scores.
In addition to the experiments above, we also demonstrate our method's efficiency and its insensitivity against the variations in the key hyper-parameters. We also verify the effectiveness of our method on not well-trained policies with sub-optimal performance. Due to the space limit, we detail these experiments in the supplementary materials. We further conduct an additional comparison to verify the necessity of minimizing the difference in the agent's expected total reward before and after applying the state mask. We consider an alternative design that directly maximizes the agent's expected total reward after applying the state mask (i.e.,  $J(\theta) = \max_{\theta} \eta(\tilde{\pi}_{\theta})$ ) and compares it with our method. Our results show that StateMask produces higher fidelity explanations on not well-trained policies than this alternative method (See Supplement S10 for more details).
Figure 5: Visualization of the identified critical steps in the Connect 4 game. The black player is our explanation target player. The states where the blue player takes actions are marked with a gray bar.
Table 1: The target agent's performance in different applications. Numbers before the brackets are means and those in the brackets are standard deviations.
Applications	Games	EDGE	Value-max	LazyMDP	Ours
Target agent's winning rate change before/after adversarial attacks	Pong	-63.04 (3.44)	-0.40 (0.40)	-36.17 (1.15)	-84.90 (1.40)
You-Shall-Not-Pass	-2.64 (0.60)	-14.80 (3.20)	-28.87 (1.21)	-35.93 (1.81)
Connect 4	-33.33 (1.56)	-26.30 (1.44)	-22.57 (1.40)	-35.37 (3.00)
DouDizhu	-0.25 (1.71)	-18.60 (0.31)	-25.20 (0.76)	-29.94 (0.64)
StarCraft II	-70.00 (14.18)	-77.33 (5.69)	-72.00 (13.53)	-83.00 (7.21)
Target agent's winning rate change before/after patching	Pong	+1.13 (0.12)	+0.53 (0.12)	+1.47 (0.12)	+1.87 (0.12)
You-Shall-Not-Pass	+0.07 (0.12)	+1.07 (0.12)	+0.12 (0.12)	+2.42 (0.18)
Connect 4	+2.00 (0.00)	+1.47 (0.12)	+1.33 (0.12)	+2.40 (0.20)
DouDizhu	+0.00 (0.00)	+0.13 (0.12)	+0.20 (0.00)	+0.47 (0.12)
5 Utility of Explanation
In this section, we demonstrate that with a higher explanation fidelity, StateMask could also achieve better utilities in three aspects: understanding agent behaviors, launching adversarial attacks, and patching agent errors.
Understanding Agent Behaviors. We visualize the identified critical steps in all 10 games to demonstrate how StateMask helps humans understand a DRL agent's behavior. Due to the space limit, we show the results of two games: Pong (normal-form game), Connect 4 (perfect-information extensive-form game), and leave the rest in Supplement S6.
Regarding the Pong game as Figure 4 shows, our explanation method highlights the frames when the ball is approaching the yellow paddle as the most important time steps whereas the states when the ball flies to the white paddle as the least important time steps. It aligns with humans' intuition since the steps when the ball is flying towards the yellow paddle are apparently critical to the final rewards. Without proper actions at these critical time steps, the DRL agent will miss the ball and lose the game.
In Connect 4, two players take turns dropping colored stones onto a grid to connect four stones in a line. Figure 5 visualizes one game where the black player wins. StateMask pinpoints the first time step (where the target agent makes the first move) as an important step. Allis et al. [53] shows that placing a stone in the center column as the first action typically brings an advantage. StateMask also highlights the last two time steps, which drive the target agent to win the game.<sup>4</sup>
Additionally, we conduct a user study  $^{5}$  to compare our explanation model with baseline methods (Value-max, EDGE, and LazyMDP) and see which one could help humans gain a better understanding of a DRL agent's behavior. Our study is determined to be exempt from an IRB review by the IRB panel. In our user-study survey, the participants are first given some background information about this project and asked for their consent to proceed. The hypothesis to be tested is that our method can offer a better explanation of the target agent's behavior in comparison with baseline methods. We use the convenience sampling method [54] to recruit 41 participants for our study. The participants come from a variety of backgrounds in explainable reinforcement learning (XRL), with  $20\%$  having never heard of XRL,  $51\%$  having some knowledge of it, and  $29\%$  being an expert or having published papers in this field. Next, the participants are presented with two groups of videos of the Kick-And-Defend game. The videos record the agent's play and highlight the critical time steps based on different explanation methods. Additionally, these two groups include trajectories played by near-optimal and sub-optimal agents. The participants are instructed to watch these two groups of videos and choose which explanation best matched their intuition.
4In the second-to-last step, the foundation for the game's victory is established. No matter which column the blue player places their piece, the black player has a winning strategy.
5The link to our survey: https://tinyurl.com/ytwj7wdt.
According to the survey results in Figure 6,  $71\%$  of the participants prefer our explanation method when the agent is near-optimal and  $59\%$  of them favor our explanation method when the agent is sub-optimal. Since the majority of participants choose our method, our hypothesis is confirmed. Our method outperforms the baseline methods in helping the user understand a DRL agent's policy.
Launching Adversarial Attacks. We follow the method developed in EDGE [18] and launch attacks against the target agent based on explanations drawn by the selected
a) Optimal
Figure 6: Results of our user study.
b) Sub-optimal
methods. Specifically, we fix the percent of time steps under attack  $T$  as  $10\%$  and launch this attack against 500 episodes. We then compute the average reward that the agent gathers before and after the attack. Table 1 shows the results in five games (same as the games in Section 4.3). Supplement S7 further shows additional results when varying the attack ratio from  $10\%$  to  $30\%$  in all ten games. As shown in the figure, the adversarial attack under the guidance of all explanation methods could dramatically drop the victim agent's performance, indicating the effectiveness of the interpretation and the exploitability of the attack. However, the performance losses observed on the adversarial attacks under the guidance of the three baseline methods are significantly lower than what we observed on the attack enabled by our method. This again implies that the attack guided by our method is more effective, and thus our interpretation has higher fidelity than the three baseline methods.
Patching Agent Errors. Rather than directly applying the rule-based patching method in EDGE [18], we propose a novel learning-based method (Supplement S8 shows the advantages of our learning-based method over the rule-based one proposed in EDGE). Given a set of trajectories, we first leverage selected explanation methods to identify a sequence of continuous time steps that are most critical to the final reward in each trajectory. Then, we replay these trajectories to the starting point of the critical steps and fine-tune the agent's policy till the game end. We validate our patching method on seven games (except CartPole, Breakthrough, and StarCraft II whose agent has a nearly optimal performance) and report the results in Table 1 and Supplement S8. As shown in the table, our method-guided agent achieves the best performance after retraining.
In this work, we propose StateMask, a novel explanation method for DRL agents. The use of StateMask as an explainer can reveal the state-reward relationship with exceptional accuracy. Compared to other types of explainers, StateMask offers the highest level of fidelity in explanation. By providing such precise explanations, we can assist humans in comprehending agent behaviors, launching adversarial attacks, and even identifying and correcting policy errors.
Our work suggests several promising directions for future research. First, in safety-critical environments, masking the agent's state and forcing it to take random actions may not be feasible due to high exploration costs. However, to train DRL agents in such environments, the model developer typically needs to build a simulator for the environment, which can be used to generate explanations. As part of future work, we will explore building simulators for such an environment if unavailable. Second, although StateMask achieves high fidelity, it may not identify cases where positively and negatively important states in the same trajectory offset each other's influence on the final reward. In the future, we will investigate additional steps to handle such cases. Third, while our explanation methods can identify important states, presenting the states to humans may not be ideal. Other future efforts will also include converting critical states to human-understandable strategies. Finally, our study in Section 5 shows the feasibility of explanation-guided policy fine-tuning. We plan to improve the fine-tuning efficiency and the generalizability of the retrained DRL agents in the future.
This project was supported in part by NSF Grant 2225234, NSF Grant 2225225, NSF Grant 2238680, and Amazon Research Award.
[37] Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proc. of ICCV, 2017.
[38] Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Proc. of NeurIPS, 2017.
[39] Andrew Elliott, Stephen Law, and Chris Russell. Explaining classifiers using adversarial perturbations on the perceptual ball. In Proc. of CVPR, 2021.
[40] Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In Proc. of BMVC, 2018.
[41] Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In Proc. of ICLR, 2019.
[42] Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. Proc. of ICML, 2019.
[43] Zuotian Li, Xiang Liu, Zelei Cheng, Yingjie Chen, Wanzhu Tu, and Jing Su. Trialview: An ai-powered visual analytics system for temporal event data in clinical trials. In Proc. of HICSS, 2024.
[44] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[45] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proc. of ICML, 2015.
[46] Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074, 2018.
[47] Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained reinforcement learning has zero duality gap. In Proc. of NeurIPS, 2019.
[48] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[49] Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. In Proc. of ICLR, 2018.
[50] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Kuttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782, 2017.
[51] Marc Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, et al. Openspiel: A framework for reinforcement learning in games. arXiv preprint arXiv:1908.09453, 2019.
[52] Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Ji Liu. Douzero: Mastering doudizhu with self-play deep reinforcement learning. In Proc. of ICML, 2021.
[53] Louis Victor Allis. A knowledge-based approach of connect-four. Journal of the International Computer Games Association, 1988.
[54] Mark Saunders, Philip Lewis, and Adrian Thornhill. Research methods for business students. Pearson education, 2009.
S1 Proof of Theorem 1
Before we proving Theorem 1, we first introduce the following lemma.
Lemma 2. Given an old and new policy of the state mask  $\tilde{\pi}_{\theta_{old}}$  and  $\tilde{\pi}_{\theta}$ , respectively, if  $\eta(\tilde{\pi}_{\theta_{old}}) \leq \eta(\tilde{\pi}_{\theta}) \leq 2\eta(\bar{\pi}) - \eta(\tilde{\pi}_{\theta_{old}})$ , we have  $|\eta(\tilde{\pi}_{\theta}) - \eta(\bar{\pi})| \leq |\eta(\tilde{\pi}_{\theta_{old}}) - \eta(\bar{\pi})|$ .
Proof of Lemma 2. The monotonicity property requires  $|\eta(\tilde{\pi}_{\theta}) - \eta(\bar{\pi})| \leq |\eta(\tilde{\pi}_{\theta_{old}}) - \eta(\bar{\pi})|$ . The following equivalence holds: