StateMask: Explaining Deep Reinforcement Learning through State Mask
Authors: Zelei Cheng
Xian Wu
Jiahao Yu
Wenhai Sun
Wenbo Guo
Xinyu Xing
Venue: NeurIPS 2023 poster (2023)
Keywords: deep reinforcement learning
interpretation
explanation
TL;DR: This paper proposes a novel method of explaining the decision-making process of a deep reinforcement learning policy.

Abstract: Despite the promising performance of deep reinforcement learning (DRL) agents in many challenging scenarios, the black-box nature of these agents greatly limits their applications in critical domains. Prior research has proposed several explanation techniques to understand the deep learning-based policies in RL. Most existing methods explain why an agent takes individual actions rather than pinpointing the critical steps to its final reward. To fill this gap, we propose StateMask, a novel method to identify the states most critical to the agent's final reward. The high-level idea of StateMask is to learn a mask net that blinds a target agent and forces it to take random actions at some steps without compromising the agent's performance. Through careful design, we can theoretically ensure that the masked agent performs similarly to the original agent. We evaluate StateMask in various popular RL environments and show its superiority over existing explainers in explanation fidelity. We also show that StateMask  has better utilities, such as launching adversarial attacks and patching policy errors.

StateMask: Explaining Deep Reinforcement Learning through State Mask
Zelei Cheng" Northwestern University Evanston, IL 60208
Xian Wu" Northwestern University Evanston, IL 60208 hawpuuu2222Qunnoottheeeeeee edu
zelei. chengdhorttheestern.edt
Jiahao Yu"
Wenhai Sun Purdue University West Lafayette, IN 47907 sun841@purdue edu
Wenbo Guo
Northwestern University Evanston, IL 60208 Jaahao.yy@horthwesterrm www.ortwwestern..ccmmmmm edu
Purdue University West Lafayette, IN 47907
j jiahao.
henrygvb@purdus edu
Xinyu Xing
Northwestern University Evanston, IL 60208
xinyu. sinyx.x.ngghoortwwsseer.....e
Abstract
Despite the promising performance deep reinforcement learning (DRL.) agents many challenging scenarios, the black-box nature of these agents greatly limits their applications in critical domains. Prior research has proposed several explanation techniques to understand the deep learning Cearning-based based policies in RL. Most existing methods explain why an agent takes individual actions rather than pinpointing the critical steps to its final reward. To fill this gap, we propose StateMask, novel method to identify the states most critical to the agent's final reward. The high-level idea of StateMasks is to learn mask net that blinds target agent and forces it to take random actions at some steps without compromising the agent's s performance. Through careful design, agent performs similarly to the original agent. We evaluate StateMask in various popular RI. environments and show its superiority over existing explainers in explanation fidelity. We also show that StateMask has better utilities, such as launching adversarial attacks and patching policy errors.
can theoretically ensure that the masked
Introduction
Deep reinforcement learning has shown promising performance in variety of challenging scenarios, such as allowing robot to learn a control policy directly from the camera or sensor inputs [ training agent master Go games (6] 8. However, DRL policies typically lack explainability, which is intrinsically difficult for humans to comprehend and establish trust. To broaden the adoption of DRL-bbsed applications in critical fields, there is pressing need to increase the transparency of DRL agents through explanation.
Prior research has proposed various methods to derive an explanation for DRL. For example, re- and model approximation methods
searchers utilized the saliency map (e.g., 9
"Equal Contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

to identify which regions in an agent's observation are most critical to the agent's current action. Considering the nature of DRL, researchers also explored the relationship between the agent's s states and its final reward. Take the work [88] as an example. Researchers demonstrated that the state-reward relationship could also be used as an explanation for DRL agent developers. Besides, they also showed that pinpointing the critical time steps towards the final reward could help agent developers better understand agent behaviors and even troubleshoot policy defects.
(e.g. 114 15 16
17
In this work, our research focuses on the DRL explanation methods that explore the state-reward relationships. We evaluate the existing DRL explanation approaches in this category on range of tasks from simple simulated environments to sophisticated real-world applications, and from normal-form games to extensive-fome games. We discovered that existing state-reward-bae -baeed DRL explanation methods provide relatively low fidelity in various reasons. For example, the method proposed [8] suffers from low explanation fidelity because of the limited capacity of their proposed approximation model and the error introduced in the final reward approximation. The methods introduced by [19 and 20 experience low fidelity because an agent's 's value function does not always reflect the state importance to its final reward.
relationship identification for
To this end, we propose StateMash a novel method pinpoint the time steps or, in other words, states most critical to a DRL agent's s expected total reward. At high level, StateMask works as follows. For target agent, we train a mask net that blinds the agent take random actions at certain time steps. If time step is critical, the mask net does not blind the target agent. The agent takes action as its policy suggests. In contrast, if the time step is less critical, the mask net blinds the agent. The agent takes random actions. The rationale behind this design is that DRL agent's final reward not equally important for each time step. Randomly taking some actions at non-critical time steps has no impact on the agent's 's final reward. Technically, to learn the mask net, we carefully design an objective function that minimizes the performance difference between the original agent and the masked agent. We demonstrate, both theoretically and empirically, that performance differences could monotonically decrease during training.
While other explanation methods could track down the time steps most or least critical to an agent's s final reward, this is the first work that could identify an agent's critical time steps with high fi- delity across various types of RL environments, especially sophisticated normal-form games and extensive-form games. Evaluating StateMask in various game environments, we show that the explanation derived from our method demonstrates significantly higher fidelity than those produced by existing explanation techniques. Compared with the existing methods, StateMask allows humans to understand the agent's 's behaviors better and calibrate appropriate trust. Agent users can easily pinpoint an agent's 's policy defects and patch the identified defeat under the guidance of explanations. Last but not least, security professionals can launch adversarial attacks using the explanation from our method.
In summary, this paper makes the following contributions.
We propose a new explanation method StateMask and provide a theoretical analysis. We show that an agent armed with explainability is equivalent to the original agent in terms of agent performance.
We evaluate StateMask on 10 different tasks ranging from simple normal-form games (e.g., Pong) to sophisticated normal-form games (e.g., StarCraft II) and extensive-fome games (e.g., Connect 4). We show that StateMask can significantly outperform existing explainers.
We demonstrate that our explanation can be highly helpful for understanding agent behavior, launching adversarial attacks, and even remediating the agent's errors.
2 Related Work
DRL Explanation. Existing DRL explanation techniques can be mainly categorized into two classes based on their different explanation objectives: (1) the observation-actto based method pinpointing which regions in an agent's observation are most important its action at an individual time step; 2) the state-reward-barr based method tracking down which time steps have the most critical influence on the agent's final reward.
"The source code of StateNask can be found in https:t:p/s attps:///githut http:////g..cccctcc/////////aaaa state, _mask

---

Recall that a DRL agent's s policy network takes as input the agent's 's current observation and outputs the corresponding action. In this setup. pinpointing regions in an observation most critical to the corresponding action is the same as identifying the most important regions in a deep neural network's (DNN) input. As such, existing works along this direction mainly utilize or extend the explanation methods developed for DNN classifiers to derive such explanations. Technically, they mainly leverage
three lines of methods: post-hoc white-box explanation methods [21 23 24], post-hoc black-box
explanation methods [15] 25 26 227. and self-explainable methods [28] 29 30 31.. In addition,
another line work identifies important regions by decomposing the agent's ' instant reward to regions
in its current observation and selecting the regions with high rewards 32 33 34 B5].
Regarding the second type of explanation methods, most of the techniques identify the critical steps based on the value function [19 (0) [6). Given that the value function measures the goodness of the current state towards the agent's expected total reward, it is natural to use it for critical time step selection. Recently, Guo ef al. [18] proposed EDGE that establishes state-reward relationship by collecting a set of trajectories and then approximating an explanation model offline. However, as discussed in [18), EDGE has two limitations. First, the approximation method is based on statistical model not effective in deriving explanations for agents that usually generate long trajectories. Second, for each new trajectory, EDGE needs to build an independent approximation model, which significantly limits its efficiency and scalability.
Our proposed technique falls into the second type of explanation, which follows different design path to overcome the limitations above. As we will discuss later, our method is superior to existing techniques in multiple aspects: higher explanation fidelity (Section 4. better utilities (Section [5) as well as better generalizability (to extensive-form games) and practicability (in the cases where the agent's 's value function/network is not available).
Perturbation- based Explanation. Perturbation techniques have been widely utilized within the field of computer vision to gain a deeper understanding of black-box image classifier [37 38 39 40).
These approaches involve introducing minor modifications the target image and observing how the classifier's 's outputs change, which enables us to identify the regions of greatest importance within the original image. Our explanation method shares similar idea with these perturbation methods. Rather than explaining supervised learning (e.g.. image classifier), our focus is on interpreting a DRL agent by introducing perturbations in non-critical time steps. This temporal perspective provides unique insights into the DRL. agent, which has not been explored in prior research.
3
Key Technique
In the following, we describe the rationales behind our design and then discuss the key technical challenges, followed by the technical details of tackling the challenges.
3.1 Overview
Assumptions. To ensure the practicability of our method, we do not assume access to the agent's s value function or policy network. Instead, we only assume having access to the agent's observations and the ability to query the agent's 's policy and thereby obtain corresponding actions. Note that our explanation method does not change the target agent's 's policy. For a multi-agent environment, we explain each agent individually.
Design Rationale. To tackle the limitations confronted by existing state-reward-barde explanation methods, our idea is to parameterize the importance of the agent's current time step as a neural network model (denoted as state mask). The state mask net takes as input the current state and outputs its importance to the agent's final reward. The key challenge of this solution is to propose proper criterion or metric to decide the importance of the current time step and thus design an objective function to train this state mask. In our work, we draw insights from existing counterfactual -aased At high level, these
explanation methods designed for supervised classifiers [37 42 4 methods identify important features in an input sample by nullifying or perturbing the features and observing the corresponding changes in the classification output. They deem the features triggering the largest prediction changes as the most important ones. Similarly, in our problem, we can decide whether a time step important by randomizing the agent's action at a certain time step and observing changes in its final reward. A small variation in the final reward indicates the reward is not sensitive
3

---

(a) The DRL agent just launched the ball.
(b) The DRI. agent attends to catch the ball.
Figure 1: Illustration of the Pong game. The DRL agent controls the yellow (right) paddle and non-RL rule-based opponent controls the white (left) paddle.
to the action change, further implying the low importance of the state. In contrast, a large reward change infers the high importance of the corresponding state.
To elaborate on this idea, we use a pong game as an example. As shown in Figure ((a) after the DRL agent launches the ball and when the ball is moving towards its opponent, the agent has few time steps of freedom to choose random actions. This is because the agent's actions at these steps will not change the movement of the ball and will be less likely influence the opponent's 's actions as well (given the opponent's ss focus is on the ball). As a result, forcing the DRL agent to take random actions at these steps will likely have no influence on the game's final result and thus trigger only minor changes to the agent's final reward. However, if we randomly perturb the DRL agent's action the time steps when the agent is about to touch the ball (Figure [(b).. will likely cause the agent to fail and thus trigger large variation in the agent's 's final reward.
State Mask Modeling. Guided by this idea, design the state mask that takes the current state as input and outputs the probability of whether to randomize the agent's 's action at the current state. The objective of this state mask is blind target agent and thus vary its actions certain states in trajectory without changing the agent's final reward. This probability can be taken as the importance of the current state (i.e., the higher the probability is, the lower the importance is). This design introduces a challenge. That is how to decide whether varying the action at the current state will influence the agent's future reward.
To address this challenge, we model as an RL problem, where fix the target agent's policy and treat it as part of the environment. Then, we introduce another agent to decide whether to randomize the target agent's 's action at each step be modeled as the state
S,
Target agent
a, (I
)))
a #(aa |s,
StateMask
not. Its policy mask above. This agent's 's goal is to preserve the target agent's total reward while randomizing actions at more states.
a Oa
Environment
Figure 2: The StateMask framework.
Following this idea, we propose a novel expla- nation framework, StateMask. Figure 2 illus-
trates this explanation system. As mentioned above, add state mask (agent) to the environment. We fit the current state 8; into both the state mask ग and the target agent म. For a given state s,, the state mask outputs binary action of of either "zero" or "one". and the target agent will sample the action a from its policy. The final action is determined by the following equation
if 0, if af = 1,
(I)
Orandom
where of 0 represents keeping the target agent's current action. On the contrary, of = 1 refers to replacing the target agent's current action with a random action Grandom uniformly sampled from its action space. Note that Grandom is different from d;-
The goal of our state mask to find the non-important time steps and randomize their actions without changing the expected total reward of the target agent. Formally, it can be expressed as (0) minimize|n/ (n(miiie//////))))) -=)])
where denotes the policy of the target agent and π denotes the policy of the perturbed agent (i.e., integration of the state mask क and target agent T). 17(-) is the expected total reward of an agent by following a certain policy. By fixing म. n) is also fixed and m7( depends only the parameters
We measure the performance gap by the Lp distance, and without loss of generality, the L1 norm.

---

of the state mask. As result, by solving Eqn. (2, we can obtain state mask. By applying this resolved mask to each state, we will be able to assess the state importance any time step.
Key Optimization Challenges and Solutions. Obtaining a non-trivial solution of Eqn. (2) presents the following challenge. To ensure the optimization function is differentiable,
need to decompose and remove the absolute value operation. A straightforward solution is to compare the value of ())) and (**) during each training iteration. If 1y(T)) larger than f7( we update the state mask by minimizing (T)). Otherwise, we maximize TT(T). With simple reformulation, we can then leverage the of-et policy gradient method Proximal Policy Optimization (PPO) 44 update the state mask in each iteration. However, the key limitation of this solution is the lack of convergence guarantee. Suppose the policy of the perturbed agent near local optimal and ()(T) is less than (T)) Then, updating the perturbed policy by maximizing (TT)) may lead to an overly large 7(T), which, in turn, requires minimizing 17( When this situation occurs, the perturbed policy a may oscillate around the local optimal without actually converging. To address this problem, we borrow the idea of TRPO technique 45 carefully designing a surrogate objective function for Eqn. (2) As we will elaborate below, through the surrogate objective function, we can guarantee the difference between (T)) and (T)) (i.e, (m([))- - (+))) monotonically decreases during the training process, and our optimization process will at least converge to a local optimal rather than oscillating around it.
3.2 Technical Details
In this section, we formally introduce our proposed StateMask framework, including modeling the state mask and designing the surrogate objective function ensure monotonicity.
Defining the State Mask. An infinite-horizon Markov Decision Process (MDP) defined as {8, A. P. R, 7}. where S and A are the state and action space. P S x A + A(S) is the state transition function. R: : x A + IR the reward function and 7 E (0, 1) is the discount factor. A policy is a mapping from S to probability distribution over A. The expected total reward of 1E (St, at)].
policy π is defined as (T)) = Eso.ao...
Based on this definition, we define our explanation as an Explanation Markov Decision Process problem (E-MDP), in which we formalize state mask as an explanation policy : S A(A"). where A {0, 13. With this definition, the integration of the explanation policy स and the target policy (that we aim to explain) then becomes (-1ss) = (a = (+})=(+1s) + (a (s)))) Grandom- where Grandom o (--ss).
As mentioned above, we fix म. In this way, we can transform the E-MDP into single-player environment, where the state-transition, reward function, and TT(T) depend only on the policy of the state mask R. As such, we can redefine (T)) as (f)). The goal here train an optimal policy ग to minimize the difference between (7T) and (+))
Solving the Explanation Policy. With the definitions above, the objective function in Eqn. could be reformulated as J(0) minimize| (mt)) - v7()),, where refers to the parameters of the explanation policy. As is discussed in Section 3.1 ensure the monotonic property, we subtly design a surrogate objective function to solve 0. To introduce our designed objective function, we first present the following lemma.
Lemma 1.
(45) The difference between Langua (fte) and 17 (0)))) is bounded by
In((t)) Lan (fre) S CKI. (#ooua (-1 (-Is))]lfo0 s)]l s)).
(3)
Ti denotes the new policy, updated from Toota- Laona (me) = n) + Σ PRo.. (s)
Σ
#o(a
8) Añe (8, aa), where PRoaia (s) = E. P(ss, = s s/froraa and Añe is the advantage function of म 0 IKT. max (No (-1 [oo (-1 s)) max, KT. (:1 [1=(0- s)) and C' is constant.
Based on Lemmal we can derive the following theorem (The derivation in Supplement S1).
Theorem 1. The following inequalities are the sufficient conditions for satisfying (n( n) s an( - [.)]]
Laoad (#o) 2 T((Ro + CKL max (Fo (-1 s)) (.1 s)),
(4)
Laggr (Te) + CKLL (Moata (:1 [-F 8)) S 2n( of (FReaa))
5

---

Algorithm The learning algorithm for training the mask net.
Input:
Target agent's policy स and the estimation of its expected total reward sy(1)) Output: Mask net Thg
Initialization: Initialize the weights 0 for the mask net āe. Lagrange multiplier À = 0 for iteration=1, 2, ... do
Run the current policy Reata in the environment for I' steps and get estimation for Añoza
Solve the objective function with respect to 0 in Eqn. (7) and update Ootd <- 0
Solve the objective function with respect to Eqn. (8 and update Xotd +- A
end for
Based on Theorem
we propose the following objective function to update 7e from Teaa
max
Liteatd (Fe)
s.t.
Titeaid (#t) 21)(() -T((torr)) 8, Enswao [a"] 2 C. Ba-Ptead [KL (.1 [| {)]] d.
(5)
where Eu [a*] 2 constrains the lower bound sparsity for the output of 7eg across all states, with e E [o, 1] as a constant syper-paramette It rules out the trivial solution where the state mask outputs O at any state. As detailed in Supplement S2, policy Ho solved from Eqn. 5 satisfies the conditions in Eqn. (4) and thus enables the desired monotonicity.
Eqn.
5 can be further transformed into the following function based on the Lagrangian duality.
min max
Ligate (Ta)
X(LFoats (Ta) 2x)(=) (T + 6),
(6)
EgewEo [a'] C, Egwpeead [KL (EL (To [[ 8.
Below we further discuss our techniques for solving Eqn. 6 In particular, we first follow the PPO algorithm [4]] and transform the Eqn. (6) into the following objective function for 0 (See Supplement S3 for more details)
maxg sgn(1 2) min
Reas (u] [x]] Asgaia clip
(a st)
se) Fu (af [x.)
to (af
1 (.)
Aseald
+ w. MASK
Here, A A (St, a)) and I. L.MASK Eug [at] sgn is the sign function which returns 1 or 1 based on whether the input is positive or negative. w is a hyperparameter controlling the weight of the corresponding term during optimization. The outer expectation is taken over a finite batch of state and action, sampled by running the current policy me....
Similarly, we can also transform Eqn. 66) into the following objective function with respect to À
min ((2m(T)) m)(me ô Lisaae (te)) A20
(8)
We follow the primal-dual method 46 47 and iteratively update 0 and À by solving Eqn. and
Eqn. ( using the gradient- gradient-based optimization method. Algorithm[[rrieff presents our learning process. Supplement S4 shows a more detailed algorithm including how to estimate (7()) and Axa..a"
As mentioned above, as an explanation method, we fix the policy of the target agent in the environment and only train state mask. For multi-agent environment, can learn individual state mask for each target agent. This ensures the generalizability of our method to multi-agent setups. As specified in Supplement S4, with minor implementation modifications, we can also generalize our method from normal-fore form games. where the agents observe the same state and take action simultaneously, to extensive-for fornssie-formmno agents, where the agents make decisions sequential order.
Evaluation
In the following, we start with our experiment setup and design, followed by experiment results and analysis. To ensure the reproducibility of our results, we specify the evaluation details in Supplement S5.

---

4.1 Experiment Setup
Environment Selection. We select 10 representative environments to demonstrate the effectiveness
of StateMask across four types of environments: simple normal-fomm game (CartPole, Pendulum, and Pong [48)), D. sophisticated normal-form game (You-Shall-Not- Shall-Not-Pass [49].
[49],
and StarCraft II [50]), perfect-information (simple) extensive-form game (Connect 4, Tic-Tac-To and Breakthrough (51), and
(sophisticated) extensive-form game (DouDizhu [2]
Baseline Selection. Sharing the same explanation goal, as discussed in Section 2 two existing methods explore the state-reward relationship the dalue-ffnctioon-aaee
method and EDGE [8]
For the value-function-base
method, we implement two methods Value-max method [9 20 and
LazyMDP (36). The value-max method associates the state importance with its value function V (s). LazyMDP pinpoints critical steps according to the lazy gap, which is defined as max Q a)
[. a)). Here, Q(s, a) is the action-value function. For EDGE, we implement it based on their code [18] We take an absolute value for the important scores drawn by these methods to align their meaning with our methods.
4.2 Experiment Design
Experiment I: Expected Reward Preservation. Recall the state mask blinds the agent and forces it to take random actions at some least important time steps. If the blinding is accurate, the agent's performance should not be affected, and the agent's final reward should not vary significantly. As result, one of our evaluations to examine whether the masked agent could preserve its expected total reward. To do so, for a target agent from a game, we compute its discounted total reward across 500 rounds. Then, we treat the averaged computation result as the estimation of the agent's expected total reward ( Following the computation of the expected reward, we further train our StateMask in the corresponding game.
As mentioned above, the combination of the state mask and the target agent's policy is the perturbed policy. Therefore. after having the state mask in hand, we further compute the discounted total reward of the perturbed policy across 500 rounds. We treat the averaged computation result as the
approximation of T)())), and then use to compute the relative value variation 1x(*)-n((tt) As [o(a)]
is shown from this equation, the relative value variation represents the change of the reward. If
this measure is low, it indicates that the state mask has minimal impact upon the target agent's performance. The state mask accurately pinpoints the time step least important to the final reward.
Experiment II: Explanation Fidelity. Apart from evaluating our explanation method based on the measure of expected reward preservation, we also employ a fidelity test proposed 18 to assess its effectiveness. The fidelity test involves using the corresponding explanation method to identify and rank the most crucial time steps in given trajectory. Subsequently, a continuous sequence of time steps is selected based the rank of the time steps. The fidelity test evaluation algorithm proposed in [18] ensures that the selected sequence includes the maximum number of critical steps, indicating the most crucial continuous time steps leading to the final reward of the agent. Additionally, the length of the sequence can be specified as an input to the fidelity test evaluation algorithm. For instance, one can limit the sequence to only the top K time steps. For more information about the fidelity test, readers can refer to Supplement S5.
Following the rest of the fidelity test introduced in 18 for a given trajectory and the most critical sequence in it, we let the agent fast-forward to the beginning of the critical sequence. Then, we force the target agent to take random actions until the end of the critical sequence. Later onwards, we use the target agent's policy to complete the game. By doing so, we simulate a situation where force the agent not to follow its policy at the most critical time steps. If the time steps truly critical to the final reward, we expect the action replacement at critical time steps could introduce the highest reward variation compared with replacing any other continuous sequence containing K time steps. In 18), Guo et al. introduce a fidelity score to measure the change of the reward after the replacement of the actions at a continuous sequence of time steps. The higher the measure is, the higher the explanation fidelity is.
In this work, we use StateMask (the state masks trained above) and baseline methods to explain 500 trajectories of each target agent and obtain the time step importance for each trajectory. Then, for each trajectory, select their top K important steps based on the importance ranking given by each
7

---

20
13
MalaOn- Nee-hhdlll4.aaa
Opentgal- Comeet
Doutuda
Figure 3: Fidelity scores of explanations generated by selected methods games. The x-axis represents the choice of K. "VALUE" stands for Value-max method, and "LAZY" stands for MDP. The black line refers to the standard deviation of 3 runs. The bar refers to the mean and the line the bar represents the standard deviation.
Figure 4: Visualization of the identified critical time steps in Pong game. The DRL agent controls the yellow paddle and non-RL rule-based opponent controls the white paddle.
method and compute the fidelity score. We compute the average score across 500 trajectories as the fidelity score for that explanation method. We set K = 10%, 20%, 30%, 40% and report the fidelity of the selected methods under each setup. We repeat each experiment 3 times with different random seeds and report the mean and standard deviation.
4.3 Experiment Result
Due to the space limit, we show the results of five games: Pong, You-Shall-Not-Par Connect DouDizhu, and StarCraft II. Supplement S5 shows the other games' results, which are aligned with those discussed this section.
Relative Value Errors. The mwximum/mininum//moooo mwtpm/mmmmiimmnnnnnmmmmmmm relative value errors across all 10 games are 14.98%/0.00%/5..77 (The exact values are Supplement S5). These results verify the effectiveness of our design objective function (Eqn (6)) in preserving the target agent's 's performance. In addition, we also show the trends and variations in (y)()) n))o)l during the training process for each game. The figures in Supplement S5 demonstrate that our design could indeed achieve the monotonical property, and further improve the stability the training process.
Fidelity Scores. Figure 3 shows the fidelity scores of StateMask against the three baseline ap- proaches in five games while the results of the remaining five games could be found in Supplement S5. First, we can observe that our method has the highest fidelity scores across all games in all settings, indicating that StateMask provides more faithful explanations for the target agents than the baseline approaches. Second, we discover that in two complex normal-form and extensive-form games (DouDizhu and StarCraft II), StateMask has a significant advantage over EDGE. This result validates our argument in Section 2 that EDGE's approximation fails to handle long trajectories. Besides, we also discover that StateMask outperforms balue-based methods in those two games. We believe that the reason is that the value function is trained to facilitate policy learning and typi- cally tends to output similar values for sequence continuous states [36), causing the time step importance to be less distinguishable. While StateMask specifically quantifies the importance of each time step via action randomization, giving more specific and distinguishable important scores.
In addition to the experiments above, we also demonstrate our method's efficiency and its insensitivity against the variations in the key on not well-trained policies with sub-optimal performance. Due to the space limit, we detail these experiments in the supplementary materials. We further conduct an additional comparison to verify the necessity of minimizing the difference in the agent's expected total reward before and after applying the state mask. We consider an alternative design that directly maximizes the agent's expected total reward after applying the state mask (i.e., (()) maxp v(fa)) and compares it with our method. Our results show that StateMask produces higher fidelity explanations on not well- trainer policies than this alternative method (See Supplement S10 for more details).
We also verify the effectiveness of method

---

Figure 5: Visualization of the identified critical steps in the Connect 4 game. The black player is our explanation target player. The states where the blue player takes actions are marked with gray bar. Table I: The target agent's performance different applications. Numbers before the brackets are means and those in the brackets are standard deviations.
Applications
Games
EDGE
Value-max
LazyMDP
Ours
Pong
-63.04 (3.44)
(0.40)
(1.13
-84.90 (1.40)
Target agent's winning rate
You-Shall-Not-Par
-2.64 (0.60)
(3.20)
-28.87 (1.21)
-35.93 (1.81)
Connect 4
33.355(1.50)
(1.44)
-22.57 (1.40)
-35.37 (3.00)
change before/after adversarial attacks
DouDizhu
(1.71)
-18.60
-25.20 (0.76)
-29.94 (0.64)
StarCrait
(14.18)
-12.00 (5.69)
-33.00 (7.21)
Target agent's winning
Pong
+1.13 (0.12)
+0.53 (0.12)
+1.47 (0.12)
+1.87 (0.12)
You-Shall-Not-Pun
+0.07 (0.12)
+1.07(00122
(0.12
+2.42 (0.18)
change before/after patching
Connect 4
+2.00 (0.00)
+1.47
+1.33 (0.12)
+2.40 (0.20)
DouDizhu
+0.00 (0.00)
(0.13
+0.20 (0.00)
+0.47 (0.12)
5 Utility of Explanation
In this section, we demonstrate that with a higher explanation fidelity, StateMask could also achieve better utilities in three aspects: understanding agent behaviors, launching adversarial attacks, and patching agent errors.
Understanding Agent Behaviors. We visualize the identified critical steps in all 10 games to demonstrate how StateMask helps humans understand DRL agent's s behavior. Due to the space limit, we show the results of two games: Pong (normal-form game). Connect 4 (perfect-informatine extensive-fore game), and leave the rest in Supplement S6.
Regarding the Pong game Figure shows, our explanation method highlights the frames when the ball is approaching the yellow paddle as the most important time steps whereas the states when the ball flies to the white paddle the least important time steps. It aligns with humans' intuition since the steps when the ball flying towards the yellow paddle are apparently critical to the final rewards. Without proper actions at these critical time steps, the DRL agent will miss the ball and lose the game.
In Connect 4, two players take turns dropping colored stones onto a grid to connect four stones in line.
visualizes one game where the black player wins. StateMask pinpoints the first time step (where the target agent makes the first move) as an important step. Allis et al. [53] shows that
placing a stone in the center column as the first action typically brings an advantage. StateMask also highlights the last two time steps, which drive the target agent to win the game.
Additionally, conduct user study to compare our explanation model with baseline methods (Value-max, EDGE, and LazyMDP) and see which one could help humans gain a better understanding of DRL agent's 's behavior. Our study is determined be exempt from an IRB review by the IRB panel. In our user-study survey, the participants are first given some background information about this project and asked for their consent to proceed. The hypothesis be tested that our method can offer a better explanation of the target agent's behavior in comparison with baseline methods. We use the convenience sampling method [54) recruit 41 participants for our study. The participants come from a variety of backgrounds in explainable reinforcement learning (XRL), with 20% having never heard XRL, 51% having some knowledge of it, and 29% being an expert or having published papers in this field. Next, the participants are presented with two groups of videos of the Kick-And-Deffed game. The videos record the agent's 's play and highlight the critical time steps based on different explanation methods.. Additionally, these two groups include trajectories played by near-optimal and sub-optimal agents. The participants are instructed to watch these two groups of videos and choose which explanation best matched their intuition.
blue player places their piece, the black player has a winning strategy. 'In second-to-len step, foundation for the game's victory established. matter which column the
'The link to our survey: atttp::/tiinya com/ytwj7wdt

---

According to the survey results in Figure [ 71% of the participants prefer our explanation method when the agent is near-optimal and 59% of them favor our explanation method when the agent is sub-optimal. Since the majority of participants choose our method, our hypothesis is con- firmed. Our method outperforms the baseline methods in helping the user understand DRL agent's policy,
12%
LAY
71N
Optimal
Sub-optimal
Launching Adversarial Attacks. We follow the method developed in EDGE [18] and launch attacks against the target agent based on explanations drawn by the selected methods. Specifically, we fix the percent of time steps under attack T as 10% and launch this attack against 500 episodes. We then compute the average reward that the agent gathers before and after the attack. Table S7 further shows additional results when varying the attack ratio from 10% to 30% in all ten games. As shown in the figure, the adversarial attack under the guidance of all explanation methods could dramatically drop the victim agent's ' performance, indicating the effectiveness of the interpretation and the exploitability of the attack. However, the performance losses observed on the adversarial attacks under the guidance of the three baseline methods are significantly lower than what we observed on the attack enabled by our method. This again implies that the attack guided by our method is more effective, and thus our interpretation has higher fidelity than the three baseline methods.
Figure Results of our user study.
the results in five games (same as the games in Section 4.3) Supplement
Patching Agent Errors. Rather than directly applying the rule-based patching method in EDGE [8] we propose a novel learning-based method (Supplement S8 shows the advantages of our learning- based method over the rule-based one proposed in EDGE). Given set of trajectories, we first leverage selected explanation methods to identify sequence of continuous time steps that are most critical to the final reward in each trajectory. Then, we replay these trajectories to the starting point of the critical steps and fine-tune the agent's policy till the game end. We validate our patching method on seven games (except CartPole, Breakthrough, and StarCraft II whose agent has nearly optimal performance) and report the results in Table and Supplement S8. As shown in the table, our method-guide -uilodd agent achieves the best performance after retraining.
6 Conclusion and Discussion
In this work, we propose StateMask a novel explanation method for DRL agents. The use of StateMask
explainer reveal the state-reward relationship with exceptional accuracy. Compared to other types of explainers, StateMask offers the highest level of fidelity in explanation. By providing such precise explanations, we can assist humans in comprehending agent behaviors, launching adversarial attacks, and even identifying and correcting policy errors.
Our work suggests several promising directions for future research. First, in safety-critical environ- ments, masking the agent's ' state and forcing it to take random actions may not be feasible due to high exploration costs. However, train DRL agents in such environments, the model developer typically needs to build a simulator for the environment, which can be used to generate explanations. As part of future work, we will explore building simulators for such an environment if unavailable. Second, although StateMask achieves high fidelity, it may not identify cases where positively and negatively important states the same trajectory offset each other's influence on the final reward. In the future, we will investigate additional steps to handle such cases. Third, while our explanation methods can identify important states, presenting the states humans may not be ideal. Other future efforts will also include converting critical states to numan-understame study in Section shows the feasibility of explanation -uided policy fine-tume fine-tuning. We plan to improve the fine-tume onn-tuning efficiency and the generalizability of the retrained DRL agents in the future.
strategies. Finally, our
10

---

Acknowledgement
This project was supported in part by NSF Grant 2225234, NSF Grant 2225225, NSF Grant 2238680. and Amazon Research Award.
References
[1] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 2016.
[2] Lei Tai, Giuseppe Paolo, and Ming Liu. Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation. In Proc. of IROS, 2017.
[3] Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, 2021.
[4] Zelei Cheng and Zuotian Li. Poisoning attack for inter-agent transfer learning. Proc. of SecureComm, 2021.
[5] Homer Rich Walke, Jonathan Heewon Yang, Albert Yu, Aviral Kumar, Jędrzej Orbik, Avi Singh, and Sergey Levine. Don't start from scratch: Leveraging prior data automate robotic reinforcement learning. In Proc. of CoRL, 2023.
[6] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che. Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
[7] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815. 2017.
[8] Julian Schritwieser,, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 2020.
[9] Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. Trans- parency and explanation in deep reinforcement learning neural networks. In Proc. of AIES, 2018.
[10] Sihang Guo, Ruohan Zhang, Bo Liu, Yifeng Zhu, Dana Ballard, Mary Hayhoe, and Peter Stone. Machine versus human attention in deep reinforcement learning tasks. In Proc. of NeurlPS, 2021.
[11] Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, and Sameer Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. In Proc. of ICLR, 2019.
[12] Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning. In Proc. of ICLR, 2020.
[13] Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Alan Fem, and Margaret Burnett. Explaining reinforeement learning to mere mortals: an empirical study. In Proc. of IJCAI, 2019.
[14] Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingean Li. Toward interpretable deep reinforce- ment learning with linear model u-trees. In Proc. of ECML-PKDD, 2018.
[15] Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Proc. of NeurIPS, 2018
[16] Alihan Hüyük, Daniel Jarrett, Cem Tekin, and Mihaela van der Schaar. Explaining by imitating: Understanding decisions by interpretable policy learning. In Proc. of ICLR, 2020.

---

[17] Zihan Ding, Pablo Hernandez-Leal, Gavin Weiguang Ding, Changjian Li, and Ruitong Huang. Cdt: Cascading decision trees for explainable reinforcement learning. arXiv preprint Stti::001100755.. 2020.
[18] Wenbo Guo, Xian Wu, Usmann Khan, and Xinyu Xing. Edge: Explaining deep reinforcement learning policies. In Proc. of NewrlPS, 2021.
[19] Dan Amir and Ofra Amir. Highlights: Summarizing agent behavior to people. In Proc. of AAMAS, 2018.
[20] Sandy H Huang. Kush Bhatia, Picter Abbeel, and Anca D Dragan. Establishing appropriate trust via critical states. In Proc. of IROS, 2018.
[21] Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning. In Proc. of ICLR, 2019.
[22] Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understand- ing atari agents. In Proc. of ICML, 2018.
[23] Piyush Gupta, Nikaash Puri, Sukriti Verma, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, and Sameer Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. In Proc. of ICLR, 2020.
[24] Janis Klaise, Arnaud Van Looveren, Giovanni Vacanti, and Alexandru Coca. Alibi explain: Algorithms for explaining machine learning models. The Journal of Machine Learning Research, 22(1):819448200 2021.
[25] Tom Bewley and Jonathan Lawry. Tripletree: A versatile interpretable representation of black box agents and their environments. In Proc. of AAAI, 2021.
[26] Joe McCalmon, Thai Le, Sarra Alqahtani, and Dongwon Lee. Caps: Comprehensible abstract policy summaries for explaining reinforcement learning agents. In Proc. of AAMAS, 2022.
[27] Himabindu Lakkaraju, Nino Arsov, and Osbert Bastani. Robust and stable black box explana- tions. In Proc. of ICML, 2020.
[28] Laurens Weitkamp, Elise van der Pol, and Zeynep Akata. Visual rationalizations in deep reinforcement learning for atari games. In Proc. of BNAIC, 2018.
[29] Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. In Proc. of NeurIPS, 2019.
[30] Wenjie Shi, Gao Huang, Shiji Song, Zhuoyuan Wang, Tingyu Lin, and Cheng Wu. Self- supervised discovering of interpretable features for reinforcement learning. IEEE' Transactions on Pattern Analysis and Machine Intelligence, 2020.
[31] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Deep reinforcement learning with relational inductive biases. In Proc. of ICLR, 2019.
[32] Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi-Velez. Explainable reinforcement learning via reward decomposition. In Proc. of IJCAI/ECAI Workshop, 2019.
[33] Zhengxian Lin, Kin- Lam, and Alan Fern. Contrastive explanations for reinforcement learning via embedded self predictions. Proc. ICLR, 2020.
[34] Tom Bewley and Freddy Lecue. Interpretable preference- reinforcement learning with tree-structured reward functions. In Proc. of AAMAS, 2022.
[35] Zichuan Lin, Li Zhao, Derek Yang, Tao Qin, Tie Liu, and Guangwen Yang. Distributional reward decomposition for reinforcement learning. Proc. of NeurIPS, 2019.
[36] Alexis Jacq, Johan Ferret, Olivier Pietquin, and Matthieu Geist. Lazy-mdps: Towards inter- pretable rl by learning when to act. In Proc. of AAMAS, 2022.
12

---

[37] Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. Proc. ICCV. 2017.
[38] Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Proc. of NeurIPS, 2017.
[39] Andrew Elliott, Stephen Law, and Chris Russell. Explaining classifiers using adversarial perturbations on the perceptual ball. In Proc. of CVPR, 2021.
[40] Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In Proc. of BMVC, 2018.
[41] Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In Proc. of ICLR, 2019.
[42] Yash Goyal, Ziyan Wu, Jan Erstt Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. Proc. of ICML, 2019.
[43] Zuotian Li, Xiang Liu, Zelei Cheng, Yingjie Chen, Wanzhu Tu, and Jing Su. Trialview: An ai-powered visual analytics system for temporal event data in clinical trials. In Proc. of HICSS, 2024.
[44] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint a7X1v:1707.06347, 2017.
[45] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proc. of ICMI,, 2015.
[46] Chen Tessler, Daniel Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv: 100. / 1074, 2018
[47] Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained reinforcement learning has zero duality gap. In Proc. of NeurlPS, 2019.
[48] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jic Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 16XX::160660114000 2016.
[49] Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. In Proc. of ICLR, 2018
[50] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv: 2017.
[51] Mare Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, et al. Openspiel: A framework for reinforcement learning in games. arXiv preprint arXiv: 1908. 09453, 2019.
[52] Daochen Zha, Jingru Xie. Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Ji Liu. Douzero: Mastering doudizhu with self-play deep reinforcement learning. In Proc. of ICML, 2021.
[53] Louis Victor Allis. A knowledge-based approach of connect-four. Journal of the International Computer Games Association, 1988.
[54] Mark Saunders, Philip Lewis, and Adrian Thornhill. Research methods for business students. Pearson education, 2009.
13

---

StateMask: Explaining Deep Reinforcement Learning through State Mask
S1 Proof of Theorem 1
Before we proving Theorem 1, we first introduce the following lemma.
Lemma 2. Given old and new policy of the state mask Agai and fhg respectivcly, if n (ftera))
n(ee) - 2m( m) have (n( - - s In((Fe.aa) --))
Proof of Lemma 2 The monotonicity property requires (n((a)) nn) S In(#e.t)) n/T)). The following equivalence holds:
In(eee) r7(5)) S ( r)
n(te)2 2n (( (5)) S n( 12 2n 2ny () + n)7)²
(1)
(m())) + 77 (o (+)( - 1y( ( S (T( (n)(t)) ny( (
Therefore, if () n)o)) and (a)) + ( (7) in other words, ( S (())) (-))
2r)(#) - (TTersaa) we have |n( [-) S Recall that Lemma 1 in Section 3.2 shows the upper bound and the lower bound of ((te)). An immediate implication of Lemma enables us to obtain the inequalities to satisfy Lemma 2
From Lemma 1, we can easily derive the lower bound of n)(a)) as n)(tt) 2 Laagee ( - CKL max (#o.ta
8) f)] 8)). the lower bound greater than r)(5auu), then we can guarantee that v7(Ta)) 2 () (Ta Similarly, we can obtain the upper bound of (ftt) as () S Li (#e) + CKL max (#g (-1 ) (-))
If the upper bound is lower than 2n(#)= 77 ( can guarantce that n)(#o) 2n((T) 1 (a Putting all these inequalities together, the following inequalities are the sufficient conditions for satislying (n(#a) ())) < (n((aat)) (T]]]:
Laoad (5o) 2 n((ratt) + CAL (-1 +)]
T (#a) CKT. max (Fe (11 (-( x)) S 2n(() n ((
S2 Derivation of Eqn. (5)
First, let Mäona (5e) = Lit (Tt) CKLLmax ( (.1 {)] (- s)). We can get Mäo (Te () By maximizing the Margee (To), we can ensure M (Ta) 2 Mar.i (#oma) = n (0 Given that n( 2 Ma (ite), we could satisly the first inequality of Eqn. (4).
By constraining the upper bound of Raola to be smaller than 2n)(#) ((F.. we could guarantee the second inequality of Eqn. (4) holds.
Therefore, combining all these constraints together, we can obtain the following optimization objective:
max Ligas (5v) C.I, (Fo < ()) s))
s.t. Liaga (8t) CKLL"M max (No a)]]Rt(-1 s)) s 2n(R) ((Keaa))
practice, Schulman et [ pointed out that utilizing the penalty coefficient C may lead to a very small step size. Therefore, we replace the penalty on the maximal KT-ddivegeece with a trust. region
constraint on an average KL-divergence (i.e., Ee- [KIL (-1 () s))] S 8) suggested
in 1
We can further obtain the following objective

---

max
Langid (#t)
s.t.
Laoaid (Rx) S 2n((5) - A 6,
(4)
Ex-exerd [KL(Fo (-1 [- a)}] 8.
Additionally, we add a sparsity constraint which will only tighten the problem without affecting the monotonicity property and obtain the following optimization objective
max
Laa (Fe)
8.1.
!
Egenay [a"] 2
(5)
Laoaad (T0) S 2T( - 10( -
Baw [KL[ (-1 ))( 8)]] S 6.
where Ea [a'] constrains the lower bound sparsity for the output of äe across all states. By maximizing the local approximation with constraint ① could guarantee that Ta (me) Legnu (#ogaa) (f0.a.)) Since Lsa, (#a) is the local approximation of n)(o),, we could further guarantee that T(aa)) 2 (..a)) Moreover, the constraint L (fØ) < 2n(( - á could ensure that the inequality T)aa))= s 2m( n is satisfied. Therefore, a policy The solved from the optimization objective above satisfies Lemma 2 and thus enables the desired monotonicity.
S3 Derivative from Eqn. (6) to of Eqn. (7)
With respect 8, Equ. (6) becomes
max (1 Ao (Te) Or Pepppaoas
[KL (frosia (-1 (-]
Bgensy [a
(6)
where C1 = (m) m)( - 5) constant that can be eliminated during the optimization of e. The KKT condition [ implies that there exists Lagrange multiplier & (dual solution) such that the set of solutions of the constrained optimization problem Eqn. (6) are equivalent to those of the following relaxed objective:
max (1-A) (#o) + Wou [a*] s.1. Bampsows
KLL (To.na (-]
S 8.
(7)
If X < 1, the optimization problem is equivalent to
max 0
LEoWn (Te) + wlE [a"] s.t. Bawpenges KKL (To (-1s)][to())) s))] s 8,
(8)
where w == E/(1 - A). The optimization problem can further be transformed as
max Σ, Progea (s) Σ (a ()) (s, ) + w.ggnnn [a*]
(9)
E Ea-pao
[KIL (Fr (-1 (- s)) 8.
Due to the summation the new policy #o, optimization is still challenging to implement even after transformation. To resolve this issue, we follow the idea from [0.. Specifically, we use importance sampling and substitute with the summation of the old policy through the Monte Carlo method [L] Then, in order to reach our ultimate optimization function, we further substitute the trust region constraints with the elipped ratio operation proposed by the Proximal Policy Optimization algorithm (PPO) M. Therefore, if X < 1, could obtain the final objective to optimize 0 as
maxe E min to (af 88) Aro clip
to (af 8t) Momi (af st)
e. c.1+e) Akoue)) ) + W.wLMAR MASK
(10)
Femi (af si)
where M.MASK BognEo (af] is added to ensure the sparsity of critical steps.

---

If X > 1, the optimization problem is equivalent to
min Lia (#o) W.g [a*] s.t. Egmprent
KL (#o (-1 ||)s)lll( s))l S 6.
(11)
Similar the analysis above, if 1, the final objective for updating 0 becomes
to (af 8t) A Aro clip
No (af 8t) c, c,1+e) Asoass
maxo E min
+ W. W.L.MAS
MASK
(12)
Feaia (af $t)
Teal
(aj st)
Putting all these objectives together, we get the final objective for updating 0 as
maxe sgn(1 min
Ra (ef (t) Aeoi [a] sc) Angard clip
Fouia (of st) e, +
Ro (až se)
Asoold
MASK
(13)
S4 Our Training Algorithm
S4.1 Algorithm Detail
Optimizing 0. Recall that the final objective function of 0 is
Ro (af sx)
Fu (af xx) Fooss (o7 a1) 1,,1+e) Araoia wLLLAAK
maxe agn(1
min
Anee clip
(14)
Jooia [o] sc)
where Ana.g A (st, a)) and LMASK(()) (0) Eag-h wio [af].
Note that the output of StateMask of is sampled from categorical distribution over all actions (i.e., 0 or 1). However, as stated in [ the categorical variable af is not capable of backpropagating through samples. Note that Bar-Ro [af] = Eurppt [Eeg~ao((ks) [aq]] = Eu-pp, [Pr(at = 1)] and Pr(a, = 1) is
differentiable. Therefore, we replace Bar=Ro [aq] with Leg,~pe [Pr(a = 1)] in practice.
When estimating Aroaus we adopt the same estimation formula as РРО
A ABo.a de + (7{)6 + ................................ (-5)
(15)
where = +V (st+1) V (st)
Therefore, during the training process, in addition to updating the actor by solving 0 in Equ. also have to update the state-value function V(s) by the Temporal-Differenee better estimating the advantage. Optimizing X. And recall that the objective function of is
14 we
Diiffeeencee (TD) method 13 for
min À (2n(T) 17((00a)) 5 Lifeor (f0)) X20
(16)
where Länge (#a) = E. (s) E.r to (a As) ond (s,
In practice, (TFeea) and (7( can be approximated via Monte Carlo method 77. nT(F) can be estimated in advance, i.e., running the game with the agent's policy स 500 times and calculating the average expected discounted reward as follows
(( Exg .o..
Σ 'r (s:)
(17)
((TTaza)) can be estimated using a similar formula in parallel with the collecting trajectories of old policy Teai while solving 0 with our training algorithm.
Nevertheless, the optimization objective remains challenging to solve because of the sunmation in La (it) involving the new policy * To address this issue, we utilize importance sampling and substitute the summation over the new policy To with sunmation over the old policy teme which can be computed using the Monte Carlo method M Mathematically, Lag (ft) can be estimated by
(a(a Hoals (a= Ato (s, a²), which has already been computed in Eqn. (14) when updating 6.

---

Algorithm 1 The training algorithm of mask net.
Input:
Target agent's policy म
Output: Mask net policy お
Initialization: Initialize the weights for the mask net म. Lagrange multiplier À = Estimate the () in advance by Equ. (17
for iteration 1.2,... do
50 D 4-D 0
for t = 1.,.... do
of < sample from Tea (-1s)
0g < sample from #-|s))
Compute actual taken action a f- de + of
Obtain current step reward rt, next state s' from the environment given action a
D D U 8')
--8 8
end for
Compute estimates for Asc.. (st. of) using samples in D by Equ. 15
Solve using the trajectory samples in DD by Eqn. (14) and update oold <- 0
Estimate the v7fT) using the trajectory samples in D by Equ. [1
Solve X by Eqn. (66 and update Andd x- end for
Extensive-form Games, Regarding extensive-form games, we train a state mask for the target agent while keeping all the agents' policies fixed. In this way, we could transform the extensive-form games into single-player normal-form games (i.e., treat all of the agents' actions part of the environment). However, the game may not end with the target agent, and the target agent will not receive the game reward in this case. To address this problem, we automatically overwrite the target agent's final reward with the final game result.
Based on what we have discussed above, we present the full algorithm in Algorithm
S4.2 Implementation and Hyperparameters
We implement our training algorithm using PyTorch g and additionally discuss the hyperparameter setting of our training algorithm. The common hyperparameter setting is as follows. Regarding the clipping parameter in Eqn. (1]. set it 0.2. As for the advantage estimation in Eqn. (1),, set y 0.99 and 0.95. Moreover, provide the game-specific hyperparameter choices in Table 1 Regarding the policy network of StateMask, we utilize a widely used network structure, commonly employed to train Deep Reinforcement Learning (DRL) agents for each game. Specifically, we use Convolutional Neural Network (CNN) for the Pong game and Residual Network (ResNet) for the Connect game since these two games' input observation contains spatial information and are relatively complicated. For other games, we use either Long Short-Term Memory (LSTM) or Multi-Layer Perceptron (MLP) network, which depends on the input type of the observation and the game's complexity. We use two separate networks with the same backbone architecture for the policy and value nelworks. The outpul dimension of the policy net work and value net are 2 and 1, respectively. As for the learning rate setting, we refer to other work of training a reinforcement learning agent in these games [ (2.. Morcover, our mothod introduces an extra hyperparameter - w (To encourage the mask network to blind the target agent at some states, we add sparsity constraint on the number of mask actions). We search the w in (0, 1e-5, 164,, 1o-3, 1e-2) and report the sensitivity experiment results in Supplement S5.3

---

Table 1: Training hyperparameter settings of our method. The numbers the bracket after "CNN" represent the number of kernels in each layer. The numbers in the bracket after "ML.P" are the hidden dimensions. 512 10 represents 10-layer MLP with hidden size 512.
Games
w
Learning rate
Optimizer
StateNask backbone
Batch Size
Num. Training Steps
Pong
Ie-4
le-5
Adam
CN, 32)
32
2e-4
You-Shall-Not-Paa
0
3e-4
Adam
MLP( 256)
2048
2e-7 7
Kick- And-Defend
0
3e-4
Adam
[512 256)
2048
2e-7 7
CartPole
0
1e-3
Adam
MLP(256, 256)
8e-2
Pendulum
0
le-3
Adam
(25, 256)
Se-2
StarCraft II
0
1e-6
Adam
MLP((18,,18,
32
8e-7
Connect 4
lo-3
lc-3
Adam
ResNet-122
1024
4o 5
Tic-Tac-Toe
lo-5
1c-2
Adam
MLP((18, 128)
128
2c
Breakthrough
1o-3
le-3
Adam
MLP(512 10)
1024
2c
DouDizhu
le-3
3e-4
Adam
MLP x 10)
42
5e-5 5
S5
Details of Evaluation
S5.1 Other Implementation Details
Baseline Implementations. Regarding baseline approaches, we use the code released by the authors or implement our own version if the authors don't release the code. Specifically, as for EDGE, we use their released open-sourced code from https: thub. com/Henrygwbbedeeeeeeeooo edge For the Value-max and the LazyMDP method, we assume that we have access the target agent's value function and implement them based on the description of the original method"
Fidelity Tests and Metrics. As mentioned in Section 4.2, adopt the experiment design in EDGE 3 to access the accuracy of the explanation. Specifically, the fidelity test first gathers N (i.e., N 500 in our setting) trajectories. Each explanation method generates the importance score of each state accordingly without altering the agent's action. For cach trajectory, we identify and record the length of the longest continuous sequence within the top-K most critical time steps. Following this, we replay the sequence of actions from the original trajectory up to the start of this identified sequence. From this point onwards, we replace the original actions with randomly selected actions for the next steps. Once the action substitution is completed, we present the newly reached state to the target agent at the end of the selected time steps. From this point forward, the agent operates according to the actions suggested by its policy network until the conclusion of the game. The reward difference before and after action replacement. is denoted as d. The fidelity score is defined as log(ps) - log(pt) where Pa = |d/dmaaz is the absolute reward difference divided by the maximum possible reward difference and Pt = I/L. is the length of the longest sequence normalized by the original trajectory length L. Note that in the original definition of the fidelity score B], lower score implies a higher level of fidelity. However, for ease of presentation, we have introduced negative sign to the fidelity score, such that a higher score indicates superior fidelity.
S5.2 Experiment Setup
Game Introduction and Target Agents. To assist in determining whether our explanation aligns with human perception, we briefly introduce cach game below.
Pong is two-dimensional sports game that simulates table tennis as Figure [ shows. The RL agent controls the right paddle while decent AI from the Atari game environment controls the left paddle. The RL agent observes itself, the opponent, and the position of the ball before deciding whether to move the paddle up, down, or take no action. each round, the RL agent wins the game and collects a +1 reward if the opponent fails return the ball; the agent loses the game and collects 0 reward if it misses the ball. To facilitate the evaluation, we terminate the game the end of each round.
For the Value-max, since it relatively easy to implement, we only need extract the state value from the value function. As to the LazyMDP, we contacted the author and implemented the explainer based the author's response.

---

Big NO Lank
Figure 1: Visualization the identified critical time steps in Pong game. The DRL agent controls the yellow paddle and a non-RL ruse-based opponent controls the white paddle.
Figure 2: Visualization of the identified critical steps in the Connect 4 game. The black player is our explanation target player. Note that the states where the blue player takes actions are marked with a gray bar.
You-Shall-Not-Pase is two-party competitive game Figure Shoows. Two humanoid robots face each other at the beginning of the game. The red robot then starts rumning towards the finish line (indicated by the red line), while the blue robot is tasked with preventing the runner from achieving its goal. In our evaluation, the red robot wins the game if it can get across the finish line. Otherwise, the blue blocker wins. The red robot gets the reward +1 once it wins the game and when it loses the game
Tacck-AndDeefone is two-party competitive game as Figure shows. Two humanoid robots compete in a soccer penalty shootout. The placements of kicker, defender, and ball are all chosen al random. the ball falls between the goalposts, the kicker wins; otherwise, the defender wins. The reward is +1 for the kicker win the game and when the kicker loscs.
CartPole is one of OpenAI's open-source environments as Figure shows. The "cartpolc" agent is a reverse pendulum in which a "cart" allempts to vertically balance a "pole" with a slight angle adjustment. The only forces that can be used are represented by +1 and -1, corresponding to left and right movement, respectively. The episode ends if the cart moves more than 2.4 units away from the center or if the angle deviates from vertical by 15 degrees. For each timestamp that the episode hasn't ended, the reward is increased by one. The maximum reward available is 500.
Pendulum simulates the inverted pendulum swingup problem as Figure g shows. The pendulum starts random position, and the goal to swing up and keep upright. The pendulum angle is represented by 0, which starts at random from - to æ. The default reward function is based on the angle 0 and the agent's action a. To maximize rewards, the agent must keep the angle 0 al zero, rotate at the slowest possible speed, and exert the least amount of effort.
StarCraft II popular real-time strategy game as Figure 10 shows. The player controls an army defeat the opponent's base by collecting resources and building units. The player's observation is the minimap and the screen. The minimap shows the entire map, while the screen shows the player's view. The player can take actions such selecting units, and issuing commands to the units. The player wins if the opponent's base is destroyed, and the game ends if the player's 's base is destroyed. The winning, and losing outcomes of the player correspond to +1 and -1 rewards, respectively.
Connect 4 uses a grid as Figure shows. Players take turns dropping tokens into the grid. The tokens land straight down, taking up the lowest available space in the column. The player wins if any four of the lokens are in row horizontally, vertically, diagonally. The game could also end in a draw if the grid is full while no player wins. The winning, draw, and losing outcomes of the player correspond to +1, O, and -1 rewards, respectivcly.
Tic- Tic-Tac-Toe uses a 3 3 grid Figure 11 shows. Two players iteratively place symbol available position. The player wins if any three of the symbols are in a row horizontally, vertically,

---

Table 2: Game information with the target agents to explain. "Discrete(X)" refers to X discrete actions
available to choose from. "Vector(X)" refers to X-dimension continuous action space.
Games
Observation shape
Action space
Reward range
Targel agent
Pong
(84,84)
Discrete(6)
0,1
Pong agent
You-Shall-Not-Pass
(380)
(17)
-1,1
Runner
Kick-and-Defend
(380)
Vector(11 17)
-1,1
Kicker
CartPole
(4)
Discrete(2)
|0,500]
CartPole agent
Pendulum
(3)
(1c
|-2000,0
Pendulum agent
StarCraft II
(919)
Discrete (165)
-1,1
StarCraft II bot
Connect 4
(3.6.7)
Discrete(7)
-1,0,1
First player
Tie-Tac-Tooe
(3,3,3)
Discrete(9)
-1,0,1
First player
Breakthrough
(8,8)
Discrete (3)
-1,1
First player
DouDizhu
(23, 54)
Discrete( Undertermined)
0,1
Landlord
diagonally. The game could also end in a draw if the grid is full while no player wins. The winning, draw, and losing outcomes of the player correspond to +1, 0, and -1 rewards, respectively.
Breakthrough uses a 8 x 8 grid as Figure 12 shows. Two players iteratively move one picce. If the target square is unoccupied, a piece may advance one space either straight forward or diagonally. Only if the square containing the opponent's piece is one step diagonally forward may a piece move into and replace it. The winner is the first player to reach the opponent's home row. A player loses if all of his pieces are taken. A draw is impossible in the Breakthrough game. The winning and losing outcomes of the player correspond to -1 and -1 rewards, respectively.
DouDizhu is a paroeeppryyyao card game as Figure 1 shows. Players begin the game by putting in bids for the "landlord". The other two players will act as "peasants". Being the first. player to out of cards is the game's goal. By removing all of their cards first, the landlord provails. If one of the peasants removes all of their cards first, the peasants win. The winning and losing outcomes of the landlord correspond to +1 and -1 rewards, respectively.
We also list some basic information (e.g., observation shape, action space, and reward range) about these selected games in Table 2 Besides, we clarify the target agent to explain in Table 2 as well (i.e., except for OpenAl gym and the Atari games, all the other games are multi-agent games, and we intend to one specific party). For the Pong game, we train an agent based on the PPO algorithm to achieve a relatively high reward (i.e., 0.906 in our setting). Regarding two MuJoCo games, we larget. explain the runner in You-Shall-Nol-Pass and the kicker in Kiek-And-Defend. The DRL models of the target agents in the two MuJoCo games are downloaded from https:// github. com/ patnnaa/mmltaaantttooom and Pendulum games, we adopt well-trained agents from the stable-baselines, model zoo: https Rw/rr//aabbaaeeeee333333 zoo| With respect to Comnect 4, Breakthrough, and Tic-Tac- Toe, we explain the first player and utilize example codes in OpenSpiel for training RL agents based on the AlphaZero algorithm [1]] The maximum simulation times for the Monte Carlo Tree Search in Connect 4, Breakthrough, and Tic-Tac-Toe are 300, 300, and 20, respectively. Concerning DouDizhu, we target at explaining the landlord and adopt the models trained by DouZero [66, which can be downloaded from https: //g thub. wwt. Setam//waa/DooZrrr When it comes to StarCraft 11, use the built-in bot with difficulty A from PySC2: https:///ithu:t com http//dppmmnd/yscc2222
As for OpenAl's CartPole
/github.
Computational Resources. In our experiment, we use a server with 2 AMD EPYC 7702 64-Core CPU Proccssors and 4 NVIDIA RTX A6000 GPUs to train and evaluate our method.
S5.3 Additional Evaluation Results
First, we demonstrate that our design could achieve the monotonicity property by showing the training curves of all games. Second, we provide the agent's performance and StateMask's performance on all games and show the relative value errors across all games. Last, we compare the fidelity scores of our

---

Table 3: Results of reward gap in all games. Each experiment contains 500 runs, We repeat all experiments three times and report the mean and standard deviation of the discounted total rewards in the table below.
Games
Agent Performance
Relative Errons (%)
Gomes
Agent Performance
Relative Errons (%)
Pong
0.20 (0.00)
0.89 (0.03)
StarCraft
0.11 (0.00)
0.40 (0.01)
You-Shail-Not-Pars
-0.13 (0.01)
2.58 (0.12)
Connect
0.91 (0:01)
2.09 (0.12)
Kick-And-
0.13 (0.01)
7.78 (0.20)
Tie- Tic-Te--To Tac-Toe
0.76 (0.02)
6.43 (0.08)
CartPole
09.34 (0.00)
0.00 (0.00)
DouDizhu
0.40 (0.02)
14.98 (0.21)
Pendulum
-1 144.16 (10.26)
5.56 (0.33)
Breakthrough
0.98 (0.00)
11.02 (0.16)
Cartpole
Pendulum
You Shall Not Pass
Kick And Defend
(p( R(
2004
Time steps (lc)
Time steps (le)
Time steps (Ie')
Time stops (1e')
Time steps (1v
StarCraft
Connect
Tic: Tac Toe
Breakthrough
DouDizhu
02
19(5 ) [(5,)]
Time steps (1c)
Time steps (1e)
Time steps (1e)
>
Time steps (1e
Time steps
Figure 3: The absolute policy value difference between the masked agent trained by StateMask and the target agent. Note that the darker solid lines represent the average absolute policy value difference across 3 random seeds. The lighter shadow indicates the corresponding variations between the maximal and minimal absolute policy value difference.
method with three baseline methods on additional five games, i.e., Kiek-And-Defend in MuJoCo games, OpenAl's Cartpole and Pendulum, Breakthrough and Tic-Tac- in Openspiel.
Monotonicity Property. Figure shows the trend of [n(()) (( (i.e., the absolute policy value difference) during the training processes of all games. As we can observe, the absolute policy value difference monotonically decreases until it converges when the training step increases. Combined with Section 3, it demonstrates that our design could guarantee the monotonicity property both theoretically and empirically.
Agent Performance. To evaluate the agent's performance before and after applying the explanation network, we run 500 tests and record the mean and standard deviation of the discounted total reward in Table 3 We can observe that the agent before and after blinding achieves similar performance when running 500 episode tests. confirms that our proposed method could achieve the objective in Section 3 (i.e,, perturbing the actions in the identified non-important time steps will not impact the expected total reward of the target agent).
Fidelity. Section 4 shows the fidelity score comparison between our method and three bascline methods on Pong, You-Shall-Nou-Pass. Connect 4, DouDizhu, and StarCraft II. Here, we additionally evaluate the fidelity of our method and three bascline mothods on the remaining five games. Figure 4 shows the fidelity score comparison across all four methods. Combined with Figure in Section 4, StateMask consistently achieves the highest fidelity score among all games, implying our method's high fidelity. Sensitivity. To show the insensitivity of our proposed method StateMask, we vary the hyperparameter from {0, 1c-5, 1c-4, 1c-3, 1c-2} and train the state mask accordingly. Figure 5 (a) and (b) shows the fidelity scores and mask ratios of our method in Pong and You-Shall-Nol-Pass games under different. settings. We can observe that through the hyperparameter w differs, the fidelity scores do not vary too much. It confirms that our mothod is insensitive to hyperparamoter choice. Morcover, adding sparsity raistraint can increase the mask ratio which can help filter out those non-important time steps.

---

23
1. Foeelry
1.
Top10 Top20 Top30 Top40
03
Top10
Top20
1op30
Top40
Top10
Top20
Top3o
Top40
Tep10 Top20 Top30 Top40
Tpp1 Top20 Top30
Top40
OpenAl Curpole
OpenAl Roxlilan
MuloCo Kicker-Aal Delonds
OpenSyid Ti-Ta:Toe
OyeaSpid Brecklloogg
Figure 4: Fidelity scores for interpretations generated by three baseline methods and proposed explanation method. Note that the definition of the fidelity score can be found in Section 4.2. A higher score implies higher fidelity.
Pong
Pong
VALUE
Fidelity Soore
Time Elapsed (8)
10'
LAZY
Mask Ratio
OURS
20'
104
160
Pong
You
Break
W
(b)
(c)
Figure 5: Sensitivity and efficiency results. In the left and middle figures, show the sensitivity results by varying the hyperparameter in Pong and You-Shall-Not-Pars games. In the right figure, we report the mean and standard deviation of the explanation time per trajectory in four games: Pong, You-Shall-Not-Pass, Tic-Tac-Toc Toe, and Breakthrough.
Efficioncy. To demonstrate our mothod's efficiency, we also record the average explanation time per trajectory in two normal-form games and two extensive-form games. Figure 3 (c) shows the explanation time of each method four games. Here, EDGE global explanation method whose explanation time negligible while the other three are local explanation methods. Regarding local explanation methods, our proposed method StateMask has similar time efficiency with the Value-max method whereas LazyMDP is much slower. The reason is that the target agents in these four games cannot provide the Q value function directly, and thus we need extra computation time to approximate Q values for deriving explanations in LazyMDP.
S6 Additional Visualization Results
You-Shall-Not-Pass. In Figure G we visualize one episode of the You-Shall-Noo-Pass game. Our
explainer pinpoints the time steps when the red robot escapes the blue robot the most important ones. The reason is that the DRL agent should be careful not to get knocked down by the opponent this moment. Otherwise, the agent will lose the balance and fall to the ground, and eventually lose the game. When the two robots are far away from each other or the blue robot is already down on the ground, the states are less important. Kick-And-Defenn
We provide one case study for the Kick-And Aad-Defend game Figure 7 Our explanation method explains that the most important time steps this game are when the blue robot is shooting. The agent should choose the proper direction and power to kick the ball the goal these critical time steps. Otherwise, it will be difficult for the agent score. When the ball is flying toward the goal, the actions of the DRL agent have impact on the final result. Therefore, these time steps are less important.
CartPole. For the CartPole game, we show an example in Figure 8 When the angle of the pole is large (no matter if it is clockwise or counterclockwise), the cart should accelerate toward the center to correct the angle, otherwise the pole will fall. StateMask treats these time steps as critical ones. When

---

Figure 6: Visualization of the identified critical time steps in You-Shall-Not-I game. The DRL agent controls the red humanoid robot and the opponent controls the blue onc. While the blue humanoid bot is trying to block the red from reaching the goal, the red one is trying to gel across the red line.
Figure 7: Visualization of the identified critical time steps in Kick-And- Aae----nd-Deeddd game. The DRL agent controls the blue humanoid robot to kick the ball and the opponent controls the red one to defend the goal.
Hightn Low
Figure 8: Visualization of the identified critical time steps Cartpole game. The DRL agent controls the cart to move left or right to keep the pole stand (within certain angle) as long as possible.
1
Figure 9: Visualization of the identified critical time steps in Pendulum game. The DRL agent controls the torque applied on the free end of the pendulum to swing it into an upright position, with its center of gravity right above the fixed point.
Figure 10: Visualization of the identified critical steps in StarCraft II. The green zerg player is the larget agent and the red one is the opponent. The big circles represent the buildings while the small circles represent the units. In the first three steps, the agent collects the minerals and vespene gas. In the next three steps in this episode, the agent builds some structures. In the last two steps, the small circles representing the zerglings and roaches of the target agent are attacking the enemy base.
the angle the pole small, the cart needs to stabilize the angle, and StateMask treats these time steps as less important ones. This is reasonable since the pole is still within a safe angle range at these time steps.
Pendulum. In the Pendulum game, show random position and the agent tries swing it into upright position. StateMask pinpoints the time steps when the pendulum is close to the upright position as the most important ones. The agent should carefully apply the lorque at these lime sleps lo make the pendulum upright with zero velocity. When the angle the pendulum large, StateMask treats these time steps as non-critical ones.
study in Figure g The pendulum starts in
StarCraft II. In the StarCraft II game, the DRL agent controls Zerg player defcat the opponent Zerg player by collecting resources and building Zerg army. Figure 10 shows one roach rush game where the DRT agent wins. StateMask identifies the first few steps critical ones, including producing the drones and overlords, well building the spawning pool, extractors, and roach warren. These steps are reasonable and will also be deemed as important by human analysis since the succossful execution of the build orders in the early stage is one key to winning the game. Also, StateMask

---

O
x
O
X
O
X
TO0T 199111
O
O
O
O
O
X
X
X
X
X
X
X
X
Figure 11: Visualization of the identified critical time steps in the Tic- Teo-acc-Te game. The DRL agent places the X mark and the opponent places the O alternately in one of the nine spaces in the grid. The states where the mark is placed are colored in gray.
4
Visualization of the identified critical time steps in Breakthrough game. The DRL agent plays as the black player and the opponent plays as the white one. They play alternatively, with each player moving one piece per turn.
Figure 12:
treats the last few steps non-important steps. This because the opponent's army has already been crashed by the roach rush, and without the army's protection, it is just matter of time before the agent destroys the opponent's base to win the game. Because of the automatic attack mechanism in Starcraft II, even if the DRL agent does not take any action, the roaches will destroy all the opponent's buildings event ually.
Tic- Tac-Tac-Tee Toee In the Tic-Tac- Toe game, we show one episode to analyze the explanation of our method in Figure [1] Our explainer only selects the last time step as the critical one since this move seals a win for the DRL agent. the agent chooses the wrong move this time step, the opponent will win the game.
hrough Finally, visualize episode of the Breakthrough game in Figure 12 Our explainer pinpoints the last four time steps in this episode as the most important. By analyzing the strategy of the agent, we could find that the agent sacrifices one piece to make sure another piece can reach the opponent's 's side in the last four time steps. If the agent docs not take such a strategy, the game will last for a longer time and the agent may lose the game eventually.
DouDizhu. In DouDizhu, a landlord (our target agent) competes with two cooperative peasants be the first to out of cards. Figure 14 shows one critical step identified by StateMask. At this step, the landlord chooses to play eight cards, i.e., Quad with Pairs. This move enables the landlord to play most low-rank cards and win the right to play first in the next round (no peasant has larger combinations than this one). If these cards are not played together, it is hard for the landlord to play them separately because the Quad and two Pairs are relatively small and cannot help the landlord to get the right to continue playing first.
We also give the winning trajectory for the DouDizhu game in Figure Our explainer pinpoints critical time steps in this episode. The first one is when the landlord plays 2. Since Red Joker in hand is the largest solo that beat any other solo, the landlord can ensure the right play cards in the next round. The second one is when the landlord plays Trio with Solo to empty his's hand. If the landlord plays Trio or Pair separately, the peasant will have a chance to win the game. Therefore, the two selected time steps are critical to the final result.

---

WINI
Figure 13: Visnalization of the identified critical time steps in the DouDizhu game. The DRL agent plays the role of the landlord and two cooperalive opponents play the role of the peasants. The landlord's goal is to win the game by emptying his hand, and vice versa. The states where the peasants take actions are marked with a gray bar.
S7 Additional Results of Attacks
this section, we provide the results of launching adversar- ial attacks under the guidance explanation when varying the attack ratio between 10% and 30% Table First, we observe that when the threshold increases, the victim agent's performance drops. Second, with the same threshold setting, the adversarial attack under the guidance of our explanation method is the most powerful one. These re- sults confirm the conclusion that our proposed explanation method has the highest fidelity.
JOKER
One critical step in the DouDizhu game: The target agent (land- lord) plays the Quad with Pains to ensure that peasants cannot beat this hand.
Figure 14:
S8 Additional Results of Policy Patching
Advantage of Our Patching Method. In this experi- ment, we run the patching method proposed in EDGE B on two MuJoCo games. At a high level, this method first replays the losing trajectory until the start of the critical steps and does random explorations at the critical steps order to find winning stralegy. The actions logether with the corresponding critical steps in the winning strategy are recorded. Then, it forms a look-up table with these collected state-action pairs and uses the look-up table to remediate the original policy. We follow the sctting in EDGE and run 500 tests to oblain the larget agents' winning rate change before and after patching. Table shows the results of EDGE's patching method. Although the agents' performances have some improvements after utilizing the EDGE's patching method, the improvements are overall marginal. Together with the results in Table 1 in Section and Table 0 we can safely conclude the superiority our proposed patching method over EDGE's patching method after retraining. The reason may be that EDGE uses hard rule (i.e., look-up table) patch the agents' policy, and the quality of the look-up table has a great influence on the patched agents' final performance. In contrast, our patching method does not require fixed look-up table to store the correct actions corresponding to critical states while encouraging the DRL agent to do more exploration at critical time steps and update the neural network parameters accordingly.
Patching Results on Other Games. Table provides the agent performance improvement results after retraining under the guidance of our explanation method and the three baselines Pendulum, Kick-And-Defenn and Tic-Tac-Tie Toc. Using our proposed retraining method under the guidance of explanation, the performance of the reinforcement learning agent generally increases. More importantly, retraining based explanation achieves the best performance.

---

Table 4: Winning rate drop varying attack ratios in all games. Each experiment contains 500 runs. We repeat all experiments three times and report the average rewards with the corresponding standard deviations in the table below.
Games
Attack ratio
EDGE
Value-max
LazyMDP
Ours
10%
-63.04 (3.44)
-0.40 (0.40)
-36.17 (1.15)
-84.90
(1.40)
Pong
20%
-65.73 (0.81)
-7.06 (8.81)
-57.00 (0.92)
-85.53 (0.75)
30%
-71.07 (1.36)
-70.13 (0.64)
-62.63 (1.63)
-86.90 (0.76)
10%
-2.64 (0.60)
-14.80 (3.20)
-28.87 (1.21)
-35.93 (1.81)
You-Shar Shall-Not- Pass
20%
-6.87 (0.61)
-55.13 (4.27)
-50.63 (0.80)
-61.67 (0.12)
30%
-11.68 (1.70)
-61.93 (0.12)
-61.87 (0.23)
-61.93 (0.12)
10%
-10.46 (0.97)
-24.57 (1.86)
-28.33 (1.76)
-28.37 (1.10)
Kick-And-Deffen
20%
-28.51 (0.72)
-61.23 (1.63)
-60.43
(1.33)
-68.63 (1.17)
30%
-37.25 (1.03)
-74.30 (0.40)
-73.57 (1.53)
-74.10 (0.80)
10%
-78.98 (4.61)
-4.85 (6.64)
-0.50 (0.51)
-17.881 (13.82)
CartPole
20%
-153.21 (11.87)
-20.96 (6.21)
-13.92 (6.42)
-198.06 (24.21)
30%
-237.03 (5.30)
-72.99 (7.98)
-56.71 (2.79)
-332.05 (16.33)
10%
-58.84 (2.79)
255.80
(15.92)
-135.23 (1.51)
-570.77 (1.13)
Pendulum
20%
-129.37 (1.86)
432.15
(43.03)
-241.20
(0.95)
-707.69 (0.74)
30%
-201.39 (3.31)
-563.63
(15.16)
-372.88
(4.21)
-796.07 (12.07)
10%
-70.00 (14.18)
-77.33 (5.69)
-72.00
(13.53)
-83.00 (7.21)
StarCraft II
20%
-93.33 (4.73)
-93.00 (3.61)
-93.00 (4.58)
-94.00 (3.00)
30%
-97.00 (0.00)
-97.00 (0.00)
-97.00 (0.00)
-97.00 (0.00)
10%
-33.33 (1.56)
-26.30 (1.44)
-22.57
(1.40)
-35.37 (3.00)
Connect 4
20%
-56.73 (2.30)
-67.57 (1.57)
-53.83 (0.81)
-71.90 (1.44)
30%
-76.23 (1.94)
-82.70 (0.80)
-69.83 (1.17)
-85.50 (0.72)
10%
-0.30 (0.23)
-8.07 (0.35)
-4.40 (0.25)
-17.24 (0.40)
Tic-Tac-Toe
20%
-6.72 (0.33)
-18.20 (0.17)
-7.60 (0.12)
-29.90 (0.45)
30%
-19.50 (0.36)
-25.12 (0.13)
-13.90 (0.41)
-42.85 (0.17)
10%
-6.53 (0.50)
-18.00 (1.00)
-18.90 (0.36)
-88.73 (1.62)
Breakthrough
20%
-13.67 (0.83)
-30.93 (1.01)
-26.87 (1.03)
-95.87 (1.21)
30%
-26.33 (1.53)
-2.67 (2.89)
-36.00 (2.65)
-98.33 (1.15)
10%
-0.25 (1.71)
-18.60 (0.31)
-25.20 (0.76)
-29.94 (0.64)
DouDizhu
20%
-10.45 (1.68)
-24.40 (0.83)
-32.27 (0.69)
-32.87 (0.20)
30%
-11.29 (1.40)
-28.07 (1.22)
-32.67 (0.20)
-34.54 (0.31)
S9 Evaluation on Explaining Sub-optimal Agents
As mentioned in Section 4.3, to verify the effectiveness of StateMask in terms of explaining sub-optimal agents, we conducted additional experiments on sub-optimal agents in Pong, Kick-And- And--endde,,,nd,, Connect 4, and Breakthrough games. These games include both normal-form form games (i.e., Pong and Kick- A And-Defend) and extensive-form games (i.e., Connect and Breakthrough). Regarding the selection of sub-optimal agents in these four games, we utilized policies from the halfway stage of training sub-optimal agents in Pong, Connect 4, and Breakthrough. In the Kick-And-Defend game, we follow the setting in Supplement S5.2 and explain the defender. Note that the original zoo model (https; //gthhu.. com/ http://aaammmaaaaaaateeeooeeeeee /ull cww.grenttcommpttiiiinoo www.oo/tttee/mttee///eeoeoeemeee aaaeeerr/gggoooooo offered three different policies for the kicker. We held the defender constant and had each kicker play against the defender separately, recording their corresponding winning rales. The kicker that displayed the lowest winning rate was chosen the sub-optimal agent.
S9.1 Fidelity
In this experiment, we compare the fidelity scores of our method and baseline methods across the four selected games. As depicted in Figure 15 our method still achieves the highest fidelity score among the four games even though the target agent is sub-optimal. The reason is that StateMask is designed to preserve the original agent's performance before and after masking the target agent at
13

---

Table 5: Results of EDGE's policy patching method with different explanation methods. Each experiment contains 500 runs. We repeat all experiments three times and report the average reward increase with the correspouding standard deviation in the table below.
Application
Games Kick-And-Defend
EDGE
Value-mas max
LazyMDP
Ours +0.00 (0.00)
Target agent's ' wimming rate improvement after patching
+ +0.00 (0.00)
+0.00 (0.00)
+ +0.00 (0.00)
You-Shall-Not-Pass
+ 0.00 (0.00)
+0.75 (0.04)
+ 0.25 (0.01) +0.25 (0.02)
Table 6: Additional results of policy patching. Each experiment contains 500 runs. We repeat all experiments three times and report the average reward increase with the corresponding standard deviation in the table below.
Application
Games
EDGE
Value-max
LazyMDP
Ours
Kick- And--eenn-Dedd Defend
1.03 (0.12)
+0.70 (0.20)
+0.97 (0.06)
+1.37 (0.12)
Target agent's winning rate improvement after patching
Pendulum
0.00 (0.00)
+0.29 (0.03)
+2.65 (0.05)
+4.30 (0.40)
Tic- Voo-Tae-Te Toe
0.03 (0.06)
0.23 (0.06)
+0020 (0.10)
2.47 (0.12)
30
3.0
50-E
2.0
WALUe
2,5
OURS
2,0
14
1.0
05
1.0
0.
0.
0.5
-03
0.5
Top10
Top20
Tep30
Top40
00
Top10
Top20
Tep30
Top40
Top10
Top20
Tep30
Top40
Top10
Top20
Tep30
Top40
Pong
MulaCo KidkerAnd--D And-Defendrr
OpenSpiel
OpenSpidl Breakthnoog
Figure 15: Fidelity scores for interpretations generated by three baseline methods and proposed explanation method for four sub-optimal agents.
certain time steps. In instances where the target agent is sub-optimal, the important time steps are those that lead the target agent to win or lose (i.e., with a higher chance) in single episode. Altering the act ions in these time steps may cause a significant. variation in performance. Therefore, StateMask has the potential to recognize and distinguish these critical time steps by learning how to unmask them accurately.
S9.2 Visualizing the Explanation for Sub-optimal Agent Behaviors
Following Section 5, we also visualize the important time steps identified by StateMask in the selected four games to demonstrate StateMask could generate human-understandaule explanations even for sub-optimal agents.
Pong. Regarding the Pong game as Figure 16 shows, our method indicates that the most crucial time step is when the ball is flying towards the yellow paddle. Nevertheless, the yellow paddle moves upward too much when trying to catch the ball. If the yellow paddle takes a different action, e.g-, moves downward, the game outcome could be flipped.
Kick-And-DD And-Defend In the Kick-And-Defendd game, as our explanation method demonstrates Figure 17 the most critical time step identified when the kicker is in contact with the ball. As shown in the figure, the kicker's positioning and approach to the ball lead to sub-optimal angle for the shot. Consequently, this results in the ball being intercepted easily by the defender, who is well-positioned to block the trajectory. If the kicker were to adjust. their angle or approach, the outcome of the game could potentially change, as the ball might bypass the defender, increasing the likelihood of scoring a goal.
Connect4. In the Counect 4 game as Figure 18 shows, StateMask pinpoints the penultimate step the most crucial step of the losing trajectory. Once the black player determines to drop his token into the seventh column in the sccond-to-the-lar step, we could observe that no matter which column the black player chooses in the last step, the game outcome will always be a failure. However, if the black player chooses the second column in the second-to-ther tose-lastt step, he would have chance to win,
14

---

Figure 16: Visualization of the identified critical time steps in the Pong game. The yellow paddle is controlled by the agent we aim to explain.
High AO LO
Figure 17: Visualization of the identified critical time steps in the Kick-And-Defent game. The kicker(blue agent) the agent we target to explain.
Figure 18: Visualization of the identified critical time steps in Connect 4 game. The black player is the sub-optimal agent we aim to explain.
Highho Low.
Figure 19: Visualization of the identified critical time steps in the Breakthrough game. The sub-optimal agent plays the black player and the opponent plays as the white one.
i.e., connecting four tokens between the first column and the fourth column cither horizontally diagonally. Therefore, the second-to-the-latt step is the most crucial step in this losing trajectory. Breakthrough. Regarding the Breakthrough game as Figure 19 shows, our method identifies the fitth-to-the-last and the fourth-to-the-las step as the most critical ones. Note that only if the square containing the opponent's piece is one step diagonally forward may a piece move into and replace it. Apparently, in the lith--o-theelast step, the black player fails replace the left white piece the fourth row, directly causing the game losc. Additionally, in the fourth-to-the-lst step, the black player's action makes him lose the most advanced piece. The identified two steps are thus the key steps for the black player to lose the game.
S9.3 Patching Results of Sub-optimal Agents
In this experiment, we follow the retraining method proposed in Section 5 and patch the sub-optimal agents in the selected four games. Table 7 shows the results of the sub-optimal agent performance improvement under the guidance of both StateMask and the baseline modhods. After retraining, the agents' performance generally has a large improvement due to these agents being far from optimal. More importantly, retraining based on our explanation achieves the best performance, which indicates the superiority of StateMask.

---

Table 7: Patching results of sub-optimal agents in four games. Each experiment contains 500 runs. We repeat all experiments three times and report the average reward increase with the corresponding standard deviation in the table below.
Application
Games
EDGE
Value-max
LDz
Ours
Pong
+5.80 (0.12)
+5.40 (0.03)
12.00 (1.24)
-
13.60 (1.22)
Target agent's 's winning rale improvement after patching
Deffenn
47.42(0.40)
(.44(()224)
+4.78(0..2) 12)
+10.26(0.62)
Connect
+24.80 (1.68)
+35.80 (1.84)
36.20 (2.20)
+37.60(1.66)
Breakthrough
68.40 (3.68)
+ +65.80 (2.77)
68.80 (3.26) 69.00 (3.07)
Table 8: Results of relative errors in four selected games. Each experiment contains 500 runs. We repeat all experiments three times and report the mean and slandard deviation of the discounted total rewards in the table below.
Games
Pong
Kick-And-Defin
Connect 4
Breakthrough
Sub-optimal agent performance
0.18 (0.00)
0.09 (0.00)
0.52 (0.01)
-0.39 (0.01)
StateMask
2.57 (0.03)
5.34 (0.54)
3.05 (0.34)
12.89 (1.86)
Relative error
PPOMask
3.86 (0.06)
7.53 (0.65)
5.77 (0.42)
28.57 (2.44)
3.0
2.0
StateMank PROM
2.5
1.5
2.0
3.5
1.0
1.0
0.5
05
0.0
Top10
Top20
Top30
Top40
0.0
Top10
Top20
Top30
Top40
0.0
Top10
Top20
Top30
Top40
Top10
Top20
Top30
Top40
Pong
MulaCo Kirker-Anl-Defenddr
OpenSpiel
Connect
OpenSpiel BreakBhroogh
Figure 20: Fidelity comparisons between StateMask and PPOMask across four selected games when the agent is sub-optimal.
S10 Evaluation on An Alternative Design
As mentioned in Section 4.3, examine alternative design of StateMask named PPOMask that trains mask norw by maximizing the masked agent's 's expected total reward (i.e., J(0) max, r)(a)) This is done by directly using the PPO algorithm to train the mask network. To avoid the trivial solution, same as Section 3.2, we also add an additional loss item I. L.MA to the PPO loss. We evaluate PPOMask in Pong, Kidk-And-Deffnd, Connect 4, and Breakthrough game.
Relative Error. Here, we report the relative errors of PPOMask in comparison StateMask across these chosen four games. As depicted in Table S PPOMask generates higher relative error than StateMask when the target agent is sub-optimal, due to its inability to maintain the target agent's performance (i.e., PPOMask trains masked agent. that performs better than the original agent)..
Fidelity. We also test the fidelity of PPOMask and compare it with StateMask in these four games. According to Figure 20 PPOMask exhibits poor fidelity on sub-optimal agents. This can be attributed to the objective function PPOMask, which to maximize the expected total reward a mixed policy. Consequently, PPOMask might disregard the critical states that contribute the failure of the original agent and instead take random actions (i.e., mask operation) during those steps in order to secure a win under the perturbed policy. However, shown in [99 our method is able to identify the critical states that contribute to the failure of the sub-optimal agent and thus has higher fidelity.
S11 Potential Social Impact
Our work focuses on improving the trustworthiness of Deep Reinforcement Learning systems, which are increasingly being used in a wide range of applications, from gaming b to robotics S to healthcare [b]

---

One of the main challenges of DRL is the black-box nature of the models, which makes it difficult to understand how they arrive at their decisions. This lack of transparency create concerns about the reliability and safety of DRL systems.
To address this challenge, our work aims to automatically generate explanations for DRL systems, without requiring human supervision. By generating explanations automatically, we reduce the cost and effort required to provide explanations, making them more accessible and scalable for various applications. Moreover, it avoids potential biases that may arise from human-generated explanations, these may be influenced by subjective factors Our approach involves developing models that can analyze the behavior of DRL systems and generale explanations based on that analysis. These explanations provide insights into how the DRL system works, what factors influence its decisions, and how can be improved. By providing this information, our work contributes to enhancing the transparency and interpretability of DRL systems, which can help users evaluate the reliability of these systems.
incomplete understanding of the system.
When it comes the potential negative societal impacts of our work, the use of StateMask for enhancing RL agents may lead to the magnification of biased and unfair RL agents. Nevertheless, method also has the capability of understanding the decision making process of biased and unfair RL agents, which enables the implementation of strategics to diminish their harmful effects. Therefore, StateMask is crucial for understanding and improving RL agents.
17

---

References
[1] Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convez optimization. Cambridge universily press, 2004.
[2] Yang Guan, Minghuan Liu, Weijun Hong, Weinan Zhang, Fei Fang, Guangjun Zeug, and Yue Lin. Perfectdow: Dominating doudizhu with perfect information distillation. In Proc. of NeurIPS, 2022.
[3] Wenbo Guo, Xian Wu, Usmann Khan, and Xinyu Xing- Edge: Explaining deep reinforcement learning policies. In Proc. of NewrIPS, 2021.
[4] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In Proc. of ICLR, 2017.
[5] Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learning. In Proc. of AAAI, volume 31, 2017.
[6] Marc Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, et Openspiel: A framework for reinforcement learning in games. arXi preprint arXio:1988.00 09453, 2019.
[7] Nicholas Metropolis and Stanislaw Ulam. The monte carlo method. Journal of the American statistical association, 44(247):335 341, 1949.
[8] Farzad Niroui, Kaicheng Zhang, Zendai Kashino, and Goldie Nejat. Deep reinforcement learning robot for search and rescue applications: Exploration in unknown cluttered environments. IEEE Robolies and Aulomation Lellers, 4(2):610- 617, 2019.
[9] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Proc. of NewrIPS, 2019.
[10] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proc. of ICML, 2015.
[11] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arxiv preprinl arXiv: 1777777700000 707.06337, 2017.
[12] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonogiou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140 --1444, 2018.
[13] Richard S Sutton. Learning to prodict by the methods of temporal differences. Machine learning, 3:9 44, 1988.
[14] Scbastian Thrun. Monte carlo pomdps. In Proc. of NeurIPS, 1999.
[15] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning healthcare: survey. ACM Computing Surveys (CSUR), 55(1):1 36, 2021.
[16] Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Ji Liu. Douzero: Mastering doudizhu with self play deep reinforcement learning. In Proc. of ICML, 2021.
18


Reviews:
[{"review_id": "bjjQMVCyQc", "reviewer": "Reviewer_ENqW", "summary": "This paper aims to explain deep RL through identifying the critical states at which the action of policy significantly impacts the final reward performance. The main idea is to learn a state-mask, which is modeled as an additional policy, to determine whether to mask original action output by a random one and minimize the performance difference in the meantime. In practice, the authors adopt a trust-region trick to guarantee the monotonic decrease of performance difference and learn the objective in a PPO style. They further apply the explanation of critical states to do adversarial attack and error patch, which exhibits a better performance than baselines.", "questions": "- When using thm 1 to derive the objective function in eq.(5), do you remove the first condition in eq.(4)?\n- How do you choose the threshold $c$ in eq.(5) since there is a trade-off between better explanation and smaller performance difference?\n- How is the fidelity score in fig. 3 computed? The authors mention it is from [1] but in the original paper the lower score indicates the higher fidelity of explanation (the third last row in page 7).\n\n[1] Guo, Wenbo, et al. \"Edge: Explaining deep reinforcement learning policies.\" Advances in Neural Information Processing Systems 34 (2021): 12222-12236.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper is clearly written and the presentation in the evaluation part is good.\n- The empirical results and the selected examples show the effectiveness of the proposed method on key state explanation.\n- The generated explanation is easily compatible with the downstream tasks, e.g., adversarial attack and defense, which lays a foundation for future work.\n", "weaknesses": "- The paper is implicitly built upon a restrictive assumption that there are some specific single timesteps/states that contribute to the final reward significantly in every episode. However, there are other cases that it is a series of actions (may be consecutive or not) that mutually influences the total reward, which is more common in complex environments but hard to be captured by this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer ENqW for the constructive and insightful comments. Please see our response to each of your questions below.\n\n**1. The paper is implicitly built upon a restrictive assumption that there are some specific single timesteps/states that contribute to the final reward significantly in every episode. However, there are other cases that it is a series of actions (may be consecutive or not) that mutually influences the total reward, which is more common in complex environments but hard to be captured by this method.**\n\nThe reviewer questioned that our paper is based on the assumption about the significant influence of specific timesteps or states on the final reward. However, we would like to clarify that our method, StateMask, does not inherently depend on such an assumption. StateMask is designed to identify not only individual states but also multiple states that contribute towards the final reward. It is capable of tracing back to multiple actions that mutually influence the reward, regardless of consecutive or not.\n\nEvidence of this can be seen in Figure 5 of our paper. Our StateMask method identifies two important non-consecutive action series in the complex game Connect 4. The first critical action series marked by our method is the initial move. This move, as explained by [1], sets the foundation for future gameplay. Furthermore, StateMask highlights the last two time steps as significant contributors to the agent's victory. The second-to-last step stands out as pivotal, as the black player has a winning strategy no matter which column the blue player places their piece next. This showcases StateMask's effectiveness in capturing joint contributions of actions in complex environments. Thus, we believe our method sufficiently addresses the scenario posed by the reviewer and is versatile enough to handle complex environments where multiple action series can mutually affect the total reward.\n\n[1] A knowledge-based approach of connect-four. Journal of the International Computer Games Association, 1988.\n\n**2. When using Theorem 1 to derive the objective function in Eqn. (5), do you remove the first condition in Eqn. (4)?**\n\nNo, we do not remove the first condition in Eqn. (4) when using Theorem 1 to derive the objective function in Eqn. (5). In fact, optimizing Eqn. (5) is equivalent to optimizing Eqn. (4). We use the same trick to deal with the first condition in Eqn. (4) as TRPO [2] and transform it to maximize the local approximation $L\\_{\\tilde{\\pi}\\_{\\theta\\_{old}}}(\\widetilde{\\pi}\\_{\\theta})$\n with a trust region constraint. We recommend referring to Supplement S2, where we give a detailed analysis to clarify this point. \n\n[2] Trust region policy optimization. In Proc. of ICML, 2015.\n\n**3. How do you choose the threshold $c$ in Eqn. (5) since there is a trade-off between better explanation and smaller performance difference?**\n\nWe appreciate your inquiry about the threshold $c$ in Eqn. (5). While we do not explicitly choose the threshold $c$ in Eqn. (5), we transform the constraint $E\\_{a^e \\sim \\tilde{\\pi}\\_{\\theta}}[a^e] \\geq c$ to an additional loss item $L^{mask}$ with coefficient $w$ when optimizing StateMask. Please refer to Supp. S3 for the detailed derivation. Now, $w$ will be utilized to regulate the proportion of the mask. As stated in Supp. S4.2, we search the $w$ in {0, 1e-5, 1e-4, 1e-3, 1e-2} for each environment. Besides, we also study the sensitivity of $w$ in Supp. S5.3. We can observe that when the hyperparameter $w$ differs, the fidelity scores of our explanation method do not vary too much. This suggests that our model is robust to the choice of \"$w$\", and thus manages the trade-off effectively.\n\n**4. How is the fidelity score in Figure 3 computed? The authors mention it is from [1] but in the original paper the lower score indicates the higher fidelity of explanation (the third last row in page 7).**\n\nWe appreciate your query about the fidelity score presented in Figure 3. We apologize if there was any confusion due to the space limit. In our work, we use a similar but slightly modified approach from [3] to calculate the fidelity scores. Unlike [3], in our method, we have introduced a negative sign, thus a higher score represents higher fidelity. We make this adjustment for ease of presentation. The details of our fidelity test can be found in Supp. S5.1. We believe this comprehensive description will provide clarity on the metric we used.\n\n[3] Edge: Explaining deep reinforcement learning policies. In Proc. of NeurIPS, 2021."}, {"review_id": "p9JYwEudJD", "reviewer": "Reviewer_oH2u", "summary": "This submission focuses on explaining which states are important to the agent’s final reward.\nBy utilizing a mask to learn and assess which actions are critical. When learning the mask, it focuses on the random actions without affecting the agent’s performance. They evaluate on 10 different tasks such as Pong, some scenarios in StarCraft II, and Connect 4. After learning the mask, they provide fidelity scores and show some examples like Pong and Connect 4. With this information from the mask, they utilize it to perform adversarial attacks and correct agent errors by fine-tuning. Their work outperforms among EDGE, lazy MDP, and value-max.\n\n", "questions": "There were no questions per say just suggestions in the weaknesses section.", "limitations": "They do mention challenges with their approach such as converge issues. So they did address some limitations. Other negative societal impacts is not an issue for this submission since they are working on the opposite part where they want to understand for reinforcement learning why decisions are being made.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Significance and originality:\nHow ubiquitous this method can be since as they stated, it does not assume access to the agent’s value function or policy network. Hence it can be utilized by multiple methods as well as individual agents in a multi-agent environment. Figures 4 and 5 are interesting to showcase which time steps are important based on the fidelity scores. These figures showcase a one player game and a two player game showcasing the differences especially in Connect 4. Using the explanations to provide adversarial attacks and correcting agent’s sub-optimal performance is interesting. In Table 1, your implementation outperforms EDGE and others in both performance drop for adversarial attacks and performance gain in patching.\n\nEdit: I have read their rebuttal and I will change the score from a 5 to a 6.", "weaknesses": "Evaluation:\nMore evaluations among other networks to see how versatile it is. Plus other ablations such as do you vary the amount of time steps for the input, like frame stacking to assess how the time step prioritization could affect among the parameter value for frame stacking.\n\nClarity:\nIn the supplementary material when reading it, the networks just said if they were CNN, LSTM, MLP, but were they DQN, A2C, and so on. This is important information to assess with your method if it can learn the different masking for them. Plus it would be interesting as another experiment to see if DQN or Double DQN focus on different time steps.\n\nRelated Works:\nRelated works to reference perturbation methods because in the design rationale what you are mentioning is very similar to what computer vision has done with perturbation methods to understand visual explanations with salience maps. You even use the nomenclature perturb like in the second paragraph of design rationale. For instance to include perturbation computer vision methods like RISE (Petsiuk, Vitali, Abir Das, and Kate Saenko. \"Rise: Randomized input sampling for explanation of black-box models.\" arXiv preprint arXiv:1806.07421 (2018).) since it creates perturbation masks and to suggest that there has been work in computer vision using perturbations. Yes, you focus on the time steps but can mention that there has been work that wants to show which pixels are affecting classification. What you are using in your masking approach is still novel for reinforcement learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer oH2u for the constructive and insightful comments. Please see our response to each of your questions below.\n\n**1. Evaluation: More evaluations among other networks to see how versatile it is. Plus other ablations such as do you vary the amount of time steps for the input, like frame stacking to assess how the time step prioritization could affect among the parameter value for frame stacking.**\n\nWe appreciate the reviewer's suggestion to perform more evaluations among different networks and conduct further ablation studies. These suggestions indeed provide avenues for enhancing the comprehensiveness of our research. \n\nIt is worth noting that we do not assume access to the target agent's network. We report the network structures of StateMask in Supp. S4.2, which vary from CNN to LSTM and MLP. This demonstrates the versatility of our method in accommodating diverse network structures for StateMask. We appreciate your suggestion to explore more evaluations across various networks, and will consider further investigations on other networks such as the Transformer to enhance the comprehensiveness of our evaluation in future research.\n\nRegarding the method of encoding the state (i.e., either considering only the current frame or stacking multiple frames), we conduct an experiment in the Pong game. Specifically, we verify the number of frames stacked for the StateMask input in the range of {1, 2, 4} and see how the fidelity changes. We show the fidelity score comparison in Figure 3(b) of the attached PDF. We can observe that stacking two and four frames share similar fidelity while the fidelity of stacking only one frame is much lower. We suspect the reason is that we need temporal information (e.g., the movement of the ball) when training the StateMask in the Pong game. In contrast, for games (e.g., Connect 4 or Tic-Tac-Toe) where temporal dependencies play a less role, the influence of frame stacking may be less pronounced. \n\n**2. Clarity: In the supplementary material when reading it, the networks just said if they were CNN, LSTM, MLP, but were they DQN, A2C, and so on? This is important information to assess with your method if it can learn the different masking for them. Plus it would be interesting as another experiment to see if DQN or Double DQN focus on different time steps.**\n\nYour suggestion to examine the impact of using different reinforcement learning methods like DQN or A2C to train the mask network is insightful. However, our current design does not permit the direct application of these methods because our method relies on iteratively updating $\\theta$ and $\\lambda$ by solving Eqn. (7) and Eqn. (8). To utilize DQN or A2C, we have to redesign the objective function that directly maximizes the agent’s expected total reward after applying the state mask. As an additional experiment, we implement DQN and A2C on the Pong game using the new objective function and compare them with our method and report the fidelity scores in Figure 3(a) of the attached PDF. The visualization results in Figure 1, show that the state masks learned by DQN and A2C focus on different time steps from ours and have a lower fidelity compared to our method.\n\n**3. Related Works: Include perturbation methods such as RISE in the computer vision area.**\n\nWe appreciate the reviewer's suggestion to reference perturbation methods in the field of computer vision, such as the RISE [1] (e.g., a saliency map method based on randomly masking the inputs and obtaining the corresponding outputs). In light of your suggestion, we will certainly include a discussion of related work in perturbation methods from the computer vision area and compare StateMask with them in the next version. Thank you for bringing this to our attention.\n\n[1] RISE: Randomized input Sampling for Explanation of Black-box Models. In Proc. of BMVC, 2018.\n"}, {"review_id": "VI7EdD7kbD", "reviewer": "Reviewer_qciU", "summary": "This paper focuses on providing an explanation for deep reinforcement learning agents by identifying the important time steps within an episode. The authors propose a module called StateMask, which replaces the original agent's policy with random actions in specific time steps. By preserving the overall episode returns, only non-important time steps are randomized.", "questions": "1. The authors should clarify the motivation for proposing the surrogate objective. \n\n1. In line 61, authors claim that they provide theoretical analysis for StateMask but it is missing. \n\n1. What is the definition of $\\tilde{\\pi}_{\\theta}$ in line 195? In the previous sections, $\\tilde{\\pi}$ denotes the policy of MaskState instead of the integration policy $\\pi$.\n\n1. What reward function is used for the advantage function in line 201? \n\n1. The sign in Eqn 8 should be reversed. \n\n1. The metric \"fidelity\" should be well-defined in the text.\n\n1. The authors should use more metrics to evaluate their method, because StateMask directly optimizes \"fidelity\" which is not fair to other methods.", "limitations": "The authors should refine the objective Eqn 2. Besides, the writing skill needs to be improved.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "One notable strength of this paper is the intriguing concept of masking actions, which allows for identifying important time steps without altering the learned agent or its learning process.", "weaknesses": "1. The objective Eqn. 2 is problematic. The optimal solution of Eqn. 2 is $\\pi=\\bar{\\pi}$, i.e. $\\tilde{\\pi}(a_t^e=0|s_t)=1$, where the integration policy $\\pi$ degenerate to the target policy $\\bar{\\pi}$. If StateMask $\\tilde{\\pi}$ becomes a constant policy, it fails to identify any important time steps.\n\n1. Even if Eqn. 2 cannot be optimized to zero but instead reduced to a small value, StateMask $\\tilde{\\pi}$ would tend  to predict $a_t^e=0$ in most states but $a_t^e=1$ in a few specific states. Since  $a_t^e=1$ indicates a non-important state, this method can only identify a limited number of non-important states and fails to capture important time steps. Identifying non-important states is not consistent to the major motivation of this paper. \n\n1. The motivation behind using the absolute error in Eqn. 2 and its surrogate objective in Section 3.2 is not clear. A more straightforward approach for regression tasks would be to minimize the squared error (MSE / L2 loss) rather than the L1 loss, as the L2 loss is differentiable everywhere. \n\n1. The true optimization challenge is that the optimization problem of StateMask is another reinforcement learning problem. The authors propose a PPO-like surrogate objective which maximize the return under the constrain of minimizing the difference to target policy $\\bar{\\pi}$. However, it seems not consistent to the major objective Eqn 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer qciU for the constructive and insightful comments. Please see our response to each of your questions below.\n\n**1. Questions about Eqn. (2)**\n\n**Q(1):** The optimal solution of Eqn. (2) is $\\pi=\\bar{\\pi}$, failing to identify any important time steps.\n\nPlease note that our goal is to find a non-trivial solution (i.e., mask some specific states) to Eqn. (2). The challenges of finding such a solution are explained in line 162-176, and our approach is designed explicitly to seek these “non-trivial” solutions. To achieve this, we introduce a mask ratio constraint (i.e., enforce the mask ratio is larger than a threshold $c$) and propose Eqn. (5) as our objective function. A comprehensive analysis of Eqn. (5) is available in Supp. S2. \n\n**Q(2):** StateMask's limited ability to identify important time steps is not consistent with the paper's major motivation.\n\nInstead of directly using the StateMask output $a_t^e$ to identify important time steps, we evaluate the significance of states based on the probability of mask net output being 0 as stated in Sec. 3.  We calculate an importance score for each time step in a trajectory based on the probabilities given by the mask network's output. These scores are ranked in descending order, enabling us to identify the states that contribute the most to the final reward in the fidelity test, regardless of whether the mask network's output is frequently 0. Consequently, even in situations where the ratio of time steps being masked is low (e.g., <10% in MuJoCo games), the ranking of importance scores enables us to identify the critical time steps effectively (see Figure 3).\n\n**Q(3):** Why use $L_1$ loss instead of $L_2$ loss in Eqn. (2)?\n\t\nRecall our goal is to minimize the performance difference between two policies $\\pi$ and $\\bar{\\pi}$ , which requires a $L_{p}$ loss to measure the performance gap. The above optimization problem will involve the gradient of $\\eta(\\pi)$ w.r.t $\\theta$. However, a common solution using the standard policy gradient method [1] to optimize $\\theta$ cannot guarantee the monotonic decrease of the $L_{p}$. The key challenge is to derive a surrogate objective that enables the monotonically decrease during training. Indeed, whether we use $L_1$ or $L_2$ loss is inconsequential to our algorithm, and Theorem 1 remains valid when optimizing with $L_2$ loss. The objective function, Eqn. (6), will be identical regardless of using $L_1$ or $L_2$ loss. Therefore, without loss of generality, we use $L_1$ loss in Eqn. (2) to introduce the challenge and motivate our design. \n\n[1] Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning,1992.\n\n**2. Motivation of the surrogate objective Eqn. (6) and consistency with Eqn. (2).** \n\nThe motivation behind the surrogate objective Eqn. (6) is to address the challenge of obtaining a non-trivial solution for Eqn. (2) with a monotonic decrease guarantee as naively optimizing Eqn. (2) may not guarantee convergence. To solve Eqn. (6), we use a prime-dual method to update $\\theta$ and $\\lambda$ by solving Eqn. (7) and Eqn. (8). As for Eqn. (7), although it resembles the PPO, it introduces a Lagrange multiplier $\\lambda$. It is worth noting that when $\\lambda$ is larger than 1, we are doing something like minimizing the return. In fact, $\\lambda$ serves as a role to control the performance difference between $\\bar{\\pi}$ and $\\tilde{\\pi}$. By updating both $\\theta$ and $\\lambda$ iteratively, we could solve Eqn. (6) and thereby monotonically decrease the major objective Eqn. (2). \n\n**3.  Lack of theoretical analysis.**\n\nWe apologize for any confusion caused. We delve into theoretical analysis in detail in Sec. 3.2. Specifically, we describe the construction of the surrogate objective, which is theoretically derived to guarantee a monotonic decrease in Eqn. (2), in accordance with Lemma 1 and Theorem 1. Furthermore, we provide a comprehensive theoretical analysis in Supp. S2, explaining why the optimization of Eqn. (5) leads to a monotonic decrease in Eqn. (2).\n\n**4. Definition of $\\tilde{\\pi}_\\theta$ in line 195.**\n\n$\\tilde{\\pi}_\\theta$ represents a parameterized policy of StateMask. It's essential to note that the parameterization is applied to $\\tilde{\\pi}$ and not the integral policy, $\\pi$. Please see line 190-193 for more details. \n\n**5. Reward function used for the advantage function in line 201.**\n\nThe reward function of the state mask is the same as the environment’s reward provided to the target agent. \n\n**6. Reverse the sign in Eqn. (8).**\n\nPlease note that we have negated the sign of the formula in parentheses associated with $\\lambda$ when deriving from Eqn. (6)  to Eqn. (8). \n\n**7. The metric \"fidelity\" should be well-defined in the text.**\n\nWe have included a detailed description of the metric \"fidelity\" in  Supp. S5.1.\n\n**8. Need more metrics to evaluate their method, because StateMask directly optimizes \"fidelity\" which is not fair to other methods.**\n\nOur optimization objective aims to minimize the performance gap between the target agent and StateMask at the policy level, specifically, by reducing the expected total reward difference between their respective policies. In contrast, \"fidelity\" is defined at the trajectory level, where we assess if altering a consecutive series of actions within a fixed trajectory results in a significant change in the reward. Hence, StateMask does not directly optimize for \"fidelity.\" \n\nIt is worth noting that, to the best of our knowledge, the fidelity score proposed in [2] is the only quantitative metric available in our setting. Additionally, we present qualitative evaluation results in  Supp. S11, where our method surpasses baseline approaches in facilitating the user's understanding of a DRL agent's policy. \n\n[2] Edge: Explaining deep reinforcement learning policies. In Proc. of NeurIPS, 2021."}, {"review_id": "sQ9dZmk9FI", "reviewer": "Reviewer_AA7o", "summary": "The paper presents an interesting model (so called statemask) to identify critical states for an agent's final reward. The goal of statemask is to find the non-important time steps and randomize their actions without changing the expected total reward of the target agent. A PPO based algorithm is leveraged to formally generate the model. Several numerical examples are shown to demonstrate the merits of the proposed model. ", "questions": "The optimization of (2) makes sense to provide a mask network. However, it is interesting to know if the state space is huge, how to find those critical states. For example, how the critical states in Figure 4 and Figure 5 are identified? ", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents an interesting model (so called statemask) to identify critical states for an agent's final reward. The goal of statemask is to find the non-important time steps and randomize their actions without changing the expected total reward of the target agent. Several numerical examples are shown to demonstrate the merits of the proposed model. ", "weaknesses": "The paper seems claim the method is suitable for all decision making processes. However, for the type of shortest path finding problems, it is questionable that the method will be effective. In fact, the critical elements are not states, but the critical paths. So, it would be interesting to know statemask fits to what kinds of processes which is missing in the current version. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer AA7o for the constructive and insightful comments. Please see our response to each of your questions below.\n\n**1. The paper seems to claim the method is suitable for all decision-making processes. However, for the type of shortest path finding problems, it is questionable whether the method will be effective. In fact, the critical elements are not states, but the critical paths. So, it would be interesting to know statemask fits to what kinds of processes which is missing in the current version.**\n\nWe do not claim that our method is suitable for all decision-making processes. Our method may not be suited to explain the agent with poor performance. In this case, the mask net may not differentiate the important and unimportant states since replacing the current action with random action or not will have little influence on the final performance. \n\nWhile our method is to identify the states most critical to the expected total reward, it is also implicitly associated with the related action at these states. Throughout treating the target agent with a fixed policy as a part of the environment as Figure 2 shows, StateMask actually takes both the state and the corresponding action of the target agent into account. Therefore, StateMask indirectly figures out the importance of the state-action pair. Back up to the shortest path finding problem, by identifying these crucial state-action pairs, we can reconstruct the corresponding path. Specifically, we apply our method in a shortest path finding environment--MiniGrid-Empty-6x6-v0 [1]. As is shown in Figure 2 of the attached PDF,  the goal of the agent is to reach the green goal square using as few steps as possible. When the agent is optimal as trajectory 1 shows, our method figures out that all steps in the path are important. When the agent is sub-optimal as trajectories 2 and 3 show, our method highlights the path starting from the last sub-optimal action as the most critical one since in the pinpointed critical path, the agent must take actions to reach the green goal square with the fewest steps possible to keep the same reward. These insights underline the adaptability of StateMask to the shortest path problem and highlight its ability to capture critical steps in diverse decision-making contexts.\n\n[1] Minimalistic Gridworld Environment (MiniGrid). https://github.com/maximecb/gym-minigrid.\n\n**2. The optimization of (2) makes sense to provide a mask network. However, it is interesting to know if the state space is huge, how to find those critical states. For example, how the critical states in Figure 4 and Figure 5 are identified?**\n\t\nOur method is independent of the size of the state space. As stated in Sec. 3, we measure the significance of states based on the probability of StateMask output being 0, i.e., the probability of keeping the target agent’s action. Take Figure 4 and Figure 5 as an example, for a single trajectory, we calculate the importance score of each time step using StateMask and then rank the corresponding importance scores from top to bottom to identify these critical states. \n\nIn addition, as depicted in Figure 4, we successfully apply our StateMask approach to the Pong game, which has a large state space ($210\\times160\\times3$). Despite the complexity of the state space, our method accurately predicts the importance of states with high fidelity as Figure 3 in the main text indicates. Moreover, in the paper, we demonstrate the effectiveness of our approach in the MuJoCo game, which has an infinite number of states due to its continuous state space. This result further validates that our approach remains unaffected by the size or complexity of the state space.\n"}]

Author rebuttal:
Dear Reviewers,

We would like to express our sincere gratitude for your thoughtful and constructive feedback on our manuscript. Your insights and suggestions have significantly enriched the quality of our work, and we appreciate the time and effort you have dedicated to reviewing our paper.

In response to your valuable recommendations, we have diligently incorporated additional experiments that align with your suggestions. Following the suggestion of Reviewer oH2u, we further investigate two alternative designs DQN and A2C, and the effect of frame stacking in the Pong game. In response to Reviewer AA7o, we do additional experiments on a shortest path finding problem -- MiniGrid-Empty-6x6-v0 and visualize the identified critical path. We have thoughtfully documented these experiments and their outcomes in the attached PDF document, which we believe adds substantial value to the overall contribution of our research.

Your input has been instrumental in shaping the evolution of our paper, and we hope that the additional experiments and results we have provided effectively address your concerns and contribute positively to the overall understanding of our methodology. 

Once again, we extend our heartfelt appreciation for your invaluable feedback, which has undoubtedly contributed to the advancement of our research.


Meta-review: This paper seeks to explain the (black-box natured) deep RL methods by identifying the critical states that would impact the final reward. It learns a StateMask network that blinds a target agent and forces it to take random actions at some steps without compromising the agent's performance. All the reviewers agree that the paper is well-written and has novel and significant contribution. The experimental results are solid in showing the effectiveness of the proposed method. Furthermore, the feedbacks provided by all the reviewers have been sufficiently addressed during the rebuttal phase.
Decision: Accept (poster)