EDGE: Explaining Deep Reinforcement Learning Policies
Authors: Wenbo Guo
Xian Wu
Usmann Khan
Xinyu Xing
Venue: NeurIPS 2021 Poster (2021)
Keywords: Explainable AI
Deep reinforcement learing
Explainable Deep reinforcement learing
Policy debugging

Abstract: With the rapid development of deep reinforcement learning (DRL) techniques, there is an increasing need to understand and interpret DRL policies. While recent research has developed explanation methods to interpret how an agent determines its moves, they cannot capture the importance of actions/states to a game's final result. In this work, we propose a novel self-explainable model that augments a Gaussian process with a customized kernel function and an interpretable predictor. Together with the proposed model, we also develop a parameter learning procedure that leverages inducing points and variational inference to improve learning efficiency. Using our proposed model, we can predict an agent's final rewards from its game episodes and extract time step importance within episodes as strategy-level explanations for that agent. Through experiments on Atari and MuJoCo games, we verify the explanation fidelity of our method and demonstrate how to employ interpretation to understand agent behavior, discover policy vulnerabilities, remediate policy errors, and even defend against adversarial attacks.

EDGE: Explaining Deep Reinforcement Learning Policies
Wenbo Guo
Xian Wu"
The Pennsylvania State University aegl
The Pennsylvania State University xkw51320psu. edu
Usmann Khan"
Georgia Institute of Technology
ukhan356gatech. edu
Xinyu Xing Northwestern University
The Pennsylvania State University xinyu wiggehottwttweeeerr..n edu
Abstract
With the rapid development of deep reinforcement learning (DRL) techniques, there is increasing need to understand and interpret DRL policies. While recent research has developed explanation methods to interpret how agent determines its moves, they cannot capture the importance of actions/states to a game's 's final result. In this work, we propose novel self-explainable model that augments a Gaussian process with a customized kernel function and an interpretable predictor. Together with the proposed model, also develop parameter learning procedure that leverages inducing points and variational inference to improve learning effi- ciency. Using proposed model, can predict agent's 's final rewards from its game episodes and extract time step importance within episodes as strategy-level explanations for that agent. Through experiments Atari and MuloCo games, we verify the explanation fidelity of our method and demonstrate how to employ inter- pretation understand agent behavior, discover policy vulnerabilities, remediate policy errors, and even defend against adversarial attacks.
Introduction
Deep reinforcement learning has shown great success in automatic policy learning for various sequential decision-making problems, such training Al agents defeat professional players in sophisticated games [74] 65| 24 37 and controlling robots to accomplish complicated tasks [33 [38] However, existing DRL agents make decisions in an opaque fashion, taking actions without accompanying explanations. This lack of transparency creates key barriers to establishing trust in an agent's 's policy and serutinizing policy weakness. This issue significantly limits the applicability of DRL techniques in critical application fields (e.g., finance [7 and self-driving cars (Ii).
To tackle this limitation, prior research (e.g., 1 73 proposes derive explanation for target agent's action at specific time step. Technically, this explanation can be obtained by pinpointing the features within the agent's observation of a particular state that contribute most to its corresponding action at that state. Despite demonstrating great potential to help users understand a target agent's
Equaal Contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

individual actions, they lack the capability to extract insights into the overall policy of that agent. In other words, existing methods cannot shed light on the general sensitivity of an agent's final reward from game in regards to the actions/states in that game episode. Consequently, these methods fall short in troubleshooting an agent's policy's 's weaknesses when it fails its task.
We propose a novel explanation method to derive strategy-level interpretations of a DRL agent. As we discuss later in Section 3 we define such interpretations as the identification of critical time steps contributing to target agent's 's final reward from each game episode. At a high level, our method identifies the important time steps by approximating the target agent's s decision Decision-making process with self- eeffexxlain model and extracting the explanations from this model. Specifically, given a well-trained DRL agent, our method first collects a set of episodes and the corresponding final rewards of this agent. Then, it fits self- explaanbie model to predict final rewards from game episodes. To model the unique correlations in DRL. episodes and enable high-fidelity explanations, rather than simply applying off-the-shelf self-explanation techniques, we develop novel self-explainable model that integrates a series of new designs. First, we augment a Gaussian Process (GP) with customized deep additive kernel to capture not only correlations between time steps but, more importantly, the joint effect across episodes. Second, we combine this deep GP model with our newly designed explainable prediction model to predict the final reward and extract the time step importance. Third, we develop an efficient inference and learning framework for our model by leveraging inducing points and variational inference. We refer our method as "Strategy- Explanation of Drl aGEnts" (for short EDGE).
With extensive experiments on three representative RL games, we demonstrate that EDGE outperforms alternative interpretation methods in terms of explanation fidelity. Additionally, we demonstrate how DRL policy users and developers can benefit from EDGE. Specifically, we first show that EDCE could help understand the agent's 's behavior and establish trust in its policy. Second, we demonstrate that guided by the insights revealed from our explanations, attacker could launch efficient adversarial attacks to cause a target agent to fail. Third, we demonstrate how, with EDGE's capability, a model developer could explain why a target agent makes mistakes. This allows the developer to explore a remediation policy following the explanations and using it to enhance the agent's 's original policy. Finally, we illustrate that EDGE could help develop defense mechanism against a newly emerging adversarial attack on DRL agents. To the best of our knowledge, this is the first work that interprets DRL agent's policy by identifying the most critical time steps the agent's final reward in each episode. This is also the first work that demonstrates how to use an explanation to understand agent behavior, discover policy vulnerabilities, patch policy errors, and robustify victim policies.
2 Related Work
Past research on DRL explanation primarily focuses on associating an agent's 's action with its obser- vation particular time step (i.e., pinpointing the features most critical to the agent's action at specific time). Technically, these methods can be summarized in the following categories.
Post-training explanation is method that utilizes and extends post-training interpretation ap- proaches (e.g., [56 28 36 355 to derive explanation from a DRL agent's 'ss policy/value network
and thus treat it as the interpretation for that DRL agent (e.g.. 9 44 32
68
20
72)).
Model approximation is an approach that employs a self-interpretable model mimic the target agent's 's policy networks and then derives explanation from the self ceff-interpretable model for the
target DRL agent (e.g., B 22 55
4
59
58
87 85
Self-interpretable modeling is an approach different from the model approximation techniques above. Instead of mimicking the target agent's policy network, self-interpretable modeling builds self-explainable model to replace the policy network. Since the new model is interpretable, one can
easily derive an explanation for the target agent (e.g., [92 82
method that re-engineers DRL agent's reward function to make the reward gained at each time step more meaningful and explainable. With the more meaningful reward in hand, at each time step, one could use the instant reward gain to interpret the agent's action (e.g.
Reward decomposition
77 46 (44)
'The source code of EDGE can be found in htttp:://ggtauu...co//eeyyyyyses

---

From the objective perspective, our work is fundamentally different from the above DRL explanation research. Rather than pinpointing the features in an observation critical for an agent's 's action, our work identifies the critical time steps contributing to the agent's is final reward. Using our explanation, one can better understand the agent's policy, unveil policy weakness, and patch policy errors (as shown in Section[)). In Supplement S7, we further conduct a user study to demonstrate the superiority of our method against the above explanation approaches in pinpointing good policies and performing policy forensics. Note that there are two other methods that also understand DRL policy through the agent's previous memories 49 23]. These works are fundamentally different from ours in two perspectives. First, both methods have different explanation goals from our work. Specifically, Koul et al. [49 focuses on identifying whether the action at each time step depends more on the current observation or the previous states. The method proposed in [3] pinpoints the important steps w.r.t. the subsequent transitions in the FSM extracted from the target agent rather than the final result of episode. Second, both methods can be applied only to white-box RNN policies, whereas our method is applicable to DRL policies with arbitrary network structures.
3 Key Technique
3.1 Problem Setup
Consider DRL game with an agent trained with Q-learning 86 60 or policy gradient 48 69
70
Our work aims to explain this agent's 's policy by identifying the important steps contributing most to a game episode's final reward/result. To ensure practicability, we allow access only to the environment states, agent's actions, and rewards. We assume the availability of neither the value/Q function nor the policy network. Formally, given N episodes T = {X (8) yu biiiN of the target agent, x {s (8) a."' )
R==1:T is the i-th episode with the length T, where s.") Rdd, and a Rd are the state and action at the time step t. is the final reward of this episode Our goal is to highlight
the top-K important time steps within each episode (+)
Possible Solutions and Limitations. The most straightforward approach of identifying important time steps is to use the output of the value/Q network as indicators and pinpoint the time steps with the top K highest value/Q function's 's outputs the top K critical steps. However, since we do not assume the availability of these networks, this method is not applicable to our problem. A more realistic method is to fit seq2one model (i.e., RNN) that takes input the state-action pairs in an episode and predicts the final reward of that episode. With this prediction model, one could utilize a post-training explanation method to derive the time step importance. However, existing post-training explanation techniques usually require approximating the target DNN with more transparent models, which inevitably introduces errors. Additionally, many post-training methods are vulnerable adversarial
attacks [61 30 [4] or generate model-independent explanations [I] 64 2 78 As we will show later in Section 4 and Supplement S3&S5, these limitations jeopardize the post-training explanations' fidelity. A more promising direction is to fit a self-explainable model to predict the final reward. Existing research has proposed a variety of self-explanation methods. Most of them do not apply to our problem because they either cannot derive feature importance as explanations 52 52 50 18).
cannot be applied to sequential data [21 27 12]. or require explanation ground truth [6 66 In this
work, we consider two self-explainable models that are designed to fit and explain sequential data
an RNN augmented with attention 10 39 and rationale net 51 Technically, both models have the form of ((((x) @ x). where 8(-) is weight RNN or an attention layer and g(-) is the prediction RNN. The output of (()) can be used identify the important steps in the input sequence. Despite reveals that the explanations given by 0(-) cannot faithfully reflect the associations (i.e., feature importance) learned by the subsequent prediction model g(.), leading to an even lower fidelity than the post-training explanations in some applications. Additionally, these models are not designed to explain an RL agent and cannot fully capture the dependencies within the episodes of that agent. Specifically, the episodes collected from the same agent tend to cxhibit two types of dependencies: dependency between the time steps within an episode and the dependency across different episodes. Although they consider the dependency
extracting meaningful explanations, recent research 45 g9 17
'For the games with delayed rewards, such as MuJoCo 84] and Atari Pong [8) where non-zero reward rr is assigned only to the last step of game, we use rr yr. For the games with instant rewards (e.g.. OpenAl CartPole (15).. ).,, compute an episode's total reward as yi. i... yi Σ, rt.
3

---

Tmnedhip Enbxdtngg
Additvo GP Model
Pred
maro Sea
Model
Episedo Stop comclations
enooder
hy
Inpui apssade x
Grie xlep
MLP oncoder
Eptsodo embedding
Step Imporrancc
Figure 1: Overview of EDGE with a constant prediction mixing weight.
within each input sequence, these methods cannot capture the correlations between different inputs. As is shown in Section 4 and Supplement S3&S5, this also jeopardizes their explanation fidelity.
3.2 Explanation Model Design of EDGE
In this work, we design a novel eelf-explainable model by adopting very different design than existing methods. First, to better capture the associations (i.e., feature importance) learned by the prediction model, we add the explainable module to the final layer rather than the input layer of the prediction model. Formally, our model can be written s((()), where ʃ()) is feature extractor and g(-) is an explainable prediction model. Second, we design a deep Gaussian Process as the feature extractor to capture the correlations between time steps and those across different episodes, which are often exhibited in a set of episodes collected from the same DRL agent. addition to capturing different levels of correlations, another advantage of deep GPs over typical DNNs is that GPs model the joint distribution of the output signals, enabling access to the output signals' uncertainty. Finally, design interpretable Bayesian prediction model to infer the distribution of final rewards and deliver time step importance. Below, we first give an overview of our proposed model. Then, we describe how to adapt the traditional GP model to our problem, followed by the design of the final prediction model.
As shown in Fig given an episode of the target agent ( EDGE first inputs into a RNN encoder, which outputs the embedding of each time step in this episode (h (0) 1- EDGE also passes the last step's embedding through a shallow MLP to obtain an episode embedding ( Then, EDGE adopts proposed additive GP framework to process (h,") and e and obtains a latent representation the whole episode - As introduced later, this representation is able to capture the correlations between time steps and those across episodes. Finally, EDGE inputs p/1) into our prediction model E/ and get the predicted final reward of the input episode. As detailed later, our prediction model is designed based on a linear regression, whose regression coefficient can be used to identify important time steps within in the input episode.
Overview.
1==1T
Additive GP with Deep Recurrent Kernels. Gaussian Process defines distribution an infinite collection of random variables, such that any finite subset of variables follows multivariate Gaussian distribution (63]. In Statistical modeling, GP defines the prior of a non-parametric function x R. Formally. if f has a GP prior, i.e., f ~ GP(O, k.), where ki (., is a positive semi- definite kernel function parameterized by ?/ any finite collections of f E R follows a multivariate Gaussian distribution Xf) (0, Xxx). Here, K Kxx N the covariance matrix, with (Kxx); by (x,, x In our model, we adopt the widely applied square exponential (SE) kernel function: k. (x; x;) exp (- _ (x; x,) Ox - x;)). with O₁. Traditional GP with SE kernel 63 assumes the input space is Euclidean, which usually invalid for real-world data with high- -ithdmiinniin inputs [3. To tackle this challenge, recent research [9 [33 proposes to conduct
dimensional reduction a DNN and then apply a GP the DNN's latent space. They show that the resultant deep kernel models achieve similar performance to DNNs on complicated datasets.
In our model capture the sequential dependency within an episode by using an RNN the deep net inside the kernel function. Specifically, given episode (+), we first concatenate the state and action (i.e., xX") [s(") (())), input them into an RNN hgg and obtain the latent representation of (4) this episode: {h, T==1.T where h e 189 is the state-action embedding the time t. We also

---

compute an episode embedding by passing the last step's hidden representation through a shallow MLP cØ, hy + e E R.. After obtaining {h1") Jt=i:T and e(*), we then adopt the additive GP framework to capture the correlations between time steps and those across episodes. Formally, an additive GP is the weighted sum of J independent GPs, i.e., f = Σ, Jz Here, J (O, ky) is the j-th GP component, in which the covariance function k, is typically applied to a subset of input features. By assigning every component a GP prior, one can ensure that the mixed- signal f also follows a GP prior (25]. Following this framework, we construct our deep GP model as the sum of two components Se and fe- Specifically, St GP ) models the correlations between time-steps, where the covariance between the f-th steps in episode and the k-th steps in episode can
be computed by ky, (hg²) ). Going beyond modeling the correlations between individual steps, fo ~ (P ky captures higher level cluster structures within the collected episodes, i.e., the similarity between episodes. Formally, the episode-level covariance between any pair of time steps in episode and j is given by ky, (e()) e Finally, our deep additive GP model can be expressed as: f f ft + Ceefe, where at and ae are the component weights. Given a set of collected episodes represented by T E R [xxxx((d,+d).) , E RR NT is given by: f|X ~ N k = ke + a² k2,),,),, where X E +ad is the flattened matrix T.
Prediction Model. To ensure explanability, we use a linear regression as the base of our prediction model, where the regression coefficients reflect the importance of each input entity. Specifically, we first convert the flattened response f back to the matrix form F E NXT, where the i-th row F(O E R is the i-th episode's s encoding given by our GP model. Then, we define the conditional likelihood for the discrete and continuous final reward, respectively. When y: is continuous, we follow the typical GP regression model [63] and define the y; F(] + C1, where w R11xT is the mixing weight and ~ NO0 o²) is the observation noise. The conditional likelihood distribution is
y F(O) ~ W(F"""" r²). For the discrete final reward with finite number of possible values, we use the softmax prediction model to perform classification. Formally, we define y: [F() follows a categorical distribution with p(y: = k)( = Σ. exp(((()
( (p(())
W E R is the mixing weight, where K is the total number of classes. Finally, we combine all the components together and write our explanation model as (A illustration of our proposed model can be found in Fig.
(alssoftmax(F"))W" (N(F w ,o").
If conducting classification otherwise
f[X N/ a2k o2k,,)) Yi F
where the mixing weight is constant. This indicates the time step importance derived from the mixing weight is a global explanation. According to the insight that time steps with high correlation tend to have a joint effect (similar importance) on the game result, we could combine the global explanation with the time step correlations in K2 (X, X) to gain fine-grained understanding of each game. Specifically, given an episode and the top important steps indicated by the mixing weight, we can identify the time steps that are highly correlated to these globally important steps and treat them together as the local explanation of that episode. Supplement S1 introduces another way of deriving episode-specific explanations by replacing the constant mixing weight with weight obtained by simple DNN. Note that the episode correlations in K2(X, X) reveal the cluster structure within a set of episodes, which helps categorize the explanations of similar episodes.
3.3 Posterior Inference and Parameter Learning
Sparse GP with Inducing Points. Direct inference of our model requires computing (Kxx -o²)) over Kxx, which incurs (NTT) computational complexity. This cubic complexity restricts our model to only small datasets. To improve scalability, we adopt the inducing points method [9]] for inference and learning. At high level, this method simplifies the posterior computation by reducing the effective number of samples in X from NT to M, where M the number of inducing points. Specifically, we define each inducing point at the latent space as z; E R2a, and u; as the GP output
"Note that our model is similar existing Ga-based state-space models 71 29 81 26 in that both
an RNN inside the kernel function. However, these models not integrate an additive GP. More importantly, their prediction models are not designed for explanation purposes and thus cannot derive time step importance.
Thee classification model gives K global explanations. one for each class derived from each row of W.
5

---

of Z5. Then, the joint prior of f and and the conditional prior flu are given by:
10]
Kxx
Kxz]
f,u ~ .
.fl. X, N(Xxx Kxx KxxKzzkkz). (2)
K
Kzz
where Xxx, kxz, K2 zz are the covariance matrices. They be computed by applying our additive kernel function to the time-step and episode embedding of the training episodes and inducing points. As is shown in Eqn (2). with inducing points, we only need to compute the inverse of K Zzz. which significantly reduces the computational cost from (NNTS) to ((m)).
Variational Inference and Learning. So far, our model has introduced the following parame- ters: neural encoder parameters e,, = {ór $1). GP parameters θ = {*ie Ye: Cle, ac), prediction model parameters @ = {w/ Kw/W,ar o²), /W,o"), and inducing points Z ( To learn these pa- rameters, we follow the idea of empirical Bayes 63 and maximize the log marginal likelihood log p X Z, O, x p.). Maximizing this log marginal likelihood is computationally expensive and, more important, intractable for models with non-Gaussian likelihood. To provide factorized approximation to marginal likelihood and enable efficient learning, we assume variational posterior over the inducing variable q(u) ~ ((,, Σ) and a factorized joint posterior q(f, u) (() where p([ u) is the conditional prior in Eqn. By Jensen's 's inequality, we can derive the evidence lower bound (ELBO):
log ply X. Z, O, O₂. Op) 2 Ey(r) [log p(y (Llg(u))pp()).
where the first part is the likelihood term. The second KL term penalizes the difference between the approximated posterior quu) and the prior p(u). Maximizing the ELBO in Eqn. (3) will automatically maximize the marginal likelihood, which is also equivalent to minimizing the KL. divergence from the variational joint posterior to the true posterior (See Supplement S1 for more details).
When conducting classification, the categorical likelihood makes the likelihood term Eqn. 3 intractable. To tackle this challenge, we first compute the marginal variational posterior distribution of f, denoted as (f() = Nuuy Ex) (See Supplement S1 for detailed computations). Then, we apply the reparameterization trick [57] to qf)). Formally, define f = vv(E)) Jus + Lyjey with Cf (OO, I) and L,L = Σ,- With this reparameterization, we can sample from the standard Gaussian distribution and approximate the likelihood term with Monte Carlo (MC) method [3]
1 ΣΣ log p(v:l (F())()))
Eyery (log pvv f)] = EEtee) (og p(ulv(s))) R
(4)
where is the number of MC samples. For the regression model, we directly compute the analyt- ical form of likelihood term and use it for parameter learning (See derivation in Supplement SI), With the above approximations, our model parameters (i.e., Ox Oz. Op. Z. and {u. E)} can be efficiently learned by maximizing the (approximated) ELBO using stochastic gradient descent method. Implementation details and hyper-parameter choices can be found in Supplement S2.
Evaluation
In this section, evaluate EDGE on three representative RL games (all with delayed rewards) Pong in Atari, You-Shall-Not-Pass in MuJoCo, and Bick-And-Defent in MuJoCo. Supplement S5 further demonstrates the effectiveness of our method on two OpenAI GYM games (both with instant rewards). For each game, we used a well-trained agent as our target agent (See Supplement S2 for more details about these agents).
Baseline Selection. Recall our goal is to take as input the episode of a target agent and identify the steps critical for the agent's 's final reward. As is discussed in Section 3.1 to do this, there are two categories of alternative approaches then deriving explanation from that model and @ fitting episode through self-cxplainable model and then obtaining interpretation directly from its interpretation component. In this section, we select some representative alternative methods as our baseline and compare them with proposed method. Below, we briefly describe these baseline approaches and discuss the rationale behind our choice.
fitting an episode through non- -ntiieerrreetble model and
With respect to the first type of alternative approaches, we first utilize the RNN structure proposed in 43 to fit the reward prediction model. Then, we apply various gradient-based saliency methods on the RNN model and thus derive interpretation accordingly. We implement three widely used saliency

---

Rusder
Salency
Amertion
Rattee
rOGE
Aben- Pong
MulaCo VousStaat-lanPaae
Figure 2: Mean and standard error of the fidelity scores obtained by each explanation method. The x-axis represents the different choices of K. "RatNet" stands for Rationale Net. For our method, we use the global explanations derived from the mixing weight in this evaluation.
methods- Vanilla gradient [76. integrated gradient [80]. and SmoothGrad 77 as well as their variants (ExpGrad [79]. VarGrad 40, and integrated gradient with uniform baseline (79 When comparing +saiency method with our proposed approach, we choose the RNN's interpretation from the saliency method with the best explanation fidelity. For the fidelity comparison between each saliency method, the readers could refer to Supplement S3. In addition to the RNN+saliency method, another method falling into the first kind of alternative approaches is Rudder 77). Technically speaking, this method also learns an RNN model to predict an agent's 's final reward. Differently, derives explanation from decomposed final reward.
Regarding the second kind of alternative approaches, we choose Attention RNN and Rationale Net. Attention RNN [10] is typically treated as self-interpretable model. From the model's 's attention layer, one could extract its output and use it as the important scores for the input dimensions. We use these important scores to pinpoint the critical time steps in the input episode. Similar to Attention RNN, Rationale Net is also self-im interpreettee.ee In our experiments we use Rationale Net's original model structure (5I) rather than the improved model structure proposed in [17. This is because, going beyond training data, the improved model training requires additional information, which is unavailable for our problem.
Evaluation Metric. An intuitive method to evaluate the fidelity of the various approaches' explana- tions is to vary the actions at the time steps critical for the final reward and then measure the reward difference before and after the action manipulation. However, this method invalidates the physical realistic property of an episode because the change of an agent's 's action at a specific time step would inevitably influence its consecutive actions and the state transitions. To address this problem, we introduce a physically realistic method to manipulate episodes. Then, we introduce a new metric to quantify the fidelity of interpretation.
Given the explanation of the i-th episode E E,, we first identify the top-K important time steps from E, From the top-K time steps, we then extract the longest sequence (i.e. the longest continuous time steps), record its length - I, and treat its elements as the time steps most critical to ys.
To evaluate and compare the fidelity of the interpretation (i.e., the most critical time steps extracted through different interpretation methods), we first replay the actions recorded on that episode to the time step indicated by the longest sequence. Starting from the beginning of the longest continuous time steps to its end (i.e., t. t241), we replace the corresponding actions at these time steps with random actions. Following the action replacement, we pass the state t,+44 to the agent's ' policy. Starting from 444411 we then use the agent's 's policy to complete the game, gather the final reward, and compute the final reward difference before/after replaying denoted as d. After computing and d, we define the fidelity score of E; as log(p;) - log(Pa). Here, P = 1/T is the length of the longest sequence normalized by the total length of the episode- T. Pa (dl/dman difference normalized by the maximum absolute reward difference of the game. When the value of the fidelity score log(pr) log(Pa) is low, it indicates E, is illustrated by short length of sequence. By varying the actions pertaining to this short sequence, we can observe great change in the agent's s final reward. such, a low score implies high fidelity of an interpretation method.
the absolute reward
Result. Fig. Fig.2isoww shows the comparison results of EDGE against the aforementioned alternative explana- tion approaches. First, we observe that existing self-explainable methods (i.e., Attention and Rational
'Note that we select these saliency methods because they pass the sanity check [ 2 Besides, it should be noted that do not consider the perturbation-base methods derive interpretation from RNN because these methods are mainly designed to explain convolutional networks trained for image recognition tasks.
If the policy network is an RNN, we also fit the observation at time f, - t441 into the policy to ensure the RNN policy's 's memory is not truncated.
7

---

Kaknning apicode sagge (g agenl
Aloing spanse tage (gewn) agerz
Time step importance of the target agent in the Pong game.
agent
booing sploode larget (pe0)
(b) Time step importance of the target regular agent in the You-Shall-Ne SIlllNNotPPSSSSSSS game.
Figure Illustrations of the critical time steps extracted by EDGE in winning/losing episode.
Table 1: The target agent's 'ss performance in different use cases. "MuJoCo-Y" represents the You- Shall-Not-Pass game and "MuJoCo-K" stands for the Kick-And-Defend game. To demonstrate the statistical significance seeds and show the mean and standard error of results on each setup. Numbers before the brackets are means and those in the brackets are standard deviations. Supplementary S6 further shows a hypothesis test result.
our results, we run all the experiments three times with different random
Applications
Games
Rudder
Saliency
Attention
RatNet
EDGR
Fong
-30.33 (4.433
-25.27(1.79
(4.24)
-65.47 (2.90)
Target agent win rate changes before/after attacks
MuJoCo-Y
-32.53
-29.33
(5.777
(L.63)
-35.13 (2.29)
MuJoCo-K
-21.80 (3.70)
-37.87 (6.31)
(4.70)
-..13
-43.47 (4.01)
Fong
+1.89 (I.25)
-1.13 (0.96)
(1.88
(1.65
+2.75 (0.65)
Target agent win rate changes before/after patching
MuJoCo-Y Y
(0.76
+0.92 (0.32)
40.44 (0.06)
+1.68 (0.50)
+2.91 (0.32)
MuJoCo-K
+0.96 (0.1)
+1.17 (0.17)
+0.57 (0.04)
+1.21 (0.16)
+1.21 (0.13)
Vietim agent win rute chunges before/after robustifying.
MuJoCo-Y
(0.75)
+12.69 (1.46)
+25.10 (1.44)
+25.42 (1.32)
+35.30 (3.02)
Net) cannot consistently outperform the post-training explanation approaches (i.e., saliency methods and Rudder). This observation aligns with our discussion in Section 3.1 Second, we discover that our method demonstrates the highest interpretation fidelity across all the games in all settings. As we discuss Section 3.2 is because our method could capture not only the inter-relationship between time steps but, more importantly, the joint effect across episodes.
In addition the fidelity of our interpretation, we also evaluate the stability of our explanation and measure the explainability of each approach with regard to the underlying model. We present the experiment results in Supplement S3, demonstrating the superiority of our method in those dimensions. Along with this comparison, we further describe how well our method could fit given episodes, discuss the efficiency of our proposed approach, and test its sensitivity against the choice of prper-parameters. Due to the page limit, we also detail these experiments and present experimental results in Supplement S3.
5 Use Cases of Interpretation
Understanding Agent Behavior. Fig. showcases some episode snapshots of the target agent in the Pong and You-Shall-Not-Pass game together with the time-step importance extracted by our method. As we can first observe from Fig. 3(a) in the winning (left) episode, EDGE highlights the time steps when the agent hits the ball the key steps leading agent wins because it sends a difficult ball bouncing over the sideline and sailing to the corner where the opponent can barely reach. Oppositely, our method identifies the last few steps that the target agent misses the ball as the critical step in the losing episode. This indicates that the agent loses because it gets caught out of position. Similarly, our method also pinpoints the critical time steps matching human perceptions in the Sou-Shall-Not-Paae
win. This explanation implies that
target
games. For example, in the left episode of Fig. 3bb] our explanations state that the runner (red agent) wins because it escapes from the blocker and crosses the finish line. Overall, Fig. demonstrates that the critical steps extracted by EDGE can help humans understand how an agent wins/loses a game. In Supplement $4, we show more examples of critical time steps and the correlations we extracted from the three games. Supplement S7 further shows user study to demonstrate that our explanation could help user understand agent behaviors and thus perform policy forensics.

---

Aeriing sode of siam (red) agant
loiing spaods the vidam (rua)) agane
Figure 4: Time step importance of the victim agent in the You-Shall Shall-Not-Paree game. By comparing this figure with Fig. 3bb we can observe that our method could derive different explanations for different policies in the same game, indicating explanation is policy dependent.
Launching Adversarial Attacks. The qualitative analysis above reveals that an agent usually wins because of its correct moves at the crucial steps. With this finding, we now discuss how to launch adversarial attacks under the guidance of the interpretation. Previous research AT 93 has proposed various attacks to fail a DRL agent by adding adversarial perturbation to its observations at each time step. We demonstrate that with the help of the explanations, an attacker could defeat an agent by varying actions at only a few critical steps rather than adding physically unrealistic perturbations. Our key idea is intuitive. If an agent's 's win mainly relies on its actions at a few crucial steps, then the agent could easily lose if it takes sub-optimal actions at those steps. Guided by this intuition, we propose an explanation- based attack that varies the agent's 's action at the critical steps identified by an explanation method. To test this attack's s effectiveness, we first collect 2000 episodes where the target agent wins and explain these episodes with EDGE and the baseline approaches. Second, we conclude the top-K commonly critical steps across all the episodes (Here, we set K=30). Finally, we run the agent in the environment and force it to take a random action at the common important steps. We test the agent for 500 rounds and record the changes in its winning rate before/after attacks in Table [ As we can observe from the 2~44 row of Table I1 all the explanation models can generate effective attacks that reduce the agent's 's winning rate. Benefiting from the high explanation fidelity, the attack obtained from our explanations demonstrates the strongest exploitability. Supplement S4 shows the results of different choices of K and discusses the potential alternatives of our attack. Note that this attack is different from the fidelity test in Section 4 in that our attack generalizes the summarized time step importance to unseen episodes while the hddeityy test replays the explained episodes.
We design an explanation-guided policy patch method. The key idea is to explore remediation policy by conducting explorations the critical time steps of losing games and use the mixture of the original policy and the remediation policy as the patched policy. Specifically, we first collected a set of losing episodes of the target agent and identified the important time steps with EDGE and the baseline approaches above. Then, we explore the remediation policy by replay those episodes with different actions at the critical steps. Here, since we do not assume an oracle knowing the correct actions to take, we perform random explorations. First, we set an exploration budget of 10, representing replaying 10 times for each losing episode. In each replaying, we take a random action at the top 5 consecutive critical steps and record the random actions and corresponding states leading to a win. Finally, we form a look-up table with these collected state-action pairs and use it as the remediation policy. When running in the environment, the target agent will act based on the table the current state is in the table. Otherwise, the agent will take the actions given by its original policy. To test the effectiveness of our method, we run 500 games and record the changes in the target agent's 's winning rate before and after patching. As is shown in row 5~7 of [able overall, the patched policies enhance the target agent's 's performance, and EDGE demonstrates the highest winning rate improvement. many false positive that even reduce the winning rate. In Supplement S4, we discuss how to mitigate this problem via a probabilistic mixture of the remediation policy and the original policy. Supplement S4 also experiments the influence of the look-up table size on the patching performance and discusses other alternatives to our patching method.
Patching Policy Errors.
AablcTalso shows that in some cases, the patched policy introduces too
Finally, we apply our methods to explain the episodes of victim agent playing against an adversarial opponent in the SSShall-Not-Paal game. The adversarial policy is obtained by the attack proposed in [II. Fig, 4 demonstrates the identified important steps. First, the losing episode in Fig. shsow the blocker takes a sequence of adversarial behaviors (i.e., intentionally falling on the ground). These malicious actions trick the runner into falling and thus losing the game.
Robustifying Victim Policies.
*For games with a continuous state space, we compute the norm difference of the current state 8, and the states s; in the table. the state difference is lower than small threshold (Ie-4 in our experiment. We tested 1e-3, le-4, and le-5 and observed similar results.), we treat s, and s; as the same state. Since the games of the same agent usually start from relatively similar states and transition following the same policy, it is possible to observe similar states in different episodes.

---

Oppositely, the similar adversarial actions in the winning episode cannot trigger the runner to behave abnormally. The explanations reveal that the different focus of the victim causes the different victim actions. In the winning episode, the victim agent focuses less on the steps pertaining to adversarial actions, whereas those steps carry the highest weights in the losing episode. This finding implies that the victim agent may be less distracted by the adversarial actions if it does not observe them. Guided by this hypothesis, we propose to robustify the victim agent by blinding its observation on the adversary at the critical time steps in the losing episode (i.e., the time steps pertaining to adversarial actions). We test the partially blind victim and record the changes in its winning rate before/after blinding. As is shown in the last row of Table I blinding the victim based on explanations significantly improves its winning rate. Table I also demonstrates the effectiveness of the baseline approaches in robustifying victim policies. Overall, we demonstrate that the explanations of victim policy could pinpoint the root cause of its loss and help develop the defense mechanism.
6 Discussion
Scalability. As is discussed in Section 3.3 by using inducing points and variational inference, our model parameters can be efficiently solved by stochastic gradient descents. Supplement S3&S5 show that EDGE imposes only small training overhead over the existing methods. We can further accelerate the training of EDGE by leveraging more advanced matrix computation methods, such as approximating the covariance matrix with kernel structure interpolation [90] replacing Cholesky decomposition with Contour Integral Quadrature when computing the zz [67.
Other games. Besides the two-party Markov games (i.e., Atari Pong and MuloCo) studied in this work, many other games also have delayed rewards - mainly multi-player Markow games (e.g., some zero-sum real-time strategy games (88) and extensive-form games (e.g., Go (74) and chess (75)). Regarding the multi-player Markov games, the associations between the episodes and final rewards will also be more sophisticated, requiring model with a high capacity to fit the prediction. As part of future work, we will investigate how to increase the capacity of our proposed model for those games, such as adding more GP components or using a more complicated DNN as the mixing weight. For the extensive- form games, only one agent can observe the game state at any given time step and thus take action. As such, these games have a different form of episodes from the Markov games. In the future, we will explore how to extend our model to fit and explain the episodes collected from extensive-form games. Supplement $5 demonstrates our method's explanation fidelity on two games with instant rewards. Future work will evaluate the effectiveness of our attack and patching methods on those games and generalize our method to more sophisticated games with instant rewards.
Limitations and Future Works. Our work has a few limitations. First, we mainly compare EDGE with some existing techniques that have been used to explain sequential data. It is possible that with some adaptions, other explanation methods can also be applied to sequential data. It is also possible that existing methods can be extended to capture the correlations between episodes. As part of future work, we will explore these possibilities and broader solutions to explain a DRL policy. Second, the fidelity evaluation method introduced in Section X multiple continuous important sequences. Our future work will investigate more rigorous fidelity testing methods and metrics. Third, our current learning strategy provides the point estimate of the mixing weight (explanations). In future work, we will explore how to place prior on the model parameters and apply Bayesian inference (e.g., MCMC [6]) to output the explanation uncertainty. Finally, our work also suggests that it may be possible to train a Transformer on MDP episodes to analyze offline trajectory data [19) and then add a GP on top to perform ablation studies. As part of future works, we will explore along this direction.
could be further improved, such as identifying
7 Conclusion
This paper introduces EDGE to derive strategy-level explanations for a DRL policy. Technically, treats the target DRL agent as a blackbox and approximates its decision- process through our proposed self-explainable model. By evaluating it on three games commonly utilized for DRL evaluation, we show that EDGE produces high-fidelit explanations. More importantly, we demonstrate how DRL policy users and developers could benefit from EDGE to understand policy behavior better, pinpoint policy weaknesses, and even conduct automated patches to enhance the original DRL policy.
10

---

Acknowledgments
We would like to thank the anonymous reviewers and meta reviewer for their helpful comments. This project was supported in part by NSF grant CNS-2045948 and CNS-2055320, by ONR grant N00014-20-1-2088, by the Amazon Research Award, and by the IBM Ph.D. Fellowship Award.
References
[1] Adebayo, J. Gilmer, M. Muelly, Goodfellow, M. Hardt, and B. Kim. Sanity checks for saliency maps. In Proc. of NeurIPS, 2018.
[2] J. Adebayo, M. Muelly, I. Liccardi, and B. Kim. Debugging tests for model explanations. In Proc. of NeurlPS, 2018.
[3] C. C. Aggarwal, A. Hinneburg, and D. A. Keim. On the surprising behavior of distance metrics in high dimensional space. In Proc. of ICDT, 2001.
[4] M. Al-Shedivat, A. G. Wilson, Y. Saatchi, Z. Hu, and E. P. Xing. Learning scalable deep kernels with recurrent structure. The Journal of Machine Learning Research (JMLR), 2017.
[5] D. Alvarez-Melis and T. S. Jaakkola. Towards robust interpretability with self-explaining neural networks. In Proc. of NeurIPS, 2018.
[6] C. Andrieu, N. De Freitas, Doucet, and M. Jordan. An introduction to meme for machine learning. Machine learning, 2003.
[7] J. A. Arjona-Medina, M. Gillhofer, M. Widrich, T. Unterthiner, J. Brandstetter, and S. Hochreiter. Rudder: Return decomposition for delayed rewards. In Proc. of NeurIPS, 2019.
[8] ATARI. Atari games. https://wws awwri. com/ 2006.
[9] A. Atrey, K. Clary, and D. Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning. In Proc. of ICLR, 2020.
[10] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR, 2015.
[11] D. Balaban. How ai is mishandled to become a cybersecurity risk. https://www.enne eweek co m/ //seeritty/oooaiiiii.
mi shandled to- a
2021.
[12] C. Bass, M. da Silva, C. Sudre, P.-D. Tudosiu, S. Smith, and E. Robinson. Icam: Interpretable classification via disentangled representations and feature attribution mapping. In Proc. of NeurlPS, 2020.
[13] O. Bastani, Y. Pu, and A. Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Proc. of ICML, 2018.
[14] Bewley and Lawry. Tripletree: versatile interpretable representation of black box agents and their environments. In Proc. of AAAI, 2021.
[15) G. Broekman, V. Cheung, L.. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint 1rXiv:1606.015400 2016.
[16] O.-M. Camburu, T. Rocktäschel, T. Lukasiewicz, and P. Blunsom. e-snli: Natural language inference with natural language explanations. In Proc. of NeurlPS, 2018.
[17] S. Chang, Y. Zhang, M. Yu, and T. Jaakkola. Invariant rationalization. In Proc. of ICML, 2020.
[18] C. Chen, O. Li, C. Tao, A. Barnett, Su, and Rudin. This looks like that: deep learning for interpretable image recognition. In Proc. of NeuriPS, 2019.
[19] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.01345, 2021.

---

[20] X. Chen, Z. Wang, Y. Fan, B. Jin, P. Mardziel, C. Joe- Joee-Wong, and A. Datta. Towards behavior- level explanation for deep reinforcement learning. arXiv preprint 2xXiv:200908800,, 2020.
[21] L. Chu, X. Hu, J. Hu, L. Wang, and J. Pei. Exact and consistent interpretation for piecewise linear neural networks: A closed form solution. In Proc. of KDD, 2018.
[22] Y. Coppens, K. Efthymiadis, T. Lenaerts, A. Nowé, T. Miller, R. Weber, and D. Magazzeni. Distilling deep reinforcement learning policies in soft decision trees. In Proc. UCCA Workshop on XAI, 2019.
[23] M. H. Danesh, A. Koul, A. Fern, and S. Khorram. Re-understanding finite-state representations of recurrent policy networks. In Proc. of ICML, 2021.
[24] DeepMind. Alphastar: Mastering the real- time strategy game stareraft ii. odia.org/wiie http:///wwiiaaasttaaraaaa (software) 2017.
[25] D. Duvenaud, H. Nickisch, and C. E. Rasmussen. Additive gaussian processes. arXiv preprint 11X1v:1112.4394, 2011.
[26] S. Eleftheriadis, T. Nicholson, M. P. Deisenroth, and Hensman. Identification of gaussian process state space models. In Proc. of NeurlPS, 2017.
[27] G. F. Elsayed, Q. V. Le, and S. Kornblith, Saccader: Accurate, interpretable image classification with hard attention. In Proc. of NeurlPS, 2019.
[28] R. C. Fong and A. Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proc. of ICCV, 2017.
[29] R. Frigola, Y. Chen, and C. E. Rasmussen. Variational gaussian process state-space models. In Proc. of NeurlPS, 2014.
[30] A. Ghorbani, A. Abid, and J. Zou. Interpretation of neural networks is fragile. In Proc, of AAAI, 2019.
[31] A. Gleave, M. Dennis, N. Kant, C. Wild, S. Levine, and S. Russell. Adversarial policies: Attacking deep reinforcement learning. In Proc. ICLR, 2020.
[32] S. Greydanus, A. Koul, J. Dodge, and A. Fern. Visualizing and understanding atari agents. In Proc. of ICMI., 2018.
[33] S. Gu, E. Holly. T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipula- tion with asynchronous off- off-policy updates. In Proc. of ICRA, 2017.
[34] T. Guo, T. Lin, and N. Antulov-Fantulin. Exploring interpretable Istm neural networks over vulti-variable data. In Proc. of ICML, 2019.
[35] W. Guo, S. Huang. Y. Tao, X. Xing, and L. Lin. Explaining deep learning models-a bayesian non-parametric approach. Proc. of NeurIPS, 2018.
[36] W. Guo, D. MU, J. Xu, P. Su, G. Wang, and X. Xing. Lemna: Explaining deep learning based security applications. In Proc. of CCS, 2018.
[37] W. Guo, X. Wu, S. Huang, and X. Xing. Adversarial policy learning in two-player competitive games. In Proc. of ICML, 2021.
[38] T. Haarnoja, V. Pong, A. Zhou, M. Dalal, P. Abbeel, and S. Levine. Composable deep reinforce- ment learning for robotic manipulation. In Proc. of ICRA, 2018
[39] J. Heo, H. B. Lee, S. Kim, J. Lee, K. J. Kim, E. Yang, and S. J. Hwang. Uncertainty-aware attention for reliable interpretation and prediction. In Proc. of NeurlPS, 2018.
[40] S. Hooker, D. Erhan, P.-J. Kindermans, and B. Kim. A benchmark for interpretability methods in deep neural networks. In Proc. of NeurIPS, 2019.
[41] S. Huang, N. Papernot, I. Goodfellow, Y. Duan, and P. Abbeel. Adversarial attacks on neural network policies. In Proc. of ICLR workshop, 2017.
12

---

[42] A. Hüyük, D. Jarrett, C. Tekin, and M. Van Der Schaar. Explaining by imitating: Understanding decisions by interpretable policy learning. In Proc. of ICLR, 2021.
[43] A. A. Ismail. M. Gunady, L. Pessoa, H. C. Bravo, and S. Feizi. Input-cell attention reduces vanishing saliency of recurrent neural networks. In Proc. of NeurIPS, 2019.
[44] R. Iyer, Y. Li, H. Li, M. Lewis, R. Sundar, and K. Sycara. Transparency and explanation in deep reinforcement learning neural networks. In Proc. of AIES, 2018.
[45] S. Jain and B. Wallace. Attention is not explanation. In Proc. of NAACL, 2019.
[46] Z. Juozapaitis, A. Koul, A. Fern, M. Erwig, and F. Doshi-Velez. Explainable reinforcement learning via reward decomposition. In Proc. of IJCAI Workshop XAI, 2019.
[47] O. Katz. Explainability: The next frontier for artificial intelligence insurance and banking.
tex-tett-e-eeeeeeeee-ee https://www, unite. wttp::uwww.t.tii.iieeeee wwwpxppaaaaaaaiiitt....aaaaaa theiitttythe--eeeee next www.tfer-otooo--aaeeeeee- forttierr-o---tiiiccaa.l--------
ntelligence- in-innurance and-rraanee-add--aae-a wanddaaniing//////a 2021.
[48] V. R. Konda and N. Tsitsiklis. Actor-critic algorithms. In Proc. of NeurlPS, 2000.
[49] A. Koul, S. Greydanus, and A. Fern. Learning finite state representations of recurrent policy networks. arXiv preprint arXiv:1811.12534 2018.
[50] G.-H. Lee, W. Jin, D. Alvarez-Melis, and T. Jaakkola. Functional transparency for structured data: game-theooo theoretic approach. In Proc. of ICML, 2019.
[51] T. Lei, R. Barzilay, and T. Jaakkola. Rationalizing neural predictions. In Proc. of EMNLP- IJCNLP, 2017.
[52] O. Li, H. Liu, C. Chen, and C. Rudin. Deep learning for case-based reasoning through prototypes:
A neural network that explains its predictions. In Proc. of AAAI, 2018.
[53] Liang, Y. Wu, D. Xu, and V. Honavar. Longitudinal deep kernel gaussian process regression. In Proc. of AAAI, 2021.
[54] Z. Lin, K.-H. Lam, and A. Fern. Contrastive explanations for reinforcement learning via embedded self predictions. In Proc. of ICLR, 2021.
[55] G. Liu, O. Schulte, W. Zhu, and Q. Li. Toward interpretable deep reinforcement learning with linear model u-trees. Proc. of ECML-PKDD,, 2018.
[56] Y. Y. Lu, W. Guo, X. Xing, and W. S. Noble. Dance: Enhancing saliency maps using decoys. In Proc. of ICML, 2021.
[57] C. J. Maddison, A. Mnih, and W. Teh. The concrete distribution: continuous relaxation of discrete random variables. In Proc. of ICLR, 2017.
[58] Madumal, T. Miller, L. Sonenberg, and Vetere. Distal explanations for model-free explainable reinforcement learning. arXiv preprini arXiv:2001.10284 2020.
[59] Madumal, T. Miller, L. Sonenberg, and F. Vetere. Explainable reinforcement learning through a causal lens. In Proc. of AAAI, 2020.
[60] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 2015.
[61] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard. Universal adversarial perturba- tions. In Proc. of CVPR, 2017.
[62] A. Mott, D. Zoran, M. Chrzanowski, D. Wierstra, and D. J. Rezende. Towards interpretable reinforcement learning using attention augmented agents. In Proc. of NeurlPS, 2019.
[63] K. P. Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
13

---

[64] W. Nie, Y. Zhang, and A. Patel. A theoretical explanation for perplexing behaviors of backpropagation-basee based visualizations. In Proc. of ICML, 2018.
[65] OpenAl. Openai at the international 2017. https://openai..omm hww.:paai.ooomml-----eeeaattttao 2017.
[66] Pedapati, A. Balakrishnan, K. Shanmugam, and A. Dhurandhar. Learning global transparent models consistent with local contrastive explanations. In Proc. of NeurlPS, 2020.
[67] G. Pleiss, M. Jankowiak, D. Eriksson, A. Damle, and J. R. Gardner. Fast matrix square roots with applications to gaussian processes and bayesian optimization. In Proc. of NeurlPS, 2020,
[68] N. Puri, S. Verma, P. Gupta, D. Kayastha, S. Deshmukh, B. Krishnamurthy, and S. Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. In Proc. of ICLR, 2020.
[69] Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In Proc. of ICML, 2015.
[70] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization 2017.
algorithms. arXiv preprint
[71] Q. She and A. Wu. Neural dynamics discovery via gaussian process recurrent neural networks. In Proc. of UAI, 2020.
[72] W. Shi, S. Song, Z. Wang, and G. Huang. Self-supervised discovering of causal features:
Towards interpretable reinforcement learning. arXiv preprint w:Xi::2003.07069 2020.
[73] T. Shu, C. Xiong, and R. Socher. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning, arXiv preprint arXiv:1712.07294, 2017.
[74] D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
[75] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre,, D. Kumaran, T. Graepel, et al. Mastering chess and shogi by self-play with general reinforce- ment learning algorithm. arXiv preprint arXiv: 2017.
[76] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. 13X2v:1312..634, 2013.
[77] D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg. Smoothgrad: removing noise by adding noise. 3xXiv1776600822. 2017.
[78] S. Srinivas and Fleuret. Rethinking the role of gradient-based attribution methods for model interpretability. In Proc. of ICLR, 2021.
[79] P. Sturmfels, S. Lundberg, and S.-I. Lee. Visualizing the impact of feature attribution baselines. Distill, 2020.
[80] M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In Proc. of ICML,, 2017.
[81] A. Svensson, A. Solin, S. Särkkä, and T. Schön. Computationally efficient bayesian learning of gaussian process state space models. In Proc. of AISTATS, 2016.
[82] Y. Tang, D. Nguyen, and D. Ha. Neuroevolution of self seff-interpretable agents. In Proc. of GECCO, 2020.
[83] S. Thrun. Monte carlo pomdps. In Proc. NeurlPS, 2000.
[84] E. Todorov, T. Erez, and Tassa. Mujoco: physics engine for model- control. In Proc. ICIRS, 2012.
14

---

[85] N. Topin and M. Veloso. Generation of policy-level explanations for reinforcement learning, In Proc. of AAAI, 2019.
[86] H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In Proc. of AAAI, 2016.
[87] A. Verma, V. Murali, R. Singh, P. Kohli, and S. Chaudhuri. Programmatically interpretable reinforcement learning. In Proc. of ICML, 2018.
[88] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 2019.
[89] S. Wiegreffe and Y. Pinter. Attention is not not explanation. In Proc. of EMNLP-IJCNLP. 2019.
[90] A. Wilson and H. Nickisch. Kernel interpolation for scalable structured gaussian processes (kiss-gp). In Proc. of ICML, 2015.
[91] A. G. Wilson, Z. Hu, R. Salakhutdinov, and E. P. Xing. Stochastic variational deep kernel learning. In Proc. of NeurlPS, 2016.
[92] V. Zambaldi, D. Raposo, A. Santoro, V. Bapst, Y. Li, I. Babuschkin, K. Tuyls, D. Reichert, T. Lillicrap. E. Lockhart, et al. Deep reinforcement learning with relational inductive biases, In Proc. of ICLR, 2018.
[93] H. Zhang, H. Chen, D. Boning, and C.-J. Hsich. Robust reinforcement learning on state observations with learned optimal adversary. In Proc. of ICLR, 2021.
[94] X. Zhang, N. Wang, H. Shen, S. Ji, X. Luo, and T. Wang. Interpretable deep learning under fire. Proc. of USENIX Security, 2020.
15


Reviews:
[{"review_id": "gefWXv1swob", "reviewer": "Reviewer_5o96", "summary": "This paper proposes a new pipeline for understanding agent behavior + actions, and identifying which timesteps contributed to the final reward. The key component of this method is the pipeline, which consists of the following:\n\n* An RNN which takes in state-action pairs, and \"compresses\" the information via its RNN-state at every timestep.\n* All RNN-states over all time steps are taken and inserted into a zero-mean Gaussian Process (GP), which consists of two independent GPs, one for correlations between time-steps, and another for correlations between episodes.\n* A feature vector $\\mathbf{f}$ is then sampled from this GP, and used to classify the reward. \n\nBecause a GP is involved, variational inference is needed for training.\n\nThe paper then experimentally verifies this result on Atari Pong + Adversarial Mujoco tasks by showing competitive performance on fidelity (i.e. metric based on replacing an intelligent agent with random actions) and visualizing which parts of an episode contribute most to the fidelity score. Finally, the method is also applied to successfully perform adversarial attacks which also beat previous baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* This method is applied to scenarios where access to the agent policy is restricted (or might not even be a neural network), and in offline settings, which makes it broad and useful in many settings, compared to saliency (which is specific to neural networks).\n* The experiments seem convincing and strong compared to saliency-based methods, and visualizations confirm the validity of the method.\n\n\nCons:\n* The entire pipeline is pretty large and complex, and involves many different components. It took quite a while for me to understand the overall picture of what was happening, mainly from staring at Figure 1 and matching with the text. There's also a lot of fine-grained details in Sections 3.2 + 3.3, some of which should be moved into the appendix - is it possible to explain the method at a high level using brief math instead? I think much of the method can be simplified for the average reader to digest.\n* The paper requires a somewhat restrictive assumption, which is that in every episode, there are specific timesteps that contributed to the overall reward. I feel like this is only applicable to a limited set of RL environments, such as Pong and Adversarial Mujoco, since they only produce a +1/-1 reward at the very end of the episode. Would this method work for environments that always produce rewards at every timestep? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "L9pQdOseK99", "reviewer": "Reviewer_1XkV", "summary": "This paper presents an explainable method for deep reinforcement learning agents. A gaussian process is used to capture the correlations and the effects between episodes, and is then used to predict the final reward. Further, the authors introduce a method (EDGE) that can provide high-level (strategy-level) explanations of the agent behaviour. Evaluation is done in 5 RL game domains, with results showing higher fidelity against the compared baseline methods. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is novel, in that it tries to identify the importance of a time step. This is in contrast to most other Rl explainability methods that are focused on feature/state based explanations. This explainable method enables the understanding of the agents’ policy, and can be used to identify policy errors and weaknesses. The work is positioned well in the explainable RL literature, through a well summarized related section. The technical definitions and equations appear to be sound.\n \nThe main weekend I see in this paper is in the limited selection of evaluation domains. It is doubtful whether the strength of the proposed model can be demonstrated in these environments (especially in the pong env), as the time step frames can look much similar to each other. This limitation is exacerbated due to the lack of a user-study to evaluate how useful the time-step importance explanations are against other explainable models. In Section 5, authors give their own interpretations of the explanation frames, but this is subjective and can vary from user to user. I encourage authors to add a discussion to address this issue. Other evaluations that focus on policy errors are reported and explained well.  \n\nQuality: \nThe technical definitions and equations are sound. As mentioned above evaluation suffers from some limitations. I believe these issues can be addressed with a careful commentary on the nature of the interpretability of the proposed method from a user perspective (or better yet a user study).\n \nClarity:  \n \nThe Paper is well written and is easy to follow. Methods are defined well enough for the reader to easily follow and understand.\n \nSignificance: \n \nThis paper has a clear contribution that is in contrast to many other explainable RL methods. The use of importance of the time-step is a novel direction. Although there are somewhat similar methods that use policy summarization [1], this method explicitly focuses on the importance of time-steps. The evaluation hinders this paper from highlighting the importance of the contributions. In its current form, it is not clear how these explanations are better than the other types of RL explainability methods (saliency, policy-based etc). The paper can significantly be strengthened by providing evaluations in other domains (can be other Open AI gym envs) that can have more strategic agents. I do believe this approach has promise and would be of great importance to the larger explainable RL community.\n \n \n \n[1] Amir, D., & Amir, O. (2018, July). Highlights: Summarizing agent behavior to people. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (pp. 1168-1176).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main technical limitation lies in the evaluation as explained above. The authors have discussed the societal impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "To the best of my knowledge, this work does not have any immediate ethical concerns.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "9AO46XEfAa", "reviewer": "Reviewer_mSsX", "summary": "This paper presents a method that uses the correlation between different episodes as well as the correlation between states within an episode to provide explanations on deep RL agents. The explanation module mainly consists of an RNN module and deep gaussian processes. Later in the paper, the authors show that the provided explanations could be used to attack deep RL agents, make them more robust, and repair them.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea presented in the paper is interesting, and the approach seems promising. However, there are a few points that need to be addressed in order to increase the quality of the paper. They are mentioned below.\n\nIn methods where an attention module is embedded in the deep RL agent's architecture, provided explanations could be easily misleading because the soundness of such approaches is not apparent. The reason for the unclear inferred explanation is that the attention module parameters are affected by the input observations and memory of the agent, which are ignored when an interpretation is given. This misleading explanation makes it hard to find if the attention patterns embed strategic information or just information highlighted by the attention that is key to decisions.\n\nRegarding lines 56-57: The authors claim that their proposed method is the first to explain deep RL policies from a sequence of its past actions and states. However, that is not an accurate claim. [1] and [2] both try to interpret deep RL policies with respect to the actions the agent has taken and the values it has in its memory, resulting in interesting insights into the workings of deep RL agents. \n\nRegarding lines 79-80: How do authors compare their method to [2]? Because in [2], the time steps in which critical decisions are made are also identified and investigated.\n\nRegarding lines 269-271: If the policy has a memory in any form of RNN, this is simply invalid because manipulating the trajectory of $(t_i, ..., t_{i+l})$ would affect the policy's memory.  So simply feeding state at $i+l+1$ would not be sound.\n\nRegarding line 275: How does $d_{max}$ is calculated?\n\nRegarding line 277: How about normalizing the first term, $log(p_l)$, of the equation by a coefficient in $[0,1]$? The reason is that it could have the most effects on the fidelity score, thus undermining the second term, $log(p_d)$.\n\nRegarding Section 5: Many other things could be extracted from these sorts of explanations (lines 298-299 and 301-302). For example, one could conclude that the agent is just focusing on hitting the ball, thus not losing, rather than focusing on hitting it in a special way. We need to be very careful about our conclusions.\n\nRegarding lines 297-298: It is mentioned that \"EDGE highlights the time steps when the agent hits the ball as the key steps leading to a win.\" Is this claim consistent across different policies?\n\nRegarding line 307: It is mentioned that the examples extracted from three games are provided in the Supp material. However, to show the soundness of the approach, a diverse set of games should be investigated.\n\nRegarding Table 1: Please provide variance as well as the mean.\n\nRegarding lines 312-313: How about strategically timed attacks? They do the same without the need for any explanations [3]. A comparison between the two methods could improve the claim in the paper.\n\nRegarding Patching Policy Errors: It is mentioned that a table could be formed from the state-action pairs to be used as a remediation policy. How is it possible to form a table from such rich and continuous observations and hoping to see them again? The observation space should become finite in a way to make it make sense.\n\n----------------------------------------------------\nGrammar, typos, etc.:\n\nLine 5: \"augments an Gaussian process\" -> \"augments a Gaussian process\"\n\nLine 16: \"success in automatically policy learning\" -> \"success in automatic policy learning\"\n\nLine 28: \"existing methods can not shed\" -> \"existing methods cannot shed\"\n\nLine 204: \"likelihood is computational expensive\" -> \"likelihood is computationally expensive\"\n\nLine 344: \"patched policy introduce too many\" -> \"patched policy introduces too many\"\n\nLine 382: \"demonstrates the our method’s\" -> \"demonstrates our method’s\"\n\n----------------------------------------------------\nReferences:\n\n[1] Koul, Anurag, Sam Greydanus, and Alan Fern. \"Learning finite state representations of recurrent policy networks.\" arXiv preprint arXiv:1811.12530 (2018).\n\n[2] Danesh, Mohamad H., et al. \"Re-understanding Finite-State Representations of Recurrent Policy Networks.\" International Conference on Machine Learning. PMLR, 2021.\n\n[3] Lin, Yen-Chen, et al. \"Tactics of adversarial attack on deep reinforcement learning agents.\" arXiv preprint arXiv:1703.06748 (2017).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations of their proposed method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3-4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new approach to explaining deep reinforcement learning policies. The approach applies to complex or black box policies, and learns Gaussian Process (GP) model with custom function to learn to predict and assess the importance of state-action pairs on game outcomes. The authors illustrate feasibility of their approach in Atari and MuJoCo domains and demonstrate several application areas.

Initial reviews assessed the contribution as strong, especially because of the method's wide applicability and strong experimental evidence. Several concerns were raised as well, including clarity (e.g., the complexity of the method and need for more precise explanations) and concerns about potentially restrictive assumptions. Suggestions for improvements included additional empirical evaluations, including a proposed user study to test the effectiveness of the approach. The discussion between reviewers and authors has been highly productive, with many concerns addressed. The authors went above and beyond by running the suggested user study during the rebuttal period. 

As a result of the productive discussion, all reviewers indicated that their concerns have been largely addressed and reached the consensus to recommend acceptance. The AC agrees with this recommendation. Given that substantial new insights were developed during the discussion, the authors are strongly encouraged to carefully incorporate all suggestions as they prepare the camera ready version.
Decision: Accept (Poster)