Contrastive Explanations for Reinforcement Learning via Embedded Self Predictions
Authors: Zhengxian Lin
Kin-Ho Lam
Alan Fern
Venue: ICLR 2021 Oral (2021)
Keywords: Explainable AI
Deep Reinforcement Learning

Abstract: We investigate a deep reinforcement learning (RL) architecture that supports explaining why a learned agent prefers one action over another. The key idea is to learn action-values that are directly represented via human-understandable properties of expected futures. This is realized via the embedded self-prediction (ESP) model, which learns said properties in terms of human provided features. Action preferences can then be explained by contrasting the future properties predicted for each action. To address cases where there are a large number of features, we develop a novel method for computing minimal sufficient explanations from an ESP. Our case studies in three domains, including a complex strategy game, show that ESP models can be effectively learned and support insightful explanations. 

Published as a conference paper at ICLR 2021
CONTRASTIVE EXPLANATIONS FOR REINFORCEMENT LEARNING VIA EMBEDDED SELF PREDICTIONS
Zhengxian Lin, Kim-Ho Lam, Alan Fern
Department of EECS
Oregon State University (1 (linzhe, lamki, alan. fern) @oregonstate.edi e
ABSTRACT
We investigate deep reinforcement learning (RL) architecture that supports ex- plaining why a learned agent prefers one action over another. The key idea is to learn action-values that are directly represented via uummannuuddrstaddable proper- ties of expected futures. This is realized via the embedded self- seff-predittion (ESP) model, which learns said properties in terms of human provided features. Action preferences can then be explained by contrasting the future properties predicted for each action. To address cases where there are large number of features, we develop a novel method for computing minimal sufficient explanations from ESP. Our case studies in three domains, including a complex strategy game, show that ESP models can be effectively learned and support insightful explanations.
INTRODUCTION
Traditional RL agents explain their action preference by revealing action A B's predicted values, which provide little insight into its reasoning, Conversely, a human might explain their preference by contrasting meaningful properties of the predicted futures following each action. In this work, we de- velop model allowing RL agents to explain action preferences by contrasting human- -umnan-underttadablle future predictions. Our approach learns deep generalized value functions (GVFs) (Sutton al.) 2011) to make the future predictions, which are able to predict the future accumulation of arbitrary features when following a policy. Thus, given human-understandable features, the corresponding GVFs capture meaningful properties of policy's future trajectories.
To support sound explanation of action preferences via GVFs, it is important that the agent uses the GVFs to form preferences. To this end, our first contribution is the embedded pelf-prediction (ESP) model, which: I) directly "embeds" meaningful GVFs into the agent's 's action-valus function, and 2) trains those GVFs to be "self preeifpredittigg" of the agent's O-function maximizing greedy policy. This enables meaningful and sound contrastive explanations in terms of GVFs. However, this circularly defined ESP model, i.e. the policy depends the GVFs and vice-versa, suggests training may be difficult. Our second contribution is the ESP-DQN learning algorithm, for which we provide theoretical convergence conditions in the basee setting and demonstrate empirical effectiveness.
Because ESP models combine embedded GVFs non-linearly, comparing the contributions of GVFs preferences for explanations can be difficult. Our third contribution a novel application of the integrated gradient (IG) (Sundararajan et al. 2017) for producing explanations that are sound in well-defined sense. To further support cases with many features, we use the notion of minimal sufficient explanation (Juozapaitis et al. 2019). which can significantly simplify explanations while remaining sound. Our fourth contribution case studies two RI benchmarks and complex real-time strategy game. These demonstrate insights provide by the explanations including both validating and finding flaws in reasons for preferences.
In Defense of Haaually-Desighee algorithms with engineered meaningful features. The key question is whether the utility of providing such features is worth the cost of their acquisition. We argue that for many applications that can benefit from informative explanations, the utility will outweigh the cost. Without meaningful features, explanations must be expressed visualizations top of lower-level perceptual information (e.g.
Features. It can be controversial to provide deep learning

---

Published as a conference paper at ICLR 2021
(8,a)
(3(5,))
Q(s.a)
Figure 1: The ESP model provides estimate of the agent's Q-function for any state-action pair. The model first maps state-action pair (s,a) a) to a GVF vector Q of the agent's greedy policy ((s) = Q(s.a).. a). This vector is then processed by the combining function Č, which produces a Q-value estimate Q(s a). The embedded GVF is self-predicting in the sense that it is predicting values of the greedy policy for which it is being used compute.
saliencylattention maps). Such explanations have utility, but they may not adequately relate to human- understandable concepts, require subjective interpretation, and can offer limited insight. Further, in many applications, meaningful features already exist and/or the level of effort to acquire them from domain experts and Al engineers is reasonable. It is thus important to develop deep learning methods, such as our ESP model, that can deliver enhanced explainability when such features are available.
2 EMBEDDED SELF-PREDICTION MODEL
An MDP is tuple (S, A, Rt,, with states S, actions A, transition function T(s, a, s'), and reward function R(s, a). A policy æ maps states to actions and has Q-function (s, a) giving the expected infinite-horizon 3-discounted reward of following и after taking action a in s. The optimal policy n² and Q-function Q* satisfy x" (s) = arg max, Q' a). Q' can be computed given the MDP by repeated application of the Bellman Backup Operator, which for any Q-function Q, returns a new Q-function [s a) R(s, a) E. T(s, a, s') max,' Q(s',a'),
We focus on RL agents that learn an approximation Q of Q" and follow the corresponding greedy policy A(s) = arg max, (ss a). We aim to explain preference for action a over 6 in a state s, i.e explain why ((s a) > ((, b). Importantly, the explanations should be meaningful to humans and soundly reflect the actual agent preferences. Below, we define the embedded self eelf-rredcction model, which will be used for producing such explanations (Section in terms of generalized value functions.
Generalized Value Functions (GVFs). GVFs (Sutoon et al. 2011) are a generalization of traditional value functions that accumulate arbitrary feature functions rather than reward functions. Specifically, given a policy and a discount factor 7. the corresponding n-diiennsonaa GVF, denoted (7, a), is the ex- pected infinite-horizon 7-discounted accumulation of F when following
n--iimensional state-action feature function F(s, a) (fi (s, ...,,... (s, a)).
after taking in s. Given an MDP, policy n, and feature function F, the GVF can be computed by iterating the Bellman GVF operator, which takes a GVF Qr and returns new GVF B7 (r,)(s ((,,a) + E T(s,a, a, x')Qr(s', Tss'))
To produce human- -unrrasnnnndettaballlleeee explanations, assume semantically-meanteee available, so that the corresponding GVFs deseribe meaningful properties the expected future-e.. expected energy usage, time spent in particular spatial region, future change in altitude.
features
ESP Model Definition. Given policy π and features F, we can contrast actions and b via the GVF difference (s, a, b) = (s, a) - Q7- b), which may highlight meaningful differences in how the actions impact the future. Such differences, however, cannot necessarily be used to soundly explain an agent preference, since the agent may not explicitly consider those GVFs for action selection. Thus, the ESP model forces agents to directly define action values, and hence preferences, in terms of GVFs of their own policies, which allows for such differences to be used soundly.
As depicted in Figure 1 the ESP model embeds a GVF O of the agent's greedy policy # into the agents Q-function Q, via ((s, a) Č(Qr(s, a)), where Ć Rn R is learned combining function from GVF vectors to action values. When the GVF discount factor y is zero, the ESP model becomes a direct combination of the features, i.e. Q(s,, a) = ((F(s, a)), which is the traditional approach to using features for function approximation. By using 7 0 leverage human-
2

---

Published as a conference paper at ICLR 2021
provided features in potentially more powerful way. Because an ESP agent represents action- values via GVF components, it is possible to produce sound contrastive explanations in terms of GVFs, as described in Section 4
In general, the ability to learn a quality Q-function and hence policy using the ESP model requires that the GVF features are sufficiently expressive. While, in concept, using a single feature equal to the reward is sufficient for learning the Q-function (i.e. the GVF is the Q-function and the identity combining function could be used), that choice does not support explainability. Thus, it is desirable to use set of features that meeningfully decompose important aspects of the environment and at the same time have GVFs that are expressive enough to combine into the Q-function. In Section 6 we describe the generic schema used for GVF features in our experimental environments.
ESP MODEL TRAINING: ESP-DQN
We will represent the learned combining function, Õ, and GVF, Qr. as neural networks with parameters 8c and 0p. The goal is to optimize the parameters so that Q(s. a) = C((r(s,)) a)) approximates Q" and ( a) approximates 0F as) The GVF accuracy condition is important since humans will interpret the GVF values in explanations. A potential learning complication is the circular dependence where Q is both an input to Q and depends on Q through the greedy policy ft Below we overview our learning algorithm, ESP-DQN, a variant of DQN (Mnih et al. 2013), which we later show to be empirically effective. Full pseudo-code is provided in the Appendix A
ESP-DQN follows an e-greedy exploration policy while adding transitions to a replay buffer D = {(s;, Ti, Fi, sf)}. where F; is the feature vector for GVF training. Each learning step updates Oc and O using a random mini- batch. Like DQN, updates are based on a target network, which uses a second set of targel parameters 0' and 0'p, defining target combining and GVF functions O' and Q'F- yield target Q-function Q' a) E² (Q's a)). The target parameters are updated to the values of the non-target parameters every learning steps and otherwise held fixed.
Combination Function Update. Since the output of Ō should approximate Q", optimizing Oc can use traditional DQN updates. The updates, however, only impact Ic while keeping 0F fixed so that the GVF output (F a) is viewed as a fixed input to C. Given batch the update to AC is based 1.2 loss with a target value for sample being y/- = r, 3Q' (5/ 04), where á² = arg max. Q' (ss' a) is the greedy action of the target network.
GVF Update. Training Q7 is similar to learning critic in actor-critic methods for the evolving greedy policy, but instead of learning to predict long-term reward, we predict the long-term accumu- lation of F. Given a mini-batch we update Op based on L2 loss at the output of QF F with respect to a target value y = F. + (5p( at), where à is the same target greedy action from above.
Convergence. Even with sufficiently expressive features, most combinations of function approx- imation and Q-learning, including DQN, do not have general convergence guarantees (Sutton & Barto 2018). Rather, for table-based representations that record a value for each state-action pair, Q-learning, from which DQN is derived, almost surely converges to Q" (Watkins & Dayan 1992), which at least shows that DQN is built sound principles. We now consider convergence for ESP-Table, a table-based analog of ESP-DQN.
Table uses size 1 mini-batches and updates target tables (i.e. analogs of target networks) every K steps. The QF table is over stale-action pairs, while for Ć' we assume a hash function h that maps its continuous GVF inputs to finite table. For example, since GVFs are bounded, this can be done with arbitrarily small error via quantization. pair of feature and hash function (F, h) must be sufficiently expressive to provide any convergence guarantee. First, we assume h is locally consistent, meaning that for any input q there exists a finite e such that for all /4' gl S <, h(()) = lu(g'). Second, we assume the pair (F, h) is Bellman Sufficient, which characterizes the representational capacity of the Ć table after Bellman GVF backups (see Section 2 with respect representing Bellman backups.
Definition 1 (Bellman Sufficiency). A feature and hash function pair (F, h) is Bellman sufficient if for any ESP model Q(+, a) = C((p(s, a)) with greedy policy ft and state-action pairs (s, a) and (x. v), if hi ( (s, = h(t(1,, v)) then B[ a) B (xx,)) v), where Q B3 (Q)
3

---

Published as a conference paper at ICLR 2021
Let Č" Q/r, Q', and #' be random variables denoting the learned combining function, GVF, cor- responding Q-function, and greedy policy after t updates. The following gives conditions for convergence of #' to n² and Q' to a neighborhood of Q7 given a large enough update interval K.
Theorem 1. ESPPPP is run under the standard conditions for the almost surely (a.s.) convergence of Q-learning and uses Bellman-ste sulliman-uufiiient pair (F, h) with locally consistent h, then for any c > 0 there exists a finite target update interval K, such that for all s and a, R (s) converges a.s. to n (s) and lime-soa (8,) - (7 a)l S with probability 1.
The full proof is in the Appendix
B It an open problem of whether stronger convergence result holds for K = 1, which would be analogous to results for traditional Q-learning.
4 CONTRASTIVE EXPLANATIONS FOR THE ESP MODEL
We focus on contrastive explanation of a preference, (ss.a) > (ss b), that decomposes the preference magnitude (ss.a) Q(s.)) b) in terms of components of the GVF difference vector Ar(s, a,,,a,b) b) Qr(s,a) Qr(s,) b). Explanations will be tuples (Ar(s, b),, ((s, a, b)) where ((s, a, b) E Rd is an attribution weight vector corresponding to Ar(s, a, b). The meaningfulness of an explanation largely determined by the meaningfulnness of the GVF features. We say that an explanation is sound if ((s. a) ((s b) (s,, a. b) (s, a. b), i.e. it accounts for the preference magnitude. We are interested in explanation methods that only return sound explanations, since these explanations can be viewed as certificates for the agent's preferences. In particular, the definition implies that W(s, a, b) A (s, a. b) > 0 if and only if Q(s, a) @ b). In the simple case of a linear combining function O with weights w E R", the preference magnitude factors as ((s,a) a) - ((,, b) = w A a, b). Thus, (A (s. a, b), w) is a sound explanation for any preference.
Non-Linear Combining Functions. Non-linear combining functions difficult to provide features that support good policies via linear combining functions. Since the above linear factoring does not directly hold for non-linear Õ, we draw on the Integrated Gradient (IG) (Sundararajan et al. 2017), which was originally developed to score feature importance of a single input relative to a "baseline" input. We adapt IG to our setting by treating the less preferred action as the baseline, which we describe below in the terminology of this paper.
necessary when it is
Let Xs = Qr(s a) and Nat = Qr (s. b) be the GVF outputs of the compared actions. Given a differentiable combining function Ć, IG computes an attribution weight (s(s, a, b) for component i by integrating the gradient of Ó while interpolating between X. and X,. That is, 0,(s, a, b) = So xxxx da, which we approximate via finite diffcrences. The key property is that 0Xan,t
the IG weights linearly autributes feature differences to the overall output difference, i.e. C(Xsa)
((xxt) = 0(s,a, b) (Xsa - xx+). Rewriting this gives the key relationship for the ESP model.
((s,a) Q(=, ( =(Qr(s,))) C(Qr(s,))))) b)) A(s a, b) as)
Thus, IGX(s, b) = (Ar(s, a, b), 0(s, a, b)) is a sound explanation, which generalizes the above we have 0(s, a, = w. In practice, we typically visualize IGX(s, a, b) by showing a bar for each component with magnitude 0, a, b) (r(s, a, b), which reflects the positive/negative contributions to the preference (e.g. Figure 3a bottom-right).
linear case, since for linear Ó with weights
Minimal Sufficient Explanations. When there are many features IGX(s, a, b) will likely overwhelm users. To soundly reduce the size, we use the concept of minimal sufficient explanation (MSX), which was recently developed for the much more restricted space of linear reward decomposition models (Juozapaitis et al.] 22019). Equation [ however, allows to adapt the MSX to our non-linear setting. Let P and N be the indices of the GVF components that have positive and negative attribution to the preference, i.e., P {i : A a. b) (s a, b) > 0} and N (1....,m) ... P. Also, for an arbitrary subset of indices E, let S(E) = -iEF (s, a, b) (, a, b)l be the total magnitude of the components, which lets the preference be expressed as S(P) > S(NN. The key idea of the required to overcome negative
MSX is that often only small subset of positive components components and maintain the preference of a over b. An MSX is simply a minimal set of such positive components. Thus, MSX is solution to arg min F
E C P. S(F) S(N)), which
4

---

Published as a conference paper at ICLR 2021
is not unique in general. We select solution that has the largest positive weight by sorting Padd including indices into the MSX from largest to smallest until the total is larger than S(N).
in
5 RELATED WORK
Prior work considered linear reward decomposition models with known weights for speeding up RL (Van Seijen et al. 2017), multi-agent RL (Russell & Zimdars, 2003; Kok & Vlassis 2004), and explanation (Juozapatiis et al. 2019). This is a special case of the ESP model, with GVF features equal to reward components and a known linear combining function. Generalized value function networks (Schlegel et al., 2018) policies) by treating GVFs as features accumulated by other GVFs. Rather, our GVFs are used as input to a combining network, which defines the policy used for the GVF definition. Integrating GVF networks and the ESP model is an interesting direction to consider.
related, but orthogonal, model that combines GVFs (with given
The MSX for linear models was originally introduced for MDP planning (Khan et al. 2009) and
more recently for reward decomposition (Juozapaitis et al., 2019). We extend to the non- non-linear linear case. recent approach to contrastive explanations (Waa al. 2018) extracts properties from policy simulations at explanation time (Waa et al.) 2018). which can be expensive or impossible. Further, the explanations are not sound, since they are not tied to the agent's 's internal preference computation. Saliency explanations have been used in RL to indicate important parts of input images (Greydanus) et al. 2018 Iyer et al. [0118 Gupta et al. 2020 Atrey et al. 2020 Olson et al. 2019). These methods lack clear semantics for the explanations and hence any notion of soundness.
6 EXPERIMENTAL CASE STUDIES
Below we introduce our domains and experiments, which address these questions: 1) (Section 6.2 Can we learn ESP models that perform as well as standard models? 2) (Section Do the learned ESP models have accurate GVFs? 3) (Section 6.3
Do our explanations provide meaningful insight?
6.1 ENVIRONMENT DESCRIPTION
Schema for Selecting GVF Features. Before introducing the environments we first describe the schema used to select GVF features across these environments. This schema can serve as general starting point for applying the ESP model to new environments. In general, episodic environments have two main types of rewards: 1) terminal reward, which occurs at the end of an episode and can depend on the final state, and 2) pre-terminal rewards, which occur during the episode depending on the states and/or actions. Since the value of policy will typically depend on both types of rewards, it is important to have GVF features that capture both terminal and pre-terminal that are potentially relevant and interpretable. Thus, in each domain, as describe below, we include simple terminal GVF features that describe basic conditions at the end of the episode (e.g. indicating if the cart went out-of-bound in Cartpole). In addition, include pre-terminal GVF features that obtained from the state variables of the environment or derived reward variables that are used to compute the reward function, which are typically readily available from a domain description.
Discrete state or reward variables can simply be encoded as indicator GVF features. For continuous state and reward variables we consider two options: a) When a variable has small number of meaningful regions, we can use indicator features for the regions as features. The GVFs then indicate how long the agent in each region; b) We also consider delta GVF features that are equal to the change in variable across a time step. The GVF value for these features can be interpreted as the future change in the variables' values. While we focus on the above generic GVF features in this paper, an agent designer can define arbitrary GVF features based on their intuition and knowledge.
Lunar Lander, We use the standard OpenAl Gym version of Lunar Lander, a physics simulation game where the agent aims to safely land a rocket ship in target region by deciding at each step which of three thrusters (if any) to activate. The raw state variables are the position and velocity vectors and the reward function penalizes crashing, rewards landing in the goal area, and includes other "shaping" reward variables. These variables are all easily extracted from the simulation enviromment. In this domain, the continuous variables do not have intuitively meaningful discretizations, use the delta features as the main features for explanation case studies (ESP-conti, However, to illustrate that learning can be done with the discretization approach in this domain, we also
5

---

Published as a conference paper at ICLR 2021
include results for the continuous features being discretized into 8 uniform bins CESP-discrete.. The pre-terminal features are based on the variables distance-to toaeeeett-gga... goal, velocity, tilt-angle, right landing leg in goal position, left landing leg in goal position, main engine use, side engine use. The terminal feature is an indicator of safely landing.
We use the standard OpenAl Gym Cart Pole environment, a physics simulation where the agent aims to vertically balance a free-swinging pole attached to a cart that can have a force applied to the left or right each step. The state variables include the position, pole angle, and their velocities, and reward is a constant +1 until termination, which occurs when either the pole falls below a certain angle from vertical, moves out of bounds, or after 500 steps.
Cart Pole.
Since the CartPole variables discretize into a small number of intuitively meaningful regions, we consider both discrete (ESP-discrete) and delta encodings (ESP-continuous) of the GVF features. For ESP-discrete, there are 8 pre-terminal GVF features that discretize the state variables into meaningful regions corresponding to an intuitive notion of safety. This includes two indicators for each variable corresponding to cart position, cart velocity, pole angle, and angle velocity. A perfectly balanced pole will always remain in the defined safe regions. These discretized features also double as the terminal features, since they capture the relevant aspects of the state at termination (being in safe or unsafe region). For continuus,,, we have 12 features, the first 8 pre-terminal GVF features corresponding to the delta features of the cart position, cart velocity, pole angle, and angle velocity for left and right sides. The 4 terminal GVF features are indicators of whether the episode ended by moving out-of-bounds to the left right or the pole fell below the termination agle to the left right.
Tug of War. Tug of War (ToW) is an adversarial two-player strategy game we designed using PySC2 for Starcraft 2. ToW is interesting for humans and presents many challenges RL including an enormous state space, thousands of actions, long horizons, and sparse reward (win/loss). A detailed description is in Appendix C ToW is played on a rectangular map divided into top and bottom horizontal lanes. Each lane has two bases structures at opposite ends, one for each player. The first player to destroy one of the opponent's bases in either lane wins. The game proceeds in 30 second waves. By the beginning of each wave, players must decide on either the top or bottom lane, and how many of each type of military production building to purchase for that lane. Purchases are constrained by the player's 's available currency, which is given at fixed amount each wave. Each purchased building produces one unit of the specified type at the beginning of each wave. The units move across the lanes toward the opponent, engage enemy units, and attack the enemy base if close enough. The three types of units are Marines, Immortals, and Banelings, which have a rock-paper paperr--sscisoorss relationship and have different costs. If no base is destroyed after 40 waves, the player with the lowest base health loses. In this work, we trained a single agent against a reasonably strong agent produced via pool-based self- eeff-play learning (similar AlphaStar training Vinyals al. 2019
We present two ToW ESP agents that use 17 and 131 structured GVF features (noting that the 131 features are very sparse). These feature sets are detailed in Appendix E For the 17 feature agent, the pre-terminal features correpsond to the delta damage to each of the four bases by each of the three types of units; allowing GVFs to predict the amount of base damage done by each type of unit, giving insight into the strategy. Note that there is no natural discretization of the numeric damage variables and hence we only consider the delta encoding. The terminal GVF features are indicators of which base has the lowest health at the end of the game and whether the game reached 40 waves. The terminal GVF features encode the possible ways that the game can end. The 131 feature agent extends these features to to keep track of damage done in each lane to and from each combination of unit types along with additional information about the economy.
6.2 LEARNING PERFORMANCE
To evaluate whether using ESP models hurts performance relative to "standard" models we compare against two DQN instances: DQN-full uses the same overall network architecture as ESP-DQN, Le. the GVF network structure feeding into the combining network. However, unlike ESP-DQN, the DQN-full agent does not have access to GVF features and does not attempt to train the GVF network explicitly. It is possible DQN-full will suffer due to the bottleneck introduced at the interface between the GVF and combiner networks. Thus, we also evaluate Vanilla DQN, which only uses the combining network of ESP-DQN, but directly connects that network to the raw agent input. Details of network architectures, optimizers, and hyperparameters are in the Appendix D

---

Published as a conference paper at ICLR 2021
Vin Ba
ESP disreooouunn)
Veeile DON
DON--ull
DON-tut
Random
-00
ESP (SP-dkcand))oung
Random
500
Lurar Lander OoT's Lees
Confolx GeTa
Wa Cot
ESP continucus(uuss)
ESP dscrrtoooo/))ss)
discrete(ours)
ESP conp-ccotnuuss/urs)
42
1590
(a) Lunar Lander
(b) Cart Pole
(c) Tug-of-war
Figure 2: Reward learning curves (top row) and GVF Loss learning curves (bottom row) for the
different agents in three environments. We show the mean +/- std over 10 independent runs.
Figure 2 (top row) shows the learning curves for different agents and for the random policy. All curves are averages of 10 full training runs from scratch using 10 random seeds. For the control problems, CartPole (with discrete and continous GVFs features) and LunarLander, we see that all agents are statistically indistinguishable near the end of learning and reach peak performance after about the same amount of experience. This indicates that the potential complications of training the ESP model did not significantly impact performance in these domains. We see that the discrete feature version of CartPole converged slightly faster than the continous version, but the difference relatively small. For ToW, the ESP-DQN agents perform as well or better than the DQN variants, with all agents showing more variance. ESP-DQN with 17 features consistently converges to a win rate of nearly 100% and is more stable than the 131-fcature version and other DQN variants. Interestingly, DQN-full with 17 features consistently fails to learn, which we hypothesize is due to the extreme 17 feature bottleneck inserted into the architecture. This is supported by seeing that with 131 features DQN-full does learn, though more slowly than ESP-DQN.
To evaluate the GVF accuracy of ESP-DQN curves. Specifically, given the ESP policy á at any point, we can use Monte-Carlo simulation to estimate (7, a) for all actions at test set of states generated by running म. Figure 2 (bottom row) shows the mean squared GVF prediction error on the test sets as learning progresses. First, for each domain the GVF error is small at the end of learning and tends to rapidly decrease when the policy approaches its peak reward performance. aunarLander and ToW show continual decrease of GVF error as learning progresses. CartPole, rather shows sharp initial increase then sharp decrease. This is due to the initially bad policy always failing quickly, which trivializes GVF prediction. As the policy improves the GVFs become more challenging to predict leading to the initial error increase.
produce ground truth GVF data along the learning
6.3 EXAMPLE EXPLANATIONS
Appendix F includes a larger set of examples with detailed analysis in each domain.
Lunar Lander. In Figure Baj the game state (top) shows a state in Lunar Lander entered by a near-optimal learned ESP policy. The state is dangerous due to the fast downward and clockwise rotational velocity depicted by arrows. The GVFs (bottom-left) shows the Q-values for the actions and the predicted GVF bars. We see that the "main engine" and "right engine" actions have nearly the same Q-values with "main engine" slightly preferred, while "Ieft engine" and "noop" are considered significantly worse We want to understand the rationale for the strong and weak preferences.
While a user can observe differences among GVFs across actions, it is not clear how they relate to the reference. The IG and MSX (bottom-right) shows the IGXs corresponding to the preference of "main engine" over the other three actions. In addition, the MSX is depicted via dashed lines over IGX components in the MSX. Focusing first the larger preferences, "main engine" is preferred to
7

---

Published as a conference paper at ICLR 2021
Drginn Store
Dawarecc Siea
77 Mia
(a) Lunar Lander
(b) Cart Pole
Figure 3: Explanation examples for Lunar Lander (left) and CartPole (right). Each example shows the game state, the Q-values and GVF predictions for actions, and the IGX and MSX.
"left engine" primarily due to GVF differences in the velocity and landing features, with the MSX showing that landing alone is sufficient for the preference. This rationale agrees with common sense, since the left engine will accelerate the already dangerous clockwise rotation requiring more extreme actions that put the future reward related landing at risk.
For the preference over "noop" the velocity feature dominates the IGX and is the only MSX feature. This agrees with intuition since by doing nothing the dangerous downward velocity will not be addressed, which means the landing velocity will have more negative impact on reward. Comparing "main engine" to the nearly equally valued "right engine" shows that the slight preference is based on the distance and right leg landing feature. This is more arbitrary, but agrees with intuition since the right engine will both reduce the downward velocity and straighten the ship, but will increase the leftward velocity compared to the main engine. This puts it at greater risk of reducing reward for missing the right leg landing goal and distance reward. Overall the explanations agreed well with intuition, which together with similar confirmation can increase our confidence in the general reasoning of the policy. We also see the MSXs were uniformly very small.
We compare Cart Pole state-action explanation to explanation produced by its reversed state as shown in Figure 3b This comparison illustrates how one case, the explanation agrees with intuition and builds confidence; while the other exposes underlying inaccuracy or flaw.
Cart Pole.
Our original game state (left) positions the cart in a dangerous position moving right, close to the end of the track. The pole is almost vertical and has small angle velocity towards the left. The action "push left" (move cart left) agrees with intuition as the cart is at the right edge of the screen and cannot move right without failing the scenario. The IG and MSX (left) concurs, showing the cart's current position close to the right edge as the main reason why it prefers the "push left" action over the "push right"; moving left will put the cart back within safe boundary.
Reversing the game state (left) by multiplying- -1 to each value in the input state vector produces a flipped game state (right). The cart now positioned in dangerous position moving left, close to the end of the track. Once again the pole is almost vertical and now has small angle velocity towards the right. One would expect the agent to perform the action "push right" (the opposite action to game state (left) as moving left will cause the agent to move off the screen and fail the scenario. However, as depicted in IG and MSX (right) we see the agent prefers "push left" over "push right". The agent justifies this action via MSX that focuses justification indicates that the agent is putting too much weight on the pole angle versus the boundary condition in this dangerous situation. The agent has not learned the critical importance of the left boundary. This indicates further training on the left side of the game map is needed. Presumably, during training the agent did not experience similar situations very often.
maintaining pole vertically to the left. This
Tug of War. In Figure 4 we give examples from a high--peforming 17 feature ESP agent, one that agrees with common sense and one that reveals flaw. Game state (top) shows the ESP agent (blue
8

---

Published as a conference paper at ICLR 2021
33
10-
Figure 4: Example Explanations for Tug-of- of War 17 feature ESP-DQN agent. Each row is decision point showing: (left) game state; (middle) Q-values and GVFs for preferred action and a non-preferred action; (right) IGX for action pair and corresponding MSX (indicated by highlighted bars). For Game 1 (top) the agent's preferred action is +4 Marine, +1 Baneling in Top Lane and the non-preferred action is +10 Marine, +1 Baneling on Bottom. For Game (bottom) the highest ranked action is +1 Baneling in Bottom Lane and sub-optimal action is +2 Marine, +4 Baneling in Bottom Lane.
player) with too few marine buildings defend against the Immortals. We show information for the best ranked action and sub-optimal action (details in caption). The best action creates top-lane units, while the sub-optimal action creates the maximum bottom-lane units. The IGX and MSX show (top) that the most responsible GVF feature for the preference is "damage to the top base from immortals", which agrees with intuition since the best action attempls to defend the top base, while the sub-optimal action does not. Indeed, the GVFs for the sub-optimal action show the top base is predicted to take 80% damage from the enemy's s Immortals compared to nearly for the best action.
In the second game state (bottom), the ESP agent plays against an opponent that it was not trained against and loses by having the bottom base destroyed. The state shows large enemy attack in the bottom with the ESP agent having enough resources (1500 minerals) to defend if it takes the right action. However, the most preferred action is to add just one Baneling building to the bottom lane, which results in losing. Why was this mistake made?
We compare the preferred action to an action that adds more buildings to the bottom lane, which should be preferred. The IGX and MSX show that the action preference dominated by the GVF feature related to inflicting damage in the top lane with Banelings. Thus, the agent is "planning" to save minerals to purchase more top Baneling buildings. The IGX does indicate that the agent understands the sub-optimal action will be able to defend the bottom lane, however, this advantage for the sub-optimal action overtaken by the optimism about the top lane. This misjudgement of relative values causes the agent to lose the game. On further analysis. we found that this misjudgement is likely due to the ESP agent never experiencing a loss due such a bottom lane attack during training.
7 SUMMARY
We introduced the ESP model for producing meaningful and sound contrastive explanations for RL agents. The key idea is to structure the agent's action-value function in terms of meaningful future predictions of its behavior. This allows for action- action-value differences be compared in terms of deltas in the future behaviors they entail. To achieve mcaningfulness. we required the agent designer to provide semantic features of the environment, upon which GVFs were learned. To achieve soundness, we ensured that our explanations were formally related to the agent's 's preferences in well-defined way. Our case studies provide evidence that ESP models can be learned in non-trivial environments and that the explanations give insights into the agent's 's preferences. An interesting direction for future work is to continue to enhance the internal structure of the GVFs to allow for explanations at different levels of granularity, which may draw ideas from GVF networks (Schlegel et al. 2018).

---

Published as a conference paper at ICLR 2021
REFERENCES
Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep (rl). In International Conference on Learning Representations, 2020.
Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynamic programming. Athena Scientific, 1996.
Samuel Greydamus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding Atari agents. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pp. 1792-1801, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Piyush Gupta, Nikaash Puri. Sukriti Verma, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishna- murthy, and Sameer Singh. Explain your move: Understanding agent actions using focused feature saliency. In International Conference on Learning Representations, 2020.
Rahul lyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia P. Sycara. Transparency and explanation in deep reinforcement learning neural networks. CoRR, abs/1 009.06061, 2018.
Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi- Velez. Explainable reinforcement learning reward decomposition. In Proceedings of the IJCAI 2019 Workshop on Explainable Artificial Intelligence, pp. 47-53, 2019.
Omar Zia Khan, Pascal Poupart, and James P Black. Minimal sufficient explanations for factored markov decision processes. In Nineteenth International Conference on Automated Planning and Scheduling, 2009.
Jelle Kok and Nikos Vlassis. Sparse cooperative q-learning. In Proceedings of the twenty-first international conference Machine learning, pp. 61. ACM, 2004.
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1rr.vv1509.0000 029.1, 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et Human-level control through deep reinforcement learning. Nature, 51 518(7540):529-51) 5235533,, 2015.
Matthew Olson, Lawrence Neal, Fuxin Li, and Weng- Wong. Counterfactual states for atari agents via generative deep learning. arXiv preprint arXiv: 1909. 12969, 2019.
Stuart J Russell and Andrew Zimdars. Q-decomposition for reinforcement learning agents, In Proceedings of the 20th International Conference on Machine Learning (ICML-00), pp. 656-663, 2003.
Matthew Schlegel, Adam White, Andrew Patterson, and Martha White. General value function networks. arXiv preprint arXiv:1807.06663, 2018.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Grarning-Voluume 70, pp. 3319-3328. JMLR. org, 2017.
Richard Sutton, Joseph Modayil. Michael Delp, Thomas Degris, Patrick Pilarski. Adam White, and Doina Precup. Horde: scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction categories and subject descriptors. In International Conference on Autonomous Agents and Multiagent Systems, volume 2, 2011.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Harm Van Seijen, Mehdi Fatemi, Joshua Romoff, Romain Laroche, Tavian Bames,, and Jeffrey Tsang. Hybrid reward architecture for reinforcement learning. In Advances in Neural Information Processing Systems, pp. 5392-5402, 2017.
10

---

Published as a conference paper at ICLR 2021
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, and et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3550 2019. doi: 10. 1038//41586-0199 1724-7.
J Waa, J van Diggelen, K Bosch, and M Neerincx. Contrastive explanations for reinforcement learning in terms of expected consequences. In Proceedings of the Workshop on Explainable Al on the IJCAI conference, Stockholm, Sweden., 37, 2018.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, (3-4))279-2922 1992.
A ESP-DQN PSEUDO-CODE
The Pseudo-code for ESP-DQN given in Algorithm
Algorithm ESS-DDN: Pseudo-code for ESP-DQN agent Learning.
Require:
Act(s, a) returns tuple (s', F, done) of next state s', reward r, GVF features F R"
and terminal state indicator done
Require: K target update interval, reward discount factor, 7 GVF discount factor
Init QF, Q'r The non-target and target GVF networks with parameters 0pr and f respectively.
Init Ć, Ć* :: The non non-target and target combining networks with 0g and 0% rspectively.
Init M +- ( initialize replay buffer
Q-function is defined by Q(s, a) ((Qr(s.))) a))
Target Q-function is defined by ('(ss,) ("'r
repeat
Environment Reset sp < Initial State totalUpdates <- for L + 0 to T do
0r 4 <(,,5) // e-greedy
(S141, Fú, done,) +- Act(si, at)
Add (3t, ae, Tt. F1, $1+1, done;) to M
:: update networks
Randomly sample mini-batch {(s, 0i, ris Fr, sy,, done;)) from M
à; arg max ((()),a)
F.
done; is true
fl
F. + 22' a.)
Otherwise
rr of + (+) + +s) a.)
done; is true
Otherwise
Update Or via gradient descent on average mini-batch loss (s² Qr(si a:))
Update &c via gradient descent on average mini-batche loss (gf Q(5 a;))
if totalUpdates mod K == 0 then O' OF 0 fc
end if
total Updates +- totalUpdates
if done; is true then break end if end for until convergence
11

---

Published as a conference paper at ICLR 2021
B CONVERGENCE PROOF FOR ESP-TABLE
Algorithm 2 gives the pscudo-code for ESP-Table based on e-greedy exploration. Note that, as for Q-learning, the convergence proof applies to any exploration strategy that guarantees all state-action pairs are visited infinitely often in the limit.
Algorithm 2 ESP-Table: Pseudo-code for table-based variant of ESP-DQN. The notation Q e shorthand for Q < (1 a + a..
Require: Act(s, a) returns tuple (s', FF) of next state s', reward r. and GVF features F E R Require: h(()) - hash
hash function from It"h to a finite set of indices
Require: K target update interval
Require: 7, discount factors for GVF and reward respectively
Init axF,0, aF, ;: learning rates for GVF and combining function
Init [s, a] GVF table indexed by state-action pairs
Init C[i]] :: Combining function table indexed by indices in
Init 2'p a,,aa== Target GVF table indexed by state-action pairs
Init [* :: Target Combining function table indexed by indices in
Q-function is defined by Q(s, a) C[h(Qp[s, al)]
Target Q-function is defined by Q'(ss, a) ("Ih(((p(ssa))
so +- Initial State =
repeat
if mod = then
O'F QF
C
end if
ar t =(Q, sc) :: e-greedy exploration
(s1+1, T2, Fi) Act(st, az)
a' t arg max, Q'' ((((s)1,a)
[F(s,, a. artt F2 + 7Q' ClN(QF (86, a.)) ac. +++++
Tt + (1'4411)
until convergence
For the proof will let / index the number of learning updates and i LtK] be the number of updates to the target tables. The formal statements refer to the "conditions for the almost surely convergence of standard Q-learing". These condition are: 1) There of updates for each state-action pair, and 2) The learning rate schedule a must satisfy Σ, ag = co and E, af < co. ESP-Table uses two learning rates, one for the GVF and one for the combining function.
be an unbounded number
We will view the algorithm as proceeding through sequence of target intervals, indexed by i, with each interval having K updates. We will let C and Q'r,; denote the target GVF and combining functions, respectively, for target interval with corresponding target Q-function (s, a) Č (',p'p al)] and greedy policy (s()) = arg max, Q' a). The following lemma relates the targets via the Bellman backup operators. Below for a GVF QF we define the max-norm as 10 = max, max, max [Q (s, a)).
Lemma 1. If ESP-Table is under the standard conditions for the almost surely (a.s.) conver- gence of Q-learning and uses a Bellman siffman-ufffiient pair (F, h) with locally consistent h, then for any 0 there exists funite target update interval K, such that, with probability 1, for all i, B[Q:) [
Q'
S <annd Q' B (': S.
That is, after a finite number learning steps during an interval, the updated target Q-function and GVF are guaranteed to be close the Bellman backups of the previous target Q-function and GVE.
12

---

Published as a conference paper at ICLR 2021
Note that since the targets are arbitrary on the first iteration, these conditions hold for any ablle-basdd ESP O-function.
Proof. Consider an arbitrary iteration with target functions Q/, C₁ Q',,, and let Q Ct and Q' be the corresponding non-target functions after t updates during the interval. Note that for t = 0 the non-targets equal to the targets. The primary technical issue is that C is based on table that can change whenever Q'p,, changes. Thus, the proof strategy to first show a convergence condition for Q P. that implies the table for Ój will no longer change, which will then lead to the convergence of CT.
Each update of Q'r is based on fixed target policy it and a fixed target GVF Q'r, so that the series of updates can be viewed as a stochastic approximation algorithm for estimating the result of a single Bellman GVF backup given by
B (F,,(ss,) a) = F(s. + E TT(s,a,s) (',')',,(((')),
(2)
which is just the expectation of F(s, a) + 1S", #{(5") with 5" ~ T(s, a, .). Given the conditions on the learning rate a it is well known that O7 will thus converge almost surely (a.s.) to this expectation, i.c to B 12' The a.s. convergence of O'F,, implies that for any e' ttere is a finite t such that for all > 41 Q', B ( s é'. This satisfies the second consequence of the lemma if é' S and K > t1-
e't < be such that satisfies the local consistency condition of h, which implies that for all / > 4 and all (s, a). h(Q'r,l> (s) al) h h [''r,,s,, a)). That is, after l updates, h will map the non-target GVF to the same table entry as the Bellman GVF Backup of the target GVF and policy. Combining this with the Bellman sufficiency of (F, h) implies that for any state action pairs (s, a) and (x, y).
if hiQO,, al) h(2r,, [ar v)) then [[ a) (zQ y). This means that after t updates all of the updates to table entry h(Qr,, a]) have the same expected value BI a). Using similar argument as above this implies that (s, a) = h (''r,,s a)) converges a.s. to (s, a) for all (s, a) pairs. Let t2 be the implied finite number of updates after t where the error is within e. The target update interval K- t + 12 satisfies both conditions of the lemma, which completes the proof.
Using Lemma T we can prove the main convergence result.
Theorem 2. ESP- TESP-Table is run under the standard conditions for the almost surely (a.s.) convergence
of Q-learning and uses there exists a finte target update interval K, such that for all and a, # ((()) converges a.s. to x: (s) and lim (s) Qf (s,a) a)l S with probability 1.
sullmun--uuficient pair (F, h) with locally consistent h, then for any 6 >
Proof. From Lemma T we can view ESP-Table as performing approximate Q-value iteration, with respect to the sequence of target functions Q' That is, the total updates done during a target interval define an approximate Bellman backup operator B, such that Q'+1 B[Q]]. Specifically, there exists a K, such that the approximate operator is e-accurate, in the sense that for any Q-function Q.
B[Q] [[Q]]
S.
Let B'[ denote i applications of the operator starting at Q so that n is the greedy policy with respect to ['()), Prior work (Bertsekas & Tsitsiklis 1996) implies that for any starting Q, the sub-optimality of this greedy policy is bounded in the limit.
lim 14000
V
v*(
< (1 3)-
23
(3)
00
'A "MDP-centric" way of sceing this view the update as doing policy evaluation in MDP with discount factor 0 and stochastic reward function R(s, a, s') = F(s,a) 72rrss' #{()))) The convergence of policy evaluation updates then implies our result.
13

---

Published as a conference paper at ICLR 2021
where V is the optimal value function and V is the value function of a policy л.
Now let
8 min min s:V (+)+V())))
(V) - (*)
be smallest non-zero difference between optimal value state and sub- sub-optimal value of a state
across all non-optimal policies. From this definition it follows that, if V V*
S 5, then
#² = n². From Equation 3 this condition is achieved in the limit as - oc if we select < (1=9)² 8.
2
Let K1 be the finite target interval implied by Lemma to achieve this constraint e. Since Lemma T hholds with probability 1, we have proven that it converges almost surely to n'" for a finite K₁. This implies the first part of the theorem.
For the second part of the theorem, similar to the above reasoning, Lemma I says that we can view the target GVF Q'r, as being updated by an approximate Bellman GVF operator B That is, for any GVF QF and policy т, [][Qr] - 1qrl I Further, it is straightforward to show that our approximate Bellman GVF operator satisfies an analogous condition to Equation B] but for GVF evaluation accuracy in the limit. In particular, for any л and initial Qr, pr if we define Q7:; to be the GVF that results after i approximate backups the following holds 2
lim sup Q7 - Q7 Q S (1-7)* 1000
(4)
Thus, for a fixed policy the approximate backup can be made arbitrarily accurate for small enough c.
From the almost sure convergence of #t. we can infer that there exists finite i' such that for all i i> i", # = n² Thus, if K > K1, then after the i" target update the target policy will be optimal thereafter. At this point the algorithm enters pure policy evaluation mode for fixed policy n", which means that the approximate GVF operator is continually being applied to x across target intervals. From Equation this means that in the limit + oo we have that
lim sup OF Q'r. 00 S (1-7)' 1-000
Thus, can achieve any desired accuracy tolerance in the limit by selecting small enough e. Let K₂ be the target interval size implied by Lemma [ for that epsilon and let the target interval be K = max(Ky, K1, K2). This implies that using target interval K, there is finite number of target updates i' after the first i' updates such that for all > + Q'F,, will achieve the error tolerance. This completes the second part of the proof.
C TUG OF WAR DOMAIN
In this section, overview the real-time strategy (RTS) game, 'Tug of War' (ToW), used for this study. Tug of War (ToW) is an adversarial two-player zero-sum strategy game we designed using Blizzard's s PySC2 interface to Starcraft 2. Tug of War is played on rectangular map divided horizontally into top and bottom lanes as shown in Figure 5 The game is viewed from an omnipotent camera position looking down at the map. Each lane has two base structures; Player 1 owns the two bases on the left of the map, and Player 2 owns the two bases on the right. The game proceeds in 30 second waves. Before the next wave begins, players may select either the top bottom lane for which to purchase some number of military-uni production buildings with their available currency.
We have designed Tug of War allowing AI AI, Human vs Human, and AI vs Human gameplay. Watch a Human vs Human ToW game from Player I's perspective here: http:: wwwp://www...m.mm yout com/ wwtthhhv=krrDzol
Each purchased building produces one unit of the specified type at the beginning of each wave. Buildings have different costs and will require players budget their capital. These three unit types,
2 This can be proved via induction on the number of exact and approximate Bellman GVF backups. showing that after i backups the difference is at most Σi=1 nd and then taking the limit i + oc.
14

---

Published as a conference paper at ICLR 2021
Marine
Baneling
Immortal
Figure 5: (left) Tug of War game map- Top lane and bottom lane, Player 1 owns the two bases on the left (gold star-shaped buildings), Player 2 owns the two bases on the right. Troops from opposing players automatically march towards their opponent's s side of the map and attack the closest enemy in their lane. (right) Unit Rock Paper Scissors - Marines beats Immortals, Immortals beats Banelings, and Banelings beats Marines. We have adjusted unit stats in our custom Starcraft map to befit ToW's balance.
Marines, Immortals, and Banelings, have strengths and weaknesses that form a rock-paper-scissors relationship shown in Figure Units automatically move across the lanes toward the opponent's side, engage enemy units, and attack the enemy base if close enough. Units will only attack enemy troops and bases in their lane. If base destroyed after 40 waves, the player who owns the base with the lowest health loses.
Both Players receive a small amount of currency at the beginning of each wave. A player can linearly increase this stipend by saving to purchase up to three expensive economic buildings, referred to as Pylon.
ToW is near full-information game; players can see the all units and buildings up to the current wave. Both player's last purchased buildings are revealed the moment after wave spawns. The only hidden information is the unspent currency the opponent has saved; one could deduce this value as the wave number, cost of each building, currency earned per wave, and the quantities of buildings up to the current snapshot are known. It would be difficult for human to perform this calculation quickly.
Tug of War is stochastic domain where there is slight randomness in how opposing units fight and significant uncertainty to how the opponent will play. Winning requires players assessing the current state of the game and balancing their economic investment between producing units immediately saving for the future. Players must always be mindful of what their opponent may do so as to not fall behind economically in unit production. Purchasing Pylon will increase one's 's's currency income and gradually allow the player to purchase more buildings, but players must be wary as Pylons are expensive, saving currency means not purchasing unit- unit-production buildings which may lead vulnerable Setttiinn.overrellyy if the opponent seems to be saving their currency, the player can only guess to what their opponent is saving for; the opponent may be saving purchase a Pylon or they may be planning to purchase a lot of units in single lane.
Tug of War presents a challenging domain to solve with Reinforcement Learning (RL). These challenges include large state space, large action space, and sparse reward. States in ToW can have conceivably infinite combinations of units on the field, different quantities of buildings in lanes, or different base health. The number of possible actions in state corresponds to the number of ways to allocate the current budget, which can range from 10s to 1000s. Finally, the reward sparse giving +1 (winning) or 0 (losing) at the end of the game, where games can last up to 40 waves/decisions.
C.I TUG OF WAR FEATURE DESIGN
While humans need continuous visual feedback to interact with video games, computer systems can use simple numeric values received in disjointed intervals to interpret game state changes. We have designed an abstract "snapshot" of the ToW game state at a single point in time represented as a 68 dimensional feature vector. Note that for this study, we have increased added additional features to capture granular details, thus bringing the total to 131 features. At the last moment before wave spawns, the Al agent receives this feature snapshot and uses it to select action for the next wave. We call this moment a decision point. The decision point is the only time when the agent receives information about the game and executes action; the agent does not continuously sample
15

---

Published as a conference paper at ICLR 2021
observations from the game. The agent's performance indicates this abstraction is sufficient for it to learn and play the game competently.
The state feature vector includes information such as the current wave number, health of all 4 bases, the agent's 's current unspent currency, the agent's current building counts in both top and bottom lanes, the enemy's last observed building counts in the top and bottom lanes, pylon quantities, and the number troops in each grid of the 4 grid sections of the map as depicted in Figure opponent's current unspent mineral count is sent the agent as this hidden information part of the game's design.
Figure 6: ToW 2 Lane 4 Grid Unit quantities and positions on the map is descretized into four sections per lane.
D AGENT DETAILS: HYPERPARAMETERS AND ARCHITECTURES
The ESP agent code is provided in Supplementary Material, including pre-trained models for all
domains we present.
Table T gives the hyperparameters used in implementation. Note that our implementation of ESP-DON supports both hard target updates as shown the pseudo-code and "soft target updates" (Lillicrap al. 2015), where at each step the target network parameters are gradually moved toward the currently learned parameters via mixing proportion т. We found that this can sometimes lead to more stable learning and use it in of our domains as indicated in the table.
T our GVF network structures used to train the agents in each domain. The choice of activation function for each GVF output component is based on the output type of the component. For GVF outputs in [0, 1] we use a sigmoid activation, for sets of mutually exclusive indicator GVFs (e.g. the win condition) we use softmax activation over the set, and for GVF outputs with arbitrary numeric ranges we use a linear activation. Specifcially, we use Sigmoid functions on F1 through F12 and F17 features for our Tug of War ESP-DQN 17-feature agent and on F131 for our Tug of War ESP-DQN 131 -featurre agent because the data ranges (0, 1). We apply SoftMax function to features F13 to F16 and Fl to for our our Tug of War ESP-DQN 17-feature and 131-feature agents because said features correspond to probabilities that sum to 1.
E TUG OF WAR 131 FEATURES
We introduce detailed description of the 131 features used to train our Tug of War ESP-DQN agent. These features capture events in ToW, namely:
Game ending win-condition probabilities; The likely-hood for each base to be destroyed or have the lowest HP at wave 40.
Pl and P2 currency; These features allow GVFs to predict the amount of money players will receive in the future.
Quantity units spawned.
The number of each type of units will be survive at different ranges we defined on the map for both players; allowing the GVFs to predict the advantage of each lane of each type of unit in the future.
'In addition to the 4 grid map regions explained in Figure 6 we add 5th map region (Grid 5) to detect units attacking bases. Grid 5 for P1 is indicates the quantity of P1 units attacking P2's hases. This is reversed for P2, where now Grid for P2 indicates P2 units attacking P1's bases.
16

---

Published as a conference paper at ICLR 2021
Jyyer-Parameters
Lunar Lander
Cart Pole
Tug-of-
Discount factors(y and (3)
0.99
0.99
0.9999
Learning Rate(a)
10-4
10-5
10-1
Start Exploration(c,)
1.0
1.0
1.0
Final Exploration(e,)
0.01
0.05
0.1
Exploration Decrease(linearly) Steps
2 * 10 128
2 10a
4 10*
Batch Size
128
32
Soft/Hard Replace
Soft
Soft
Hard
Soft Replace(r)
5 * 10-1
5* 10-4
N/A
Hard Replace Steps
N/A
N/A
10
GVF Net Optimizer
Adam
Adam
Adam
Combiner Net Optimizer
SGD
SGD
SGD
Training Episodes
5 * 103
10*
1.3 10*
Evaluation Intervals
200
100
40
Evaluation Episodes
100
100
10
Riemann approximation steps of IGX²I
30
30
30
Table I: Syyper-parameters and optimizers used to train our ESP-DQN and DQN agents on Lunar Lander, Cart Pole and Tug of War.
Lunar Lander
Cart Pole
War(17f)
Tug-of-War(1311
GVF Net
layers MLP
3 layers MLP
4 layers MLP
4 layers MLP
GVF Output Activation
Linear
Linear
Sigmoid FI-F1,, F17)
SoftMax(F) F--F8)
Function
SoftMax(F13-F16)
Sigmoid(F131)
Combiner Net
layers MLP 3 layers MLP
4 layers MLP
4 layers MLP
Table Network structures we used train our ESP-DQN and DQN agents on Lunar Lander, Cart
Pole and Tug of War.
Delta damage to each of the four bases by each of the three unit types. These features allow GVFs to predict the amount of damage each unit type will inflict on the opponent's ' base in the unit's respective lane.
The amount of damage inflicted by which type of units another type of units for both players, like the damage the friendly Marine inflicted on enemy immortal; Allows the GVFs to predict the amount damage for each type of units inflicting each type of units.
An indicator of whether the game reaches waves of fie-breaker.
F EXAMPLE EXPLANATIONS
Cart Pole. Figure 7a shows a Cart Pole state encountered by learned near-optimal ESP policy, where the cart and the pole are moving in the left direction with the pole angle being in dangerous range already. The action "push left" is preferred over "push right", which agrees with intuition. We still wish verify that the reasons for the preference agree with our common sense. From the IGX and MSX in Figure [C the primary reason for the preference is the "pole angle left" GVF, which indicates that pushing the right will lead future where the pole angle spends more time the dangerous left region. Interestingly we see that "push right" is considered advantageous compared to "push left" with respect to the left boundary and left velocity features, which indicates some risk for push left with respect to these components. All of these preference reasons agree with intuition and along with similar examples can build our confidence in the agent.
Lunar Lander. Figure Sal illustrates Lunar Lander state achieved by near-optimal ESP policy. The lander is moving down to the left and is close to landing within the goal. Additionally, the left leg has touched the ground marked by the green dot. Figure shows the GVF values of all actions and expects the lander to land successfully with the right leg touching down. The GVFs values the distance, velocity, and angle small because the lander is close to the goal.
Although this state allows the lander to successfully land after taking any action, the IGX shown in Figure 8c illustrates the agent prefers "use left engine". This is because using the right engine will
17

---

Published as a conference paper at ICLR 2021
GVTS
(a) Game State
GVFs
IG
Gaundaay
Cla
FEG
Pil
Rii Merdy
Figure 7: Explanation example for Cart Pole. Three Figures show the game state, the Q --auuesand GVF predictions for actions, and the IGX and MSX respectively.
OVS
(a) Game State
(b) GVFs
(c) IG and MSX
MSX
Milddy
Log
Engine
Enginn Ubez
Lancng
Figure 8: Explanation example for Lunar Lander. Three Figures show the game state, the Q-values and GVF predictions for actions, and the IGX and MSX respectively.
increase the velocity of the lander, pushing it towards the left and increasing "Fl Distance" from the goal. This action and justification makes intuitive sense as the lander is unlikely fail in this state and has chosen an action that reduces its velocity and decreases its landing delay.
The "use main engine" action also delays the landing increases distance to the goal, as indicated by the MSX bars in Figure &c The IGX also shows the "use main engine" engine risks the left leg leaving the ground which agrees with intuition as moving up pushes the lander back into space. However, "us main engine" gives the lander another opportunity to adjust its velocity and angle. That may be why the IGX of velocity and tile-angle are negative. The "no-op" action has lower preference than the best action because the lander is slightly drifting and may move out of the goal. Two largest IGXs of "noop" action agrees with this rationale. However, the IGX of landing is negative that may be arbitrary indicates doing the "noop" action will lead the lander to land faster since the lander is moving down already, but sometimes landing faster gets less reward because moving to the center of the goal gain reward by reducing the distance between the center of goal and lander.
Tug of War: 17 Features. Figure Da depicts screenshot of Tug of War game where our ESP agent (P1, blue) is playing against a new AI opponent (P2, orange) it has never encountered. The ESP agent's 's top base destroyed after two waves thus losing the game. The annotated game state shows the ESP agent doesn't 't have enough units to defend its top base as its opponent's 's banelings can kill almost all units, and the agent's Top lane base has approximately 35% hit points (HP) remaining. We can regard this state money to defend the top lane and still looses the base in two waves after taking the highest ranked action. Given our deep ToW game knowledge, want to understand why the ESP agent chose to purchase Banelings in the Bottom Lane (arguably sub-optimal) rather than purchase Immortals the Top Lane (intuitively a better action).
critical moment in the game because the agent spends all its
To understand why the agent prefers the action that worse than an action we intuitively recognize to be better, we analyze both action's 's GVFs (Figure 9b), and IG & MSX (Figure Dc). The sub-optimal action's 's GVFs shows the sub-optimal action is expected to reduce damage from the enemy's 's top Banelings. This indicates the agent understands taking the sub-optimal action can pose better
18

---

Published as a conference paper at ICLR 2021
(a) Game State
(b) GVFs
(c) IG and MSX
Figure 9: Explanation example for Tug-of-Wo War 17 feature ESP-DQN agent. Three Figures show the game state, the Q-values and GVF predictions for actions, and the IGX and MSX respectively. The top ranked action +2 Baneling in Bottom Lane and sub-optimal is +1 Immortals in Top Lane
defense. However, the MSX bar shows positive IGX of the self bottom Baneling damage still can cover the negative IGX of enemy top Bancling damage; indicating the agent is focusing on destroying the enemy's bottom base while ignoring the damage its top base will take. This misjudgement can be attributed to the agent over-fitting to its fixed-agent opponent during training.
(a) Game State
Tug of War: 131 Features. Figure depicts screenshot of a Tug of War game where our (P1, blue) ESP agent is playing against the same fixed-policy AI opponent (P2, orange) was trained against. The ESP agent wins by destroying the opponent's 's bottom base. The state in Figure TOa indicates both players have balanced quantity of units in the top lane. We also observe P2 has an advantage in the bottom lane as the ESP agent doesn't 't have enough units to defend. The ESP agent has determined its best action is to spend all its money on producing +8 Marine buildings in the Bottom Lane to defend, which agrees with intuition as Marines counter Immortals. To justify why one can regard this choice as optimal, we compare the agent's 's best-determined action, +8 Marine buildings in Bottom Lane, to sub-optimally ranked action, +5 Baneling buildings in Bottom Lane, due to Immortals counter Banclings.
Figure 10b shows the GVF value of both action. Given the dense nature of the 131 Features, summarize the following:
The values concerning accumulated quantity features such as future currency to be earned are higher in the sub-optimal action than the best action because the game is expected to be prolonged if sub-optimal action is taken. The probability to end the game by tie- breaker(F131) as shown in Figure 10b graph "Probability End by Tie-breaker" agrees taking the best action leads to a faster win.
The sub-optimal action raises the probability of our ESP agent's bottom base getting de- (F)) and lowers the probability of the opponent's 's bottom base getting (Fs))) This assessment agrees with the game rules as Banelings do little to counter Immortals.
Agent's Expected Bottom Marine to Spawn 14) is higher of if it takes the best action, and Expected Bottom Baneling to Spawn(F15) F15) is higher if takes sub-optimal action.
By taken the best action, the agent expects its future surviving bottom marines to be closer to P2's bottom base F47 and F50); indicating the agent's 's units are able to push the enemy back. Contrasted to the sub-optimal action, where the opponent's surviving bottom Immortal
19

---

Published as a conference paper at ICLR 2021
Dastory and Probabilly
Gurnnny be Carred
Expeceed Uniaat Spaee
Tutan Sunking Cremy (Debon)
Unb Araddigg Tea Tofto Dasa
P1 o8 F2 UN Cavage
011 Voa
02-300H
(b) GVFs
is expected to be closer to the ESP agent's bottom base(F70, F73 and F76), indicating the opponent pushed the agent back.
If the ESP agent purchases +8 marines in the bottom lane (best ranked action), the agent expects to take no damage from the enem to F94). This can be contrasted to the expected damage if the agent were to purchase +5 baneling buildings in the bottom lane (sub-optimal action) where the agent expects to take base damage from P2's iFm9rtals(F94) shown in Figure 10b] graph "Units Attacking Top/Bottom Base".
We can validate the agent understands the pock-aape--scisee interaction between marines, banelings, and immortals from the GVF graphs as shown in Figure Tob graph "P1 Unit on P2 Damage" and "P2 Unit on P1 Damage". If the agent produces marines, the ESP agent correctly expects to inflict a large amount of damage on P2's immortals. If the agent produces banelings, the ESP agent correctly expects to inflict a large amount of damage on P2's s marines.
There exist some flaws in the agent's 's GVF predictions. Some values such as Future Surviving Units in Figure [Ob] should not be negative, indicating some flaw in the agent's training. This suggests an engineer can add ReLU function on the output to prevent negative values.
Explanations produced by our ESP model are sound because said explanations do not depend on GVF comparisons alone. Figure T0cl graph "Units Attacking Top/Bottom Base" illustrates P2 Immortal Damage on bottom base (I94); the primary MSX contribution for why the agent ranked +8 marine buildings as its best action. Given the notion that P2's Immortals in the bottom lane presents a significant threat, producing marines defend the immortals makes good intuitive sense. Banelings are a sub-optimal choice in this scenario, and would do little to defend against Immortals. We summarize the IG and MSX graph in Figure aS follows,
20

---

Published as a conference paper at ICLR 2021
Destory
Oueweyylh Farrel Saeepie
Uiiis
Der sereeee Call spene Vike
Takin Sunviing Enerry (oaior)
Lrbs Top Dara
Lizt Tarage
Necivies All
P2 LHi or P1 UnI Damage
Pobes is Enaily Te-ansaarr
14-
P2
(c) IG and MSX
Figure 10: Explanation example for Tug-of- of- War 131 feature ESP-DQN agent. Since there too much features to show as one figure, we separate them into 11 clusters. Three Figures show the game state, the Q-values and GVF predictions for actions, and the IGX and MSX respectively. The top ranked action +8 Marines in Bottom Lane and sub-optimal is +5 Banelings in Bottom Lane.
The best action adds more Marine buildings; thus increasing the quantity of marines spawned per wave, but the agent doesn't care about the quantity of marines(F14) the IGX is close to O. However, the agent cares about the damage the Marine inflict (F86), although this is not important as defending against opponent's 's Immortals.
Graph "Destroy and Lowest HP Probability" illustrates the two mutually exclusive win types in ToW; winning by destroying one of P2's bases, or winning by making sure one of P2's base has the lowest HP at wave 40. The probability Base Destroyed IGX indicates the agent expects to destroy the opponent's 's bottom base(F4) and defend its own bottom base
Graph "Future Surviving Friendly (Bottom)" illustrates the contribution of Pl's surviving troops in the bottom lane. The positive IGX contribution of feature "PI Bottom Marine Grid 4(F47)" and "Grid 5(F50)" indicates the agent cares about its marines moving closer the enemy's ' bottom base. The IGX of the "Pl Bot Marine Grid 3(F44)" is negative, possibly because Grid is too far from the opponent's base to be considered a disadvantage.
Given the large number of features, the MSX is critical to get a quick understanding of the agent's s preference. In general, user interface design will be an important consideration when the number of features is large. Such interfaces should allow users to incrementally explore the IGX and GVFs of different actions flexibly and on demand.
21


Reviews:
[{"review_id": "mB7GEoU8EK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThe paper attempts to improve the interpretability of RL agents' action selection process by (a) proposing embedded self-prediction (ESP), a model that embeds generalised value functions (GVFs) in the action-value function of the agent with a \"combining\" function to and (b) ESP-DQN, an extension of DQN that augments experience replay tuples with a GVF feature vector and that decomposes the model into separate combining and GVF parameters. These enable to define action-values with respect to predefined feature maps, thus providing more \"resolution\" into the behaviour of the policy.\n\n### Good stuff\n\n1. The idea of decomposing the policy into GVFs as a way to force explanations wrt. some features is *brilliant*, and it is well executed when combined with the contrastive explanation system.\n2. Sections 2-4 provide both a clear introduction to GVFs as well as a detailed and sound description of the overall framework.\n3. The related work section is fairly tight, but actually covers a good amount of necessary and relevant related work, which makes it easy to scan through the literature.\n4. The experimental section does employ mostly fairly similar environments, but it is clear in the hypotheses being tested, and it is fairly satisfactory considering what it is attempting to evaluate.\n\n### Uncategorised notes\n\n- This is more of a meta-comment, but I enjoyed reading the paragraph about manually-designed features in Section 1. I understand why the authors felt the need to write it, and it is a sad state of affairs that it is now often a requirement to argue what to many people is just reality.\n- I wonder if it'd be worth it to test the method on Atari, through possibly the use of MinAtar: https://github.com/kenjyoung/MinAtar -- It feels like there's a gap in difficulty (of learning and analysis) between the ToW setting and the rest of the environments, and something aking to a middle ground would probably be a good setting to add. Complex gridworlds such as BabyAI and the NetHack Learning Environment are probably also good options.\n\n### Final comments\n\nOverall, I'm extremely happy to strongly recommend this paper towards acceptance. It is well written, it introduces a method that attempts to move forward towards solving an important problem in Reinforcement Learning, and there's a significant amount of details in the paper that would make it fairly straightforward to reproduce.\n\n### Typos\n#### Sec 1\n\n- RL agents explain its... -> their\n- directly \"embed\"... -> embeds\n- train those... -> trains\n\n#### Sec 3\n- Combination function update -> Combining? The manuscript is at times a little inconsistent.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NNbbsJSpEf-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Edit: \n\nI have read the authors' response as well as the other reviews. Based on the additional results and added feature selection details, I now agree that ESP is generally applicable. \n\nSummary: \n\nThe authors present ESP, an RL system that can then explain action choices in terms of future feature values. Generalized value functions (GVFs) are used to learn an estimate of total future discounted feature values. These estimated total feature values are then used to estimate each actions Q-value. Since Q-values are based on GVF outputs, these intermediate values can be used as an explanation. The authors select a subset of these values to form a Minimal Sufficient Explanation (MSX). The proposed system is evaluated using three domains. The authors show that performance is comparable to non-GVF agents.\n\nReasons for score: \n\nThough the authors present a novel explanation format, the applicability of the method is uncertain. The results appear to rely on specific GVF feature choices. Non-general methods are still of interest, but the limited information on feature construction prevents a fair comparison to other approaches. Additionally, the explanations are not evaluated quantitatively. \n\nPros:\n\n-The use of GVFs for explanations in terms of future feature values is a novel line of work. MSXs are a natural way to then produce more concise explanations, and the authors extend MSX to their non-linear use case in a well-justified way. \n\n-The analysis of the Tug of War explanations was thorough. It clearly showed how ESP explanations would be used to investigate agent behavior. \n\nCons:\n\n-ESP is built upon the GVF features, but the choice of GVF features is suspect. Each environment uses a different style for its features. Lunar Lander and CartPole both have continuous features, yet the authors use \"deltas\" for Lunar Lander and region discretization for Cart Pole. Tug of War uses a bunch of features, including information about feature values when a game ends, and non-linearities are applied to the outputs of the GVF features. Note that these non-linearities are used *only* for Tug of War, and different non-linearities are used for different features (Table 2 in Appendix D). Effectively, each environment appears to use carefully engineered features. Given that DQN-Full performs very poorly for specific settings (i.e., in some cases, the agent cannot learn without the GVF features), the choice of features seems to be important. The authors should indicate the process used for selecting them and how these features should be chosen for other environments. ESP may not be robust to GVF feature choice, but this is insufficiently addressed in the paper.\n\n-In Section 6.3, the authors present potential conclusions that can be drawn from an ESP agent. These conclusions can be evaluated to determine whether valid conclusions can be drawn from the explanations. Such an evaluation would allow the hypotheses of the authors to be tested.\n\n-A substantial reorganization of the paper would improve clarity. Various definitions and descriptions are provided a few sections after the terms/methods are first used. Terms are unnecessarily over-loaded (such as \"sound\"). \n\nQuestions During Rebuttal Period:\n\n-Please address and clarify the \"Cons\" above.\n\n-In particular:\n\na) How were the GVF features chosen? Why does each environment use different features? \n\nb) What happens when Lunar Lander is given \"CartPole-style\" (region discretization) features? What happens when CartPole is given \"Lunar-Lander-style\" (change in features) features? \n\nMinor comments:\n\n-This work would benefit from another editing pass for tense/plurality matching between subject and verb.\n\n-The proofs in Appendix B would benefit from a re-write; currently, they are hard to parse. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DEar7tddQqn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method that offers explanation of action preference in a deep RL agent based on given features by the human. In other words, the model explains why action A is preferred to action B based on some given features. This is done through the embedded self-prediction (ESP) model. The authors also proposed a method for evaluation of importance of features in the learned policy. While the paper benefits from extended experimental result and interesting theoretical analysis in the tabular RL, I think its readability could be improved. For example, I believe generalized value functions should be explained more extensively as (I think) it is less known to the community compared to concepts like MDP and DQN. Also, an analysis  (or at least some discussion) on the effect of the number of features on learning QFs would be helpful. Another question I have, is about the dependence of features. What would happen to the evaluation if some given features are dependent?\nSome minor points: \n- It would be helpful if the authors add a figure of their networks similar to what is common in the field, specifying input and output of the networks. \n- The phrase of \"greedy policy\" was a little confusing for me, especially because \"greedy algorithm\" is usually one-step look ahead search.  Is it just argmax Q(s, a) (as suggested in page 3)? \n- I think the defense on manually-designed features could be transferred to conclusion since the importance of interpretability has been already mentioned in the first paragraph. \n   ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the important problem of endowing deep RL agents with added interpretability. Action values are decomposed as the combination of GVFs learned on externally-specified features, offering action explanations in terms of discounted future returns in the space of interpretable quantities. Reviewers praised the approach, as well as the level of detail for reproducibility purposes. R3 had concerns about the generality of the method but follow-up experiments have allayed these concerns. Given the reviewer response and the central importance of the problem considered to the field, I can wholeheartedly recommend acceptance.
Decision: Accept (Oral)