Explainable Reinforcement Learning via Model Transforms
Authors: Mira Finkelstein
Nitsan Schlotterbeck levy
Lucy Liu
Yoav Kolumbus
David C. Parkes
Jeffrey Rosenschein
Sarah Keren
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Deep Reinforcement Learning
Explanability
XAI
Model Based Reasoning

Abstract: Understanding emerging behaviors of reinforcement learning (RL) agents may be difficult since such agents are often trained in complex environments using highly complex decision making procedures. This has given rise to a variety of approaches to explainability in RL that aim to reconcile discrepancies that may arise between the behavior of an agent and the behavior that is anticipated by an observer. Most recent approaches have relied either on domain knowledge, that may not always be available, on an analysis of the agent’s policy, or on an analysis of specific elements of the underlying environment, typically modeled as a Markov Decision Process (MDP). Our key claim is that even if the underlying model is not fully known (e.g., the transition probabilities have not been accurately learned) or is not maintained by the agent (i.e., when using model-free methods), the model can nevertheless be exploited to automatically generate explanations. For this purpose, we suggest using formal MDP abstractions and transforms, previously used in the literature for expediting the search for optimal policies, to automatically produce explanations. Since such transforms are typically based on a symbolic representation of the environment, they can provide meaningful explanations for gaps between the anticipated and actual agent behavior. We formally define the explainability problem, suggest a class of transforms that can be used for explaining emergent behaviors, and suggest methods that enable efficient search for an explanation. We demonstrate the approach on a set of standard benchmarks.

Explainable Reinforcement Learning via Model Transforms
Mira Finkelstein', Lucy Liu', Nitsan Levy Schlot', Yoav Kolumbus', David C. Parkes? Jeffrey S. Rosenschein' and Sarah Keren*
The Hebrew University of Jerusalem, Benin School of Computer Science and Engincering 2 Harvard University, School of Engineering and Applied Sciences DeepMind
Technion Israel Institute of Technology. Taub Faculty of Computer Science
Abstract
Understanding emerging behaviors of reinforcement learning (RL) agents may be difficult since such agents are often trained in complex environments using highly complex decision making procedures. This has given rise variety of approaches to explainability in RL that aim to reconcile discrepancies that may arise between the behavior of an agent and the behavior that anticipated by an observer. Most recent approaches have relied either on domain knowledge, that may not always be available, on an analysis of the agent's policy, or on an analysis of specific elements of the underlying environment, typically modeled as Markov Decision Process (MDP). Our key claim is that even if the underlying model is not fully known (e.g., the transition probabilities have not been accurately learned) or is not maintained by the agent (i.e., when using model-free methods), the model can nevertheless be exploited to automatically generate explanations. For this purpose, we suggest using formal MDP abstractions and transforms, previously used in the literature for expediting the search for optimal policies, to automatically produce explanations. Since such transforms are typically based on a symbolic representation of the environment, they provide meaningful explanations for gaps between the anticipated and actual agent behavior. We formally define the explainability problem, suggest class of transforms that can be used for explaining emergent behaviors, and suggest methods that enable efficient search for an explanation. We demonstrate the approach on of standard benchmarks.
1 Introduction
The ienttormancc-transpaaeee trade-off is a major challenge with many artificial intelligence (AI) methods: as the inner workings of an agent's 's decision making procedure increases in complexity, becomes more powerful, but the agent's 's decisions become harder to understand. Accordingly, interest in explainable Al and the development of transparent, interpretable, Al models has increased rapidly in recent years [1]. This increase in complexity is particularly prevalent in reinforcement learning (RL) and deep reinforcement learning (DRL), where agent autonomously learns how to operate in its environment. While RL has been successfully applied to solve many challenging tasks, including traffic control [2], robotic motion planning [3], and board games [4], it is increasingly challenging to explain the behavior of RL agents, especially when they do not operate as anticipated.
To allow humans to collaborate effectively with RL-based AI systems and increase their usability, it is therefore important to develop automated methods for reasoning about and explaining agent behaviors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

While there has been recent work on explainability of DRL (see [5] for a recent survey), most of these methods either rely on domain knowledge, which may not be available, or involve post-processing the policy learned by the agent (e.g., by reasoning about the structure of the underlying neural network [6]). Moreover, most existing methods for explainability do not fully exploit the formal model that assumed to represent the underlying environment, typically a Markov Decision Process (MDP) [7], and analyze instead one chosen element of the model (e.g., the reward function [8])
We focus on RL settings in which the model of the underlying environment may be partially known, i.e. the state space and action space are specified, but the transition probabilities and reward function are not fully known. This is common to many RL settings in which the action and state spaces are typically known but the agent must learn the reward function and transition probabilities, either explicitly as in mooel-bbsee RL or implicitly as when learning to optimize its behavior in model-free RI.. For example, in a robotic setting. the agent may have some representation of the state features (e.g,, the location of objects) and of the actions it can perform (e.g., picking up an object), but not know its reward function or the probabilities of action outcomes.
Our key claim is that even if the underlying model is not fully known (or not explicitly learned), it can nevertheless be used automatically produce meaningful explanations for the agent's behavior, i.e., even if the agent is using a model-free method, the partial model can be manipulated using a model- based analysis to produce explanations. Specificallly, we suggest producing explanations by searching for a set of formal abstractions and transforms that when applied to the (possibly incomplete or approximate) MDP representation will yield behavior that is aligned with an observer's expectations. For this purpose, we exploit the rich body of literature that offers MDP transforms [9. 10, 11, 12, 13, 14] that manipulate different elements of the model by, for example, ignoring the stochastic nature of the environment, ignoring some of the effects of actions, and removing or adding constraints. While these methods have so far been used to expedite planning and learning, we use them to automatically produce explanations. That is, while for planning the benefit of using such transforms is in increasing solution efficiency, we use them to isolate features of the environment model that cause an agent to deviate from behavior that is anticipated by an observer.
Formally, we consider an explainability setting, which we term Reinforcement Learning Policy Explanation (RLPE), that comprises three entities. The first entity, the actor, is an RL agent that seeks to maximize its accumulated reward in the environment. The second entity, the observer, expects the actor to behave in some way and to follow a certain policy, which may differ from the one actually adopted by the actor. We refer to this as the anticipated policy, and this specifies which actions an observer expects the actor to perform in some set of states.' The third entity, the explainer, has access to a (possibly partial) model of the environment, to the anticipated policy, and to a set of MDP transforms. The explainer seeks sequence of transforms to apply to the environment such that the actor's policy in the transformed environment aligns with the observer's anticipated policy.
To demonstrate RLPE, consider Figure 1, which depicts a variation of the Taxi domain [15]. In this setting, the actor represents a taxi that operates an environment with a single passenger. The taxi can move in each of the four cardinal directions, and pick up and drop off the passenger. The taxi incurs a small cost for each action it performs in the environment, and gains a high positive reward for dropping off the passenger at her destination. There are walls in the environment that the taxi cannot move through. The observer has a partial view of the environment and knows which actions the taxi can perform and how it can collect rewards. With the information available and the, possibly incorrect, assumptions she makes about the actor's 's reasoning, the observer anticipates that the taxi will start its behavior by moving towards the pussenger. This description of the anticipated behavior over a subset of the reachable states in the environment is the anticipated policy. The prefix of this policy is depicted by the green arrow in the figure. However, the actual policy adopted by the actor, for which the prefix is represented by the red arrow, is to visit some other location before moving towards the passenger.
Example 1
In order to explain the actor's 's behavior, the explainer applies different action and state space transforms to its model of the environment. The objective is to find transformed model in which the actor follows
'Our formalism can be extended to support cases in which the observer anticipates any one of set of policies to be realized.
'In some settings, the actor and explainer may represent the same entity. We use this structure to separate the role of an actor from the attempt to explain its behavior.

---

Environment
Observer
Explainer
Transformed
Envirenmmntt
Observer
TI
7 T T33
Figure I: Reinforcement Learning Policy Explanation, example (left) and model (right). The policy prefix of the taxi (yellow square) is depicted in red and leads to the fuel station (purple mark). The observer, who is not aware of the fuel constraint, anticipates that the taxi heads towards the passenger (the anticipated policy is depicted in green). An explanation is generated by applying a transform that removes the fuel constraint.
the anticipated policy. We note that our suggested approach can produce meaningful explanations only the explainer uses transforms that are meaningful to the observer. In our example, the explainer first applies an action transform that allows the taxi to move through walls and trains the actor in the transformed environment. Since the policy in the transformed model still does not match the anticipated policy, the explainer can infer that the reason for the discrepancy is not the fact that observer may unaware of the walls the environment, and therefore this transform would not represent meaningful explanation. As second attempt, the explainer applies transform that relaxes the constraint that a car needs enough fuel to be able to move, and allows the taxi to move regardless of its fuel level. After training, the actor's 's policy in the transformed environment aligns with the anticipated policy. This indicates the observer may not be aware of the fuel constraint, and does not expect the actor first drive towards the gas station. This transform is consistent with the discrepancy between the anticipated and actual policies and represents a suitable explanation, as long as this constraint can be conveyed to the observer.
Beyond this illustrative example, the ability to understand the "anticipation gup" (the gap between anticipated and observed behavior) is important in many applications. Examples include autonomous driving, where critical to know why a vehicle deviates from an anticipated course of action, medical applications, where it is crucial to explain why an AI system recommends one treatment over another, and search and rescue missions, where a robot is moving in an unknown enviroument with observations that are differrntt from those of its operator and may behave in unpredictable ways.
The translation of the transform sequence that reconciles the gap between the observer and actor to natural language is beyond the scope of this work. Nevertheless, since the transforms manipulate the underlying MDP model, they incorporate the symbolic information represented by the MDP representation, and this can reasonably be expected to translate to an intuitive explanation (e.g., notifying the observer about a missing precondition in model of an action). Thus, our approach can be used to automatically generate explanations without compromising generality. Moreover, while we used a single-agent setting to demonstrate the approach, the same ideas can apply to multi-agent settings, where the set of applicable transforms include, in addition to the transforms used for single-agent settings, transforms that deal with the multi-agent aspects of the system (e.g.. shared resource constraints).
The recent interest explainability in RL has yielded approaches that vary in the kind of questions the explanations are aimed to address and in the methods applied to find them (e.g., [16, 17, 8, 18, 19]). Ours an example of a post-processing approach, accounting here for settings in which the observer has an anticipated behavior that is not aligned with the actual behavior, and where the objective is to find an explanation by transforming the underlying environment one in which the agent behaves as expected.
3

---

Typically, post-hoc methods focus on particular element of the model and investigate its effect on the agent's 's behavior. For example, some propose that the reward function be decomposed into an aggregation of meaningful reward types according to which actions are classified [8], or that human-designed features, such as the estimated distance to the goal, are used to represent action-value functions [18]. In other work, human-user studies have been used to extract saliency maps for RL agents in order to evaluate the relevance of features with regard to mental models, trust, and user satisfaction [19], while [6, 20] use saliency maps to produce visual explanations. Others suggest producing summary of an agent's behavior by extracting important trajectories from simulated behaviors [21].
Our approach supports arbitrary transforms and abstractions that can be applied to the environment model and combined with any learning approach in both single- and multi-agent settings. The variety of transforms that can be used for generating explanations relies on the various methods suggested for expediting planning [13] and RI. [11]. Previous work has considered an optimal planning agent in deterministic environment and suggested learning a partial model of the environment and task, and identifying missing preconditions to explain the behavior [22]. We generalize this to stochastic environments with partially-infommn RL agents and to arbitrary transforms (beyond only those that consider action preconditions).
The contributions of this work are threefold. First, we present a novel use of model transforms and abstractions, formerly mainly used for planning, to produce explanations of RL agent behaviors. Second, we present formal definition of the Reinforcement Learning Policy Explanation (RLPE) problem and specify classes of state and action space transforms that can be used to produce explanations. Finally, we empirically demonstrate our approach on a set of standard single-agent and cooperative multi-agent RI. benchmarks.
2 Background
Reinforcement learning (RL) deals with the problem of learning policies for sequential decision making in environment for which the dynamics are not fully known [23]. A common assumption is that the environment can be modelled as a Markov Decision Process (MDP) [7], typically defined as tuple (S, 80, A, RG P, where S is finite set of states, 80 E S is an initial state, A a finite set of actions, R: S x A x S - IR is a Markovian and stationary reward function that specifies the reward r(s, a, 8²) that agent gains from transitioning from state s to s by the execution of action a, P S x A + P[S] is a transition function denoting probability distribution p(s, , a, s') over next states s' when action a exccuted at state s, and y E [0. 1] is a discount factor. In this work we use factored MDPs [24]. where each state is described via set of random variables X = Xi...... X.. and where each variable X, takes on values some finite domain Dom(X.) state is an assignment of a value X; E Dom (X;) for each variable X,. To model multi-agent setting, we use a Markov game [25], which generalizes the MDP by including joint actions = { representing the collection of action sets A'z for each of the agents. We will hereon refer to an MDP the model of the underlying environment, and highlight as needed the specific considerations to a Markov game.
A solution to an RL problem is either stochastic policy, indicated π : S + P[A], representing a a probability of taking an action a at that state, or deterministic policy,
mapping from states s indicated π : S + A, mapping from states to single action. The agent's objective is to find a policy that maximizes the expected, total discounted reward.
There are a variety of approaches for solving RL problems [26, 23], these generally categorized as either policy gradient methods, which learn a numerical preference for executing each action, value-based methods, which estimate the values of rate-actionn pairs, or actor-critic methods, which combine the value and policy optimization approaches. Another important distinction exists between model-based methods, where a predictive model is learned, and model-free methods, which learn a policy directly. We support this variety by assuming the algorithm that is used by the actor compute its policy is part of our input.
3 MDP Transforms
We use MDP transforms to explain the behaviors of RI. agents. Given a large set of possible trans- forms, an explanation is generated by searching for set of transforms to apply the environment's

---

model such that the actor's behavior in the modified model aligns with the observer's expectations. Since the transition from the original to the transformed environment is done by manipulating the symbolic MDP representation of the environment, the difference between the models can help the observer reason about the actor's 's behavior, thus providing an explanation.
In this section, we describe various transforms suggested in the literature for expediting planning and RL, and that we apply here for the purpose of explainability. We define transform any mapping T: M + M that can be applied to an MDP to produce another MDP. We use the term "transforms" to refer to various kinds of mappings, including "abstractions" (or "relaxations") that are typically used to simplify planning, as well as other mappings that may yield more complex environments. Moreover, the set of transforms used for explanation may modify different elements of the MDP instead of focusing on a specific element (e.g, the reward function). We provide some examples of transforms, but our framework is not restricted to particular transforms. We start by defining transforms that modify the MDP's state space.
Definition (State Mapping Function) A state-mapping function Φ - S maps each state E S, into state s' 8 The inverse image o "s') with s' E S is the set of states in S that map to s' under mapping function ø.
When changing the state space of an MDP, we need to account for the induced change to the other elements of the model. For this, we use a state weighting function that distributes the probabilities and rewards of the original MDP among the states in the transformed MDP.
Definition 2 (State Weighting Function) [11] A state weighting function of state mapping func- tion é is function w + [0, where for every E S Es w(s) 1
Definition 3 (State-Space Transform) [11] Given a state mapping function & and a state weighting function W, a state space transform Te maps an MDP M = (S, 50, A, R, P,7) to T(M) = (S, So, A, R, P,7) where:
mupping.function
$ 8
((50)
Va A, R(3,a) Ese ( w(s)R(ssa)
Va A, P(3,a,3') =
=cco P(5.a,8)==
Eveo 1(z) w(s) P(s,,))
State-space transforms can, for example, group states together. In factored representations, this can be easily implemented by ignoring a subset of the state features. In Example 1, state-space transform can, for example, ignore the fuel level, grouping states that share the same taxi and passenger locations.
Another family of transforms changes the action space.
Definition 4 (Action Mapping Function) An action mapping function ab :::AA A - A maps every action in A an action in A The inverse image 2 (aa for a' A the set of actions in A that map to a' under mapping function tb.
Various action space transforms have been suggested the literature for planning with MDPs [27, 28). Since such transforms inherently bear the MDP's symbolic meaning with regard to the environment and agent, a sequence of transforms that yields the anticipated policy can provide a suitable explanation.
As an example, even if the exact transition probabilities of actions are not fully known, it is possible to apply the single-outcome determinization transform, where all outcomes of an action are removed (associated with zero probability) except for one, perhaps the most likely outcome or the most desired outcome [29]. Similarly, the all outcome determinization transform allows a planner to choose a desired outcome, typically implemented by creating separate deterministic action for each possible outcome of the original formulation [29, 27]. If such transforms yield the anticipated policy, this implies that the observer may not aware of the alternative outcomes of an action, or of the stochastic nature of the environment. In settings where actions are associated with preconditions,
5

---

is possible to apply a precondition relaxation transform, where subset of the preconditions of an action are ignored [22]. For example, for MDPs represented via factored state space, each action is associated with a set pre(a) specifying the required value of a subset of its random variables. A precondition relaxation transform removes the restriction regarding these variables. Similarly, is possible to ignore some of an action's 8 effects, for example by applying delete relaxation transform and ignoring an actions' effect on Boolean variables that are set to false [9]. As another example, a precondition addition transform would add preconditions to an action, perhaps those that may be considered by the observer by mistake. In all cases, if one or more transforms produce the anticipated policy, a plausible explanation is that the observer is not aware of the preconditions or effects of actions, such as in the setting we describe in regard to fuel in Example 1.
The transforms mentioned above are also applicable to multi-agent settings. In addition, we can apply multi-agent specifie transforms, such as those that allow collisions between agents, or allow for more flexible communication. In multi-agent extension of our taxi example, an observer may not be aware that taxis cannot occupy the same cell-a discrepancy that can be explained by applying a transform that ignores the constraint (precondition) that a cell needs to be empty for taxi to be able to move into it.
Transforms as Explanations
We formalize the explainability problem as composed of three entities: an actor, which is an agent operating in the environment, an observer, which is an agent with some anticipation about the behavior of the actor, and an explainer, which is agent that wishes to clarify the discrepancy between the anticipated and actual behaviors. The input to a Reinforcement Learning Policy Explanation (RLPE) problem includes description of the environment (which may be inaccurate), a description of the behavior (policy) of an RL agent in the environment, the anticipated behavior an observer expects the actor to follow, and set of possible transforms that can be applied the environment.
Definition 5 (RLPE Model) A Reinforcement Learning Policy Explanation (RLPE) model is defined as R (M, A. म, T), where
M an MDP representing the environment,
A: M - II the actor, which is associated with an RL algorithm that uses to compute policy E 11,
* # is the anticipated policy the observer expects the actor to follow, and
. : + -M is a.fintte set of transforms.
We assume the actor is a reward-maxken maximizig RL agent'. The anticipated behavior of the observer de- scribes what the observer expects the actor to do in some subset of the reachable states", Since we do require the anticipated policy to be defined over all states, we refer this as a partial policy, The settings of interest here are those in which the actual policy differs from the anticipated policy. We denote by T the set of all transforms. Each transform T E is associated with a mapping function for each of the MDP elements that it alters. We let ΦT and urT denote the state and action mapping functions, respectively (when the MDP element is not altered by the transform, the mapping represents the identity function). When sequence of transforms is applied, we refer to the composite state and action mapping that it induces, and define this as follows.
Definition 6 (Composite State and Action Space Function) Given sequence T = (T,,..., Tn), T, E T, the composite state space function of T, is Φ)((5) = ΦT. ..... action space function is (s) = VT,," ...., VT, (s).
",...,,TTT &T, (s). The composite
The explainer seeks sequence of transforms that produce an environment where the actor follows a policy that corresponds to the observer's anticipated policy. Formally, we seek a transformed environment where the actor's policy satisfies the anticipated policy, i.e., for every state-action
$ For the multi agent case, instead single agent we have group agents. All other elements are unchanged.
'The model can be straightforwardly extended to support a set of possible anticipated policies.

---

pair in the anticipated policy, the corresponding state in the transformed model is mapped to its corresponding action. Given a policy π, we let (( represent the set of states for which the policy is defined.
Definition (Policy Satisfaction) Given a partial policy a defined over MDP M (S, so, A. R. P. 7),, a partial policy x' defined over MDP M' (S', A', R', P', "",) state mapping function o S + S and an action mapping function ib A + A', at' satisfies n, denoted x' = n, for every E S(T)), we have ((s) E S(T') and ()))(s)) x (ó(s))
Intuitively, policy x' satisfies x if they agree on the agent's selected action on all states for which a is defined. We note that definition above is suitable only if (s)) and x (o)()) are well-defined, i... if the policies are deterministic or, if they are stochastic, a deterministic mapping from states to actions given (e.g., selecting the maximum probability action).
Clearly, for any two policies, there exist state and action mappings that can be applied to cause any policy to satisfy another policy. In order to produce valuable explanations, the input needs to include suitable transforms, i.c., transforms that change the environment in a way that highlights the elements of the model that cause unanticipated behaviors. In addition, and inspired by the notion of Minimal Sufficient Explanation [8], we want to minimize the change that is applied to the environment. Intuitively, the more similar the original and transformed MDPs are, the better the explanation. We therefore assume the input to an RLPE problem includes some distance metric, d .M x M - Rt+, between pair of MDPs [30]. In our evaluation, the distance represents the number of atomic changes that change a single element of the MDP (see the supplementary material for a description of several other distance metries from the literature).
The objective of the explainer is to find sequence of transforms that yield an MDP M' such that the actor's 's policy in M' satisfies न. Among the sequences that meet this objective, we are interested in sequences that minimize the distance between the original and the transformed MDP. Formally:
Definition 8 (RLPE Problem] Given RI.PE model R and metric function d M x M R+. an RLPE problem seeks a transform sequence T (Tr..... Tn), T₁ T, S.I.
1. the actor's policy x' in T(M) satisfies म,, i.e, n' F t,, and
2. among the sequences that satisfy (1.), T minimizes the distance ((M,T(M)).
Finding Explanations
In an RLPE setting, the explainer has access to set of transforms, but does not know priori which transform sequence will produce meaningful explanations. This means that the explainer may need to consider a large set of possible transform sequences. This makes a naive approach impractical, as the number of transform combinations is exponential in (T).
To address this computational challenge, we offer several approaches for expediting the search, Inspired by the search for an optimal MDP redesign in [31], basic approach is a Dijkstra-like search through the space of transform sequences. Assuming a successor generator is available to provide the MDP that results from applying each transform, the search graph is constructed in the following way. The root node is the original environment. Each edge (and successor node) appends single transform to the sequence applied to the parent node, where the edge weight represents the distance between the adjacent MDPs according to the distance measure d. For each explored node we examine whether the actor's 's policy in the transformed MDP satisfics the anticipated policy. The search continues until such model is found, or until there are no more nodes to explore. The result is transform sequence that represents an explanation. This approach is depicted in Figure 2, where the top of the figure depicts the search in the transform space and the lower part depicts the MDPs corresponding each sequence.
The suggested approach is guaranteed to return an optimal (minimum distance) solution under the assumption that the distance additive and monotonic with respect to the transforms in T, in that a transform cannot decrease the distance between the resulting MDP and the original one. From a computational perspective, even though in the worst case this approach covers all the possible sequences, in practice it may find solutions quickly. In addition, in cases where the transforms
7

---

M
Initial Environment
7
T2
TylMal
Ts Tortiial
T4
Ts
T
Farraageeet
Figure 2: A search for transform- based explanations in the transform space.
Figure 3: Evaluation domains including
single-agen and multi multi-agent settings.
independent, in that their order of application does not affect the result, it is possible to expedite the search by maintaining a closed list that avoids the re-computation of examined permutations. The depth of the search can also be bounded by predefined fixed number of transforms.
In spite of these computational improvements, the above solutions require learning from scratch an actor's policy in the transformed environment for each explored node. One way to avoid this is by preserving the agent's policy in given environment and using it for bootstrapping re-training in the transformed environment. Another way expedite the search is to group together set of transforms and examine whether applying the set leads to a change in the actor's 's policy. If this compound transform does not change the actor's policy, individual transforms. This approach is inspired by pattern database (PDB) search heuristics [32], as well the relaxed modification heuristic [31]. Even though this heuristic approach compromises optimality, it can potentially reduce the computational effort in settings in which aggregation can be done efficiently, such when transforms have parameterized representations. In our example, if allowing a taxi to move through (all) walls a given environment does not change the actor's 's policy, avoid computing the value of all individual transforms that remove single wall. Finally, we examine the efficiency of performing a focused policy update: when applying transform, instead of collecting random experiences from the environment and updating the policy for all states, we start by collecting new experiences from states that are directly affected by the transform, and then follow the propugated effect of this change. In Example 1, when removing a wall in the taxi domain, we start by collecting experiences and updating the policy of states that are near the wall, and iteratively follow the propagated effect of this change on the policy in adjacent cells.
avoid computing the values of the
6 Empirical Evaluation
The empirical evaluation was dedicated to examining the ability to produce meaningful explanations via MDP transforms and to examining the empirical efficiency of the suggested approaches for finding satisfying explanations. Each RLPE setting included a description of the underlying environment, the actual policy followed by the actor, and the anticipated policy. We describe each component below, before describing our results².
We conducted experiments with 12 different environments, including both determin- istic and stochastic domains and single and multi-agent domains (see Figure 3). Frozen Lake [33] represents stochastic grid navigation task, with movements in all four cardinal directions and a probability of slipping (and terminating). As demonstrated in Example 1, Taxi is an extension of the similar Open-Al domain (which in turn is based on [15]), with fuel constraint that needs to be satisfied in order to move and actions that correspond to refueling the car at a gas station. Apple-Picking is our stochastic extension of the Taxi domain: reward is achieved only when picking up passenger (i.e., an "apple") and the session can terminate with some probability when an agent encounters thorny wall. We also used seven PDDLGym domains [34]: Sokoban, Blocks World, Towers of Hanoi, Snake, Rearrangement, Triangle Tireworld, and Exploding Blocks. The PDDLGym
Environments:
Additional results and extensions can be found in the supplementary material. Our complete dataset and code can be found
8

---

BASE
PRE-TRAIN
CLUSTER
DQN
Frozen Lake
Blocks World
Taxi
Tax
Snake
*trnnol
0.98
Rearranpement
Petrrangement
Sovobon
Henoi
Apple Picking
Blocks World State
Exploding Blocks
Policy satisfaction
3.94
Rearzar prrgementt
Apole
Apple Picking
Tire world
Frozen
0.92
Bllod Stisee
Sollobon
Feplading Mlodes
0.9
Sokotan
3.66
Exllddin Blocks
Solution time
Figure 4: Single-Agent DQN [36]: Average running time for finding explanation for each domain (x-axis), and policy satisfaction ratio, measuring the ratio of instances for which satisfying policy was found (y-axis).
BASE
PRE-TRAIN
PRE-CLUSTER
PPO
MultiTae
SulliSn
0.96
0.94
Multi Sokoban
0.92
MultiTasi
0.9
MultiSollbban
0.88
Solution time
Figure 5: Multi-Agent: Comparing average running time for finding explanation for the multi-agent domains (x-axis), and policy satisfaction ratio (y-axis). The actors are using PPO [37].
framework aligns with the OpenAl Gym interface while allowing the user to provide model-based relational representation of the environments using PDDL [35]. This representation is not available to the actor, which operates using standard algorithms. For multi-agent domains, we created two-agent Sokoban in which agents need to avoid colliding with each other and also provide a Multi-Taxi domain that includes multiple taxis that may collide and need transport multiple passengers All these domains have delayed rewards and require multi-step reasoning, making them challenging for standard RL methods.
We considered a partially informed observer that has access to subset of the environment features. For example, Taxi the observer may be unaware of the fuel constraint or may not be able to see the walls. For all environments we assume the observer anticipates that the actor follows a policy that is optimal w.r.t. the observer's possibly incomplete or inaccurate model. Plans were produced using [38].
Observer:
SSee https: http:::///ithie aaasssaraaaaeeaae taxi

---

Actor:
For the single-agent settings, we used DQN [36], CEM [39], and SARSA [23] from the keras-rl library", as well as Q-learning [40]. For the multi-agent domains, we used PPO [37] from keras-rl. Agents were trained for 600,000-11,00,0000 episodes in each environment, with a maximum of 60 steps per episode.
Explainer: We used five paramterized transform types: state space reduction [29]. likely outcome relaxation [29], precondition relaxation [22], all outcome determinization (for stochastic domains) [41], and delete relaxation [9]. Grounding (i.e., the instantiation of the parameterized representations) was performed automatically for each transform for all environments in which it is applicable. Each grounded transform modifies a single action or variable. For the Frozen Lake, Taxi, and Apple Picking domains, where the dynamics are not defined explicitly, we first learn the transition matrix to generate the precondition relaxation transform.
We used three methods for searching for explanations. BASE is Dijkstra search, PRE-TRAIN is a Dijkstra search using the learned policy in a given environment to bootstrap learning in the modified environment, and with a focused policy update to avoid iteratively updating the entire policy, PRE+CL CLUSTER extends PRE-TRAIN by computing values of groups of transforms ge.g., applying the delete relaxation to multiple actions) and using them to prune individual transforms for which the superset did not change the ratio of states for which the anticipated policy is satisfied. Experiments were run on a cluster using six CPUs, each with four cores and 16GB RAM. We limited the depth of the search tree to three.
To assess the ability produce explanations using environment transforms, measured the satisfaction ratio of each transform sequence. This measure is defined as the fraction of states for which the anticipated policy and actor policy agree among all states for which the anticipated policy is defined, i.e., the number of states E S(T)) for which @(s) E S(x') and (s)) = a r((((s)) For distance measure d, we used the length of the explanation, i.e., the number of atomic transforms (each changing single element of the MDP) that were applied.
Results:
Figure 4 gives the results achieved by each method for the aggle-agent domains and with actor that uses DQN. Figure 5 gives the results for the multi-agent settings, with PPO used by the agents. Each plot represents, for each domain and each method, the average computation time for finding an explanation (x axis) and the average satisfaction ratio (y axis), i.e., the average ratio of the expected policy that was satisfied before the search exhausted the computational resources. Results for the single agent domains show that while BASE achieves the highest satisfaction ratio (which is to be expected from optimal algorithm), its computation time is much higher, requiring more than 7x the time of PRE+CLUSTER in Triangle Tireworld. In contrast, PRE+CLUSTER outperforms all other methods in terms of computation time, still with 84% success in the worst case domain, and with a maximum average variance of 0.03 over the different domains. The results are similar for the multi-agent settings, where the PRE+CLUSTER approach achieved best run time results on both domains while compromising the policy satisfaction rate by up to 10%.
Conclusion
We introduced a new framework for explainability in RL based on generating explanations through the use of formal model transforms, which have previously been primarily used for planning. The empirical evaluation on a set of single and multi-agent RL benchmarks illustrates the efficiency of the approach for finding explanations among large set of transforms.
Possible extensions include integrating human users or models of human reasoning into the process of generating anticipated policies and in the process of evaluating the quality of the explanations generated by our methods. In addition, while this work uses a restrictive satisfaction relation that requires full match between the anticipated policy and the actor's behavior in discrete domains, it may be useful to account for continuous domains and to use more flexible evaluation metrics for satisfaction that allow, for example, finding transforms that get close as possible to the anticipated policy. Finally, our current account of multi-agent settings focuses on fully cooperative settings and it would be interesting extend this framework account for adversarial domains.
10

---

8 Acknowledgments
This research has been partly funded by Israel Science Foundation grant #1340/18 and by the Euro- pean Research Council (ERC) under the European Union's Horizon 2020 Research and Innovation Programme (grant agreement no. 740282).
References
[1] Alejandro Barredo Arrieta, Natalia Diaz Rodrigodr,,,,, Javier Del Ser, Adrien Bennetot, Siham Tabik, A. Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence: Concepts, taxonomies, opportunities and challenges toward responsible AI. Journal Advances in Information Fusion, 2020.
[2] Itamar Arel, C. Liu, Thomas Urbanik, and Airton G. Kohls. Reinforcement learning- baaningg-bsedd multi-agent system for network traffic signal control. Journal of Intelligent Transportation Systems, 2010.
[3] Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 2013.
[4] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. Mastering Atari, Go, Chess and Shogi by planning with a learned model. Nature, 2020.
[5] Erika Puiutta and Eric MSP Veith. Explainable reinforcement learning: survey. ArXiv, 2020.
[6] Sam Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. ICML, 2018
[7] Dimitri P. Bertsekas. Dynamic Programming and Optimal Control, volume I. Athena Scientific, 3rd edition, 2005.
[8] Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi-Veled. Explainable reinforcement learning via reward decomposition. In IJCAI-XAI, 2019.
[9] Jörg Hoffmann and Bernhard Nebel. The ff planning system: Fast plan generation through heuristic search. Journal of Artificial Intelligence Research (JAIR), 2001.
[10] Blai Bonet and Hector Geffner. mgpt: A probabilistic planner based on heuristic search. Journal of Artificial Intelligence Research (JAIR), 2005.
[11] Lihong Li, Thomas J. Walsh, and Michael Littman. Towards a unified theory of state abstraction for mdps. In ISAIM, 2006.
[12] Malte Helmert. The fast downward planning system. Journal of Artificial Intelligence Research (JAIR), 2006.
[13] Hector Geffnnr and Blai Bonet. A concise introduction models and methods for automated planning. Morgan and Claypool Publishers, 2013.
[14] David Abel, Dilip Arumugam, Lucas Lehnert, and Michael Littman. State abstractions for lifelong reinforcement learning. In ICML, 2018.
[15] Thomas G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of artificial intelligence research (JAIR), 2000.
[16] Zhengyao Jiang and Shan Luo. Neural logic reinforcement learning. In ICML, 2019.
[17] Stephanie Milani, Nicholay Topin, Manuela Veloso, and Fei Fang. Explainable reinforcement learning: A survey, ArXiv, 2022.

---

[18] Zhengxian Lin, Kim-Ho Lam, and Alan Femn Contrastive explanations for reinforcement learning via embedded self predictions. ICLR, 2021.
[19] Tobias Huber, Katharina Weitz, Elisabeth Andre, and Ofra Amir. Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps. Artificial Intelligence Journal (AIJ), 2021.
[20] Terrell N. Mundhenk, Barry Y. Chen, and Gerald Friedland. Efficient saliency maps for explainable AL ArXiv, 2019.
[21] Dan Amir and Ofra Amir. Highlights: Summarizing agent behavior to people. In AAMAS, 2018.
[22] Sarath Sreedharan, Utkash Soni, Mudit Verma, Siddharth Srivastava, and S. Kambhampati. Bridging the gap: Providing post-hoc symbolic explanations for sequential decision-making problems with black box simulators. ArXiv, 2020.
[23] Richard S Sutton and Andrew G Barto. Reinforcement learning: An Introduction. MIT press. 2018.
[24] Craig Boutilier, Thomas L. Dean, and Steve Hanks. Decision-theoretic planning: Structural assumptions and computational leverage. Journal of Artificial Intelligence Research (JAIR), 1999.
[25] Michael Littman. Markov games as framework for multi-agent reinforcement learning. Machine Learning Proceedings, 1994.
[26] Csaba Szepesvari. Algorithms for Reinforcement Learning. Morgan and Claypool Publishers, 2010.
[27] Mausam and Andrey Kolobov. Planning with markov decision processes: an Al perspective. Morgan and Claypool Publishers, 2012.
[28] David Abel, Nathan Umbanhowar, Khimya Khetarpal, Dilip Arumugam, Doina Precup, and
M. Littman. Value preserving state-action abstractions. In AISTATS, 2020.
[29] Sung Wook Yoon, Alan Fern, and Robert Givan. FF-replan: A baseline for probabilistic planning. In ICAPS, 2007.
[30] Jinhua Song, Yang Gao, Hao Wang, and Bo An. Measuring the distance between finite markov decision processes. In AAMAS, 2016.
[31] Sarah Keren, Luis Pineda, Avigdor Gal, Erez Karpas, and Shlomo Zilberstein. Efficient heuristic search for optimal environment redesign. In ICAPS, 2019.
[32] Stefan Edelkamp. Automated creation of pattern database search heuristics. In MoChArt, 2006.
[33] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. ArXiv, 2016.
[34] Tom Silver and Rohan Chitnis. Pddlgym: Gym environments from pddl problems. In ICAPS- PRL, 2020.
[35] Maria Fox and Derek Long. PDDL2.1: An extension to pddl for expressing temporal planning domains. Journal of artificial intelligence research (JAIR), 2003.
[36] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing Atari with deep reinforcement learning. CoRR, 2013.
[37] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Explainable reinforcement learning: A survey. AzXiv, 2017.
[38] Mau Magnaguagno, Felipe Meneguzzi, and Anwar. pucssattoonttdlllaninngdddd.dppp Version I.I. December 2020.
12

---

[39] István Szita and András Lörinez. Learning Tetris using the noisy cross-entropy method. Neural computation, 2006.
[40] C. J. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, King's College, Oxford, 1989.
[41] Thomas Keller and Patrick Eyerich. A polynomial all outcome determinization for probabilistic planning. In ICAPS, 2011.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative social impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material or URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "mieK-h3iejF", "reviewer": "Reviewer_Bq81", "summary": "The paper proposes the use of MDP transforms for autonomously producing explanations. The work also introduces and formally defines the RLPE problem and empirically demonstrates the performance of their proposed approach in single and multiagent environments.", "questions": "Major comments:\n\n1. The key idea is that an observer has an anticipated policy in mind which is assumed to arise from partial knowledge of the MDP. The explanation then corresponds to an MDP transformation/sequence of MDP transformations whose solution closely matches (satisfies) the anticipated policy. Although this approach might be reasonable in some cases, it is possible that the transformation sequence found to satisfy the anticipated policy may not actually constitute a good explanation. Since there may be multiple transformation sequences whose solution would satisfy the anticipated policy, only some of these may correspond to a meaningful explanation. \n\n2. Regarding the distance metric criterion used: - if the anticipated policy was indeed derived from a transformed MDP whose distance to the original MDP is large, then choosing a transformation sequence with a low distance measure may not be ideal (because the anticipated policy actually corresponds to a transformed MDP with a large distance to the original). It would help to include some lines discussing what motivates the inclusion of the distance metric. Wouldn’t it be better to instead consider for example the length of the transformation sequence and select ones with lower sequence lengths (as smaller lengths might mean simpler explanations)\n\n3. How does one choose a suitable set of transformation functions? This is probably a critical choice because a poor set of transformations may still be able to match the anticipated policy, but may not lead to meaningful explanations.\n\nMinor comments:\n\n1. The figure captions need to be more detailed. Eg: It is not clear what the yellow rectangle/purple ‘x’ in fig 1 mean.\n2. The phrasing of lines 164-165 is confusing\n3. In lines 351-355, I am curious as to why A* was used – was it because it assumes the existence of some heuristic signal which might simulate an observer’s assumptions? Why not use say standard Q learning for example?\n4. It would be good to see results in more complex/high dimensional environments. The chosen environments all seem relatively simple.\n5. The results only report the satisfaction ratio with the transformed MDPs. Reporting the satisfaction ratios on the original environments might better indicate how much the transformation sequence improves the satisfaction ratio relative to the original environment.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The article is clear and fairly well written, and the proposed approach seems novel and original. I believe the paper tries to tackle a very relevant and important problem. My main concerns with the work (as listed later) are related to the validity of the assumptions made with respect to the quality of explanations produced by the proposed approach. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Tn-xc9G_gf", "reviewer": "Reviewer_2xom", "summary": "This paper presents a new framework and algorithm for explainable RL.  Given some limited domain information and observed behavior by an actor, the authors propose to search through a space of possible model transformations in order to find a model and associated policy that matches the observed behavior.  The transformations then represent some aspect of deviation, and serve as a partial explanation for observed behavior.  The transforms are derived from RL literature on state-space aggregation; the search algorithm involves chaining them together. Each node in the search tree must be evaluated by computing a full policy for the transformed model, which can be computationally expensive, and which the authors discuss.  The paper has one set of experimental results illustrating an implementation of the ideas.", "questions": "\n- Why do we need an observer?  Why not eliminate that idea and simply state that there is an anticipated policy?\n\n- The claim on lines 274-275 isn't obviously true to me. Can you please explain this further? (I think I understand why you might think it's \"clear\", and I think you might be wrong, so it would be helpful to have it fleshed out)", "limitations": "I think the authors could have done a better job of explaining the limitations of their method (although I don't think they tried to hide anything).", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The framework is novel\n+ This is a reasonable problem to be working on\n+ The ideas are straightforward, and the paper is clearly written\n+ The experiment seems clear and helpful\n\nWeaknesses:\n\n- While I generally liked this paper, I consistently found myself wanting more. The paper is very \"chatty\" and does not do a good job of balancing exposition of the ideas (which are straightforward) with some sense of application or empirical evaluation (which is very, very thin).\n\n- I ultimately felt the problem (explainable RL) is important, but that the proposed framework should have been more general. I mean this in two specific ways:\n\n1) A more general characterization of the idea could be \"search for a model and corresponding optimal policy that matches observed behavior\". This subtly includes a host of ideas that involve /complexifying/ policies, instead of just /simplifying/ them.  For example, you discuss how removing a constraint from a symbolic planning problem might suggest that the agent was unaware of the constraint, but the opposite is also true - you could /add/ constraints to the problem,  to similarly better match behavior, that would suggest the agent is laboring under constraints that it doesn't need to.  People do this all the time!\n\n2) I am generally dissatisfied by the approach because I feel that it can only explain the /difference/ between the baseline policy and the transformed policies. Suppose, for example, that the optimal set of transforms is the null set -- in other words, the agent is behaving as expected.  This does NOT imply to me that the agent's behavior is explainable!  In other words, this framework does not contribute any explainability of the base policy. In my mind, this also motivates the more general approach of searching for matching models.  \n\n- The framework seems a bit limited to what feels like counterfactual reasoning \n\n- The computational cost is real.  As the space of transforms grows, the importance of searching that space effectively becomes increasingly important. You discuss how important a distance metric is, but it's unclear to me how often such a thing exists or is practical to compute.  I like the idea of somehow transferring partial solutions or bootstrapping solvers, but these seem to dance around the issue.  Ultimately, I'm lead to wonder: will this method ever be useful for real problems?\n\n- The experiments in the paper are underhwhelming. While the single provided experiment is clear and helpful, I feel like a much stronger empirical evaluation is necessary.  I would recommend trimming the writing to make room for experiments.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QqDPLcGSztY", "reviewer": "Reviewer_bxZ3", "summary": "This paper studies explanations of RL policies. The idea is quite slick. An agent executes a policy learned using RL. An observer has some expectations with respect to the behaviour of the agent (e.g. it may expect certain actions to be taken in certain states). The goal of explanations is to explain to the observer why the agent's policy does not match observer's expectations. This may explain to the observer that their expectations are incorrect since they are not compatible with the MDP. The expected behaviour may simply be unfeasible given the MDP, and the goal is to find those infeasible situations and offer them as explanations to the observer. The problem is well formulated, and the results on 12 domains show that the new method is feasible, and it can be implemented even in tasks that don't have a PDDL representation.", "questions": "The results are excellent, but it would be useful if the appendix, for example, showed a few examples of observer's expectations, and how they were addressed by the algorithm. I can imagine that the method works, but a few examples, perhaps one example per domain, would be very useful.\n\nLines 18-24 in the appendix explain how to deal with non-symbolic domains? Will the derivative vector always work? I am asking because perhaps there is a counterexample that could be mentioned or a theorem that could be cited that shows that there is no counterexample.\n\nThis is a strong piece of research and I don't have other questions. The paper is well written, and it was a pleasure to read it.", "limitations": "There is no negative societal impact in this work.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very clear and writing if of high quality. The authors took care to introduce all the necessary concepts, and the key terms have accurate formal definitions (e.g., definition 7 is excellent).\n\nThe main idea is innovative, and it appears original. The relevant literature is presented, and discussed in a way that allows the reader to see where this paper sits in the related work.\n\nExample 1 is very good, and it (along with the paragraph in lines 82-94) clarifies the goals. \n\nThe fact that the method is domain independent is a strength.\n\nThe fact that both an intuitive explanation and formal definitions are provided is a plus.\n\nSection 5 is very competent. The authors explain the challenges and propose a feasible solution.\n\nOne weakness is that it would be good to see what the explanations were found on the 12 domains that were evaluated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7I_uvQGeKJu", "reviewer": "Reviewer_5vMh", "summary": "This work proposes an explainable Reinforcement Learning framework, which seeks to explain the discrepancies between the learned \"actor\" policy and an anticipated \"observer\" policy of a partially informed observer. This is done automatically by (an \"explainer\") searching \n over an available set of (state and action) transforms and selecting (a composition) of transforms that tweak elements in the state of the policy (producing counterfactual states), such that the (re-learned) policy in the transformed MDP aligns with the expected policy. Since the authors work on a symbolic (relational) environments, the transforms can given an indication of the key elements that had previously been overlooked by the partially informed observer (which also explains the agent's behavior, hence explainable RL). The authors do an excellent job in formalizing the definitions and explaining the intuitions. Several heuristics are proposed to perform quicker searches over the set of transforms to reach perfect alignment while making minimal changes to the states/actions. The proposed framework is shown to be applicable to a variety of domains including that of single and multi-agent RL environments, and in stochastic settings. ", "questions": "1. lines 82:94 - In both examples, the reason behind the anticipation gap is due to the crucial information missing in the the observer's anticipated policy. How about the other way round: the actor's policy overlooking some crucial information. In other words, is the actor's policy always optimal? If not, the authors should clarify this just to avoid confusion.\n\n2. line 106-108: I am not fully convinced with this statement. Anticipated behavior needs some domain knowledge, no matter how general it is. Moreover, I believe that the more general an anticipated behavior is, the more number of transforms would be required to align both the policies (hence, more retraining in the transformed environments), which in some complex environments might be infeasible. \n\n3. line 185: Do these mapping functions modify multiple random variables X_i of the state or just one?\n\n4. Line 327-328: I failed to understand how the authors can tell this without training the policy in the transformed environment?\n\n5. Is exhaustive search used for all methods: Base, pre-train, pre-cluster? Or do you prune the transforms after a certain distance is reached?\n\nMinor Comments: \n1. To my understanding, there can be different definitions of explainability: for instance, an observer might not have any anticipations, however he/she might be interested in understanding \"how\" (or even \"why\") [1] a particular decision was arrived at. In this work, the authors focus on setups where an observer already has some expected behavior in mind. It would be good if the authors contrast this definition with those of others, or explicitly state that this is just one form of explainability they are dealing with. \n\n2. lines 252-254: Might be good to mention this upfront to avoid confusion. As a reader, I would definitely be in suspense otherwise.\n\n3. Scatter plots: The points should be made bigger and more distinct. Also please make the captions more detailed (like define what X-axis and Y-axis units are). As a reader, it is inconvenient to search for such details in the text.\n\n4. Line 374: We also measured the length of the explanation (i.e. the number of atomic ... ): I couldn't find this result in the main paper. It possible I may have missed it.\n\nline 61: \"is comprised of\" -> comprises\nline 147-148: \"where the set of states\" -> where each state\nline 198: s \\in \\phi_{-1}(\\bar{s})\nline 237: \"as comprised of\" -> as composed of\n\n[1] Neural Logic Reinforcement Learning, Zhengyao Jiang, Shan Luo Proceedings of the 36th International Conference on Machine", "limitations": "The authors do propose some limitations and future work to extend their framework in the Conclusion Section. However, to my understanding, there is a key limitation that may have been overlooked:  The authors consider a Relational (symbolic) domain, hence a discretized state with limited values. In continuous domains, with the introduction of feature extraction networks, the number of elements in a state (random variables) and its corresponding values would increase, thus increasing the set of transforms required to align the two policies. Will this render the search for transforms infeasible? I understand that this might not be withing the scope of this work (since it is in the preliminary stages), however it might be good to just comment on this for future developments in this direction.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This work raises valid questions on explainability in (Deep) RL, which is of great significance, given the black-box nature of the policies and their usage especially in safety critical domains like autonomous driving and treatment recommendation.\n2. The authors have done an excellent job in writing the paper. I thoroughly enjoyed going through the content. The explanations and intuitions are very clear.\n3. The proposed framework seems to be novel and is also generally applicable (agnostic of the RL framework). Empirical evaluations are performed on both single-agent and multi-agent environments with added stochasticity.\n4. The authors have submitted code for reproducibility (although I did not run the code).\n\nWeaknesses:\n1. Results: It is rather unfortunate that the authors chose to devote just half a page to the Results sections (with just one figure) and pushed the rest of the (significant and interesting) results to the appendix. The authors have devoted a lot of space to explain things that might not be required. For instance, Figure 2 is unnecessary and can be substituted with results from the Appendix. Also I would prefer detailed explanations like those in Appendix 4 in the main paper.\n2. Proposed Framework: Although the authors state that they use a RL-setting where the transition function is unknown, some action transforms that they propose work on precondition or postcondition (action effect) relaxation which is readily available in planning domains but NOT in RL domains and must be learned by the policy. Moreover, the experiments only demonstrate explainability on a limited set of transforms like all-outcome determinization (see Appendix 4) and ignore the rest.\n3. Comparisons with prior work: The authors state in lines 126-130 that the work by Sreedharan et. al., 2020 is similar to theirs in deterministic environments. However, there are not comparisons presented for the deterministic experiments. I wonder why.\n4. The first and the second contribution seem to go hand-in-hand: To use model transforms in the explainable RL setup, one needs to define it formally. \n5. Details from the experimental section (like comparisons with previous work, model details, algorithm details) are either vaguely mentioned or are pushed to the Appendix: See Questions 3,4,5\n\nOverall, the paper would benefit a lot from a rewrite, especially the Experiments and the Results sections. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept