Interpreting Reinforcement Policies through Local Behaviors

1 INTRODUCTION . Deep reinforcement learning has seen stupendous success over the last decade with superhuman performance in games such as Go ( Silver et al. , 2016 ) , Chess ( Silver et al. , 2018 ) as well as Atari benchmarks ( Mnih et al. , 2015 ) . With increasing superior capabilities of automated ( learning ) systems , there is a strong push to understand the reasoning behind their decision making . One motivation is for ( professional ) humans to improve their performance in these games ( Rensch , 2021 ) . An even deeper reason is for humans to be able to trust these systems if they are deployed in real life scenarios ( Gunning , 2017 ) . Safety , for instance , is of paramount importance in applications such as self-driving cars or deployments on unmanned aerial vehicles ( UAVs ) ( Garcia & Fernandez , 2015 ) . The General Data Protection Regulation ( Yannella & Kagan , 2018 ) passed in Europe demands that explanations need to be provided for any automated decisions that affect humans . While various methods with different flavors have been provided to explain classification models ( Ribeiro et al. , 2016 ; Lundberg & Lee , 2017 ; Lapuschkin et al. , 2016 ; Dhurandhar et al. , 2018 ) and evaluated in application-grounded manner ( Doshi-Velez & Kim , 2017 ; Dhurandhar et al. , 2017 ) , the exploration of different perspectives to explain reinforcement learning ( RL ) policies has been limited and user study evaluations comparing methods are rarely employed in this space . In this paper , we provide a novel perspective to produce human understandable explanations with a task-oriented user study that evaluates which explanations help users predict the behavior of a policy better . Our approach involves two steps : 1 ) learning meta-states , i.e. , clusters of states , based on the dynamics of the policy being explained , and 2 ) within eat meta-state , identifying states that act intermediate goals , which we refer to as strategic states . Contrary to the global nature of recent explainability works in RL ( Topin & Veloso , 2019 ; Sreedharan et al. , 2020 ; Amir & Amir , 2018 ) , our focus is on local explanations ; given the current state , we explain the policy moving forward within a fixed distance from the current state . This key distinction allows us to consider richer state spaces ( i.e. , with more features ) because the locality restricts the size of the state space we consider , as will be demonstrated . It is also important to recognize the difference from bottlenecks ( Menache et al. , 2002 ; Simsek & Barto , 2004 ) which are policy-independent and learned by approximating the state space with randomly sampled trajectories ; rather than help explain a policy , bottlenecks are used to learn efficient policies such as through hierarchical RL ( Botvinick et al. , 2008 ) or options frameworks ( Ramesh et al. , 2019 ) . Strategic states , however , are learned with respect to a policy and identified without assuming access to the underlying topology . An example of this is seen in Figure 1a . Each position is a state and a meta-state is a cluster of possible positions ( states sharing a color/marker ) . Within each meta-state , we identify certain states as strategic states ( shown with larger markers ) , which are the intermediate states that moving towards will allow the agent to move to another meta-state and get closer to the goal state , which is the final state that the agent wants to get to . In Figure 1a , each room is ( roughly ) identified as a meta-state by our method with the corresponding doors being the respective strategic states . Topology refers to the graph connecting states to one another ; our method only has access to the knowledge of which states are connected ( through the policy ) , whereas reinforcement learning algorithms might have access to properties of the topology , e.g. , the ability to access similar states using successor representations ( Machado et al. , 2018 ) . In Figure 1 , the topology is a graph connecting the different positions in each room or the doors connecting one room to another . A key conceptual difference between our approach and others is that other methods aggregate insight ( i.e . reduce dimension ) as a function of actions ( Bastani et al. , 2018 ) or formulas derived over factors of the state space ( Sreedharan et al. , 2020 ) to output a policy summary , whereas we aggregate based on locality of the states determined by the expert policy dynamics and further identify strategic states based on these dynamics . Still other summarization methods simply output simulated trajectories deemed important ( Amir & Amir , 2018 ; Huber et al. , 2021 ) as judged by whether or not the action taken at some state matters . We use the term policy dynamics to refer to state transitions and high probability paths . We use the term dynamics because this notion contrasts other methods that use actions to explain what to do in a state or to identify important states ; strategic states are selected according to the trajectories that lead to them , and these trajectories are implicitly determined by the policy . The example in Figure 1 also exposes the global view of our explanations when the state space is small because local approximations of the state space are not needed . We show that this perspective leads to more understandable explanations ; aggregating based on actions , while precise , are too granular a view where the popular idiom can ’ t see the forest for the trees comes to mind . We conjecture that the improved understanding is due to our grouping of states being more intuitive with strategic states indicating tractable intermediate goals that are easier to follow . An example of this is again seen in Figures 1b and 1c , where grouping based on actions for interpretability or for efficiency leads to less intuitive results ( note that Figure 1c replicates Figure 4b from ( Abel et al. , 2019 ) ) . A more detailed discussion of this scenario can be found in section 5 , where yet other domains have large state spaces and require strategic states to explain local scenarios . As such , our main contributions are two-fold : 1 . We offer a novel framework for understanding RL policies , which to the best of our knowledge , differs greatly from other methods in this space which create explanations based on similarity of actions rather than policy dynamics . We demonstrate on three domains of increasing difficulty . 2 . We conduct a task-oriented user study to evaluate effectiveness of our method . Task-oriented evaluations are one of the most thorough ways of evaluating explanation methods ( Doshi-Velez & Kim , 2017 ; Lipton , 2016 ; Dhurandhar et al. , 2017 ) as they assess simulatability of a complex AI model by a human , yet to our knowledge , have rarely been used in the RL space . 2 NOTATION . We use the following notations . Let S define the full state space and s ∈ S be a state in the full state space . Denote the expert policy by πE ( · , · ) : ( A , S ) → R where A is the action space . The notation πE ∈ R|A|×|S| is a matrix where each column is a distribution of actions to take given a state ( i.e. , the policy is stochastic ) . Note that we assume a transition function fE ( · , · ) : ( S , S ) → R that defines the likelihood of going from one state to another state in one jump by following the expert policy . Let Sφ = { Φ1 , ... , Φk } denote a meta-state space of cardinality k and φ ( · ) : S → Sφ denote a meta-state mapping such that φ ( s ) ∈ Sφ is the meta-state assigned to s ∈ S. Denote m strategic states of meta-state Φ by GΦ = { gΦ1 , ... , gΦm } where gΦi ∈ S ∀i ∈ { 1 , ... , m } . 3 METHOD . We now describe our algorithm , the Strategic State eXplanation ( SSX ) method , which involves computing shortest paths between states , identifying meta-states , and selecting their corresponding strategic states . However , we first define certain terms . Recall that all paths discussed below are based on transitions dictated by an expert policy because we want to explain the policy ; the well-known concept called bottlenecks are identified from paths generated as random walks through the state space and are meant to help learn policies rather than explain them . Maximum likelihood ( expert ) paths : One criterion used below is that two states in the same metastate should not be far away from each other . The distance we consider is the most likely path from state s to state s′ under πE . Consider a fully connected , directed ( in both directions ) graph where the states are vertices and an edge from s to s′ has weight − log fE ( s , s′ ) . By this definition , the shortest path is also the maximum likelihood path from s to s′ . Denote by γ ( s , s′ ) the value of this maximum likelihood path and Γ ∈ R|S|×|S| a matrix containing the values of these paths for all pairs of states in the state space . Γ , along with a predecessor matrix P that can be used to derive the shortest paths , can be computed using Dijkstra ’ s shortest path algorithm in O ( |S|2 log |S| ) because all edge weights are non-negative . Section 3.4 below discusses how our algorithm is applied with a large state space . Note that computation of Γ means that SSX requires access to a policy simulator for πE , and in practice , might require simulation for estimation when Γ can not be computed exactly . This is a common requirement among related explanation methods , e.g. , in order to simulate important trajectories Amir & Amir ( 2018 ) or samples to train a decision tree Bastani et al . ( 2018 ) , that are discussed below in Section 4 . Counts of Out-paths : Another criterion used below for assigning states to meta-states is that if state s lies on many of the paths between one meta-state Φi and all other meta-states , then s should be assigned the meta-state Φi , i.e. , φ ( s ) = Φi . We define , for fixed state s and its assigned meta-state φ ( s ) , the number of shortest paths leaving φ ( s ) that s lies on . Denote T ( s , s′ ) as the set of states that lie on the maximum likelihood path between s and s′ , i.e. , the set of states that define γ ( s , s′ ) . Then 1 [ s ∈ T ( s′ , s′′ ) ] is the indicator of whether state s lies on the maximum likelihood path between s′ and s′′ , and we compute the count of the number of such paths for state s and meta-state φ ( s ) via C ( s , φ ( s ) ) = ∑ s′ 6=s , φ ( s′ ) =φ ( s ) ∑ s′′ : φ ( s′′ ) 6=φ ( s ) 1 [ s ∈ T ( s′ , s′′ ) ] . ( 1 ) C ( s , φ ( s ) ) can be computed for all s ∈ S in O ( |S|2 ) by iteratively checking if predecessors of shortest paths from each node to every other node lie in the same meta-state as the first node on the path . Note this predecessor matrix was already computed for matrix Γ above . One may also consider the likelihood ( rather than count ) of out-paths by replacing the indicator in eq . ( 1 ) with γ ( s′ , s′′ ) .

Summary: The paper proposes an approach to interpret a black-box control policy of a reinforcement learning (RL) problem such that its interpretations can be understood by a human user. For a given expert policy (i.e., a fitted deep neural network), the proposed approach uses transition probabilities induced by the expert policy to define regions of the state space that are “similar.” These policy-dependent regions are referred to as meta-states and are computed using an algorithm similar to spectral clustering. The proposed method then computes so-called strategic states for each meta-state, where strategic states of a meta-state are those states that belong to the meta-state and bridge the meta-state to other ones. In other words, the expert policy should pass through the strategic states frequently when it goes from one meta-state to another. The paper performs numerical experiments using standard RL applications, i.e., four rooms, door-key, and mini-Pacman, to show the effectiveness of the proposed approach.