Ranking Policy Decisions

1 Introduction . Reinforcement learning is a powerful method for training policies that complete tasks in complex environments . The policies produced are optimised to maximise the expected reward provided by the environment . While performance is clearly an important goal , the reward typically does not capture the entire range of our preferences . By focusing solely on performance , we risk overlooking the demand for models that are easier to analyse , predict and interpret [ 16 ] . Our hypothesis is that many trained policies are needlessly complex , i.e. , that there exist alternative policies that perform just as well or nearly as well but that are significantly simpler . This tension between performance and simplicity is central to the field of explainable AI ( XAI ) , and machine learning as a whole [ 11 ] ; our method aims to help by highlighting the most important parts of a policy . ∗The work in this paper was done while at the University of Oxford . †The work in this paper was done prior to joining Amazon . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The starting point for our definition of “ simplicity ” is the assumption that there exists a way to make a “ simple choice ” , that is , there is a simple default action for the environment . We argue that this is the case for many environments in which RL is applied : for example , “ repeat previous action ” is a straightforward default action for navigation tasks . The key contribution of this paper is a novel method for ranking policy decisions according to their importance relative to some goal . We argue that the ranked list of decisions is already helpful in explaining how the policy operates . We evaluate our ranking method by using the ranking to simplify policies without compromising performance , hence addressing one of the main hurdles for wide adoption of deep RL : the high complexity of trained policies . We produce a ranking by scoring the states a policy visits . The rank reflects the impact that replacing the policy ’ s chosen action by the default action has on a user-selected binary outcome , such as “ obtain more than X reward ” . It is intractable to compute this ranking precisely , owing to the high complexity and the stochasticity of the environment and the policy , complex causal interactions between actions and their outcomes , and the sheer size of the problem . Our work uses spectrum-based fault localisation ( SBFL ) techniques [ 20 , 31 ] , borrowed from the software testing domain , to compute an approximation of the ranking of policy decisions . SBFL is an established technique in program testing for ranking the parts of a program source code text that are most likely to contain the root cause of a bug . This ranking is computed by recording the executions of a user-provided test suite . SBFL distinguishes passing and failing executions ; failing executions are those that exhibit the bug . Intuitively , a program location is more likely to be the root cause of the bug if it is visited in failing executions but less ( or not at all ) in passing ones . SBFL is a lightweight technique and its rankings are highly correlated with the location of the root cause of the bug [ 31 ] . We argue that SBFL is also a good fit for analysing complex RL policies . Our method applies to RL policies in a black-box manner , and requires no assumptions about the policy ’ s training or representation . We evaluate the quality of the ranking of the decisions by the proxy of creating new , simpler policies ( we call them “ pruned policies ” ) without retraining , and then calculate the reward achieved by these policies . Experiments with agents for MiniGrid ( a more complex version of gridworlds ) [ 7 ] , CartPole [ 4 ] and a number of Atari games [ 4 ] demonstrate that pruned policies maintain high performance ( similar or only slightly worse than that of the original policy ) when taking the default action in the majority of the states ( often 90 % of the states ) . As pruned policies are much easier to understand than the original policies , we consider this an important step towards explainable RL . Pruning a given policy does not require re-training , and hence , our procedure is relatively lightweight . Furthermore , the ranking of states by itself provides important insight into the importance of particular decisions for the performance of the policy overall . The code for reproducing our experiments is available on GitHub3 , and further examples are provided on the project website4 . 2 Background . 2.1 Reinforcement learning ( RL ) . We use a standard reinforcement learning ( RL ) setup and assume that the reader is familiar with the basic concepts . An environment in RL is defined as a Markov decision process ( MDP ) and is denoted by 〈S , A , P , R , γ , T 〉 , where S is the set of states , A is the set of actions , P is the transition function , R is the reward function , γ is the discount factor , and T is the set of terminal states . An agent seeks to learn a policy π : S → A that maximizes the total discounted reward . Starting from the initial state s0 and given the policy π , the state-value function is the expected future discounted reward as follows : Vπ ( s0 ) = E ( ∞∑ t=0 γtR ( st , π ( st ) , st+1 ) ) ( 1 ) A policy π : S → A maps states to the actions taken in these states and may be stochastic . We treat the policy as a black box , and hence make no further assumptions about π . 3https : //github.com/hadrien-pouget/Ranking-Policy-Decisions . Experiments done at commit c972414 4https : //www.cprover.org/deepcover/neurips2021/ 2.2 Spectrum-based fault localization ( SBFL ) . The reader is likely less familiar with spectrum-based fault localization ( SBFL ) , as to the best of our knowledge , it has not yet been used in RL . We therefore give a detailed description . SBFL techniques [ 31 ] have been widely used as an efficient approach to aid in locating the causes of failures in sequential programs . SBFL techniques rank program elements ( say program statements ) based on their suspiciousness scores , which are computed using correlation-based measures . Intuitively , a program element is more suspicious if it appears in failed executions more frequently than in correct executions , and the exact formulas differ between the measures . Diagnosis of the faulty program can then be conducted by manually examining the ranked list of elements in descending order of their suspiciousness until the cause of the fault is found . It has been shown that SBFL techniques perform well in complex programs [ 2 ] . SBFL techniques first execute the program under test using a test suite . A test suite comprises of a set of inputs and an expected output for each input . A test passes when the output produced by the program under test matches the expected output given by the test suite , and has failed otherwise . In addition to the outcome of the test , SBFL techniques record the values of a set of Boolean flags that indicate whether a particular program element was executed by that test . The task of a fault localization tool is to compute a ranking of the program elements based on the values of the Boolean flags recorded while executing the test suite . Following the notation from [ 20 ] , the suspiciousness score of each program statement s is calculated from a set of parameters 〈asep , asef , asnp , asnf 〉 that give the number of times the statement s is executed ( e ) or not executed ( n ) on passing ( p ) and on failing ( f ) tests . For instance , asep is the number of tests that passed and executed s. Many measures have been proposed to calculate the suspicious scores of program elements . In Equations ( 2a ) ∼ ( 2d ) we list a selection of popular and high-performing measures [ 1 , 9 , 15 , 30 ] ; these are also the measures that we use in our ranking procedure . Ochiai : asef√ ( asef + a s nf ) ( a s ef + a s ep ) ( 2a ) Tarantula : asef asef +a s nf asef asef +a s nf + asep asep+a s np ( 2b ) Zoltar : asef asef + a s nf + a s ep + 10000asnf a s ep asef ( 2c ) Wong-II : asef − asep ( 2d ) SBFL-based tools present the list of program elements in descending order of their suspiciousness scores to the user . There is no single best measure for fault localization ; different measures perform better on different types of programs , and it is best practice to use multiple measures [ 18 ] . In our experiments , we combine the four measures listed above for this very reason . While more sophisticated versions of SBFL exist [ 3 , 5 ] , in this work we prefer to stick to the simpler approach , which was sufficient for producing notable results . 3 Method : ranking policy decisions using SBFL . Inspired by the use of SBFL for localising the cause of a program ’ s outcome , we propose a new SBFL-based method to identify the states in which decisions made by an RL policy are most important for achieving its objective . Our method is modular and is composed of two phases : ( 1 ) generating mutant policies and ( 2 ) ranking states based on the importance . 3.1 Definitions . Executions of RL policies We apply the SBFL technique to a set of executions ( sometimes called trajectories in the literature ) of a given RL policy π with mutations . An execution τ of π describes a traversal of the agent through the environment MDP using the RL policy π and is defined as a sequence of states s0 , s1 , . . . and actions a0 , a1 , . . . , where s0 is an initial state and each subsequent state si+1 obtained from the previous state si by performing action ai , as chosen using π ( si ) . The last state must be a terminal state . As π or the environment can be stochastic , each execution of π may result in a different sequence of actions and states , and hence in a different τ . The set of all possible executions is denoted by T . A decision of a policy π in a state s is a pair 〈s , π ( s ) 〉 . Note that π ( s ) is the learned probability distribution from which an action in this state is obtained ; in a deterministic policy , π ( s ) is a single action for each s. Passing and failing executions An execution is either successful or failed . We define the success of an execution as a ( binary ) value of a given assertion on this execution . For example , the assertion can be that the agent reaches its destination eventually , or that the reward of this execution is not lower than 0.75 of the maximal reward for π . The assertion induces a Boolean function C : T → { 0 , 1 } . We say that an execution τ is a pass if C ( τ ) = 1 , and is a fail otherwise . We use a binary condition for simplicity , as SBFL is designed to work with passing and failing executions . This condition can be relaxed [ 5 ] , and we plan to investigate generalising this in future work . Mutant executions We use SBFL to understand the impact of replacing actions by a default action . The choice of the default action d is context dependent and can be configured by the user . For example , an obvious default action for navigation is “ proceed in the same direction ” . The default action can in principle be as basic as a single action , or as complex as a fully-fledged policy . In our experiments , we evaluate two default actions . The first is “ repeat the previous action ” , defined as : d ( s0 , . . . , si , a0 , . . . , ai−1 ) = ai−1 , ( 3 ) and the second is “ take a random action ” . For A being the action space , d ( s0 , . . . , si , a0 , . . . , ai−1 ) = asi ∼ Unif ( A ) . ( 4 ) Once asi has been sampled , it is not re-sampled if the state si is revisited ; the same action is used . Using “ take a random action ” as the default action can be useful in cases where there is no other obvious default action . However , we generally expect this to be a worse option than a default action tailored to the environment . As the choice of d depends on the context and the user ’ s goals , it is ultimately a user choice . Using the default action , we create mutant executions , in which the agent takes the default action d whenever it is in one of the mutant states . More formally , given a set of mutant states SM , we act according to the policy πSM defined as : πSM ( s0 , . . . , si , a0 , . . . , ai−1 ) = { π ( si ) si /∈ SM , d ( s0 , . . . , si , a0 , . . . , ai−1 ) si ∈ SM . ( 5 ) Decisions made in states in which the default action is a very good option are deemed less important . In these states , not following the policy would be less consequential .

Summary: This paper studies the problem of "ranking policy decisions," which uses spectrum-based fault localization to compute an offline score for each action, conditioning on a state. The main idea is that the action with a higher success frequency should be ranked higher than other actions. Studying the action rank of policies presents an interesting angle to investigate RL policy from simplifying the DRL policy, interpretability, offline evaluation, while the experiments conducted might not be sufficient to support either one aspect solidly.