Ranking Policy Decisions
Authors: Hadrien Pouget
Hana Chockler
Youcheng Sun
Daniel Kroening
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning
Explainable AI
Statistical Fault Localization

Abstract: Policies trained via Reinforcement Learning (RL) without human intervention are often needlessly complex, making them difficult to analyse and interpret. In a run with $n$ time steps, a policy will make $n$ decisions on actions to take; we conjecture that only a small subset of these decisions delivers value over selecting a simple default action. Given a trained policy, we propose a novel black-box method based on statistical fault localisation that ranks the states of the environment according to the importance of decisions made in those states. We argue that among other things, the ranked list of states can help explain and understand the policy. As the ranking method is statistical, a direct evaluation of its quality is hard. As a proxy for quality, we use the ranking to create new, simpler policies from the original ones by pruning decisions identified as unimportant (that is, replacing them by default actions) and measuring the impact on performance. Our experimental results on a diverse set of standard benchmarks demonstrate that pruned policies can perform on a level comparable to the original policies. We show that naive approaches for ranking policies, e.g. ranking based on the frequency of visiting a state, do not result in high-performing pruned policies. To the best of our knowledge, there are no similar techniques for ranking RL policies' decisions. 


Ranking Policy Decisions
Hadrien Pouget"
Hana Chockler causaLens and King's College London UK hana@causalens.. com hana. www.khleewkl......c..oo
University of Cambridge
UK pougeth@gmail.com com
Youcheng Sun Queen's University Belfast UK boucheng.sun@qub... sun@qub ac. uk
Daniel Kroening'
Amazon
UK Ganiel..roeninggnaagd ox ac uk
Abstract
Policies trained via Reinforcement Learning (RL) are often needlessly complex, making them difficult to analyse and interpret. In run with time steps, policy will make n decisions on actions take; we conjecture that only a small subset of these decisions delivers value over selecting simple default action. Given trained policy, we propose novel black-box method based on statistical fault localisation that ranks the states of the environment according to the importance of decisions made in those states. We argue that among other things, the ranked list of states can help explain and understand the policy. As the ranking method is statistical, a direct evaluation its quality hard, As proxy for quality, we use the ranking to create new, simpler policies from the original ones by pruning decisions identified as unimportant (that is, replacing them by default actions) and measuring the impact performance. Our experiments benchmarks demonstrate that pruned policies can perform on level comparable to the original policies. Conversely, show that naive approaches for ranking policy decisions, e.g., ranking based on the frequency of visiting a state, do not result berfhorerorming pruned policies.
diverse set of standard
Introduction
Reinforcement learning powerful method for training policies that complete tasks in complex enviromments. The policies produced are optimised to maximise the expected reward provided by the environment. While performance is clearly an important goal, the reward typically does not capture the entire range of our preferences. By focusing solely on performance, we risk overlooking the demand for models that are easier to analyse, predict and interpret [66 Our hypothesis is that many trained policies are needlessly complex, i.e., that there exist alternative policies that perform just as well or nearly as well but that are significantly simpler. This tension between performance and simplicity is central to the field of explainable Al (XAI), and machine learning as whole (II] our method aims to help by highlighting the most important parts policy.
'The work in this paper was done while at the University of Oxford. t 'The work this paper was done prior joining Amazon.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

The starting point for our definition of "simplicity" is the assumption that there exists way to make a "simple choice", that is, there is simple default action for the environment. We argue that this is the case for many environments in which RI is applied: for example, "repeat previous action" is straightforward default action for navigation tasks.
The key contribution of this paper is a novel method for ranking policy decisions according to their importance relative to some goal. We argue that the ranked list of decisions is already helpful in explaining how the policy operates. We evaluate our ranking method by using the ranking to simplify policies without compromising performance, hence addressing one of the main hurdles for wide adoption of deep RL: the high complexity of trained policies.
We produce a ranking by scoring the states policy visits. The rank reflects the impact that replacing the policy's 's chosen action by the default action has on a user-selected binary outcome, such as "obtain more than X reward". It is intractable compute this ranking precisely, owing to the high complexity and the stochasticity of the environment and the policy, complex causal interactions between actions and their outcomes, and the sheer size of the problem. Our work uses spectrum-based fault localisation (SBFL) techniques [1). borrowed from the software testing domain, to compute an approximation of the ranking of policy decisions. SBFL is an established technique in program lesting for ranking the parts of a program source code text that are most likely to contain the root cause of a bug. This ranking computed by recording the executions of user-provides test suite. SBFL distinguishes passing and failing executions; failing executions are those that exhibit the bug. Intuitively, a program location more likely to be the root cause of the bug if is visited in failing executions but less (or not at all) in passing ones. SBFL. is a lightweight technique and its rankings are highly correlated with the location of the root cause of the bug 31. We argue that SBFL is also a good fit for analysing complex RL policies.
Our method applies RL policies in a black-box manner, and requires no assumptions about the policy's s training or representation. We evaluate the quality of the ranking of the decisions by the proxy of creating new, simpler policies (we call them "pruned policies") without retraining, and then calculate the reward achieved by these policies. Experiments with agents for MiniGrid (a more complex version of gridworlds) 77. CartPole (4) and a number of Atari games 4 demonstrate that pruned policies maintain high performance (similar or only slightly worse than that of the original policy) when taking the default action in the majority of the states (often 90% of the states). As pruned policies are much easier to understand than the original policies, we consider this an important step towards explainable RL. Pruning a given policy does not require re-training, and hence, our procedure is relatively lightweight. Furthermore, the ranking of states by itself provides important insight into the importance of particular decisions for the performance of the policy overall.
The code for reproducing our experiments is available on GitHut and further examples are provided on the project website
2 Background
2.1 Reinforcement learning (RL)
We use a standard reinforcement learning (RL) setup and assume that the reader is familiar with the basic concepts. An environment in RL is defined as a Markov decision process (MDP) and is denoted by (S. A, P. R. y, T), where S is the set of states, A is the set of actions, P is the transition function, R the reward function, 7 is the discount factor, and T is the set of terminal states. An agent seeks to learn a policy
S + A that maximizes the total discounted reward. Starting from the initial state so and given the policy æ, the state-value function is the expected future discounted reward as follows:
Va (50) = E
rd R(s,, ((8)),
(I)
A policy π: : 5 A maps states to the actions taken in these states and may be stochastic. We treat the policy black box, and hence make no further assumptions about I.
http://giihhh..ommmmaaeeeenggggu Decteeeee/eeemniiioooloolieeiioo Deccisios Experiments done at commit c972414 http://wwww.ppoeeeeegeepperreepp

---

(a)
(b)
(a) CartPole, in a state where the cart and pole are moving rapidly. The heatmap represents the frequency of appearance of the possible pole angles (left) and the importance scores following SBFL (right). While more frequent for the pole more important when the pole is close to falling. (left) and prumed policy (right). States in which we take the default action are in blue. Both policies succeed, but pruning unimportant actions simplifies the policy.
Figure 1:
centred, SBFL successfully identifies that the policy's 's decisions are (b) MiniGrid. Traces of executions with the original policy
2.2 Spectrum-based fault localization (SBFL)
The reader is likely less familiar with spectrum-based fault localization (SBFL), as to the best of our knowledge, it has not yet been used in RL. We therefore give detailed description. SBFL techniques [1 have been widely used as an efficient approach to aid in locating the causes of failures in sequential programs. SBFL techniques rank program elements (say program statements) based on their suspiciousness scores, which are computed using correlation- program element is more suspicious it appears failed executions more frequently than correct executions, and the exact formulas differ between the measures. Diagnosis of the faulty program can then conducted by manually examining the ranked list of elements descending order their suspiciousness until the cause of the fault is found. It has been shown that SBFL techniques perform well in complex programs [2].
measures. Intuitively,
SBFL techniques first execute the program under test using a test suite. A test suite comprises of set of inputs and an expected output for each input. A test passes when the output produced by the program under test matches the expected output given by the test suite, and has failed otherwise. In addition to the outcome of the test, SBFL techniques record the values of set of Boolean flags that indicate whether particular program element was executed by that test.
The task of fault localization tool is compute ranking of the program elements based the values of the Boolean flags recorded while executing the test suite. Following the notation from (20), the suspiciousness score of each program statement is calculated from set of parameters (a aef. app o.g)) that give the number of times the statement is executed (e) or not executed (n) on passing (p) and on failing (f) tests. For instance, 0%p is the number of tests that passed and executed 8.
Many measures have been proposed to calculate the suspicious scores of program elements. In Equations (2)) we list selection of popular and tigh-performing measures 9 15 30 these are also the measures that we use in our ranking procedure.
a ag m
Ochiai:
(2a)
Tarantula:
(2b)
Vaa' + May asp))
4- +o
a2 tot
Zoltar:
(2c)
Wong-Il: def a
(2d)
100000 azg
" azf + app
SBFL--ased tools present the list of program elements descending order their suspiciousness scores to the user. There is no single best measure for fault localization; different measures perform better on different types of programs, and it is best practice to use multiple measures [8]] In our experiments, combine the four measures listed above for this very reason.

---

While more sophisticated versions of SBFL exist B in this work we prefer to stick to the simpler approach, which was sufficient for producing notable results.
3 Method: ranking policy decisions using SBFL
Inspired by the use of SBFL for localising the cause of program's outcome, propose new SBFL-based method to identify the states in which decisions made by an RL policy are most important for achieving its objective. Our method is modular and is composed of two phases: (1) generating mutant policies and (2) ranking states based on the importance.
3.1 Definitions
Executions of RL policies We apply the SBFL technique to set of executions (sometimes called trajectories in the literature) of a given RL policy л with mutations. An execution T of π describes traversal of the agent through the environment MDP using the RL policy æ and defined sequence of states 50, 81.... and actions do, ai..... where so is an initial state and each subsequent state 8 obtained from the previous state s; by performing action a, chosen using (s:). The last state must be terminal state. As т or the environment can be stochastic, each execution of π may result in different sequence of actions and states, and hence in a different т. The set of all possible executions is denoted by T. A decision of a policy in state is a pair (s, TT (s)). Note that π (s) is the learned probability distribution from which an action in this state obtained; in a deterministic policy, (s) is a single action for each 8.
Passing and failing executions An exccution is either successful or failed. We define the success of an execution as a (binary) value of a given assertion on this execution. For example, the assertion can be that the agent reaches its destination eventually, or that the reward of this execution is not lower than 0.75 of the maximal reward for æ. The assertion induces a Boolean function C C - {0. 1). We say that execution T is pass if C(r) for simplicity, as SBFL is designed to work with passing and failing executions. This condition can be relaxed and we plan investigate generalising this in future work.
and is a fail otherwise. We use binary condition
Mutant executions We use SBFL to understand the impact replacing actions by a default action. The choice of the default action d is context dependent and can be configured by the user. For example, an obvious default action for navigation "proceed the same direction". The default action can in principle be basic single action, complex as fully-fledged policy. In our experiments, we evaluate two default actions. The first is "repeat the previous action", defined as:
d(so 8/s ag. a4-1) dg-1:
and the second "take a random action". : For being the action space,
d(su. S;, 0;-1) = Oss (A).
(4)
Once ay; has been sampled, it not samsede if the state s; is revisited; the same action is used. Using "take a random action" as the default action can be useful in cases where there is no other obvious default action. However, we generally expect this to be worse option than default action tailored to the environment. As the choice of d depends on the context and the user's goals, it is ultimately a user choice.
Using the default action, we create mutant executions, in which the agent takes the default action whenever it in one of the mutant states. More formally, given set of mutant states SM, we act according to the policy TSs defined as:
8i $ SM TeA (Su.... Si, Qu. ..-11))111))) [d(o...... 0g: (1-1)
((())
a;-1)
(5)
8, SM-
Decisions made in states in which the default action a very good option are deemed less important. In these states, not following the policy would be less consequential.

---

3.2 Generating the test suite and mutant executions on-the-fly
The naive approach to generating a comprehensive suite of mutant executions for applying SBFL would be to consider all possible sets of mutant states Sw is, we would need to consider all possible subsets of the state space S. However, the state space of most RL. environmmss is too large to enumerate, and enumerating all possible subsets of is intractable even for simplistic environments.
We use two algorithmic techniques to address this problem: (a) we generate mutant executions on-the-ffy, and (b) we use an abstraction function a S + S to map the full set of states S to smaller, less complex set of abstract states S. Examples of these are in the supplementary material, and may for example include down-scaling or grey-scaling images that are input to the agent, or quantising a continuous state space. The set of possible mutations is then the set of subsets of Š, instead of the set of subsets of S. We then score the abstract states, rather than the full state space. The test suite of mutant exccutions produced this way for π is denoted T (Tx) CT.
On-the-fly mutation We maintain set SM C $ per execution. We begin each execution by initialising the set SM with the empty set of states. At each step of the execution, upon visiting a state 8, check the current SM- If (s) E SM, we add a(s) Sas according the predefined mutation rate ya (and take the default action); otherwise, we use π to determine the action in this state. In case we re-visit an (abstract) state, we maintain the previous decision of whether to mutate state s or not. This way, the states that are never visited in any of the executions are never mutated; hence, we never consider "useless mutations" that mutate a state that is never visited. We finish by marking as pass or fail according to C(T). Note that a mutant execution may visit states not typically encountered by π,, meaning that we are even able to rank states that are out of distribution. This is especially important when trying to understand how the policy behaves in parts of the environment it unfamiliar with.
Overall, our algorithm has five (tunable) parameters: the size of the test suite IT(x)), the passing condition C, the default action d, the mutation rate μ and the abstraction function a.
SBFL benefits from a balanced test suite of passing and failing executions [31, a ratio largely determined by the choices of and C. The choice of C' depends on the context. In our experiments in Section 4.3 our goal was to find the states with highest impact on the expected reward. We set the condition to be "receive more than X reward" for some X R. and chose X to yield a balanced suite (i.e. if there were too many failed runs, we lowered X). In most cases actions important for achieving at least X reward are important for maximising reward in general, we found that this worked well. In the cases where some actions are not important for achieving X, but later important for getting even higher rewards (e.g., states that only appear after achieving X reward), not have considered them important. The choice of ja is also significant. If it is too large, executions fail too often, and the behaviour in mutant executions is uninteresting. If μ is too small, we do not mutate states enough to learn anything, and in larger environments fail to mutate many of the states we encounter. In our experiments, we selected μ manually, but this could easily be automated by parameter tuning algorithm.
would
3.3 Computing the ranking of the policy decisions
We now explain how to rank states according to the importance of policy decisions made in these states, with respect to satisfying the condition. The ranking method is based on SBFL as described in Section 2.2 We first create the test suite of mutant executions ((r) as described above. We denote the set of all abstract states encountered when generating the test suite ST S Š; these are the states to which we assign scores. Any unvisited state is given the lowest possible score by default.
Similarly to SBFL for bug localisation, for each state E ST we calculate vector (aep. aef. Shp d.y)) We use this vector to track the number of times that s was unmutated (e) or mutated (n) on passing (p) and on failing (f) executions, and we update these scores based on those executions in which the state was visited. In other words, the vector keeps track of success and failure of mutant executions based on whether an execution took the default action in s or not. For example, a is the number of passing executions that took the action (s)) in the state s, and a is the number of failing executions that took the default action in the state
5

---

Definition 1 (Ranking). Given an SBFL measure m. and vector (dep, anz Ceg) for each (abstract) state in the set of (abstract) states of the policy, we define the ranking function rank: $ - (I....,|$1} as the ordering of the states in S in the descending order according to the values m(s); that is, the state with the maximal value will be the first in the ranking.
4 Experimental evaluation
4.1 Research Questions
Our goal is to demonstrate the applicability of our ranking method variety of standard environ- ments and to provide evidence of the utility of the generated ranking. We aim to answer the following research questions:
RQ1: How measure the relative importance of decisions for achieving the reward? What is good proxy for measuring this?
RQ2: Does the approach we present this paper scale to large policies and complex environments?
We answer these questions in Section 4.3) by performing extensive experiments with various envi- ronments and policies. In Section 4.4, we discuss possible applications our ranking (including interpretability), and the effect of choosing a different default action.
4.2 Experimental setup
We experimented in several environments. The first is Minigrid [7]. gridworld in which the agent operates with cone of vision and can navigate many different grids, making more complex than a standard gridworld. In each step the agent can turn or move forward. We also used CartPole [4], the classic control problem with a continuous state-space. Finally, to our ability scale, we ran experiments with Atari games 4
We policies that are trained using third-party code. No state abstraction is applied the gridworld environments (i.e., a is the identity). The state abstraction function for the CartPole environment consists of rounding the components of the state vector between and decimal places, and then taking the absolute value. For the Atari games, as is typically done, we crop the game's border, grey-scale, down-sample to 18 14, and lower the precision of pixel intensities to make the enormous state space manageable. Note that these abstractions are not a contribution of ours, and were primarily chosen for their simplicity. For our main experiments, we use "repeat previous action" as the default action.
Examples of some environments, and important states found in them, are given in Figs. la and Details about the state abstraction functions, policy training, hyperparameters, etc., are provided in the full version of this paper 22
We define two further measures in addition to the SBFL ones in Eq. (2). for comparison. Eq. (6a) measures how frequently the state was encountered in the test suite. Eq. (6b) is a random ranking of the states visited by the test suite. We use the Freq Vis measure as a baseline because we are not aware of any previous work with the same goals as ours for ranking policy decisions, and naïve approach to determining importance may be simply looking at how frequently a state is visited.
Freq Vis: app + ale + amp + a'e (6a)
Rand: Unif(O
(6b)
4.3 Experimental results
Performance of pruned policies The precise ranking of decisions according to their importance for the reward is intractable for all but very simple policies. To answer RQ1, in our experiments, we use the performance of pruned policies as proxy for the quality of the ranking computed by our algorithm. In pruned policies, the default action is used in all but the top ranked states. For given r (a fraction or percentage), we denote by rank (r) the subset of r top-ranked states. We denote by x" the pruned policy obtained by pruning all but the top-r ranked states. That is, an execution of n" retains actions in the fraction of the most important states from the original policy and replaces

---

Originnl Reward (Ne)
Original Reward (%6
Original States
Policy's Action Steps)
(a) CartPole States Pruned Policy
(b) CartPole Actions Taken
Original Reward (80
Original newward (36)
Original
Policy's Action Taken 1% Steps)
(c) Krull States in Pruned Policy
(d) Krull Actions Taken
Figure 2: Performance of the pruned policies, measured as percentage of the original reward. (e), The æ-axis is the % of states where the original action taken out of the of all states encountered in the test suite. (d): The r-axis the expected % of steps in which original policy is followed over the default action during an execution of the pruned policy. See the supplementary material for more detail.
the rest by default actions. The states that are in ronk(r] are called the original states. We measure the performance of the pruned policies for increasing values of relative to the performance of the original policy π.
These results are given in first four columns of Tab. 1 and some are represented graphically in Figs. 2h. In these, SBFL stands for the SBFL portfolio, l.e., the combination of the four measures in Eq. 27. where the best result is taken at each point. The results show that the pruned policies obtained using the SBFL ranking can achieve performance that is comparable to π with less than 40% of the original decisions (and in some cases the number is as low as 20%). The performance of SBFL-based pruning is compared with random pruning Eq. (6b) and Freq Vis pruning in Eq. (6a). At first glance, seems that the Freq Vis pruned policies perform well in Figs. _4. To better understand this, show in the last two columns of Tab. [ and in Figs. 2b.d how performance evolves with the proportion of steps in which the original policy is used over the default policy (i.e. not replaced by the default action). As shown in Figs. 2b and 2d FreqVis does much worse by this metric, because yields a pruned policy that prefers use the original policy often possible.
We observe that using our ranking method enables significant pruning of the policies, while maintaining performance. To answer RQ2, we experimented with larger and more complex Atari environments. The results demonstrate that our framework is reasonably scalable, but also that the quality of the ranking is significantly improved by using a state abstraction. The results in the larger and more complex Atari environments show the effect of using a test suite that is too small: many states not encountered during the computation of the ranking. This means that rank [1], in which the original policy is used in all of the states discovered while making the ranking, does not recover the performance of the original policy. Increasing the size of the test suite would help, but this is not scalable solution. Instead, we use better state abstractions, which reduce the state space. In CartPole, this allows to tackle continuous domain. In the Atari games, the generic abstraction in all the games sometimes not enough ("x" in Tab.
To show the potential utility of specialised abstractions, we create one for Breakout in which we extract the coordinates of the ball and paddle. The results obtained with this abstraction are given in Tab, i in the row labelled "Breakout (abs)". While the new abstraction does worse in terms of states
7

---

Table 1: Minimum percentage of original states in pruned policies, and percentage of steps in which the original policy used, hefore recovering 90% of original performance. Using default action "repeat previous action". Results are reported for the SBFL portfolio ranking and the random ranking. "x" denotes that the required reward was never reached, cf. Sec. 4.3
% of original states restored
% of steps that use
Environment
SBFL
random
SBFL
random
MiniGrid
49 00
99 = 00
76 = 00
98 + 01
Cartpole
31 - 04
= 02
22 = 04
+ 02
Alien
x 100 + 00
x 100 + 00
Assault
45 = 07
93 + 01
Atlantis
50 + 00
100 + 00 X
99 + 00
100 + 00
BankHeist
BattleZone
30 = 00
86 = 07
84 = 07
84 + 08
Berzerk
47 12
100 = 00 X 100 +
88 = 03
100 = 00
Boxing
X
Breakout
10 00
54
+ 81 + 06 x 99 + 02
Breakout (abs)
40 + 00
85 + 05 X
41 + 00
ChpperCmmnd
DemonAttack
20 = 00
99 - 01
98 + 01
Hero
48 = 04
- 03 X
86 + 08
96 + 04 X + 06
IceHockey
65 = 20
91 = 08
Jamesbond
30 + 13
+ 06 99 = 01
59 +
Krull
75 = 12
35 = 21
98 + 02
Phoenix
30 = 00
92 = 07
97 = 00
92 07
Pong
21 = 03
79 = 03
42 = 01
78 03
Qbert
40 + 00
100 + 00
84 + 04
100 + 00
Riverraid
95 05
100 + 00 X
99 + 01 X
100 + 00 X
RoadRunner
Seaquest
48 = 04
94 = 04
92 - 04
91 + 05
SpaceInvaders
30 + 00
100 + 00
93 - 00
100 + 00
StarGunner
40 00
100 + 00 X
99 + 00
100 + 00
YarsRevenge
X
pruned within the policy, it allows to reach 90% of the policy's original performance with 13% fewer steps in which we use over the default action.
4.4 Discussion
SBFL ranking for better understanding the RL policy Any strong claim about the ranking's application to interpretability requires a user study, which is out of scope this paper. However, we can look to existing research looking the usefulness of SBFL. While some studies suggest that the users typically do not go over the list of possible causes generated by SBFL linearly (and hence question the usefulness of the ranking) [21], recent large-scale study demonstrates statistically significant and substantial improvements for the users who use an SBFL tool, and the results hold for "mediocre" SBFL tools be used to explain policy decisions, as the ranked list itself would be helpful to identify the most important decisions. In addition, the pruned policies that we construct are simpler than the original policies while achieving a comparable performance, which can make identifying problems more straightforward Examples are presented the CartPole and Minigrid sections of the website. Finally, in Fig. 3 we show a heatmap of the scores of each state of a minigrid environment. In each grid square, the agent can be facing four directions, We show the score based on the tarantula measure for these directions from blue (lowest) to red (highest). We show this only for the states visited along path to the goal. Our heatmap gives information about the general behaviour of the policy. this case, points at which the agent needs to turn are more red (more important) than points where the agent walking straight line. To understand why the downward-facin states in the right-most column are considered important, refer to our website for full heatmap and explanation.
32 Based this evidence, we suggest that the ranking can

---

Figure 3: Example of heatmap made based scores. Colour from least most important blue, white and red. (a) Our healmap based on the Tarantula measure, showing all the states an agent encounters while walking to the goal, including the direction in which the agent was facing. For example, in the top left grid location, we show the importance of the state in which the agent is facing right, and the state in which it is facing down.
A Different Default Action Not all environments have an obvious default action. In this case, straightforward choice is set the default action to "take random action". We measured the effect of this choice by running the same experiments with the changed default action, and detailed results are available in the full version of this paper [22] The results are similar or slightly worse to the ones obtained with the default action "repeat the previous action". In most games, the difference was small (S 10%); a few games show a marginal improvement. Theee results indicate that the choice of a default action has no effect on our conclusions.
Good vs. bad policies Finally, we performed some initial work towards using the ranking to understand the difference between high and low-performing policies. To this end, we produced a ranking according to the high-performing policy, and then compared how the two policies behaved in the highest ranked states. Interestingly, our experiments demonstrate that the high performing and low performing policies agree on 80% of the actions in the top 10% of states. This suggests that the policy training (in CartPole) first picks the 'low-hanging fruit' by performing well in the most important states. The difference between a high-performing and a low-performing policy is mostly in the lower-ranked states. Full results for this experiment are available in the full version this paper (22].
Related work
There a significant body of work on identifying the important parts of trained algorithms, but to the best of our knowledge none suggested to rank the states as we do. Prioritised experience replay [2
looks for the most important transitions for training. Saliency maps [TO 29 33 identify the parts of the state that most influence influence the agent's 's decision. Sun et al. (26) have previously applied SBFL to visual feature importance for input images given to image classifiers. Other attempts include identifying the important parts of the representation of the policy by looking at the parameters of trained model and pruning it to reduce its size [17]. None of these methods attempt to understand what the important decisions of the policy
whole are.
Much of the recent work focuses on making deep learning models more interpretable 23 26
Many approaches [10 29 33 to explaining deep reinforcement learning methods explain the decision made a single state, without the context of the past or the future behaviour. Iyer et al. [4] explain single decision via an object-level saliency map by leveraging the pixel-level explanation and object detection. As these methods focus on single decisions, the explanation is typically not sufficient to understand the overall decision-making of the trained agent.
Other work has also attempted to explain entire policies, rather than individual decisions. Ehsan et al. g produce natural language explanations for state-action pairs, based on a human- -pooiddd corpus of explanations. Topin and Veloso [27] create a Markov chain which acts as an abstraction of the policy, making it easier to reason about the policy. They create the Markov chain by grouping states into abstract states based on how similarly the policy acts those states; our method substantially different in that it ranks states based on importance, rather than grouping similar states. While our method allows for the use of abstract states, it is not always necessary, and we are not contributing any specific abstraction function. Similarly, Sreedharan et al. [25) propose method for creating

---

temporal abstraction of policy by using bottleneck 'landmarks' in the policy's 's executions. A robot's behaviour can be explained using operator "important program state variables" and "important functions" [12]. We find that the policy-wide decision ranking in this paper is an easier and more general method for understanding the policy. For more work in this vein, we encourage the reader to consult the overview from Chakraborti et al. [6] of the rapidly growing field of Explainable AI Planning (XAIP).
There have been attempts to make more interpretable models, either from scratch 13 or by approxi- mating a trained neural network [28]. In the latter case, our method may be useful for determining in which states the approximation must be most accurate.
6
Conclusions
We have applied SBFL-based ranking of states to reinforcement learning and demonstrated that this ranking correlates with the relative importance of states for the policy's performance. The ranking can be used to explain RI. policies, similarly to the way ranked program locations are used to understand the causes of bug. We evaluate the quality of our ranking by constructing simpler pruned policies, where only the most important decisions are made according to the policy, and the rest are default. Our experiments show that the performance the pruned policies is comparable to the performance of the original policies, thus supporting our claim that the SBFL-based ranking is accurate. Moreover, the pruned policies may be preferable many use-cases, as they are simpler. Our approach can be scaled with the use of abstractions, demonstrated by the larger Atari environments.
In the future, we hope to explore different upplications of the ranking, new measures, more nuanced hyperparameter selection, and relaxing the binary constraint over the assertion C. We do not expect our work to have any negative societal impacts, as it only serves to improve our understanding of the policies we train; the main concern is incorrectly increasing confidence in a policy.
Funding transparency statement The authors acknowledge funding from the UKRI Trust woothy Autonomous Systems Hub (EP/V00784X/I) and the UKRI Strategic Priorities Fund to the UKRT Research Node on Trustworthy Autonomous Systems Governance and Regulation (EP/V026607/1).
References
[1] Rui Abreu, Peter Zoeteweij. and Arjan J.C. van Gemund. On the accuracy of spectrum- based fault localization. In TAICPART-MUTATION, pages 89-98. IEEE, 2007.
[2] Rui Abreu, Peter Zoeteweij, Rob Golsteijn, and Arjan J.C. van Gemund. A practical evaluation of spectrum-based fault localization. Journal of Systems and Software, 82811):1780-1792. 2009.
[3] Rui Abreu, Peter Zoeteweij, and Arjan J.C. van Gemund. A new bayesian approach to multiple intermittent fault diagnosis. In IJCAI, pages 653 658, 2009. URL http://ijcai.org/Pooeedddi 14..
[4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl gym. CoRR, abs/1606.01540, 2016. URL https
//gym.openai.com/
[5] Nuno Cardoso, Rui Abreu, Alexander Feldman, and Johan de Kleer. A framework for automatic debugging of functional and degradation failures. ECAL pages 569 576. IOS Press, 2016. doi: 10.3233/978-
URL http:://ooooog/00333333333314610
[6] Tathagata Chakraborti, Sarath Sreedharan, and Subbarao Kambhampati. The emerging land- scape of explainable automated planning & decision making. In International Joinr Conference on Artificial Intelligence (ICAAI, pages 4803-4811, ijeai.org, 2020. doi: 10.24963/ijcai 2020/
669.
[7] Maxime Chevalier-Boisvert,, Lucas Willems, and Suman Pal. Minimalistic gridworld environ- ment for OpenAl Gym.
2018.
10

---

[8] Upol Ehsan, Brent Harrison, Larry Chan, and Mark O. Riedl. Rationalization: A neural machine translation approach to generating natural language explanations. In Al, Ethics, and Society, pages 81-87, 2018.
[9] Alberto Sonzalez-Sancher Automatic emror detection techniques based on dynamic invariants. M.S. Thesis, Delft University of Technology, 2007.
[10] Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understand- ing Atari agents. In ICML, pages 1787-1796, 2018.
[11] David Gumning and David W. Aha. DARPA's explainable artificial intelligence program. Al Magazine, 40(2):44-55 58, 2019.
[12] Bradley Hayes and Julie A. Shah. Improving robot controller transparency through autonomous policy explanation. In Human-Roo Robot Interaction (HRI), pages 303-312. ACM, 2017.
[13] Daniel Hein, Steffen Udluft, and Thomas A. Runkler. Interpretable policies for reinforcement learning by genetic programming. Engineering Applications of Artificial Intelligence, 76: 158-169, 2018.
[14] Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. Trans- parency and explanation in deep reinforcement learning neural networks. In Al, Ethics, and Society, pages 144-150. ACM, 2018
[15] James A. Jones and Mary Jean Harrold. Empirical evaluation of the Tarantula automatic fault-localization technique. Automated Software Engineering (ASE), pages 273-282, ACM, 2005.
[16] Michael Lewis, Huao Li, and Katia Sycara. Deep learning, transparency, and trust in human robot teamwork. In Trust in Human-Robot Interaction, pages 321 3-1-352. Elsevier, 2021.
[17] Dor Livne and Kobi Cohen. PoPS: Policy pruning and shrinking for deep reinforcement learning. IEEE Selected Topics in Signal Processing, 14(4))799-801, 2020.
[18] Lucia, David Lo, Lingxiao Jiang, Ferdian Thung, and Aditya Budi. Extended comprehensive study of association measures for fault localization. Journal of Software: Evolution and Process, 26(2):172-219, 2014.
[19] Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In NIPS, pages 3765-4774 Curran Associates, 2017.
[20] Lee Naish, Hua Jie Lee, and Kotagiri Ramamohanarao. A model for spectra-based based software diagnosis. ACM TOSEM, 20(3):11, 2011.
[21] Chris Parnin and Alessandro Orso. Are automated debugging techniques actually helping programmers? In International Symposium on Software Testing and Analysis (ISSTA), pages 199-209. ACM, 2011. doi: 10. 10.1 2,.144/200142222004442
[22] Hadrien Pouget, Hana Chockler, Youcheng Sun, and Daniel Kroening, Ranking policy decisions. CoRR, abs/2008. 13607, 2021. httph//s////xx..or///////0000000 13607
[23] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you? Explaining the predictions of any classifier. In KDD, pages 1135-1144, 2016.
[24] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay.
In International Conference on Learning Representations (ICLR), 2016. URL http://arxivvorg/
abs/1511.00995
[25] Sarath Sreedharan, Siddharth Srivastava, and Subbarao Kambhampati. TLdR: Policy sum- marization for factored SSP problems using temporal abstractions. In International Con- ference on Automated Planning and Scheduling, pages 272-280. AAAI Press, 2020. URL

---

[26] Youcheng Sun, Hana Chockler, Xiaowei Huang, and Daniel Kroening. Explaining image classifiers using statistical fault localization. In European Conference on Computer Vision (ECCV), volume 12373 of LNCS, pages 391-406. Springer, 2020.
[27] Nicholay Topin and Manuela Veloso. Generation of policy-level explanations for reinforcement learning. In AAAI, volume 33, pages 2514-2521, 2019.
[28] Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In ICML, pages 5045-5054 2018.
[29] Ziyu Wang. Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando de Freitas. Dueling network architectures for deep reinforcement learning. In ICML, pages 1995-2003, 2016.
[30] W. Eric Wong, Yu Qi, Lei Zhao, and Kai- Yuan Cai. Effective fault localization using code coverage. In Computer Software and Applications Conference (COMPSAC), volume 1, page 449. IHEE, 2007.
[31] W. Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. survey on software fault localization. JEEE TSE, 7208):707-740, 2016.
[32] Xin Xia, Lingfeng Bao, David Lo, and Shanping Li. Automated debugging considered harm- ful: A user study revisiting the usefulness of spectra-based fault localization techniques with professionals using real bugs from large systems. In ICSME, pages 267-278. IEEE, 2016.
[33] Tom Zahavy, Nir Ben -rihem, and Shie Mannor. Graying the black box: Understanding DQNs. In ICML, pages 1899 1999-1908, 2016.
12


Reviews:
[{"review_id": "uAeVPW_VrmZ", "reviewer": "Reviewer_8qfB", "summary": "The goal of this work is to leverage fault localization techniques from software development and debugging as a way to simply learned RL policies. The main premise is that learned policies are unnecessarily complex. In order to simplify their complexity, it is critical to rank the states where the learned policy decisions are critical. Motivated by spectrum-based fault localization, authors leverage it in RL by using similar cost functions to compare complex policy functions to so called mutant executions which are policies that recommend default or previous actions as a way to simplify the complexity of the policy class. Based on such mutant policies (identified on the fly by parametrizing specific aspects), the authors demonstrate they are able to rank important states (based on the policy's impact on outcome). The proposed method is evaluated on standard RL environments (atari, minigrid, cartpole) to quantify efficacy and visually demonstrate interpretability via saliency maps. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Vague definition of simple policy can be formalized. If the authors are restricting to stochastic policies, why not formalize the notion of simple vs complex policy functions with something like entropy? I think it will also improve the evaluations done in the experiments section. \n\n2. Worthwhile to consider other mutant policies (instead of just the one of replacing with default previous action) to test generalizability of the contributions. I suggest potentially breaking down the atari experiments to try different mutant policies. \n\n3. What is $\\hat{S}$ in reference to an on-the-fly mutation (last para, pg 4, line 148). Are you suggesting that abstract state representations are used in combination with on-the-fly mutations to collect trajectories?\n\n4. In experiments, it is unclear why the same set of environments do not achieve the desired reward with either the proposed or the random policy. Can the authors comment on whether it is an issue for the environment or something else?\n\n5. The saliency map comparison is odd because semantically they're comparing very different things depending on the goal. It adds to the confusion to analyze interpretability of the policies. Overall another measure like entropy might be a better motive to formalize what this work is trying to do.\n\n6. From the four different measures used for ranking in states in SBFL, I really would've liked to see more insight into what to use versus not. Instead the authors choose the best performing for every environment(?) which is not adding much insight. Since you've already experimented with all measures, can all measures be reported and discussed? \n\nOverall I am happy to reconsider the score if these concerns are addressed. \n\n----------------------------------------------------------------------------------------\nI have gone through author responses and I have increased my score to 6. I trust that authors will make all the changes that came up during the discussions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't believe there are negative social implications of the proposed work. The authors do address limitations of their methods sufficiently. ", "review_text": "", "extra_scores": {"ethical_concerns": "I don't see any immediate ethical concerns of this method.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hrs"}, "extra_text": {}}, {"review_id": "XUm8dOg2BdA", "reviewer": "Reviewer_3xFH", "summary": "This paper studies the problem of \"ranking policy decisions,\" which uses spectrum-based fault localization to compute an offline score for each action,\nconditioning on a state. The main idea is that the action with a higher success\nfrequency should be ranked higher than other actions. Studying the action rank\nof policies presents an interesting angle to investigate RL policy from simplifying the DRL policy, interpretability, offline evaluation, while the experiments conducted might not\nbe sufficient to support either one aspect solidly.  ", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Overall\n\nMy main concern of this paper is that it tries to cover a range of topics:\npruning DRL policy,  improving DRL interpretation, offline policy evaluation(comparison),\nsolving any single one of them would make a significant contribution already, while\nthis work couldn't provide sufficient evidence to demonstrate the proposed method \ncan work well on any one of those aspects. \n\n### Methodology\nWhat is the reason for accessing a power set of state?\nWhat is the formal definition of \"mutant state\"? \nFrom (2a) to (2d), we have the formula to rank actions given a state. What is the\nformula to rank states as mentioned in Line 107?\nAs the frequency count relies on the behavior policies used to collect data, it would be better to detail how the trajectories are collected. \n\n### Missing reference\nA related work [1] formally learns the rank of actions (decisions in this paper's terminology) in either an online or offline setting. \n\nWhat is the advantage of the proposed method comparing to learn the rank directly from\ndata, then applying the learned rank to the downstream task? \n\n\n\n### Experiments\n\nOverall, the baseline compared in this paper is relatively weak. Even though the classic\nRL algorithm [Q-learning based: DQN, DDQN, etc.][policy gradient based: PPO, A3C, etc.] does not learn the rank of action as their goal, we can use the action value or policy logits learned from those algorithms to serve as a baseline comparison. Based on [1]\nthe policy gradient approach is learning the listwise rank of actions. \n\nThe experiments on the interpretation of RL policy is lack quantitative and\nqualitative support. Figure 3 seems not convincing to verify the proposed method is interpretable as any method can plot such a \"heatmap\".\n\n\n[1]Lin, Kaixiang, and Jiayu Zhou. \"Ranking policy gradient.\" ICLR (2020).\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "LmScm9vmtUd", "reviewer": "Reviewer_ezL2", "summary": "The paper uses an existing ranking mechanism borrowed from software testing to rank the states of trajectories in reinforcement learning. After ranking the states, the authors propose to replace the top performing states with a default decision/action showing that that proposed ranking correlates with the `importance` of the trajectory states. The results show that the performance of the pruned policies (after replacing the states with a pre-defined action) is comparable with the original (non-pruned) policies.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper seems to tackle an important problem in the reinforcement learning domain, moving towards better understanding of the importance of the decisions made by the RL policies. The paper claims that the proposed ranking mechanism of the RL states followed by mutating the low-performing states with a pre-defined action/decision can still achieve comparable performance to the baseline non-pruned policies. In addition, to tacked the intractability of the states (those that are chosen to be mutant) the authors propose an abstraction mechanism that maps the large state space of RL environments to a simpler and tractable state space. The paper reads well and it seems interesting, however, I feel some of the claims are not sufficiently tested and substantiated.\n- If I understand correctly, the order of states also affect the overall performance of policies. That is, if a mutant state occurs at the very beginning of the trajectory that could affect the whole trajectory and may land the agent into a low-performing state. Would you please clarify how the proposed ranking mechanism would deal with this ordering of states? Do you ignore replacing the early states in the trajectory or the ranking mechanism automatically detect these important states?\n- Following on the previous question, are there cases where mutating the early states in the trajectory has minimal impact on the overall performance of the policy?\n- It seems the mutation policy could have an important impact on the overall performance. That makes me wonder if the proposed approach can be scaled to more complex environment where forming an abstraction mechanism is not straightforward. An interesting result to see is where the mutation is simply `picking a random action`.\n-  While I appreciate the authors bringing up RQ3-4, I think the claims are unsubstantiated and not well-supported (as mentioned in the paper, these claims require more thorough study).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As I think the mutation policy and the abstraction mechanism could have significant impact on the overall ranking and performance of the pruned policies, one of the limitations of this work could be its applicability to more complex environment. Also, some of the claims in the paper are not well studied.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "JZl0TSAwj2j", "reviewer": "Reviewer_iJVm", "summary": "This paper introduces a novel method based on spectrum-based fault localization (SBFL) from the software testing domain to identify the states in which decisions made by an RL policy are most influential in determining whether or not the agent achieves its objective. More specifically, the authors create \"mutant trajectories\" where, with probability $\\mu$, a state is mutated and the action that the agent must take in that state is the \"default action\" (e.g. \"repeat the previous action\") or unmutated, where the agent acts according to policy $\\pi$ when in that state.  From these mutant trajectories, the authors use SBFL to determine a ranking of states.  SBFL takes as input the vector $\\<a_{ep}^{s}, a_{ef}^{s}, a_{np}^{s}, a_{nf}^{s}\\>$, where $a_{ep}^{s}$ is the number of successes achieved when state $s$ is a non-mutant state, $a_{ef}^{s}$ is the number of failures achieved when state $s$ is a non-mutant state, and $a_{np}^{s}$, $a_{nf}^{s}$ log the number of successful and failed trajectories when state $s$ is a mutant state.  In order to assess the quality of the ranking returned by SBFL, the authors create \"pruned policies\" where an agent can only use the policy $\\pi$ for the top $r$ states, and must use the default action for all other states.  Overall, in a wide variety of settings ranging from MiniGrid and Cartpole to Atari games, the authors find that, by ranking states using SBFL, they are able to encourage agents to vastly reduce the length of their trajectories while still attaining high reward.  This is a significant contribution from the perspective of creating interpretable AI.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To the best of my knowledge, the method that this paper introduces is very novel, and is an exciting contribution in itself.  The paper is well written and clear.  The authors conceive of reasonable baselines for comparing their SBFL pruned policies against, despite the fact that the question they tackle here is novel, so there aren't readily available baselines from existing work.  That the authors show the performance of their framework on a wide variety of tasks is a great strength of this paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In general, I was very impressed by this paper.  The authors introduced a novel framework to tackle an interesting new problem. The results of applying this framework to a wide variety of tasks were largely impressive.  My only concerns are the following:\n- The biggest concern that I have about the widespread adoption of this framework is the fact that there are many parameters that are hard to tune in an automatic way (and, as such, the method fails on some Atari games).  Do you have any suggestions for setting these parameters in an automatic way?  For example, I found it a bit arbitrary how pruned policies were created: namely by keeping top-r.  You could perform a clustering to see if there are 2 clusters of states according to SBFL metrics\n- Saturation of Figure 2c and 2d at less than 100% - I may not be understanding something, but I would have expected these to saturate at 100% for freqVis and rand when the original states and policy's actions taken tended to 100%.  Can you explain why this is not the case?\n- Concern (Line 155/156): \"Note that a mutant execution may visit states not typically encountered by $\\pi$, meaning that we are able to even rank states that are typically out of distribution.\"  The converse is also true though: mutant execution may mean that certain states that are typically encountered by $\\pi$ are never reached.  Was this a problem that the authors came across?  If so, did they have any ideas for how to resolve this problem?", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The direction of the paper is based on the premise that learned RL policies are unnecessarily complex and only a small subset of the overall set of decisions yields considerable improvements over some simple baseline. Thus the paper considers ranking of RL policy decisions, and in particular, identifying states where the RL policy's decisions most influence the outcome of the agent achieving its objective. The paper presents a novel method based on the software testing domain, spectrum-based fault localization (SBFL).  The paper uses this to simplify the policy and shows that their approach can help agents considerably reduce the length of their trajectories while still attaining high reward in various settings.


The paper tackles an important problem with a novel approach. Their results show a vast improvement over a variety of settings, however, some concerns are raised by reviewers about the generality especially to complex environments / continuous settings. There are also concerns raised about the interpretability part and RQ3 and 4.


While the paper is far from perfect, in my opinion, the novel approach to an important problem and the empirical results for the settings studied push it over the bar. The authors, however, should carefully revise the paper accommodating all of the points that arose in the reviews, including but not limited to mentioning the use of NoFrameskip-v4 versions of the environments in which actions aren’t repeated, a clarification (ore even better, a formal definition) of the ranking, and addressing the saliency map interpretation.
Decision: Accept (Poster)