Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper uses an existing ranking mechanism borrowed from software testing to rank the states of trajectories in reinforcement learning. After ranking the states, the authors propose to replace the top performing states with a default decision/action showing that that proposed ranking correlates with the `importance` of the trajectory states. The results show that the performance of the pruned policies (after replacing the states with a pre-defined action) is comparable with the original (non-pruned) policies.
main_review: The paper seems to tackle an important problem in the reinforcement learning domain, moving towards better understanding of the importance of the decisions made by the RL policies. The paper claims that the proposed ranking mechanism of the RL states followed by mutating the low-performing states with a pre-defined action/decision can still achieve comparable performance to the baseline non-pruned policies. In addition, to tacked the intractability of the states (those that are chosen to be mutant) the authors propose an abstraction mechanism that maps the large state space of RL environments to a simpler and tractable state space. The paper reads well and it seems interesting, however, I feel some of the claims are not sufficiently tested and substantiated.
- If I understand correctly, the order of states also affect the overall performance of policies. That is, if a mutant state occurs at the very beginning of the trajectory that could affect the whole trajectory and may land the agent into a low-performing state. Would you please clarify how the proposed ranking mechanism would deal with this ordering of states? Do you ignore replacing the early states in the trajectory or the ranking mechanism automatically detect these important states?
- Following on the previous question, are there cases where mutating the early states in the trajectory has minimal impact on the overall performance of the policy?
- It seems the mutation policy could have an important impact on the overall performance. That makes me wonder if the proposed approach can be scaled to more complex environment where forming an abstraction mechanism is not straightforward. An interesting result to see is where the mutation is simply `picking a random action`.
-  While I appreciate the authors bringing up RQ3-4, I think the claims are unsubstantiated and not well-supported (as mentioned in the paper, these claims require more thorough study).
limitations_and_societal_impact: As I think the mutation policy and the abstraction mechanism could have significant impact on the overall ranking and performance of the pruned policies, one of the limitations of this work could be its applicability to more complex environment. Also, some of the claims in the paper are not well studied.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper studies the problem of "ranking policy decisions," which uses spectrum-based fault localization to compute an offline score for each action,
conditioning on a state. The main idea is that the action with a higher success
frequency should be ranked higher than other actions. Studying the action rank
of policies presents an interesting angle to investigate RL policy from simplifying the DRL policy, interpretability, offline evaluation, while the experiments conducted might not
be sufficient to support either one aspect solidly.  
main_review: ### Overall

My main concern of this paper is that it tries to cover a range of topics:
pruning DRL policy,  improving DRL interpretation, offline policy evaluation(comparison),
solving any single one of them would make a significant contribution already, while
this work couldn't provide sufficient evidence to demonstrate the proposed method 
can work well on any one of those aspects. 

### Methodology
What is the reason for accessing a power set of state?
What is the formal definition of "mutant state"? 
From (2a) to (2d), we have the formula to rank actions given a state. What is the
formula to rank states as mentioned in Line 107?
As the frequency count relies on the behavior policies used to collect data, it would be better to detail how the trajectories are collected. 

### Missing reference
A related work [1] formally learns the rank of actions (decisions in this paper's terminology) in either an online or offline setting. 

What is the advantage of the proposed method comparing to learn the rank directly from
data, then applying the learned rank to the downstream task? 



### Experiments

Overall, the baseline compared in this paper is relatively weak. Even though the classic
RL algorithm [Q-learning based: DQN, DDQN, etc.][policy gradient based: PPO, A3C, etc.] does not learn the rank of action as their goal, we can use the action value or policy logits learned from those algorithms to serve as a baseline comparison. Based on [1]
the policy gradient approach is learning the listwise rank of actions. 

The experiments on the interpretation of RL policy is lack quantitative and
qualitative support. Figure 3 seems not convincing to verify the proposed method is interpretable as any method can plot such a "heatmap".


[1]Lin, Kaixiang, and Jiayu Zhou. "Ranking policy gradient." ICLR (2020).





limitations_and_societal_impact: No potential negative societal impact.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper introduces a novel method based on spectrum-based fault localization (SBFL) from the software testing domain to identify the states in which decisions made by an RL policy are most influential in determining whether or not the agent achieves its objective. More specifically, the authors create "mutant trajectories" where, with probability $\mu$, a state is mutated and the action that the agent must take in that state is the "default action" (e.g. "repeat the previous action") or unmutated, where the agent acts according to policy $\pi$ when in that state.  From these mutant trajectories, the authors use SBFL to determine a ranking of states.  SBFL takes as input the vector $\<a_{ep}^{s}, a_{ef}^{s}, a_{np}^{s}, a_{nf}^{s}\>$, where $a_{ep}^{s}$ is the number of successes achieved when state $s$ is a non-mutant state, $a_{ef}^{s}$ is the number of failures achieved when state $s$ is a non-mutant state, and $a_{np}^{s}$, $a_{nf}^{s}$ log the number of successful and failed trajectories when state $s$ is a mutant state.  In order to assess the quality of the ranking returned by SBFL, the authors create "pruned policies" where an agent can only use the policy $\pi$ for the top $r$ states, and must use the default action for all other states.  Overall, in a wide variety of settings ranging from MiniGrid and Cartpole to Atari games, the authors find that, by ranking states using SBFL, they are able to encourage agents to vastly reduce the length of their trajectories while still attaining high reward.  This is a significant contribution from the perspective of creating interpretable AI.
main_review: To the best of my knowledge, the method that this paper introduces is very novel, and is an exciting contribution in itself.  The paper is well written and clear.  The authors conceive of reasonable baselines for comparing their SBFL pruned policies against, despite the fact that the question they tackle here is novel, so there aren't readily available baselines from existing work.  That the authors show the performance of their framework on a wide variety of tasks is a great strength of this paper. 
limitations_and_societal_impact: In general, I was very impressed by this paper.  The authors introduced a novel framework to tackle an interesting new problem. The results of applying this framework to a wide variety of tasks were largely impressive.  My only concerns are the following:
- The biggest concern that I have about the widespread adoption of this framework is the fact that there are many parameters that are hard to tune in an automatic way (and, as such, the method fails on some Atari games).  Do you have any suggestions for setting these parameters in an automatic way?  For example, I found it a bit arbitrary how pruned policies were created: namely by keeping top-r.  You could perform a clustering to see if there are 2 clusters of states according to SBFL metrics
- Saturation of Figure 2c and 2d at less than 100% - I may not be understanding something, but I would have expected these to saturate at 100% for freqVis and rand when the original states and policy's actions taken tended to 100%.  Can you explain why this is not the case?
- Concern (Line 155/156): "Note that a mutant execution may visit states not typically encountered by $\pi$, meaning that we are able to even rank states that are typically out of distribution."  The converse is also true though: mutant execution may mean that certain states that are typically encountered by $\pi$ are never reached.  Was this a problem that the authors came across?  If so, did they have any ideas for how to resolve this problem?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The goal of this work is to leverage fault localization techniques from software development and debugging as a way to simply learned RL policies. The main premise is that learned policies are unnecessarily complex. In order to simplify their complexity, it is critical to rank the states where the learned policy decisions are critical. Motivated by spectrum-based fault localization, authors leverage it in RL by using similar cost functions to compare complex policy functions to so called mutant executions which are policies that recommend default or previous actions as a way to simplify the complexity of the policy class. Based on such mutant policies (identified on the fly by parametrizing specific aspects), the authors demonstrate they are able to rank important states (based on the policy's impact on outcome). The proposed method is evaluated on standard RL environments (atari, minigrid, cartpole) to quantify efficacy and visually demonstrate interpretability via saliency maps. 
main_review: 1. Vague definition of simple policy can be formalized. If the authors are restricting to stochastic policies, why not formalize the notion of simple vs complex policy functions with something like entropy? I think it will also improve the evaluations done in the experiments section. 

2. Worthwhile to consider other mutant policies (instead of just the one of replacing with default previous action) to test generalizability of the contributions. I suggest potentially breaking down the atari experiments to try different mutant policies. 

3. What is $\hat{S}$ in reference to an on-the-fly mutation (last para, pg 4, line 148). Are you suggesting that abstract state representations are used in combination with on-the-fly mutations to collect trajectories?

4. In experiments, it is unclear why the same set of environments do not achieve the desired reward with either the proposed or the random policy. Can the authors comment on whether it is an issue for the environment or something else?

5. The saliency map comparison is odd because semantically they're comparing very different things depending on the goal. It adds to the confusion to analyze interpretability of the policies. Overall another measure like entropy might be a better motive to formalize what this work is trying to do.

6. From the four different measures used for ranking in states in SBFL, I really would've liked to see more insight into what to use versus not. Instead the authors choose the best performing for every environment(?) which is not adding much insight. Since you've already experimented with all measures, can all measures be reported and discussed? 

Overall I am happy to reconsider the score if these concerns are addressed. 

----------------------------------------------------------------------------------------
I have gone through author responses and I have increased my score to 6. I trust that authors will make all the changes that came up during the discussions.
limitations_and_societal_impact: I don't believe there are negative social implications of the proposed work. The authors do address limitations of their methods sufficiently.
review_title: null

Meta-review: The direction of the paper is based on the premise that learned RL policies are unnecessarily complex and only a small subset of the overall set of decisions yields considerable improvements over some simple baseline. Thus the paper considers ranking of RL policy decisions, and in particular, identifying states where the RL policy's decisions most influence the outcome of the agent achieving its objective. The paper presents a novel method based on the software testing domain, spectrum-based fault localization (SBFL).  The paper uses this to simplify the policy and shows that their approach can help agents considerably reduce the length of their trajectories while still attaining high reward in various settings.


The paper tackles an important problem with a novel approach. Their results show a vast improvement over a variety of settings, however, some concerns are raised by reviewers about the generality especially to complex environments / continuous settings. There are also concerns raised about the interpretability part and RQ3 and 4.


While the paper is far from perfect, in my opinion, the novel approach to an important problem and the empirical results for the settings studied push it over the bar. The authors, however, should carefully revise the paper accommodating all of the points that arose in the reviews, including but not limited to mentioning the use of NoFrameskip-v4 versions of the environments in which actions aren’t repeated, a clarification (ore even better, a formal definition) of the ranking, and addressing the saliency map interpretation.
Decision: Accept (Poster)