Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes learning decision trees over latent features using MCRTS to improve interpretability of Deep Reinforcement Learning.

The paper leverages the Information Bottleneck principle to find explainable trees with minimum description length. 

The proposed method uses VAE to learn a disentangled representation on which to build regression trees that more easily interpreted.
main_review: # Originality: 
The problem of understanding and being able to interpret DNN used for RL is an important problem, both for developing and validating new ideas, as for increasing the safety and confidence on the learnt models.

The paper introduces a novel combination of VAEs with MCTRS to learn a more interpretable policy that can mimic the behavior of the DNN.

# Quality: 
It is not clear how IDOE is capable of learning an identifiable disentangling object encoder, and how identifiable and disentangled are the variables and whether they correspond to objects or not.

Missing comparisons with Gradient Boosting Trees.

There is no validation of the claim that VAEs learn disentangled representations, or that the latent variables are interpretable by humans. 

There is not proof that the latent variables learnt by VAEs are more interpretable than DNNs latent features.

In Section 4.2 it's unclear that the latent representation can capture objects and its relations in the different tasks.

In section 4.4 the claim that IDOE learns a symbolic abstraction of state space by representing object variations is not sustained. Showing 2 cherry picked examples are enough.

It is not clear how the number of objects and latent size was chosen and those impact the results.

The human evaluation with 12 participants is very weak. The paper didn't present a good protocol for human evaluation. For instance not clear how the participant were chosen and interviewed.

# Clarity:
The is not very clear about its main contributions and its significance. The writing is not very clear or well organized. 

For example section 4.3 is confusing on how the model is implemented and how the different CVAE are defined or learnt.

Not explanation of why Flappy Bird, space Invaders and Assaults where chosen for the experiments. 

The Figure 5 representing the decision tree is hard to interpret the changes on the outputs. For example how to read the  advantage of different actions (down vs up) on different branches.

# Significance:
The paper claims a promising mimic performance, but not clear how good is the mimic policy at actually solving the tasks.

The paper shows some interesting and promising results, but its applicability to complex RL tasks is missing.


# Minor issues:
In Figure 4, only the right figure has confidence bands around the different algorithms.
limitations_and_societal_impact: The paper briefly mention some limitations of the work in section 6.3, although focus on computational complexity rather than interpretability complexity. It is not clear if the proposed method increase a shallow interpretability of RL.

The claim that "A potential negative societal impact of our work is that it encourages the government to endorse laws that regulate the level of transparency in AI systems." has no base, and also it is not clear it would be negative societal impact, instead of positive.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on building a tree representation from pixels to evaluate interpretability in deep reinforcement learning. The proposed approach relies on features learned by a variational autoencoder. A mimic tree then extracts disentangled object representations for showing the causal impact. The method is evaluated on three game environments: Flappy Bird, Space Invaders and Assaults.
main_review: Strength:
The paper is well-written. Increasing interpretability in learned features for RL is very useful for real-world tasks. The methodology and experiments are well explained.
Weakness:
* Line 18: as far as I know vision RL still remains a challenging task except some known environments such as atari. It would be useful if authors cite works outperforming human level control.
* Line 188: Authors mention that one can collect enough data to learn object representation in RL. This in real world scenarios is not applicable and is the main reason researchers focus on sample efficiency.
* Line 185: Can authors provide more details on how unet was used for attention network? This seems to make it computationally expensive.

limitations_and_societal_impact: Authors mention “A potential negative societal impact of our work is that it encourages
the government to endorse laws that regulate the level of transparency in AI systems. This may
increase the difficulty of deploying advanced AI systems.” as a negative impact which can be seen as positive in most cases.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a novel method of interpreting individual decisions of a DRL policy trained by deep Q learning. The explanation they want to deliver is to identify the important features within an input state that contribute most to the corresponding Q function output. Technically, it first learns an encoding layer to transform the high-dimensional input into low-dimensional hidden representations. To enable explainability, the encoding layer is designed to preserve the physical meaning of the objects in raw inputs. Using the hidden space representations, it proposes a Monte Carlo Regression Tree Search (MCRTS) algorithm to train a tree to mimic the action advantages.  To validate the effectiveness of the proposed model, the authors evaluate the proposed technique from two aspects: fidelity and interpretability. The evaluation on three RL environments demonstrates the superiority of the proposed method over some of the existing explanation methods. 

main_review: This paper is overall well organized and written. The proposed technique is interesting and clear explained. I like the idea of the two-step interpretation: hidden representation + mimic model, and I appreciate the authors' effort in keeping the identifiability and interpretability of the latent representations. They are very important properties towards providing explanations. The evaluations are relatively well designed and comprehensive. I appreciate that the authors include a relatively complete comparison with the baselines. IMHO, the paper would be more convincing if the authors could address my following comments.

My main concern is in the fidelity evaluation. I appreciate that the authors evaluate this important property. However, the current evaluation is not comprehensive. More specifically, approximation accuracy is essential for mimic-based explanations. IMHO, it is only the first step towards a comprehensive fidelity evaluation. More importantly, I would suggest the authors evaluating whether the derived explanations actually reflect the decision-making process of the neural networks. The authors could refer to some existing fidelity evaluation metrics used in the papers that try to derive important features as explanations[1,2]. The basic idea of these metrics is to perturbing the identified features and observing the changes in the corresponding NN's outputs. If the identified features are truly important ones, perturbing them will trigger a large change to the output. The authors could borrow this idea and design a fidelity metric to evaluate the explanation fidelity or just use the existing ones (If they are applicable). In summary, enabling an accurate approximation is only the first step towards achieving faithful explanations. The authors need to conduct another evaluation directly attached to the derived explanations. 

Some minor comments:

1. I may miss this detail. I am wondering whether the MCRTS can be applied to the raw features. If yes, I would like to see the results. It could further demonstrate the effectiveness of the feature extraction step.

2. The current version only showcases the explanations on one of the three environments. I would suggest demonstrating the explanations on the other two applications and conducting the human evaluation on those applications too.

3. The proposed technique utilizes some existing methods to address some of the challenges. Among them, VAE is a well-known technique, which may not need an introduction in this work. However, [34] may not be that well-known. I would appreciate that if the authors could provide some background knowledge regarding these non-well-known techniques.

======== Post rebuttal Comments===============

After reading the authors' response, my concern about fidelity evaluation still stands. I would still suggest designing a quantitative fidelity evaluation w.r.t. the RL agents together with an evaluation metric. The authors could design their own metric by following the methodology similar to the counterfactual analysis (remove and rerun). With the metric, the authors could compare the fidelity of their proposed approach with that of existing explanation methods. As things stand, I would be inclined to maintain my current score.

limitations_and_societal_impact: The authors discussed two limitations in Section 6.3. Additional limitations and future works could be (1) The proposed technique can only be applied to environments using images as inputs; (2) Generalizability of the proposed technique to more complicated environments, multi-agent environments; (3) Generalizability of the proposed technique to policy networks.

I appreciate that the authors include a social impact discussion. And I agree with the discussion points. To make it more comprehensive, the authors could discuss the possible negative impact from the adversarial point of view, .e.g, attackers may leverage the proposed explanation methods to develop stronger attacks.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposed a new explanation approach for reinforcement learning.
main_review: The research is a little bit incremental. There are already many interpretation methods in the context of reinforcement learning. The work proposed is yet another method that generates explanation in higher fidelity. However, the evaluation of higher fidelity interpretation is insufficient. Some existing techniques are not compared (see below). 

In addition to the methods based on interpretable trees, there are many other explanation methods for RL applications (e.g., saliency map-based approaches and attention-based approaches). The paper does not conduct an experimental comparison with these alternative approaches. It would be helpful to show the performance difference between alternative approaches and the proposed method.

To evaluate the effectiveness of the proposed method, the authors measure the fidelity of the model's capability in approximation. This, to some extent, indicates fidelity, but it does not imply a good mimicking performance is a good explanation. I am happy to see the authors conduct counterfactual analysis to assess the fidelity of interpretability. However, the authors only show a case but do not quantify the fidelity. I think it would be more convincing if the authors could follow existing and commonly adopted methods to assess the fidelity of the interpretation (e.g., blinding some critical component in the explanation and see how that influences the action of the target agent quantitatively). 

The paper shows some interesting results on some simple games. I wonder if the proposed method is applied to more sophisticated games (e.g., starcraft or super Mario that have a more complicated background and foreground), will the proposed method still generate a decent latent representation for background and objects? It would be more promising if the authors could show results on more sophisticated games.

I am happy to see the authors evaluate the explainability by conducting a user study. First, I am not sure if the authors received IRB approval before conducting this human study. Second, the human study is subjective. Even if an interpretation makes sense to a human, it does not imply it is the right explanation. Can the authors justify the validity of this human-based assessment?
limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: This paper explores the problem of learning an interpretable version of the decision procedure used by neural RL, which reviewers agree is interesting. There was some disagreement about how substantial the results are, especially regarding fidelity and practicality. While the majority of reviewers ended up mildly positive, I believe this paper can be substantially improved by taking the comments to heart in guiding further experiments. 
Decision: Accept (Poster)