Learning Tree Interpretation from Object Representation for Deep Reinforcement Learning

Abstract: Interpreting Deep Reinforcement Learning (DRL) models is important to enhance trust and comply with transparency regulations. Existing methods typically explain a DRL model by visualizing the importance of low-level input features with super-pixels, attentions, or saliency maps. Our approach provides an interpretation based on high-level latent object features derived from a disentangled representation. We propose a Represent And Mimic (RAMi) framework for training 1) an identifiable latent representation to capture the independent factors of variation for the objects and 2) a mimic tree that extracts the causal impact of the latent features on DRL action values. To jointly optimize both the fidelity and the simplicity of a mimic tree, we derive a novel Minimum Description Length (MDL) objective based on the Information Bottleneck (IB) principle. Based on this objective, we describe a Monte Carlo Regression Tree Search (MCRTS) algorithm that explores different splits to find the IB-optimal mimic tree. Experiments show that our mimic tree achieves strong approximation performance with significantly fewer nodes than baseline models. We demonstrate the interpretability of our mimic tree by showing latent traversals, decision rules, causal impacts, and human evaluation results.


Reviews and author responses:

This paper proposes Represent and Mimic (RAMi), a framework for extracting an interpretable mimic tree from the advantage function learned by a deep RL agent. This is achieved by (1) using a variation of MONet [34] to extract disentangled object representations from visual state observations that are interpretable, and (2) fitting a distribution over regression trees to explain the advantage of an action in a particular state in terms of causal relationships between these representations.

The main innovation is the objective function for the regression tree search, which is inspired by the information bottleneck approach and the principle of minimum description length. Here, the distribution over regression trees is viewed as an "information bottleneck" for predicting the advantage of an action-state pair (i.e. the target) from their input representation. This suggests that the tree search should prioritize trees having small description length (i.e. that are compressed and "simple") while accurately modeling the target values (i.e. having high fidelity), which is achieved by incorporating suitable heuristics.

RAMi is compared on three standard RL datasets, and compared to a number of baselines that primarily vary in terms of how they extract the mimic tree. Qualitatively it can be observed that RAMi is able to balance the trade-off of simplicity and fidelity quite well, and outperforms prior approaches on standard metrics. It can also be observed how encoding pixel-level observations to disentangled object representations is helpful. Qualitatively it is demonstrated how the trees learned by RAMi are interpretable and allow for a causal explanation of how certain state-action combinations affect the prediction made by the advantage function.
  This paper is concerned with an interesting and relevant problem: given a trained deep neural network, how can we explain their predictions? In the case of RL, as is the context of this paper, the prediction may be the learned advantage function, and the goal to explain how certain state-action pairs lead to higher expected future normalized return compared to others.

The approach taken is that of "mimic learning", where an interpretable learner such as a decision tree, is fit post-hoc to the predictions made by the deep neural network. Prior approaches use linear model trees to achieve this in the context of RL, however due to them being applied to low-level pixels, their extracted rules are complex. Another consequence is that the resulting trees grow quite complex, while it is not clear how they should be balanced.

This paper aims to address both of these problems by (1) applying the mimic learning process to a learn feature space that distinguishes high-level variables such as objects, and (2) using a tree search method that considers the global tree structure and is better able to trade-off fidelity and simplicity. 

To address these problems it is proposed to address them from within a single framework called Represent and Mimic (RAMi) based on the Information Bottleneck (IB) principle [13]. Here, the mimic learner is viewed as an "information bottleneck" for encoding information about the target variable (advantage) from the input (state, action, reward), which suggests a means of learning the mimic tree to trade-off fidelity and simplicity. In particular the variational lower-bound to the information bottleneck as proposed in Alemi et al. [27] is modified to include latent variables z for encoding information about the input x, as well as a reconstruction term, which gives rise to a three part objective function. The first part is the standard ELBO used to train latent variable models such as VAEs. The second part can be understood as capturing the description length for encoding the targets using the mimic model as well as the description length of the mimic model itself. The final term is an entropy-regularizer.

The derivation of this objective (theorem 1) largely follows prior work, while incorporating the reconstruction term was rather adhoc. Since this training objective is not actually used to train a single system end-to-end, but rather separate systems are trained on separate terms, it is not quite clear what is gained by modifying the objective from Alemi et al. [27]. Would it not have been much simpler (and more principled) to treat the problem of representation learning separately outside the IB framework to give rise to the standard ELBO objective? And then apply the IB objective directly to latent observations z as opposed to x? It seems to me that one can reach a similar objective for the tree search that can also be understood in terms of MDL. This would certainly be preferable since the connection between MDL terms and the ELBO is not exploited during learning as far as I can tell.

To learn good representations using the ELBO objective, an approach called Identifiable Disentangling Object Encoder (IDOE) is proposed based on VAEs. As far as I can tell, IDOE is really just MONet [34] but having a conditional encoder and prior (conditioned on actions and reward) inspired by the Identifiable VAE from Khemakhem et al. [33]. Further the prior factorizes across latents as is a standard assumption. The fact that IDOE is really just MONet with only very slight modifications is not sufficiently made clear in section 4.1-4.3 and I urge the authors to avoid renaming prior work in this way and claiming innovations that belong to others! The choice for MONet with a conditional and factorized prior is well motivated, since the goal is to obtain an input representation that is identifiable and disentangled to avoid having confounders when modeling relations between variables when interpreting the mimic tree learned later one. I think the MCC experiment is quite nice here, although details about the set-ups are lacking. The latent traversals in section 4.4 are less convincing since in the bottom right it can be seen how Z_2,3 not only captures the length of the left pillas, but also the size of the bottom pillars, which go up and down during traversals. I would therefore suggest that the authors tone down claims regarding the impossibility of confounders (such as in 321).

Given the representations learned by MONet, what remains is inferring the mimic-trees, which in my view is where the main innovation is at. To balance simplicity and fidelity as suggested by the IB principle one should search for trees that can be encoded efficiently, and that represent the data well (require encoding few exceptions). The cost of encoding a tree (proposition 1) is computed using the metric from Quinlan et al. [42], where again the authors fail to make clear that they are using an existing method as opposed to inventing a metric themselves. The derivation and "proof" provided in appendix B.3 is identical to [42] (equation 6 & 8)  other than inserting different values for the parameters. To be able to measure the effect of exceptions gaussian distributions are used at the leaf nodes, whose variance reflects the amount of exceptions that need to be encoded. The log of the variance can then be used as a cost metric. Both of these measures make sense to me and the connection to the IB-principle and MDL is clear.

The search itself is implemented using Monte Carlo Regression Tree Search (MCTRS) making use of these cost functions. The explanation in section 5.2 is quite dense and I was not able to understand in detail how the search proceeds. I believe that section 4 can be shortened significantly, which should make it possible to provide a gentler discussion and provide an intuitive explanation of the various update rules. 

The reported results are quite promising and MCTRS is compared to a number of other tree search algorithms that act on the raw observation and on the encoded observations by either a VAE or MONet. In terms of standard metrics such as variance reduction and root mean squared error it can be observed that MCTRS is the best performing model. Perhaps in the interest of space, the comparison to modeling raw pixels could be moved to the appendix in favor of the RMSE results, since it is quite clear that this is suboptimal. The leaf-by-leaf regression performance is quite interesting and indeed suggests that MCRTS performs better decision-making under a limited budget of leaf nodes.

The qualitative evaluation is impressive and clearly shows how this approach can be utilized to infer interpretable mimic trees. The benefit of learning a representation in terms of object representations is clearly shown, since intermediate leave nodes can be visualized nicely in this way. I wonder if these visualizations could be combined with pixel-level techniques such as attention maps to obtain the best of both worlds. A limitation of the counterfactual analysis in the way it is presented is that it is not clear how to obtain latent vectors that correspond to the kind of interventions one would want to perform at a pixel-level, which should be discussed. The human evaluation results are interesting, but also limited, since it is likely that the 12 participants from 3 different universities do not at all reflect the general population. Rather, I would encourage the authors to include additional analysis for MCRTS focusing on the implementation of the two cost functions, as well as sensitivity to hyper-parameters. Qualitative analysis on the other 2 datasets is also missing, which should be added (to the appendix)

In general I would like to see this paper accepted, since the results are impressive and the proposed approach is sensible and well motivated. However, the clarity of the paper should be improved significantly, and the authors should clearly indicate what parts are novel contributions and what parts belong to prior work. I would also encourage the authors to consider separating the ELBO and the IB principle, since their connection is currently not exploited (no end-to-end system is trained) while the derivation is ad hoc. The experiment section can also be improved as indicated above. Currently is it difficult to gauge whether the demonstrated examples are cherry-picked or not.

**Minor Comments**

* Error bars in Figure 4 are only shown for MCRTS in the right plot, but not in the others. Please try to include them for the left and middle plot as well

* I would have liked to see additional examples of the mimic trees extracted in the appendix, as well as examples of mimic trees extracted by other methods

* I would like to see the related work section expanded to include a discussion of other approaches to learning representations that satisfy the criteria met here with MONet. An overview of other methods to learning object representations can be found in Greff et al., 2020 "On the binding problem ..."

* It is an assumption that there are no confounders after training MONet, which is not evident from the current discussion. This is problematic since this ideal representation is likely not achieved in practice.

* At times the paper conflates a representation that is disentangled with one in which the posterior factorizes (and so does the prior). This is not how disentanglement is treated in other works and it would be good to indicate that this particular notion is adopted here.

* highlighting the best method is missing in Table C.1

* I believe the value for Figure 5c should be 0.11 as opposed to 0.23. Currently there is no change as far as I can tell. 

* The entropy sign in equation (1) should be a + I believe

**Post-rebuttal Update**

I have read the other reviews and the author response. I am satisfied with the author response insofar the authors agree to do a better job at framing their contributions wrt. prior work and discussing limitations. On the other hand, I remain of the opinion that the shared RAMi objective contributes little understanding and makes the overall approach needlessly difficult to understand. In particular, there is little to no interaction between the representation learning parts (ELBO) and the mimic tree learning parts (MDL), and presenting this method as a 2-stage approach based on different principles is more natural. 

The qualitative empirical results look impressive, although I would have liked to see additional examples to understand how reliable this approach is. The quantitative results also look good and the authors appear to provide a thorough comparison to prior methods. I note that I don't consider myself an expert in the area of mimic tree learning (or explainable ML in general), and it is possible that I have misjudged the significance of these results. I intend to keep my current score of a 6 with medium confidence (3). Both statements are provided albeit quite brief. Regarding limitations, it would also be good to comment on current failure cases and let the reader develop an understanding of how cherry-picked the results shown are.

 Thanks for pointing out the important issues of our work. Our reply to your concerns are as follows:

1. We will carefully revise and motivate our presentation.

2. We will rename IDOE according to your suggestion. We believe I-MONet could be an ideal candidate name for our method.

3. Our object interventions are performed on latent space. In terms of a "particular intervention", we assume you indicate manually or randomly hiding some parts (pixels) of an object and checking the influence to model output. Note that such an intervention is actually performed on pixel space, which does not fit our framework. Within our framework, the intervention will only influence the variation (e.g., length and width) of an object and these variations are automatically determined by the IDOE (or I-MONet). To perform an intervention, the user should first understand the variations captured by the latent variables with latent traversals and then pick a latent variable to intervene. We agree this intervention is not as straightforward as the pixel-level intervention, which humans are generally more comfortable with. We will add this discussion to our limitations.

 Thank you for the initial reply. I have two additional comments and a request for clarification.

Regarding (1), this seems like a weak connection/dependency, while it obfuscates the overall presentation of the idea. If the authors wish to keep the current derivation as is, I would at least suggest that they try to improve the presentation.

Regarding (2), I agree with the significance of this change, even though it is small. However, you are still combining two existing ideas: IVAE + MONet but re-brand it under a different name: IDOE, which I disagree with. In order to not take away from the authors of these existing methods, I strongly suggest that you adopt a name for the encoder that reflects that this is a combination of existing ideas, e.g. I-MONet, IVAE+MONet or similar (in addition to clarifying this connection much more in the main text as promised above).

Regarding (4), I understand the benefit of working in object-latent space compared to pixel space, however, it is still unclear how to design an object-latent to perform a particular intervention. For example, lets say that I would want to perform an intervention other than setting an object latent to 0, how would I choose such latent? One way could be to look at the dataset and consider the latent of another segmented image, but this is still rather impractical. It would be great if you could comment on the practicality of the current approach for conducting arbitrary interventions in latent space, which seems a current limitation and is what my initial comment tried to get at.


 Thanks for your comments

* For your concerns in (1), we set K as a hyper-parameter.  For the majority of the RL environments, the number of objects in a state is consistent throughout a game, so we empirically locate the **range** of K by observing RL states and then determine the **smallest** K that provides a satisfying **reconstruction performance**. Note that we only need to ensure $k\geq$ "Number of Objects" for a satisfying reconstruction performance, but increasing K will expand the dimension of learned latent representation and thus add complexity to the following tree learning.

 Thank the authors for replying to my questions.
* (1) I get the general idea, but I am still not sure if I understand the object mast correctly. Do the authors mean the number $k$ should be pre-defined according to the environment? If true, how to determine this number?
* (2) The authors’ response is satisfactory. I suggest the authors add the discussion to their limitations. 
* (3) The authors’ response fully answers this question.
* (4) The authors’ response fully answers this question.


 Dear Reviewer,

We appreciate your constructive comments, and we will carefully revise our paper according to your comments. Here are our response to your concerns. 

1. " *Why can IDOE decompose a state into object-level components? I thought IDOE is a carefully designed encoder-decoder structure with disentangled latent variables, then where does the decomposition happen?* "

* **Response** The decomposition is learned by the spatial masks (see lines 173-174). These masks are generated by an attention network to capture the objects in an image. For example, the $k^{th}$ mask $m_{k}$ dynamically captures which part of the input image belongs to the $k^{th}$ object. To better illustrate this idea, Figure 2 visualizes the learned masks (see $m_{1}, m_{2}, m_{3}$) and explains how we decompose an image into different objects (background, pillars, and birds). 

2. " *It seems that all 3 RL environments have a clean background and clear objects separation. Will the IDOE model still work in more complex real-world environments?* "

* **Response** This paper is aimed at interpreting RL algorithms, and a majority of the existing RL algorithms are evaluated on the benchmark of gaming environments (e.g, Atari). This is the main reason why we study these environments. In terms of more complex environments, we believe you are referring to some real-world applications of RL, such as autonomous driving. It is true that extracting objects from a real-world environment will be harder because of the background noise. However, we believe learning an object representation is still the key to solve the complex task since it is generally easier for the policy layer to distinguish reward-relevant information from noise at the object level rather than pixel level.

3. " *If the IDOE model fails, e.g. encodes entangled latent variables, how much it will influence the mimic tree generation?* "

* **Response** We have studied the influence of sub-optimal representation on the following mimic tree generation. Our experiment has applied VAE as a baseline. We show that VAE captures inconsistent variations in Section 4.2 (line 148) and provide some failure examples of VAE in Section E.1 (in appendix).  The gap of the mimic performance between VAE-based trees and IDOE-based trees can explain the influence of a failure representation. %We will present it more clearly in our revised version.

4. " *If I understand Section 4.4 correctly, human efforts are needed to check the interpretability of latent variables with latent traversing. Then, how about the scalability for complex tasks?* "

* **Response** The goal of exploring RL interpretability is to let humans understand how RL works. Reasonable interaction between humans and machine is necessary for better understanding. It explains why previous works on interpretable RL often involved some human effort, for example, attention-based or super-pixels based methods require manually concluding what information has been captured by the attentions or pixels. We agree that more human effort will be required for our model in order to scale from simple to complex tasks, but on the other hand, we think this cost is inevitable for the majority of the methods. How to minimize the cost of human effort is an important challenge for future work.


 Dear Reviewer,
We appreciate your constructive comments and provide our response as follows.

1. " *The originality of MCRTS algorithm compared to the decision tree policy of VIPER model is not clear by the given text and Prop 1.* "

* **Response:** MCRTS is a heuristic search algorithm that utilizes our IB-MDL objective as its search heuristic, whereas VIPER learns policy trees with the traditional Classification And Regression Tree (CART) algorithm that determines a split in a greedy manner. Prop 1 introduces how we infer mimic trees based on the IB-MDL objective in practice, according to which MCTS builds its reward function (or heuristic). We will elaborate this point in the revised version.

2. " *The paper should have provided more information and supportive experiments about the excellence of the model, as this work aims to learn "interpretable" representation.* "

* **Response:** We are not sure what additional experiments should be included.  We believe our experiments are comprehensive. In summary, we evaluate our model with respect to fidelity, simplicity and interpretability in Sections 6.1 and 6.2. The interpretability is evaluated from multiple perspectives, including 1) the MCC experiments to show the identifiability of IDOE (Section 4.2) 2) the illustrative examples to show the interpretability of object representation (Figure 2) and the mimic tree from MCRTS (Figure 5) 3) the human evaluations to empirically qualify the level of interpretability. Many of these results are moved to an appendix (Section C and Section D) due to space limit. Potential limitations are discussed in Section 6.3. 

3. " *The inductive bias to disentangle latent space used in this paper is the spatial mask of objects in state scene, which is a reasonable implementation choice only for limited environments. Using a mask will be an inappropriate choice for long-term, sparse environments like Montezuma’s Revenge.* "

* **Response:** The masks are learned for identifying and representing objects in each frame. Our RAMi framework splits this representation component from the decision component, which is about modeling the interaction of these objects and their relation to the long-term reward. For example, in the setting of Object-Oriented RL (briefly introduced in line 190), to solve Montezuma’s Revenge, the agent should extract objects at every frame and learn a value function to capture their relationship to future cumulative (sparse) rewards. Since the mimic learning applies neural network output as the oracle, our mimic tree can discover this long-term relationship if the teacher model (or neural network) can properly model it with the value function. 

4. " *Though experiments show that disentangled latent was unique to problem space unlike normal VAE and the IDOE reduced variance of regression, the most experimental contribution to variance reduction per leaf looks due to MCRTS.* "

* **Response:** We agree that the Variance Reduction Per Leaf (VR-PL) metric is designed to evaluate the efficiency of tree search and show the benefit of MCRTS as mentioned. However, apart from VR-PL, we include other metrics (e.g., total Variance Reduction,  Mean Absolute Error (MAE), and Root Mean Square Error (RMSE)) to evaluate the fidelity of our model. We also include baseline mimic trees that apply raw state features and latent features from a normal VAE as input. They together can demonstrate the importance of IDOE. The complete results as well as the corresponding variances are in table C.1 (page 7) in the appendix due to the space limit.

5. " *Most importantly, the paper has weak relation to RL framework unlike its title.* "

* **Response:** Our IDOE is specially designed for capturing objects in RL environments since 1) an RL state can generally be represented by a limited number of objects, and 2) the agent can interact with the RL environment to generate sufficient data for learning such a representation (see line 187 - line 193 in our paper). 

6. " *The objectives of the suggested framework don’t exploit any properties of RL assumptions, thus the regression target can be replaced to any utility functions for continuous state space.* "

* **Response:** In terms of RL assumptions, we believe you indicate the properties of the Markov Decision Process (MDP) and Bellman equation. For the MDP, our target does apply the Markov assumption by conditioning decisions on the current state and action pair. Our target is the advantage function. Advantage function has a clear definition in RL that it provides unbiased action-specific evaluations with less variance (see lines 89-99). The proof of this property is based on the Bellman equation (We didn't add it to our paper since it is a well-known proof.). Although our RAMi framework can potentially apply other mimic targets, the mimic performance will have to be compromised without these key RL assumptions.


 Dear Reviewer, we appreciate your constructive reviews and detailed comments. We will revise our paper according to your comments. Here are some of our responses to your concerns.

1. " *I would also encourage the authors to consider separating the ELBO and the IB principle, since their connection is currently not exploited (no end-to-end system is trained) while the derivation is ad hoc.* "

* **Response:** Ideally, our IDOE + MCRTS model can be trained end-to-end by optimizing both ELBo and MDL objectives together. However, in practice, since IDOE and MCRTS are based on different model structures (neural net and decision tree), training them together will be hard, so we approximate this procedure by first training IDOE and then running MCRTS based on the well-trained encoder. This approximation is supported by the fact that targets Y are conditionally independent of raw inputs X which is $p(Y|X, Z) = p(Y|Z)$. In this sense, ELBo and MDL objectives have a dependency. In an earlier version, we had a graph that clearly indicated this relation, but then we removed it due to the space limit. We will put it back in the revised version.

2. " *The fact that IDOE is really just MONet with only very slight modifications is not sufficiently made clear in section 4.1-4.3* "

* **Response:**  IDOE combines the idea of both MONet and Identifiable VAE (IVAE, see lines 158-161). Utilizing MONet by itself can not ensure the learned representation is identifiable (see Figure 1), that's why we replace the VAE encoders in MONet with IVAEs [33] that enforce conditionally factored priors. In the theory of [33], the prior can be modeled by any univariate exponential family distribution, and in this work, we pick the Gaussian distribution since the KL divergence between two Gaussians has a closed-form expression. This update might involve only a slight modification to the model structure of MONet, but the improvement is significant. We have referred to both IVAE and MONet when introducing IDOE. We did not mean to reclaim their contributions. We will highlight their works in the revised version to prevent any potential misunderstanding.

[33] Ilyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, and Aapo Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework. In The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020.

3. " *The cost of encoding a tree (proposition 1) is computed using the metric from Quinlan et al. [42], where again the authors fail to make clear that they are using an existing method as opposed to inventing a metric themselves.* "

* **Response:**  The MDL objective required encoding both the tree structure and exceptions. For tree structure encoding, we have referred to Quinlan et al. [42] when introducing the cost of encoding the tree structure. Proposition 1 is mainly based on the theory in [42], which we mentioned in the proof of Proposition 1. Again, we did not mean to reclaim their contributions. We will highlight their works. However, for exception encoding, Quinlan et al. [42]'s encoding method is not applicable since it focuses on only the decision tree instead of a regression tree, so we utilize a log-variance metric which is inspired by CART (line 219-226).

4. " *A limitation of the counterfactual analysis in the way it is presented is that it is not clear how to obtain latent vectors that correspond to the kind of interventions one would want to perform at a pixel-level, which should be discussed.* "

* **Response:** Our intervention is based on latent variables instead of pixels since we believe pixel-level causal inference is impractical. The difficulty lies in the fact that pixels are often entangled and image data often has high dimensionality. We explained this in Section 4.1. 

5. " *I would encourage the authors to include additional analysis for MCRTS focusing on the implementation of the two cost functions, as well as sensitivity to hyper-parameters. Qualitative analysis on the other 2 datasets is also missing, which should be added (to the appendix).* "

* **Response:** In terms of hyper-parameters, we believe you indicate $\lambda$ in our MDL objective. We empirically determine this parameter with the validation dataset since it is a key parameter that controls the balance of simplicity and fidelity. We will report more detailed results and illustrative examples for the other two environments.

* For Minor Comments, we will carefully revise our paper according to your suggestions.

## Summary
This paper deals with learning interpretation of objects and decision making using RAMi framework. The framework first learns an identifiable latent representation, and then mimics a casual relationship using a tree. As a result, the framework can construct approximations to an interpretable, high-order decision making procedure. The paper shows that it can be applied to standard deep reinforcement learning problems.  ## Main Review
### Originality and Significance.
I think the originality of the paper can be evaluated by the theoretical arguments (Thm 1 and Prop 1). As far as I understood, the proposed framework and models are somewhat novel by adopting information theoretic perspectives for modeling representation. However, the originality of MCRTS algorithm compared to the decision tree policy of VIPER model is not clear by the given text and Prop 1. This work also lacks providing qualitative results of representation (visualization, comparison, etc.).

### Clarity.
- I do not believe that the paper is clearly written. Also, the paper should have provided more information and supportive experiments about the excellence of the model, as this work aims to learn "interpretable" representation.
- Though its derivation of IB objective (Theorem 1) is sound, its relations to individual objectives of modules are a bit ambiguous. The inductive bias to disentangle latent space used in this paper is the spatial mask of objects in state scene, which is a reasonable implementation choice only for limited environments. Using a mask will be an inappropriate choice for long-term, sparse environments like Montezuma’s Revenge. Also, only latent visualizations for the Flappy Bird are attached, which makes an evaluation of interpretability less reliable. Though experiments show that disentangled latent was unique to problem space unlike normal VAE and the IDOE reduced variance of regression, the most experimental contribution to variance reduction per leaf looks due to MCRTS.
- Most importantly, the paper has weak relation to RL framework unlike its title. A VIPER model, which the paper referenced as previous mimic model combined with reinforcement learning, implemented mimic tree policy and clearly supported its relation to RL using RL’s terminology and experiment results. This paper, however, just uses the state value function of a trained RL framework as a mimic tree’s regression target. The objectives of the suggested framework don’t exploit any properties of RL assumptions, thus the regression target can be replaced to any utility functions for continuous state space. It can be considered that the paper mentions RL because it just used RL environments without sincere consideration.

### Additional Comments.
I am a bit concerned about the title. I think the paper does not show much perspectives about "Deep Reinforcement Learning". The paper does not provide usefulness of the model in the sense of score acquisition, and it is ambiguous how the model helps DRL studies (which are indeed a specific field of decision making).

### Minor.
- Module terminologies are inconsistent in a supplement.
supplement line 119: “IDE” -> “IDOE”
supplement line 122: “IDE” -> “IDOE”
supplement line 144: “IDE” -> “IDOE”

- Typos:
supplement line 1: “Flappy Bird” -> “The Flappy Bird”
supplement line 1: “we upload” -> “we uploaded”
supplement line 23: “applies” -> “apply
supplement line 98: empirically -> empirical
supplement line 142: “entangle” -> “entangled”
supplement line 159: “We” -> “we”
supplement line 161: “minimizing” -> “minimize”

 N/A

This paper proposes a tree-based method to interpret causality between actions and states in reinforcement learning algorithms. They firstly use an encoder-decoder model name IDOE to extract object-level and identifiable latent variables, then infer a mimic tree with information bottleneck as the objective function. Compared with previous methods and baselines, their generate trees have fewer leaf nodes, therefore are easy for humans to interpret and understand.  **Originality:** As far as I know, this work is original in this area. Although the idea of using a tree structure to explain RL algorithms is not new, their method still has contributions in controlling the size of the mimic tree.

**Quality:** The method proposed in the paper is technically correct and their experiments on three RL environments can also support their statements. In the evaluation part, they also provide human evaluation results to show the advantage of their method. Although this is not common in the ML community and the number of participants is relatively small, the conclusion is acceptable. 

**Clarity:** The writing of this paper is good but I feel that some details of both the IDOE part and MCRTS part are missing. Maybe it is because of the complexity of this method and the limited pages, but it would be better if the methodology part could be consecutive. One example is section 4.3. Why can IDOE decompose a state $s$ into object-level components? I thought IDOE is a carefully designed encoder-decoder structure with disentangled latent variables, then where does the decomposition happen?
There are also some typos that influence the reading:

(1)	In Line 251, Appendix (E.1) -> Appendix (E.2)

(2)	According to the description of Section 6.2, I think the bottom figure in Figure 5 (c) should be the same as $cell_{3,3}$.

**Significance:** This paper targets a very important problem about XAI and the results shown in the paper are interesting. One concern from me is that this method is too complex to be extended. Also, I have three questions about the IDOE part:

(1)	It seems that all 3 RL environments have a clean background and clear objects separation. Will the IDOE model still work in more complex real-world environments? 

(2)	If the IDOE model fails, e.g. encodes entangled latent variables, how much it will influence the mimic tree generation?

(3)	If I understand Section 4.4 correctly, human efforts are needed to check the interpretability of latent variables with latent traversing. Then, how about the scalability for complex tasks?
 The authors mention two limitations of this work: (1) the computational complexity for the MCRTS generation part. (2) lack of theoretical guarantees for interpretability. Also, as a method attempts to explain the AI system, one potential societal impact, said by the authors, is that this method is abused by the government or users.

Meta-review: The paper proposes an approach to learning an interpretable distillation of an RL agent's advantage function by first adopting a object-disentangling model (MONet) and subsequently fitting regression trees to allow extraction of causal relationships therein. A key issue of deciding between tree size/complexity and expressiveness is handled using the Information Bottleneck (IB).

The reviewers agree that the paper tackles an interesting and relevant problem, and the use of IB to drive the tree learning is novel, and the experiments are quite thorough.

The primary issues with the work however appear to be with some of the framing (with a conditioned version of MONet implying novelty) and the model setup, with questions on whether the end-to-end setup is overly complicated over a two-stage pipelined model.

On balance, it appears as if the paper has more merits than issues, and most of the issues raised could be addressed with a bit of work. I would strongly urge the authors to actually make the edits for model renaming, detail, and the utility of end-to-end here, as requested by the reviewers.

Decision: accept