Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This work presents a disentanglement method for compact NNs in policy learning. There is no ground truth for the factors of variation, since emergent behaviors are unknown a priori, so decision trees from observable states to specific neural responses are extracted from the policy, along with an inverse mapping predicting paths through the decision tree from neural responses. The metrics for quantifying interpretability are: Neuron-response variance, the concentration of the neuron response for a particular decision path. Mutual Information Gap, modified by an additional calibration term, the gap between the highest and second highest mutual dependencies. Modularity, 1 - the mutual information between a neuron response and all non-matched decision paths. Inverse correlation between metrics and decision tree quality metrics. Empirical results and qualitative illustrations of the interpretability of this method are shown for pendulum, locomotion, and visual servoing environments.
review: Quality: Overall, the paper is well-written, equations appear correct, and the diagrams are aesthetically pleasing and provide a vision for how this method could be used in a real-world robotics application.

Clarity: The paper’s explanations of the motivation for the method and the theory used are mostly clear, and most concerns are addressed by the rebuttal. The remaining concerns I have are about the amount of cognitive overload it takes to qualitatively interpret the decision tree branch analysis. I still think a video of the trajectories would make this a bit clearer, but in the meantime, the diagrams the authors produced are of good quality.

Originality: The paper claims to introduce a novel set of quantitative metrics for interpretability; however, all of the metrics appear to be taken from (cited) established works on disentanglement, decision trees, or interpretability. The paper’s most original contribution appears to be the use of extracted logical programs as a factor of variation for disentanglement in simulated control settings.

Significance: Interpretability methods like the one presented in this paper are important and relatively underexplored in robotics, where it could be highly impactful for debugging real-world issues and monitoring safety-critical applications. For future work the authors will need to address the limitations with using privileged states used in the logic programs which are not available to the policy.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper's premise is interesting and potentially useful to robotics. The authors have integrated together an impressive body of knowledge of decision trees, logical programs, neural networks, reinforcement learning, and interpretability metrics. They have clearly conducted extensive experiments using this technique, which shows promise for future work. Future work will need to show how it can be extended to different neural network architectures, how to deal with privileged states, and clearer presentations of how the branch analysis and discovered factors of variation relate to intuitively critical states in the environment trajectory.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: The goal of this paper is to develop methods for interpreting learned policies. To that end, the paper proposes learning a decision tree to model each neuron of a policy network. The interpretability is evaluate with three metric that they introduce and three metrics proposed in prior work.

The three metrics that they introduce are:
1. The average **variance** of a finite set of decision paths across neurons
2. The **mutual information gap** to compute the disentanglement of different decision paths among neurons
3. The **modularity** of each neuron with respect to different decision paths

The authors measure the interpretability of 6 different architectures on classical control, locomotion, and autonomous driving tasks.
review: Strengths:
- The main text and appendices are highly detailed.
- The findings in the paper were interesting to me/sparked a lot of ideas. I imagine many readers will feel the same way. (e.g., the behavior of command neurons in locomotion experiments)
- The idea of modeling neurons with decision trees is elegant and can be repurposed across many kinds of robotic tasks.

Weaknesses:
- The paper is motivated by interpretability but it's not clear to me that the metrics evaluate interpretability with respect to human judgments. Although three of the metrics are derived from an evaluation of human interpretability, they are ultimately re-interpreted for the paper. Adding a small interpretability experiment with human subjects would make for a much stronger claim.
- The presentation of the metrics could be made more clear. In particular the intuition behind/motivation for some of the metrics could be made more clear. This is especially true for MIG.

Nits (not material to review, just recommendations for revisions):
- Add a caption to the first figure
- Slight grammatical errors in L35, L54
- Defining K_i as the decision paths more explicitly somewhere would help with readability
- The first cited work for MIG (numbered [8] in the bibliography) does not discuss mutual information. As a reader I was confused how this paper was inspirational for this section.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I am recommending the paper for acceptance because the authors explore an elegant idea thoroughly. Although the robotics focus is not direct (as the experiments are in sim) I think many of the members of the corl community will find the method relevant and the findings interesting.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors propose a novel way of studying interpretability of learned policies for robot learning from the perspective of disentangled representations. They aim to learn a logical program for each neuron by training a decision tree over trajectories containing state information and individual neural response. At inference time, the decision tree takes as input a neural response, and generates decision paths that correspond to logical programs to aid interpretability. The authors then suggest 3 metrics to evaluate these decision paths (variance of neural response, mutual information with ground truth factors, and modularity). They then compare different neural architectures on Pendulum, HalfCheetah, and Visual Servoing tasks .
review: I overall found the paper a solid contribution towards interpretability for control policies, and the proposed method demonstrating how to map neural responses and trajectories to decision paths to analyze intuitive. The analysis was also pretty thorough, comparing 3 different metrics across several neural architectures for different control tasks.

I did however find the writing could be improved, particularly more explanation about the high-level purpose of the different metrics, defining notational symbols again as they are split across papers, and more clarity on whether the motivation of the work comes more from a robotics perspective (i.e. its a domain where the authors claim ground truth factor is harder to define) or policy learning perspective (i.e. the authors claim novelty for study neuron-level interpretability of policies). Because of this, it is also unclear how robot-centric this work is, and if CoRL is the right fit. Finally, I feel that a more complete work would run some kind of user study that show why these logical programs are useful, and evaluates whether they actually help human users on some kind of safety checking or decision-making task. 

Other comments:

- It seems like the set of works on skill-discovery algorithms could be relevant enough to cite, as they seek to identify sub-trajectories that align with meaningful skills for the task.  Many of them do have a similar API in that they take in trajectories as input, and the key difference is that they just aren't trying to attribute these skills to a particular neuron. 
- In the first paragraph of the top of page 2, it's a bit unclear how "Besides, directly observing the neuron response along with sensory information provided as input to the policy can be extremely inefficient and tedious for identifying behaviors and interpreting decision-making" follows from the earlier sentences --> the text here is a bit confusing on what the contrast is. 
- Could the authors explain a bit more why compactness is necessary for interpretability? Why is it important that we can attribute behavior to only one particular neuron? 
- Lines 124-129 could benefit from more clarity on the process of taking the inverse of a decision tree, as well as maybe re-introducing the notation meanings (z and P) to make it easier to follow . 
- It would be great if semantic labels could be attached to the Visuo-Servoing figures for neuron activation to better understand what we should be associating with the neural response in that example, even if they are just the authors' suggestions. 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, I think the paper is solid and added clarifications to my points above would help increase my score.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper introduces a method to "disentangle" learned policies. Using a set of world states and a trained policy decision trees are extracted as logic programs based on a specific neuron's activation.

These decision trees can then be used to generate metrics defined by the authors in order to measure and compare the interpretability of different models trained for a specific task. The paper focuses on compact neural networks but fails to properly define the scale of what they mean as compact.
review: The paper is clear and to the point at the expense of providing reasoning or motivation for the specific metrics chosen. The limitations outlined are very superficial. The problem space is interesting and applicable to the broader robotics community. Interpretability and guarantees of learned policies are highly sought after.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper touches on interpretability of policies for compact networks. Although it is unclear how this may generalize for bigger models / harder tasks, the interpretability and visualization of the decision trees attached to states is quite encouraging.

The metrics presented are interesting and overall the work is highly relevant for the community.
review_title: null

Meta-review: This paper introduces an interesting approach for measuring interpretability of robot policies. Overall reviewers found the paper to be well-written and a clear contribution to the field. As such, I am happy to recommend the paper be accepted.
Decision: Accept (Oral)