Measuring Interpretability of Neural Policies of Robots with Disentangled Representation
Authors: Tsun-Hsuan Wang
Wei Xiao
Tim Seyde
Ramin Hasani
Daniela Rus
Venue: CoRL 2023 Oral (2023)
Keywords: Interpretability
Disentangled Representation
Neural Policy

Abstract: The advancement of robots, particularly those functioning in complex human-centric environments, relies on control solutions that are driven by machine learning. Understanding how learning-based controllers make decisions is crucial since robots are mostly safety-critical systems. This urges a formal and quantitative understanding of the explanatory factors in the interpretability of robot learning. In this paper, we aim to study interpretability of compact neural policies through the lens of disentangled representation. We leverage decision trees to obtain factors of variation [1] for disentanglement in robot learning; these encapsulate skills, behaviors, or strategies toward solving tasks. To assess how well networks uncover the underlying task dynamics, we introduce interpretability metrics that measure disentanglement of learned neural dynamics from a concentration of decisions, mutual information and modularity perspective. We showcase the effectiveness of the connection between interpretability and disentanglement consistently across extensive experimental analysis.

Measuring Interpretability of Neural Policies of Robots with Disentangled Representation
Tsun-Hsuan Wang, Wei Xiao, Tim Seyde, Ramin Hasani, Daniela Rus
Massachusetts Institute of Technology (MIT)
Abstract: The advancement of robots, particularly those functioning complex human-centric environments, relies on control solutions that are driven by ma- chine learning. Understanding how learning-base based controllers make decisions is crucial since robots are often safety-critical systems. This urges formal and quantitative understanding of the explanatory factors in the interpretability of robot learning. In this paper, we aim to study interpretability of compact neu- ral policies through the lens of disentangled representation. We leverage decision trees to obtain factors of variation [1] for disentanglement robot learning; these encapsulate skills, behaviors, strategies toward solving tasks. To assess how well networks uncover the underlying task dynamies, introduce interpretability metrics that measure disentanglement of learned neural dynamics from a concen- tration of decisions, mutual information and modularity perspective. We showcase the effectiveness of the connection between interpretability and disentanglement consistently across extensive experimental analysis.
Keywords: Interpretability, Disentangled Representation, Neural Policy
Introduction
Interpretability of learning-basee based robot control is important for safety-critical applications as it af- fords human comprehension of how the system processes inputs and decides actions. general, achieving interpretability diffi- cult for learning-based robot con- trol. The robot learning models make decisions without being ex- plicitly programmed to perform the task and are often very large, thus it impossible to synthesize and explain their reasoning processes. This lack transparency, often referred to the "black box" problem, makes hard interpret the workings of learning based robot control systems. Understanding why particular decision made predicting how the system will behave in future scenarios remains a challenge, critical for physical deployments.
Compact Neural Policy for Robot Learning
Interpretability Messure
by Logic Program
by Disentanglement
Neuron Re sponse
Stowegy 1 (Srshlion)
Rehawier #1
Observation
Seratigy [Reccruer At. The Hightt
Concentration
Mutual Information Modularity
Figure I: Understand robot behaviors by extracting logic programs as factors of variation
interpretability with disentanglement.
Through the lens of representation learning, we assume that neural networks capture a set of pro- cesses that exist in the data distribution; for robots, they manifest learned skills, behaviors, or strate- gies, which are critical understand the decision-making of a policy. However, while these factors of variation [1] (e.g., color or shape representations) are actively studied in unsupervised learning for disentangled representation, in robot learning, they are less well-defined and pose unique chal- lenges due to the intertwined correspondence of neural activities with emergent behaviors unknown a priori. In the present study, we aim to (i) provide a useful definition of factors of variation for policy learning, and (ii) explore how to uncover dynamics and factors of variation quantitatively as measure interpretability in compact neural networks for closed-loop end-to-end control applica-
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

tions. In this space, an entanglement corresponding to multiple neurons responsible for an emergent behavior can obstruct the interpretation of neuron response even with a small number of neurons [2, 3, 4, 5]. To this end, the disentanglement of learned representations [6. 7, 8] in compact neural networks is essential for deriving explanations and interpretations for neural policies.
We posit that each neuron should learn abstraction (factor of variation) related to specific strat- egy required for solving sub-component of task. For example, in locomotion, one neuron may capture periodic gait, where the numerical value of the neuron response may be aligned with dif- ferent phases of the gait cycle; another neuron may account for recovery from slipping. Retrieving the abstraction learned by a neuron is, however, non-trivial. Directly observing the neuron response along with sensory information provided as input to the policy can be extremely inefficient and tedious for identifying behaviors and interpreting decision-making.
In this work, our objective is to formulate an abstraction that represents the decision-makie of parametric policy to quantify the interpretability of learned behaviors, specifically from the perspec- tive of disentangled representations. To this end, we make the following key contributions:
Provide a practical definition of factor of variation for robot learning by programmatically extract- ing decision trees from neural policies, in the form of logic program grounded by world states.
Introduce a novel set of quantitative metrics of interpretability to assess how well policies uncover task structures and their factors of variation by measuring the disentanglement of learned neural dynamics from a concentration of decisions, mutual information, and modularity perspective.
Experiment in series of end-to-end policy learning tasks that (a) showcase the effectiveness of leveraging disentanglement measure interpretabilityy, (b) demonstrate policy behaviors extracted from neural responses, (c) unveil interpretable models through the lens of disentanglement.
2 Related Work
Compact neural networks. Compact neural networks are ideal in resource constraints situations such as robotics and by nature easier interpret due to smaller number of neurons [2, 4]. Com- pact networks can be obtained by pruning [9] or compressing neural networks in end-to-end training [10]. Regularization has also been used to generate compact neural networks [11]. Compact repre- sentation of features can be learned using discriminative masking [12]. Neural Ordinary Differential Equations have also been used for learning compact network [13, 4J. In this work, we formally study the interpretability of compact neural policies through the lens of disentangled representation.
Interpretable neural networks. An interpretable neural network could be constructed from a phys- ically comprehensible perspective [14, 15]. Knowledge representation is used obtain interpretable Convolutional Neural Network [16]. An active line of research focuses on dissecting and analyzing trained neural networks in generic yet post-hoc manner [17, 18, 19, 20]. Another active line of research is to study disentangled explanatory factors in learned representation [8]. A better repre- sentation should contain information in a compact and interpretable structure [1, 21J. Unlike prior works that study disentanglement based on factors of variations such as object types, there no no- tion of ground-truth factors in robot learning and thus we propose use decision trees construct pseudo-ground-rruth factors that capture emergent behaviors of robot for interpretability analysis.
Interpretability in policy learning. Explainable AI has been recently extended to policy learn- ing like reinforcement learning [22] or for human-Al shared control settings [23]. One line of research analyzes multi-step trajectories from the perspective of options or compositional skills [24, 25, 26, 27, 28, 29]. A more fine-grained single-step alternative is to extract policies via imitation learning to interpretable models like decision tree [30]. Another line of work directly embeds the decision tree framework into the learning- beasengg-based model to strike balance between expressiveness and interpretability [31, 32, 33). Explanation of policy behaviors can also be obtained by searching for abstract state with value function [34] or feature importance [35]. In this work, we aim to offer
new perspective of disentangled representation to measure interpretability in robot policy learning.
3 Method

---

Algorithm Extract Abstraction via Decision Tree
Data Trajectories rollout from a compact neural policy Date {Oo, So, do, 20.... .......)). )5) )5
Result Interpreters of neuron response {
for i E Z do
Train decision tree Tp: from state {st) to neural response {aí).
Collect dataset Dap with neuron response {zi} } and decision paths {Ps,
Train neuron response classifier 9sz R + (P) with Dap.
Obtain decision path parser r² {P] + by tracing (Pk) in Tgt.
Construct the mapping I's r² qai.
end for
In this section, we describe how to obtain factor of variation by predicting logic programs from neuron responses that reflect the learned behavior of the policies (Section 3.1), followed by a set of quantitative measures interpretability in the lens of disentanglement (Section 3.2).
3.1 Extracting Abstraction via Decision Tree
Our goal is to formulate a logic program that represents the decision-making of parametric policy to serve as an abstraction of learned behaviors, summarized in Algorithm 1. First, we describe a decision process as a tuple {O, S, A, Pa, h). where at a time instance i, o E O is the observation, 8t is the state, ag E is the action, Pa S x A + [0, 1] is the (Markovian) transition probability from current state 8, to next state 5 under action a, and h + O is the observation model. We define a neural policy as : O + A and the response neuron i where I refers to set of neurons be interpreted. For each neuron i, aim to construct mapping that infers a logic program from neuron response, I's R + L, where L is set of logic programs grounded on environment states S. Note that f3 does not take the state as an input as underlying states may be inaccessible during robot deployment. In the following discussion, we heavily use the notation P. for the decision path associated with the i'th neuron, where the subscript refers to the dependency on state with parenthesis (like (ex)) and otherwise indexing based on the context.
{2] e RRJez,
From states to neuron responses. Decision trees non-parametric supervised learning algo- rithms for classification and regression. Throughout training, they develop set of decision rules based on thresholding one or a subset of input dimensions. The relation across rules is described by a tree structure with the root node as the starting point of the decision-making process and the leaf nodes as the predictions. The property of decision trees to convert data for decision making to a set of propositions is a natural fit for state-grounded logic programs. Given a trained neural policy m, collect set of rollout trajectories D& J.W where Tj (oo, So, ao. 20, oI,...). We first train decision tree Tg. to predict the ith neuron response from states,
0²* arg min
Σ
Lai zt), where 2 = Tg: (8t)
(1)
where La represents the underlying classification or regression criteria. The decision tree Tg: de- scribes relations between the neuron responses and the relevant states as logical expressions. During inference, starting from the root node, relevant state dimensions will be checked by the decision rule in the current node and directed to the relevant lower layer, finally arriving at of the leaf nodes and providing information to regress the neuron response. Each inference traces out route from the root node to leaf node. This route is called decision path. A decision path consists of a sequence of decision rules defined by nodes visited by the path, which combine to form a logic program,
(8) S Cn) + Behavior extracted from & via Tg:
nEP (=)))))(())
where A is the logical AND, Pi ((x) is the decision path of the tree To. that takes as inputs, gives the state dimension used in the decision rule of node n (assume each node uses one feature for notation simplicity), and c, is the threshold at node n.
From neuron responses to decision paths. So far, we recover correspondence between the neuron response ,, and the state-grounded program based on decision paths Pt P(e.); however, this is not
3

---

sufficient for deployment since the decision tree Tp. requires as input the ground-truie state and not the observable data to the policy (like 0,, 21). To address this, we find an inverse of To with neuron responses as inputs and pre-extracted decision paths as classification targets. Based on the inference process of Tgs, we can calculate the numerical range of neuron responses associated with a certain decision path P(.s) from the predicted & and then construct the pairs of 2t and P We collect another dataset Day up and train classifier 4sr to predict decision paths from neuron responses,
arg min ( (DDDD
Σ
(apl9s((1)) P(o.))
(3)
where Lup is a classification criterion. While P(s.) is state-dependent, there exists finite set of decision paths {Pi AK given the generating decision tree. We define the mapping from the decision tree to the logic program as {P) + L, which can be obtained by tracing out the path as described above. Overall, the desired mapping is readily constructed as I's = 1² o 96
3.2 Quantitative Measures of Interpretability
Programmatically extracting decision trees for constructing mapping from the neuron response to logic program offers representation that facilitates the interpretability of compact neural poli- cies. Furthermore, building on the computational aspect of our approach, we can quantify the inter- pretability of a policy with respect to several metrics through the lens of disentanglement.
A. Neuron-Response Response Variance. Given decision paths (P associated with a tree Ty. at the ith
neuron, we compute the normalized variance of the neuron response averaged across decision paths,
Var (sG,))) (E(u)P().)) P))
where Z' is normalization factor that depends the range of response of the ith neuron. The set [ulP/ (a_) = } contains all time steps that exhibit the same behavior as entailed by Pi: For example, suppose we have trajectory consisting of behaviors including walking and rumning, and that walking is depicted as P the set refers to all time steps of walking. This metric captures the concentration of the neuron response that corresponds to the same strategy represented by the logic program defined by Tg. In practice, we discretize all neuron responses N bins, compute the index of bins which value belongs, divide the index by N and compute their variance.
B. Mutual Information Gap. Inspired by [21, 8]. we integrate the notion of mutual information in our framework to extend disentanglement measures for unsupervised learning to policy learning. Specifically, while previous literature assumes known ground truth factors for disentanglement such as object types, viewing angles, etc., there is no straightforward equivalence in neural policies since the emergent behaviors or strategies are unknown priori. To this end, we propose to leverage the decision path sets to construct secudo-ground-treg -trut factors Máp (P) Note that there may be correlation across decision paths, i.e., P(P.,P,) # P(P.) for logic program of the robot moving
(P
i * j. For example, one decision path corresponding forward at high speed has correlation another decision path for moving forward low speed. This may occur because neuron of policy can learn arbitrary behaviors. However, this leads to a non-orthogonal ground-truth factor set and can be undesirable since high correlations of a neuron to multiple ground-truth factors (e.g.. /[2 Pi] and [[z are large) can result from not only entanglement of the neuron but also the correlation between factors (e.g.. T(P;; is large). Hence, this urges the need to calibrate mutual information for computing disentanglement measures. We start by adapting the Mutual Information Gap (MIG) [21] our framework:
K
1
[1[2"", (112"";Aal ;M] max ; - 122 Phol) j
(5)
K
[P]]
R=1
where H is entropy, I is interaction information that take an arbitrary number of variables (with 2 being mutual information), i" arg max; [|2;; Pk], and ky arg max, 1[2 Pi]. Intuitively,

---

Table 1: Quantitative results of classical control.
Table 2: Alignment between disen-
tanglement and explanation quality classical control.
Network Architecture
Disentanglennnt Mi-Gap1
Explanation Sizz
Cugnliee Chunks
Versacee
Modularit
Vertical
Horsontta
LSTM
GRU
0,0242 405
0.8229
0.0216
0.2764
0.2303-900
0.3008 mns
0,9355
0.9412 014
0,9096 922
4,90
4.75
500 10
2.02
1.91
1.96
1 190-24
165
165 90
Correlation
agned
Venical
Explanation Size
Honsontal
Cogntiive
OUE-KNN
0,0187 207
0. 0.3062-041
0.9326 917
4.90-36
1,91
1.30-7
Variance
.0.146
0.002
0,400
201
0.2822-13 113
009
4.20 60
1.82-33
1.50-47
MI-Gup
0427
0.505
0,44P
90*
(L3653 093
0.9551 919
3
1,30 37
Modalarity
-0.144
0.155
0,002
(a) Phase Portrait Neuron Response
Emergent Strategles from Logic Programs
[c) Decision of Neuron
True
Faloe
-113
s -1.
Static
Downward
Upright
Downward (Right]
(l+ft)
Positions
Clockoniie Angular Welacity & 2:
Figure 2: In classical control (Pendulum): (a) Phase portrait with empirically measured closed-loop dynamics and neuron response. Each arrow and colored dot the results averaged around the binned state space. (b) Emergent strategies from logic programs. (c) Decision tree extracted for command neuron in NCP.
this measures the normalized difference between the highest and the second highhsst mutual informa- tion of each decision path with individual neuron activation, i.c., how discriminative the correlation between the neuron response is with one decision path opposed to the others. For example, neuron response correlated to multiple factors of variation will have lower MIG than those to one only. The last term PP; Pke] is for calibration and captures the inherent correlation between 2¹ and Pa resulted from potentially nonzero 1 [P;; P with Pks being a proxy random variable of 2) in the ground truth factor set. We show how to compute [[2;; [
- [ Pk; Pko]] in Appendix Section C. pPx-
C. Modularity. We compute modularity scores from [36] with the same calibration term,
Exta- (Tz=) Pk] (s:
Σ
(6)
I
(K - ()-1)/(2') P
RET
where k* = arg max, [[=;; For a ideally modular representation, each neuron will have high mutual information to a single factor of variation and low mutual information with all the others. Suppose for each neuron i has the best "match" with decision path (ground-truth factor) k", non- modularity of that neuron is computed as the normalized variance of mutual information between its neuron response and all non- matched decision paths [Px heget. In practice, we discretize neuron responses into N bins to compute discrete mutual information.
Experiments
We conduct a series of experiments in various effective is disentanglement to measure the interpretability of policies? (ii) What can we extract from neural responses? (iii) What architecture is more interpretable through the lens of disentanglemen?
tasks to answer the following: (i) How
4.1
Setup
Network architecture. We construct compact neural networks for each end-to-end learning to con- trol task. For all tasks, our networks are constructed by the following priors: (i) Each baseline net- work is supplied with a perception backbone (e.g.. a convolutional neural network) (ii) We construct policies based on different compact architectures that take in feature vectors from the perception backbone and output control with comparable cell counts (instead of actual network size in memory as we assess interpretability metries down to cell-level). The perception backbone is followed by a neural controller designed by compact feed-forward and recurrent network architectures includ- ing fully-connected network (FCs), gated recurrent units (GRU) [37], and long-short term memory (LSTM) [38]. Additionally, we include advanced continuous time baselines designed by ordinary differential equations such as ODE-RNN [39], closed-form continuous-time neural models (CfCs) [40], and neural circuit policies (NCPs) [4]. We interpret the dynamics of the neurons in the last layer before the output in FCs, the nommand-neurod layer of NCPs, and the recurrent state of the rest. We then extract logic programs and measure interpretability with the proposed metrics.
5

---

Table 3: Quantitative results of locomotion.
Table 4: Alignment between disentan-
glement and explanation quality in lo- comotion.
Netwonk Architecture
Disentanglnnnnt Mi-Gap1
Explanation Sizc
Cugnliee Chunks
Varsacce
Modularity
Vertical
Horsonnta
LSTM
PCs
GRU
0.0259 402
0,0187 40>
0.0108
01823-013 m1
0.1530
0.1455-925
0,9600
0,9622 omx
0,9713
5.78
562-10
566 10
2,52-40
2.52
2.54 1X
3.92-9
394
4.0 ho
Resignal
Comelation
Vertiaal
Exglanation Size
Horizontal
Cognitive Chunks
OUE-KNN
0,0210 903
0.180 929
0.9701 987
6.00 50
2,59-13
4.16
Virianes MI-Gap
0,502
0.456
0.443
0,0234 204
01596 015
0,9628 000
594 15
04
4.0022
0422
0,504
0.481
NCP
201
0.2164
0.9791 900
394
272 14
Mosiabrity
0.170
0.150
0.173
Notation
[CMD7] Periodic Galt Sequence
Pro 2 -03
[CMD3] Failure Modes
y,, -00
Robot States
-84
0.5
AR
Boy s -0.1
Ay 2002
rra 02
dra
0g7 Opr
Gyn
Figure 3: Neural activations along gait sequence HalfCheetah [43]. We focus neurons and for illustration. Neuron exhibits periodic activation pattern that reacts to different phases of the gait cycle (left). Neuron displays peak activity in situations with the potential to destabilize gait, such as early touchdown (left) and forward flipping (right). Our approach aids in failure detection by monitoring key neurons" responses.
Evaluation. To evaluate the effectiveness of measuring interpretability through the lens of disen- tanglement, we adopt the metries proposed in [41], which studies human interpretability of decision sets [42] (a representation of explanation similar that in this work). They show human response time and subjective satisfaction are highly correlated with explanation size and cognitive chunks. Explanation size consists of vertical size, the number of cases (the number of decision paths per neu- ron), and horizontal size, the complexity of each case (the length of each decision path). Cognitive chunks refer to the presentation of newly defined concepts, which we quantify as the introduction of new symbols in the logic program. Furthermore, we measure the alignment of disentanglement quantification and the above-mentioned explanation quality metrics. We compute re-signed rank correlation by re-signing Spearman's 's rank correlation coefficient to make larger values always refer to better alignment, e.g. given higher modularity being better while lower explanation size being better, better alignment corresponds to negative correlation and we thus negate the coefficient.
4.2 Classical Control
Environment and policy learning. We use the OpenAl Gym Classical Control Pendulum task [43]. The environment has simple yet nonlinear dynamics and allows for straightforward visualization of the entire state space. The environment states include / (joint angle) and ê (joint angular velocity). 0 is in the range of म with 0 = 0 as the upright position. Á is along the clockwise direction. The control is (joint torque). The goal is to stabilize at the upright position (# 0 = 0) with limited control energy consumption (u 4). We use Proximal Policy Optimization (PPO) [44] to train the policy with early stop by reaching episode reward -500 or a maximal number of training iterations.
We run each model with different random seeds and report average results.
Quantitative analysis. Table shows that, among all models, NCP achieves the best performance in disentanglement and explanation quality (i.e., explanation size and cognitive chunks), suggesting that it is more interpretable from the perspective of both our work and [41]. Beyond alignment of the best performance, Table 2 indicates the consistency of overall ranking between disentanglement and explanation quality. We found that while variance and modularity are (partially) aligned in the best performance in Table 1, only the mutual information gap is correlated to explanation quality in the overall ranking. Another interesting finding is that CfCs have the lowest logic conflict. By empirically checking the decision trees, they construct non-trivial but ovvelapinn decision paths, thus leading to considerably fewer conflicts in logic programs across neurons.
Neuron responses and underlying behaviors. While all models learn reasonable strategies, as exemplified by focusing on the sign of and 0, we now dive deeper into understanding individual neural dynamics. To this end, we focus on NCPs they provide a lower variance from the disentan- glement perspective in their logic programs. We found different neurons roughly subdivide the state space into quadrants and focus on their respective subsets. In Figure 2, we show the interpretability

---

Table 5: Quantitative results of visual servoing.
Table 6: Alignment between disentan-
Network Architecture
Disenianglement MI-CGap
Explasation Size
Cognitive Chunks
glement and explanation quality in vi-
Variance
Modalarity
Vertical
Horizonial
sual servoing.
RCs
0.1354
0.9704
2.88
Re-signed Rank
Explanation Size
Cognitive Chunks
0.1614
1.88
1.75
Comelation
Vertical
Horizontal
LSTM
0.0172
0.1950
0,9851
4.25
2.06
2.38
ODE-RNN
0.01511
0.1588
0.9766
5.25
2.24
2.75
Variance
0.371
0.314
0314
CIC
0.0191
0.1391
0.9677
5.50
2.43
3,00
M1-Gup
0.677
0.771
0,771
NCP
0.3902
Modularity
0.257
0.314
0.314
(a) Interpretation of a single neuron
(b) Front-view image (observation) different neuron activation
Dedition of
Phase Portralt
Neuron
Neuron Activation at
Different Control Modes
16
2 600u
Stable center
Right side
Highh sitle lift oriented
Left s:0
Lattfiggt Crienee
Fabz
Lelk/hg gf Curder
Figure 4: Explanation of neural policies for end-to-end visual servoing (Image-hased Driving). (a) Phase portrait of local heading error l and lateral deviation d with empirically measured mean neuron response and closed-loop dynamics. (b) Front-view images retrieved based on neuron response.
analysis of command neuron 3 example. This neuron developed fine-grained strategies for different situations like swinging clockwise in the right left downward positions, and stabilizing positive angular velocity around the upright position, as shown in Figure 2 (b)(c). We further provide phase portrait in Figure 2 (a). The arrows indicate empirically measured closed-loop dynamics (with control from the policy) and the color coding indicates average neuron response at a specific state from evaluation. The color of the neuron response (corresponding to logic programs) and the arrows (which implicitly capture actions) highlight different fine-grained strategies in the phase portrait. Notably, this finding applies not just to NCPs but also other networks with similar functions.
4.3 Locomotion
Environment and policy learning, We consider a planar locomotion task based on OpenAl Gym's HalfCheetah environment [43]. The agent is rewarded for forward locomotion based on a simple base veloeity reward. We optimize our policies with PPO until a maximum number of episodes has been reached. For each model, we run five trials with different random seeds and report average results. Here, our objective is extend our interpretability framework control task. Specificallyy, we investigate whether our approach is capable of extracting consistent single-neuron activation patterns that align with individual phases of a periodic gait cycle.
higher-dimensional
Quantitative analysis. In Table 3, we observe consistent results with classical control that NCP achieves the best performance disentanglement and explanation quality. We further observe that the LSTM achieves a desirable low disentanglement variance comparable to NCPs. LSTMs and most networks compared to NCPs the other hand show lower Mutual Information Gap. This suggests that in these networks neuron responses are concentrated for different decision paths but not quite identifiable from a probabilistic perspective, as certain neuron activation cannot be uniquely mapped to a decision path. Besides, Table 4 shows consistent findings that the mutual information gap has the best ranking correlation with explanation quality. As opposed to classical control, the ranking correlation is overall much higher and we hypothesize that more complex tasks may yield better aligmment overall ranking between disentanglement and explanation quality.
Exploring Gait Pattern. As the state space is significantly larger than the Pendulum task, we com- plement our quantitative interpretability results with qualitative results that focus on two exemplary neurons, namely command neurons (CMDs) and 7. Figure 3 provides the extracted decision trees for CMDs (left) and (right). We find that the former displays periodic activation patterns that align very well with individual phases of regular gait. particular, leverages position readings of the back thigh joint in conjunction with fore shin velocity to coarsely differentiate between stance and flight phase. More fine-graine grained coordination of lift-off and touchdown is handled by the leftmost and rightmost branches, respectively. In addition to periodic neuron activations following regular
7

---

gait, we also observe more specialized decision trees that respond to potential safety-critich sit- uations. For example, the decision tree of CMD 3 includes two branching options that align with variations of tripping due to premature touchdown during the flight phase corresponding to a forward trip (6A) and a forward flip (6B). More quantitative analysis is shown in Appendix Section I.
4.4 End-to-end Visual Servoing
Environment and policy learning. We consider vision-based end-to-end autonomous driving where the neural policy learns steering commands for lane-following. The model takes front-view RGB images of the vehicle as input, and outputs control commands for the steering wheel and speed. We use the high-fidelity data-driven simulator VISTA [45] as our environment. We adopt a training strategy called guided policy learning that leverages VISTA to augment a real-world dataset with diverse synthetic data for robust policy learning. The training dataset contains roughly 200k image-and-conie pairs and mean squared error is used as the training objective. For evaluation, initialize the vehicle at a random position throughout the entire track and evaluate the policy for 100 frames (roughly 10s) for 100 episodes. The performance is estimated as the ratio of the length of the path traversed without a crash and the total path length. Notably, this task has two additional major distinctions: (1) policy learning based on supervised learning as opposed to reinforcement learning (2) policies take in data (images) different from states on which logic program is grounded.
Quantitative analysis. In Table 5, we have consistent findings that NCP achieves the best perfor- mance in both disentanglement and explanation quality (more precisely, comparable with the best in modularity, horizontal explanation size, and cognitive chunks). In Table 6, the mutual information gap achieves consistently the best alignment in the overall ranking. Also, the correlation are much higher than classical control and comparable with or higher than locomotion. This suggests again that more complex tasks yield better alignment between disentanglement and explanation quality.
Maneuver strategies from visual inputs. In Figure 4, we show extracted behaviors for neuron in the NCP driving policy. While the state space of driving is higher dimensional, we focus on local heading error and lateral deviation from the lane center d in the lane following task. We compute the statistics and plot neuron response and closed-loop dynamics in the d-p phase portrait. This specific neuron develops more fine-grained control for situations when the vehicle is on the right of the lane center, as shown in Figure 4(a), with images retrieved from neuron response in Figure 4(b).
5 Discussion and Limitation
We summarize all consistent findings to answer the questions asked at the beginning of Section 4. First, disentanglement is highly indicative of explanation quality in the best performance across all tasks, suggesting that, among all compact neural policies, the ones with more disentangled represen- tation are more interpretable for humans [41] (with robustness analysis across hyperparameters in Appendix Section F). In addition, compared to neuron response variance and modularity, the mutual information gap consistently has the best alignment in the overall ranking with explanation quality. Besides, there certain network architectures (NCPs) that exhibit superior performance in disen- tanglement and explanation quality consistently across experiments. Another interesting finding is that more complex tasks yield better alignment between disentanglement and explanation quality (by comparing between Table 2, 4, and 6). Finally, qualitative results showed that learned behaviors of neural policies, e.g.. gait patterns or maneuver strategies, can be extracted from neuron responses. Limitation. The proposed framework involves extracting factor of variation relevant to strategies and task structures; however, the empirical implementation only considers abstraction (i.e., logic program) in a single time step. Extensions to temporal reasoning include temporal logic [46] or using decision trees with temporal capability [33, 47]. Furthermore, the abstraction is grounded on a set of world states pre-determined by human; however, these states may not be sufficiently expres- sive to capture the learned behavior of the policy. This requires estimating the information carried between observation and grounding symbols or methods to extract the latter from the former.
Acknowledgments. This work is supported by Capgemini Engineering, the Toyota Research In- stitute (TRI). This research was also supported in part by the A12050 program at Schmidt Futures (Grant G-22-905 63172). It reflects only the opinions of its authors and not TRI or Toyota entity.
8

---

References
[1] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new per- spectives. IEEE transactions pattern analysis and machine intelligence, 35(8))1798-1888) 2013.
[2] M. Lechner, R. Hasani, M. Zimmer, T. A. Henzinger, and R. Grosu. Designing worm- inspired neural networks for interpretable robotic control. In 2019 International Conference on Robotics and Automation (ICRA), pages 87-94 94.9. IEEE, 2019.
[3] R. Hasani, M. Lechner, A. Amini, D. Rus, and R. Grosu. Liquid time-constant networks. arXiv preprint arXiv:2006..omm 04439, 2020.
[4] M. Lechner, R. Hasani, A. Amini, T. A. Henzinger, D. Rus, and R. Grosu. Neural circuit policies enabling auditable autonomy. Nature Machine Intelligence, 2(10):642-652, 2020.
[5] C. Vorbach, R. Hasani, Amini, M. Lechner, and Rus. Causal navigation by continuous- time neural networks. Advances in Neural Information Processing Systems, 34, 2021.
[6] J. Schmidhuber. Learning factorial codes by predictability minimization. Neural computation, 4(6):863-87) 1992.
[7] Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: foundations and learn- ing algorithms. The MIT Press, 2017.
[8] Locatello, S. Bauer, M. Lucic, G. Ractsch, Gelly, B. Schölkopf, and O. Bachem. Chal- lenging common assumptions in the unsupervised learning of disentangled representations, In international conference machine learning, pages 41 44144... PMLR, 2019.
[9] C. Baykal, L. Liebenwein, I. Gilitschenski, D. Feldman, and D. Rus. Sensitivity-informed provable pruning of neural networks. SIAM Journal on Mathematics of Data Science, 4(1); 26-45, 2022. doi: (0000111770003333233000 URL httth::t//att http//ii..gg///////// wrw. 1137 11....1113//001883333
[10] C. Hawkins, X. Liu, and Z. Zhang. Towards compact neural networks via end-to-end training: A bayesian tensor approach with automatic rank determination. SIAM Journal Mathematics of Data Science, 4(1):46-71, 2022. 11/21..1111.//3 137/21111114, URL httpt::////do org/ hrgg//.. 10.
1137/21M1391
[11] Oymak. Learning compact neural networks with regularization. In Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3966-3975. PMLR, 10-15 Jul 2018. URL attps://proceedinng mlr ppress/v00/yyee oomak18a...aaa. html.
[12] J. Bu, A. Daw, M. Maruf, and A. Karpatne. Learning compact representations of neural net- works using discriminative masking (dam). Advances in Neural Information Processing Sys- tems, 34, 2021.
[13] M. Torkamani, P. Wallis, S. Shankar, and A. Rooshenas. Learning compact neural networks using ordinary differential equations as activation functions. arXiv preprint arXiv:1905.07633 2019.
[14] B. A. Toms, E. A. Barnes, and Ebert- Physically interpretable neural networks for the geosciences: Applications to earth system variability. Journal of Advances in Modeling Earth Systems, 1299)=2011MMS002002 2020.
[15] R. Hasani, A. Amini, M. Lechner, Naser, R. Grosu, and D. Rus. Response characterization for auditing cell dynamics in long short-term memory networks. In 2019 International Joint Conference Neural Networks (IJCNN), pages 1-8, IEEE, 2019.

---

[16] Q. Zhang, N. Wu, and S.-C. Zhu. Interpretable convolutional neural networks. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 8827-8836, 2018.
[17] D. Bau, Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying inter- pretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541-6549, 2017.
[18) B. Zhou, D. Bau, A. Oliva, and A. Torralba. Interpreting deep visual representations via net- work dissection. IEEE transactions on pattern analysis and machine intelligence, 41(9):2131- 2145, 2018.
[19] D. Bau, J.-Y. Zhu, H. Strobelt, Zhou, B. Tenenbaum, W. Freeman, and A. Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. arXiv preprint 2018.
[20] D. Bau, J.-Y. Zhu, H. Strobelt, A. Lapedriza, B. Zhou, and A. Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 117(48):3001130007 2020.
[21] R. Chen, X. Li, R. B. Grosse, and Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
[22] A. Heuillet, F. Couthouis, and N. Díaz-Rodríguez Explainability in deep reinforcement learn- ing. Showledge-Baser Systems, 314:106685, 2021.
[23] Q. Li, Z. Peng, H. Wu, L.. Feng, and B. Zhou. Human-ai shared control via policy dissection. Advances in Neural Information Processing Systems, 35:8853-8888 2022.
[24] M. Wulfmeier, A. Abdolmaleki, R. Hafner, J. Springenberg, M. Neunert, Hertweck, T. Lampe, N. Siegel, N. Heess, and M. Riedmiller. Compositional transfer in hierarchical reinforcement learning. arXiv preprint arXiv: 1906. 1228, 2019.
[25] A. Sharma, M. Ahn, S. Levine, V. Kumar, K. Hausman, and S. Gu. Emergent real-world robotic skills via unsupervised off-policy reinforcement learning. arXiv preprint arXiv:2004.12974, 2020.
[26] V. Campos, A. Trott, C. Xiong, R. Socher, X. Giró-i Nieto, and Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pages 1317-1327. PMLR, 2020.
[27] A. Bagaria, J. K. Senthil, and G. Konidaris. Skill discovery for exploration and planning using deep skill graphs. In International Conference on Machine Learning, pages 521-531. 2021.
[28] D. Tanneberg, K. Ploeger, Rueckert, and Peters. Skid raw: Skill discovery from raw trajectories. IEEE robotics and automation letters, 6(3):469-47703,, 2021.
[29] T. Seyde, W. Schwarting, I. Gilitschenski, M. Wulfmeier, and D. Rus. Strength through diver- sity: Robust behavior learning via mixture policies. In Conference Robot Learning, pages 1154-1155. PMLR, 2022.
[30] O. Bastani, Y. Pu, and A. Solar-Lezama. Verifiable reinforcement learning via policy extrac- tion. Advances in neural information processing systems, 31, 2018.
[31] A. Silva, M. Gombolay, T. Killian, I. Jimenez, and S.-H. Son. Optimization methods for interpretable differentiable decision trees applied to reinforcement learning. In International conference artificial intelligence and statistics, pages 1855-1865. PMLR, 2020.
10

---

[32] Z. Ding, P. Hernandez-Leal, G. W. Ding, C. Li, and R. Huang. Cdt: Cascading decision trees for explainable reinforcement learning. arXiv preprint 1.00:2001.007553. 2020.
[33] A. Pace, A. Chan, and van der Schaar. Poetree: Interpretable policy learning with adaptive decision trees. arXiv preprint urXiv:2203.08055 2022.
[34] D. Amir and O. Amir. Highlights: Summarizing agent behavior to people. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 1168-117,, 2018.
[35] N. Topin and M. Veloso. Generation of policy-level explanations for reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 2514 2521, 2019.
[36] K. Ridgeway and M. C. Mozer. Learning deep disentangled embeddings with the f-statistic loss. Advances in neural information processing systems, 31, 2018.
[37] K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: 1259, 2014.
[38] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 98)) 1780, 1997.
[39] Y. Rubanova, R. T. Chen, and K. Duvenaud. Latent ordinary differential equations for irregularl-samplled time series. Advances in neural information processing systems, 32, 2019.
[40] R. Hasani, M. Lechner, A. Amini, L. Liebenwein, M. Tschaikowski, G. Teschl, and D. Rus. Closed-form continuous-depth models. arXiv preprint arXiv:2 13898, 2021.
[41] I. Lage, Chen, J. He, M. Narayanan, B. Kim, S. Gershman, and Velez. evalua- tion of the human-interpretablitty of explanation. arXiv preprint arXiv: 0000000000 2019.
[42] H. Lakkaraju, S. H. Bach, and J. Leskovec. Interpretable decision sets: A joint framework for description and prediction. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1675-1684, 2016.
[43] G. Brockman, V. Cheung, L. Pettersson, Schneider, Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv: 1606.01140 2016.
[44] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (eo:::777.06444 2017.
[45] A. Amini, T.-H. Wang, Gilitschenski, W. Schwarting, Z. Liu, S. Han, S. Karaman, and D. Rus. Vista 2.0: An open, data-driven simulator for multimodal sensing and policy leamn ing for autonomous vehicles. In 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022.
[46] A. Camacho and S. A. Mcllraith. Learning interpretable models expressed linear temporal logic. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 29, pages 621-630, 2019.
[47] L. Console, C. Picardi, and D. T. Duprè Temporal decision trees: Model-based diagnosis of dynamic systems on-board. Journal of artificial intelligence research, 19:469-55 -122 2003.
[48] R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[49] J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural net- works. In International Conference Learning Representations, 2018.

---

[50] L. Liebenwein, R. Hasani, A. Amini, and D. Rus. Sparse flows: Pruning continuous-depth models. Advances in Neural Information Processing Systems, 34, 2021.
[51] I.. Liehenwein, C. Baykal, B. Carter, D. Gifford, and D. Rus. Lost in pruning: The effects of pruning neural networks beyond test accuracy. Proceedings of Machine Learning and Systems, 1393-138, 2021.
[52] L. Li, J. Walsh, and M. L. Littman. Towards unified theory of state abstraction for mdps. In AI&M, 2006.
[53] T. D. Kulkarni, K. Narasimhan, A. Saeedi, and Tenenbaum. Hierarchical deep reinforce- ment learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[54] E. S. Spelke and K. D. Kinzler. Core knowledge. Developmental science, 10(1):89-96, 2007.
[55] T. Dean and R. Givan. Model minimization in markov decision processes. In AAAI/IAAI, pages 106-111, 1997.
[56] E. van der Pol, Kipf, F. A. Olichock, and M. Welling. Plannable approximations to mdp homomorphisms: Equivariance under actions. arXiv preprint arXiv:2002. /963, 2020.
[57] I. Higgins, D. Amos, D. Pfau, S. Racaniere, L. Matthey, D. Rezende, and A. Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv: 2018.
[58] H. Caselles-Dupré, M. Garcia Ortiz, and D. Filliat. Symmetry-based disentangled represen- tation learning requires interaction with environments. Advances in Neural Information Pro- cessing Systems, 32, 2019.
[59] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikow, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenherg, et al. generalist agent. arXiv preprint arXiv:2205.0611 2022.
[60] L. Chen, Lu, A. Rajeswaran, Lee, Grover, M. Laskin, Abbeel, A. Srinivas, and
I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 1504- 2021.
[61] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748-8763. 8763. PMLR, 2021.
[62] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-trainin with frozen image encoders and large language models. arXiv preprint arXiv:2301 2023.
12

---

A Compact Networks for Neural Policies
To obtain compact neural representations, there are three common approaches: 1) simply choose an RNN with small number of units densely wired to each other (e.g., long short-term memory, LSTM, network [38], or continuous-time network such an ordinary differential equation, ODE, based network [48, 391). 2) sparsify a large network into a smaller system (c.g., lottery ticket winners [49], or sparse flows [50]), and 3) use neural circuit policies that are given by sparse architectures with added complexity to their neural and synaptic representations but have a light-weighted network architecture [3, 4, 40].
In the first approach the number of model parameters inversely affect interpretability, i.e., interpret- ing wider and/or deeper densely wired RNNs exponentially makes the interpretation of the system harder. Sparsity has been shown to help obtain a network with 95% less parameters compared to the initial model. However, recent studies show that such levels of sparsity affect the robustness of the model, thus make more susceptible to perturbations [51]. Neural circuit policies (NCPs) [4] on the other hand have shown great promise in achieving attractive degrees of generalizabil- ity while maintaining robustness environmental perturbations. This representation learning pability is rooted in their ability to capture the true cause and effect of given task [5]. NCPs are sparse network architectures with their nodes and edges determined by liquid time-constant (LTC) concept [3]. The state of a liquid network is described by the following set of ODEs [3]: dx(()) di = ((x)),
0 f(x()) [(t), t, 0) @ A. Here, x (P)))() is the hidden state with size D. (t) is an input signal, 7(Dx1) is the fixed internal time-constant vector, ( is bias parameter, and @ is the Hadamard product. In tasks involving spatiotemporal dynamics these networks showed significant benefit over their counterparts, both in their ODE form and in their closed-form representation termed Closed-form continuous-time (CIC) models [4, 5, 40].
Interpretation of Neuron Responses. Compact neural representations promise to enable the in- terpretability of decision-maken making by focusing post-hoc analysis on limited number of neural sponses. However, having merely a lower-dimensional space for visualization is not sufficient to identify consistent behaviors or strategies acquired by a learning agent. Emergent behaviors may distribute responses across numerous neurons with a high degree of entanglement. Even for models with a small number of neurons, it can be challenging to identify and interpret the behavior corre- lated with observed response patterns. In this paper, we hypothesize that abstraction with respect to type of learned strategy within single neuron is necessary for better interpretability of neural policies. We further desire semantic grounding of the neuron response, that is, associating neuron response to human-readable representation. The representation space should be abstract enough to be undeessaddett and expressive enough to capture arbitrary types of emergent behaviors or strategies. We adopt the framework of logic programs due to their simple yet effective represen- tations of decision making processes.
B A Motivating Perspective Disentangled Representation
The underlying behaviors of neural policies involves descriptions with multiple levels of abstraction, from detailed states at every time instance to high-level strategies toward solving a task, spanning a continuum where the details can be summarized and reduced to gradually construct their concise counterparts. Among these descriptions of behaviors, right amount of abstraction should be con- cise enough for human interpretability yet being sufficiently informative of how neural policies act locally toward solving the overall task. Relevant concepts about abstraction have been explored in the context of state abstraction in Markov Decision Process (MDP) [52], hierarchical reinforcement learning [53], and developmental psychology [54]. In the following, we aim to more formally define such abstraction for interpretability of neural policies and draw connection to disentangled repre- sentations. First, we define a MDP as a tuple (S, .A Pa, R). where at time instance t, 8, E S is the state, a E A is the action, Pa : S x A x S + [0, is the transition function, R R:S x A + R is the reward function. The goal of policy learning in a MDP to find a policy я SSx x A - [0,
13

---

we would like that maximizes the expected future return (accumulated reward), The closed-loop dynamics (in deterministic setting) can then be written as
Sc++1 (a(s dz), where o, π (st)
Then, we construct an abstract MDP, with state presumably being the abstraction we are looking for interpretability, as a tuple { Pa. R) that follows similar definition to the above-mentioned regular MDP. follows the deterministic MDP homomorphism [55, 56] as follows,
Vst, S1+1 E ES,a E A Pa(st, az) - St-1 + P. A(a.)) - ((s=-))
Ast E a, E E R(s;, az) = R(Q())),
the state embedding function and A + Á is the action embedding function. The state embedding functiin can also be seen as an action-equivarian map that precisely satisfies the MDP homomorphism [56]. Next, we start to draw connection disentangled representation from one of its formalism using symmetries and group theory [57]. Informally, disentanglement refers to the level of decomposition in representation that reflects the factor of variation. For ample, one dimension of vector representations corresponds to color and the other corresponds to shape. In [57], these factor of variations are formally defined as symmetries of world state (S in our case). Given group G, binary operator a x G + G, group decomposition into a direct product of subgroups = G1 x G2 x ...., and group action action act upon, the idea is to "commute" symmetries from one set to the other X'. Suppose there group of geometries acting on the world state S via action & G
where S
G x X + with as a set which the group
to find corresponding action acting on representation '2 G Z structure of S in Z (in our case neuron response E Z). This entails the equivariance condition,
Z that reflects the symmetric
2 Es+ E 's st)
where Es+z commutes action across S and Z, and can be called G-morphism or equivariant map.
GxS
S
idg x Es+z
ES*2
G Z
2 Z
A more concrete connection of group action to MDP can be seen in the analogy of agent- environment interaction [58],
g s 8, = 8141 (st 0r)
It is worth emphasizing the distinction group action 's and regular action az: not all regular action a, exhibit symmetry, as pointed out in [58]. And the group action upon neural state z be viewed as the transition dynamics of neural policies,
9. 24 = E++1 = (Z() = T (P Ta (z+))
where To TT. Z A,T. S + Z is simply the decomposition of neural policies to explicitly extract neuron responses and π. 7 + Z is the transition function of neural states (note that this does not necessarily require recurrence structure of neural policies; instead this is more of a convenient notation here). Following the definition of [57], an agent's representation Z is disentangled with respect to the decomposition G G, G2 x...
1. There is a group action iz G Z Z.
2. The map Es.z + Z is equivariant between the group actions on S and Z.
3. There decomposition Z Z1 Zn x ... such that each Z; is fixed by the actions of all G * i and affected only by G..
For the first condition, We already define in the above. For the second condition, we show that the equivariant map can follow the definition Es+2 = Te o T., i.e. 24+1 = Es+2(8t). This follows the proof as,
g z Es+ = 99.2 Et+1 *s(2(+1) To (T (T(+11)) (To (5)411))) = Es+z(g i st)
14

---

Next, extending the formalism of disentangled representation in [57] with the above- mentioned MDP homomorphism [55], we define the equivariance condition between the regular MDP {S, A. Pa. R) and the abstract MDP {S, A, Pa. R),
s ( = Es 8t)
where Eigi commutes action across S and S, and can be defined with MDP homomorphism,
St+1 ((s++1)
Pa 0z) - ((Pa(st,.e))
9 & Q(g 's st)
Q (g s 81) 9 is Q Q
Es+s Q
Note that theoretically the state embedding function Q may not have an inverse mapping since going from to S supposed to be more abstract (and thus concise with equal or less information). However, this does not matter since we don't 't necessarily require this recipe tell us how exactly group actions in & commute to Overall, we establish the following group homomorphism across set S, S, and Z,
G S
8
S
id
Egy8
G ---.- S
idg x Estz
Es+z
G .
This connects the right amount of abstraction for interpretability discussed in the beginning, then associated with MDP homomorphism, to factor of variation in disentangled representation, which is formalized by symmetry and group theory. Disentanglement in can then be lifted to symmetries in abstract state space S. In [57], disentanglement of representation is lifted up to the symmetries in the world state space S, e.g., factor of group decomposition G; can be color of an object. However, this is sufficient describe the behavior of policies since S lacks task structure. Hence, further go from S to S with MDP homomorphism to capture the essence of solving task. The factor of group decomposition G; can then be task-related, e.g., relative pose to target object (which may be of high interest for tasks like object tracjing, and less so for tasks like locomotion). Overall, this provides a motivation to cast the problem of searching for proper description of the behavior neural policies (for interpretability) to searching for disentanglement in neuron responses. In this paper, we therefore study how to measure interpretability of compact neural policies with disentangled representation.
C Calibration Of Mutual Information
Lemma C.1. The calibration term [(2) Pu;:: in both MIG (5) and Modularity (6) metrics, for j it, without loss of generality has the following lower bound:
[[2;;Px]
2 max(0, [[2)PAs] [PP::Pe)
Lemma C.1 is necessary because to compute the calibration term, we need access to the conditional distribution of the random variable (Pkv |2'), which is normally inaccessible. Hence, we derive a lower bound for the calibrated mutual information.
Proof. In the main paper, we adapting Mutual Information Gap (MIG) [21] our framework as,
k
(I1E"';
max ;(2) [[ P Pto)) j
H[PW]
k=1
15

---

and Modularity score [36] as,
Enser ([[* - Pks (K (X-1)][2::P>-1
+1
1
E11 iEI
I
Both involve the computation of 1[=; the notation of MIG), we simplify the calibration term for j * i" as follows,
; Pa; Pki]. Without loss generality for both cases (and with
PM Pk:::Pa:PPM)
[[>; - (I[22; PA] 1[2/; Pk)P
= 1/>l:P//Pul
= P[=);Pa] H HPPw [P]] H(P , ,Pk] ] H[pps] -[[= ( H/ (Pt) + PH [2)] H(Pks |2,, PP))
Pb] - 1[Px; /Psps
max(0, 1 I[Pu;
Most steps simply follow identities of mutual information and entropy. The last step requires access to the conditional distribution of random variable (Pxa (22), which is normally inaccessible. Hence, we introduce an approximation that serves as a lower bound for the calibrated mutual information in our implementation.
D Other Quantitative Measures
Decision Path Accuracy. During deployment, we use an inverse proxy 9ø for the decision tree To. and hence we compute the approximation error by measuring the accuracy of state-grounded decision path inferred from the neuron response with 46 compared to true states,
Σ
Σ (=
1ss S cp)
(8)
T
Dat
4s (zt)
(sD,
j=g(n)
where is an indicator function, U0 (2f) is the inferred decision path with norm as number of decision rules. The condition of I Cn validates if the current state sy complies with the inferred rule defined by (which is from T. Since the discrepancy is computed at the decision rule level, it captures not only the error of the classifier model go. but also how accurately I's parses 2/-
Cross-neuron Logic Conflict. When interpreting a neural policy as a whole instead of inspecting individual neuron response, it is straightforward find the intersection across logic programs tracted from different neurons 4 reduce(Aiez ()), where reduce summarizes and reduces logic programs to more compact one. Intuitively, the neuron-wise logic program should summarize the operational domain of the strategy currently executed by the neuron, where intersection describes the domain of a joint strategy across neurons. However, the reduction intersection can be invalid there is conflict the logical formulae across neurons, e.g., a < 3 from the first neuron and 2 from the second neuron. The conflict may imply, under the same configuration of fs. that (1) the policy fails to learn compatible strategies across neurons or (2) there is error induced by the interpreter due to insufficient ambiguous connection between the logic program and the neuron response, which implicitly indicates lack of interpretability.
Experimental Results. For classical control, we verify in Table that all models achieve compara- ble performance when learning toward target -500 episode reward. For locomotion, in Table 8, most models achieve comparable task performance except for GRU and ODE-RNN being slightly worse. For end-to-end visual servoing, in Table 9, all models achieve good performance (> 0.9) except for ODE-RNN, which fails to learn good policy within maximal training iterations.
16

---

Table 7: Other quantitative results of classical control.
Network
Decision Path
Logic
Performance
Architecture
Accuracy f
Conflict !
FCs
0.3015 069
0.2104 065
488,55
GRU
0.2504
0.2832
-559.82 114.07
LSTM
0,2392 031
0.5072 103
4677.95244558
ODE-RNN
0.2980 D65
0.2506
-533,93 12..931222.47
CIC
0.2509 1338
0.1556 4b9
-489.88 (199228007.00
NCP
0.4726 114
0.2026 088
-556.64 11.75
1666.6441..7
Table 8: Other quantitative results of locomotion.
Network Architecture
Decision Path
Logic
Performance
Accuracy f
Conffict |
FCs
0.5285 U64
0.1035 011
11866004468884
GRU
0,4924-0m 054
0.1500 032
3857.2 1 1448.37
LSTM
0,5283 073
0.2155 042
4122.74
ODE-RNN
0.4959 057
0.1474-024
3472.69 1734.91
CIC
0.4841
0.1581 .031
5195.46 4292.46
NCP
0.5859 019
0.1105 020
5822.730012.77
Implementation Details
NCPs are designed by four-layer structure consisting of sensory neurons (input layer), interneu- rons, command neurons (with recurrent connections), and motor neurons (output layer). To make fair comparison, we augment all non-NCP models by a feed-forward layer, which of equivalent size to the inter-neuron layer in NCPs.
E.1 Classical Control (Pendulum)
Network Architecture. With 3-dimensional observation space and I-dimensional action space, use the following network architecture for compact neural policies.
FCs: 3 10 + 4 + fully-connected network with tanh activation.
GRU: a 3 - 10 fully-connected network with tanh activation followed by GRU with cell
size of 4, outputting a 1-dimensional action.
LSTM: + 10 fully-connected network with tanh activation followed by LSTM with hidden size of 4, outputting 1-dimensional action. Note that this effectively gives 8 cells by considering hidden and cell states.
ODE-RNN: a 3 - 10 fully-connected network with tanh activation followed by neural ODE with recurrent component both size 4, outputting a 1-dimensional action.
CfC: with backbone layer 1, backbone unit 10. backbone activation silu, hidden size 4 without gate and mixed memory, outputting 1-dimensional action.
NCP: with 3 sensory neurons, 10 interneuron, 4 command neurons, 1 motor neuron, output sensory synapses, 3 output inter-synapses, 2 recurrent command synapse, 3 motor synapses.
For all policies, we use 3 - 64 > 64 + fully-connected networks with tanh activation as value function. We interpret the layer of size for each policy.
Training details. We use PPO with the following parameters for all models. Learning rate is 0.0003. Train batch size (of epoch) is 512. Mini-batch size is 64. Number of iteration within batch is 6. Value function clip parameter is 10.0. Discount factor of the MDP is 0.95. Generalized advantage
17

---

Table 9: Other quantitative results of visual servoing.
Network
Decision Path Accuracy
Logic Conflict !
Performance 1
Architecture
FCs
0.5379
0.1354
1.0000
GRU
0.6160
0.1884
0,9210
LSTM
0,5174
0.4504
1.0000
ODE-RNN
0.5483
0.3786
0.4239
CIC
0,5549
0.2274
0,9922
NCP
0.5960
0.1067
1.0000
estimation parameter is 0.95. Initial coefficient of KL divergence is 0.2. Clip parameter is 0.3. Training halts if reaching target average episode reward 150. Maximal training steps is 1M.
Interpreter details. For the decision tree Tgi, we set minimum number of samples required to be leaf node 10% of the training data, criterion of split mean squared error with Friedman's improvement score, the maximum depth of the tree 3, complexity parameter used for minimal cost-complexity pruning as 0.003; we use scikit-learn implementation of CART (Classification and Regression Trees). For simplicity. we use another decision tree as decision path classifier Qz with maximal depth of tree as 3, minimum number of samples in a leaf node as 1% of data, complexity parameter for pruning as 0.01, criterion as Gini impurity. The state grounding of the interpreter f's is {0, f). where / joint angle and é is joint angular velocity. We use the offline data collected during the closed-loop policy evaluation for the training dataset, which consists of 100 trajectories with each having maximally 100 time steps (default in the environment).
E.2 Locomotion (HalfCheetah)
Network Architecture. With 17-dimensional observation space and 6-dimensional action space, we first use feature extractors shared architecture as a 17 - 256 fully-connected network, which then output features compact neural policies with the following architectures,
FCs: a 256 + 20 + 10 - fully-connected network with tanh activation.
GRU; 256 > fully-connected network with tanh activation followed by GRU with cell size of 10, outputting 6-dimensional action.
LSTM: 256 20 fully-connected network with tanh activation followed by LSTM with hidden size of 10, outputting a 6-dimensional action. Note that this effectively gives 20 cells by considering hidden and cell states.
ODE-RNN: 256 - 20 fully-connected network with tanh activation followed by a neural ODE with recurrent component both size 10, outputting a 6-dimensional action.
CJC: with backbone layer 1, backbone unit 20, backbone activation silu, hidden size 10 without gate and mixed memory.
NCP: with 256 sensory neurons, 20 interneuron, 10 command neurons, 6 motor neuron, 4 output sensory synapses, 5 output inter-synapses, 6 recurrent command synapse, 4 input motor synapses.
For all policies, value function. We interpret the layer of size 10 for each policy.
17 256 256 fully-connected networks with tanh activation as
Training details. We use PPO with the following parameters for all models. Learning rate is 0.0003. Train batch size (of an epoch) is 65536. Mini-batch size is 4096. Number of iteration within a batch 32. Value function coefficient is 10.0. Discount factor of the MDP is 0.99. Generalized advantage
estimation parameter is 0.95. Initial coefficient of KL divergence is 1.0. Clip parameter is 0.2. Gradient norm clip is 0.5. Training halts if reaching target average episode reward - -500. Maximal training steps is 12M.
18

---

Interpreter details. For the decision tree Tpi, we set minimum number of samples required to be at a leaf node as 10% of the training data, criterion of split as mean squared error with Friedman's improvement score, the maximum depth of the tree as 3, complexity parameter used for minimal cost-complexity pruning as 0.001; we use scikit-learn implementation of CART (Classification and Regression Trees). For simplicity, we use another decision tree as decision path classifier qøi with maximal depth of tree 3, minimum number of samples in a leaf node as 1% of data, complexity parameter for pruning as 0.01, criterion as Gini impurity. The state grounding S of the interpreter I's is (hR, On, Or,n, Os,B. F,,,, Or,r, Os,r, Or, In; hn, 1z, Or,B, Os.B, Or,B, Ör, 0s, Or,F), where ha, ha are position and velocity of z-coordinate of the front tip, 9n, OR gular velocity of the front tip, 0r,n,0T, Or,n are angle and angular velocity of the thigh in the back, Os, B, 0g, are angle and angular velocity of the shin in the back, FF, OF,B are angle and angular velocity of the foot the back, Or,r, Ôr,7 are angle and angular velocity of the thigh in the front, 0S.T, 0s.T are angle and angular velocity of the shin in the front, 0F,T, OF are angle and angular velocity of the foot in the front, žR is the velocity of x-coordinate the front tip. We use the offline data collected during the closed-loop policy evaluation for the training dataset, which consists of 100 trajectories with each having maximally 1000 time steps (default in the environment).
angle and an-
E,3 End-to-end visual servoing (Image-based Driving)
Network Architecture. With image observation space of size (200, 220,320033 3) and 2-dimensional ac- tion space, we first use feature extractors of a shared architecture as a convolutional neural network (CNN) in Table 10, which then output features to compact neural policies with the following archi- tectures.
FCs: a 1280 20 + 8 fully-comnected network with tanh activation.
GRU: a 1280 - 20 fully-connected network with tanh activation followed by GRU with cell size of 8, outputting a 2-dimensional action.
LSTM: a 1280 + 20 fully-connected network with tanh activation followed by LSTM with hidden size of 8, outputting 2-dimensional action. Note that this effectively gives 20 cells by considering hidden and cell states.
ODE-RNN: a 1280 - 20 fully-connected network with tanh activation followed by a neural ODE with recurrent component both of size 8, outputting a 2-dimensional action.
CjC: with backbone layer 1, backbone unit 20, backbone activation silu, hidden size 8 without gate and mixed memory.
NCP: with 1280 sensory neurons, 20 interneuron, command neurons, 2 motor neuron, 4 output sensory synapses, 5 output inter-synapses, recurrent command synapse, 4 input motor synapses.
Training details. Batch size is 64. Sequence size is 10. Learning rate is 0.001. Number of epochs is 10. We perform data augmentation RGB images with randomized gamma of range [0.5, 1.5], brightness of range [0.5, 1.5), contrast of range [0.7, 1.3], saturation of range [0.5, 1.5].
Interpreter details. For the decision tree To:, we set minimum number of samples required to be leaf node as 10% of the training data, criterion of split as mean squared error with Friedman's improvement score, the maximum depth of the tree 3, complexity parameter used for minimal cost-complexity pruning as 0.003; use seikit-learn implementation of CART (Classification and Regression Trees). For simplicity, we use another decision tree as decision path classifier 2øz with maximal depth of tree as 3, minimum number of samples in a leaf node as 1% of data, complexity parameter for pruning as 0.01, criterion as Gini impurity. The state grounding S of the interpreter f3s is (u, 8, d, Al, μ4, k).. where e is vehicle speed, 5 is heading, is lateral deviation from the lane center, Al is longtitudinal deviation from the lane center, 1s is local heading error with respect to the lane center, 6 is road curvature. We use the offline data collected during the closed-loop policy evaluation for the training dataset, which consists of 100 trajectories with each having maximally 100 time steps.
19

---

Layer
Hyperparameters
Conv2d
(3, 2, 2) (16, le-5)
GroupNorm2d
ELU
Dropout
0.3 (24, 36, 5, 2, 2) (16, le-5)
Conv2d
GroupNorm2d
ELU
Dropout
0.3 (36, 48, 3, 2, 1) (16, le-5)
Conv2d
GroupNorm2d
ELU
Dropout
0.3
Conv2d
(48, 64, 3, 1, 1) (16, le-5)
GroupNorm2d
ELU
Dropout
0.3 (64, 64, 3, 1, 1)
Conv2d
AdaptiveAvgPool2d
reduce height dimension
Table 10: Network architecture of CNN feature extractor for end-to-end visual servoing. Hyperparameters for Conv2d are imput channel, output channel, kernel size, stride, and padding; GroupNorm2d, they are group size and epsilon; for Dropout, is drop probability.
F Robustness Analysis
We propose study the interpretability of neural policies through decision trees and present several quantitative measures interpretability by analyzing various properties on top of neuron responses and corresponding decision trees, including Veural-Responts Variance, Mutual Information Gap, Modularity, Decision Path Accuracy, and Logic Conflict. However, the extracted decision trees may differ across different configurations. Hence, to validate the robustness of the proposed metrics to hyperparameters, we compute all metrics with different decision tree parameters in classical control environment (Pendulum). We report the averaged results with 5 random seeds in Table 11 (Neural- Response Variance), Table 12 (Mutal Information Gap), Table 13 (Modularity). Table (Decision Path Accuracy), Table 15 (Logic Conflict). Most metrics (variance, MI-gap, decision path accuracy, logic conflict) yield consistent top-1 results and agree with similar rankings among network archi- tectures, except for modularity that is slightly less robust against hyperparameters yet still consistent in the top-3 set of models. This results demonstrate the reliability of the proposed interpretability analysis for neural policies.
Table 11:
Robustness hyperparameters for Neural- Neural-Responsee Variance. The results averaged across random seeds in classical control (Pendulum).
\Variance J] Network Architecture
FCs
GRU
LSTM
ODE-RNN
CíC
NCP
0.001
0.0232
0.0304
0.0209
0.0266
0,0254
0.0207
Cost Complexity Pruning
0.003
0.0242
0.0329
0.0216
0.0287
0.0272
0.0240
0.01
0.0261
0.0371
0.0221
0.0315
0.0267
0.0305
0.01
0.0154
0.0261
0.0138
0.0193
0.0189
0.0186
Minimal Leaf Sample Ratio
0.1
0,0242
0.0329
0.0216
0.0287
0.0272
0.0240
0.2
0.0334
0.0387
0.0284
0.0354
0.0295
0.0285
20

---

Table 12:
Robustness to hyperparameters for Mutual Information Gap.
The results are averaged across 5
random seeds in classical control (Pendulum).
[Mp-Gap 11 Network Architecture
FCs
GRU
LSTM
ODE-RNN
CIC
NCP
0.001
0.0284
0.2686
0.2026
0.2891
0.2544
0,3403
Cost Complexity Pruning
0.003
0.3008
0.2764
0.2303
0.3062
0.2892
0.3653
0.01
0.3482
0.3065
0.2547
0.3142
0.3567
0.3664
0.01
0.2824
0.2632
0.2040
0.2819
0.2433
0,3456
Minimal Leaf Sample Ratio
0.1
0.3008
0.2764
0.2303
0.3062
0.2892
0.3653
0.2
0.3798
0.3387
0.2528
0.3168
0.3342
0.3429
Table 13: Robustness hyperparameters for Modularity. results are averaged across random seeds classical control (Pendulum).
[Modularity
FCs
GRU
LSTM
ODE-RNN
CTC
NCP
Network Architecture
0.001
0.9519
0,9558
0,9327
0,9485
0,9228
0,9438
Cost Complexity Pruning
0.003
0.9471
0.9550
0.9402
0,9486
0,9116
0,9551
0.01
0.9532
0.9598
0.9445
0.9487
0.8970
0.9593
0.01
0.9638
0.9702
0.9547
0,9630
0.9333
0.9651
Minimal Leaf Sample Ratio
0.1
0.9471
0.9550
0.9402
0,9486
0.9116
0.9551
0.2
0.9475
0.9372
0.9197
0.9404
0.8755
0.9301
G Counterfactual Analysis via Removal of Neurons
There exist neurons with logic programs that sensible but may have little effect task performance. For example, in NCPs (not confined to this specific architecture but just focus it for discussion), we find neuron that aligns its response purely with vehicle speed. Given the task objective is lane following without crashing, such neuron pays attention to useful (for temporal reasoning across frames) but relatively unnecessary (to the task) information. Furthermore, there are neurons that don't exhibit sufficient correlation with any of the environment state and fail to induce decision branching. In light of these observation, try to remove neurons that we suspect to have little influence the performance by inspecting their logic program. We show the results in Table 16. Removing neurons 3, 4, 7 has marginal impact task performance. Among them, neuron 3 and 4 mostly depends vehicle speed v with a small tendency to the lateral deviation d. Neuron 7 fails to split a tree.
H Interpretation Of Driving Maneuver
In Figure 4, we describe interpretations similar to classical control (for neuron in NCP). While the state space of driving is higher dimensional (5 with bicycle model for lane following), states of interest only include local heading eттоr μ and lateral deviation from the lane center d in lane following task. We compute the statistics and plot neuron response and closed-loop dynamics in the d-ju phase portrait. This specific neuron develops more fine me-igraieed control for situations when the vehicle on the right of the lane center, as shown in Figure 4(a). We further show front-view images retrieved based on neuron response Figure 4(b).
I Additional Quantitative Analysis on Locomotion Behaviors
In Figure 3 (right), we demonstrate interesting qualitative examples on discovered gaits critical for failure modes like early touchdown or forward flipping. Table 17, we conduct quantitative analysis to further justify our findings. Since Gym Half-Cheetah does not have early termination, did analysis the stepwise reward, specifically the run reward (horizontal distance incremented across the consecutive time steps). Recall in Section 3.1 paragraph "From neuron responses to decision paths", , can infer the decision path (or branch) by the range of neuron responses. We use
21

---

Table 14; Robustness to hyperparameters for Decision Path Accuracy. The results averaged across 5 random seeds classical control (Pendulum).
[Decisoon Path Accuracy 1]
GRU
LSTM
ODE-RNN
CfC
NCP
Network Architecture
0.001
0.2815
0.2415
0.2195
0.2904
0.2250
0.4294
Cost Complexity Pruning
0.003
0.3015
0.2504
0.2392
0.2980
0.2509
0,4726
0.01
0.3074
0.3330
0.3161
0.3707
0.2864
0.4390
0.01
0.2950
0.2637
0.2270
0.2574
0.2452
0,4287
Minimal Leaf Sample Ratio
0.1
0.3015
0.2504
0.2392
0.2980
0,2509
0.4726
0.2
0.3572
0.3587
0.2794
0,3322
0.2784
0.4684
Table Robustness to hyperparameters Logic Conflict. The results are averaged across random seeds
classical control (Pendulum).
LLogic Conflict !J
FCs
GRU
LSTM
ODE-RNN
CTC
NCP
Network Architecture
0.001
0,2451
0.3348
0.5240
0.2641
0.2048
0,3159
Cost Complexity Pruning
0.003
0.2104
0.2832
0,5072
0,2506
0.1556
0,2026
0.01
0.1766
0.1877
0.4325
0.1401
0.1121
0.2924
0.01
0.2672
0.4298
0.6791
0,3575
0.2654
0.2607
Minimal Leaf Sample Ratio
0.1
0.2104
0.2832
0.5072
0,2506
0.1556
0.2026
0.2
0.1796
0.1664
0.3842
0.2001
0.1089
0.1111
Table 16: Removing single neuron hased on explanation.
Remove Neuron
5
Performance f
1.00
0.969
0.29
0.03
reuutre
0.0
Progress
etror d/in
0.6
Reervee neumn
Al reulrr
Nanove mouror and
50
200
3.25
150
175
200
Progress e/m
Figure Driving profile when removing neurons according to decision tree interpretation.
this technique (e.g., s..... 6B corresponds to CMD03 ;= -0.914) retrieve all time steps that fall into the branch 6A (early touchdown) or 6B (forward flipping) and compute average reward of the next few time steps. We also compute the non-branch results to serve as reference, i.c., comparison between branch X and not branch X. For early touchdown (6A), we can observe from the quantitative results that the reward drops immediately when such a branch is "activated"; this makes sense as early touchdown brakes the robot right away, leading to smaller distance increments. For forward flipping (6B), we observe higher reward at the closer time steps, which then quickly falls off relatively much lower value; this is also reasonable as flipping motion carries the robot body forward a lot in the beginning stage yet leaves the robot body in very bad pose to move forward afterward.
22

---

Table 17: Quantitative analysis failure mode of locomotion.
Step Run Reward At Time
t+1
t+2
t+3
f+4
Branch 6A (early touchdown)
5.405
5,414
6.048
6.730
Not Branch 6A
6.235
6.233
6.031
5,813
Branch 6B (forward flipping)
7.145
6.431
5,392
4.998
Not Branch 6B
5.360
5.794
6.425
6.664
Table 18: Extension larger models with Decision Transformer [60] example.
Method
Variance
MI-Gap
Modularity
Decision Transformer
0.007 003
0.167 -68
0.981 017
Potential Extension to Larger Models
While work focuses only compact neural networks, method be extended to larger- scale models by selecting a "bottleneck" layer and extract interpretation from neurons in that layer. Take transformers as examples. To start with, at a high level, transformer-based policies are con- structed with an encoder-decoder architecture [59, 60] with inputs/outputs either being tokenized or kept to be directly mapped from continuous values to embeddings. The natural selection of the bot- tleneck then the last hidden state of the encoder, which is common way do feature extraction from large transformer-based models [61, 62]. Overall, believe there are promising extensions of our work toward larger-scale scale models.
Furthermore, in Table 18, we conduct an experiment using Decision Transformer [60] to demonstrate the future potential of our work. We apply our method to a pre-trained checkpoint that can achieve 10,000 reward in Gym Half-Cheetah. Given Decision Transformer adopts an encoder-decoder transformer-basee architecture (similar to most language models), we extract the latest time step of the "last hidden state" of the encoder (following the terminology of Huggingface- Tluggingface-Transformeee here); specifically, last hidden state refers to the last layer of the stacked attention blocks and the latest time step refers to the last time step of the input sequence, e.g., in natural language, the "robot" in the sentence "I like robot". As discussed earlier, such technique is commonly adopted to extract features from large-scale transformer-based models. So far, in Decision Transformer, we get a (3, 128) feature, where corresponds to return, action, and state prediction respectively. We take the dimension that used to predict action, eventually leading to a 128-dimensional feature vector. We apply our method this 128-dimensional feature vector and report the metrics.
Interestingly, Decision Transformer gives very good performance in neuron response variance and modularity, and slightly below average performance in MIG. Besides, we also show some samples of extracted logic programs. (Please refer to Section E.2 paragraph "Interpreter details" for detailed description of each symbol)
hR <= -1.59 A 0s.B <= -0.256
ds.B <= -2.139 A θT, > -0.995
ds.B -0.945 hg <= 0.109
Furthermore, as there are 128 neurons interpreted, which leads to much larger set of possible decision paths and hence larger-sized logic programs, it drastically increases the cognitive load of humans interpreting the programs. To remedy this issue, we try to extract the decision path set from a smaller number of neurons by performing dimension reduction and only considering the principal components. Such an approximation can surprisingly effective in practice as from the perspective of representation learning, the feature may exhibit highly-structurr distribution in the 128-dimensional space. We can empirically verify this by checking the explained variance (E.V.) of the principal components (P.C.) shown in Table 19. We can achieve over 80% of explained variance. Note that the dimension reduction is only performed
that the 10 principal components
23

---

Table 19: Explained variance of the dimension-reduueed space of Decision Transformer's features.
Principle
1
3
4
6
7
8
9
10
Components
Explained
0.247
0.179
0.133
0.109
0.044
0.028
0.025
0.022
0.017
0.013
Variance (E.
Accumulated
0.247
0.426
0.559
0.669
0.713
0.742
0.767
0.789
0.807
0.821
V.
Table 20: Results for the human study.
Method
Accuracy
Subjective Satisfaction
Non-NCP
0.603.053
0.8 0.814 021
NCP
0.648 065
0.971
at constructing the decision path set for factors of variation and we are still interpreting all 128 neurons. We will then get much smaller-sized logic programs after performing the logic reduction step (as discussed the Section D paragraph "Cross-neuron Logic Conflict"). Immediate research questions then arise here like does this dimension reduction step still produce the factors of variation with similar amounts of information to the original ones?, or how sensitive is it to the methods and hyperparameters that extract the factors of variation?, etc. These studies are extremely interesting yet go beyond this work and we leave these to future exploration.
While this additional study only provides a minimal experiment and analysis, we believe it demon- strates the potential of extending the proposed concept to larger-scale models and we will keep on exploring along this research direction in the future.
K Minimal Human Study as Validation
We design a questionnaire adapted from [41] to measure accuracy, response time, and subjective satisfaction. An example is shown in Figure We show the human subject the observation of the policy (the image below what the robot sees) and the logic programs extracted from neuron responses (the text below In the mind of the robot), and ask the subject to guess what the robot will do next (the two different angles of the steering wheels). One of the options corresponds to the actual control predicted by the policy and the other is randomly sampled with the opposite sign from the actual control to avoid ambiguity. The user can choose between the two angles or non-selected (i.e., I don't know or I am not sure). Also, we put checkbox below if the user thinks the logic program is not helping to measure subjective satisfaction. We record the answers of the subjects along with their response time.
The questionnaire consists of 144 questions, which takes roughly 20 minutes to finish. We sample 62 questions based on NCP's results and 62 questions based on non-NCP's since this specific archi- tecture has the most distinction from the others across all tasks, thus potentially easier for minimal human experiment. We collected the results from 10 subjects, shown in the below table with super- script as standard deviation. Both accuracy and subjective satisfaction are between 0 and 1, where the worst and the best. The subjective satisfaction is the rate of the subject not checking the checkbox that indicates the logic program is not helping. The results are shown in Table 20. First, can see that all accuracy are larger than 0.5 (random guess), which indicates that the logic pro- grams are indeed useful for human users to understand the decision making of robots. Besides, we see positive correlation between subjective satisfaction and accuracy, which means when the human users think the logic programs are useful, they are indeed useful
Note that, however, these results should be only viewed the evaluation and analysis from [41, 42], which performs detailed user study the human inter- pretability of the decision set (conceptually the same as logic programs used in our work). The
minimal experiment that augments
24

---

(What the robot sees)
On roact. -0.05
Relative Heading Roarl furvwdtre 0.00 Speed
7.55
(Question) on) What do think the robot
Option
Option B
unowiecee
[| information from roctio (In the mine otthe nothelpig
Subret
You colected 0ww.0000000052 recond
Figure 6: The demo of the user study questionnaire (adapted from [41]). We show the human subject the observation of the policy (the image below what the robot sees) and the logic programs extracted from neuron responses (the text below In the mind of the robot), and ask the subject to guess what the robot will do next (the two different angles of the steering wheels). One of the option corresponds the policy and the other is randomly sampled with opposite sign from the actual control avoid ambiguity. The user can choose between the two angles or unselected (i.e., don't know or am not sure). Also, we put a checkbox below if the user thinks the logic program is not helping to measure subjective satisfaction. We record the answers of the subjects along with their response time.
the actual control predicted by
more thorough and rigorous study with human subjects on interpretability robot learning should be further explored in the future research.
L Logic Program from Decision Trees
Here we show the corresponding logic program of the finite set of decision path {r(Pi)) for every interpreted neuron in all network architectures. The symbols used in the logic program follow the state grounding definition in Section E. We also briefly summarize the size of associated decision trees by computing the number of decision rules for each model (before logic program reduction and conflict checking).
In classical control (Pendulum), the extracted logic program are shown in Table 21 (FC; of size 39), Table 22 (GRU; of size 54), Table 23 (LSTM; of size 43), Table 24 (ODE-RNN; of size 40), Table 25 (CfC; of size 20), Table 26 (NCP; of size 26).
In locomotion (HalfCheetah), the extracted logic program are shown in Table 27 (FC; of size 171), Table 28 (GRU; of size 158), Table 29 (LSTM; of size 148), Table 30 (ODE-RNN; of size 156), Table 31 (CfC; of size 149), Table 32 (NCP; of size 81).
In end-to-end visual servoing (Image-based Driving), the extracted logic program are shown in Table 33 (FC; of size 92), Table 34 (GRU; of size 60), Table 35 (LSTM; of size 70), Table 36 (ODE-RNN) of size 94), Table 37 (CfC; of size 107), Table 38 (NCP; of size 66).
In logic program, "conflict" indicates there are conflict between predicates within the logic pro- gram as claborated in Section 3.2.
25

---

Neuron
Lagic Program
0: (6 <= 0.69) A (9 <= 2.18)
I: (0 > 0.69) A (0 <= 2.18)
0
(conflict)
3: (8 <= 2.41) A -2.18)
4: > 2.41)
(: <= - 1.16) (8 <= -0.34)
1: <= 1.73) A (6 -1116) A (0 <= 0.34)
2: > 1.73) A (9 <= -0.31)
3: 2.03)
A0 -0.34)
4: <== 2.62)
A (0 2.03)
5: 2.62)
FC
16 <= - 1.54) A (0 <= 11:=-110 1168)
1: <= 1.47) A (6 -1.54) A (0 <= -1.6
2
2: (6 > 1.47) A (9 <= - 1.68)
(conflict)
4:16 <= 2.48) A (0 -1.68)
5: 2.48)
(0 <= -2.76)
(conflict)
(8 < 0.05) A (0 -2.66)
0.05)
Table 21: Logic program of FC in classical control (Pendulum).
Model
Neuron
Logic Program
D: (0 <= -0.06) (conffict)
(6 0.30) A -0.06)
[6 <= 1.75)
-0.30) -0.66)
(: > 1.75) (0 -0.06)
(0 <= -2.30) A (0 <= -1227
[: <= 1.83) A -2.30) A <= 1.27)
(6 -0.37)
-1.27)
3: (6 1.83) (6 -0.37)
1.27)
4: 3.10)
(0 1.83)
5: > 3.10)
GRU
(: < -0.11) <= 0..5)
1: -0.11)
0005)
(conffict) (0 -0.05)
2
(: > 0.11) <= -2.09)
4: > 0.11) <= <= 0.41) A -2.99)
5: <= 1.61) A -0.11) 0.41)
6: > 1.61) A (0 0.41)
0: <= 22611)
1: <= 2.44) <= (0 -2.61)
> (9
-2.11)
3
3: <= 1.76) A (0 0.21)
4:18 <= 0.39) A (8 -1776 A (9 0.21)
5: (0 > 0.39) A (0 0.21)
Table 22: Logic program of GRU in classical control (Pendulum).
26

---

Model
Neuron
Logle Program
-2.16)
(conflict)
0.01)
-2.16)
(0 0.48)
0.01)
(6 <= 3.00) A (0 0.48) A (#> 0.01)
5: (0 > 3.00) A (0.0 0.01)
(6 -2.57)
I: (conffict)
2: (6 > 2.57) A (0 <= 0.35)
(: > 2.57) A (9 <= 2.03) A (0 0.35)
4: > 2.57) A > 2.03)
LSTM
( <= 4.79) A (8 <= -2.31)
1: < 4.79) A (0 <= 1.72) A (9 -2.31)
2
2: <= -3.13) A (0 1.72)
3: <= 4.79) A (0 -3.13) A 1.72)
4: 1.79)
(9 <= -232))
[:
-2.32)
(0 -3.98)
0.93)
(8 > -3.98) A (9 <= 2.04) A (0 0.93)
4: (0 3.98) A (0 2.04)
Table 23: Logic program of LSTM in classical control (Pendulum).
Neuron
Logic Program
D: (d <= 1.48) A (0 <= 0000
[: <= -1 1.48) A (9 > 0.022
(0 > -1.48)
0: (6 <= -0.08) A <= -1.45)
[: > -0.08) A (0 <= - -1.45)
(: <= 2.05) A -1.45)
(: <= 2.50) A
2.05)
4: <= -0.40) A (0 2.50)
5: <= 0.03) A (6 > -0.40) A (0 2.50)
ODE-RNN
6: > 0.03) A (9 2.50)
(6 0.56)
(6 -0.56) A
2216)
2
2: (0 > 00566 A (0 <= 2.44) A (0 2.16)
(conflitt A 2.44)
(6 0.56)
2.44)
(8 <= -2.18)
(0 0.04)
-2.18)
3
(0 <= 2.65) A (0 > 0.04)
(8 <= -0.21) A (9 2.65)
4: 0.21)
2.65)
Table 24: Logic program of ODE-RNN in classical control (Pendulum).
27

---

Model
Neuron
Logic Program
-0.03)
1: (contlict)
(conilict)
3: (0 > - 0.03)
(0 <= 0.05)
1: (conffict)
(conflict)
CTC
3: (0 > 0.05)
0: (6 <= -2002)
1: -2.02)
(0 <= -0.12)
I: 0.24) A -0.12)
2: (0 <= 2.14) A (0 0.24)
2.14)
Table 25: Logic program of CTC in classical control (Pendulum).
Model
Neuron
Logic Program
(0 <= 0.33)
0
> 0.33)
(9 <= -0.07)
(conflict)
2: (9 <== 0.27) A (9 -0.07)
> 0.27)
NCP
(9 <= 4.80) A (9 <= -1.27)
1: <= 4.80) A (0 <= 1.66) A (0 1.27)
2
< 4.80)
1.66)
3: 4.80)
(: <= -0.33)
[: <= 0.44) A (0 > -0.33)
(: > 0.44) A <= -131)
(9 > 0.44) A
1.44) A (0 -131))
(0 0.44) (0 1.44)
Table 26: Logic program of NCP in classical control (Pendulum).
28

---

Model
Lugic Program
(ē -0:22) (0z,F <= -3.26)
1: (8n -0.22) (67,p <= -3.26)
[0y,e e <= 6.46) A (8r,r > -3.26) A (0p, <= -0.50)
(Ar.F
($r,F -3.26) (0p,F -0.50)
(6r.F > 6.46) A (hi <= 0.05)
(Or.F 6.46) (hi 0.05)
-0.05) [9s.F
(fr,8
(6F,8 <= -0.05) (fg,F 0.61) (6r, <= 0.05)
-00r -11.24) (9y,8 -0.05) A <= 0.05)
3: (0r,F > -11.24) (8 -0,05) A (0r,8 <= 0.05)
(,r. 0.05) A (8T,F 0.40)
(8r, 0.05) ( 0.62) (8r
6 10r,8 <= 2.08) A (0r,8 0.05) A (0T,F > 0.62)
200r. 2.08) (0T,2 0.08) (Pr,F 0.82)
(, <= -12.45) ($r,p 0.06)
1: (8p, 12.45) A (8p,8 <= 0.06) (f <= 0.65)
2: (0p,r 12:45) (0 <= 0.08) A (0g 0.65;
2
[0r.
(0p,8 0.06) (Pr.F 0.33)
4: (,r, <= 6.33) A (fp.8 0.06) A (0r > 0.38)
(Br,8
(9r,8 0.06)
(,
(0
($r
(8F,9 <= 0.38) (95,F <= 0.17) (fI.F 0.58) 2: (0 <= 0.38) A (9, > 0.17) A (0 <= 0.07)
(8
(0y 0.97)
(8F,8 0.38)
0: 00,,, <= L.79) A (0 (0r.8 <= 0.06) (0g <= (fT <=
105.
2 (0 <= 9.97) A (9p 0.199) A (TT.8 <= 0.06)
0.19) (0r.8 0.06)
RC
4: 400 <= -0.03) (0r,8 0.06)
x: (0g -d.u3) (9g.F <0-00000
10 0.06)
(An -0.03) ($p -0.49)
(fr, 0.06)
(0r,a 1.10) (ór, <=
(fr, 0.67)
I: 1000, <= 1.10) A (0T,F -6.72) A (fr <= 0.67)
(êr.8 <= 1.10) A (fr,F > 0.67)
3: (êr,0 1.10) A (bg
-0.33) A (8p 0.29)
(00r,n 1.10) A (hn > -0.33) A (0p <= 0.29)
(Ar. 1.10)
(8 0.29)
(0r 1.10) A (óg -0.48) A (AR > 0.29)
0: 10g <= -0.83) (0r,y <= 4.63) ( <= -0.07)
(0g -0.83) (0y,g 4.83) (0 -0.07)
2 (0 -0.83) (6p.F
o.07) (88, -0.33)
(Bn -0.83) (6p,F <= -0.07)
(8g,8 -033)
4: (0 -0,07) A (0R 0.52)
5: [0 -0.07) A (9g 0.52)
(0r.F <= -0.36) (OR <= 0.33) (8s,r <= 0.56)
(0T.F -0.36) (8n
(#5,
2: 100, <= -0.36) (0g, 0.56)
(0T. 6.91) A (6T,F -0,36)
(05.8 <= 0.09)
4: (0r,F 6.91) A (9g.g <= 0.09)
(8r,r -0.36) A (fs 0.09)
a. (θ <= -0.10) (êr, <= -..1)
(8n
(6r,8
2: 100 <= 0.97) A (0,,1 -3.31)
(0 0.97) A (0s,8
4: (0r,0 0.97) A (8g <= 0:20) A (05.8 -0330
(êr.n 0.97) A (9 > 0.20) A (8s,p > 0.35)
(dv,n <= 0.63) (0g.r 3.07)
E00F <= 0.63) A (0g,p
(0g
(0 <= u.63) A (6, 3.07) A (9g > 0.46]
(#f.9
(#T.F
0.03)
(0 > 0.63) A (fr <= 5.96) ^ (bn > 0.03)
(0 8 0.63) ( F 5.96) (0T,P 0.38)
& (0 0.63) A (0T.F 5.98) (fT,F 0.38)
Table 27: Logic program of FC in locomotion (HalfCheetah).
29

---

Model
Logic Program
0:( <= 1.54) (6r,8 <= -2.33 (0p <= 0.52)
1: > 1.54) (9T,B <= -2.33) (5g <= 0.52)
2:10..2 -2.33) (8R <=0.11)
3:10r.8
(0n <= 0.52)) (On 0.111
4:(8r, <- -7.48) A (0p > 0.62)
(.
0.97)
0.52)
6:(9,.F -7.48) (OR > 0.97;
0:(8g,8 <= 10.33) [9R <= 0.50) A (0p.8 <= -0.41
1: <= 10.33) (OR <- 0.50) (9g.8 -0.41)
2: <= 10.33) (kn -0.07) [fn > 0.50)
3, <= 10.33) (hq -0.07) (0g 0.50)
400,,8 10.33)
0.(8..8
0.12)
1: 3.33) (0g <= 0;12;
2:0r,,n x= (9n 0.12) (0r, <=
3. <= 6.59) (9g 0.12) A (8T,F > 0.70)
4. 6.59) A (8g <= 0.54) (8R 0.12)
5. 6.59) (0R 0.54)
0.60g
(0r.F <= 0.17)
1: > -0.78) A (Op <= 0.68) A (8T.F (.107) 0.17)
2:
0.68)
<.
(bg <= <= 0.64) A (0r,8 <= -0.14) (0z, 0.17)
4:hk 0.64) (6r,8 -0.14) A 0.17)
5:4k 0.64) (6r,F 0.17)
(: <= 1.92) ($g,8 <= 0.02)
1:00,, 1.92) A <=0.22)
2.0g,8 (fg,F <=2.1) ( 0.0)
GRU
2. <= 6.10) ( 7.21) (95,F 0.02)
4: 6.10) (05,7 0.02)
0. <= 2.59) (fr,8 <- 0.10)
(#r.8 -0.16)
(. <= 2.59) A [#r,8 0.100) (9r,8 -0.16)
2: 2.59) (0y, <=
3:
(0T.F
(9
4.1TT 1.40) (6r,5 <= 6.43) A (9 > 0.10)
5: (conflic)) ( 0.10)
6.6r, > A
0.10)
0. <= A00 -0.12)
1: <= 0.62) A (fg 0.17) (8p,B <= -0322)
2:
(0p.n -0.12)
3: > 6.66) (0R <= <= 0.62) (0,,8 -0.12)
4.1T.8 -5.47) (9g 0.62)
5. -5.47) (8 <= 0.89)
(9p 0.62)
6 5..47) (9n 0.89)
ah <= -0.27) A (0T <= 0.20)
1: -0.25) (bg
(0r.F 0.20)
2 -0.25) (ha -0.27) (0r, <= 0.20)
lhR <=
< (0T.F 0.20)
4. 0.12) (fg 0.77) (0r,F 0.20)
s 0.12) ( 0.200
(hh <= (0p, <= -0.16
(hg <= 0.07)
1:hhg <=
-0.16) (hg 0.077
2: 0.13) A (bg > 0.07)
3
(bg
(kg
4th > A (kg <= 0.08)
5:
(kg
0. 0.51)
-0.07)
1.(9,.F <= 3.81) (6y, <= (0gr -0.07)
2: 3.81) (f <=5.1) (0g
3: (: 0.51) (0g <= 0.34)
4. -1.78) ( 0.51)
(0 8...2.. > 0.34)
5: -1.88) (fr, 0.51) (9 0.34)
Table 28: Logic program GRU locomotion (HalfCheetah).
30

---

Model
Neuron
Logic Program
0: (8n (Ps,n 0.58) (0r, -0.25)
1: <= 2.01) (#s,8 <= k.5033) (fr,P 00228
(9,,8 (.5.33)
(92, 0.53)
(: <= d.u2) A (#r,F <= -0.35)
(.5,5 <= 1.23) A (8p.F <= 0.02) A (0r,F -0.35)
2:66 > 1.23) A (9 <= 0.02) A (0r, > 0.33)
(. > 0.02) A (9r.F <= -0.07)
4:fg.FF <= 0.95) A (8p.p > 0.02) (6r,F > -0.07)
5.55 0.95) A (9r,F 0.02) ^ (0r,F -0.07)
0: <= -0.46) A (hg <= -0.07)
1:(F,BB <= -4.78) (0 -0.46) (bg -0.07)
2: -4.76) A (0F,F -0.46) A (bR <= -0.07)
<= (kg -0.07)
4: > ( (bn -0.07)
0:66, <= 0.14) A (0r, -001) A (ha <= -0.11)
1:66,
(0T -0.01) (ng <= -0.11)
2. > 0.14) A (9, <= 0.42] A (hg <= -0.11)
3
(6 0.14) (0s,8
<0.0.1
(,r,, <= 0.27) A (hg -0.11)
8.T.88 <= 0.53) A (0T, > 0.27)
A (bg -0.11)
6.T. 0.53)
-0.11)
8: (6 -0.00) (canflic))
2: <= 0.12) (0r.F <=
0.399 0.39) (0F,F -0.00) -0000)
LSTM
3: 0.12) A (fy.r <= 0.39) (0p, -0.00)
4: <= -0.51) (9 0.39)
5:0R
(0F,8
0:
(0F,F -0.05)
1:68 <= 0.07) (0r, <= 0.35) (0p,P > -0.05)
2: a.07) (9p, <= 0.35) (bR <= -0.1))
3 > 0.07) (9g.F <= 0.35) A (bg -0.18)
4: <= -2.15) (0 0.35)
5: -2.15) (0p,p 0.35)
8: 0. 1.81) (hg -1.08) (0T,F -0.14)
16 <= 1.81) A (bn > -1.08) A (5r,F <= -0.14)
26
(0r,P c0.144
3: 3:hR <= -0.47) (9T
-0.14)
4:65, <= -1.89) A (&R > -0.47) A (0 > -0.14)
5: 5. -1.89) A (hg -0.47) A (87,8 > -0.14)
0: 0: <= -0.07) A (0T, <= 0.336)
1.0g.F
(0T.F 0.36)
2 <= 4.14) (9p,F F..... >0007) à (fr <= 0.34)
3: 3 > 4.14) (9p, F -0.07) A (0r, <= 0.34)
4:6T <= -2.37) (0 -0.07) (0r, > 0.34)
5: 5. -2.37) A (0 -0.07) A (5r.8 > 0.34)
0: 0. <= 11.09) A (0r <= 1.05) A (0r <= -0.46)
1: <= 11.09) A (9r.F <= 1.05) A (0r.F > -0.46)
2: 11.09)
1.05)
3: 3. 1.05) A (9 <= -11.38)
4:00, 1.05) (0
(0R 0.16)
5: 8. 1.05) A (4T.B -11.38) (0R > 0.16)
8: (5p.5 <= -D.40) (0s.8 <= -0.33)
[9 -0.40)
-0.33)
2: 20 <= -10.21) A (hg <= -0.80) A f0 00400
(6
-0.80) (0p.F
(hR > -0.80) A (0p,8 <= -0.14) (85,F -0.40)
5: -0.80) (0p, A -0.10)
Table 29: Logic program of LSTM in locomotion (HalfCheetah).
31

---

Neuron
Lagic Program
0: 0: <= -0.27) A 10 <= -0.49)
1:g -0.27) A (0s.8 <= -0.49)
2: <= -0.00) A 10F,F <= -0.22) A (85.8 -0.49)
(h -0.00; (8r,F -0.22) (8,, -0.49)
4 > -0.00) A 105,0 > -0.49) (0r,a <= 0.22)
(: > -0.00) (0s.8 -0.49) (0r.8 0.22)
0: 0.( <= -2.91) A (8p.2 <= 0.339)
(: 2.91) (kn <= -0.21) A (0p, <= 0.399
2: > -2.91) (ha -0.211 (0p, <= -0.39)
3:10 -0.00) [0F,B -0.39)
4:15R > -9.00) A (Og <= -0.04) A (9 -0.39)
5: -0.00) (8R -0.04) (8p,B 0.330
0.(8g
(
-0.16)
1:(9R <= 0.02) A (fr,B > -0.16) A (Agg <= 0.00)
2:10 <= 0.02) A (Cr,8 -0.16) A (hn > 0.00)
4:
(conflict)
5: > 0.02) A (Ag > -0002)
0:th <= 0.56) A (0R <= 0.08) A 10 < 0.42)
1:6k 0.55)
0.08) (fr.8
2: <= 0.08) A (0 <= 0.16) A 10 0.42)
3:19 <= 0.08) A (95. 8 > 0.16) A (6r.8 0.42)
4:18% 0.08) (8
0.42)
00.05)
5:(9n 0.08) A (9r,8 <= 0.42) A (kn 0.00)
6:19R > 0.08) A (9r.8 > 0.42)
0.00r.B <= -3.63) A (6r.F <= -0.87)
1:00,,8 <= -3.53) A (6r,F > -0.87) A (hn <= 0.088
2. -3.53) (ôr,F
-0.05)
3, > 3.83) A (0g,8 <= -0.11) A (bg <= -0.09)
4
(0g,8 -0.11)
5: > -3.63) A (0T,8 <= -0.33) A (hg > -0.09)
6, > -3.83) (0r.n -0.33) (An -0.09)
(:a > 3.31) (0g,B <- 0.01) (#r.F <- 0.67)
2: > 3.31) ( > 50.01) (4r,F <= 0.57)
3: 3.311) ( 0.577)
0:(0r.B 1.22) (0s.8
1:10. > 1:22) (0s.n <= <= -0.54)
2:00,,1 -3.70) A (8s,n > -0.54) (0r,8 0.18)
(,r, <= -3.70) A (8g.8 -0.44) A (67,8 > 0.16)
4: > 3.700) (conflia)
5:00 -3.70)
-0.54)
0.(hg <= 0.02) A (0g.F -0.19) (0r,
1. <= 0.02) A (03.F <= -0.19) A (0r.8 > 0.53)
2h >
(#g,F
3: <= 0.53) A (0s <= 0.16) A (0g, -0.19)
4:10R 0.52) (0 0.18)
5: 0.63) A (8s,F -0.19)
0:(0r.8 <= -0.39) A
<= -0.37)
1:18, <= -0.39) A (8y, <= 0.04) A (0r.8 > -037))
299.
($2.B
(9T.8
3: (.T > -0.39) A (#T.B <= 0.56) A (0r.F <= 0.09)
4:10..8 -0.39) (0T.8 0.55) (0r,F 0.09)
( -0.39) A (0r,n 0.55)
0: (, <= -0.24)
1: -6.60)) (svefliit
2: 2:00. > -6.60) A (confict)
(.
-0.11)
(Br, p <= 0.32) (0r, > -0.24) (6x -0.13)
(: > 0.32) A (ER -0.11)
Table 30: Logic program of ODE-RNN in locomotion (HalfCheetah).
32

---

Model
Nearon
Logic Program
D: (6p,F <= -9.40)
1:6p -9.40) 10r.n <= -1.68) (0 0.23)
2: (0 -9.40) (0r, -1.68) (0p <= 0.23)
3: (FF.FF -9.40) A (0r,F <= -5.09) (9F,F 0.23)
4. -9.40) (#r,F -6.69) (8r.F 0.23)
0: ( 0.06) (0r,
-0.46)
1:(9
(9r,F <= -0.48)
2: <= 6.02) A (Fr.F. -0.46)
3: ( 0.02) (confiic)
4: 0.02) (Pr,P -0.46)
(.6r. <= 1.92) f <= -0.59) A (kg 0.43)
1: (6r,8 <= 1.92) ( -0.50) A (hn <= 0.43)
266,, > 1.92) A (0r, <= 5.02) (hn 0=433)
3: (6r 1.92) (0r. F 5.02) (ng 0.43)
(.g.F <= 13.53) A (ig 0.43) A (0 <= -0.01)
5. 13.53
0.43) (#F,8 -0.01)
8: (65, 13.53) A (bn 0.43)
0: 1. 8.70) A [0,,F <= 0.53) A (bg <= -0.09)
1:6T.F S.70) (0 0.53) (bg -0.09)
2: (6r, <= 5.70) A (#g.F 0.53)
3,6, 8.70) A (0n <= 0.20)
460, 8.70) (0g 0.20)
8: (0 <= -0.51) A (hn <= -0.04)
1:0T.FF <= -0.51) (&g -0.04)
2: (6r.8 <= 0.5a) A (0T.F <= 0.00) A (6r.F -u.51)
3,6,, <= 0.58) A (fr. 0.50)
4: (6r 0.58) (hn -0.44)
(0r -0.51)
(. 0.58) A (hg -0.44)
(6r,F 0.511
8: (6r, <= 2.61) A (05,F <= 0.08) A (hg <= 0.03)
1. 2.61) (9,.F 0.08) (hg
2: 2: <= 0.63)) (0r,n <= 2.61) (05.P 0.08)
3:
(6r,8 <= 2.61) (0,,F 0.08)
4: 4. 2.61) A (0r,F <= -0.13)
5. 2.61) (0r,F -0.13)
8: (. <= -0.60) A (6p,F <= -0.11)
F -0.11] (lg -0.09)
2: (6T.8 > -0.60) A (9 <= -0.11) A (hg > -0.09)
(
(0
(9T,8 -0.10)
4: (8g. <= 11.48) A (0F,F -0.11) A (9T.B -0.10)
5. 11.48) (0g,F > -0.11)
0: <= -8-388)) (hn <= 0.45) A (8r, <= 0.47)
1:66, -6.38) (hg 0.45) (6 0.47)
2: 0.45) A (0r,r <= 0.14) (0r. <= 0.47)
7
3. 0.45) (8p
(
4: 4,6r <= 0.69) A 1 0.47)
5: S: 0.69)
0: (61.8 <= -6.34) (bg <= -0.02)
(0 > -6.34) A (hR <= -0002) 2: (8r, -6.34) (conflist)
3: (6 <= 10.03) A (8s.F <= 0.50) A (li > -0.02)
(6T,
(9g,F 0.50) (ng
5: 0.50) A (hn > -0.02)
8: (6 <= 1.14) A (0s,8 <= -0.66] A (9 <= U.46)
1, 1.14) (45,8 -0.06)
($ 0.46)
2: (6 <= 1.14) A (95.P 0.46)
(6 > 1.14) (0r,F 0.56) (hR -0.10)
4: (8T.8 1.14) A (0r,F <= 0.58) A (kg -0.10)
5: (, > 1.14) (fr.F 0.56)
Table 31: Logic program of CTC in locomotion (HalfCheetab).
33

---

Model
Logic Program
(0F,a <= -0.05)
I: (0p, -0.05)
0: (8F.8 -0.04)
[00, -0.04)
0. (fr.B <= -0.29)
1: (0 n -0.29)
0: (Ar.F <= -6.91) A (hR <= -0.08)
1: (OT.F <= -6.91) A (hx > -0.08)
2: (ÔT_F -6.91) A (8r.F <= -0.13)
3: (0r,r -6.91) A (0 -0.13)
0. (Ôp.a <= -6.37) (θr.8 <= -0.36)
1: (0 -6.37) A (0r,g <= -0.36)
2: (hn <= -0.59) A [6r,a <= 0.59) A (07, > -0,36)
3: (hx -0.59) A (fr,8 <= 0.59) A (9r. > -0.36)
4: (00, <= 0.64) A (0r,a > 0.59)
5: (. 0.64) (#r.8 0.50)
NCP
0: (00, <= -0.02) A (0r,a <= 0.40)
(.0, -0.02) (0T.8 00.40)
2 (8F.B -0002)
0: (0F.F -3.61) (0F,8 -0.06)
1: (0p. -3.61) (0p, <= -0.06)
2: (hR <= 0.51) A (6p.8 -0.06] (0r.F <= -0.70)
3: (hn <= 0.51) A (0p.8 -0.06) (0r.F > -0.70)
4: (hR 0.51) A (0p.8 > -0.66)
0: (0r <= -0.27)
(conflicc)
(0s.r -8.45) A (hn 0.18) (0r, -0.27)
3: (As.r -8.45) (hg 0.18) (fr,8> -0.27]
4: (0s,r -S.15) A (9F,8 <= 0.22) (9r.8 -0.27)
5: (0s.F > -8.45) A (0r, 0.22) (fI 0.27)
0: (dv, 6.33) (dr.r <= 12.79) A (0s, n <= -0.04)
1: [ 6.33) (67,F <= 12.79) (83.B -0.04)
2: (1T.F 12.79) (05,, -0.04)
3: (0r.F 12.79)
(0F.8 <= 6.97) A (ha <= -0.12)
1: 6.97) (0p.F <= 0.15) (kn > 00122)
2: (05.8 6.97) (0p.F 0.15) (bg -0.12)
3: (0p,n 6.97) (0g.n <= -8.533
4: (θr.8 6.97) (65, -8.53)
Table 32: Logic program of NCP in locomotion (HalfCheetah).
34

---

Model
Neuron
Logle Program
[x 0.00)
7.40)
1: <= 0.00) A <= 7.71) A 7.40)
2: [x > 0.00) A <= 7.71)
[ 0.00)
0.12) 7.71)
4: [K <= 0.00) A (d A (m 7.71)
( 0.00)
7.71)
(e < 7.30)
[6 < 0.00) (d 0.19) 7.30)
2: <= 0.00)
(d 0.19) (.330
(6 0.00)
0.29) 7.30)
(6 > 0.00) A (d> 0.29) (v 7.30)
(8 <= -0.02) (s <= 0.02) A (a <= 0.01)
> 0.02)
<= 0.02) A <= 0.01)
2: [K > 0.02) A (a <= 0.01)
(p > 0.01)
0: (K <= -0.00)
0.00)
0.01)
0.07)
3
2: [K > 00.00
A [a <= 0.01)
A (d > 0.07)
[K 0.00)
(p 0.02)
(μ > 0.01)
4: (M 0.00)
A (a 0.02)
PC
(K 0.00)
0.01)
(v <= 7.66)
[K <= 0.00) A [a <= 0.01)
A (v 7.66)
<= 0.00)
0.01)
3: [K > 0.00) A [a <= -0.02)
4: > 0.00) A (a <= 0.01)
(6 0.02)
[: 0.00)
0.01)
(j 0.01) A <= 7.49)
[8 < -0.02) A (a <= <= 0.01) ( 7.49)
(6 0.02)
0.01) ( 7.49)
3: (j < 0.02) A (R 0.01)
(p > 0.02)
(K < -0.01)
[6 -0.01) A -0.01) A 0.01)
(8 > 0.01) A (x -0.01) A (μ <= 0.01)
(x -0.01)
0.01)
(ju 0.022)
[6 < 0.02) (j <= 0.01) A (a -0.02)
(8 0.02) (a <= 0.01) A (4 -0.02)
3: (y 0.01)
Table 33: Logic program of end-to-end visual servoing (Image-based Driving).
35

---

Model
Neuron
Logic Program
0: (d <= -0.06)
[: 0.11)
-0.66)
2: (d > 0.11)
(j <= 0.04)
[: <= 0.09) A (# > 0.04)
2: 0.09)
[j
2
I: (e <= 0.10) A (a 0.04)
[y > 0.10)
(e <= 5.26)
[; 0.01)
6.88) (v 5.26)
3
2: [E > 0.01) A (v <= 6.88) A (v > 5.26)
7.34)
{: > 7.34)
GRU
D: None
0: (e 5.26)
I: (u <= -0.01) A <=200 A (
[p
A
(v 5.26)
3: [j > -0.01) A 6.81) A (v> 5.266
(μ > 0.01) A ( 6.81)
0: <= -0.04)
[4 0.05]
-0.04)
7.81)
2: (8 <= 0.05) A (8 -0.44) A (8> 7.81)
<= 0.09)
0.05)
(6 0.09)
0: [x < 0.02) A (j < -0.05) A (d < 0.61)
[x <= 0.02) A (p -0.05) (d <= 0.61)
[x 0.02)
<= 0.61)
3: (8 <= 0.06) (u <= <= -0.04) A (d 0.61)
0.06)
-0.01) (d 0.61)
[: > 0.06) (d > 0.61)
Table 34: Logic program of GRU end-to-end visual servoing (Image-based Driving).
36

---

Model
Neuron
Logic Program
(x <= 0.00) A (d <= 0.07)
0.00) A 0.07)
2: (N <= 0.00) A (x 0.00)
(: > 0.00) A (v <
(N > 0.00)
7.58)
(d
7.54)
[: <= 0.64] A (d -0.08) A <=
(d <= 0,04) A
7.68) 7.54)
(: <= 0.04) A 7.68)
(d <= 0.30)
A (d 0.04)
(: 0.300
0. 0.00)
-0.11)
[: > 0.00) A <d -0.11]
<= 0,03) A (d -0.11)
<= 7.50)
2
(d <= 0.03) A P) A 7.50)
4: ( 0.29) A > 0.03)
5: 0.29)
LSTM
<x 7.25)
1: <= 0.03) A A (u (v6=7..6)
7.25)
2: 0.03) A <= 7.66) (1 7.25) 3: <= 0.00) ( > 7.66)
4: > 0.00) A (v 7.66)
0: <= 0.05) (s <= 0.00) A (d <= 0.23)
1:
0.00)
(d 0.23)
2: <= 0.05) A (d > 0.23)
36 > 0.05)
(6 <= 0.04)
5
1: 0.04)
6
None
<= -0.01)
<= 7.39)
(6 > -0.01) A (v <= 7.39)
7
(K <= 0.02) A (d <= 0.01) A/e 7.39) ( 0.02) A (d 0.01)
7.39)
(d 0.01) A (e 7.39)
Table 35: Logic program of LSTM in end-to-end visual servoing (Image-bra based Driving).
37

---

Model
Lagic Program
0: (6 <= 0.033 (μ <= 0.02)
(: 0.03)
-0.00)
2: -0.03) [j <= 0.02) A (a 0.000
(pu >
6.79)
4: > 0.02) A (o 6.79)
(ju <= -0.05)
[: <= -0.03) (μ > -0.05)
2: 00.03
0.00)
-0.05)
3: -0.03) (x 0.00) A (u -0.05)
0: <= 0.02) A (d <= 0.12) A (v <= 7.23)
[: <= 0.02) A <= 0.12) A (# 7.23)
2: (μu <= 0.00) A (d 0.12)
(μ <= 0.02) A (μ 0.00)
(d 0.12)
6.79)
5: (J > 0.02) A (0 > 6.79)
(: <= -0.04)
(conficc) (6 0.04) (d <= 0.16)
(6 0.04) (d 0.16)
0: (N <= 0.00) A (μ <= -0.00)
CIC
>
-0.00)
2: (μ <= 0.02) A [f -0.00)
A (d <= 0.40)
[j <= 0.02) A (μ -0.00) (d 0.40)
4: (: 0.02) A <= 6.87)
(ju > 0.02) A (e 6.87)
(: <= 6.41)
(ju 0,00)
7.15
6.11)
2: (μ <= 0.00) A ( ( >/0>7.113 7.15)
[pu > 0.00) A <=
6.41)
4: > 0.00) A (d> 0.51) 6.41)
0: (8 <= 0.00)
(conffict
2: 0.04)
-0.00)
7.18)
(6 <= 0.04)
(8 -0.00)
A (v > 7.18)
0.07)
0.04)
(: 0.07)
0: 0.02) (v 7.266
[: > 0.02) (v <= 6.56)
(6 -0.02)
7.20)
6.56)
(8 <= 0.00) A (v <= 7.55) A (e > 7.26) 0.00) A <= 7.50) A > 7.26)
(v 7.55)
Table 36: Logic program of ODE-RNN in end-to-end visual servoing (Image-based Driving).
38

---

Model
Neuron
Logle Program
0: (d < 0.04) A (u 1::0-m58.4 5.48)
(d <= -0.04) A (1 <= 7.46) A (v 5.48)
[n 0.02) (d -0.04)
<= 7.46)
3: (a > 0.02) A (d -0.04] <= 7.46)
0.05)
(v T.46)
(: <= 0.05) A (e > 7.84)
(8 0.05) A (# 7.46)
(: <= -0.72)
I: -0.22)
5.80)
2: <= -0.00) (d 0..22 (1 5.80)
3: -0.00)
-0.72)
(x <= 5.00)
[: <= 0.09) (v <= 6.99) A 5.00)
2
(: <= 0.09) A (e > 6.99)
36 0.00)
5.00)
4: > 0.090 A (a > 6.98)
0: (d <= -0.04)
(d < 0.04)
-0.04)
3
2: (( <= -0.11) A (d (d>> 0.04)
[:4 > -0.11) (d <d<<=00 0.92) A (d 0.04)
( -0.11) (d
0,02)
0.60) <= 7.01)
ODE-RNN
[: <= 0.02) A [d (d 0.60) A <= 7.04)
( 0,02) (d -0.43) (v <= 7.04)
3: > 0.02) (d -0.43) A <= 7.04)
3: <= <= 7.3)
T.04)
7.53)
6: (d > 0.38) ( 7.53)
(v <= 5.38)
(8 <= -0.03)
<= 0.10) ( ( 0.10) ==
(6 -0.03) (d <=
(.388.8)
(: <= -0.02) ( 0.10) (+ (
4: -0.02) (P 0.10)
5.38)
(μ 0,01)
[: <= 0.01) A (d A (d 0.18)
2: <= 0.10) (4 0.01)
0.47)
3: (.100 /
(
4. 0.47) ( 6.60)
(d>
6.60)
6. > - 0.000 A (d>00.47) (v 6.00)
(v <= 5.15)
(14 <= 0.12) A (d <= 0.08) (v 5.15)
7
2: (μ <= 0.12) A (d -0.88 (v 5.15)
3: (4 > 0.12) A (v 5.15)
Table 37: Logic program of CTC in end-to-end visual servoing (Image- basage-based Driving).
39

---

Model
Nenron
Lagic Program
(8 00.05)
[: <= 0.02) (8> -0.55) A (p <= -0.0))
(6 0.02) (6 -0.05) (p -0.01)
3: <= 0.09) A (6> 0.02) A (M <= -0.01)
(8 <= 0.09) (000002)) A (p -0.01)
(: 0.09)
(# <= 0.G5)
(: > 0.05)
(8 <= 0.02) A (μ <= -0.03)
1: > 0.02) (M -0.03)
2.
2: <= -0 0.02) (и -0.03)
(6 > -0.02) A (μ -0.03)
(u <= 7.41)
[ 7.11)
2: <= -0.02) [
3
3: 0.10) A (d -0.02) 3
4: 4 A A
<= (v (004=8.00)) 8.05) (r 7.72)
NCP
5: (.10) A 8.05)
(: <= 7.45)
[: <= 7.78)
[ 7.45)
2: <= 8.08)
A (e 7.78)
3: > 8.08)
(d <= -0.15) A (v <= 7.65)
[: -0.15)
7.65)
5
(M < 0.06) A (d <= 0.04) A (8> 0.11)
(d> 0.04)
4: ( > 0.06) A (d -0.15)
0: (8 <= 0.03) A (u <= 7.73)
(8 <= -0.03) (v > 7.73)
(8 0.03)
<= 0.07) (d <= 0.02)
(8 0.03) (M <= 0.07) A (d 0.02)
(6 0.03)
0.07)
Table 38: Logic program of NCP in end-to-end visual servoing (Image-based Driving).
40


Reviews:
[{"review_id": "lMsABeS6eyN", "reviewer": "Reviewer_75ky", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a novel way of studying interpretability of learned policies for robot learning from the perspective of disentangled representations. They aim to learn a logical program for each neuron by training a decision tree over trajectories containing state information and individual neural response. At inference time, the decision tree takes as input a neural response, and generates decision paths that correspond to logical programs to aid interpretability. The authors then suggest 3 metrics to evaluate these decision paths (variance of neural response, mutual information with ground truth factors, and modularity). They then compare different neural architectures on Pendulum, HalfCheetah, and Visual Servoing tasks .", "summary_of_recommendation": "Overall, I think the paper is solid and added clarifications to my points above would help increase my score. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "\n- the authors claim \"there is no notion of ground-truth factors in policy learning\" --> is this a property of policy learning, or the domain of robotics/control? It seems like the issue is more the latter, as surely there are simply domains where there would be a clear sense of ground truth factors for a policy? \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I overall found the paper a solid contribution towards interpretability for control policies, and the proposed method demonstrating how to map neural responses and trajectories to decision paths to analyze intuitive. The analysis was also pretty thorough, comparing 3 different metrics across several neural architectures for different control tasks.\n\nI did however find the writing could be improved, particularly more explanation about the high-level purpose of the different metrics, defining notational symbols again as they are split across papers, and more clarity on whether the motivation of the work comes more from a robotics perspective (i.e. its a domain where the authors claim ground truth factor is harder to define) or policy learning perspective (i.e. the authors claim novelty for study neuron-level interpretability of policies). Because of this, it is also unclear how robot-centric this work is, and if CoRL is the right fit. Finally, I feel that a more complete work would run some kind of user study that show why these logical programs are useful, and evaluates whether they actually help human users on some kind of safety checking or decision-making task. \n\nOther comments:\n\n- It seems like the set of works on skill-discovery algorithms could be relevant enough to cite, as they seek to identify sub-trajectories that align with meaningful skills for the task.  Many of them do have a similar API in that they take in trajectories as input, and the key difference is that they just aren't trying to attribute these skills to a particular neuron. \n- In the first paragraph of the top of page 2, it's a bit unclear how \"Besides, directly observing the neuron response along with sensory information provided as input to the policy can be extremely inefficient and tedious for identifying behaviors and interpreting decision-making\" follows from the earlier sentences --> the text here is a bit confusing on what the contrast is. \n- Could the authors explain a bit more why compactness is necessary for interpretability? Why is it important that we can attribute behavior to only one particular neuron? \n- Lines 124-129 could benefit from more clarity on the process of taking the inverse of a decision tree, as well as maybe re-introducing the notation meanings (z and P) to make it easier to follow . \n- It would be great if semantic labels could be attached to the Visuo-Servoing figures for neuron activation to better understand what we should be associating with the neural response in that example, even if they are just the authors' suggestions. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "ap3cc1PVg5Q", "reviewer": "Reviewer_tXoG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a method to \"disentangle\" learned policies. Using a set of world states and a trained policy decision trees are extracted as logic programs based on a specific neuron's activation.\n\nThese decision trees can then be used to generate metrics defined by the authors in order to measure and compare the interpretability of different models trained for a specific task. The paper focuses on compact neural networks but fails to properly define the scale of what they mean as compact.", "summary_of_recommendation": "The paper touches on interpretability of policies for compact networks. Although it is unclear how this may generalize for bigger models / harder tasks, the interpretability and visualization of the decision trees attached to states is quite encouraging.\n\nThe metrics presented are interesting and overall the work is highly relevant for the community.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "How do compact networks fit in a world where most planning and robot policies are tending to become larger?\nUp to what size models is this method practical?\nHow does this method fare as complexity of the task increases?\nHow do interpretability metrics correlate with policy performance? Better performing, but more complex policies may fare worse under interpretability metrics, but better performing at the task.\n\nBy design you need a set of states to generate your metrics and decision trees. However the interesting scenarios will usually be those for which you have no examples. How useful is this method in practice given that limitation?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is clear and to the point at the expense of providing reasoning or motivation for the specific metrics chosen. The limitations outlined are very superficial. The problem space is interesting and applicable to the broader robotics community. Interpretability and guarantees of learned policies are highly sought after.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "7niznNe_0fc", "reviewer": "Reviewer_BCVj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents a disentanglement method for compact NNs in policy learning. There is no ground truth for the factors of variation, since emergent behaviors are unknown a priori, so decision trees from observable states to specific neural responses are extracted from the policy, along with an inverse mapping predicting paths through the decision tree from neural responses. The metrics for quantifying interpretability are: Neuron-response variance, the concentration of the neuron response for a particular decision path. Mutual Information Gap, modified by an additional calibration term, the gap between the highest and second highest mutual dependencies. Modularity, 1 - the mutual information between a neuron response and all non-matched decision paths. Inverse correlation between metrics and decision tree quality metrics. Empirical results and qualitative illustrations of the interpretability of this method are shown for pendulum, locomotion, and visual servoing environments.", "summary_of_recommendation": "The paper's premise is interesting and potentially useful to robotics. The authors have integrated together an impressive body of knowledge of decision trees, logical programs, neural networks, reinforcement learning, and interpretability metrics. They have clearly conducted extensive experiments using this technique, which shows promise for future work. Future work will need to show how it can be extended to different neural network architectures, how to deal with privileged states, and clearer presentations of how the branch analysis and discovered factors of variation relate to intuitively critical states in the environment trajectory.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Can you clarify the exact reason why we need the inverse classifier?\nWhat are the specific details of the classifier training process (technique, optimizer, classification criterion, dataset size)?\nWhat are the specific details of the decision tree training (similar details to above)?\nCan you clarify what are the pre-determined grounding states of the program and how these are distinct from the states observed in the environment?\nIs the same data used for policy learning used in decision tree/classifier training?\nWhat are the error bars on the numbers cited in the results?\nDo the “failure states” for the locomotion task discovered by the decision tree actually correlate to episode terminations?\nCould you use other sources of ground truth as factors of variation, for example, a handcrafted PID controller for the pendulum task?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality: Overall, the paper is well-written, equations appear correct, and the diagrams are aesthetically pleasing and provide a vision for how this method could be used in a real-world robotics application.\n\nClarity: The paper’s explanations of the motivation for the method and the theory used are mostly clear, and most concerns are addressed by the rebuttal. The remaining concerns I have are about the amount of cognitive overload it takes to qualitatively interpret the decision tree branch analysis. I still think a video of the trajectories would make this a bit clearer, but in the meantime, the diagrams the authors produced are of good quality.\n\nOriginality: The paper claims to introduce a novel set of quantitative metrics for interpretability; however, all of the metrics appear to be taken from (cited) established works on disentanglement, decision trees, or interpretability. The paper’s most original contribution appears to be the use of extracted logical programs as a factor of variation for disentanglement in simulated control settings.\n\nSignificance: Interpretability methods like the one presented in this paper are important and relatively underexplored in robotics, where it could be highly impactful for debugging real-world issues and monitoring safety-critical applications. For future work the authors will need to address the limitations with using privileged states used in the logic programs which are not available to the policy.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "-xHydIwpV3", "reviewer": "Reviewer_Z6qk", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The goal of this paper is to develop methods for interpreting learned policies. To that end, the paper proposes learning a decision tree to model each neuron of a policy network. The interpretability is evaluate with three metric that they introduce and three metrics proposed in prior work.\n\nThe three metrics that they introduce are:\n1. The average **variance** of a finite set of decision paths across neurons\n2. The **mutual information gap** to compute the disentanglement of different decision paths among neurons\n3. The **modularity** of each neuron with respect to different decision paths\n\nThe authors measure the interpretability of 6 different architectures on classical control, locomotion, and autonomous driving tasks.", "summary_of_recommendation": "I am recommending the paper for acceptance because the authors explore an elegant idea thoroughly. Although the robotics focus is not direct (as the experiments are in sim) I think many of the members of the corl community will find the method relevant and the findings interesting.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address the weaknesses listed above. Additionally:\n- L143: What does it mean that the set contains time steps that correspond to the behavior of P^i_k? Specificall, how is \"corresponding behavior determined?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The main text and appendices are highly detailed.\n- The findings in the paper were interesting to me/sparked a lot of ideas. I imagine many readers will feel the same way. (e.g., the behavior of command neurons in locomotion experiments)\n- The idea of modeling neurons with decision trees is elegant and can be repurposed across many kinds of robotic tasks.\n\nWeaknesses:\n- The paper is motivated by interpretability but it's not clear to me that the metrics evaluate interpretability with respect to human judgments. Although three of the metrics are derived from an evaluation of human interpretability, they are ultimately re-interpreted for the paper. Adding a small interpretability experiment with human subjects would make for a much stronger claim.\n- The presentation of the metrics could be made more clear. In particular the intuition behind/motivation for some of the metrics could be made more clear. This is especially true for MIG.\n\nNits (not material to review, just recommendations for revisions):\n- Add a caption to the first figure\n- Slight grammatical errors in L35, L54\n- Defining K_i as the decision paths more explicitly somewhere would help with readability\n- The first cited work for MIG (numbered [8] in the bibliography) does not discuss mutual information. As a reader I was confused how this paper was inspirational for this section.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an interesting approach for measuring interpretability of robot policies. Overall reviewers found the paper to be well-written and a clear contribution to the field. As such, I am happy to recommend the paper be accepted.
Decision: Accept (Oral)