CDT: Cascading Decision Trees for Explainable Reinforcement Learning
Authors: Zihan Ding
Pablo Hernandez-Leal
Gavin Weiguang Ding
Changjian Li
Ruitong Huang
Venue:  (2021)
Keywords: Explainable Reinforcement Learning
Decision Tree
Matrix Factorization

Abstract: Deep Reinforcement Learning (DRL) has recently achieved significant advances in various domains. However, explaining the policy of RL agents still remains an open problem due to several factors, one being the complexity of explaining neural networks decisions. Recently, a group of works have used decision-tree-based models to learn explainable policies. Soft decision trees (SDTs) and discretized differentiable decision trees (DDTs) have been demonstrated to achieve both good performance and share the benefit of having explainable policies. In this work, we further improve the results for tree-based explainable RL in both performance and explainability. Our proposal, Cascading Decision Trees (CDTs) apply representation learning on the decision path to allow richer expressivity. Empirical results show that in both situations, where CDTs are used as policy function approximators or as imitation learners to explain black-box policies, CDTs can achieve better performances with more succinct and explainable models than SDTs. As a second contribution our study reveals limitations of explaining black-box policies via imitation learning with tree-based explainable models, due to its inherent instability.

Under review as a conference paper at ICLR 2021
CDT: CASCADING DECISION TREES FOR EXPLAIN- ABLE REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep Reinforcement Learning (DRL) has recently achieved significant advances in various domains. However, explaining the policy of RL agents still remains open problem due to several factors, one being the complexity of explaining neural networks decisions. Recently, a group of works have used models to learn explainable policies. Soft decision trees (SDTss and discretized differentiable decision trees (DDTs) have been demonstrated achieve both good performance and share the benefit of having explainable policies. this work, we further improve the results for tree-based explainable RL in both performance and explainability. Our proposal, Cascading Decision Trees (CDTs) apply representa- tion learning the decision path to allow richer expressivity. Empirical results show that in both situations, where CDTs are used as policy function approxi- mators or as imitation learners to explain black-box policies, CDT's can achieve better performances with more succinct and explainable models than SDTs. As second contribution our study reveals limitations of explaining black-box poli- cies via imitation learning with tree-based explainable models, due to its inherent instability.
INTRODUCTION
Explainable Artificial Intelligence (XAI), especially Explainable Reinforcement Learning (XRL) (Puiutta and Veith, (020) is attracting more attention recently. How interpret the action choices in reinforcement learning (RL) policies remains a critical challenge, especially as the gradu- ally increasing trend of applying RL in various domains involving transparency and safety (Cheng DRL agents neural networks
et al. 2019 Junges al. 2016). Currently, many state-of-the (NNs) as their function approximators. While NNs are considered stronger function approximators (for better performances), ., RL agents built on top of them are generally lack of interpretability (Lip) ton 2018). Indeed, interpreting the behavior of NNs themselves remains an open problem in the helld MMontavon et al. 2018 Albawi et al. 2017).
In contrast, traditional DTs (with hard decision boundaries) are usually regarded as models with readable interpretations for humans, since humans can interpret the decision making process by visualizing the decision path. However, DTss may suffer from weak expressivity and therefore low accuracy. An early approach to reduce the hardness of DT was the soft/fuzzy DT (shorten as SDT) proposed by (Súárez and Lutsko (1999). Recently, differentiable SDTs (Frosst and Hinton| 2017) have shown both improved interpretability and better function approximation, which lıe in the middle of traditional DTss and neural networks.
People have adopted differentiable DTs for interpreting RL policies in two slightly different settings: an imitation learning setting (Coppens et al., [019 Liu al. 2018), in which imitators with inter- pretable models are learned from RL agents with black-box models, or full RL setting (Silva et al.. 2019), where the policy is directly represented as an interpretable model, e.g., DT. However, the DTs in these methods only conduct partitions in raw feature spaces without representation learning that could lead to complicated combinations of partitions, possibly hindering both model interpretabil- ity and scalability. Even worse, some methods have axis-aligned partitions (univariate decision nodes) (Wu et al. 2017 Silva et al. 2019) with much lower model expressivity.
In this paper, we propose Cascading Decision Trees (CDTs) striking a balance between model interpretability and accuracy, this is, having an adequate representation learning based on interpretable models (e.g. linear models). Our experiments show that CDTs share the benefits of having a significantly smaller number of parameters (and a more compact tree structure) and better performance than related works. The experiments are conducted on RL tasks, in either imitation-learni or RI. settings. We also demonstrate that the imitation-learning approach is less reliable for interpreting the

---

Under review as a conference paper at ICLR 2021
RL policies with DTs. since the imitating DTss may be prominently different in several runs, which also leads to divergent feature importances and tree structures.
RELATED WORKS
A series of works were developed in the past two decades along the direction of differentiable DTss (Irsoy et al. 2012 Laptev and Buhmann] 2014). Recently, [Frosst and Hinton (2017) proposed to distill SDT from a neural network. Their approach was only tested on MNIST digit classification tasks. Wu et al. (017)) further proposed the tree regularization technique to favor the models with decision boundaries closer compact DTs for achieving interpretability. To further boost the prediction accuracy of tree-based models, two main extensions based on single SDT were proposed: (I) ensemble of trees, or (2) unification of NNs and DTs.
An ensemble of decision trees is common technique used for increasing accuracy or robustness of prediction, which can be incorporated in SDTs (Rota Bulo and Kontschieder] 2014 Kontschieder al. 2015 Kumar et al. 2016). giving rise to neural decision forests. Since more than one tree needs to be considered during the inference process, this might yield complications in the interpretability. A common solution is to transform the decision forests into single tree (Sagi and Rokach 2020).
As for the unification of NNs and DT's, Laptev and Buhmann (2014) propose convolutional decision trees for feature learning from images. Adaptive Neural Trees (ANTs) (Tanno et al., 2018) incorporate representation learning in decision nodes of a differentiable tree with nonlinear transtormations like convolutional neural networks (CNNs). The nonlinear transformations of an ANT, not only in routing functions on its decision nodes but also in feature spaces, guarantee the prediction performances in classification tasks on the one hand, but also hinder the potential of interpretability of such methods on the other hand. Wan et al. (2020) propose the neural-backed decision tree (NBDT) which transfers the final fully connected layer of a NN into a DT with induced hierarchies for the ease of interpretation, but shares the convolutional backbones with normal deep NNs, yielding the ttateoof-the-at performances on CIFARI0 10 and ImageNet classification tasks.
However, these advanced methods either employ multiple trees with multiplicative numbers of model parameters, or heavily incorporate deep learning models like CNNs in the DTs. Their interpretability severely hindered due to their model complexity.
To interpret an RL agent, Coppens al. (2019) propose distilling the RL policy into a differentiable DT by imitating a pre-trained policy. Similarly, Liu et al. (2018) apply an imitation learning framework but to the Q value function of the RL agent. They also propose Linear Model U-trees (LMUTs) which allow linear models in leaf nodes. (Silaa et al. (2019) propose to apply differentiable DTs directly as function approximators for either Q function or the policy in RL. They apply discretization process and a rule list tree structure to simplify the trees for improving interpretability. The VIPER method proposed by Basttani et al. (2018) also distills policy as NNs into a DT policy with theoretically verifiable capability, but for imitation learning settings and nonparametrie DT's only.
Our proposed CDT is distinguished from other main categories of methods with differentiable DTs for XRL in the following ways: (i) Compared with SDT (Frosst and Hinton| (017), partitions in CDT not only happen in original input space, but also in transtormed spaces by leveraging intermediate features. This is well documented in recent works (Kontschieder et al., 2015; Xiao 2017 Tanno al. 2018) improve model capacity, and can be further extended into hierarchical representation learning with advanced feature learning modules like CNN (Tanno et al. 2018). (ii) Compared with work by Coppens et al. (2019), space partitions are not limited to axis-aligned ones (which hinders the expressivity of trees with certain depths), but achieved with linear models of features as the routing functions. Moreover, the adopted linear models are not a restriction (but as an example) and other interpretable transformations are also allowed in our CDT method. (iii) Compared with ANTs (Tanno) et al. 2018). our CDT method unifies the decision making process based on different intermedhate teatures with single decision making tree, which follows the low-rank decomposition of large matrix with linear models. It thus greatly improves the model simplicity for achieving interpretability. About model simplicity and interpretability in DTs, see our motivating example in Appendix A
3 METHOD
3.1 SOFT DECISION TREE (SDT)
A SDT is a differentiable DT with probabilistic decision boundary at each node. Considering we have a DT of depth D, each node in the SDT can be represented as weight vector (with the bias as
2

---

Under review as a conference paper at ICLR 2021
an additional dimension) w/ where i and indicate the index of the layer and the index of the node The corresponding node represented as nu. where
in that layer respectively, as shown in Fig. [ 2 + j uniquely indices the node.
6-th layer jeth nade
we
a()
1 a())
(4 1)-th layer 2j-th node
(2) 1)-th node
Figure 1: A SDT node. (-) is the sigmoid function with function values on decision nodes as input.
The decision path for single instance be represented set of nodes P N, where / is the set for all nodes on the tree. We have p = arg max [u} I P [3/21=1 where p |//21=5 is the probability of going from node 22/-2+ to T22-1+j- The {u} indicates that the arg max taken over a set of nodes rather than a single one. Note that p. 1 will always be I for a hard DT (Safavian and Landgrebe 199T). Therefore the path probability
specific node nh is: pa -p0=1 II = Pi 13 1372 ut' P. In the following, we name all DT's using probabilistic decision path as SDT-based methods, shorten as SDT.
Silva et al. (019) propose to discretize the learned differentiable SDTs into univariate DT's for improving interpretability. Specifically, for decision node with a (k + 1) 1-dimensional vector w (the first dimension wn is the bias term), the discretization process (i) selects the index of largest weight dimension as k* = arg max, Wg and (ii) divides wi by w,., to construct a univariate hard DT. Without further description, the default discretization process in our experiments for both SDTs and CDTs also follows this manner. The SDTs are therefore the same DDTs in Silva et al. 2019)
3.2 CASCADING DECISION TREE (CDT)
CDT an extension based on SDT, allowing it to have the capability of representation learning well as decision making in transformed spaces. In simple CDT architecture as shown on the left of Fig. 2 a feature learning tree cascaded with decision making tree 'D. In tree F, each decision node 18 a simple function of raw feature vector x given learnable parameters w: <>æ;; w), while each leaf of it is feature representation function: f [(x; w) parameterized by w. In tree D. each decision node is simple function of learned features f rather than raw features æ given learnable parameters w': (f;;; w'), The output distribution of D is another parameterized function p(-; wO) independent of either x or f. For simplicity and interpretability. all functions db, f and v are linear functions in our examples, but they are free to be extended with other interpretable models.
Methods. We propose
Specifically, we provide detailed mathematical relationships based on linear functions as follows. For an environmen with input state vector æ and output discrete action dimension O, suppose that our CDT has intermediate features of dimension K (not the number of leaf nodes on F, but for each leaf node), we have the probability of going the left/right path on the u-th node on F::
Pu
Go Left (w0 x).
Pu
Go Right
1
Ps
Go Left
(1)
which is the same as in SDTs. Then we have the linear feature representation function for each leaf node on F, which transforms the basis of the representation space with:
fa क.. xz, k = 0, 1,..,, K -
(2)
which gives the K K-dimensional intermediate feature vector f for each possible path. Due to the
symmetry in all internal layers within tree, all internal nodes satisfy the formulas in Eq. 2 In
tree D, it is also SDT but with raw input replaced by learned representations for each node n' in D:
Go Left P
(wi f).
Go Right P
p.
Leit
(3)
A motivation of CDT method is the duplicative structures in the heuristic solution of LunarLander-v2. as discussed in Appendix(B B
3

---

Under review as a conference paper at ICLR 2021
Raw Inpur
F
Raw Iupu:
Raw Input
x
o( w)
f(x; ib)
D
#ff; w')
D
Output Distribution
p
Output Distribution
(a)
(b)
Output Distribattin
Figure 2: CDT methods. Left: simple CDT architecture, consisting of feature learning tree F and decision making tree D; Right: two possible types of hierarchical CDT architectures, where (a) is an example architecture with hierarchical representation learning using three cascading F before one D, and (b) an example architecture with two F in parallel, potentially with different dimensions of as inputs.
Finally, the output distribution is Teature-independent which gives the probability mass values across output dimension O for each leaf of D as:
exp( (')
PM
W 00,,,,,... -......000 0
(4)
Ex kk = exp w(p.))
Suppose we have a CDT of depth N1 for and depth N₂ for D. the probability of going from root of either F or D to u-th leaf node on each sub-tree both satisfies previous derivation in SDTs:
p" = - 1'/2 PF
-1-1- u' E P, where P is the set of nodes on path. Therefore the overall path probability of starting from the root of F to u1 u1-th leaf node of J and then ug-th leaf node of D is:
P pu p
(5)
Each leaf of the feature learning tree represents one possible assignment for intermediate feature values, while they share the subsequent decision making tree. During the inference process, we simply take the leaf on F or D with the largest probability to assign values for intermediate features (in F) or derive output probability (in D), which may sacrifice little accuracy but increase interpretability. The detailed architecture of CDT with relationships among variables plotted figures Appendix
Model Simplicity
We analyze the simplicity of CDT compared with SDT in terms of the numbers of learnable parameters in the model. The reason for doing this is that in order to increase the interpretability. we need to simplify the tree structure reduce the number of parameters including weights and bias the tree.
We can analyze the model simplicity CDT against normal SDT with linear functions in matrix decomposition perspective. Suppose we need a total of M multivariate decision nodes in the R- dimensional raw input space X to successfully partition the space for which can be written as a matrix W M CDT tries achieve the same partitions through learning
prediction,
a transformation matrix T TRXR: X + F' for all leaf nodes in F and a partition matrix w
Mx K for
all internal nodes in D in the K- -imieensional feature space F, such that:
W = w f W/Tr Tr
(6)
W W
Therefore the number of model parameters be learned with CDT is reduced by M x R (M x K + K R) compared against a standard SDT of the same total depth, and is positive value as long as K M+R
MxR while keeping the model expressivity.
detailed quantitative analysis of model parameters for CDT and SDT is provided in Appendix D
4

---

Under review as a conference paper at ICLR 2021
Hierarchical CDT
From above, a simple CDT architecture in Fig. 2 with a single feature learning model F and single decision making model D can achieve intermedhate feature learning with a significant reduction in model complexity compared with traditional SDT. However, sometimes the intermediate features learned with F. may be unsatisfying for capturing complex structures in advanced tasks, therefore we further extend the simple CDT architecture into more hierarchical ones. As shown on the right side in Fig. 2 two potential types of hierarchical CDT are displayed: (a) a hierarchical feature abstraction module with three feature learning models { Fa in cascading manner before inputting to the decision module D; (b) parallel feature extraction module with two feature learning models F2) before concatenating all learned features into D.
One needs to bear in mind that whenever the model structures are complicating, the interpretability of the model decreases due to the loss of simplicity. Therefore we did not apply the hierarchical CDTs in our experiments for maintaining interpretability.. However, the hierarchical structure is one of the most preferred ways to keep simplicity as much possible if trying increase the model capacity and prediction accuracy, so it can be applied when necessary.
4 EXPERIMENTS
We compare CDT and SDT two settings for interpreting RL agents: (1) the imitation learning setting, whereas the RL agent with black-box model (e.g. neural network) to interpret first generates a state-action dataset for imitators to learn from, and the interpretation derived on the imitators; (2) the full RL setting, whereas the RL agent is directly trained with the policy represented with interpretable models like CDTs or SDTs, such that the interpretation can be derived by directly spying into those models. The environments OpenAl Gym (Brockman sections, where d1 is the depth of feature learning tree F and d2 is the depth of decision making tree D. Each setting is trained for five runs in imitation learning and three runs in RI..
CartPole-vI, LunarLander-v2 and MountainCar-vo in al.) 2016). The depth of CDT is represented as "d, + d2 " in the following
Both the fidelity and stability of mimie models reflect the reliability of them interpretable models. Fidelity is the accuracy of the mimic model, w.r.t. the original model. is an estimation of similarity between the mimic model and the original one in terms of prediction results. However, fidelity is not sufficient for reliable interpretations. An unstable family of mimic models will lead to inconsistent explanations of original black-box models. The stability of the mimic model is deeper excavation into the model itself and comparisons among several runs. Previous research (Bastani et al.) 2017) has investigated the fidelity and stability of decision trees as mimic models, where the stability is estimated with the fraction of equivalent nodes in different random decision trees trained under the same settings. In our experiments, the stability analysis is conducted via comparing tree weights of different instances in imitation learning settings.
4.1 IMITATION LEARNING
The datasets for imitation learning are generated with heuristic agents for environ- ments CartPole-vl and LunarLander-v2, containing 10000 episodes of state-action data for each environments. See Appendix
Performance.
E for other training details. The results are provided in Table T and 2
Tree Type
Depth
Discretized
Accuracy
Episode Reward
Parameters
2
X
94.11:0.11
500.0:10.0 0
23
49.7++0.02
39.9*7.6
14
SDD
3
5000 +0000
10.
500.0.00.0
42.3+7.3
30
4
F4.0
S0.1 10.1
SDRLUEEO.
40.4
107
62
X
95.4-11.1
500.0+10.0
38
142
F only
34.4
500.0.0.00
35
27 only
84.1
500.0+0.0
35
3 D
K3NE2.6
297.828.4
95.6450.1
500.0
54
CDT (ours)
2+1
F only
only
92.7+-04
48.1
500.0=0.0
SI0UUE0.0
45
F +
89.020.4
500.0.0..0
X
V6.610.9
500.0.000
64
only
91.6+1.3
500.0-00.0
2+2
D only
82.1337
4948119.8
J D
81.911.8
488.8431.4
52
Table 1: Comparison of CDT and SDT with imitation-learning settings on CartPole-vl.
5

---

Under review as a conference paper at ICLR 2021
CDTs perform consistently better than SDTs before and after discretization process in terms of prediction accuracy, with different depths of the tree. Additionally, for providing similarly accurate model, CDT method always has a much smaller number of parameters compared with SDT, which improves its interpretability as shown in later sections. However, although better than SDTs, CDTs also suffer from degradation in performance after discretization, which could lead to unstable and unexpected models. We claim that this is a general drawback for tree-based methods with soft decision boundaries in XRL with mmitationlearning settings, which is further studied in the following.
Free Type
Depih
Discretized
Accuracy (3)
Episode Reward
Parameters
85.410.4
58.241246.1
199
14.8
=1371=1221.9
5
87.610.3
191331155.5
407
SDT
51.64-4.6
93.7+102.
190
6
50.233.9
1724=1220 122.0
11142101.4
382
7
154
138.8
1055
62.742.8
233.4262.4
766
X
68.241.66
107.4470 190.7
116
2+2
J only
-22.922377
95
27 only FF
68 10.3
301.6=1188 13368
113
64.4112.1
2229.=2256.
92
88.311
168 18.51166 169.0
144
2+3
F only
only
40.71119
70.242.3
3106.3
139.2
123
CDT (ours)
J +
35.991.5
1302211553
116
V..E1177
1993 T28.7
216
3+2
only
12.248.3
2 only
18.122.5
150 148.
219
F D
64.6-24.
173.6
160
904E1.2
173.0:1224
244
F only
72.0
-55.3 1178.6
195
3+3
only
58.718.6
237
J +
46.845566
-210.3 17:3-1225 121.9
188
Table 2: Comparison of CDT and SDT with imitation-learning settings on LumarLander-v2.
Stability. To investigate the stability of imitation learners for interpreting the original agents, we measure the normalized weight vectors from different -mitation-learning trees. For SDTs, the weight vectors are the linear weights on inner nodes, while for CDTs {w,, w' are considered. Through the experiments, we would like show how unstable the imitators {L} are. We have a tree agent X E (L', H, R). where L' is another imitator tree agent trained under the same setting, R is a random tree agent, and H is heuristic tree agent (used for generating the training dataset). The distances of tree weights between two agents L, X are calculated with the following formula:
M
Σ X(L,X) = 2N n=1
min 1.2.....M
1m I,1111 + 2M
Σ m==
min 12.,.....,N
am - -n1l1
(8)
while D(L, X) are averaged over all possible Ls and Xs with the same setting. Since we have the heuristic agent for LunarLander-12 environment and we transform the heuristic agent into multivariate DT agent, we get the decision boundaries of the tree on all its nodes. So we also compare the differences of decision boundaries in heuristic tree agent H and those of the learned tree agent L. But we do not have the official heuristic agent for CartPole-vi in the form of a decision tree. For the decision making trees in CDTs, we transform the weights back into the input feature space to make fair comparison with SDT and the heuristic tree agent. The results are displayed in Table 3 all trees use intermediate features of dimension 2 for both environments. terms of stability, CDTs generally perform similarly as SDTs and even better on CartPole-vl environment.
We further evaluate the feature importance with at least two different methods on SDTs to demonstrate the instability of imitation learning settings for XRL, see Appendix F.I] We also display all trees (CDTs and SDTs) for both environments in Appendix F.3 Significant differences can be found in different runs for the same tree structure with the same training setting, which testifies the unstable and unrepeatable nature by interpreting imitators instead of the original agents.
Conclusion. We claim that the current initation-learning setting with rree-based models is not suitable for interpreting the original RL agent, with the following evidence derived from our experi- ments: (i) The discretization process usually degrades the performance (prediction accuracy) of the agent significantly, especially for SDTs. Although CDTs alleviate the problem to certain extent, the performance degradation is still not negligible, therefore the imitators are not expected to be alternatives for interpreting the original agents; (ii) With the stability analysis our experiments, we find that different imitators will display different tree structures even they follow the same training setting on the same dataset, which leads to significantly different decision paths and local feature importance assignments.

---

Under review as a conference paper at ICLR 2021
Type
Depth
DLL, L/)
DLL, R)
DIL, H)
SDT
CarPPle-vi
3
0.21
0.90
Lunarl 22
0.50
0.22
0:84
CariPole-vl
1+2
0.07
105
CDT (ours)
242
0.19
10000 10
LunarLanderr-v2
10
383
0.53
U.V1U.0
D.56
Table 3: Tree Stability Analysis. D(L, I.), D(L, R) and D(L, H) are average values of distance between an imitator L and another imitator L', or random agent R, or a heuristic agent H with metric D. CDTs generally more stable, but still with large variances over different imitators.
4.2 REINFORCEMENT LEARNING
Performance. We evaluate the learning performances of different DTs and NNs policy function approximators RL, as shown Fig. Every setting trained for three runs. We use Proximal Policy Optimization (Schulman et al. 2017) algorithm in our experiments. The multilayer perceptron (MLP) model is two-layer NN with 128 hidden units. The SDT has depth of 3 for CartPole-vi and 4 for Lundrrander-22. The CDT has depths of 2 and 2 for feature learning tree and decision making tree respectively on CartPole-vI, while with depths of 3 and 3 for LunarLander-v2. Therefore for each environment, the SDTs and CDTs have similar number of model parameters, while MLP model has at least 6 times more parameters. Detailed training settings are provided in Appendix G From Fig. 3 we can see that CDTs can at least outperform SDTs as policy function approximators for RL in terms of both sampling efficiency and final performance, although may not learn as fast as general MLPs with a significantly larger number of parameters. For MountainCar-vo environment, the learning performances are less stable due to the sparse reward signals and large variances in exploration. However, with CDT for policy function approximation, there are still near-optimal agents after training with or without state normalization, as displayed in Appendix
[8] CartPole State Norm
CartPole-v1 w/o
LunarLander-v2 State Norm
(d)
Lunart.ander-v2 w/o
201
402
302
507
507
3290 2003 tslook
2500
3030
1000
2053
1000
2000
1200
2000
2000
4200
5000
Figure 3: Comparison of SDTs and CDTs on CartPole-v/ and LunarLander--2 in terms of average rewards in RL setting: (a)(c) use normalized input states while (b) use unnormalized ones.
Tree Depth. The depths of DTs are also investigated for both SDT and CDT, because deeper trees tend to have more model parameters and therefore lay more stress on the accuracy rather than interpretability. Fig. shows the learning curves of SDTs and CDTs in RL with different tree depths for the two environments, using normalized states as input, while the comparison with unnormalized states is in Appendix with similar results. From the comparisons, we can see that generally deeper trees can learn faster with even better final performances for both CDT' and SDTs, but CDTs are less sensitive to tree depth than SDTs.
CartPole SDT
[b]
300
CartPole v1 CDT
120
Lunarlander-w2 SOT
[d]
LunarLander-v2CCD CDT
Sarrgg tanerd
loregg Braar
Rowar
100
C7T tepth: 2+7
deptr:
CDT supth: 212
G0T degtt:
dupli: 312
507 depti:
CD7
depth:
1002
2000
3000
1006
2003
1000
1800
2000
3000
0000
5050
Spaicod
Figure 4: Comparison of SDTs and CDTs with different depths (state normalized), (a) and (b) are trained on CartPole-vl, while (c) and (d) are on LunarLander-v2.
Interpretability. We display the learned CDTs in RL settings for three enviromments, compared against some heuristic solutions or SDTs. A heuristic solution for CartPole-vl is: if 30 + 0 > 0,
Provided by Zhiqing Xiao OpenAl Gym Leaderboard: http://giiiaacccoooopgiiiiginina caderrbrrr
7

---

Under review as a conference paper at ICLR 2021
push right; otherwise, push left. As shown in Fig. in learned CDT of depth 1+2, the weights of wwo-dimensional intermediate features (f(0) and fTI are much larger on the last two dimensions of observation than the first two, therefore we can approximately ignore the first two dimensions due to their low importance in decision making process. So we get similar intermediate features for two cases in two dimensions, which are approximately W1x[2] [2] + [22[33] - w6 + & after normalization (w >> 0). Based on the decision making tree in learned CDT, gives close solution the heuristic one, yielding if w& + 0 < push left otherwise push right. The original CDT before discretization and SDT for comparison are provided Appendix [
For MountainCar-vvl, due to the complexity in the landscape as shown in Fig. 6 interpreting the learned model is even harder. However, through CDT, we can see that the agent learns intermediate features combinations of car position and velocity, potentially being estimated future position previous position, and makes action decisions based on that. The original CDT before discretization has depth 2+2 with one-dimensional intermediate features, and its structure is shown Appendix !]
Due to the page limitation, the learned CDT as an example for Lunar LunarLander-v2 is provided in Appendix which also captures some important feature combinations like the angle with angular speed and X-Y coordinate relationships for decision making.
The
08 Di 18 B8 (xll) 20 ( boll ri II 21-22 (222))0007.. Tx.t
01)-0.00 -ost) safb 1e
XII: Oct Valasty
Teus.
Fake
<3: Pole Augl
angle @
XB: Velccity
+18
406
Thoe
position
<0): Hight
Figure 5: Left: learned CDT (after discretization). Right: game scene of CartPole-vl.
Toe
Pala
W 11 Su
x00 Car Postion
x]1] Cu Veleityy
Tmm
Polee
001-0.2
all: None
Trul
Fabi
velocity
<22 Rignt
AM
position
Figure 6: Left: learned CDT (after discretization). Right: game scene of MountainCar--v..
5 CONCLUSION
In this work, we have proposed new architecture of differentiable DT, the Cascading Decision Tree (CDT). A simple CDT cascades a feature learning DT and a decision making DT into a single model. From our experiments, we show that compared with traditional differentiable DTs (i.e., DDTs SDTs) CDTs have better function approximation in both imitation learning and full RL settings with a significantly reduced number of model parameters while better preserving the tree prediction accuracy after discretization. We also qualitatively and quantitively corroborate that the SDT-bbased methods with imitation learning setting may not be proper for achieving interpretable RL agents due to instability among different imitators in their tree structures, even when having similar performances. Finally, we contrast the interpretability the intermediate features. Our analysis supports that CDTs lend themselves to be further extended to hierarchical architectures with more interpretable modules, due to its ts richer expressivity allowed via representation learning. More work needs to be done to fully realize the potential of our method, which involves the investigation of hierarchical CDT settings and well-regularized intermediate features for further interpretability. Additionally, since the present experiments are demonstrated with linear transformations in the feature space, non-linear transformations are expected to be leveraged for tasks with higher complexity continuous action space while preserving interpretability.
learned DT's RL settings, especially for
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
S. Albawi, T. A. Mohammed, and S. Al-Zawi. Understanding of a convolutional neural network, In 2017 International Conference on Engineering and Technology (ICET), pages 1-6, IEEE, 2017.
O. Bastani, C. Kim, and H. Bastani. Interpreting blackbox models via model extraction. arXiv preprint arXiv: 705.08504, 2017.
O. Bastani, Y. Pu, and A. Solar-Lezama. Lezama. Verifiable reinforcement learning via policy extraction. In Advances in Neural Information Processing Systems, pages 2494-2504, 2018.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint I 004000 2016.
R. Cheng, Orosz, M. Murray, and J. W. Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3387-3395, 2019.
Y. Coppens, K. Efthymiadis, T. Lenaerts, A. Nowé, T. Miller, R. Weber, and D. Magazzeni. Distilling deep reinforcement learning policies in soft decision trees. In Proceedings of the IJCAI 2019 Workshop on Explainable Artificial Intelligence, pages 1-6, 2019.
N. Frosst and G. Hinton. Distilling a neural network into a soft decision tree. arXiv preprint arXiv:1711.0974/ 2017.
O. Irsoy, O. T. Yıldız, and E. Alpaydın. Soft decision trees. In Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), pages 1819-1822. IEEE, 2012.
S. Junges, N. Jansen, C. Dehnert, U. Topcu, and J.-P. Katoen. Safety-constrainee reinforcement learning for mdps. In International Conference Tools and Algorithms for the Construction and Analysis of Systems, pages 130-146. Springer. 2016.
Kontschieder, M. Fiterau, A. Criminisi, and S. Rota Bulo. Deep neural decision forests. In Proceedings of the IEEE international conference on computer vision, pages 1467-1475, 2015.
G. K. Kumar, P. Viswanath, and Rao. Ensemble of randomized soft decision trees for robust classification. Sädhanā, 41(3):273-282, 2016.
D. Laptev and J. M. Buhmann. Convolutional decision trees for feature learning and segmentation. In German Conference on Pattern Recognition, pages 106. Springer, 2014.
7. C. Lipton. The mythos of model interpretability. Queue, 16(3):31-57, 2018.
G. Liu, O. Schulte, W. Zhu, and Q. Li. Toward interpretable deep reinforcement learning with linear model u-trees. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pages 414 429. Springer, 2018
C. Molnar, G. Casalicchio, and B. Bischl. Quantifying model complexity via functional decomposition for better post-hoc interpretability. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 193-204. Springer, 2019.
G. Montavon, W. Samek, and K.-R. Müller. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
A. B. Murray. Reducing model complexity for explanation and prediction. Geomorphology, 90(3-4): 178-191, 2007.
E. Puiutta and E. Veith. Explainable reinforcement learning: A survey, arXiv:2005.06247, 2020.
arXiv preprint
Rota Bulo and Kontschieder. Neural decision forests for semantic image lahelling. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 81-88, 2014.
S. R. Safavian and D. Landgrebe. A survey of decision tree classifier methodology. IEEE Transactions on Systems, Man, and Cybernetics, 21 21(3):660-674, 1991.
O. Sagi and I.. Rokach. Explainable decision forest: Transforming a decision forest into an inter- pretable tree. Information Fusion, 2020.

---

Under review as a conference paper at ICLR 2021
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (70.00637700 2017.
P. Schwab and W. Karlen. Cxplain: Causal explanations for model interpretation under uncertainty. In Advances in Neural Information Processing Systems, pages 10220-10230, 2019.
A. Silva, T. Killian, I. D. J. Rodriguez, S.-H. Son, and M. Gombolay. Optimization methods for inter- pretable differentiable decision trees in reinforcement learning. arXiv preprint arXiv: vrXiv:/903,000 0ee1/99330033 2019.
A. Suárez and F. Lutsko. Globally optimal fuzzy decision trees for classification and regression. IEEE Transactions on Pultern Analysis and Machine Intelligence, 21(12):1297-1311 1999.
R. Tanno, K. Arulkumaran, D. C. Alexander, A. Criminisi, and A. Nori. Adaptive neural trees. arXiv preprint arXiv:1807.0660 2018.
A. Wan, L. Dunlap, D. Ho, J. Yin, S. Lee, H. Jin, S. Petryk. S. A. Bargal, and J. E. Gonzalez. Nbdt: Neural-backer decision trees. arXiv preprint arXiv:2004.0022 2020.
M. WU, M. C. Hughes, S. Parbhoo, M. Zazzi, V. Roth, and E. Doshi- Velez. Beyond sparsity: Tree regularization of deep models for interpretabillty. arXiv preprint arXiv: 1711066/7 2017.
H. Xiao. Ndt: Neual decision tree towards fully functioned neural graph. 172.0:171,.05594, 2017.
arXiv preprint
A SIMPLICITY AND INTERPRETABILITY A MOTIVATING EXAMPLE
Figure 7: Comparison of three different tree structures simple classification problem. From left to right: (1) multivariate DT; (2) univariate DT; (3) differentiable rule lists.
People have proposed variety of desiderata for interpretability (Lipton 2018), including trust, causality, transferability, informativeness, etc. Here we summarize the answers in general into two aspects: (1) interpretable meta- variables that can be directly understood; (2) model simplicity. Understandable variables with simple model structures comprise most of the models interpreted by humans either by physical and mathematical principles human intuitions, which is also in accordance with the Occam's razor principle.
For model simplicity, a simple model in most cases is more interpretable than a complicated one. Different metrics can be applied to measure the model complexity (Murray) 2007 Molnar et al.) 2019), like the number of model parameters, model capacity, computational complexity, non-Ineartyy ctc. There are ways to reduce the model complexity: sub-space, merging the replicates in the model, etc. Feature importance (Schwab and Karlen 2019) (e.g., through estimating model sensitivity to changes of inputs) is one type methods for projecting complicated model into a scalar space across feature dimensions. The proposed method CDT in this paper is a way to improve model simplicity by merging the replicates through representation learning.
projection from a large space into a small
For binary classification problem, three different tree structures and their decision boundaries are compared in Fig- 7 (1) multivariate DT;; (2) univariate DT;; (3) differentiable rule lists (Silva et al.) 2019). We need to define the simplicity of DT for achieving interpretability and choose which type of tree the one we prefer.
For the first two structures, we may not be able to draw conclusions for their simplicity since it seems one has a simpler tree structure but more complex decision boundaries while the other one is the opposite. We will have another example to clarify it. But draw conclusion for the second
10

---

Under review as a conference paper at ICLR 2021
and the third ones since the structure of differentiable rule lists is simpler, as it has asymmetric structure and the left nodes are always leaves. However, the problem of differentiable rule lists also obvious, that it sacrifices the model capacity and therefore hurts the accuracy. For the first left node in the example, it can only choose either one of the two labels without distinctiveness, which is clearly not correct.
big
Wy, 2, Mi
bu
D1
60), 40,,,,
2t, b.,.
Figure 8: Comparison two different tree structures on a complicated classification problem.
To clarify the problem choosing between the first two structures, a more complicated example is provided in Fig. 8 It shows the comparison of a multivariate DT and univariate DT for binary classification task. Apparently, the multivariate DT is simpler than the univariate in its structure. The conclusion is that for complex cases, multivariate tree structure has greater potentials of achieving necessary space partitioning with simpler model structures.
B DUPLICATIVE STRUCTURE IN HEURISTIC AGENTS
As shown in Fig. found that the heuristic solution") for Lunarl.nder-v22 contains the duplicative structure after being transformed into a decision tree, and the duplicative structure can be leveraged to simplify the learning models. Specifically, the two green modules (feature learning ones) in the tree are basically assigning different values to two intermediate variables (ht and at) under different cases, while the grey module (decision making one) takes the intermediate variables make action selection. Both the decision making module and the second feature learning module are used repeatedly on different branches on the tree, which forms a duplicative structure. This can help with the simplicity and interpretability of the model, which motivates our idea of CDT methods for XRL.
II the code repository of OpenAl Gym: http://wwiicccooooiiimmmmmmmmmaa

---

Under review as a conference paper at ICLR 2021
x
at-0 N=0.55
at-d 21---15444///111 M2j0.00
at 0.5x44 xll
2t-0
htval
hb-0.05
20.0.00
2n2 at-0.05
atccuns
Figure 9: The heuristic decision tree for LunarLander-v2. is 8-dimensional observation, is the univariate action given by the agent. at, ht are two intermediate variables, corresponding to the "angle-to-do" and "hover-to-do" in the heuristic solution.
C DETAILED SIMPLE CDT ARCHITECTURE
Raw Input
Feature Learing Tree
Zoron Mort.
Mart
1-0(1
18
R
ff
R-
D
f En 2
Intermerjate Feature
Decision Making
&
Output
Figure 10: A detailed architecture of simple CDT: the feature learning tree is parameterized by w and b, and its leaves are parameterized by w the inner nodes of decision making tree are parameterized by w' and b', while the leaves parameterized by w²"
D QUANTITATIVE ANALYSIS OF MODEL PARAMETERS
Considering the where have feature dimension of inputs R. we choose the interme- diate feature dimension be K < R. A CDT with two cascading trees of depth d1 and da and SDT with depth are compared. Supposing the output dimension is O, we can derive the number of parameters in CDT as:
NCCDT) = [[R + 1) 1) + K R 2d1 + [K 1)(2 - 1) +0.2
(9)
while the number of parameters in SDT is:
N(SDT) (R 1) 1) .2
(10)
12

---

Under review as a conference paper at ICLR 2021
Considering example for Eq. and Eq. with SDT being depth of while CDT has d1 2, d2 raw feature dimension R 8, intermediate feature dimension
4, and output dimension = 4, we can get NCCDT) = 222 and ((SDT) = 343. It indicates a reduction around 35% parameters in this case, which will significantly increase interpretability.
In another example, when R 8, K 4, O 4. the numbers of parameters in CDT SDT models di d2 for a total depth of range 20, The Ratio of numbers of model parameters is derived with: N(CDT)
compared in Fig. [1 assuming
N(SDT)
Comparison of Model Parameters
107
Ratio
SDT CDT
2.00
1.75
10a
1.50
Parameters
10a
1.25
Parameter eiaameeers
10
0.75
10a
0.50
0.25
10a
0.00
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Tree Depth
Figure 11: Comparison of numbers of model parameters in CDTs and SDTs, The left vertical axis is the number of model parameters in log-scale. The right vertical axis is the ratio of model parameter numbers. CDT has decreasing ratio of model parameters against SDT as the total depth of model increases.
E HYPERPARAMETERS IN IMITATION LEARNING
Tree Type
Env
Hyperparameter
Value
learning rate
x 10
CartPole-vl
batch size
1280
epochs
80
Common
learning rate
X 10
LunarLander-v2
batch size
1280
epochs
80
CartPole-vl
depth
SDT
LunarLander-v.
depth
FL depth
CartPole-vl
DM depth
intermediate variables
CDT
FL depth
LunarLander-v2
DM depth
intermediate variables
Table 4: Imitation learning hyperparameters. The "Common" hyperparameters are shared for both
SDT and CDT.
F ADDITIONAL IMITATION LEARNING RESULTS FOR STABILITY ANALYSIS
Both the fidelity and stability of mimic models reflect the reliability of them interpretable models. Fidelity is the accuracy of the mimic model, w.r.t. the original model. is estimation of similarity between the mimic model and the original one terms of prediction results. However, fidelity is not sufficient for reliable interpretations. An unstable family of mimic models will lead to inconsistent explanations of original black-box models. The stability of the mimic model is deeper excavation
13

---

Under review as a conference paper at ICLR 2021
into the model itself and comparisons among several runs. Previous research (Bastani et al., 2017) has investigated the fidelity and stability of decision trees as mimic models, where the stability is estimated with the fraction of equivalent nodes in different random decision trees trained under the same settings. However, in our tests, apart from evaluating the tree weights in different imitators, we also use the feature importance given by different differentiable DT instances with the same architecture and training setting measure the stability.
F.1 FEATURE IMPORTANCE ASSIGNMENT ON TREES
For differentiable DT methods (e.g. CDT and SDT), since the decision boundaries within each node are linear combinations of features, we can simply take the weight vector un as the importance assignment for those features within each node.
After training the tree, local explanation is straightforward to derive with the inference process of single instance and the decision path on the tree. A global explanation can be the average local explanation across instances, egg. in an episode or several episodes under the RL settings. Here we list several ways of assigning importance values for input features with SDT, to derive the feature importance vector with the same dimension the decision node vectors and input feature:
For local explanation:
I. A trivial way of feature importance assignment on SDT would be simply adding up all weight vectors of nodes on the decision path: I (xx) Ex, w (a))
II. The second way is a weighted average of the decision vectors, w.r.f. the confidence of the decision boundaries for a specific instance. Considering the soft decision boundary on each node, we assume that the more confident the boundary is applied to partition the data point into specific region within the space, the more reliable we can assign feature importance according to the boundary. The confidence of a decision boundary can be positively correlated with the distance from the data point to the boundary, or the probability of the data point falling into one side of the boundary. The latter one is straightforward in our
settings. We define the confidence as p(x) choosing node j in i-th layer from its parent on instance x's decision path. It indicates how far the data point is from the middle of the soft boundary in probabilistic view. Therefore the importance value is derived via multiplying the confidence value with each decision
p///2/=- (x)). which is also the probability of
node vector: [(x) = Ei, P pl- - (z).
Fig. [hhelps to demonstrate the reason for using the decision confidence (i.e., probability) as a weight for assigning feature importance, which indicates that the probability of belonging one category is positively correlated with the distance from the instance to the decision boundary. Therefore when there are multiple boundaries for partitioning the space (e.g.. two in the figure), we assign the boundaries having a shorter distance to the data point with smaller confidence in determining feature importance, since based on the closer boundaries the data point is much easier to be perturbed into the contrary category and less confident to remain in the original.
P1 - (+)(
11
P2 o(02(x))
d2
(2(2)
Figure 12: Multiple soft decision boundaries (dashed lines) partition the space. The dots represent input data points, and different colored regions indicate different partitions in the input space. The boundaries closer to the instance are less important in determining the feature importance since they are less distinctive for the instance.
14

---

Under review as a conference paper at ICLR 2021
III. Since the tree we use is differentiable, we can also apply gradient-based methods for feature importance assignment, which is: I(x)
By
where y SDT(x).
For global explanation:
We can simply average the feature importance at each time step (L.e, local explanation) to get global feature importance over an episode or across episodes, where the local explanations can be derived in either of the above ways.
F.2 RESULTS OF FEATURE IMPORTANCE IN IMITATION LEARNING
To testify the stability of applying SDT method with imitation learning from given agent, we compare the SDT agents of different runs and original agents using certain metrics. The agent to be imitated from is a heuristic decision tree (HDT) agent, and the metric for evaluation is the assigned feature importance across an episode on each feature dimension. As described in the previous section, the feature importance for local explanation can be achieved in three ways, which work for both HDT and SDT here. The environment Lunarl with an 8-dimensional observation in our experiments here.
Considering SDT of different runs may predict different actions, even if they are trained with the same setting and for considerable time to achieve similarly high accuracies, we conduct comparisons not only for an online decision process during one episode, but also on a pre-collected offline state dataset by the HDT agent. We hope this can alleviate the accumulating differences trajectories caused by consecutively different actions made by different agents, and give more fair comparison on the decision process (or feature importance) for the same trajectory.
Different Tree Depths. First, the comparison of feature importance (adding up node weights on decision path) for HDT and the learned SDT of different depths in an online decision episode is shown as Fig. 13 All SDT agents are trained for 40 epochs to convergence. The accuracies of three trees are 87 35%, 95, 23%, 97.50%, respectively.
(c) HDT
SDT, depth 3
(c) SDT, depth
[d] SDT, depth
Figure 13: Comparison of feature importance (local explanation 1) for SDT of depth 5, 7 with HDT on an episodic decision making process.
we can tell significant differences among SDTs with different depths, as well as in comparing them against the HDT even on the episode with the same random seed, which indicates that the depth of SDT will not only affect the model prediction accuracy but also the decision making process.
From Fig. 13
Same Tree with Different Runs. We compare the feature importance on an offline dataset, con- taining the states of the HDT agent encounters in one episode. All SDT agents have a depth of 5 and are trained for 80 epochs to convergence. The three agents have testing accuracies of 95.88%, 97.93%, and 97.79% respectively after training. The feature importance values are evalu- ated with different approaches as mentioned above (local explanation episode, as shown in Fig 14 In the results, local explanation II and III looks similar, since most decision nodes in the decision path with greatest probability have the probability values close to (i.e. close to hard decision boundary) when going the child nodes.
and III) on the same offline
From Fig. 14 considerable differences can also be spotted in different runs for local explanations, even though the SDTs have similar prediction accuracies, no matter which metric is applied.
F.3 TREE STRUCTURES IN IMITATION LEARNING
We display the agents trained with CDTs and SDT's on both CartPole-vl and LunarLander-v2 before and after tree discretization in this section, as in Fig. 15
T6
17
18
20
21
22
Each figure contains
15

---

Under review as a conference paper at ICLR 2021
SDT, id=1
SDT, id=2
SDT, id=3
local explanation
00
199
(b)
local explanation II
02-
(d)
(e)
(I)
local explanation III
15
(g)
(h)
Figure 14: Comparison of feature importance for three SDT' (depth=5, trained under the same setting) with three different local explanations. All runs are conducted on the same offline episode.
16

---

Under review as a conference paper at ICLR 2021
trees trained in four rums with the same setting. Each sub sub-figure contains learned tree (plus input example and its output) with inference path (i.e., the solid lines) for the same imput instance. The lines and arrows indicate the connections among tree nodes. The colors of the squares on tree nodes show the values of weight vectors for each node. For feature learning trees in CDTs, the leaf nodes are colored with the feature coefficients. The output leaf nodes of both SDTs and decision making trees CDTs are colored with the output categorical distributions. Three color bars displayed on the left side for inputs, tree inner nodes, and output leaves respectively, as demonstrated in Fig. 1 It remains the same for the rest tree plots. The digits on top of cach node represent the
output action categories.
Among all the learned tree structures, significant differences can be told from weight vectors, as well as intermediate features in CDTs, even the four trees are under the same training setting. This will lead to considerably different explanations or feature importance assignments on trees.
SDT !
SDT 2
11322
SDT 3
SDT (
01224
Figure 15: Comparison of four runs with the setting for SDT (before discretization) imitation learning on L The dashed lines with different colors on the left top diagram indicate the valid regions for each color bar, which the default setting for the rest diagrams.
SDT
SDT 8
Figure 16: Comparison of four runs with the same setting for SDT (after discretization) imitation learning LanarLander-v2.
17

---

Under review as a conference paper at ICLR 2021
254
input
SDT
input
SDT 2
1348
3.640
0,1
0,1
0,1
0,1
12.5
35.0
0,1
0,1
382.0
523
3.50
23.00
input
0,1
SDT 3
input
0,1
SDT
4
41033
0,1
0,1
0,1
0,1
0,1
0
0,1
0,1
15.09
2.58
11.25
Figure 17: Comparison of four runs with the same setting for SDT (before discretization) imitation learning CartPole-vl.
input
0,1
SDT 1
input
0,1
SDT ?
2.345
1.303
0,1
0
0,1
0,1
0,1
14.35
input
0,1
SDT
input
0,1
SDT
4
2334
0,1
0.1
0,1
0,1
0,1
13:00
21.35
Figure 18: Comparison of four runs with the same setting for SDT (after discretization) imitation learning CartPole-vI.
18

---

Under review as a conference paper at ICLR 2021
CDT-FL
CDT-FL
CDT-FL
CDT-DM
CDT-DM
Figure 19: Comparison of four runs with the same setting for CDT (before discretization) imitation learning on Lnder-ander--2: feature learning trees (top) and decision making trees (bottom).
CDT-FL Ø
CDT-FL
Figure 20: Comparison of four runs with the same setting for CDT (after discretization) imitation
learning LunarLander-v2: feature learning trees (top) and decision making trees (bottom).
19

---

Under review as a conference paper at ICLR 2021
CDT-FL
CDT-FL 3
CDT-FL
CDT-FL
CDT-DM &
CDT-DM
Figure 21: Comparison of four runs with the same setting for CDT (before discretization) imitation learning on CartPole-vI: feature learning trees (top) and decision making trees (bottom).
CDT-FL
CDT-FL
CDT-DM &
CDT-DM
CDT-DM &
CDT-DM
Figure 22: Comparison of four runs with the same setting for CDT (after discretization) imitation learning CartPole-vI: feature learning trees (top) and decision making trees (bottom).
20

---

Under review as a conference paper at ICLR 2021
G
TRAINING DETAILS IN REINFORCEMENT LEARNING
Tree Type
Env
Hyperparameter
Value
learning rate
5 x
0.98
0.95
0.1
CartPole-vl
update iteration
hidden dimension (value)
128
episodes
3000
time horizon
TO00
learning rate
5 x 10
0.98
0.95
0.1
Common LunarLaaneer-v2
update iteration
3
hidden dimension (value)
128
episodes time horizon
5000
1000
learning rate
5 10-3
0.999
0.98
0.T
MountainCar-v0
update iteration
hidden dimension (value)
32
episodes
5000
time horizon
TOO0
CartPole-vi
hidden dimension (policy)
128
MLP
LunarLander--v
hidden dimension (policy)
128
MountainCar Car-vO
hidden dimension (policy)
32
CartPole-vi
depth
3
SDT
LunarLander-v2
depth
MountainCar-v0
depth
FI. depth
2
CartPole-vl
DM depth
intermediate variables
CDT
FI. depth
3
Lunarl
DM depth
3
# intermediate variables
FL depth
MountainCar-v0
DM depth
2
# intermediate variables
Table 5: RL hyperparameters. The "Common" hyperparameters are shared for both SDT and CDT.
To normalize the states we collect 3000 episodes of samples for each environment with well- trained policy and calculate mean and standard deviation. During training, each state input is subtracted by the mean and divided by the standard deviation.
The hyperparameters for RL are provided in Table for MLP, SDT, and CDT on three environments.
'We found that sometimes the state normalization affect the learning performances significantly, espe- cially RI. settings.
21

---

Under review as a conference paper at ICLR 2021
H ADDITIONAL REINFORCEMENT LEARNING RESULTS
s0p
CartPole-v SDT
(9)
320
CartPoievvl CDT
(4)
Lunart ander-v2 SDT
(5) 200
LunarLandervz
600
100
200
300
300
300
300
CDI depth e12
depth:
depti:
depth:
100
507 0usti:
501 0esith
CD1 lmptr) 2+2
CDT Septh: 3+2
-300
SDT depth:
SDT depth:
0
2000
2003
3000
1003 2000
3000
2003
2000
3200
4000
5002
4003 5200
Eptoods
Editade
Figure 23: Comparison of SDTs and CDTs with different depths (state unnormalized). (a) and (b) trained on CartPole-vI, while (c) and (d) are on ander-v2..
Fig. 23 displays the comparison learning curves for SDTs and CDTs with different depths, under the RL settings without state normalization. The results are similar as those with state normalization in the main paragraph.
MountainCar-s ar-vo wy State Norm
MountainCar-vo w/o State Norm
[6]
492
42:33
Eplodd
Figure 24: Comparison of SDTs and CDTs on MountainCar-v0 in terms of average rewards in RL
setting: (a) uses normalized input states while (b) uses unnormalized ones.
Fig. 24 shows the comparison of MLP, SDT, and CDT as policy function approximators RL for the MountainCar-v0 environment, where the learning curves for each run, as well as their means and standard deviations, are displayed. The MLP model has two layers with 32 hidden units. The depth of SDT is 3. CDT has depths 2 and 2 for the feature learning tree and decision making tree respectively, with the dimension of the intermediate feature as The training results less stable due to large variances in exploration, but CDTs generally perform better than SDTs with near-optimal agents learned considering both cases.
TREES STRUCTURES COMPARISON
0M 1 axt oxld mell
6oj 1of 2ell -3%
0e edl
( 111-1111 (..5 22.00// +20.7419
2.2 1111
x:9 Carr Poecion
a.)
x(1]: Cast Vokdity
Pf P(AT]]: Q.D
Pole
a()))
Pile Valuiit
U]: Left
plald 0.0 plal[
*,1]: Right
Figure 25: The learned CDT (before discretization) of depth 1+2 for CartPole-v..
22

---

Under review as a conference paper at ICLR 2021
400 Cart Poeition
xt1I Cart Velocity
Velsrity
+00 Teft
.0136-M 162 104 S1.548
A|l Right (7)-)
ploble pillij 00
PLolEje: L.C
o
PANDIC
piaiit plalfj:: 10
PLANI: Q.D POATIE 1.0
piatep L3 P 0.
Figure 26: The learned SDT (before discretization) of depth for CartPole-vl.
(M6x) 30.24]]
a
[0] 1
8el0 1K
1.13/1 2.84
X0I: Cur Position
x11 Cur Valunity
0.9-3.8001
*|t: Laft
a[1] None
plalih
PENUI:
4|2): Righa
Pi D.02
plallh: 0.9L
peall 0.16
p(al 0.62
P64R2: 0.04
BANII 0.02
plat2 0.30
Figure 27: The learned CDT (before discretization) of depth 2+2 for MountainCar-vv.
x|1 0 8x
Xeposition
Y-pundtion
Tre
Fuboe
Troe
Filao
Xepped
EUJ Axl 2xll
AA -4.1 1..1//9
79 20/2222/2--- 2/-33
Augle
2. 2
1m -
111-2.886 sol 4x77
Q1 -2 2.0]] (3.54/
x/3] Augular Speed
>/6] Log Contact
E0j-0.2
XIN Second Leg
Teu
Fullar
aB): None
Q1//0.4
a|lk: Fire Led:
False
wil: Fire Main
w(Bi: Fire Righs
x|
Figure 28: The learned CDT (after discretization) depth 2+2 for Lunarl Lunarl.ander-v2. two dimensions are reserved for weight vectors in both F and 'D, as well as the intermediate features.
23


Reviews:
[{"review_id": "yWCvWXbpzHC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes cascading decision trees (CDTs), which applies representation learning on the decision path of trees, and applies it to explain reinforcement learning models. The main empirical results are that CDTs are claimed to achieve better performance with more compact models than SDTs (soft decision trees), and CDTs as a post-hoc explainer of RL models using imitation learning is unstable.\n\nThe results don't seem fully convincing yet. The paper only compares to SDT, although the references mention several other tree-based methods to explain RL models. VIPER would be a nice baseline to add for the imitation learning task, and ANT would also be a nice baseline in general. \n\nSome important results are not yet reported. For example, what is the performance of the black-box model on the imitation learning task? This would contextualize the accuracy numbers of SDT and CDT. Also, discretized SDT has around 50% accuracy while undiscretized SDT has around 94% accuracy, which is a surprising gap. What is the accuracy of a vanilla, hard decision tree on this problem? In Figure 4, CDT doesn't seem much better than SDT for the Lunar Lander problem. Do the authors have any explanation for why this is the case? Perhaps it was mentioned in the paper but I missed it.\n\nBelow are some suggested improvements to the presentation of the results: \n- For Table 1, perhaps separate out the bracketed results into a different row, e.g. CDT with discretization for the feature learning tree vs. CDTs with only discretization for the decision making tree and discretization for both sub-trees. In fact, maybe 2 columns for SDT — SDT not discretized, SDT discretized — followed by 3 columns for CDT — CDT not discretized, CDT discretization for decision making tree, CDT discretization for both sub-trees. And then make 3 rows: accuracy, depth, # parameters. This is a very detailed suggestion but the gist is that the presentation of this table could be much improved. And similarly for Table 2.\n- Why are the Figure 5 cart-pole results distributed over 2 plots? 6 lines on one plot is possible if done with care. Splitting them over 2 plots makes the SDT (Figure 5a) and CDT (Figure 5b) results less comparable, especially when tables are not provided in that section, only figures. The y-axes are also inconsistent, e.g. there is a -300 for the y-axis of Figure 5d but not for 5c, which makes them less comparable.\n\nWhile the trees being proposed may be more compact, are they more interpretable? I found it hard to parse the linear coefficients and retain all of them in my brain. The trees presented in Figures 6-7 had 4 features, but what if you have more features, or one-hot-encoded features? The literature has user studies to test if human subjects can use trees, but this addition of the linear coefficients to the nodes perhaps warrants a user study to see if human subjects can retain that additional information and make use of it. If a user study is not feasible, could the authors comment on how the interpretability (or lack of) of the feature representation part can be quantified? \n\nI found the findings about instability of trees as a post-hoc explainer interesting and appreciate that the authors including several runs of the methods with the same setting (Figures 15-22). However, I found the figures hard to read, and the heatmap in the tree nodes also needed more explanation, which made me further concerned about the interpretability of the feature representation part of the trees.\n\nMinor points:\n- Typo in capitalization in \"In this paper, We propose Cascading Decision Trees\"\n- Citation needed for \"some methods have axis-aligned partitions (univariate decision nodes) with much lower model expressivity\"\n- Some sentences are not precise and perhaps too casual -- “it basically gives a similar solution”, “kind of like an estimated future position” ", "extra_scores": {}, "extra_text": {}}, {"review_id": "ih3OXjn8Ipc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Edit:\n\nI have read the authors' response and the other reviews. I still believe that this paper is not ready for acceptance. \n\nSummary: \n\nThe authors propose to use Cascading Decision Trees (CDTs) to express the policy for an RL agent. The authors describe CDTs and evaluate their use in imitating a trained expert as well as representing a policy during training. \n\n\nReasons for score: \n\nThe interpretability of CDTs is not convincingly demonstrated by the examples provided. CDTs do perform better than the tested SDTs, but the experiments are insufficient to conclude that CDTs perform well compared to other models which are less interpretable than SDTs. \n\n\nPros:\n\n-CDTs are explained well.\n\n-CDTs are shown to solve basic RL environments which are commonly used for evaluation in XRL work. \n\n\nCons:\n\n-The authors argue that linear partitions are superior to axis-aligned partitions based on the number of parameters. However, the authors do not consider the number nor complexity of operations required for a \"forward pass\" through the model. A fairer evaluation of explainability would include the average or worst-case number of operations required to select an action (e.g., 4 multiplications, 4 additions, and 2 comparisons in the worst case for Figure 1a). By using a single D tree for all F trees, the number of parameters is reduced, but the length of any given path is still long (with many operations). \n\n-The authors claim that CDTs are more \"explainable\" than SDTs, but this is not sufficiently demonstrated. One advantage of DTs over MLPs is that all splitting operations occur on the original features. If the original features are interpretable, then the partitioning process operates on meaningful features. When learned features are used (as in CDTs), this property is lost. \n\n-Between the leaf of a feature learning tree and an internal node in the decision tree, the input is multiplied by a set of weights and put through a non-linearity. When this is performed several times in sequence, this begins to resemble a MLP. The authors do acknowledge this potential problem (as motivation for not evaluating hierarchical CDTs), but this concern also applies (to a lesser degree) to the \"single F, single D\" case. \n\n-The authors use \"heuristic agents\" as experts in their experiments. This does not follow the procedure established by prior work, and this is at odds with the motivation of CDTs as useful for explaining RL agents.\n\n-The experimental evaluation is lacking in a number of ways: \n\n--The authors should report policy performance (in imitation learning experiments). Accuracy is a useful metric, but not sufficient on its own. A model can have a high accuracy without learning a well-performing policy. Also, results are reported for a different set of configurations in Table 2 as compared to Table 1 (without any justification). This suggests that discretization of CDTs must be performed in a more nuanced way than otherwise stated in this work.\n\n--The authors do not compare to VIPER in the imitation learning experiments they perform though VIPER was shown to yield higher-performing policies than standard classification-based learning.\n\n--In the RL experiments, the authors compare to a MLP. They find that CDT has a similar (but sometimes worse) final performance despite having fewer parameters. However, the authors should also compare to a smaller MLP, ideally with the same number of parameters as CDT. Without this comparison, conclusions cannot be drawn about the \"parameter vs performance\" benefits of CDT.\n\n-State normalization (based on a \"well-trained policy\") is not standard and generally not feasible when applying RL. It is unclear why this was done. \n\n-The second paragraph on page 8 attempts to explain a learned MountainCar-v0 model. The lack of certainty and vagueness of the insights (\"kind of like an estimated future position or previous position, and makes action decisions based on that\") suggests that additional work is required to make CDTs interpretable. Why are \"future position\" and \"previous position\" both options for this explanation? The environment has two features (position and velocity), so discovering that the agent selects actions based on intermediate features derived from position is a given. Ideally, the authors select a way to measure interpretability and perform a quantitative evaluation. \n\n\nQuestions During Rebuttal Period:\n\nPlease address and clarify the \"Cons\" above.\n\n\nMinor Comments:\n\n-The method's motivation is best placed in the main paper, not in the Appendix (page 3, footnote 2).\n\n-The figures would be more helpful if they appeared on the same page as the corresponding text.\n\n-The caption for Table 1 is not clear with respect to which CDT accuracies are for which discretization schemes.\n\n-The authors note that discretization decreases performance and \"claim that this is a general drawback for tree-based methods in XRL...\" However, this is only applicable to soft DTs. This should be made clear (e.g., VIPER does not have this drawback).\n\n-The y-limits of Figures 5a and 5b should match so that performances can be more readily compared across plots (given that CDT and SDT are not plotted within one figure). The same applies for Figures 5c and 5d.\n\n-The paper would benefit from another editing pass for grammar. \nSome Typos:\n\n-Abstract: \"trees (DDTs) have [been] demonstrated to achieve\"\n\n-Introduction: \"are generally lack[ing] interpretability\"; \"In this paper, [w]e propose\"\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TpXh9HUDS6S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## After Rebuttal and Discussion Period\nI want to first say that I really appreciated the opportunity to review this paper. It was awesome to see the authors willing to respond to the various suggestions and questions that the reviewers provided. The paper has improved over time but some significant areas of improvement remain. There continues to be some discontinuity between the stated scope of this paper as XRL and the proposed CDT approach. During the rebuttal period, I felt that the authors didn't do enough to soften their claims to match the support provided by the contributions present in the paper through both the methods and experimentation. This is not to say that this is not valuable work. As discussed, the concepts and ideas are strong, I however feel that the execution and scoping of this work is a little off from clearly communicating the contributions it makes. One aspect will be through a full evaluation of the utility of their approach as explainable through user studies or other qualitative means. The claims made in this paper regarding explainability are currently unsupported.\n\nAdditionally, I believe that there is significant room for improvement in the experimental portion of this work. If there were a way to provide some ablations of the CDT approach as additional baselines as well as the SDT/DDT benchmark, it would significantly improve this portion of the paper.\n\nAs it stands, I have not chosen to adjust my score. I do however urge the authors to continue in this line of work. I believe that there is strong merit with the direction they've begun. I look forward to seeing a future completed version of this work.\n\n\n#### **Summary**\nThis paper builds from recent developments in differentiable decision tree approaches to address explainable/interpretable Reinforcement Learning. A primary focus of this paper is on the development of rich feature representations to improve the expressivity of the probabilistic splits in the downstream decision nodes of the tree function approximators. Extensive experiments are run to compare the proposed CDT against a general representation of prior differentiable decision tree approaches in two tasks: imitation learning and online policy development.\n\n#### **Assessment**\nThis paper does a great job introducing the proposed CDT approach in the context of the relevant literature, addressing valid criticisms about prior work. The experiments are extensive, it is clear that a lot of care and thought was put into demonstrating the apparent benefits of CDT. The ideas of cascading improved feature representations to the differentiable decision trees is a reasonable improvement over prior approaches. However, I found the claims of improved explainability to be tenuous at best. Multivariate decision boundaries, and fewer parameters do not generally equate to improved explainability, especially in high dimension input spaces. This challenge of preserving explainability is complicated further by allowing for multiple layers of multivariate decision rules. Even linear functions, when using several variables, lose their ability to be clearly understood after two or three parameters. Traditionally the notion of explainability, especially within RL, is built around the interaction between observed features and the criteria a model uses to arrive at the suggested action. By transforming the decision criteria of the function approximator to be based on a, now uninterpretable, representation of the input, this explainability is no longer preserved. I'd be open to changing my mind if user studies showed the proposed CDT model to be more explainable but the major claims of improved explainability rest on an unsupported conjecture that fewer parameters in linear equations are better. This may have been true in analyzing the results presented in this paper but the experiments are performed over low-dimensional state spaces that have relatively similar oscillatory dynamics admitted by optimal policies. Other weaknesses and suggestions for improvement can be found below.\n\n#### **Strengths**\n- The paper does a great job outlining the prior literature in framing the proposed CDT approach. It is clear where the proposed contributions lie within the space of what has been done before. The idea of improving the feature representation for use in a decision tree model is similar to Konstschieder, et al yet the insistence on maintaining model simplicity is refreshing. I'm not convinced that this automatically equates to model explainability but the effort to maintain simplicity is appreciated.\n- The paper highlights two central tasks that CDT can be used within RL and extensively evaluates its performance against a generalized form of the prior literature (SDT).\n- Within the experimental evaluation the paper evaluates several architectural settings of both CDT and SDT providing a reliable analysis over possible options for the reader to build from.\n\n#### **Weaknesses**\nIn general, I found the technical development in Section 3 to be unclear. The path $\\mathcal{P}$ is described to be a set of nodes yet the only formal definition of $\\mathcal{P}$ is in selecting a single node. Within the arg max over possible nodes, it's not clear how a choice of $u$ affects the product of path probabilities. Overall, the discussion about path probabilities is confusing. Continuing, The development for CDT in Section 3.2 only describes a single layer of decision nodes and doesn't account for multiple layers. The transformation matrix $T_{K\\times R}$ is only defined for a single decision node. Other points that are unclear throughout section 3 are:\n- In the paragraph following Equation 5, it is written: \"During the inference process, we simply take the leaf on $\\mathcal{F}$ or $\\mathcal{D}$ with the largest probability...\" Aren't these path probabilities input dependent? Don't they vary stochastically due to the probabilistic definition?\n- After equations 6-7, there is some analysis about the reduction of parameters in  CDT. Yet this is hard to place any significance on because a similar analysis is not provided about standard DTs or SDTs. This comparison is also a little tenuous given the different types of SDTs.\n\nTouching on this last point. As defined, SDT is a class of models with very different configurations. It is probably not appropriate to lump them all together in a generalized setting. Even then, the explicit setting of the SDT used to compare with CDT is never described. Without a clear description of the baselines, the experimental results are difficult to fully interpret and place much confidence in. It's not clear why the authors didn't choose several SDT models to evaluate CDT against. In particular, it appears that Silva, et al [AISTATS; 2020] accomplishes a lot of the same goals as CDT albeit without learning a feature representation. It would've been nice to see a more dedicated comparison between CDT and the specific approaches in the literature. More discussion or justification for the use of a single SDT approach is necessary if the current experimental analysis is going to be the final one.\n\nOn this point, it is unclear why discretizing the decision trees lead to reduced parameter counts as well as a reduction in overall performance. In most of the prior literature using differentiable or soft decision trees, the discretization is done after training to speed up inference as well as secure the splits at the decision nodes to allow for model explainability. It appears that perhaps the \"discretized\" form of each of these evaluated models was indicative of how the models were trained? If so, this is a deviation from how the baseline methods were actually developed, leading to an inaccurate comparison. Further clarity about what is mean by discretization here would improve this initial set of experiments greatly. \n- Accuracy of an imitation learned model isn't very indicative of how effective that model is in solving the actual task. While informative, the results presented in Tables 1 and 2 don't really communicate how effective CDT or SDT are in solving the tasks. Average return from executing the imitation learned policies would be a better metric here. This would help separate out the differences between a 94% accurate model and a 91% accurate model. Ultimately, we care about how a policy performs on the task. It would be nice to have that analyzed.\n\n- The definition of stability is never formalized. Stability of what? What is the ideal score or metric for stability? Are we looking for the weight vectors to be equivalent between different random initializations of the same tree architecture? If so, I'm not certain that Equation 8 is appropriate as it's measuring the distance a between weight vectors between arbitrary nodes. There is no relation between where in the tree compared weight vectors are coming from. Also, the distance metric will be heavily influenced by the number of parameters and depth of the tree. This makes it an insufficient comparison between tree settings as is done in Table 3.\n\n- In Section 4.2 The results figures over experimental domain are inconsistent. Figures 4 and 5 show results from Cartpole and LunarLander, omitting Mountain Car while Figures 6 and 7 omit LunarLander. Looking at the appendix, it seems that the learning curves for Mountain Car are not as clearly separable (with a lot of variance) and the decision tree for Lunar Lander is not as interpretable. This feels like these results being less aligned with the desired narrative got swept under the rug, hoping that they wouldn't be looked at.\n\n- While the magnitude of weights within the decision node do certainly help determine feature importance, the weights alone doesn't tell me anything about the value of the input and why it leads to one decision being made over another.\n \n\n#### **Additional Comments**\nOne paper that I felt was overlooked within the great literature review was:\n*Wu, Mike, et al. \"Beyond Sparsity: Tree Regularization of Deep Models for Interpretability.\" AAAI. 2018.*\n\nThis omission doesn't negatively affect my estimation of how the authors framed their work but I do feel that it stands to be mentioned alongside other distillation approaches. Not only does this paper distill a Neural Network into a Decision Tree for interpretability, it also regularizes the neural network based on the complexity of that distilled decision tree, ensuring that the end interpretation of the network is simple yet informative. \n\nAt present, I do not feel as though this paper is ready for publication. I am not confident in the claims being made by the authors with regards to explainability and would suggest that they either fully justify these claims or place more emphasis on the performance improvements made through CDT at the expense of some explainability. Being more honest with the limitations and assumptions being made in the development of a model is always the best way to go. I do however find the idea of cascading feature representations to encourage more expressive decision nodes within the decision tree to be intriguing. Particularly within the RL space. Representation learning within RL is an open problem and I wonder if this simple modeling strategy may highlight unique aspects of learning representations that deep RL is unable to.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QNXpQMQQXo", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new cascading decision trees (CDT) for interpretable RL tasks. As an extension to soft decision trees (SDT), CDT adds a feature learning tree before the decision making tree. By utilizing a low-rank matrix model in the feature learning tree, CDT reduces the parameters of SDT. \n\n1. Novelty is limited. The key novelty of this paper is to add a low-rank representation learning, i.e., feature learning tree, before the decision making tree. Therefore, it is expected that the number of parameters of CDT is less than that of SDT. The improved interpretability is in fact mainly due to such representation learning procedure. \n\n2. It is unclear how to select the tree depths of both the feature learning tree and the decision making tree in a data-driven way. Although the authors provide some preliminary experiments on the tree depth in Figure 5, it is unclear why only $1+2$, $2+2$, $3+2$ are considered in CartPole-v1 and $2+2$, $2+3$, $3+2$, $3+3$ are considered in LunarLander-v2. As shown in Table 5 in Appendix G, the feature learning tree depth, the decision making tree depth, and the number of intermediate variables of CDT are different in three examples.\n\n3. The hierarchical CDT is included in Section 3.2. The authors claimed the hierarchical CDT might be able to improve the prediction accuracy while increasing the model capacity. No experimental study was provided on hierarchical CDT. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers and authors have had a significant and healthy discussion around this manuscript. The reviewers remain concerned about the some of the central claims in this manuscript. While they have appreciated the clear communication and willingness of the authors to clarify most of their concerns, this central issue unites the reviewers in maintaining their desire to see a more significant revision of this work before publication. I recommend that the authors take the reviewers' recommendations in improving the presentation and comparison of their ideas.
Decision: Reject