Learning Tree Interpretation from Object Representation for Deep Reinforcement Learning
Authors: Guiliang Liu
Xiangyu Sun
Oliver Schulte
Pascal Poupart
Venue: NeurIPS 2021 Poster (2021)
Keywords: Interpretable Reinforcement Learning
Mimic Learning
Information Bottleneck
Identifiable Disentangling Object Encoder
Monte Carlo Regression Tree Search

Abstract: Interpreting Deep Reinforcement Learning (DRL) models is important to enhance trust and comply with transparency regulations. Existing methods typically explain a DRL model by visualizing the importance of low-level input features with super-pixels, attentions, or saliency maps. Our approach provides an interpretation based on high-level latent object features derived from a disentangled representation. We propose a Represent And Mimic (RAMi) framework for training 1) an identifiable latent representation to capture the independent factors of variation for the objects and 2) a mimic tree that extracts the causal impact of the latent features on DRL action values. To jointly optimize both the fidelity and the simplicity of a mimic tree, we derive a novel Minimum Description Length (MDL) objective based on the Information Bottleneck (IB) principle. Based on this objective, we describe a Monte Carlo Regression Tree Search (MCRTS) algorithm that explores different splits to find the IB-optimal mimic tree. Experiments show that our mimic tree achieves strong approximation performance with significantly fewer nodes than baseline models. We demonstrate the interpretability of our mimic tree by showing latent traversals, decision rules, causal impacts, and human evaluation results.

Learning Tree Interpretation from Object Representation for Deep Reinforcement Learning
Guiliang Liul.3, Xiangyu Sun', Oliver Schulte', Pascal Poupart
Cheriiton School of Computer Science, University of Waterloo 2Scooo of Computing Science, Simon Fraser University
Vector Institute
g2331 wu@uwaterooo. ca, oschul tedcs. sfu. ca, ppoupart@uxaterloo.. ca.
sunusfu.
Abstract
Interpreting Deep Reinforcement Learning (DRL) models is important to enhance trust and comply with tramsparency regulations. Existing methods typically explain a DRL model by visualizing the importance of low-level input features with super- pixels, attentions, saliency maps. Our approach provides interpretation based on high-level latent features derived from a disentangled representation. We propose Represent And Mimic (RAMi) framework for training 1) an identifiable latent representation to capture the independent factors of variation for the objects and 2) mimic tree that extracts the causal impact of the latent features action values. To jointly optimize both the fidelity and the simplicity of a mimic tree, derive novel Minimum Description Length (MDL) objective based on the Information Bottleneck (IB) principle. Based on this objective, we describe a Monte Carlo Regression Tree Search (MCRTS) algorithm that explores different splits to find the IB-optimal mimic tree. Experiments show that our mimic tree achieves strong approximation performance with significantly fewer nodes than baseline models. We demonstrate the interpretability of our mimic tree by showing latent traversals, decision rules, causal impacts, and human evaluation results.
DRL
Introduction
Deep neural networks have enabled Reinforcement Learning (RL) agents to extract relevant features from image observations and achieve human-level control 2 by modeling action-value functions. Despite their promising performance, the learned knowledge remains implicit in these black-box neural structures, which hinders understanding the importance of input features and how they influence decisions. Most previous DRL interpretations aimed to visualize attention masks 4 [ for input states. These interpretations are commonly based on entangled raw features in nigh-dimensional input space, and some recent studies g 10 showed that the generated attention masks are inconsistent for local samples in that very different attention distributions can yield equivalent predictions. Moreover, these point-wise importance maps do not identify the underlying causal relationships between target variables and complex inputs.
saliency maps 6 7
An alternative approach for generating post-hoc global explanations is mimic learning I1 which allows distilling the knowledge from an opaque DRL model to a transparent tree model. Some recent works [12 [3] built linear model trees to imitate the DRL action-valus functions. These trees learned for the nigh-dimensional input space; therefore they require numerous splits for building a tree with promising regression performance. The large tree size increases the complexity of their interpretations and makes difficult to generate human-understandaale decision rules. Previous models [4 15 constrained the tree complexity by empirically setting some
(e.g.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

maximum depth), which, however, limits the mimic performance. It is unclear how to jointly optimize the simplicity and the fidelity of tree models find a "sweet-spot".
To find an attractive trade-off between simplicity and fidelity, we develop a Represent And Mimic (RAMi) framework based the Information Bottleneck (IB) principle [T6]. The goal of IB is learning a bottleneck representation to compress input signals while maximizing the information about a target variable, which defines an anfformation-thhoreee a data compression process. Our novel IB objective facilitates learning a mimic tree as the bottleneck to approximate the action advantages from the DRL model with minimum number of splits. To learn IB- IB-optimal mimic trees for DRL interpretation, RAMi integrates two components.
objective that views mimic learning as
1) An Identifiable Multi-Object Network (IMONet) that detects objects from high- -itheninnnional input space and learns an object representation to embed the objects with a limited number of latent variables. This representation is a) well-disentangled, meaning that each variable independently captures one factor of variation in a detected object, b) identifiable, allowing the mimic tree to uniquely extract the underlying causal relation from DRL models, and c) interpretable, in order to generate understandable DRL explanations. The interpretability is demonstrated with illustrative examples of latent traversals and human evaluation.
2) A Monte Carlo Regression Tree Search (MCRTS) algorithm that learns mimic trees based on the latent features from the object representation. MCRTS incorporates the IB principle into its search heuristic by deriving a Minimum Description Length (MDL) objective. Unlike previous tree learners that deterministically select split by evaluating only its local influence, MCRTS looks ahead to global mimic performance and generates a compact distribution of candidate trees with explorations.
Our empirical evaluation shows that IMONe++MCRTS achieves promising mimic performance with significantly fewer splits than other baseline models and an important reduction in the size of tree interpretation. To demonstrate how the learned mimic tree makes its target action-advantage function interpretable we show causal relations and results of counterfactual interventions can be extracted from it. We validate the tree interpretations by 1) comparison with previous interpretable RL models, and 2) a survey that collects human evaluations.
Contributions. 1) We propose a RAMi framework that enables representing the object information with our novel IMONet and searching for the optimal mimic tree with the MCRTS algorithm. 2) We derive an cnormatiionthhoretti IB-MDL objective that incorporates both the fidelity and simplicity for mimic tree learning. 3) We introduce our method of leveraging the mimic tree to compute feature importance and extract causal relations from DRL model.
2 Represent and Mimic Framework
We propose a Represent And Mimic (RAMi) framework based on the Information Bottleneck (IB) principle. RAMi separately interprets the feature extraction and the decision making process of DRL model with 1) an interpretable latent representation and 2) a transparent mimic tree.
2.1 Mimic Learning for DRL
RAMi applies mimic learning 11 to learn post-hoe interpretations for a DRL model by transferring its knowledge to mimic tree Φ. To facilitate the knowledge distillation, mimic learners utilize soft-outputs from deep models as targets to supervise the training process of mimic trees. DRL models typically compute value functions (V(s) or Q(s, a)) estimate the expected cumulative rewards at state s. These value functions explicitly influence the RL decisions by controlling the policy gradient: Q(s, a V (als) [17] or determining the action: à = arg max, (ss, a) [8]. Compared to the policy π ( s), value functions evaluate candidate actions or states more directly. Based on the value functions, one can compute an advantage value for each action by subtracting a baseline: ((s a) [= which creates unbiased action-specific evaluations with less variance. Our mimic learner utilizes action advantages as the mimic targets, allowing us to understand when an action can outperform others by a certain advantage.
Action advantages can be defined V(s') +r V(s) if Q function is unknown (e.g., in A3C).

---

2.2 IB Method for Representing and Mimicking
We derive an objective for our RAMi framework based on the Information Bottleneck (IB) princi- ple [16]. An ideal mimic tree achieves a promising approximation performance (fidelity) with a minimum number of splits (simplicity). The IB objective naturally integrates both goals, by encourag ing a bottleneck representation to compress the input signals X and preserve as much of the relevant information about mimic targets Y (action advantages) as possible. The IB objective is:
max [T) (T,
where I denotes the mutual information, and À is Lagrange multiplier. -w (Ø, X) controls how much the mimic model compresses the input data, and I (#, Y) measures how well the mimic model preserves information about targets The IB principle defines what we mean by a good representation, in terms of the fundamental trade-off between having a concise representation and one with good mimic performance [19]. However, in practice, is difficult to learn mimic tree from high-dimensional and entangled raw inputs (e.g., images and text embeddings), so we learn a latent object representation from input space and then build mimic upon latent features. Inspired by the deep variational IB [19), The objective for our RAMi framework is given by the following result.
Theorem Consider dataset size N with input features x and prediction target Let (Bm)
log qlyn ()) denote the description length for encoding the target you let cx(()) = log po(Φ) denote the description length for encoding the mimic model &. Optimizing the IB objective is equivalent to maximizing its lower bound, which is:
N
ce w..)) 1or AD x
Bg(= m..) Bi((blz) (C,( + xC,(O)) - {H(o(()))]]
(2)
ELBo objective + IB-MDL objective + Entropy Regularizer
The proof can be found in the appendix. This lower bound defines two sub-objectives: 1) the Evidence Lower Bound (E) objective enables learning an object representation for inputs {(][ with our IMONet (Section 3. 2) The IB-MDL objective, for which we design MCRTS algorithm (Section 4 MCRTS learns a compact distribution over mimic trees q( where we select the optimal mimic tree & to achieve "sweet-spot" between fidelity and simplicity.
3 Learning Object Representation
This section introduces the Identifiable Multi-Ohject Network (IMONet). The observed inputs from the RL environment are sate-action-rewart triplets nn = s.,, an rn]. Actions and rewards are often well-disentangled in low dimensions, so our IMONet learns a disentangled latent representation for capturing the object features from states. We 1) introduce the key properties of IMONet that enable extraction causal relationships, 2) describe the model structure and 3) discuss the interpretability for the learned latent representation.
3.1 Disentangled Representation for Causal Interpretation
The high-dimensional state features from the RL environment often entangled (e.g.. highly correlated pixels in an image), which makes causal inference from the raw input space impossible. IMONet utilizes a Variational Auto-Encoder (VAB) framework (20] that converts state features to a disentangled latent representation (i.e., p(Z) 12 P(Z))). After optimizing a generative model for the observed states that utilizes the latent representations, the independent latent variables capture complete state information. The independence implies that there no (unobserved or observed) confounder between two latent variables [21 22) This property enables mimic learners to model causal relations between latent variables Z and the target variable Y following a Causal Decision Tree [23]. (Section 5.2 shows examples of causal relations). Since the latent representation encodes complex RL states with few latent variables, the efficiency of causal inference is improved significantly.
2x. Y, Z. and & are random variables; æ. z, and & are instances of random variables, so p(X) represents distribution while p(æ) defines probability. (-) is the functional parameterized by
3

---

3.2 Identifiability of Latent Representation
Identifiability is one of the most fundamental properties for uniquely identifying the causal relations between imputs and targets. If a latent representation is unidentifiable, there exists multiple distri- butions with the same generative performance, and the variables between distributions can be completely entangled, meaning that they develop inconsistent factorizations for the same inputs [24] (see Section E.1 in appendix for examples). With unidentifiable representations, the mimic learner can return many different causal relations between the latent and target variables.
To better understand this problem, we define identifiability in the unsupervised representation learning setting as follows: Z(Z) is identifiable up to equivalence ~, Vs E S. p(s) = ((){=)p((2))) (T(p(Z')) + c, where function computes sufficient statistics of a distribution, is a vector, and B is an invertible matrix (that represents bijective mapping), If B is a permutation matrix, we have identfiability up to a permutation, otherwise up to scaling 225 Previous work (24] proved that without including model designs or additional observations), it is impossible to learn identifiable representations without supervising signals. To ensure the identifiability of the latent object representation, IMONet 1) utilizes a Multi-Object Network (MONet) [26] structure that is specially designed to capture variations of objects, and 2) employs conditionally factored prior p(Z [A, R) II pf Za .A, R) where each variable Zμ has a univariate Gaussian distribution. Following an Identifiable VAE (IVAE) design 25 IMONet learns conditional approximate posterior q( (Z SS, A, R) the state representation.
p((] (222))((2)))))))))) )22 = 3B,C s.t. ())(
inductive bias (e.g., specific
0:2
0.55
0.50
moc
26
moc
05
0.40
15
20
20 90 25
25 25 Running Slep (by 10
Running Slep (by 10
Running Step (by 10K)
Figure 1: Mcan Correlation Coefficients (MCC) for the examined variational encoders (including VAE, Condi- tional VAE, MONet and IMONet) trained in the Flappy Bird (left), Space Invaders (middle), and Assault (right) environments. We report the variance MCC scores computed from three independent runs.
Evaluation. We evaluate the identifiability of IMONet by comparing its Mean Correlation Coefficient (MCC) with other variational encoders. MCC measures whether it is possible to identify latent variables from one model with variables from another model (up point-wise transformations). To compute MCC, we 1) independently train two encoders based on the same structure, 2) pair latent variables from the encoders, by solving a linear sum assignment problem for the best overall correlation, 3) average the correlation coefficients between paired latent variables at different training steps. The results (Figure [ demonstrate that conditioning variables and an object network can consistently improve the identifiability of latent representations, whereas VAEs show no evidence of identifiability. This finding is consistent with 24 25
3.3 Model Implementation
IMONet decomposes a state into object-level components with spatial masks (m1..., mm) from a multi-layer convolutional attention network (m2 indicates whether the 5th object is present in each pixel, see examples in Figure Conditioning on m, r, and a. IMONet independently encodes each component with an encoder-decoder Conditional VAE (CVAD). To train IMONet, we decompose the ELBo from our IB objective (Theorem 1) into an object-oriented variational lower bound:
(3)
log mapa(s|zz) 3Dx (II geme (2 a, mm) (a,
XDRL (gimuls, a, r))
where 1) the first term is decoder log-likelihood that defines the reconstruction performance for mixtures of decoder outputs. The state decoder pa(s 2xx) implemented by deconvolution layers. 2)
The second term is the KL-divergence (KLD) between the conditional approximate posteriors for

---

each component and conditional prior. The approximate posterior is implemented by: gene 8, O, m&) :- (ug diag(o()) s.t. 1Mp: v/ [yboone (8, me), , si [a,r)]
(4)
shmdpp and v/ are multi-layer perceptrons while sioone denotes convolutional network. The prior p(zla, a. is implemented by another Gaussian function without grony. 3) The last term defines the KLD between the mask generated by the attention network and the mask reconstructed by the CVAE. Our attention network q( mfm)) s, a, r) is implemented by U-Net [27]. which is implemented layer-by-layer convolutional neural network with instance normalization, followed by sigmoid function that transfers logistics to attentions. The mask decoder pa(ma (2x)) applies the same structure state decoder.
Learning an object representation for the RL states is applicable in practice since 1) a RL state can typically be represented by limited number of objects, and 2) the agent can interact with RL. environment to generate sufficient data for learning such a representation. Object-Oriented RL [28 29 30] has demonstrated that agent master RL games by modeling the object dynamics, so we utilize features from the object representation as the inputs to mimic learners: we sample latent features & from each component posterior Jene [S 8, A a, R M my), concatenate them with the conditioning values, and feed the entire vector [= I....., a, r] to MCRTS.
3.4 Interpreting Latent Variables
IMONet learns a symbolic abstraction of state space by representing object variations (e.g.. shapes and locations) with latent variables. A latent variable Zk,d (the dth variable for the 1 object) captures an independent factor of object variations. A common approach to reveal the information captured by Zx.a is latent traversing 26 31 31]:2]: we randomly choose images and inspect how the reconstruction components change as Zu.a a is traversed from (empirical) minimum to maximum values. Figure iillusraaees the objects captured by attention masks from IMONet and shows two examples of latent traversals in a flappy bird environment. We include human evaluation to demonstrate the interpretability of latent variables (Section 5.2p. The appendix includes more examples (Section E.3 )
Latent Traversals
Syeat
m2
m?
min,
max(2,.)
Ei M
$1
$
Figure Visualizations for IMONet outputs. IMONet decomposes (for the background), my (for pillars) and ms (for the bird), where white/dark colors mark captureed/uncaptured regions. The generations from the decoder are $1, 82, and 83. We show two latent traversals [33 for interpreting
Sreat into three objects with masks
Za.1 (the sf variable for the 3dd object) and Z2.3 (the 3rd variable for the ndd object). Za,1 captures the vertical distance between the bird and pillar, and Z2.3 captures the length of the left pillar (highlighted by blue frames).
Learning Mimic Tree Interpretations
This section introduces the approach infer mimic tree with the IB-MDL objective (in Theorem and Monte Carlo Regression Tree Search (MCRTS) algorithm to find the optimal tree.
4.1 Inferring Mimic Trees with IB-MDL
We define (x) and (y,)) in the IB-MDL objective for mimic tree inference. The goal of MDL is to find regularity and simplicity the data by viewing learning as a compression problem. There are two agents with complete copies of the latent features 2n, but only the sender agent knows the target labels Va- The sender agents transmits the receiver agent a description of the missing labels using as few bits possible;
a mimic tree to encode the information. The optimal mimic tree is
5

---

defined to be the one that enables sending the minimum description length of (a) encoding the tree structure C,(Ø), and (b) encoding the exceptions C. (yn) at each leaf node.
(a) Encoding Tree Structure: We convert the binary tree structure to string that records the splits (f) and leaf predictions (i) (e.g., 1, fi, ,, O, Do, O, U1 0, D2 where mark splits and leaves with 1 and 0) with depth-first search 34 The next proposition gives the encoding cost (6)
Given a regression tree with L splits, the total cost (in bits) of describing the tree structure with the string encoding method is:
Proposition 1
(2T. 1)2
L
()( = llogg
( 1) + -2L-
2L 21-1 + +
(5)
(b) Encoding the Exceptions: Traditional MDL tree algorithms
35 are proposed for classification
trees to handle discrete labels. Their exception-encoding methods are infeasible with continuous predictions, so we utilize an alternative approach that models the distribution of action advantages at a leaf node i with Gaussian distribution N (0)"0(()) 0, ())). Given a mimic tree o, ge and of model the prediction and the scale of exception at the leaf respectively. Inspired by the variance reduction objective in classic regression trees, we utilize the log- aag-anei to measure the cost of
where N and Nylea denote the /++1 N
encoding exceptions, so Ep(,..) IC ] == log
Σ
i= N o
total data size and the number of data points the tt leaf node respectively.
4.2 MCRTS Implementation
MCRTS (Figure takes latent fea- tures 2n R R (K and D de- note the number of objects and latent size respectively) from our IMONet and the conditioning values an, I' as inputs. It explores different splits by maintaining multiple candidate mimic trees that are trained to approx- imate action advantages Y. To initi- ate a tree search, MCRTS stores all the instances (data points) cell((21, a1, T1; 31), (z2, a2, r2: y2), (23, a3 T3i ys)...... ...)...) at the root node. An edge in the MCRTS search tree represents a split f in the mimic tree. This binary split passes the instances from a parent cell to two partition cells in child nodes. Each MCRTS edge records number of visits (V f) and an estimate QMC (J, f) of the expected splitting influence. Each node J contains a series of partition cells constructed by splits from the root node: J : (cell, ((z1, a1, 7 31); (23, 0g2 F3; ya),,............ ... cella((22, a2 r25 y2), (=1, a4 F4i ...,,
Jo (I, F, (xy), ag.r21 (5.)...))
Visiling number: value:
NY, (6.6))
N%, (6.
NY, 06 ) 0.e U..
My. Co - 0
0t (6..6')
)
2
16's <
86 16 < x
-6
N
cell,)
Jr = frelif pell
a cell
Figure 3: MCRTS structure, where tree node records partition cells and edge stores number of visits NV and estimate QJJ, f). S : Z. < denotes the kth split at layer which checks whether Za is smaller than a value v. The layer number equals the number of splits in a path. "fi denotes splitting ends.
,...,),,ellg(................... cellg(...)) ). MCRTS learns a compact distribution of mimic trees p( /b (2) A mimic tree Φ E 4 can be extracted from MCRTS by following path from the root to a leaf node Jr. so Φ = -..oo.off......... J1). A partition cell; in Jj defines a leaf node in the mimic tree Φ and the leaf prediction is 0.rec=- Execelt,, M/N, M/NN/eaf Appendix (E.1) provides mimic tree extraction example.
Searching: MCRTS implements a tree search by running M plays from a starting node J. (which is initialized to be the root node and updated after each move) with root parallelization (36). At the mhth play, MCRTS implements four phases including:
1) Selection: Traverse the tree from Jj leaf node by selecting the split ft.m at each layer to maximize the Upper Confidence Bound (UCB) [37]:
ft.m arg max,
MC
Vlog(m-1) 1 f) + Cpuct NV (J:
(6)
where Cyact controls the scale of exploration. Similar to the entropy regularizer (Theorem 1), large Cpuct prevents over-frequent visits to node. 2) Evaluation: Evaluate the selected leaf
Jieaf with reward: (Jteaf)) (yn( XL((d). 3) Expansion: Expand the leaf node

---

Table I: Mimic performance. We use the Variance Reduction (VR) and Variance Reduction Per Leaf (VR-PI.) metrics since they common regression objectives and optimized by the IB-MDL objective. Results for other regression metrics (Root Mean Square Error and Mean Absolute Error) and the corresponding variances are
recorded in Table B.2 in appendix. W4 indicates that each leaf node has an extra linear regression model. We
omit the results for the raw-input -nsed based MCRTS since it is computational intractable.
Flappy Bird
Space Invaders
Assault
Method Carl VIPER
VR-PL
Leaf
VR
VR-PL
Leaf
VR
VR-PL
Leaf
8.51E-2
8.430.5
1007
4,96E-2
7.02E-5
705
4.79E-2
7.461-5
642
8.57F-2
1.88F-4
453
4.63F-2
8.80F:5
525
5.28F-2
8.09F-5
653
M5-RT
9,59E-2
8.37B-5
1144
4,54E-2
2.92E:5
1558
4.37E-2
2.73E-5
1605
M5-MT
9.56E-2
1.55E-4
612%s
1.0E-2
1.23E:5
1303
3.42E-2
2.54E-5
1351
GM-LMT
8.99F-2
2.99F-4
303
2.07F-2
8.32F-5
249"s
5.55E-2
1.83E-4
307 44
VR-LMT
8.46B-2
5.36E-4
157
2.65E-2
1.61E4
166
5.80E-2
1.98E4
291 W1
VAB+CART
7.25B-2
"3.4E--
212
3.99E:2
7.8GE:5
"SOT"
5.ISE:2
I.IGE-4
448
VAE+VIPER
7.63E-2
5.32R-4
143
4.12F-2
9.89E-5
417
4.57E:22
1.29E-4
356
VAE+GM-LMT
6,35B-2
3.51E-4
180
3.39E-2
2.75E-4
123"4
4.20E-2
1.44E-5
293
VAE+VR-LMI
7,95-2
5.12E-4
154
3.52E-2
2.08E-4
171
5.10E-2
1.99E-4
258
VAE-MCRTS
7,83E-2
1.27E-3
61
4.82E-2
5.66E-4
85
6,58E-2
7.75E-4
85
MONNC--CRRT
8.23E-2
4.02E-T
204"
5521EE-2
133E--
335
3.67E-2
TSE4
315
IMONeL+
8,50E-2
4.48E-4
191
5.26E-2
1.69-4
313
6.05E-2
1.90E-4
319
IMONct+GM-LMT
7,87E-2
3.74E-4
212
4.79E-2
3.23E-4
149"
5.45E-2
2.15E-4
256
IMONeL- VR-LMT
8.211-22
7.160-4
115%
4,54E-2
3.79E-4
120"
6.03E-2
2.271-4
268
IMONS
8,53E-2
1.37E-3
62
5.37E-2
7.08E-4
76
7.53E-2
9.07E-4
83
with G (the maximum exploration width) child nodes. 4) Back Up: Update the action-values:
QMC = (QMSI - ( on all the traversed edges.
and increment the visit count: N Vm = -V
Move: After M plays, we move J. to child node by selecting the split f. = arg m.ex s(m))
NV (Ja: fae and setting J, to a child node J,+1 (connected by the edge fa). The next play will start from the new starting node J&+1. It allows MCRTS to prune the sub-optimal nodes (38]
Empirical Evaluation
We evaluate the mimic performance and demonstrate the interpretability of the mimic tree.
Environment and Running Settings: We study the Flappy Bird, Space Invaders, and Assaults environments. Flappy Bird is procedural game, where the game states randomly generated at each episode. Space Invaders and Assaults are commonly studied Atari games from the Gym toolkit [39.. (Check data generation details and model hyper-parameters in Appendix).
We compare previous tree-based mimic learners. The first baseline model is a Classification and Regression Tree (CART) ) [40] We include VIPER 41 our second baseline by replacing its policy (decision) tree with a regression tree that imitates action values with a Q-dagger algorithm. The third baseline is the M5 42 regression tree training algorithm. M5 constructs piecewise constant tree by pruning the sub-optimal nodes. We include M5 with the Regression Tree option (M5-RT) and the Model-Tree option (M5-MT). M5-MT builds linear regression model for the instances at each leaf node while M5-RT maintains only a constant value. The last baseline Linear Model Tree (LMT) [12 for interpreting action values. A recent work [13] explored different heuristics for selecting splitting features, including Variance Reduction (VR-LMT) and Gaussian Mixture clustering (GM-LMT). The aforementioned models are directly learnt from raw input space (states and actions) [4). The aforementioned models were previously learnt directly from raw input space (states and actions). In this study of latent representation, we evaluate their regression performance based on the latent features from both VAE and IMONet (Check appendix for the implementation details of baseline models).
Baseline Models:
5.1 Fidelity versus Simplicity
This experiment evaluates mimic trees by their 1) fidelity: how well a mimic model approximates the DRL model [12 and 2) simplicity: the size of these nonparametric trees. We divide the dataset (50k) into training (80%), validation (10%), and testing (10%) sets and generate independent runs. Fidelity and simplicity are evaluated by the mimic performance and the number of leaves respectively.
7

---

Table shows the regression performance. MCRTS+IMONet achieves promising regression performance with significantly fewer leaves than other baseline models. This is because IMONet learns concise object representation from RI. states and this representation outperforms that modelled by an unidentifiable VAE. 2) MCRTS selects splits to maximize the regression performance at global level, yielding mimic tree with better fidelity. Apart from these obvious advantages, we find that some latent features may be indistinguishable for predicting action values, and this explains why some trees have better regression performance by building large tree from raw inputs, while the large tree size compromises their interpretability.
Leaf-by-Leaf Regression Performance: To study the regression efficiency, we evaluate the per-leaf regression performance of examined models based on the latent features from IMONet. Figure illustrates the performance of examined mimic trees when we constrain their number of leaves. MCRTS achieves a leading regression performance, as it selects a split by looking ahead to the future cumulative rewards instead of its local influence. Another key observation is that well-explored split from MCRTS can beat a greedy split with extra linear regressors leaf nodes (e.g.. LMTs), allowing MCRTS to substantially improve the learning efficiency.
0.00
0.00
Veriencs Reduation
Varianne doduudtonn
Ventncc Rodk cotuuctoo
VS
6.04
6.03
0.00
0:00
0.00
Leaf Number
Leaf Number
Leaf Number
Figure 4: Leaf-by-leat tree regression results based on latent features from IMONet in the Flappy Bird (left), Space Invaders (middle), and Assault (right) environments.
5.2 Illustrative Examples of Interpretable Mimic Trees
We demonstrate the interpretability of our mimic tree by 1) illustrating the extracted rules and causal relations, 2) comparing with previous RL interpretations and 3) conducting human evaluations. This section uses the Flappy Bird environment as an example, where DRL agent earns rewards by controlling a bird to pass pillars.
cell
f foachon 'dorr??
cell fe
cell,
00.1
Plad"baby Tbeb)
Sample
Sample
cella:
&e 2 2012
collo2 fills
Intervened
Sallengy Riglon
f3l4a 071,
de 071,
cellan
Fel Res 2.177
Celtl
celle
Fst Zas 0.71
callee
exilyy
SEAM 0.17
8 0.00
90tt 0.15
Ragreescnn Trmes
Counerfacuats
Figure 5: Mimic tree. ft Zu.d v indicates the 1 tree split at Zu.s (d variable for the 8tth object) with splitting value v. In the decision rules (b&c), solid dash lines indicate tree path / causal relations.
Causal Explanations: Our mimic tree & captures the influence of latent features action advantages y. Figure 15 (a) illustrates the top three layers of the mimic tree. We can interpret the influence of split by visualizing the expected latent values after this split. For example, after implementing the split f1, we obtain two child nodes (cell2,1 and cell2,2). We compute the expected latent values for the instances on these nodes (Engceli( (Zn)) and visualize the 3rd object with the decoder (since 7 captures the variation of the 3rd object). These visualizations capture the difference of object information after splitting. They show that the instances celly,2 have shorter (vertical) distance

---

between the bird and the upper pillar, and thus their advantages for the action 'down' are larger. We verify this observation by sampling an instance from both child nodes respectively. This observation is consistent with game rules since performing the action "down' when the bird approaches the upper pillar can prevent potential crashes and thus has larger advantage.
(a) Causal Relation Extraction. Since there is no (observed/unnobserved) confounder between latent variables (section 3.1) our mimic tree can capture causal relations between splitting features and predictions. Similar to Causal Decision Trees (CDTs) 23 a tree split ff in our mimic tree represents a context- context-specific causal relationship between the splitting condition ff and the action advantage jf The context is a series of value assignments of the latent features along the path from the root to the parent node of f. For example, the decision rule in Figure 55 states that having Z3.1 2 0.71 is casually related to an action advantage of 0.23 when the action is 'down' and Z 2 0.12 (context).
(b) Counterfactual Analysis. Leveraging the generation ability of the IMONet decoder, our mimic tree enables counterfactual analysis to answer "what if" questions: we show the influence on tree prediction by intervening to set the value of Zx.dd equal some particular value, For example, in Figure ce)), we compute Loo fi,..., ft. X', (g =a,1=0)) =0]] 0)]. To achieve this, 1) we set Za,1 to for all input latent vectors and get z'. 2) The IMONet decoder computes pa( ((') and the DRL model predicts Yj with the generated observations X'. 3) The mimic tree follows previous splits to construct tree predictions (the splits on the intervened variables are removed). The difference of tree prediction before and after the intervention (0.23 versus 0.11) reveals the influence of (o(23,1 = 0).
Comparison to other interpretations. We compare other interpretable DRL models based on 1) Super-pixels [12) (Figure 5d) that highlight the pixels used as splitting features in a regression tree, 2) Saliency Map 6 (Figure which is generated by perturbing the pixels within region and evaluating their impact on target outputs, and 3) Attention Mask 4 (Figure 5D (from the convolutional attention network) that represents the importance of pixels on model outputs. These traditional models used the raw images as input during training. The learned interpretations visualize the pixels that are important for decisions. They generally agree that the distance between the bird and the pillar is most influential. This observation is consistent with rules extracted by our method (Figure b))b)) since this distance determines how likely the bird can pass the pillars win more rewards.
Human Evaluation. We evaluate the interpretability of the aforementioned methods with RL practitioners from four universities. We first introduce the saliency map, mimic trees based on super- pixels and latent variables, and then ask the participants to 1) describe the information captured by these methods and 2) rank the corresponding interpretations. To eliminate potential bias, participants are interviewed independently, and they are not told which methods are the baselines or the proposed model (check Section D in appendix for further details). We present a summary of the results by discussing the two aspects of interpretability regarding our method:
(a) Interpretability of Latent Variables. After observing the latent traversals, all (12/12) participants correctly recognized the physical meaning of the latent variables whereas 58%(7/12) and 33. participants managed to recognize the features captured by saliency maps and pixels due to the inconsistency of local samples and the complexity of tree paths.
(b) Interpretability of mimic trees. 83% (10/12) participants preferred the mimic tree based on latent features since its tree path captures globally-consisennt knowledge and its tree size is smaller than that of the super-pixel tree (62 versus 1007, check Table The rest voted for the saliency map because they preferred froight-forward explanations.
5.3 Limitations
Computational Complexity. Since MCRTS conducts multiple simulations to explore and generate the global-optimal mimic tree, it generally consumes more computational resources than classic re- gression. We compare the computational complexity of our method and the baselines in the appendix. Although IMONet compresses dige--imensional inputs into latent vectors and we parallelize the tree search in MCRTS, empirically, it takes MCRTS longer determine split.
Guarantee for Interpretability. It generally hard to theoretically justify or numerically quantity the level of interpretability, so we utilize illustrative examples and human evaluation to empirically

---

show the interpretability. Although previous works demonstrated interpretability of latent features [26] there no guarantee that these features always have a clear meaning across all RL games.
31 32,
6 Related Works
We introduce previous works for interpreting DRL models.
Seff--xplannble Approaches: Some recent works have explored self-explainable DRL models. To derive human-underssanddbbe policies, these models modified existing performance-driven DRL models (c.g.. DQN) according to some specific designs, such as symbolic planning [43], decision
trees 44 45 Generalized Value Functions (GVFs) 146), Moore Machine Networks (MMNs) (77)
transparent decision dynamics [48] or explainable feature extraction 4 44]. These modifications
influence the decision process of DRL agents to enhance interpretability.
Post-hoc Interpretation: An alternative approach is generate posterior explanations for DRL models to achieve post-hoc interpretability. Existing methods commonly apply feature perturbation to generate saliency maps G or learn attention masks SS] 50 with the attention neural network. These explanations are driven locally, based computing a localization map that highlights the important regions for current actions. is hard for these point-wise interpretations maintain their consistency with explanations from other samples, which leads to unstable explanations [10). Mimic learning (i.e., model distillation) [I]] enables the generation of globally consistent explanations from performance-driven DRL model. Previous works [13 12 utilize mimic regression trees to interpret action values from DRL models, but their training objective did not control the model complexity and thus encouraged oversized mimic trees. Another recent work [51] proposed a tree regularizel that adjusts the loss function of deep models by integrating the regression performance of mimic trees. A recently proposed VIPER model [4] designed Q-dagger algorithm to derive tree-based policy function following online imitation learning [52]. Our experiments evaluate an updated VIPER model for estimating action advantages.
7 Conclusion and Societal Impacts
This paper introduced a RAMi framework based on the IB principle, following which we 1) introduced an IMONet to learn object representation for RL states and 2) proposed a MCRTS algorithm to find mimic tree with promising fidelity and simplicity. Experiments demonstrated the leading mimic performance of our MCRTS+IMONer model. We illustrated causal relations and human evaluation. A potential negative societal impact of our work is that it encourages the government to endorse laws that regulate the level of transparency in Al systems. This may increase the difficulty of deploying advanced Al systems.
tree interpretations with
Acknowledgments and Disclosure of Funding
This project supported by Strategic Project Grant from the Natural Sciences and Engineering Research Council of Canada and MITACS Research Training funding. Our computations were facilitated by a GPU donation from NVIDIA. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 5 1877550):529-533 2015.
[2] Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Çaglar Gülçehre, Ziyu
10

---

Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature, 575(7722)3300-3335 2019.
[3] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, llya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. CoRR, abs/1912.06680, 2019.
[4] Wenjie Shi, Shiji Song, Zhuoyuan Wang, and Gao Huang. Self-supervised discovering of causal features: Towards interpretable reinforcement learning. CoRR, abs/2003.07066 2020.
[5] Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, Vancouver, BC, Canada, 8-14 December 2019, pages 12329-12338, 2019.
[6] Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understand- ing atari agents. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüsssnn Stockholm, Sweden, July 10-15, 2018, pages 1787-1796, 2018.
[7] Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding dqns. In Proceedings of the 33nd International Conference on Machine Learning. ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 1899-1908, 2016.
[8] Sarthak Jain and Byron C. Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HILT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 3543-3556, 2019.
[9] Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNUP 2019, Hong Kong, China, November 3-7, 2019, pages 11-20, 2019.
[10] David Alvarzz-elis and Tommi S. Jaakkola. On the robustness of interpretability methods. CoRR, abs/ 808.000000899 2018
[11] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, Neurips 2014, Montreal, Quebec, Canada, December 8-13 2014, pages 2654- 2662, 2014.
[12] Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingcan Li. Toward interpretable deep reinforce- ment learning with linear model u-trees. In Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2018, Dublin, Ireland, September 10-14, 2018, Proceedings, Part II, pages 414-429, 2018
[13] Xiangyu Sun, Jack Davis, Oliver Schulte, and Guiliang Liu. Cracking the black box: Distilling deep sports analytics. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Virtual Conference, August 23-27, 2020, 2020.
[14] Olcay Boz. Extracting decision trees from trained neural networks. In Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Edmonton, Alberta, Canada, July 23-26, 2002, pages 456-461, 2002.
[15] Darren Dancey, Zuhair Bandar, and David McL.an. Logistic model tree extraction from artificial neural networks. IEEE Transactions Systems, Man, and Cyberneties, 37(4):794-88) 802, 2007.
11

---

[16] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004055, 2000.
[17] Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information Processing Systems 12: Anmual Conference on Neural Information Processing Systems 1999, Neurips 1999, Denver, Colorado, USA, November 29 December 4, 1999, pages 1018-1014, 1999.
[18] Martin A. Riedmiller. Neural fitted iteration first experiences with a data efficient neural reinforcement learning method. In 16th European Conference on Machine Learning, ECML 2005, Porto, Portugal, October 3-7, 2005, Proceedings, pages 317-328, 2005.
[19] Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
[20] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[21] Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Structured causal disentanglement in variational autoencoder. CoRR, abs/2004.08697 2020.
[22] Bernhard Schölkopf. Causality for machine learning. CoRR,
2019.
[23] Jiuyong Li, Saisai Ma. Thuc Duy Le, Lin Liu, and Jixue Liu. Causal decision trees. IEEE Transactions on Knowledge and Data Engineering, 29(2):257-271, 2017.
[24] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, Long Beach, California, USA, 9-15 June 2019, pages 4114-4124, 2019.
[25] Ilyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, and Aapo Hyvärinen. Variational autoencoders and nonlinear ICA: unifying framework. In 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, Online, 26-28 August 2020, pages 2207-2217, 2020.
[26] Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. CoRR, aass//91.1.. 1390, 2019.
[27] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Inter- vemtion - MICCA/ 2015 18th International Conference Munich, Germany, October 5 9, 2015, Proceedings, Part III, volume 9351 of Lecture Notes in Computer Science, pages 234-241. Springer, 2015.
[28] Carlos Diuk, Andre Cohen, and Michael L. Littman. An object-oriented representation for efficient reinforcement learning. In Proceedings of the 25th International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008, volume 307 of ACM International Conference Proceeding Series, pages 240-247. ACM, 2008
[29] Vikash Goel, Jameson Weng, and Pascal Poupart. Unsupervised video object segmentation for deep reinforcement learning. In Advances Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montréal, Canada, December 3-8, 2018 pages 5688-5699, 2018
[30] Thomas N. Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
12

---

[31] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholnsmässan, Stockholm, Sweden, July 10-15, 2018, pages 2654-2663, 2018.
[32] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a con- strained variational framework. In 5th International Conference Leurning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
[33] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
[34] J. Ross Quinlan and Ronald L. Rivest. Inferring decision trees using the minimum description length principle. Information and Computation, 80(3):227-248, 1989.
[35] Manish Mehta, Jorma Rissanen, and Rakesh Agrawal. Mdl-based decision tree pruning. In Proceedings of the First International Conference Knowledge Discovery and Data Mining (KDD-95), Montreal, Canada, August 20-21, 1995, pages 216-221, 1995.
[36] Guillaume Chaslot, Mark H. M. Winands, and H. Jaap van den Herik. Parallel monte-carlo tree search. In Computers and Games, 6th International Conference, CG 2008, Beijing, China, September 29 October I, 2008. Proceedings, pages 60-71, 2008.
[37] Adrien Couëtoux, Jean-Baptise Hoock, Nataliya Sokolovska, Olivier Teytaud, and Nicolas Bonnard. Continuous upper confidence trees. In Learning and Intelligent Optimization Sth International Conference, LION 5, Rome, Italy, January 17-21, 201 Selected Papers, pages 433 445, 1.
[38] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354-355) 2017.
[39] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.011400, 2016.
[40] Roman Timofeev. Classification and regression trees (cart) theory and applications. Humboldt University, Berlin, pages 1-40, 2004.
[41] Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montréal, Canada, December 3-8, 2018, pages 2499-2509, 2018.
[42] John R Quinlan et al. Learning with continuous classes. In 5th Australian joint conference on artificial intelligence, volume 92, pages 343-34-3 -38.. World Scientific, 1992.
[43] Daoming Lyu, Fangkai Yang, Bo Liu, and Steven Gustafson. SDRL: efficient deep reinforcement learning leveraging symbolic planning. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, AAAÍ 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, pages 2970 (900229777770 2019.
interpretable and data-
[44] Ujjwal Das Gupta, Erik Talvitie, and Michael Bowling. Policy tree: Adaptive representation for policy gradient. In Proceedings of the 29th AAAI Conference on Artificial Intelligence, Austin, Texas, USA, January 25-30, 2015, pages 2547-2555 2015.
[45] Aaron M. Roth, Nicholay Topin, Pooyan Jamshidi, and Manuela Veloso. Conservative q-improvement: Reinforcement learning for an interpretable decision-tree policy. CORR, Abs/1907.011800 2019.
[46] Zhengxian Lin, Kin-Ho Lam, and Alan Fern. Contrastive explanations for reinforcement learn- ing via embedded self predictions. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
13

---

[47] Anurag Koul. Alan Fern, and Sam Greydanus. Learning finite state representations of recurrent policy networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, IA, USA, May 6-9, 2019. OpenReview.net, 2019.
[48] Alihan Hüyük, Daniel Jarrett, Cem Tekin, and Mihaela van der Schaar. Explaining by imitating: Understanding decisions by interpretable policy learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, ,021,
[49] Tom Bewley and Jonathan Lawry. Tripletree: A versatile interpretable representation of black box agents and their environments. CoRR, abs/2009.044 04444, 2020.
[50] Anthony Manchin, Ehsan Abbasnejad, and Anton van den Hengel. Reinforcement learning with attention that works: A self sepf-supe approach. In Neural Information Processing 26th International Conference, ICONIP 2019, Sydney, NSW, Australia, December 12-15, 2019, Proceedings, Part pages 223-230, 2019.
[51] Mike Wu, Michael Hughes. Sonali Parbhoo, Maurizio Zazzi, Volker Roth, and Finale Doshi- Velez. Beyond sparsity: Tree regularization of deep models for interpretability. In Proceedings of the 32nd AAAI Conference on Artificial Intellience, (AAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 1670-1678, 2018.
[52] Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011, volume of JMLR Proceedings, pages 627-635, 2011.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5.3
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 7 For a detailed discussion of ethics statement, please check section F in appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Please check the assumption of the IB objective in the Theorem I and the assumption of causal inference section B
(b) Did you include complete proofs of all theoretical results? [Yes] For the complete proof of Theorem please check Section B in appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] Please check the attached code and Section A.2.2 in appendix for the data generation details.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please check Section A.2.3 in appendix for the implementation details (including hyperparameters) of our model and all baseline models.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] Please check Section C in appendix for the complete results (including mean and variance for 5 independent runs).
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please check Section A.2.7 in appendix for the details of computing resource. Please check Section A.2.4 in appendix for the computational complexity of our method and other basclines.
14

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the creators of Pygame learning environment and OpenAl gym toolkit in reference.
(b) Did you mention the license of the assets? [N/A] The assets creators posted the
instructions about how to cite their works. We are not aware of mentioning the licenses.
(c) Did you include any assets either in the supplemental material as URL? [N/A] We have not created new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We have used data from other people.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] Data from video game does not contain personal information.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] Check Section in appendix for the details of human evaluation. (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, upplicable? [N/A] Participants answered some questions regrading model evaluation. We did not collect any personal data.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] Participants are volunteers. This message was properly delivered before any interview.
15


Reviews:
[{"review_id": "nPnY4u4vmEe", "reviewer": "Reviewer_RYit", "summary": "This paper proposes a tree-based method to interpret causality between actions and states in reinforcement learning algorithms. They firstly use an encoder-decoder model name IDOE to extract object-level and identifiable latent variables, then infer a mimic tree with information bottleneck as the objective function. Compared with previous methods and baselines, their generate trees have fewer leaf nodes, therefore are easy for humans to interpret and understand.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** As far as I know, this work is original in this area. Although the idea of using a tree structure to explain RL algorithms is not new, their method still has contributions in controlling the size of the mimic tree.\n\n**Quality:** The method proposed in the paper is technically correct and their experiments on three RL environments can also support their statements. In the evaluation part, they also provide human evaluation results to show the advantage of their method. Although this is not common in the ML community and the number of participants is relatively small, the conclusion is acceptable. \n\n**Clarity:** The writing of this paper is good but I feel that some details of both the IDOE part and MCRTS part are missing. Maybe it is because of the complexity of this method and the limited pages, but it would be better if the methodology part could be consecutive. One example is section 4.3. Why can IDOE decompose a state $s$ into object-level components? I thought IDOE is a carefully designed encoder-decoder structure with disentangled latent variables, then where does the decomposition happen?\nThere are also some typos that influence the reading:\n\n(1)\tIn Line 251, Appendix (E.1) -> Appendix (E.2)\n\n(2)\tAccording to the description of Section 6.2, I think the bottom figure in Figure 5 (c) should be the same as $cell_{3,3}$.\n\n**Significance:** This paper targets a very important problem about XAI and the results shown in the paper are interesting. One concern from me is that this method is too complex to be extended. Also, I have three questions about the IDOE part:\n\n(1)\tIt seems that all 3 RL environments have a clean background and clear objects separation. Will the IDOE model still work in more complex real-world environments? \n\n(2)\tIf the IDOE model fails, e.g. encodes entangled latent variables, how much it will influence the mimic tree generation?\n\n(3)\tIf I understand Section 4.4 correctly, human efforts are needed to check the interpretability of latent variables with latent traversing. Then, how about the scalability for complex tasks?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mention two limitations of this work: (1) the computational complexity for the MCRTS generation part. (2) lack of theoretical guarantees for interpretability. Also, as a method attempts to explain the AI system, one potential societal impact, said by the authors, is that this method is abused by the government or users.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Yf3K4uG-3ys", "reviewer": "Reviewer_NQFz", "summary": "## Summary\nThis paper deals with learning interpretation of objects and decision making using RAMi framework. The framework first learns an identifiable latent representation, and then mimics a casual relationship using a tree. As a result, the framework can construct approximations to an interpretable, high-order decision making procedure. The paper shows that it can be applied to standard deep reinforcement learning problems.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Main Review\n### Originality and Significance.\nI think the originality of the paper can be evaluated by the theoretical arguments (Thm 1 and Prop 1). As far as I understood, the proposed framework and models are somewhat novel by adopting information theoretic perspectives for modeling representation. However, the originality of MCRTS algorithm compared to the decision tree policy of VIPER model is not clear by the given text and Prop 1. This work also lacks providing qualitative results of representation (visualization, comparison, etc.).\n\n### Clarity.\n- I do not believe that the paper is clearly written. Also, the paper should have provided more information and supportive experiments about the excellence of the model, as this work aims to learn \"interpretable\" representation.\n- Though its derivation of IB objective (Theorem 1) is sound, its relations to individual objectives of modules are a bit ambiguous. The inductive bias to disentangle latent space used in this paper is the spatial mask of objects in state scene, which is a reasonable implementation choice only for limited environments. Using a mask will be an inappropriate choice for long-term, sparse environments like Montezuma’s Revenge. Also, only latent visualizations for the Flappy Bird are attached, which makes an evaluation of interpretability less reliable. Though experiments show that disentangled latent was unique to problem space unlike normal VAE and the IDOE reduced variance of regression, the most experimental contribution to variance reduction per leaf looks due to MCRTS.\n- Most importantly, the paper has weak relation to RL framework unlike its title. A VIPER model, which the paper referenced as previous mimic model combined with reinforcement learning, implemented mimic tree policy and clearly supported its relation to RL using RL’s terminology and experiment results. This paper, however, just uses the state value function of a trained RL framework as a mimic tree’s regression target. The objectives of the suggested framework don’t exploit any properties of RL assumptions, thus the regression target can be replaced to any utility functions for continuous state space. It can be considered that the paper mentions RL because it just used RL environments without sincere consideration.\n\n### Additional Comments.\nI am a bit concerned about the title. I think the paper does not show much perspectives about \"Deep Reinforcement Learning\". The paper does not provide usefulness of the model in the sense of score acquisition, and it is ambiguous how the model helps DRL studies (which are indeed a specific field of decision making).\n\n### Minor.\n- Module terminologies are inconsistent in a supplement.\nsupplement line 119: “IDE” -> “IDOE”\nsupplement line 122: “IDE” -> “IDOE”\nsupplement line 144: “IDE” -> “IDOE”\n\n- Typos:\nsupplement line 1: “Flappy Bird” -> “The Flappy Bird”\nsupplement line 1: “we upload” -> “we uploaded”\nsupplement line 23: “applies” -> “apply\nsupplement line 98: empirically -> empirical\nsupplement line 142: “entangle” -> “entangled”\nsupplement line 159: “We” -> “we”\nsupplement line 161: “minimizing” -> “minimize”\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "WzryzbyGpPB", "reviewer": "Reviewer_hnrw", "summary": "This paper proposes Represent and Mimic (RAMi), a framework for extracting an interpretable mimic tree from the advantage function learned by a deep RL agent. This is achieved by (1) using a variation of MONet [34] to extract disentangled object representations from visual state observations that are interpretable, and (2) fitting a distribution over regression trees to explain the advantage of an action in a particular state in terms of causal relationships between these representations.\n\nThe main innovation is the objective function for the regression tree search, which is inspired by the information bottleneck approach and the principle of minimum description length. Here, the distribution over regression trees is viewed as an \"information bottleneck\" for predicting the advantage of an action-state pair (i.e. the target) from their input representation. This suggests that the tree search should prioritize trees having small description length (i.e. that are compressed and \"simple\") while accurately modeling the target values (i.e. having high fidelity), which is achieved by incorporating suitable heuristics.\n\nRAMi is compared on three standard RL datasets, and compared to a number of baselines that primarily vary in terms of how they extract the mimic tree. Qualitatively it can be observed that RAMi is able to balance the trade-off of simplicity and fidelity quite well, and outperforms prior approaches on standard metrics. It can also be observed how encoding pixel-level observations to disentangled object representations is helpful. Qualitatively it is demonstrated how the trees learned by RAMi are interpretable and allow for a causal explanation of how certain state-action combinations affect the prediction made by the advantage function.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is concerned with an interesting and relevant problem: given a trained deep neural network, how can we explain their predictions? In the case of RL, as is the context of this paper, the prediction may be the learned advantage function, and the goal to explain how certain state-action pairs lead to higher expected future normalized return compared to others.\n\nThe approach taken is that of \"mimic learning\", where an interpretable learner such as a decision tree, is fit post-hoc to the predictions made by the deep neural network. Prior approaches use linear model trees to achieve this in the context of RL, however due to them being applied to low-level pixels, their extracted rules are complex. Another consequence is that the resulting trees grow quite complex, while it is not clear how they should be balanced.\n\nThis paper aims to address both of these problems by (1) applying the mimic learning process to a learn feature space that distinguishes high-level variables such as objects, and (2) using a tree search method that considers the global tree structure and is better able to trade-off fidelity and simplicity. \n\nTo address these problems it is proposed to address them from within a single framework called Represent and Mimic (RAMi) based on the Information Bottleneck (IB) principle [13]. Here, the mimic learner is viewed as an \"information bottleneck\" for encoding information about the target variable (advantage) from the input (state, action, reward), which suggests a means of learning the mimic tree to trade-off fidelity and simplicity. In particular the variational lower-bound to the information bottleneck as proposed in Alemi et al. [27] is modified to include latent variables z for encoding information about the input x, as well as a reconstruction term, which gives rise to a three part objective function. The first part is the standard ELBO used to train latent variable models such as VAEs. The second part can be understood as capturing the description length for encoding the targets using the mimic model as well as the description length of the mimic model itself. The final term is an entropy-regularizer.\n\nThe derivation of this objective (theorem 1) largely follows prior work, while incorporating the reconstruction term was rather adhoc. Since this training objective is not actually used to train a single system end-to-end, but rather separate systems are trained on separate terms, it is not quite clear what is gained by modifying the objective from Alemi et al. [27]. Would it not have been much simpler (and more principled) to treat the problem of representation learning separately outside the IB framework to give rise to the standard ELBO objective? And then apply the IB objective directly to latent observations z as opposed to x? It seems to me that one can reach a similar objective for the tree search that can also be understood in terms of MDL. This would certainly be preferable since the connection between MDL terms and the ELBO is not exploited during learning as far as I can tell.\n\nTo learn good representations using the ELBO objective, an approach called Identifiable Disentangling Object Encoder (IDOE) is proposed based on VAEs. As far as I can tell, IDOE is really just MONet [34] but having a conditional encoder and prior (conditioned on actions and reward) inspired by the Identifiable VAE from Khemakhem et al. [33]. Further the prior factorizes across latents as is a standard assumption. The fact that IDOE is really just MONet with only very slight modifications is not sufficiently made clear in section 4.1-4.3 and I urge the authors to avoid renaming prior work in this way and claiming innovations that belong to others! The choice for MONet with a conditional and factorized prior is well motivated, since the goal is to obtain an input representation that is identifiable and disentangled to avoid having confounders when modeling relations between variables when interpreting the mimic tree learned later one. I think the MCC experiment is quite nice here, although details about the set-ups are lacking. The latent traversals in section 4.4 are less convincing since in the bottom right it can be seen how Z_2,3 not only captures the length of the left pillas, but also the size of the bottom pillars, which go up and down during traversals. I would therefore suggest that the authors tone down claims regarding the impossibility of confounders (such as in 321).\n\nGiven the representations learned by MONet, what remains is inferring the mimic-trees, which in my view is where the main innovation is at. To balance simplicity and fidelity as suggested by the IB principle one should search for trees that can be encoded efficiently, and that represent the data well (require encoding few exceptions). The cost of encoding a tree (proposition 1) is computed using the metric from Quinlan et al. [42], where again the authors fail to make clear that they are using an existing method as opposed to inventing a metric themselves. The derivation and \"proof\" provided in appendix B.3 is identical to [42] (equation 6 & 8)  other than inserting different values for the parameters. To be able to measure the effect of exceptions gaussian distributions are used at the leaf nodes, whose variance reflects the amount of exceptions that need to be encoded. The log of the variance can then be used as a cost metric. Both of these measures make sense to me and the connection to the IB-principle and MDL is clear.\n\nThe search itself is implemented using Monte Carlo Regression Tree Search (MCTRS) making use of these cost functions. The explanation in section 5.2 is quite dense and I was not able to understand in detail how the search proceeds. I believe that section 4 can be shortened significantly, which should make it possible to provide a gentler discussion and provide an intuitive explanation of the various update rules. \n\nThe reported results are quite promising and MCTRS is compared to a number of other tree search algorithms that act on the raw observation and on the encoded observations by either a VAE or MONet. In terms of standard metrics such as variance reduction and root mean squared error it can be observed that MCTRS is the best performing model. Perhaps in the interest of space, the comparison to modeling raw pixels could be moved to the appendix in favor of the RMSE results, since it is quite clear that this is suboptimal. The leaf-by-leaf regression performance is quite interesting and indeed suggests that MCRTS performs better decision-making under a limited budget of leaf nodes.\n\nThe qualitative evaluation is impressive and clearly shows how this approach can be utilized to infer interpretable mimic trees. The benefit of learning a representation in terms of object representations is clearly shown, since intermediate leave nodes can be visualized nicely in this way. I wonder if these visualizations could be combined with pixel-level techniques such as attention maps to obtain the best of both worlds. A limitation of the counterfactual analysis in the way it is presented is that it is not clear how to obtain latent vectors that correspond to the kind of interventions one would want to perform at a pixel-level, which should be discussed. The human evaluation results are interesting, but also limited, since it is likely that the 12 participants from 3 different universities do not at all reflect the general population. Rather, I would encourage the authors to include additional analysis for MCRTS focusing on the implementation of the two cost functions, as well as sensitivity to hyper-parameters. Qualitative analysis on the other 2 datasets is also missing, which should be added (to the appendix)\n\nIn general I would like to see this paper accepted, since the results are impressive and the proposed approach is sensible and well motivated. However, the clarity of the paper should be improved significantly, and the authors should clearly indicate what parts are novel contributions and what parts belong to prior work. I would also encourage the authors to consider separating the ELBO and the IB principle, since their connection is currently not exploited (no end-to-end system is trained) while the derivation is ad hoc. The experiment section can also be improved as indicated above. Currently is it difficult to gauge whether the demonstrated examples are cherry-picked or not.\n\n**Minor Comments**\n\n* Error bars in Figure 4 are only shown for MCRTS in the right plot, but not in the others. Please try to include them for the left and middle plot as well\n\n* I would have liked to see additional examples of the mimic trees extracted in the appendix, as well as examples of mimic trees extracted by other methods\n\n* I would like to see the related work section expanded to include a discussion of other approaches to learning representations that satisfy the criteria met here with MONet. An overview of other methods to learning object representations can be found in Greff et al., 2020 \"On the binding problem ...\"\n\n* It is an assumption that there are no confounders after training MONet, which is not evident from the current discussion. This is problematic since this ideal representation is likely not achieved in practice.\n\n* At times the paper conflates a representation that is disentangled with one in which the posterior factorizes (and so does the prior). This is not how disentanglement is treated in other works and it would be good to indicate that this particular notion is adopted here.\n\n* highlighting the best method is missing in Table C.1\n\n* I believe the value for Figure 5c should be 0.11 as opposed to 0.23. Currently there is no change as far as I can tell. \n\n* The entropy sign in equation (1) should be a + I believe\n\n**Post-rebuttal Update**\n\nI have read the other reviews and the author response. I am satisfied with the author response insofar the authors agree to do a better job at framing their contributions wrt. prior work and discussing limitations. On the other hand, I remain of the opinion that the shared RAMi objective contributes little understanding and makes the overall approach needlessly difficult to understand. In particular, there is little to no interaction between the representation learning parts (ELBO) and the mimic tree learning parts (MDL), and presenting this method as a 2-stage approach based on different principles is more natural. \n\nThe qualitative empirical results look impressive, although I would have liked to see additional examples to understand how reliable this approach is. The quantitative results also look good and the authors appear to provide a thorough comparison to prior methods. I note that I don't consider myself an expert in the area of mimic tree learning (or explainable ML in general), and it is possible that I have misjudged the significance of these results. I intend to keep my current score of a 6 with medium confidence (3).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Both statements are provided albeit quite brief. Regarding limitations, it would also be good to comment on current failure cases and let the reader develop an understanding of how cherry-picked the results shown are.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach to learning an interpretable distillation of an RL agent's advantage function by first adopting a object-disentangling model (MONet) and subsequently fitting regression trees to allow extraction of causal relationships therein. A key issue of deciding between tree size/complexity and expressiveness is handled using the Information Bottleneck (IB).

The reviewers agree that the paper tackles an interesting and relevant problem, and the use of IB to drive the tree learning is novel, and the experiments are quite thorough.

The primary issues with the work however appear to be with some of the framing (with a conditioned version of MONet implying novelty) and the model setup, with questions on whether the end-to-end setup is overly complicated over a two-stage pipelined model.

On balance, it appears as if the paper has more merits than issues, and most of the issues raised could be addressed with a bit of work. I would strongly urge the authors to actually make the edits for model renaming, detail, and the utility of end-to-end here, as requested by the reviewers.
Decision: Accept (Poster)