Learning Tree Interpretation from Object Representation for Deep Reinforcement Learning
Authors: Guiliang Liu
Xiangyu Sun
Oliver Schulte
Pascal Poupart
Venue: NeurIPS 2021 Poster (2021)
Keywords: Interpretable Reinforcement Learning
Mimic Learning
Information Bottleneck
Identifiable Disentangling Object Encoder
Monte Carlo Regression Tree Search

Abstract: Interpreting Deep Reinforcement Learning (DRL) models is important to enhance trust and comply with transparency regulations. Existing methods typically explain a DRL model by visualizing the importance of low-level input features with super-pixels, attentions, or saliency maps. Our approach provides an interpretation based on high-level latent object features derived from a disentangled representation. We propose a Represent And Mimic (RAMi) framework for training 1) an identifiable latent representation to capture the independent factors of variation for the objects and 2) a mimic tree that extracts the causal impact of the latent features on DRL action values. To jointly optimize both the fidelity and the simplicity of a mimic tree, we derive a novel Minimum Description Length (MDL) objective based on the Information Bottleneck (IB) principle. Based on this objective, we describe a Monte Carlo Regression Tree Search (MCRTS) algorithm that explores different splits to find the IB-optimal mimic tree. Experiments show that our mimic tree achieves strong approximation performance with significantly fewer nodes than baseline models. We demonstrate the interpretability of our mimic tree by showing latent traversals, decision rules, causal impacts, and human evaluation results.

Learning Tree Interpretation from Object Representation for Deep Reinforcement Learning
Guiliang Liul.3, Xiangyu Sun', Oliver Schulte', Pascal Poupart
Cheriiton School of Computer Science, University of Waterloo 2Scooo of Computing Science, Simon Fraser University
Vector Institute
g2331 wu@uwaterooo. ca, oschul tedcs. sfu. ca, ppoupart@uxaterloo.. ca.
sunusfu.
Abstract
Interpreting Deep Reinforcement Learning (DRL) models is important to enhance trust and comply with tramsparency regulations. Existing methods typically explain a DRL model by visualizing the importance of low-level input features with super- pixels, attentions, saliency maps. Our approach provides interpretation based on high-level latent features derived from a disentangled representation. We propose Represent And Mimic (RAMi) framework for training 1) an identifiable latent representation to capture the independent factors of variation for the objects and 2) mimic tree that extracts the causal impact of the latent features action values. To jointly optimize both the fidelity and the simplicity of a mimic tree, derive novel Minimum Description Length (MDL) objective based on the Information Bottleneck (IB) principle. Based on this objective, we describe a Monte Carlo Regression Tree Search (MCRTS) algorithm that explores different splits to find the IB-optimal mimic tree. Experiments show that our mimic tree achieves strong approximation performance with significantly fewer nodes than baseline models. We demonstrate the interpretability of our mimic tree by showing latent traversals, decision rules, causal impacts, and human evaluation results.
DRL
Introduction
Deep neural networks have enabled Reinforcement Learning (RL) agents to extract relevant features from image observations and achieve human-level control 2 by modeling action-value functions. Despite their promising performance, the learned knowledge remains implicit in these black-box neural structures, which hinders understanding the importance of input features and how they influence decisions. Most previous DRL interpretations aimed to visualize attention masks 4 [ for input states. These interpretations are commonly based on entangled raw features in nigh-dimensional input space, and some recent studies g 10 showed that the generated attention masks are inconsistent for local samples in that very different attention distributions can yield equivalent predictions. Moreover, these point-wise importance maps do not identify the underlying causal relationships between target variables and complex inputs.
saliency maps 6 7
An alternative approach for generating post-hoc global explanations is mimic learning 11 which allows distilling the knowledge from an opaque DRL model to a transparent tree model. Some recent works [12 [3] built linear model trees to imitate the DRL action-valus functions. These trees learned for the nigh-dimensional input space; therefore they require numerous splits for building a tree with promising regression performance. The large tree size increases the complexity of their interpretations and makes difficult to generate human-understandaale decision rules. Previous models [4 15 constrained the tree complexity by empirically setting some
(e.g.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

maximum depth), which, however, limits the mimic performance. It is unclear how to jointly optimize the simplicity and the fidelity of tree models find a "sweet-spot".
To find an attractive trade-off between simplicity and fidelity, we develop a Represent And Mimic (RAMi) framework based the Information Bottleneck (IB) principle [T6]. The goal of IB is learning a bottleneck representation to compress input signals while maximizing the information about a target variable, which defines an anfformation-thhoreee a data compression process. Our novel IB objective facilitates learning a mimic tree as the bottleneck to approximate the action advantages from the DRL model with minimum number of splits. To learn IB- IB-optimal mimic trees for DRL interpretation, RAMi integrates two components.
objective that views mimic learning as
1) An Identifiable Multi-Object Network (IMONet) that detects objects from high- -itheninnnional input space and learns an object representation to embed the objects with a limited number of latent variables. This representation is a) well-disentangled, meaning that each variable independently captures one factor of variation in a detected object, b) identifiable, allowing the mimic tree to uniquely extract the underlying causal relation from DRL models, and c) interpretable, in order to generate understandable DRL explanations. The interpretability is demonstrated with illustrative examples of latent traversals and human evaluation.
2) A Monte Carlo Regression Tree Search (MCRTS) algorithm that learns mimic trees based on the latent features from the object representation. MCRTS incorporates the IB principle into its search heuristic by deriving a Minimum Description Length (MDL) objective. Unlike previous tree learners that deterministically select split by evaluating only its local influence, MCRTS looks ahead to global mimic performance and generates a compact distribution of candidate trees with explorations.
Our empirical evaluation shows that IMONe++MCRTS achieves promising mimic performance with significantly fewer splits than other baseline models and an important reduction in the size of tree interpretation. To demonstrate how the learned mimic tree makes its target action-advantage function interpretable we show causal relations and results of counterfactual interventions can be extracted from it. We validate the tree interpretations by 1) comparison with previous interpretable RL models, and 2) a survey that collects human evaluations.
Contributions. 1) We propose a RAMi framework that enables representing the object information with our novel IMONet and searching for the optimal mimic tree with the MCRTS algorithm. 2) We derive an cnormatiionthhoretti IB-MDL objective that incorporates both the fidelity and simplicity for mimic tree learning. 3) We introduce our method of leveraging the mimic tree to compute feature importance and extract causal relations from DRL model.
2 Represent and Mimic Framework
We propose a Represent And Mimic (RAMi) framework based on the Information Bottleneck (IB) principle. RAMi separately interprets the feature extraction and the decision making process of DRL model with 1) an interpretable latent representation and 2) a transparent mimic tree.
2.1 Mimic Learning for DRL
RAMi applies mimic learning 11 to learn post-hoe interpretations for a DRL model by transferring its knowledge to mimic tree Φ. To facilitate the knowledge distillation, mimic learners utilize soft-outputs from deep models as targets to supervise the training process of mimic trees. DRL models typically compute value functions (V(s) or Q(s, a)) estimate the expected cumulative rewards at state s. These value functions explicitly influence the RL decisions by controlling the policy gradient: Q(s, a V (als) [17] or determining the action: à = arg max, (ss, a) [8]. Compared to the policy π ( s), value functions evaluate candidate actions or states more directly. Based on the value functions, one can compute an advantage value for each action by subtracting a baseline: ((s a) [= which creates unbiased action-specific evaluations with less variance. Our mimic learner utilizes action advantages as the mimic targets, allowing us to understand when an action can outperform others by a certain advantage.
Action advantages can be defined V(s') +r V(s) if Q function is unknown (e.g., in A3C).

---

2.2 IB Method for Representing and Mimicking
We derive an objective for our RAMi framework based on the Information Bottleneck (IB) princi- ple [16]. An ideal mimic tree achieves a promising approximation performance (fidelity) with a minimum number of splits (simplicity). The IB objective naturally integrates both goals, by encourag ing a bottleneck representation to compress the input signals X and preserve as much of the relevant information about mimic targets Y (action advantages) as possible. The IB objective is:
max [T) (T,
where I denotes the mutual information, and À is Lagrange multiplier. -w (Ø, X) controls how much the mimic model compresses the input data, and I (#, Y) measures how well the mimic model preserves information about targets The IB principle defines what we mean by a good representation, in terms of the fundamental trade-off between having a concise representation and one with good mimic performance [19]. However, in practice, is difficult to learn mimic tree from high-dimensional and entangled raw inputs (e.g., images and text embeddings), so we learn a latent object representation from input space and then build mimic upon latent features. Inspired by the deep variational IB [19), The objective for our RAMi framework is given by the following result.
Theorem Consider dataset size N with input features x and prediction target Let (Bm)
log qlyn ()) denote the description length for encoding the target you let cx(()) = log po(Φ) denote the description length for encoding the mimic model &. Optimizing the IB objective is equivalent to maximizing its lower bound, which is:
N
ce we..) 1or AD x
Bg(= m..) Bi((blz) (C,( + xC,(O)) - {H(o(()))]]
(2)
ELBo objective + IB-MDL objective + Entropy Regularizer
The proof can be found in the appendix. This lower bound defines two sub-objectives: 1) the Evidence Lower Bound (E) objective enables learning an object representation for inputs {(][ with our IMONet (Section 3. 2) The IB-MDL objective, for which we design MCRTS algorithm (Section 4 MCRTS learns a compact distribution over mimic trees q( where we select the optimal mimic tree & to achieve "sweet-spot" between fidelity and simplicity.
3 Learning Object Representation
This section introduces the Identifiable Multi-Ohject Network (IMONet). The observed inputs from the RL environment are sate-action-rewart triplets nn = s.,, an rn]. Actions and rewards are often well-disentangled in low dimensions, so our IMONet learns a disentangled latent representation for capturing the object features from states. We 1) introduce the key properties of IMONet that enable extraction causal relationships, 2) describe the model structure and 3) discuss the interpretability for the learned latent representation.
3.1 Disentangled Representation for Causal Interpretation
The high-dimensional state features from the RL environment often entangled (e.g.. highly correlated pixels in an image), which makes causal inference from the raw input space impossible. IMONet utilizes a Variational Auto-Encoder (VAB) framework (20] that converts state features to a disentangled latent representation (i.e., p(Z) I2 P(Z))). After optimizing a generative model for the observed states that utilizes the latent representations, the independent latent variables capture complete state information. The independence implies that there no (unobserved or observed) confounder between two latent variables [21 22) This property enables mimic learners to model causal relations between latent variables Z and the target variable Y following a Causal Decision Tree [23]. (Section 5.2 shows examples of causal relations). Since the latent representation encodes complex RL states with few latent variables, the efficiency of causal inference is improved significantly.
2x. Y, Z. and & are random variables; æ. z, and & are instances of random variables, so p(X) represents distribution while p(æ) defines probability. (-) is the functional parameterized by
3

---

3.2 Identifiability of Latent Representation
Identifiability is one of the most fundamental properties for uniquely identifying the causal relations between imputs and targets. If a latent representation is unidentifiable, there exists multiple distri- butions with the same generative performance, and the variables between distributions can be completely entangled, meaning that they develop inconsistent factorizations for the same inputs [24] (see Section E.1 in appendix for examples). With unidentifiable representations, the mimic learner can return many different causal relations between the latent and target variables.
To better understand this problem, we define identifiability in the unsupervised representation learning setting as follows: Z(Z) is identifiable up to equivalence ~, Vs E S. p(s) = ((){=)p((2))) (T(p(Z')) + c, where function computes sufficient statistics of a distribution, is a vector, and B is an invertible matrix (that represents bijective mapping), If B is a permutation matrix, we have identfiability up to a permutation, otherwise inductive bias (e.g., specific
p((] (222))((2)))))))))) )22 = 3B,C s.t. ())(
up to scaling 225 Previous work (24] proved that without including model designs or additional observations), it is impossible to learn identifiable representations without supervising signals. To ensure the identifiability of the latent object representation, IMONet 1) utilizes a Multi-Object Network (MONet) [26] structure that is specially designed to capture variations of objects, and 2) employs conditionally factored prior p(Z [A, R) II pf Za .A, R) where each variable Zμ has a univariate Gaussian distribution. Following an Identifiable VAE (IVAE) design 25 IMONet learns conditional approximate posterior q( (Z SS, A, R) the state representation.
0:2
0.55
0.50
moc
26
moc
05
0.40
15
20
20 90 25
25 25 Running Slep (by 10
Running Slep (by 10
Running Step (by 10K)
Figure 1: Mcan Correlation Coefficients (MCC) for the examined variational encoders (including VAE, Condi- tional VAE, MONet and IMONet) trained in the Flappy Bird (left), Space Invaders (middle), and Assault (right) environments. We report the variance MCC scores computed from three independent runs.
Evaluation. We evaluate the identifiability of IMONet by comparing its Mean Correlation Coefficient (MCC) with other variational encoders. MCC measures whether it is possible to identify latent variables from one model with variables from another model (up point-wise transformations). To compute MCC, we 1) independently train two encoders based on the same structure, 2) pair latent variables from the encoders, by solving a linear sum assignment problem for the best overall correlation, 3) average the correlation coefficients between paired latent variables at different training steps. The results (Figure [ demonstrate that conditioning variables and an object network can consistently improve the identifiability of latent representations, whereas VAEs show no evidence of identifiability. This finding is consistent with 24 25
3.3 Model Implementation
IMONet decomposes a state into object-level components with spatial masks (m1..., mm) from a multi-layer convolutional attention network (m2 indicates whether the 5th object is present in each pixel, see examples in Figure Conditioning on m, r, and a. IMONet independently encodes each component with an encoder-decoder Conditional VAE (CVAD). To train IMONet, we decompose the ELBo from our IB objective (Theorem 1) into an object-oriented variational lower bound:
(3)
log mapa(s|zz) 3Dx (II geme (x a, mm) (a,
XDRL (gimuls, a, r))
where 1) the first term is decoder log-likelihood that defines the reconstruction performance for mixtures of decoder outputs. The state decoder pa(s 2xx) implemented by deconvolution layers. 2)
The second term is the KL-divergence (KLD) between the conditional approximate posteriors for

---

each component and conditional prior. The approximate posterior is implemented by: gene 8, O, mp) :- (ug diag(o))) s.t. 1Mp: v/ [yboone (8, me), , si [a,r)]
(4)
shmdpp and v/ are multi-layer perceptrons while sioone denotes convolutional network. The prior p(zla, a. is implemented by another Gaussian function without grony. 3) The last term defines the KLD between the mask generated by the attention network and the mask reconstructed by the CVAE. Our attention network q( mfm)) s, a, r) is implemented by U-Net [27]. which is implemented layer-by-layer convolutional neural network with instance normalization, followed by sigmoid function that transfers logistics to attentions. The mask decoder pa(ma (2x)) applies the same structure state decoder.
Learning an object representation for the RL states is applicable in practice since 1) a RL state can typically be represented by limited number of objects, and 2) the agent can interact with RL. environment to generate sufficient data for learning such a representation. Object-Oriented RL [28 29 30] has demonstrated that agent master RL games by modeling the object dynamics, so we utilize features from the object representation as the inputs to mimic learners: we sample latent features & from each component posterior Jene [S 8, A a, R M my), concatenate them with the conditioning values, and feed the entire vector [= I....., a, r] to MCRTS.
3.4 Interpreting Latent Variables
IMONet learns a symbolic abstraction of state space by representing object variations (e.g.. shapes and locations) with latent variables. A latent variable Zk,d (the dth variable for the 1 object) captures an independent factor of object variations. A common approach to reveal the information captured by Zx.a is latent traversing 26 31 31]: we randomly choose images and inspect how the reconstruction components change as Zu.a a is traversed from (empirical) minimum to maximum values. Figure iillusraaees the objects captured by attention masks from IMONet and shows two examples of latent traversals in a flappy bird environment. We include human evaluation to demonstrate the interpretability of latent variables (Section 5.2p. The appendix includes more examples (Section E.3 )
Latent Traversals
Syeat
m2
m?
min,
max(2,.)
Ei M
$1
$
Figure Visualizations for IMONet outputs. IMONet decomposes (for the background), my (for pillars) and ms (for the bird), where white/dark colors mark captureed/uncaptured regions. The generations from the decoder are $1, 82, and 83. We show two latent traversals [33 for interpreting
Sreat into three objects with masks
Za.1 (the sf variable for the 3dd object) and Z2.3 (the 3rd variable for the ndd object). Za,1 captures the vertical distance between the bird and pillar, and Z2.3 captures the length of the left pillar (highlighted by blue frames).
Learning Mimic Tree Interpretations
This section introduces the approach infer mimic tree with the IB-MDL objective (in Theorem and Monte Carlo Regression Tree Search (MCRTS) algorithm to find the optimal tree.
4.1 Inferring Mimic Trees with IB-MDL
We define (x) and (y,)) in the IB-MDL objective for mimic tree inference. The goal of MDL is to find regularity and simplicity the data by viewing learning as a compression problem. There are two agents with complete copies of the latent features 2n, but only the sender agent knows the target labels Va- The sender agents transmits the receiver agent a description of the missing labels using as few bits possible;
mimic tree to encode the information. The optimal mimic tree is
5

---

defined to be the one that enables sending the minimum description length of (a) encoding the tree structure C,(Ø), and (b) encoding the exceptions C. (yn) at each leaf node.
(a) Encoding Tree Structure: We convert the binary tree structure to string that records the splits (f) and leaf predictions (i) (e.g., 1, fi, ,, O, Do, O, U1 0, D2 where mark splits and leaves with 1 and 0) with depth-first search 34 The next proposition gives the encoding cost (6)
Given a regression tree with L splits, the total cost (in bits) of describing the tree structure with the string encoding method is:
Proposition 1
(2T. 1)2
L
()( = llogg
( 1) + -2L-
2L 21-1 + +
(5)
(b) Encoding the Exceptions: Traditional MDL tree algorithms
35 are proposed for classification
trees to handle discrete labels. Their exception-encoding methods are infeasible with continuous predictions, so we utilize an alternative approach that models the distribution of action advantages at a leaf node i with Gaussian distribution N (0)"0(()) 6, ())). Given a mimic tree o, ge and of model the prediction and the scale of exception at the leaf respectively. Inspired by the variance reduction objective in classic regression trees, we utilize the log- aag-anei to measure the cost of
encoding exceptions, so Ep(,..) IC [l]]] == log
Σ
/++1 N i= N
o
where N and Nylea denote the
total data size and the number of data points the tt leaf node respectively.
4.2 MCRTS Implementation
MCRTS (Figure takes latent fea- tures 2n R R (K and D de- note the number of objects and latent size respectively) from our IMONet and the conditioning values an, I' as inputs. It explores different splits by maintaining multiple candidate mimic trees that are trained to approx- imate action advantages Y. To initi- ate a tree search, MCRTS stores all the instances (data points) cell((21, a1, T1; 31), (z2, a2, r2: y2), (23, a3 T3i ys)...... ...)...) at the root node. An edge in the MCRTS search tree represents a split f in the mimic tree. This binary split passes the instances from a parent cell to two partition cells in child nodes. Each MCRTS edge records number of visits (V f) and an estimate QMC (J, f) of the expected splitting influence. Each node J contains a series of partition cells constructed by splits from the root node: J : (cell, ((z1, a1, 7 31); (23, 032 F3; ya),,............ ... cella((22, a2 r25 y2), (=1, a4 F4i ...,,
Jo (I, F, (xy), ag.r21 (5.)...))
Visiling number: value:
NY, (6.6))
N%, (6.
NY, 06 ) 0.e U..
My. Co - 0
0t (6..6')
)
2
16's <
86 16 < x
-6
N
cell,)
Jr = frelif pell
a cell
Figure 3: MCRTS structure, where tree node records partition cells and edge stores number of visits NV and estimate QJJ, f). S : Z. < denotes the kth split at layer which checks whether Za is smaller than a value v. The layer number equals the number of splits in a path. "fi denotes splitting ends.
,...,),,ellg(................... cellg(...)) ). MCRTS learns a compact distribution of mimic trees p( /b (2) A mimic tree Φ E 4 can be extracted from MCRTS by following path from the root to a leaf node Jh. so Φ = -o.oo.off......... J1). A partition cell; in Jj defines a leaf node in the mimic tree Φ and the leaf prediction is 0.rec=- Execelt,, M/N, M/NN/eaf Appendix (E.1) provides mimic tree extraction example.
Searching: MCRTS implements a tree search by running M plays from a starting node J. (which is initialized to be the root node and updated after each move) with root parallelization (36). At the mhth play, MCRTS implements four phases including:
1) Selection: Traverse the tree from Jj leaf node by selecting the split ft.m at each layer to maximize the Upper Confidence Bound (UCB) [37]:
ft.m arg max,
MC
Vlog(m-1) 1 f) + Cpuct NV (J:
(6)
where Cyact controls the scale of exploration. Similar to the entropy regularizer (Theorem 1), large Cpuct prevents over-frequent visits to node. 2) Evaluation: Evaluate the selected leaf
Jieaf with reward: (Jteaf) (yn( XL((d). 3) Expansion: Expand the leaf node

---

Table I: Mimic performance. We use the Variance Reduction (VR) and Variance Reduction Per Leaf (VR-PI.) metrics since they common regression objectives and optimized by the IB-MDL objective. Results for other regression metrics (Root Mean Square Error and Mean Absolute Error) and the corresponding variances are
recorded in Table B.2 in appendix. W4 indicates that each leaf node has an extra linear regression model. We
omit the results for the raw-input -nsed based MCRTS since it is computational intractable.
Flappy Bird
Space Invaders
Assault
Method Carl VIPER
VR-PL
Leaf
VR
VR-PL
Leaf
VR
VR-PL
Leaf
8.51E-2
8.430.5
1007
4,96E-2
7.02E-5
705
4.79E-2
7.461-5
642
8.57F-2
1.88F-4
453
4.63F-2
8.80F:5
525
5.28F-2
8.09F-5
653
M5-RT
9,59E-2
8.37B-5
1144
4,54E-2
2.92E:5
1558
4.37E-2
2.73E-5
1605
M5-MT
9.56E-2
1.55E-4
612%s
160E-2
1.23E:5
1303
3.42E-2
2.54E-5
1351
GM-LMT
8.99F-2
2.99F-4
303
2.07F-2
8.32F-5
249"s
5.55E-2
1.83E-4
307 44
VR-LMT
8.46B-2
5.36E-4
157
2.65E-2
1.61E4
166
5.80E-2
1.98E4
291 W1
VAB+CART
7.25B-2
"3.4E--
212
3.99E:2
7.8GE:5
"SOT"
5.ISE:2
I.IGE-4
448
VAE+VIPER
7.63E-2
5.32R-4
143
4.12F-2
9.89E-5
417
4.57E:22
1.29E-4
356
VAE+GM-LMT
6,35B-2
3.51E-4
180
3.39E-2
2.75E-4
123"4
4.20E-2
1.44E-5
293
VAE+VR-LMI
7,95-2
5.12E-4
154
3.52E-2
2.08E-4
171
5.10E-2
1.99E-4
258
VAE-MCRTS
7,83E-2
1.27E-3
61
4.82E-2
5.66E-4
85
6,58E-2
7.75E-4
85
MONNC--CRRT
8.23E-2
4.02E-T
204"
5521EE-2
133E--
335
3.67E-2
TSE4
315
IMONeL+
8,50E-2
4.48E-4
191
5.26E-2
1.69-4
313
6.05E-2
1.90E-4
319
IMONct+GM-LMT
7,87E-2
3.74E-4
212
4.79E-2
3.23E-4
149"
5.45E-2
2.15E-4
256
IMONeL- VR-LMT
8.211-22
7.160-4
115%
4,54E-2
3.79E-4
120"
6.03E-2
2.271-4
268
IMONS
8,53E-2
1.37E-3
62
5.37E-2
7.08E-4
76
7.53E-2
9.07E-4
83
with G (the maximum exploration width) child nodes. 4) Back Up: Update the action-values:
QMC = (QMSI - ( on all the traversed edges.
and increment the visit count: N Vm = -V
Move: After M plays, we move J. to child node by selecting the split f. = arg f.exi.en.... s(m))
NV (Ja: fae and setting J, to a child node J,+1 (connected by the edge fa). The next play will start from the new starting node J&+1. It allows MCRTS to prune the sub-optimal nodes (38]
Empirical Evaluation
We evaluate the mimic performance and demonstrate the interpretability of the mimic tree.
Environment and Running Settings: We study the Flappy Bird, Space Invaders, and Assaults environments. Flappy Bird is procedural game, where the game states randomly generated at each episode. Space Invaders and Assaults are commonly studied Atari games from the Gym toolkit [39.. (Check data generation details and model hyper-parameters in Appendix).
We compare previous tree-based mimic learners. The first baseline model is a Classification and Regression Tree (CART) ) [40] We include VIPER 41 our second baseline by replacing its policy (decision) tree with a regression tree that imitates action values with a Q-dagger algorithm. The third baseline is the M5 42 regression tree training algorithm. M5 constructs piecewise constant tree by pruning the sub-optimal nodes. We include M5 with the Regression Tree option (M5-RT) and the Model-Tree option (M5-MT). M5-MT builds linear regression model for the instances at each leaf node while M5-RT maintains only a constant value. The last baseline Linear Model Tree (LMT) [12 for interpreting action values. A recent work [13] explored different heuristics for selecting splitting features, including Variance Reduction (VR-LMT) and Gaussian Mixture clustering (GM-LMT). The aforementioned models are directly learnt from raw input space (states and actions) [4). The aforementioned models were previously learnt directly from raw input space (states and actions). In this study of latent representation, we evaluate their regression performance based on the latent features from both VAE and IMONet (Check appendix for the implementation details of baseline models).
Baseline Models:
5.1 Fidelity versus Simplicity
This experiment evaluates mimic trees by their 1) fidelity: how well a mimic model approximates the DRL model [12 and 2) simplicity: the size of these nonparametric trees. We divide the dataset (50k) into training (80%), validation (10%), and testing (10%) sets and generate independent runs. Fidelity and simplicity are evaluated by the mimic performance and the number of leaves respectively.
7

---

Table shows the regression performance. MCRTS+IMONet achieves promising regression performance with significantly fewer leaves than other baseline models. This is because IMONet learns concise object representation from RI. states and this representation outperforms that modelled by an unidentifiable VAE. 2) MCRTS selects splits to maximize the regression performance at global level, yielding mimic tree with better fidelity. Apart from these obvious advantages, we find that some latent features may be indistinguishable for predicting action values, and this explains why some trees have better regression performance by building large tree from raw inputs, while the large tree size compromises their interpretability.
Leaf-by-Leaf Regression Performance: To study the regression efficiency, we evaluate the per-leaf regression performance of examined models based on the latent features from IMONet. Figure illustrates the performance of examined mimic trees when we constrain their number of leaves. MCRTS achieves a leading regression performance, as it selects a split by looking ahead to the future cumulative rewards instead of its local influence. Another key observation is that well-explored split from MCRTS can beat a greedy split with extra linear regressors leaf nodes (e.g.. LMTs), allowing MCRTS to substantially improve the learning efficiency.
0.00
0.00
Veriencs Reduation
Varianne doduudtonn
Venincc Rodk cotuuctoo
VS
6.04
6.03
0.00
0:00
0.00
Leaf Number
Leaf Number
Leaf Number
Figure 4: Leaf-by-leat tree regression results based on latent features from IMONet in the Flappy Bird (left), Space Invaders (middle), and Assault (right) environments.
5.2 Illustrative Examples of Interpretable Mimic Trees
We demonstrate the interpretability of our mimic tree by 1) illustrating the extracted rules and causal relations, 2) comparing with previous RL interpretations and 3) conducting human evaluations. This section uses the Flappy Bird environment as an example, where DRL agent earns rewards by controlling a bird to pass pillars.
cell
f foachon 'dorr??
cell fe
cell,
00.1
Plad"baby Tbeb)
Sample
Sample
cella:
&e 2 2012
collo2 fills
Intervened
Sallengy Rigion
f3l4a 071,
de 071,
cellan
Fel Res 2.177
Celtl
celle
Fst Zas 0.71
celles
exilyy
SEAM 0.17
8 0.00
90tt 0.15
Ragreescnn Trmes
Counerfacuats
Figure 5: Mimic tree. ft Zu.d v indicates the 1 tree split at Zu.s (d variable for the 8tth object) with splitting value v. In the decision rules (b&c), solid dash lines indicate tree path / causal relations.
Causal Explanations: Our mimic tree & captures the influence of latent features action advantages y. Figure 15 (a) illustrates the top three layers of the mimic tree. We can interpret the influence of split by visualizing the expected latent values after this split. For example, after implementing the split f1, we obtain two child nodes (cell2,1 and cell2,2). We compute the expected latent values for the instances on these nodes (Engceli( (Zn)) and visualize the 3rd object with the decoder (since 7 captures the variation of the 3rd object). These visualizations capture the difference of object information after splitting. They show that the instances celly,2 have shorter (vertical) distance

---

between the bird and the upper pillar, and thus their advantages for the action 'down' are larger. We verify this observation by sampling an instance from both child nodes respectively. This observation is consistent with game rules since performing the action "down' when the bird approaches the upper pillar can prevent potential crashes and thus has larger advantage.
(a) Causal Relation Extraction. Since there is no (observed/unnobserved) confounder between latent variables (section 3.1) our mimic tree can capture causal relations between splitting features and predictions. Similar to Causal Decision Trees (CDTs) 23 a tree split ff in our mimic tree represents a context- context-specific causal relationship between the splitting condition ff and the action advantage jf The context is a series of value assignments of the latent features along the path from the root to the parent node of f. For example, the decision rule in Figure 55 states that having Z3.1 2 0.71 is casually related to an action advantage of 0.23 when the action is 'down' and Z 2 0.12 (context).
(b) Counterfactual Analysis. Leveraging the generation ability of the IMONet decoder, our mimic tree enables counterfactual analysis to answer "what if" questions: we show the influence on tree prediction by intervening to set the value of Zx.dd equal some particular value, For example, in Figure ce)), we compute Loo fi,..., ft. X', (g =a,1=0)) =0]] 0)]. To achieve this, 1) we set Za,1 to for all input latent vectors and get z'. 2) The IMONet decoder computes pa( ((') and the DRL model predicts Yj with the generated observations X'. 3) The mimic tree follows previous splits to construct tree predictions (the splits on the intervened variables are removed). The difference of tree prediction before and after the intervention (0.23 versus 0.11) reveals the influence of (o(23,1 = 0).
Comparison to other interpretations. We compare other interpretable DRL models based on 1) Super-pixels [12) (Figure 5d) that highlight the pixels used as splitting features in a regression tree, 2) Saliency Map 6 (Figure which is generated by perturbing the pixels within region and evaluating their impact on target outputs, and 3) Attention Mask 4 (Figure 5D (from the convolutional attention network) that represents the importance of pixels on model outputs. These traditional models used the raw images as input during training. The learned interpretations visualize the pixels that are important for decisions. They generally agree that the distance between the bird and the pillar is most influential. This observation is consistent with rules extracted by our method (Figure b))b)) since this distance determines how likely the bird can pass the pillars win more rewards.
Human Evaluation. We evaluate the interpretability of the aforementioned methods with RL practitioners from four universities. We first introduce the saliency map, mimic trees based on super- pixels and latent variables, and then ask the participants to 1) describe the information captured by these methods and 2) rank the corresponding interpretations. To eliminate potential bias, participants are interviewed independently, and they are not told which methods are the baselines or the proposed model (check Section D in appendix for further details). We present a summary of the results by discussing the two aspects of interpretability regarding our method:
(a) Interpretability of Latent Variables. After observing the latent traversals, all (12/12) participants correctly recognized the physical meaning of the latent variables whereas 58%(7/12) and 33. participants managed to recognize the features captured by saliency maps and pixels due to the inconsistency of local samples and the complexity of tree paths.
(b) Interpretability of mimic trees. 83% (10/12) participants preferred the mimic tree based on latent features since its tree path captures globally-consisennt knowledge and its tree size is smaller than that of the super-pixel tree (62 versus 1007, check Table The rest voted for the saliency map because they preferred froight-forward explanations.
5.3 Limitations
Computational Complexity. Since MCRTS conducts multiple simulations to explore and generate the global-optimal mimic tree, it generally consumes more computational resources than classic re- gression. We compare the computational complexity of our method and the baselines in the appendix. Although IMONet compresses dige--imensional inputs into latent vectors and we parallelize the tree search in MCRTS, empirically, it takes MCRTS longer determine split.
Guarantee for Interpretability. It generally hard to theoretically justify or numerically quantity the level of interpretability, so we utilize illustrative examples and human evaluation to empirically

---

show the interpretability. Although previous works demonstrated interpretability of latent features [26] there no guarantee that these features always have a clear meaning across all RL games.
31 32,
6 Related Works
We introduce previous works for interpreting DRL models.
Seff--xplannble Approaches: Some recent works have explored self-explainable DRL models. To derive human-underssanddbbe policies, these models modified existing performance-driven DRL models (c.g.. DQN) according to some specific designs, such as symbolic planning [43], decision
trees 44 45 Generalized Value Functions (GVFs) 146), Moore Machine Networks (MMNs) (77)
transparent decision dynamics [48] or explainable feature extraction 4 44]. These modifications
influence the decision process of DRL agents to enhance interpretability.
Post-hoc Interpretation: An alternative approach is generate posterior explanations for DRL models to achieve post-hoc interpretability. Existing methods commonly apply feature perturbation to generate saliency maps G or learn attention masks SS] 50 with the attention neural network. These explanations are driven locally, based computing a localization map that highlights the important regions for current actions. is hard for these point-wise interpretations maintain their consistency with explanations from other samples, which leads to unstable explanations [10). Mimic learning (i.e., model distillation) [I]] enables the generation of globally consistent explanations from performance-driven DRL model. Previous works [13 12 utilize mimic regression trees to interpret action values from DRL models, but their training objective did not control the model complexity and thus encouraged oversized mimic trees. Another recent work [51] proposed a tree regularizel that adjusts the loss function of deep models by integrating the regression performance of mimic trees. A recently proposed VIPER model [4] designed Q-dagger algorithm to derive tree-based policy function following online imitation learning [52]. Our experiments evaluate an updated VIPER model for estimating action advantages.
7 Conclusion and Societal Impacts
This paper introduced a RAMi framework based on the IB principle, following which we 1) introduced an IMONet to learn object representation for RL states and 2) proposed a MCRTS algorithm to find mimic tree with promising fidelity and simplicity. Experiments demonstrated the leading mimic performance of our MCRTS+IMONer model. We illustrated causal relations and human evaluation. A potential negative societal impact of our work is that it encourages the government to endorse laws that regulate the level of transparency in Al systems. This may increase the difficulty of deploying advanced Al systems.
tree interpretations with
Acknowledgments and Disclosure of Funding
This project supported by Strategic Project Grant from the Natural Sciences and Engineering Research Council of Canada and MITACS Research Training funding. Our computations were facilitated by a GPU donation from NVIDIA. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 5 1877550):529-533 2015.
[2] Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Çaglar Gülçehre, Ziyu
10

---

Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature, 575(7722)3300-3335 2019.
[3] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, llya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. CoRR, abs/1912.06680, 2019.
[4] Wenjie Shi, Shiji Song, Zhuoyuan Wang, and Gao Huang. Self-supervised discovering of causal features: Towards interpretable reinforcement learning. CoRR, abs/2003.07066 2020.
[5] Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, Vancouver, BC, Canada, 8-14 December 2019, pages 12329-12338, 2019.
[6] Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understand- ing atari agents. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüsssnn Stockholm, Sweden, July 10-15, 2018, pages 1787-1796, 2018.
[7] Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding dqns. In Proceedings of the 33nd International Conference on Machine Learning. ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 1899-1908, 2016.
[8] Sarthak Jain and Byron C. Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLL 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 3543-3556, 2019.
[9] Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNUP 2019, Hong Kong, China, November 3-7, 2019, pages 11-20, 2019.
[10] David Alvarzz-elis and Tommi S. Jaakkola. On the robustness of interpretability methods. CoRR, abs/ 808.000000899 2018
[11] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, Neurips 2014, Montreal, Quebec, Canada, December 8-13 2014, pages 2654- 2662, 2014.
[12] Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingcan Li. Toward interpretable deep reinforce- ment learning with linear model u-trees. In Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2018, Dublin, Ireland, September 10-14, 2018, Proceedings, Part II, pages 414-429, 2018
[13] Xiangyu Sun, Jack Davis, Oliver Schulte, and Guiliang Liu. Cracking the black box: Distilling deep sports analytics. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Virtual Conference, August 23-27, 2020, 2020.
[14] Olcay Boz. Extracting decision trees from trained neural networks. In Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Edmonton, Alberta, Canada, July 23-26, 2002, pages 456-461, 2002.
[15] Darren Dancey, Zuhair Bandar, and David McL.an. Logistic model tree extraction from artificial neural networks. IEEE Transactions Systems, Man, and Cyberneties, 37(4):794-88) 802, 2007.
11

---

[16] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004055, 2000.
[17] Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information Processing Systems 12: Anmual Conference on Neural Information Processing Systems 1999, Neurips 1999, Denver, Colorado, USA, November 29 December 4, 1999, pages 1018-1014, 1999.
[18] Martin A. Riedmiller. Neural fitted iteration first experiences with a data efficient neural reinforcement learning method. In 16th European Conference on Machine Learning, ECML 2005, Porto, Portugal, October 3-7, 2005, Proceedings, pages 317-328, 2005.
[19] Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
[20] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[21] Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Structured causal disentanglement in variational autoencoder. CoRR, abs/2004.08697 2020.
[22] Bernhard Schölkopf. Causality for machine learning. CoRR,
2019.
[23] Jiuyong Li, Saisai Ma. Thuc Duy Le, Lin Liu, and Jixue Liu. Causal decision trees. IEEE Transactions on Knowledge and Data Engineering, 29(2):257-271, 2017.
[24] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, Long Beach, California, USA, 9-15 June 2019, pages 4114-4124, 2019.
[25] Ilyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, and Aapo Hyvärinen. Variational autoencoders and nonlinear ICA: unifying framework. In 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, Online, 26-28 August 2020, pages 2207-2217, 2020.
[26] Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. CoRR, aass//91.1.. 1390, 2019.
[27] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Inter- vemtion - MICCA/ 2015 18th International Conference Munich, Germany, October 5 9, 2015, Proceedings, Part III, volume 9351 of Lecture Notes in Computer Science, pages 234-241. Springer, 2015.
[28] Carlos Diuk, Andre Cohen, and Michael L. Littman. An object-oriented representation for efficient reinforcement learning. In Proceedings of the 25th International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008, volume 307 of ACM International Conference Proceeding Series, pages 240-247. ACM, 2008
[29] Vikash Goel, Jameson Weng, and Pascal Poupart. Unsupervised video object segmentation for deep reinforcement learning. In Advances Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montréal, Canada, December 3-8, 2018 pages 5688-5699, 2018
[30] Thomas N. Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
12

---

[31] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholnsmässan, Stockholm, Sweden, July 10-15, 2018, pages 2654-2663, 2018.
[32] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a con- strained variational framework. In 5th International Conference Leurning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
[33] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
[34] J. Ross Quinlan and Ronald L. Rivest. Inferring decision trees using the minimum description length principle. Information and Computation, 80(3):227-248, 1989.
[35] Manish Mehta, Jorma Rissanen, and Rakesh Agrawal. Mdl-based decision tree pruning. In Proceedings of the First International Conference Knowledge Discovery and Data Mining (KDD-95), Montreal, Canada, August 20-21, 1995, pages 216-221, 1995.
[36] Guillaume Chaslot, Mark H. M. Winands, and H. Jaap van den Herik. Parallel monte-carlo tree search. In Computers and Games, 6th International Conference, CG 2008, Beijing, China, September 29 October I, 2008. Proceedings, pages 60-71, 2008.
[37] Adrien Couëtoux, Jean-Baptise Hoock, Nataliya Sokolovska, Olivier Teytaud, and Nicolas Bonnard. Continuous upper confidence trees. In Learning and Intelligent Optimization Sth International Conference, LION 5, Rome, Italy, January 17-21, 201 Selected Papers, pages 433 445, 1.
[38] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354-353) 2017.
[39] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.011400, 2016.
[40] Roman Timofeev. Classification and regression trees (cart) theory and applications. Humboldt University, Berlin, pages 1-40, 2004.
[41] Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montréal, Canada, December 3-8, 2018, pages 2499-2509, 2018.
[42] John R Quinlan et al. Learning with continuous classes. In 5th Australian joint conference on artificial intelligence, volume 92, pages 343-34-3 -388. World Scientific, 1992.
[43] Daoming Lyu, Fangkai Yang, Bo Liu, and Steven Gustafson. SDRL: efficient deep reinforcement learning leveraging symbolic planning. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, AAAÍ 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, pages 2970 (900229777770 2019.
interpretable and data-
[44] Ujjwal Das Gupta, Erik Talvitie, and Michael Bowling. Policy tree: Adaptive representation for policy gradient. In Proceedings of the 29th AAAI Conference on Artificial Intelligence, Austin, Texas, USA, January 25-30, 2015, pages 2547-2555 2015.
[45] Aaron M. Roth, Nicholay Topin, Pooyan Jamshidi, and Manuela Veloso. Conservative q-improvement: Reinforcement learning for an interpretable decision-tree policy. CORR, Abs/1907.011800 2019.
[46] Zhengxian Lin, Kin-Ho Lam, and Alan Fern. Contrastive explanations for reinforcement learn- ing via embedded self predictions. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
13

---

[47] Anurag Koul. Alan Fern, and Sam Greydanus. Learning finite state representations of recurrent policy networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, IA, USA, May 6-9, 2019. OpenReview.net, 2019.
[48] Alihan Hüyük, Daniel Jarrett, Cem Tekin, and Mihaela van der Schaar. Explaining by imitating: Understanding decisions by interpretable policy learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, ,021,
[49] Tom Bewley and Jonathan Lawry. Tripletree: A versatile interpretable representation of black box agents and their environments. CoRR, abs/2009.044 04444, 2020.
[50] Anthony Manchin, Ehsan Abbasnejad, and Anton van den Hengel. Reinforcement learning with attention that works: A self sepf-supe approach. In Neural Information Processing 26th International Conference, ICONIP 2019, Sydney, NSW, Australia, December 12-15, 2019, Proceedings, Part pages 223-230, 2019.
[51] Mike Wu, Michael Hughes. Sonali Parbhoo, Maurizio Zazzi, Volker Roth, and Finale Doshi- Velez. Beyond sparsity: Tree regularization of deep models for interpretability. In Proceedings of the 32nd AAAI Conference on Artificial Intellience, (AAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 1670-1678, 2018.
[52] Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011, volume of JMLR Proceedings, pages 627-635, 2011.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5.3
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 7 For a detailed discussion of ethics statement, please check section F in appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Please check the assumption of the IB objective in the Theorem I and the assumption of causal inference section B
(b) Did you include complete proofs of all theoretical results? [Yes] For the complete proof of Theorem please check Section B in appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] Please check the attached code and Section A.2.2 in appendix for the data generation details.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please check Section A.2.3 in appendix for the implementation details (including hyperparameters) of our model and all baseline models.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] Please check Section C in appendix for the complete results (including mean and variance for 5 independent runs).
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please check Section A.2.7 in appendix for the details of computing resource. Please check Section A.2.4 in appendix for the computational complexity of our method and other basclines.
14

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the creators of Pygame learning environment and OpenAl gym toolkit in reference.
(b) Did you mention the license of the assets? [N/A] The assets creators posted the
instructions about how to cite their works. We are not aware of mentioning the licenses.
(c) Did you include any assets either in the supplemental material as URL? [N/A] We have not created new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We have used data from other people.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] Data from video game does not contain personal information.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] Check Section in appendix for the details of human evaluation. (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, upplicable? [N/A] Participants answered some questions regrading model evaluation. We did not collect any personal data.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] Participants are volunteers. This message was properly delivered before any interview.
15


Reviews:
[{"review_id": "rk1yPnRqiOi", "reviewer": "Reviewer_zwwU", "summary": "The paper proposed a new explanation approach for reinforcement learning.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The research is a little bit incremental. There are already many interpretation methods in the context of reinforcement learning. The work proposed is yet another method that generates explanation in higher fidelity. However, the evaluation of higher fidelity interpretation is insufficient. Some existing techniques are not compared (see below). \n\nIn addition to the methods based on interpretable trees, there are many other explanation methods for RL applications (e.g., saliency map-based approaches and attention-based approaches). The paper does not conduct an experimental comparison with these alternative approaches. It would be helpful to show the performance difference between alternative approaches and the proposed method.\n\nTo evaluate the effectiveness of the proposed method, the authors measure the fidelity of the model's capability in approximation. This, to some extent, indicates fidelity, but it does not imply a good mimicking performance is a good explanation. I am happy to see the authors conduct counterfactual analysis to assess the fidelity of interpretability. However, the authors only show a case but do not quantify the fidelity. I think it would be more convincing if the authors could follow existing and commonly adopted methods to assess the fidelity of the interpretation (e.g., blinding some critical component in the explanation and see how that influences the action of the target agent quantitatively). \n\nThe paper shows some interesting results on some simple games. I wonder if the proposed method is applied to more sophisticated games (e.g., starcraft or super Mario that have a more complicated background and foreground), will the proposed method still generate a decent latent representation for background and objects? It would be more promising if the authors could show results on more sophisticated games.\n\nI am happy to see the authors evaluate the explainability by conducting a user study. First, I am not sure if the authors received IRB approval before conducting this human study. Second, the human study is subjective. Even if an interpretation makes sense to a human, it does not imply it is the right explanation. Can the authors justify the validity of this human-based assessment?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concern", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "LYGjBFQkyGN", "reviewer": "Reviewer_nsW5", "summary": "This paper proposes a novel method of interpreting individual decisions of a DRL policy trained by deep Q learning. The explanation they want to deliver is to identify the important features within an input state that contribute most to the corresponding Q function output. Technically, it first learns an encoding layer to transform the high-dimensional input into low-dimensional hidden representations. To enable explainability, the encoding layer is designed to preserve the physical meaning of the objects in raw inputs. Using the hidden space representations, it proposes a Monte Carlo Regression Tree Search (MCRTS) algorithm to train a tree to mimic the action advantages.  To validate the effectiveness of the proposed model, the authors evaluate the proposed technique from two aspects: fidelity and interpretability. The evaluation on three RL environments demonstrates the superiority of the proposed method over some of the existing explanation methods. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is overall well organized and written. The proposed technique is interesting and clear explained. I like the idea of the two-step interpretation: hidden representation + mimic model, and I appreciate the authors' effort in keeping the identifiability and interpretability of the latent representations. They are very important properties towards providing explanations. The evaluations are relatively well designed and comprehensive. I appreciate that the authors include a relatively complete comparison with the baselines. IMHO, the paper would be more convincing if the authors could address my following comments.\n\nMy main concern is in the fidelity evaluation. I appreciate that the authors evaluate this important property. However, the current evaluation is not comprehensive. More specifically, approximation accuracy is essential for mimic-based explanations. IMHO, it is only the first step towards a comprehensive fidelity evaluation. More importantly, I would suggest the authors evaluating whether the derived explanations actually reflect the decision-making process of the neural networks. The authors could refer to some existing fidelity evaluation metrics used in the papers that try to derive important features as explanations[1,2]. The basic idea of these metrics is to perturbing the identified features and observing the changes in the corresponding NN's outputs. If the identified features are truly important ones, perturbing them will trigger a large change to the output. The authors could borrow this idea and design a fidelity metric to evaluate the explanation fidelity or just use the existing ones (If they are applicable). In summary, enabling an accurate approximation is only the first step towards achieving faithful explanations. The authors need to conduct another evaluation directly attached to the derived explanations. \n\nSome minor comments:\n\n1. I may miss this detail. I am wondering whether the MCRTS can be applied to the raw features. If yes, I would like to see the results. It could further demonstrate the effectiveness of the feature extraction step.\n\n2. The current version only showcases the explanations on one of the three environments. I would suggest demonstrating the explanations on the other two applications and conducting the human evaluation on those applications too.\n\n3. The proposed technique utilizes some existing methods to address some of the challenges. Among them, VAE is a well-known technique, which may not need an introduction in this work. However, [34] may not be that well-known. I would appreciate that if the authors could provide some background knowledge regarding these non-well-known techniques.\n\n======== Post rebuttal Comments===============\n\nAfter reading the authors' response, my concern about fidelity evaluation still stands. I would still suggest designing a quantitative fidelity evaluation w.r.t. the RL agents together with an evaluation metric. The authors could design their own metric by following the methodology similar to the counterfactual analysis (remove and rerun). With the metric, the authors could compare the fidelity of their proposed approach with that of existing explanation methods. As things stand, I would be inclined to maintain my current score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed two limitations in Section 6.3. Additional limitations and future works could be (1) The proposed technique can only be applied to environments using images as inputs; (2) Generalizability of the proposed technique to more complicated environments, multi-agent environments; (3) Generalizability of the proposed technique to policy networks.\n\nI appreciate that the authors include a social impact discussion. And I agree with the discussion points. To make it more comprehensive, the authors could discuss the possible negative impact from the adversarial point of view, .e.g, attackers may leverage the proposed explanation methods to develop stronger attacks.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "5QKuxVvXpp1", "reviewer": "Reviewer_SaJC", "summary": "The paper proposes learning decision trees over latent features using MCRTS to improve interpretability of Deep Reinforcement Learning.\n\nThe paper leverages the Information Bottleneck principle to find explainable trees with minimum description length. \n\nThe proposed method uses VAE to learn a disentangled representation on which to build regression trees that more easily interpreted.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality: \nThe problem of understanding and being able to interpret DNN used for RL is an important problem, both for developing and validating new ideas, as for increasing the safety and confidence on the learnt models.\n\nThe paper introduces a novel combination of VAEs with MCTRS to learn a more interpretable policy that can mimic the behavior of the DNN.\n\n# Quality: \nIt is not clear how IDOE is capable of learning an identifiable disentangling object encoder, and how identifiable and disentangled are the variables and whether they correspond to objects or not.\n\nMissing comparisons with Gradient Boosting Trees.\n\nThere is no validation of the claim that VAEs learn disentangled representations, or that the latent variables are interpretable by humans. \n\nThere is not proof that the latent variables learnt by VAEs are more interpretable than DNNs latent features.\n\nIn Section 4.2 it's unclear that the latent representation can capture objects and its relations in the different tasks.\n\nIn section 4.4 the claim that IDOE learns a symbolic abstraction of state space by representing object variations is not sustained. Showing 2 cherry picked examples are enough.\n\nIt is not clear how the number of objects and latent size was chosen and those impact the results.\n\nThe human evaluation with 12 participants is very weak. The paper didn't present a good protocol for human evaluation. For instance not clear how the participant were chosen and interviewed.\n\n# Clarity:\nThe is not very clear about its main contributions and its significance. The writing is not very clear or well organized. \n\nFor example section 4.3 is confusing on how the model is implemented and how the different CVAE are defined or learnt.\n\nNot explanation of why Flappy Bird, space Invaders and Assaults where chosen for the experiments. \n\nThe Figure 5 representing the decision tree is hard to interpret the changes on the outputs. For example how to read the  advantage of different actions (down vs up) on different branches.\n\n# Significance:\nThe paper claims a promising mimic performance, but not clear how good is the mimic policy at actually solving the tasks.\n\nThe paper shows some interesting and promising results, but its applicability to complex RL tasks is missing.\n\n\n# Minor issues:\nIn Figure 4, only the right figure has confidence bands around the different algorithms.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper briefly mention some limitations of the work in section 6.3, although focus on computational complexity rather than interpretability complexity. It is not clear if the proposed method increase a shallow interpretability of RL.\n\nThe claim that \"A potential negative societal impact of our work is that it encourages the government to endorse laws that regulate the level of transparency in AI systems.\" has no base, and also it is not clear it would be negative societal impact, instead of positive.", "review_text": "", "extra_scores": {"ethical_concerns": "It is not clear if proper ethical guidelines were followed while doing the human evaluation, for example, how the participants were chosen and how well informed they were.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "1p97BabXCgC", "reviewer": "Reviewer_8RF7", "summary": "This paper focuses on building a tree representation from pixels to evaluate interpretability in deep reinforcement learning. The proposed approach relies on features learned by a variational autoencoder. A mimic tree then extracts disentangled object representations for showing the causal impact. The method is evaluated on three game environments: Flappy Bird, Space Invaders and Assaults.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe paper is well-written. Increasing interpretability in learned features for RL is very useful for real-world tasks. The methodology and experiments are well explained.\nWeakness:\n* Line 18: as far as I know vision RL still remains a challenging task except some known environments such as atari. It would be useful if authors cite works outperforming human level control.\n* Line 188: Authors mention that one can collect enough data to learn object representation in RL. This in real world scenarios is not applicable and is the main reason researchers focus on sample efficiency.\n* Line 185: Can authors provide more details on how unet was used for attention network? This seems to make it computationally expensive.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors mention “A potential negative societal impact of our work is that it encourages\nthe government to endorse laws that regulate the level of transparency in AI systems. This may\nincrease the difficulty of deploying advanced AI systems.” as a negative impact which can be seen as positive in most cases.\n", "review_text": "", "extra_scores": {"ethical_concerns": "No general concern.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the problem of learning an interpretable version of the decision procedure used by neural RL, which reviewers agree is interesting. There was some disagreement about how substantial the results are, especially regarding fidelity and practicality. While the majority of reviewers ended up mildly positive, I believe this paper can be substantially improved by taking the comments to heart in guiding further experiments. 
Decision: Accept (Poster)