POETREE: Interpretable Policy Learning with Adaptive Decision Trees
Authors: Alizée Pace
Alex Chan
Mihaela van der Schaar
Venue: ICLR 2022 Spotlight (2022)
Keywords: Imitation Learning
Interpretable ML
Clinical Decision Support
Sequential Decision-Making

Abstract: Building models of human decision-making from observed behaviour is critical to better understand, diagnose and support real-world policies such as clinical care. As established policy learning approaches remain focused on imitation performance, they fall short of explaining the demonstrated decision-making process. Policy Extraction through decision Trees (POETREE) is a novel framework for interpretable policy learning, compatible with fully-offline and partially-observable clinical decision environments -- and builds probabilistic tree policies determining physician actions based on patients' observations and medical history. Fully-differentiable tree architectures are grown incrementally during optimization to adapt their complexity to the modelling task, and learn a representation of patient history through recurrence, resulting in decision tree policies that adapt over time with patient information. This policy learning method outperforms the state-of-the-art on real and synthetic medical datasets, both in terms of understanding, quantifying and evaluating observed behaviour as well as in accurately replicating it -- with potential to improve future decision support systems.

Published as a conference paper at ICLR 2022
POETREE: INTERPRETABLE POLICY LEARNING WITH ADAPTIVE DECISION TREES
Alizée Pace
Alex J. Chan
ETH AI Center, Switzerland ETH Zürich, Switzerland
University of Cambridge, UK
ajo3 4 400cam. ac.uk
MPI for Intelligent Systems, Tübingen, Germany al i zee pwceppcceeaae.eee weteeBai.ehhz..c. ethz.ch
Mihaela van der Schaar University of Cambridge, UK Cambridge Centre for Al in Medicine, UK The Alan Turing Institute, UK mv4720 cam. wwc.maacc..... ac. uk
ABSTRACT
Building models of human decision-makinn from observed behaviour is critical better understand, diagnose and support real-world policies such as clinical care. As established policy learning approaches remain focused on imitation perfor- mance, they fall short of explaining the demonstrated decision-making process. Policy Extraction through decision Trees (POETREE) is novel framework for in- terpretable policy learning, compatible with fully-offline and partially-observabbee clinical decision environments and builds probabilistic tree policies determin- ing physician actions based on patients' observations and medical history. Fully- differentiable tree architectures are grown incrementally during optimization to adapt their complexity to the modelling task, and learn representation of pa- tient history through recurrence, resulting in decision tree policies that adapt over time with patient information. This policy learning method outperforms the state- of of-the-an art on real and synthetic medical datasets, both in terms of understanding, quantifying and evaluating observed behaviour as well as in accurately replicating it with potential to improve future decision support systems.
INTRODUCTION
Different approaches to integrating, analysing and acting upon clinical information leads to wide, unwarranted variation in medical practice across regions and institutions (McKinlay et al. 2007] Westert al. 2018). Algorithms designed to support the clinical decision-making process
help overcome this issue, with successful examples in oncology prognosis (Beck et al. 2011 Es- teva et al. 2017), retinopathy detection (Gulshan al., (016) and radiotherapy planning Valdes et al. 2017 ) Still, these support systems focus on effectively replacing physicians with autonomous agents, trained optimise patient outcomes or diagnosis accuracy. With limited design concern for end-users, these algorithms lack interpretability, leading to mistrust from the medical community
LLii et al. 2020 The Royal Society 2017).
Instead, our work aims to better describe and understand the decision-making process by observing physician behaviour, in a form of epistemology. These transparent models behaviour can clearly synthesise domain expertise from current practice and could thus form the building blocks of future decision support systems (Li al. 2015), designed in partnership with clinicians more reliable validation, better trust, and widespread acceptance. In addition, such policy models will enable quantitative comparisons of different strategies of care, and may in turn lead to novel guidelines and educational materials to combat variability of practice.
The associated machine learning challenge, therefore, is to explain how physicians choose clinical actions, given patient observations and their prior medical history and thus describe inter- pretable decision- policy. Modelling clinical diagnostic and treatment policies is particularly challenging, with observational data involving confounding factors and unknown evolution dynam-

---

Published as a conference paper at ICLR 2022
C+-3
38.5CC
T> 37.5"C
Histories 7
Observations 2
Infection
Antibiotics
Actions
Antibiotics-
Figure Schematic representation of behavioural policies as trees, adapting over time.
ics, which we wish to then translate into a transparent representation of decision-making, State-of- the-art solutions for uncovering demonstrated policies, such inverse reinforcement learning (Ng
& Russell 2000) or imitation learning (Bain & Sammut 1996 Piot et al. 2014 Ho & Ermon 2016), fall short of interpretability- - mainly proposing black-box models as behaviour representations.
In contrast, transparent policy parametrisations based on visual decision boundaries (Hüyük et al.
2021 high-level programming syntax (Verma et al. 2018) B) or outcome preferences (Yau et al.] 2020) can explain learned behaviour, at the cost of restrictive modelling constraints and obscure dynamics. Among possible interpretable representations, decision trees offer the flexibility to capture high- dimensional environments and, we demonstrate, can model time-series through recurrence. In addition, decision trees are highly familiar to the medical community, with many clinical guidelines having this form (Chou al., 2007; McCreery & Truelove 1991 Burch et al., 2012). Cognitive search suggests human decision decision-making al. 2017), making these models straightforward to understand. As result, our algorithm Pol- icy Extraction through decision Trees (POETREE) represents clinical policies as decision trees, evolving over time with patient information as illustrated in Figure [
making does follow a hierarchical, branching process (Zylberberg
Contributions The main contributions of this work are as follows: (i) A novel framework for policy learning in the form of incrementally grown, probabilistic decision trees, which adapt their complexity to the task at hand. (ii) An interpretable model for representation learning over time- series called recurrent decision trees. (iii) Illustrative analyses of our algorithm's ''s expressive power in disambiguating policies otherwise unidentifiable with related benchmarks; integration of domain knowledge through inductive bias; and formalisation and quantification of abstract behavioural con- cepts (c.g. uncertain, anomalous or low-value actions) thus demonstrating its effectiveness in both understanding and replicating behaviour.
2 PROBLEM FORMALISM
Our goal is to learn an interpretable representation of demonstrated decision-making process to understand agent behaviour. The clinical context requires offline learning using observational data only, as experimenting with policies would be both unethical and impractical. Finally, as previ- observations and actions affect treatment choice, decision-making environment.
concerned with partially-observabee
We assume access to dataset of m demonstrated patient trajectories in discrete time D {(zi, (fff,at... ai,..., 2t,, of } where T; is the length of trajectory i. We drop index i to denote generic draw from the population. At timestep the physician-agent observes patient features, denoted by random variable Z, e Z, and chooses treatment or diagnostic action A, E A. where A is a finite set of actions. Let 24 and a, denote realisations of these random variables. In contrast to traditional Markovian environments, the agent's 's reward function Rt, state space S and transition dynamics are all inaccessible.
Let f denote a representation function, mapping the history of patient observations and actions to a representation space H. and he f(2 (211,11......... 11. at-1) f(210-1, Q1:1-1 E H. represent the patient history prior to observing new set of features 2;. Following the partially-observable Markov Decision Process (POMDP) formalism, information (h,, 21] can be combined to form a belief over the inaccessible patient state st. In contrast to recent policy learning work on POMDPs, which uncover mappings from belief to action space (Sun et al., 2017 Choi & Kim 2011 Makino & Takeuchi 2012 Hüyük et al. 2021), however, aim is to highlight how newly- aewyuieq obser- vations 2, conditions action a,, to ensure decisions are based on interpretable, meaningful variables. As result, agent behaviour must be represented as mapping from observation action space.
An adaptive decision-nen maasiinn-aainggaa policy is a stationary mapping क : x H X A + [0, 1]. where (ao)21, hi) encodes the probability of choosing action a; given patient history hy and latest obser-
2

---

Published as a conference paper at ICLR 2022
vation 2 thus Vt, Ea,ed (a) h4) 1. We assume trajectories in D generated by agent following policy TE, such that de TE (- ht). The goal of our work is to find interpretable policy representation x which matches and explains the behaviour of mg.
3 INTERPRETABLE POLICY LEARNING WITH DECISION TREES
3.1 SOFT DECISION TREES
Motivated by the interpretability and pervasiveness of trees in the medical literature (Chou et al. 2007), Policy Extraction through decision Trees (POETREE) represents observed deciston-making policies as trees. Our model architecture is inspired by soft decision trees structures with binary nodes, probabilistic decision boundaries and fixed leaf outputs (Irsoy et al., 2012) for their transparency and modelling performance (Frosst & Hinton 2017). This supervised learning framework be adapted for policy learning by setting input variables as 2g (h,, = (for fully- partially-observabee envi- ronments), and targets as y 0g (Bain Sammut 1996).
inner node fgg
P Pgate
2
a leaf mode to
Pgace
Inner nodes Illustrated in Figure 2a the path followed by normalised input x E ID_ is determined by gating functions encoding the probability of taking the rightmost branch at each lonf node nt: Prate(()
softmmee R
a (x" w" + 6"). where (w" E IRP: 5" E In are trainable parameters and a is the sigmoid function. This design choice is moti- vated by its demonstrated flexibility and expressivity (Silva al.) 2020), and be made interpretable by retaining the largest component of w" to form an unidimensional axis- aligned threshold each node, illustrated Figure and detailed in Appendix [C] Model simplification can be made more robust through I.I regularisation, encouraging sparsity in (w") Jneeer The path probability of each node, P" (x), therefore given by the product gating functions leading to it from the tree root.
(a) DT policy.
time delay
P gate!
hg
p
Paate
he
2e
Lanie
Leaf nodes In our tree architecture, leaves define fixed probability distribution over K categorical output classes. Leaf I encodes relative activation values à softmax (O where 0% E IRK learnable parameter. For ease of com- putation and interpretation, the output of the maximum-probabilitty leaf Öfmas where Lmax arg max, pt (xx), taken overall classification decision, following Froossss Hinton (2017).
(b) RDT policy.
Figure 2: POETREE policy structures.
3.2 RECURRENT DECISION TREES
To account for previously acquired information, our algorithm must jointly extract and depend on a representation of patient history h, As in Figure 2b] RDTs take history h, and new observation 2, as inputs, and output both predicted action a, and subsequent history embeddings he+1 from decision leaves. These are computed as M = (anh(0))). where 0 TR is an additional leaf parameter - alternative leaf models are discussed parameters 0- IR our model also predicts patient evolution,
Appendix B Finally, as third leaf output with
observations the next timestep Æ411, effectively capturing the expected effects of a treatment on the patient trajectory as a policy explanation Yaau al. 2020). A single tree structure is used at all timesteps: unfolded over time, our recurrent tree can be viewed as a sequence of cascaded trees (Ding et al. 202T) with shared parameters O {[w", 8") ) (0', (ee
Leaf) and topology
3

---

Published as a conference paper at ICLR 2022
3.3 TREE GROWTH AND OPTIMISATION
Optimisation objective The policy learning objective is to recover the tree topology T and param- eters @ which best describe the observation-to-actie mapping demonstrated by physician agents. Our loss function combines the cross-entropy between one-hot encoded target a, E {0, 1] and each leaf"s action output, weighted by its respective path probability under input { ²x):
L(( Fe. ); T, 0)
Σ tELeaf
Pl 21) Σ Ll.kk log [dt,x]
(1)
where d² is the output probability for action class k in leaf 1. Additional objectives are necessary for the multiple outputs to match their intended target: regularisation term Lg is designed to learn patient evolution ε as expected by the acting physician. It minimises prediction error on 2 and ensures the policy is consistent between timesteps by constraining predicted observations to lead to similar action choices as true ones, under the new history h141:
(= Rt, Z 1 T, 5, ||+ - 82 DKL. (T(-)he+1, = (-1//++111 Re+1)
where {81, are tunable hyperparameters, weighting fidelity to true evolution and to demonstrated behaviour. For a set of training trajectories D. and an overall loss L = + Le, our policy learning optimisation objective becomes:
arg min L(D) T,e
(3)
The probabilistic nature of our model choice makes it fully-differentiable, allowing optimisation of parameters @ through stochastic gradient descent and backpropagation through the structure for a fixed topology T. Optimising with respect to T, however, requires more involved algorithm.
Algorithm 1: Tree growth optimisation.
Initialise to inner node with two suboptimal leaves;
1. Initialisation
3. Global optimisation and path pruming
Optimise θ via gradient descent of loss C; while 3 suboptimal leaves in T do
imer made
2. For each suboptimal leaf:
Split suboptimal leaf into inner node with leaves;
a. Split node b. Label as optimal
2a. Locally optimise via gradient descent L; if validation performance improves then retain split;
Local opttimistionn
buef Keep hoo-porfornng validation
else 2b. Retain leaf as optimal;
Globally optimise @ via gradient descent of L; Prune branches in T with low path probability p (Deat)-
Tree growth Following the work of Irsoy et al. (2012) and Tanno al. (2019) our decision tree architecture is incrementally grown and optimised during the training process, allowing the model independently adapt its complexity to the task hand. Algorithm [ summarises the growth and optimisation procedure. Starting from an optimised shallow structure, tree growth is carried out by sequentially splitting each leaf node and locally optimising new parameters the split is accepted if validation performance is improved. After final global optimisation, branches with path probability below a certain threshold on the validation set are pruned. Further details on the tree growth procedure are given in Appendix
Complexity Policy training can be computationally complex due to the multiple local tree optimi- sations involved and significant number of parameters in comparison to other interpretable models (Hüyük et al.) (00T). Still, at test time, computing tree path probabilities for given input is inex- pensive, which is advantageous to allow fast decision support. A detailed comparison of runtimes involved with our method and related work is proposed in Appendix D
4 RELATED WORK
In this section, compare related work interpretable policy learning, and contrast them in terms of key desiderata outlined in Section 2 Table summarises our findings.
Policy learning While sequential decision-making problems are traditionally addressed through reinforcement learning, this framework is inapplicable to our policy learning goal due to the unavail- ability of a reward signal R. Instead, focus on the inverse task of replicating the demonstrated behaviour of an agent, known as imitation learning (IL), which assumes no access to R. Com- approaches include behavioural cloning (BC) where this task is reduced supervised learning.
4

---

Published as a conference paper at ICLR 2022
Table 1: Related work. Comparison to policy learning methods in terms of interpretability.
Offliee Learning
Partial Observability
Interpretable Policy
Modelling Assumptions
Related work
BC-II. Bain Sammut 1996
PO-BC-IL (Sun et 2017
Interpretable BC-IL (Hiyiik et al 2021
known; low-dim. env.
DM-IL (Ho Ermon 2016
MB-IL (Egglett et al. 2013
Simple dynamics
IRL [gg & Russell 2000
X X
X
x
TE optimal wrt. R
PO-IRL (Chor & Kim 201
/
X
TE optimal wrt. R
Offline PO-IRL (Makino & Takeuchi 2012
TE optimal wrt. R
Interpretable RL (Sivv at. a.l... 2020
R. known
POETREE (Ours)
mapping states to actions (Bain & Sammut 1996 Piot et al. 2014); and distribution matching (DM) methods, where state-action distributions between demonstrator and learned policies are matched through adversarial learning (Ho & Ermon| 2016 Jeon et al. 2018 Kostrikov et al.] 2020). Ap-
prenticeship learning (AL. aims to reach or surpass expert performance on a given task a closely related but distinct problem. The state-of-the-ar solution is inverse reinforcement learning (IRL),
which recovers A from demonstration trajectories (Ng & Russell 2000 Abbeel & Ng 2004 Ziebart
al. 2008); ; followed by forward reinforcement learning algorithm to obtain the policy.
Towards interpretability for understanding decisions Imitation and apprenticeship learning so- lutions have been developed to tackle the challenges of fully-offline learning in partially-observee Perpaartiiii
decision environments, as required by the clinical context: in Sun et al. (2017) learn belief states through RNNs, and IRI. was adapted for partial-observabiiity (Choi & Kim 2011 and offline learn- ing (Makino & Takeuchi 2012). However, while black-box neural networks typically provide best results, transparent description of behaviour requires parametrising the learned policy to be in- formative and understandable. This often involves sacrificing action-matching performance (Lage) al. 2018), and introduces assumptions which limit the applicability of the proposed method, such as Markovianity (Silva et al. 2020) or low-dimensional state spaces (Hüyük et al., (021). The latter work, for instance, parametrises action choice through decision boundaries within the agent's 's belief space. In addition to leaving the observations-to-actee mapping obscure, this imposes constraints on modelled states and dynamics which limit predictive performance, scalability and policy iden- tifiability (Biasutti et al. 2012).
One approach to achieve interpretability in decision models is to implement interpretable reward functions for IRL, providing insight into demonstrating agents' inner objectives. Behaviour has thus been explained as preferences over counterfactual outcomes (Bica et al. 202I), or over information value in time-prssuree context (Jarrttt & van der Schaar 2020). Still, the need for black-box RL algorithms to extract a policy obscures how observations affect action choice. Instead, the agent's 's policy function be directly parametrised as an interpretable structure. RI. policies have been represented through a high-level programming syntax (Verma et al., (2018), or through explanations in terms of intended outcomes (Yau et 2020). Related work on building interpretable models for time-series in general is also discussed in Appendix
Adaptive decision trees for time-series Related soft and grown tree architectures have been pro- posed for image-based tasks (Frosst & Hinton 2017 Tanno et al., 2019) little research has been reported on probabilistic decision trees for the dynamic, heterogeneous and sparse nature of time- series data. Tree structures have been proposed for representation learning, relying for instance on projecting inputs to more informative features in between decision nodes (Tanno al., 2019) or on cascaded trees, sequentially using their outputs as input to subsequent ones (Ding et al. 2021). In all cases, however, models lose interpretability, as decisions are based on unintuitive variables: in contrast, we build thresholds over native observation variables, meaningful to human experts. Our work is further contrasted with established tree models in Appendix
ILLUSTRATIVE EXAMPLES
5.1 EXPERIMENTAL SETUP
Decision-Making Environments Three medical datasets were studied to evaluate our work. First, as ground-truth policies ag are inaccessible in real data, synthetic dataset of 1000 patient trajec-
5

---

Published as a conference paper at ICLR 2022
tories over 9 timesteps, denoted SYNTH, was simulated. In a POMDP with binary disease state space, observations include oneddimenaa diagnostic tests (2+, 2-} as well as 10 noisy dimen- sions disregarded by the agent (e.g. irrelevant measurements);; and actions {a., a- correspond to treatment and lack thereof. Assuming treatments require exposure beyond the end of symp- 1996), the expert policy treats patients who tested positive over the last three timesteps: a₂ = Tg [1, 2 = a. zf E {z 2t J a- else. Next, a real medical dataset was explored, following 1.625 patients from the Alzheimer's s Disease Neuroimaging Initiative (ADNI), as in Hüyük et al. (202T) for benchmarking. We consider the task of predicting, at each visit, whether a Magnetic Resonance Imaging (MRI) scan is ordered for cognitive disorder diagnosis (NICE, 2018). Patient observations consist of the Clinical Dementia Rating (CDR-SB) on a severity scale (normal; questionable impairment; severe dementia) following O' D'Bryant et al. 2008); and the MRI outcome of the previous visit, categorised into four possibilities (no MRI scan; and below- average, average and above-average hippocampal volume V,,) Finally, we also consider a dataset of 4,222 ICU patients over up to 6 timesteps extracted from the third Medical Information Mart for In- tensive Care (MIMIC-III) database (Johnson et al.) 2016) and predict antibiotic prescription based on 8 observations temperature, white blood cell count, heart rate, hematocrit, hemoglobin, blood pressure, creatinine and potassium; as in Bica et al. (2021). By dataset design or by the nature of real-world clinical practice, observation history must be considered by the acting policies making our decision -aaiing environments
toms (Entsuah et al.
Success Metries Reflecting our unconventional priority for policy interpretabilityy over imitation performance, we provide illustrative examples evidencing greater insight into observed behaviour through decision tree representations. For more quantitative assessment of interpretability, we also surveyed five practising UK clinicians of different seniority level, asking them to score models out of ten on how well they could understand the decision-makee G Action-matching performance was also evaluated through the areas under the receiver-operating characteristic curve (AUROC) and precision-recall curve (AUPRC), and through Brier calibration.
process with details Appendix
Benchmark Algorithms For benchmarking, we implemented different behavioural cloning mod- els, mapping observations or belief states to actions: (i) decision tree (Tree BC-IL) and logistic regression models with no patient history; (ii) partially-observable BC algorithm (PO-BC-IL) extracting history embeddings through an RNN (Sun et al.) 2017), on which a feature importance analysis is carried out (Lundberg al. 2017): (iii) and interpretable PO-BC-IL model, INTER- Benchmarks also include (iv) model-based imitation learning PO-MB-IL (Englert al.) 2013), adapted for cartial-obeevvabiliiy with learned environment model; (v) Bayesian Inverse Reinforcement Learning for POMDPs (PO-IRL), based on Jarrett& & van der Schaar (2020); and (vi) Offline PO-IRL version of this algorithm as in Makino & Takeuchi (2012). Implementation details are provided in Appendix E
POLE (Hüyüik et al., (2021). described in Section
5.2 INTERPRETABILITY
In light of our priority on recovering an interpretable policy, we first propose several clinical exam- ples and insights from the ADNI dataset to highlight the usefulness of our tree representation.
Explaining patient trajectories Our first example studies decision-making within typical ADNI trajectories. Let Patient A be healthy with normal CDR score and average Vi, on initial scan. Let Patient B initially have mild cognitive impairment (MC1), progressing towards dementia: their CDR degrades from questionable to severe the last timestep and below- V,, is measured at each visit. Patient C is diagnosed with dementia at the first visit, with severe CDR and low VA: no further scans are ordered they would be uninformative (NICE 2018). Figure 3a illustrates our learned tree policy, both varying with and determining patient history at each timestep (see Appendix F for the distinct policy of each patient). Patient is not ordered any further scan, as they never show low V,, or abnormal CDR, but may be ordered one if they develop cognitive impairment. For Patient with low Vn, but a CDR score not yet severe, another MRI is ordered to monitor disease progression. At the following visit, patient history conditions the physician to be more cautious of scan results: even if V,, is found to be acceptable, another scan may be ordered the CDR changes. Finally, Patient C's observations already give certain diagnosis of dementia: no scan is necessary. At following timesteps, unless the patient unexpectedly recovers, their history suggests to not carry out any further scans. We must highlight the similar strategy between our decision tree policy and published guidelines for Alzheimer's ' diagnosis in Biasutti et al. (2012), reproduced in Appendix F our policy correctly leams that investigations are only needed until diagnosis is confirmed.

---

Published as a conference paper at ICLR 2022
MCI
Vis low
Healthy
Patient
CDR SD nonmal
MRI
Va low
Parient B
Patient C
MRI
No previous MRI
CDR-SB nermal
CDR-SB severe
Healthy
Dementia
MRI
MRT
MRI
(b) Decision houndary over helief
(c) Static decision tree policy
t+1
MC1
Dementia
Savee CES4
CDR
Vh low
CDR severe
Eridddy Mi
CDR-SB questiorable
CDR-SB severe
MRI
Snor
Strnal
MRI
Trobilibyy lo
Jourze
(d) Logistic regression
(e) Feature importance
(a) Adaptive decision tree policy
Figure 3: Comparison of interpretable policy learning methods on ADNI. Crosses stand for no action. Fig.
(b) follows method from Hüyük (02T))
Vh low
Vh low
CDR-SH normal
CDR-SB severe
CDR-SH questionable
CDR-SB severe
29%
64%
11%
81%
MRI
MRI TOPN
MRI
MRI
*6% MSRIjp..)
'Lanssipp
6.78
Figure 4: Policy confidence ADNI. Maximum probability paths are highlighted in blue, with action proba- bilities in green/red.
In contrast, the black-box history extraction procedure required by most benchmarks for partial- observability (PO-BC-IL, PO-IRL., Offline PO-IRL, PO-MB-II.) is difficult to interpret, with no clear decision-making process. The policy learned by INTERPOLE (Hüyük et al., [022]) in Figure 3b defined by decision boundaries over a subjective disease belief space, provides more insight: this model suggests respectively few and frequent scans for healthy Patient A and MCI Patient B, at different vertices of the belief space, but cannot account for scan reduction in diagnosed Patient C as this requires discontinuous action regions. This policy representation only relies on disease beliefs and thus cannot identify the effect of different observations on treatment strategies. Finally, com- mon interpretable models proposed in Figures 3c-Br Be fail to convey an understanding of behavioural strategy and evolution.
Decision-making uncertainty decision confidence thanks to its probabilistic nature:
Our behaviour model also inherently captures the useful notion of
ा (az = k h4, 21)
Σ P'(he, 21) af.x
(4)
LELeaf
where, for each leaf node P' 2,) and di, the path and output probability for action k. Figure highlights how policy confidence varies over time for each typical patient. Uncertainty for Patient A can be attributed to variability in healthy patient care in the demonstrations dataset, as scans may provide information about their state, but must be balanced with cost and time considera- tions (NICE 2018). For Patient C with a dementia diagnosis, further investigations are not required (Biasutti et al. 2012), reflected in confidence increase. As result, for Patient B with degrading symptoms, scans are initially ordered with low confidence to monitor progression, but as symptoms worsen and the patient is diagnosed, decision confidence increases. Figure 4 also evidences both greater inter-path and intra-leaf uncertainty for MRI prediction: our policy model identifies clear conditions not warranting scan after diagnosis, whereas conditions warranting one are more am- biguous. This example illustrates the value of uncertainty estimation afforded by our framework, modulating probability values within tree paths and leaf outputs.
IInter-path uncertainty can be measured as entropy within leaf path probabilities - Σ, (''((x)log P' while lntra-lcaf uncertainty can he measured as the average leaf output entropy: -1/1,, E, òr log
7

---

Published as a conference paper at ICLR 2022
Vh low
Asundion ucleo
Belated verectusa
Uninfommaati Mile (90)
Aneragg
CDR-SB normal
CDR-SB
Proquency (%)
palset
95%
4%
MRI
MRI
60%
x(x
MT) 0.95 0.92 0.04 x (1 0.00) +
0.90
ym
Apr54
Sutly agoi
Roodde
Male
difeatty
<(o, AR,
(a) Anomalous behaviour per cohort.
(b) Uninformative actions.
Figure 5: Anomalous and low-value actions various patient cohorts: patients over 75 years old, patients carrying the c4 allele the apolipoprotein E gene (ApoE4), patients with signs of MCI or dementia at their first visit, female and male patients.
Anomalous behaviour detection Learned models of behaviour are also valuable to flag actions incompatible with observed policies. Visits where an MRI is predicted with 90% certainty, yet the physician agent does not order as in Figure Sa make up 8.4% of ADNI. This may highlight demonstration flaws, as the learned policy is confident and has well-calibrated probability estimates (low Brier score Table 2). Anomalous behaviour followed by an MRI 'correcting' the off- -pllic action can also be detected, suggesting 6.7% of patients are thus investigated late comparable to belated diagnoses found in 6.5% of patients by HHyüük et al. (202T). As shown in Figure 5a high- risk cohorts due to age, female gender or ApoE4 allele (Launer et al. 1999 Lindsay et al., 2002) are more often subject to late actions. Most anomalous actions on patients with ApoE4 or female patients end up corrected, in contrast to older patients or ones with early symptoms immaing is less paramount for the latter, as they may be diagnosed clinically (NICH) 2018).
Action value quantification through counterfactual evolution Our joint expected evolution and policy learning model allows action value to be quantified YYau et al. 2020 Bica et al. (021). Low- value actions correspond when this could already be foreseen from expected evolution ă (less than average variation minus one standard deviation). As shown in Figure 5b) ill patients are more often monitored with uninformative MRIs than healthy ones. This analysis explains the high rate of uninformative actions for patients with early signs of disease, found in previous work (Hüyük et al., 202T): the policy recommends investigation for diagnosis, yet their expected evolution is typically unambiguous. Overall, our decision tree architecture enables straightforward assessment of counterfactual patient evolution under different observation values.
ordered scan showing similar observations to the previous timestep.
Overall, our decision tree policy parametrisation explains demonstrated decision-making behaviour in terms of how actions chosen based observations and prior history, and, in the above illustrative analyses, allows a formalisation abstract notions such as uncertain, anomalous or low-value choices for quantitative policy evaluation. Next, we show that these insights are not gained at the cost of imitation performance.
5.3 POLICY FIDELITY
Action-matching performance As shown Table our approach outperforms or comes second- best on all success metrics compared to benchmark algorithms. This confirms the expressive power of our recurrent tree policies over decision boundaries (Hüyük et al. (02I) and even over neural- network-based methods (Sun et al. 2017), specifically optimised for action-matching The poor action-matching performance static decision trees (Tree BC-IL) highlights the importance his- tory representation learning in overcoming the partial-observabiiityy of the docision-making environ- ment. Mean relative evolution error between 2, and & was measured as 13 + 4% and 26 + 9% on ADNI and MIMIC respectively, giving reasonable and insightful values. Our model was scored as the second most understandable our user survey, following the overly-simplistic static decision tree (Fig. Bcp. Collected feedback is provided in Appendix G
Policy identifiability Figure 6 illustrates policies learned from our high-dimensional SYNTH dataset with POETREE and the closest related work (Hüyük et al. 202I). Despite the apparent simplicity of the decision decision-making problem, INTERPOLE fails to identify the outlined treatment strat- egy, as evidenced by its inferior performance in Table This illustrates greater flexibility of our
8

---

Published as a conference paper at ICLR 2022
Table 2: Comparison of the performance of policy learning algorithm pN medical datascts. Interpretabil- ity scores out of ten were obtained through our clinician survey. Lower is better for Brier calibration. Standard emrors for MIMIC and SYNTH were v 0.04.
Task
ADNI MRI seans
MIMIC antibiotics
SYNTH
Algorithm
Interpretability
AUROC
AUPRC
Brier
AUROC
Brier
AUROC
Brier
Tree BC-II.
9.3 0.3
0.53 0.01
0.72 0.01
0.25
0.01
0.50
0.23
PO-BC-IL.
47
0.3 + 0.3
0.59 0.04
0.80 +: 0.08
0.18 +: 0.05
0.67
0.16
0.98
0.02
INTERPOLE
7.3 = 0.5
0.44 = 0.04
0.75 + 0.09
0.19 + 0.07
0.65
0.21
0.84
0.12
PO-MB-II.
13
0.54 0.08
0.81 0.07
0.19 + 0.03
PO-IRL
0.50 0.08
0.82 0.04
0.00
Offline PO-IRL 37
0.54 0.06
0.83 + 0.04
0.23 + 0.01
POETREK
8.3 +: 0.3
0.62 0.01 0.82 +: 0.01
0.18 0.01 0.68
0.19
0.99
0.01
tree representation in disambiguating decision-making factors which simply cannot captured by previous work; and, due to not relying on HMM-like dynamics, can be credited to fewer modelling assumptions and scalability to tigh-dimmnsional environments.
Treatnert
Healihy
Dactwal
(a) Decision- Decision-boundary policy HHöyük et al. 2021).
Inductive bias Having demonstrated that our policy learning framework allows us to recover domain expertise, we investigated the effect of incorporating prior policy knowledge on learning per- formance. In practice, our model's action prediction structure can be initialised based on a published guideline (Biasutti al., 2012) as in Appendix F training time was reduced by a factor of 1.4 and cetion-matchiin AUPRC was improved to 0.84 0.01. This 'warm-start" is promising strategy to learn from physician obser- vation while building on established clinical practice standards.
Positive test
Trentmentt
Positive test
Acurray-interpretabllit trade-off: complexity analysis In- crementally grown trees outperform fixed structures by eliminating unnecessary partitions, with ~40% fewer parameters for the same depth, greater readability (Lage et al. (0018) and reasonable training avoid
Treumment
Treocment
Postiive test
time. Grown structures adapt their depth and complexity overfitting while capturing the modelling task, in a form of Occam's 's razor - jointly optimising policy accuracy and interpretability. In particular, sample complexity modulates learned policies in train- ing. Detailed experimental results are provided in Appendix D
Theatment
Treatment
(b) Adaptive decision tree policy.
Figure 6: Recovered policies for SYNTH.
6 CONCLUSION
Established paradigms replicate demonstrated behaviour fall short of explaining learned decision policies, whereas the state-of-the-art in interpretable policy learning was criticised by end-users for restrictive modelling assumptions (Hüyük et al., 202T) thus not fully capturing diagnostic strate- gies. Policy Extraction with Decision Trees (PoEtREE) overcomes these challenges, outputting most likely actions and uncertainty estimates, given patients' observations and medical history. Our algorithm carries out joint policy learning, history representation learning and evolution prediction tasks through recurrent, multi-output tree model - justifying agent decisions by comparing coun- terfactual patient evolutions. Optimised structures outperform the state-of-the-art in interpretability, policy fidelity and metthnnmatching, thus providing an understanding of demonstrated behaviour without compromising accuracy. In particular, we can formalise and quantify concepts of practice variation with time; uncertain, anomalous, or low-value behaviour detection; and domain knowledge integration through inductive bias. Finally, validate clinical insights through survey of intended users.
Beyond policy learning, our novel interpretable framework for time-series modelling shows promise for alternative tasks relevant to clinical decision support, such as patient trajectory prediction, out- come estimation human-in-the-loop learning for treatment recommendation promising avenue of research for further work.
References: 47
Sun et al.
2017:
23
Hüyük al. 2021
1k]
Englert al.
2013
25
Jarrett &
der Schaar 20200
Makino & Takeuchi 2012)

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We thank all NHS clinicians who participated in our survey and the Alzheimer's Disease Neuroimag- ing Initiative and Medical Information Mart for Intensive Care for access to datasets. This publica- tion was partially made possible by an ETH AI Center doctoral fellowship to AP. AJC would like to acknowledge and thank Microsoft Research for its support through its PhD Scholarship Program with the EPSRC. Many thanks to group members of the van der Schaar Lab and our reviewers for their valuable feedback.
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning, In- ternational Conference on Machine Learning (ICML), 2004.
Michael Bain and Claude Sammut. framework for behavioural cloning. MACHINE INTELLI-
GENCE 15, pp. 103 129, 1996. URL htttp:/cittseeexx: ist psu..uu..... edu/viewdoc/
commmaryydoi=10-oo
25.1759
AH Beck, AR Sangoi, Leung, RJ Marinelli, TO Nielsen, MJ van de Vijver, RB West, M van de Rijn, and D Koller. Systematic analysis of breast cancer morphology uncovers stromal fea- tures associated with survival. Science translational medicine, 3, 11 2011. ISSN 1946-6242. doi: 10. 1122////TTTAAALE.00000..0 22C72638/
URLh:::::: http::::://pbmmddd...m ncbi wwieciinn.....mmmmm nih. niv.gov/
Yoshua Bengio and Paolo Frasconi. An input output hmm architecture. Adv. Neural Inf. Process. Syst., 7:427-434, 1995.
Maria Biasutti, Natacha Dufour, Clotilde Ferroud, William Dab, and Laura Temime. Cost- effectiveness of magnetic resonance imaging with new contrast agent for the early diagnosis of alzheimer's s disease. PLOS ONE, 7:e35559, 4 2012. ISSN 1932-6203. doi: 10.137/JOURRNA.
PONE.0033559. URL //tp:/:://////osssss// pll..s..pppl......om.....ooo..l. http://oooooooeearccceeeeeeeeee. http:///ooo.aaaaaaoooooo/a/////r
1371/journa pone. 0035559
Ioana Bica, Daniel Jarrett, Alihan Hüyük, and Mihaela der Schaar. Learning "what-if" explana- tions for sequential decision-making. Proc. 9th International Conference on Learning Represen- tations (ICLR 2021). 2021. URL http::ttt//aattt
org/ abs/2007.135.
Leo Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification and regression frees. Wadsworth & Brooks/Cole, 1984.
J Burch, S Hinde, S Palmer, Beyer, J Minton, A Marson, U Wieshmann, N Woolacott, and M Soares. The clinical effectiveness and cost-effectiveeess of technologies used to visualise the seizure focus in people with refractory epilepsy being considered for surgery: a systematic review and decision-anaal analytical model. Health Technology Assessment, 16:1-163, 9 2012. ISSN
ISSN: 2046-4924. doi: 10.3310/HTAL 16340. URL www. ht..a.ta..... ac..uk
Edward Choi, Mohammad Taha Bahadori, Joshua A. Kulas, Andy Schuetz, Walter F. Stewart, and Jimeng Sun, Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Avances Neural Information Processing Systems (NIPS), 29, 8 2016. URL
1/arxiv.org/ abs/ / 608 045
Jaedeug Choi and Kee-Fung Kim. Inverse reinforcement learning in partially observable environ- ments. Journal of Machine Learning Research, 12:691-730, 2011. ISSN 1533-7928. URL
hattp://jmle http://ll.o./////...///../oo httppp///ill..ooggapppeee////// html
Roger Chou, Amir Qaseem, Vincenza Snow, Donald Casey, Thomas J. Cross, Paul Shekelle, and Douglas K. Owens. Diagnosis and treatment of low back pain: A joint clinical practice guide- line from the american college of physicians and the american pain society. Annals of Internal Medicine, 147:478-491, 10 2007. doi: 10.7326/0003-48/ 1100--88011111-22220000000000000
Zihan Ding, Pablo Hernandez-Leee Leal, Gavin Weiguang Ding, Changjian Li, and Ruitong Huang. Cdt: Cascading decision trees for explainable reinforcement learning. arXiv preprint arXiv:2011.07533, 2021. URL htttp:://githu. com httpp//a///ym///////mattttseeeee
sym/ennvsbox2d//un
10

---

Published as a conference paper at ICLR 2022
Peter Englert, Alexandros Paraschos, Mare Peter Deisenroth, and Jan Peters. Probabilistic model- based imitation learning. Adaptive Behavior, 21:388 403, 10 2013. ISSN 10597123. doi: 10. sagepub....pp................... com/ doi/ wttp:/ccccdccoou////e// full/IC.
1177/10597123114491114 URL
(177/10597123134991
A. R. Entsuah, R. L. Rudolph, D. Hackett, and S. Miska. Efficacy of venlafaxine and placebo during long-term treatment of depression: pooled analysis of relapse rates. International Clin- ical Psychopharmacolooyy 11:137-145, 1996. doi: 00997/00004450--19961022222 (06e0r0/1996-00055000
URL
Andre Esteva, Brett Kuprel, Roberto A. Novoa, Justin Ko, Susan M. Swetter, Helen M. Blau, and Sebastian Thrun. Dermatologiistlevel classification of skin cancer with deep neural networks. Nature 2017 542:7639, 542:115- 2017. ISSN 1476-4687. doi: 10. 1050 URL
https:
/ www. naturattueeee..e... coo art wew.ee/eaaanneeeeeeeeeee
Wael Farag, Safe-driving cloning by deep learning for autonomous cars. International Journal of Advanced Mechatronic Systems, 7:390-397, 2017. doi: 10. 10.5004IAMECHSS017..00
Nicholas Frosst and Geoffrey Hinton. Distilling a neural network into soft decision tree. CEX workshop at AI*IA 2017 conference, 11 2017. URL http:://axxivv.rrg/bbb////
09784
Matthew Golub, Steven Chase, and Byron Yu. Learning internal dynamics model from control demonstration. pp. 606-614, 2 2013. ISSN 1938-7228. URL http::tttpccceeeiiiiiii/ mmr..
press/v28/golubl/ html
Gulshan, L. Peng, M Coram, MC Stumpe, Wu, A Narayanaswamy, S Venugopalan, Widner, T Madams, J Cuadros, R Kim, R Raman, PC Nelson, JL Mega, and DR Webster. Development and validation of a deep learning algorithm for detection of diabetic retinopathy retinal fundus photographs. JAMA. 3 31:2422-24100, 12 2016. ISSN 1538-3598.
doi: 10. 1001/JAM..2016.
17216. URL https: / hwwp//d.bdddeeeeeeee ncbi. nlm. nih. gtt.gog////ogeee Nttihh.oo/7778866668
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. 30th Conference on Neural Information Processing Systems, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9:1735-
1780, 11 1997. ISSN 08997667. doi: 10. w62///cc..oooo 1997.9.9...8 1735. URL http://direct. 1997. 9. 8. 1735. 17f
mit mit. wdw.needu/ecoo///o
1735/ /tt.:///////11131117863383
Stephanie L. Hyland, Martin Faltys, Matthias Hüser, Xinrui Lyu, Thomas Gumbsch, Cristóbal Es- teban, Christian Bock, Max Horn, Michael Moor, Bastian Rieck, Mare Zimmermann, Dean Bo- denham, Karsten Borgwardt, Gunnar Rätsch, and Tobias M. Merz. Early prediction of cireu- latory failure in the intensive care unit using machine learning. Nature Medicine 2020 26:3, 26:364-373, 3 2020. ISSN 1546-170X. doi: 10. 1011038/441511--0--078888
URL https:
// //www. www.nattur.en com articles setlee//411999 020-0789---
Alihan Hüyük, Daniel Jarrett, Cem Tekin, and Mihaela Van Der Schaar. Explaining by imitating: Understanding decisions by interpretable policy learning. ICLR, 2021.
Ozan Irsoy, Olcay Taner Yıldız, and Ethem Alpaydın. Soft decision trees. 21st International Con- ference on Pattern Recognition (ICPR), 2012.
Daniel Jarrett and Mihaela van der Schaar. Inverse active sensing: Modeling and understanding timely decision-making, Proceedings ofthe the 37 th International Conference Machine Learning, 2020.
Wonseok Jeon, Seokin Seo, and Kee-Eung Kim. A bayesian approach to generative adversarial imitation learning. Advances in Neural Information Processing Systems, 31, 2018.
Alistair E. W. Johnson, Tom Pollard, Lu Shen, Li Wei H. Lehman, Mengling Feng. Mohammad Ghassemi, Benjamin Moody, Peter Szolovits. Leo Anthony Celi, and Roger G. Mark. Mimic-iii, a freely accessible critical care database. Scientific Data, 3:1-9, 9, 2016. ISSN 20524463. doi: 10. 10088//dtta2211.... URL www. nature. come.com/ssee coom/ddata/
11

---

Published as a conference paper at ICLR 2022
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, 12 2015. URL
abs/ 1412. 6980v9
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. ICLR, 12 2020. URL http:://rxlii wttp://...gg//b//1/../.... 05032
Isaac Lage, Andrew Slavin Ross, Been Kim Google Brain, Samuel Gershman, and Finale Doshi- Velez. Human-in-the-loop interpretability prior. 32nd Conference on Neural Information Pro- cessing Systems (NIPS 2018), 2018.
LJ Launer, K Andersen, ME Dewey, L Letenneur, A Ott, LA Amaducci, C Brayne, JR Copeland, JF Dartigues, P Kragh-Sorensen, Lobo A, Martinez-Lage JM, T Stijnen, and A Hofman. Rates and risk factors for dementia and alzheimer's ' disease: results from eurodem pooled analyses. eurodem incidence research group and work groups. european studies of dementia. Neurology, 52:78-84, 1999. ISSN 0028-3878. doi: 10. 10.1212/WNL.52..1. URL httph::tt//ss..... http::://pubmdd.. ncbi mlm m. nih /ttp//99299222
M.-C Lai, Brian, and M.-F Mamzer. Perceptions of artificial intelligence healthcare: findings from qualitative survey study among actors in france. J Trans! Med, 18:14, 2020. doi: 10.1 186/ /ww. 512967-011-022044.y URL http:ttp///.....r///1111111///// 1ttp:/21.11116/111616611
A Li, S Jin, L Zhang, and Y Jia. A sequential decision-theorede model for medical diagnostic system. Technology and health care official journal of the European Society for Engineering and Medicine, 23 Suppl 1 1:837-$42, 5 2015. ISSN 1878-7401 doi: 10.3233/THC- HBN/THC--19926 URI.
https: /tt::///pbbmddde ncbi nlm. nih, gov/26410330 6332667
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from vi- sual demonstrations. Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 3815-3825, 2017. URL /tatts::/gitinn comp http://rmoooooo////oo
InfoGAIL.
J Lindsay, D Laurin, R Verreault, R Hébert, B Helliwell, GB Hill, and McDowell. Risk factors for alzheimer's is disease: a prospective analysis from the canadian study of health and aging. American journal of epidemiology, 156:445 9 2002. ISSN 0002-9262. doi: 10. N0.1093/JJ//KKH004 ncbi mnm. nih. 0000//21963314
URL httts:://pubbreeee
Scott Lundberg, Paul Allen, and Su-In Lee. A unified approach interpreting model pre- dictions. 31st Conference on Neural Information Processing Systems (NIPS 2017), 2017. URL
https:// com/ slundberg/ shap
Takaki Makino and Johane Takeuchi. Apprenticeship learning for model parameters of partially ob- servable environments. Proceedings of the 29 th International Conference on Machine Learning, pp. 891-898, 2012.
R. G. Masterton, A. Galloway, G. French, M. Street, J. Armstrong, F. Brown, J. Cleverley, P. Dil- worth, C. Fry, A. D. Gascoigne, Alan Knox, Dilip Nathwani, Robert Spencer, and Mark Wilcox. Guidelines for the management of hospital- asspiaal-cquiree pneumonia the uk: Report of the work- ing party on hospital-acquired pncumonia of the british society for antimicrobial chemotherapy. Journal of Antimicrobial Chemotherapy, 62:5 34, 7 2008 ISSN 7453. doi: 10. 1093/JA/////// DKN162. ht /tt.l:::://aaaeeeee aaa.emiaammmmccm oup. com/ jac/ http/rttilleee. /2//:///1111//// 8444812
Ann M. McCreery and Edmond Truelove. Decision making in dentistry. part i: A historical and methodological overview. The Journal of Prosthetic Dentistry, 65:447-451, 3 1991. ISSN 0022- 3913. doi: 10.
B. McKinlay, C. L. Link, K. M. Freund, L. D. Marceau, A. B. O'Donnell, and K. L. Lutfey. Sources of variation in physician adherence with clinical guidelines: Results from a factorial experiment. Journal of General Internal Medicine, 22:289, 2007. doi: 10.1007/ 0000000000000005--0 URL http/////tttiiic hat/aartiicclse PMC PMMC122476880 http://///mmm/tttrtieeeec ncbi nlm. nih gww.
rrt i www.llee//ee PMC 824760/
12

---

Published as a conference paper at ICLR 2022
Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. Proc. 17th International Conf. on Machine Learning, Pp. 663-670, 2000.
NICE. Dementia: Assessment, management and support for people living with dementia and their
carers. National Institute for Health and Care Excellence (NICE) Guideline No. 97, 2018. URL
http::/wwet nice. org. uk/ wwtpk/ii/uuuuuu
SE O' O'Bryant, SC Waring, CM Cullum, J Hall, I. Lacritz, PJ Massman, PJ Lupo, JS Reisch, and R Doody. Staging dementia using clinical dementia rating scale sum of boxes scores: a texas alzheimer's research consortium study. Archives of neurology, 65:1091-1095, 8 2008. ISSN 1538-3687. doi: 10. D..000//RCHEEUR.6... 1091. URL https: /twpmeed/pppbmddd..m ncbi .nlm. ni.. (ttb.oov/18655059
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. Foundations and Trends @ in Robotics, 7:
1-179, 2018. doi: 10.1561/230000053.
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted and reward-regularized classification for apprenticeship learning. Proceedings of the 13th Inter-national Conference on Autonomous Agents and MultiagentSystems (AAMAS 2014), 2014. URL www www.ifaamas orq
Andrew Silva, Taylor Killian, Ivan Rodriguez Jimenez, Sung-Hyun Son, and Matthew Gombolay. Optimization methods for interpretable differentiable decision trees in reinforcement learning- Proceedings of the 23rdInternational Conference on Artificial Intelligence and Statistics (AIS-
TATS), 2020.
Wen Sun, Arun Venkatraman, Geoffrey Gordon, Byron Boots, and J Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. Proceedings of the 34th International Conference on Machine Learning, pp. 3309-3318, 2017. ISSN 2640-3498. URL http::/proceeddiin .arpsssm.l..... mlr. press /v0/sss/vvv//// sun 7d. html.
Ryutaro Tanno, Kai Arulkumaran, Daniel C Alexander, Antonio Criminisi, and Aditya Nori. Adap- tive neural trees. Proceedings of the 36th International Conference Machine Learning, pp-
(166,6175,, 5 2019. ISSN 2640-3498. URL http: http://pcccecen...eeeepp mlr. P...pprees/v97 tannol html]
The Royal Society. Machine learning: The power and promise of computers that learn by example. The Royal Sociery, 2017.
Gilmer Valdes, Charles B. Simone, Josephine Chen, Alexander Lin, Sue S. Yom, Adam J. Pat- tison, Colin M. Carpenter, and Timothy D. Solberg. Clinical decision support of radiotherapy treatment planning: A data-driven machine learning strategy for patient-specific dosimetric de- cision making. Radiotherapy and Oncology, 125:392-397, 12 2017. ISSN 0167-8140. doi: W...RA....2.1...004....44
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. 35th International Conference on Ma- chine Learning, ICML 2018, 11:8024-800 8033, 2018. URL https://arxiis ergg abs/
1804. 02477v3
Gert P. Westert, Stef Groenewoud, John E. Wennberg, Catherine Gerard, Phil Dasilva, Femke Atsma, and David C. Goodman. Medical practice variation: Public report-
ing a first necessary step to spark change. International Journal for Quality in Health Care, 30:731-735, 11 2018. ISSN 14643677
doi: 10.1093/imtqhc/mzyy92. URL
(mmc/aariicless/MM6333733 http:///...................////.
abstrracthtps:://wwe ncbi n nmm nih. http://///ppeee.....eee////eeeee
Herman Yau, Chris Russell, and Simon Hadfield. What did you think would happen? explaining agent behaviour through intended outcomes. Advances in Neural Information Processing Systems,
33:18375-18886. 2020. URL http:::ttt//gtt thu..u.......... com/ hmhyau/ r1 intention
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. Proceedings of the Twenty-Than -hird AAAI Conference on Artificial Intelli- gence, 2008. URL.www. www.wwwwaaaa. .org
13

---

Published as a conference paper at ICLR 2022
Ariel Zylberberg. Jeannette AM Lorteije, Brian G Ouellette, Chris I De Zeeuw, Mariano Sigman, and Pieter Roelfsema. Serial, parallel and hierarchical decision making in primates. eLife, 6, 6 2017. doi: 10.7554/ELIFE URL /pmm//rrtiles
PMC54 prc.a////7/oonnttttte//ss/////ep
7/www. ncbi _nmm. http:///////////aaaaaaaa////////
A ADDITIONAL RELATED WORK
A.1 IMITATION LEARNING
A.1.1 BEHAVIOURAL CLONING
Behavioural cloning (BC) is the simplest form imitation learning, as it reduces policy learning to supervised learning of a direct mapping from states to actions, assuming fully-observable setting where S Z (Bain & Sammut 1996; Piot et al. 2014). For a trajectories dataset D {[11T, (1:T)). the objective is to recover the deterministic policy mapping ı :ZZ A, such that (24 is the action following observation 24:
π arg min E [C (T(Z)), ar]]
(5)
where C(.,8) y) is supervised learning loss function for target value y.
This approach is typically compatible with offline learning, as it only requires access to demonstrated trajectories of system states and actions (Zt, 0z) at each timestep t. Its excellent learning efficiency has motivated its use and deployment, including in safety-critical applications such as autonomous driving (Farag 2017).
With no explicit handling of system dynamics, however, this approach is highly susceptible to dis- tributional shift (Ho & Ermon 2016); at test time, the agent might enter states unseen in training trajectories and thus drift away from its training domain and perform poorly. Likewise, BC is not designed for non-Markovian or partially-observable environments, where the latest observation 2 is insufficient to determine action choice. Representation learning of POMDP belief states through recurrent neural networks (Sun et al. 2017) can help tackle these issues at the cost of interpretability, as it introduces black-box elements in the model.
A.1.2 STATE-ACTION DISTRIBUTION MATCHING
More recently, an alternative approach to dealing with covariate shift was proposed, based on match- ing state-action distributions between the demonstrator and learned policies. Defining occupancy measures 14x : S x A E for each policy as Ma (s, a) Taals) Ee 7 P(s r)) this can be achieved by minimising the Censen-Shannod divergence in occupancy measures for each policy, while maximising the entropy of the learned one H(m) through generative adversarial training (Ho
& Ermon 2016 Jeon et al. 2018 Kostrikov et al. 2020):
arg min Dis (H
and
arg max ((T)
(6)
In contrast to our descriptive goal, this approach therefore focuses on matching feature expectations rather than explicitely recovering the demonstrated policy mapping from observations or belief states to actions, based on the assumption that this behaviour is optimal and thus aims to maximise performance on a given task. In addition, it suffers from the same issue as BC-IL providing interpretable representations of belief states in partially-observable settings. despite Li et al. (2017) improving explainability by clustering similar demonstration trajectories.
In all cases, online roll-outs of the learned policy required to train a dynamics model of the environment, as P(s = s/TT) is needed to estimate occupancy, making this work incompatible with purely observational clinical data. To overcome this, model-based imitation learning (MB- IL) proposes to train autoregressive exogenous models of environment dynamics (Englert et al.] 2013) for offline feature expectation. Encouraging results were obtained for fully- observable robotic applications, but these dynamics remain excessively simple for the healthcare context.
A comprehensive review of imitation learning methods be found in Osa et al. 2018
14

---

Published as a conference paper at ICLR 2022
A.2 APPRENTICESHIP LEARNING
Apprenticeship learning is another approach to modelling expert behaviour, concerned with match- ing the performance of the demonstrating expert in terms of some unknown utility measure R. The state-of-the-ar solution to this task is inverse reinforcement learning (IRL), which involves recov- ering this reward function from demonstration trajectories (Ng & Russell 2000) identifying the expert's 's hidden objectives. A forward reinforcement learning (RL) algorithm is subsequently ap- plied to obtain the policy.
The inverse task of uncovering the reward function is fundamentally ill-posed, as the observed be- haviour be optimal under multitude of reward signals. Additional constraints must thus be specified to obtain a unique solution, for instance by maximising policy entropy (Ziebart et al. 2008 Choi Kim 2011) described above for distribution-matee matching imitation learning, or by maximising the margin between induced policies in terms of feature expectations
(Ng & Russell
2000 Abbeel & Ng 2004):
arg min Es-pTg [V* V**
where (s) is a parametrisation of the value function for policy т, the expected sum of total dis- counted rewards following this policy, assuming start in state (*(s) = E, 2" s).
IRL can deal with partially environments (Choi & Kim 2011) and even learn dynam- ics models fully offline (Makino & Takeuchi] 2012). Still, in terms of providing insight into the decision-making process, the indirect nature of the IRL RL framework results in uninterpretable policies - despite attempts to parametrise reward functions for explainability (Bica et al. 2021).
Table 3: Comparison of related work in light of our key policy learning goals. DM-IL stands for distribution-matehee imitation learning. f stands for representation learning step; by is the agent's belief over the true St.
Related
Optimisation Objective
Offine
Interpretable Policy
Leamnnng
Observability
BC-IL (Bain Sammot 1996
Clarg max, (ala))mu)
PO-BC-I (Stn an 2017
Interpretable BC-LL Hoyuk ctal,
2021
Carrg max, x(albe), aclibe ((s
DM-IL (Ho Ermon
2016
Interprediiiee DI
2017
Dxu (+(a))))))=(((())))
x
MB-IL (Boglert <t
2013
/
IRL Ng Russell
2000
i arg ming Es,-n,,, [V* (xx)
x
PO-IKL Choi
arg min, Es.-... (b.) (-*
Offlin: PO-RRL (i0 blogchh 2012
Subjective PO-IKL Golub 2013
Table 3 compares how related research addresses the key goals outlined for our work. Our policy learning algorithm must be operable offline on observational data; must accommodate a partially- observable environment to allow adaptive behaviour over time; and must result in interpretable representation of the policy. In comparison to Hüyük et al. (202T) which addresses all three goals, propose to implement policy learning algorithm with fewer assumptions and restrictions on the belief state and dynamics model structure, as well as a more interpretable and expressive policy parametrisation.
B SOFT DECISION TREE OPTIMISATION
In this section, additional details are provided on our decision tree structure and optimisation proce- dure.
To improve optimisation, Frosst & Hinton (017 encourage soft decision tree models to split data evenly across tree inner nodes through an additional regularisation term Lxptit:
Leptit =
Σ Inner
2 2-d |0.5 10.llg a + 0.5 log(1 a])]
(8)
where
Ex P" Px(() Poate (x)
a
(9)
Σx Px(
and À is a splitting penalty weight. This additional loss is essentially a cross-entropy between the discrete binary distribution (0.5; 0.5) and the inner node's data split (a"; 1 a=);; and decays with
15

---

Published as a conference paper at ICLR 2022
depth d' to allow uneven splits the tree learns more complex hierarchies. This regularisation was not found to significantly affect optimisation but was nevertheless included in our objective in light of Frosst & Hinton 2017)'s 's favourable results.
The path probability Pa (x) of node n, conditioned on tree input x, is formally defined as follows:
(*(x)= = II
(xate(() (nEfrihhe(m) (1 P (x)) 1-2n (igghehee())
(10)
SESppcth())
where Spath (n) are the set of nodes on the path from the tree root to node n; 1 is the indicator function (returns if is true, else); and (mi)h((m) are the set of nodes descending from m 's right branch.
Non-categorical action types can also be modelled by passing leaf parameters 0' through a hy- perbolic tangent function, and replacing the cross-entropy in equation I with the mean-squared error. Validation performance in Algorithm [ can be determined with traditional supervised learn- ing metrics, such as target prediction accuracy, as well as with alternative success measures such as interpretability ratings or domain-expert approval.
A particular implementation challenge is to minimise the loss of information from previous opti- misations during growth of the history tree, as new leaves are created (step 2a in Figure (5).. To overcome this, split leaves are initialised to the previous value of the parent node, with added Gaus- sian noise of variance o² = 1/d where the leaf depth. This facilitates local optimisation as the tree grows, by ensuring new leaves still capture similar distributions to optimised parent nodes.
Table Balleecomnnn contrasts mappings learned by our recurrent decision trees (RDTs) and RNNs for different time-series tasks. Note that for action or evolution estimation beyond the input timestep. i.. for т > 0 in this table, state-action distribution matching methods become important in training to overcome compounding errors test time. Our differentiable recurrent tree structures can also be optimised by backpropagation through time as for RNNs.
B. INTERPRETABLE REPRESENTATION LEARNING FOR TIME-SERIES
Related work. The saae-oofthhe-art in interpretable time-series modelling representation learn- ing is to highlight the relative importance of features in contributing to changes in model predictions. This can be achieved with attention mechanisms in Choi al. (2016), or with Shapley explana- tions (Lundberg et al.) 2017). Overall, these methods remain tools to gain approximate insight into complex models, rather than transparent description of their inner workings. In addition, patient observations and medical actions are all treated as independent variables as illustrated in Table 4 significant assumption for policy learning, making models susceptible to confounding in obser- vational data. In contrast, our work aims to distill decision-making pathways from observations to actions in clear format, comparable the human thought process.
Recurrent decision trees This section demonstrates the gains achieved by our choice of history representation learning model to tackle the challenges of partially-observabbe environments. In par- ticular, Recurrent Decision Tree (RDT) models address our requirement for interpretable policies. While the history embedding ht may not be interpretable, experimentally find that the brsed model is more understandable than an RNN. Different structures shown in Figures 7b and 7
were
Table 4: Comparison of time-series methods with actions.
Framework
Recurrent Neural Network (RNN)
Recurrent Decision Tree (RDT)
Tasks & targets (r e 2)
Policy learning (explored in this work): deer R1ze, 0i:1-1 Evolution prediction (decision support): Retr (Ris--1 0ist-1
Outcome prediction (decision support): 9647 Fi 1141::1
Mapping
(hi ==
{2), } tuggs
+ he41
(hi 31 + {au he+1)
other target)
recurrence
Typical model
h = (W,he + Wyit + bz)
a Amer hees Alyas
Imaz arg max, Pl (he, #c)
Merits
Modelling performance flexibility
Interpretahle model; ht modulates mapping from 2, to target Disentanglement of ht and o, via multiple outputs
Joint recurrence and target prediction
16

---

Published as a conference paper at ICLR 2022
also studied, in which hy is first extracted from 21:1-1 through an RNN RDT, and is concatenated to latest observation 2 as imput to distinct action-decision tree. Results in Table[Shighlighh that our RDT architecture performs as well the RNN history extractor in action-matching performance, and, as expected, is preferred by surveyed physicians for its interpretability. Performance remained acceptable as history dimensionality decreased (< 5% loss in accuracy), which is valuable for visualisation purposes.
delay
h
2
time detay
1 Fc-1
time delay
RNN
Z
4
h1 21
Gt
he+1
(a) DT policy.
(b) RNN+DT policy.
(c) RDT+DT policy.
RDT policy.
Figure 7: Decision tree policies with different history extraction models.
Table 5: Performance different history extraction models for decision tree policies on ADNI Unless shown, standard errors were all < 0.02. History-extraction and action-prediction trees have respective depths dy and da.
History extraction model
dи
dA
# Parameters
Interpretabiliity
AUPRC
Brier
None (DT)
3.7 + 0.4
36 4
9.3 + 0.3
0.72
0.25
RNN+DT RDT+DT
4.8 1 0.2
253 1 6
5.0 1 0.5
0.83
0.17
3.3 1
0.4
2.5 1 0.3
167 + 12
6.3 1 0.3
0.80
0.21
RDT
3.3
+
0.7
122 18
8.3 0.3
0.82
0.18
The unexpected result that decision trees perform history representation nearly well neural networks can be attributed to the sparse and heterogeneous nature of our time-series data, often better captured through models based on decision trees (Frosst & Hinton| 2017). however not expected that RDTs would be applicable to complex, structured history embeddings as in speech or language modelling.
17

---

Published as a conference paper at ICLR 2022
Leaf recurrence models An important step in model design consists of choosing leaf parameters for history prediction. In increasing order of complexity in terms numbers parameters per leaf, Table highlight number of possibilities, where {0, R 8, RM 0, RMMM Og RMXDD are trainable parameters for each leaf (for history and observation dimensionality M and D respectively) and @ denotes element-wise multiplication. As for action prediction, the simplest solu- tion is for leaves to be independent of inputs, outputting fixed history vector either distribution over categorical values with equation| [1, or a vector of continuous values through equation [2 De- pendence on previous history - and on the newest observation žr-1 be incorporated through element-wist wise or matrix multiplications (equations 13 17 simple RNN unit.
In particular, equation corresponds to
Table Performance different Ieaf models for history recurrence on ADNI. Standard error on tree depth d is omitted for brevity.
Recurrence equation
# Parameters
Accuracy
Warm Start Acc.
h++1 softmax (0..)
(11)
2.7
114 + 11
0.72 0.02
h++1 tanh(O,)
(12)
3.0
130 + 23
0.76 0.03
h++1 tanh(e; + 0, @ h4)
(13)
2.7
154 + 26
0.68 = 0.01
h++1 lanh(On, + 0)
(14)
3.0
335 - 46
0.76 0.01
0.73 + 0.01
he+1 tanh(0, + 0r @ lee 0f2))
(15)
2.3
306 + 19
0.75 0.02
0.73 + 0.01
h++1 tanh(0), + h,)
(16)
3.3
522
+ 110
0.73 = 0.02
0.77 : 0.02
h++1 tanh(6), (6), + 0, + 0z2))
(17)
3.7
847
1 152
0.76 0.02
0.77 1 0.01
More complex leaf models tend perform best on ADNI, at the cost of tree readability and computa- tion cost. Still, using fixed Icaf history parameters through a hyperbolic tangent activation (equation
12 result in second-best performance with minimal number of parameters, and intuitive readabil- ity justifying the use of this parsimonious model for all other investigations. Initialising parameters with values previously optimised for the RNN+DT recurrence model, in warm start, found to achieve similar performance, but did not systematically improve results. A caveat should be added, as some Docision-making environments require more complex recurrence models to capture sub- SYNTH dataset
tleties in patient history. For instance, policy optimisation for our proof-of- was challenging with iistory-indepenie models, and recurrence equation [16 was used instead. Overall, this architecture choice depends on the complexity of the modelling task and on the nature of demonstrated trajectories.
B.2 IMPROVEMENT ON ESTABLISHED DECISION TREE MODELS
We conclude this section with a comparison of our method with the traditional Classification And Regression Tree (CART) and soft decision tree (SDT) algorithms in Table 7 to highlight tributions. In particular, our work combines soft and grown tree architectures (Frosst & Hinton 2017 Tanno et al. 2019) and pioneers their application time-series data instead of images. We formalise and study the conversion of multidimensional gating functions to axis-aligned ones for in- terpretability in Section C Our final contribution is an entirely novel approach to model time-series through a multi-output recurrent tree structure, extending the representation learning model of Ding al. (202T) to the sequential setting and improving its interpretability by marginalising out obscure latent variables.
Table 7: Comparison of our proposed architecture with traditional, soft and cascaded decision frees
(SDT) as in Breiman et al. (1984), Frosst & Hinton (2017) and Ding et al. (201)).
CART
SDT
CDT
POETREE
MODELLING TASKS
Discrete, categorical, continuous outputs Multiple outputs (e.g.. (as, kt, =c}) Multidimensional decision boundaries
X
/
/
Interpretahle decision boundaries
X
X
demendepede handling via recurrence
X
OPTIMISATION
Optimisation objective Probabilistic decision boundaries
Gini impurity
Prediction error
Prediction error
Prediction error
/
Gradient-descent optimisation Tree depth growth & path pruning
X
/
/
X
X
X
18

---

Published as a conference paper at ICLR 2022
While mudieent-booste decision tree structures have been reported obtain good performance time-series forecasting tasks (Hyland et al. 2020p. such ensemble methods result multiple tree structures to navigate which negatively affects interpretability (Lage et al. 2018). In addition, man- ual pre-procsssing of the time-series data is required to include history information, which our model avoids by independently learning history embeddings and marginalising them at interpretation time.
C AXIS-ALIGNT DECISION TREE MODELS
For easier human interpretation, decision policies should be represented with deterministic, axis-aligned thresholds at each inner node. We propose a post-processing method to achieve this from multidimensional trees, well variant of proposed architecture directly learn axis- aligned decision boundaries during training, and optimise policy closer to the structure used at inference time.
C.I MULTIDIMENSIONAL TREE PRST-SSOGSSIIN
final technical consideration required by the model architecture proposed in Section policy post post-processing for human interpretation. After optimisation, our decision trees consist of multidi- mensional gating functions over the history and observations variables which are difficult to under- stand as illustrated in Figure Sa and must be converted to axis-aligned, unidimensional thresholds.
Each inner node thus consists of learned parameters {b, w) and computes the probability of taking the right-most branch as follows:
M h4 + w;
D Σ
Paote (lte 2:) = b+
Wm-+s Zz,i
(18)
i=1
where M and D are history and observation space dimensionalities. indices are omitted for clarity.
Superscripts denoting node
The history vector is first marginalised out into the bias term, to condition the resulting observation- to-action policy on the history as desired. This results in specific tree model for each timepoint, which only depends on observation 2t for easier human interpretation. Gating functions are re- expressed as follows:
Poate (21) = 2008'' L Wm+s Z4,6
(19)
where = E w. he is now a history-dependent bias term.
As in Silva al. (202), these time-dependent multidimensional policies are then converted into axis-aligned unidimensional trees by taking the largest component of the remaining gating parame- ters at each node, wmazi
Pgate (2t) > 0.5
b + Wmar ",,mar > O, where Wmaz = max(wm+=))
(20)
= 71,ma bb Wrraa
(21)
(or 2t,mar b' WWmax if Wmma
0)
(22)
Figures 8b and 8c are examples of axis-aligned policies at different timesteps for typical patient (with degrading MCI symptoms), much easier to follow than the multidimensional tree in Figure 8a
Table 8: Performance of multidimensional and different axis-aligned decision tree policy structures. Stan- dard errors all 1%.
Adaptive decision tree policy
ADNI Accuracy (%)
MIMIC Accuracy (%)
Multidimensional Axis-aligned; bias b'
77.6
79.9
74.7
75.9
Axis-aligned; bias D' + immax Wi 24,1
76.9
77.8
Axis-aligned; bias /' + Lifmar Wi 3r.i
75.4
76.0
Axis-aligned thresholds were also improved by exploiting our framework to predict expected obser-
vations at the subsequent timestep ž,1 Thresholds were adjusted by incorporating the predictions
19

---

Published as a conference paper at ICLR 2022
12
42
a2
at
(a) Multidimensional policy.
low
Gl right
Vi not lowe
CDR- sa guestionable
Go right
CDR SB nocmaal
No prevsous MRL
CDR-SR severo
Vy
CDR: SB noemal
No previous MRI
MRI
[MRI]
MRI
MEII
MERE
MRI
MRR
MRI
SB noemal
[Mа]]
MKI
[MK]]
(b) Axis adisnaligned policy,
(c) Axis aligned policy, t
Vh low
CDR-SB severe
CDR-SI
No previous
MRI
MRI
MRI
(d) Axis axis-aligned policy after pruning, t
(c) Axis-aligned policy after pruning,
Figure Transformation of multidimensional tree policy into axis-aligned and pruned structure. History and expected evolution outputs are not represented for clarity.
20

---

Published as a conference paper at ICLR 2022
for all components except mar in the bias term, transforming equation 21 into:
6' + Liymaz Wi 2
2t,max
(23)
Wmar
As shown in T incorporating true and expected observation values in the thresholds improved the axis-aligned model performance. Naturally, true observations for the corresponding timestep are unavailable, and can therefore not be used to adjust the threshold; only expected observations can be used for this. This investigation highlights another benefit of the expected patient evolution model, in addition to explanatory insights.
Further post-processing of axis-aligned policies in Figures 8b and 8c is required in the form of node pruning, setting minimum path probability threshold on the validation set, to address the following concerns:
Axis-aligned thresholds are often sensible and typically in the [0, 1] range for categorical variables, converting straightforwardly to meaningful decisions. Negative or out-of-scope thresholds, however, exclude all data from child path (e.g., left branch of the "Go right" node in Figure 8b), which can be pruned from the tree.
Learning interdependencies between variables within the hicrarchical structure is also chal- lenging. For example, in Figure [8b] the first right branch (low Vn previous scan) is incompatible with the following right branch (not low V₁). In practice, we obtained this structure due (1) soft partitions, with examples shared across incompatible branches; and as (2) multi-dimensional gating affects thresholds in training. Again, the solution is to eliminate non-sensical children nodes in post-processing.
Accuracy loss was found to be virtually null (< 2%) with pruning probability threshold of 0.05 on the validation set Dual Formally, average path probability for node was estimated as:
Pd (Dvvt)
Σ
P' 2t)
(24)
DDuat
(NNiie) E Doat
and nodes with (DDaa) ) < 0.05 were eliminated from the structure. Indistinguishable performance of the multidimensional and axis-aligned tree mode was even obtained on ADNI by training the multidimensional structure with an increasing L1 regularisation weight, to allow greater flexibility initial stages and induce more axis- axis-aligned structure at the end of training. With an LI weight from 10 to 10 this resulted in an action-matching accuracy of 0.75 + 0.01 and AUROC of 0.52 + 0.02 with both multidimensional and post-processed models.
Overall, axis- trees therefore offered excellent performance, comparable to multidimensional ones. As the observation space dimensionality increases, however, approximations become less accurate. To overcome this, trees can be trained closer to their axis-aligned version during opti- misation, for instance with LI regularisation to induce sparsity the gating functions. On a 15- dimensional observation space with the SYNTH simulated policy, an Ll-regularisation weight of 0.01 on gating parameters decreased multidimensional action matthhnn accuracy by 0.7% overall, but improved axis-aligned performance by 5%. This suggests promising avenue ensure the scalability of our model.
C.2 AXIS-ALIGNED TREE TRAINING
In this section, propose variant to optimisation algorithm in order to directly train structure that is closer to the one used in inference time.
The following gating function was designed to allow for axis-aligned training while keeping the tree structure differentiable:
d
Pgate (x) = II =
(m:w; + bz)
(25)
for an input x E Rdd. This function defines a soft AND gate over d axisealigned thresholds, where threshold value -b; w. is learned for each input dimension At inference time, each inner node consists of d axis-aligned thresholds, one for each input dimension, which must all be satisfied to proceed to the right child branch.
21

---

Published as a conference paper at ICLR 2022
In recurrent architecture, multidimensional gating functions on the history embedding hy can be kept for greater model flexibility, since these are marginalised at interpretation time. Axis-aligned gating functions can be applied to the observation values 2, only, such that:
Poate (he: 21) = a (hą w' +6) x II a (2,, + b.) 1=1
(26)
where a is the sigmoid function and (w',b, (wy, by 3 are leaf parameters.
Action-matching performance. Overall, as shown in Table 2 slightly superior performance was still obtained from training and post-processing multidimensional soft architectures in comparison to axis-aligned ones. This greater performance may be explained by greater flexibility during training time, particularly as tree structures are grown from low-depth trees with poor discrimination if only considering unidimensional partitions.
Table 9: Performance of multidimensional and axis-aligned decision tree policy structures on ADNI. The static setting corresponds learning mapping 2; to dç-
Static setting
Recurrent setting
Tree structure
Accuracy
AUROC
Accuracy
AUROC
Multidimensional
0.75 + 0.01
0.53 + 0.01
0.78 + 0.02
0.62 = 0.01
exis-aligned processeded from multi.)
0.73 0.01
0.52 + 0.02
0.76 = 0.02
0.59 = 0.01
Trained axis-aligned (SDT)
0.72 0.02
0.52 + 0.02
0.75 + 0.02
0.56 = 0.02
Trained axis- axis-aligned (CART)
0.69 + 0.01
0.50 = 0.04
Better performance over a deterministic tree structure (CART) was also obtained. On our relatively small datasets of few thousand patients, sharing information from each patient across every node through probabilistic gating functions ensures no part of the tree is trained on excessively few exam- ples this may help combat overfitting, which decision trees are often subject (Frosst Hinton, 2017]. An additional source of difference in performance at test time is that our differentiable struc- ture trained by gradient descent, while traditional trees are built by Gini impurity splitting.
D ALGORITHM IMPLEMENTATION DETAILS
Our POETRER algorithm implemented as follows for our experimental investigations Models were built using the open-source automatic differentiation framework JAX
All observation inputs were normalised prior modelling. Hyperbolic tangent outputs for ž and axis-aligned thresholds were mapped back to the observation space after training for human- readability and interpretation. History representation learning was integrated in the decision, through the recurrent decision tree structure described in Section 3.2 Leaf models for history prediction were independent of previous history: h,41 tanh(b), where R.m a trainable parameter for each leaf and m is the history dimension.
Following Algorithm [ tree structures were initialised from depth 2, to avoid excessively simple local optima. For tree growth, limited to depth 5, the AUROC score was used a measure of validation performance, to determine whether to split leaves into new inner nodes. All global and local optimisations were carried out until convergence no improvements on the validation set for 50 update iterations. We used the Adam optimiser (Kingma & Ba 2015) with a step size of 0.001 and batch size of 32. Model hyperparameters in Table 10 were optimised through grid search on validation datasets- random subsets of 10% of the training data.
All experiments were with 5 different model initialisations and data splits and the average results and standard errors were reported. Optimisation of tree structures was found to be temperamental, with some initialisations not improving in performance with training. A restarting procedure was therefore implemented if loss did not decrease by more than 5% over the first five epochs; and was also applied benchmark algorithms if necessary. Experiments were performed on a Microsoft Azure virtual machine with 6 cores and powered by Tesla K80 GPU.
*Code made available at Attps://gght con/a" zeepace//oet and https: :
github. wtchub.com/vanderrr... wtt..///wwwe.caaaaaaa.aeo ab hwtpb//mloorrirrr forhealth abpub
https://githhb. wtt.::ttbbb..c.coooooooooteeoote
22

---

Published as a conference paper at ICLR 2022
Table 10: Hyperparameter grid for POETREE optimisation, with values optimised for the ADNI dataset.
Hyperparameter
Search range
Optimal values
History dimension m
- 20
Spliting penalty À
10-4 10a
10-1
Evolution prediction 81
10 10-3 - 100
10
Evolution prediction 82
10-4 10-4-10-1
10-3
The results of a brief study of the impact of different loss terms is provided in Table 11 The choice for the loss of the tree structure itself was first studied, comparing the cross-entropy between targets and each leaf's output distribution at, weighted by their respective path probability P' (2(), the cross-entropy with the weighted average of all leaf outputs (I'), and the cross-entropy with the output of the cokinuum-rrobablin
Ablation Study
leaf (moa (L"). The first objective
function, proposed by Frosst & Hinton (0017). returned marginally better results, as it may better capture the different contribution from each element of the structure.
Table 11: Performance of decision tree policies optimised with different objective functions ADNL CE(, a) is to the categorical cross-entropy loss with respect to target a.
Objective function
Action-matching accuracy
Relative MSE on & (%)
I. = E, (2) Ca(a'
0.78 0.01
60 + 10
L'= = -E P')))
0.76 = 0.01
L" CE(a'maz a) where bouz max; P'(2)
0.77 = 0.01
L ())(2(11)
0.75 + 0.02
7 + 4
I. - Log
0.77 + 0.02
16 + 6
L L Lippitt
0.77 + 0.01
The performance of our model was also evaluated under the addition of the expected evolu- tion regulariser Izg, itself composed of a mean-squared error term on the predicted observation, MSE(=++1) = 61 ||=++11 E++1ll², 1|2 and a term penalising inconsistent action choices based on this prediction, 5g DKL (- #t+1 1|+11|=([
=)+1)) (-1+))=1=+++)) Results suggest that the additional evolution loss term Lg allows to balance consistency to the policy (by restoring the action-matching perfor- mance degraded with only the MSE term and fidelity to the observation evolution. Finally, the
splitting regularisation term Lsplit improved consistency between runs but did not largely improve performance.
Complexity Analysis This section highlights the benefit of our tree growth procedure for both interpretability, complexity and performance optimisation. Figure 9a compares results obtained from trees of fixed depth and trees grown during training. Incrementally grown trees outperformed all fixed structures with a minimal number of parameters (with, on average, 40% fewer parameters than fixed tree of the same depth) and reasonable training time (faster than fixed trees of depth 4 or 5, despite multiple local optimisations). Trees grown during the training process thus learn to capture general hierarchies in the data while eliminating unnecessary partitions, which otherwise make the tree more difficult to read as depth increases (Lage et al. 2018).
comparison of performance against tree depth after optimisation is also proposed in Figure 9h demonstrating that our incrementally grown structure adapts its complexity to the the task at hand. Indeed, while performance on the training set continuously increases with tree depth, test accuracy is optimal at intermediate model complexities which avoid overfitting while capturing subtleties in the data. With an average tree depth 3.3 + 0.7, our optimisation procedure therefore success- fully implements this Occam's 's razor. Tree depth was also found to adapt sample complexity, as optimised depth decreased to 2.4 4 0.3 when input dataset size was halved.
Table 12: Complexity analysis of different policy learning algorithms on ADNL Runtime is measured as computation time for the prediction of all test actions (10% of demonstrations trajectories).
Model
Optimisation time (s)
Runtime (s)
POETREE
205 + 32
POETREE with inductive bias (Figure INTERPOLE (Hyyük al. 2021
141 + 24
<
752 - 63
26
23

---

Published as a conference paper at ICLR 2022
Grown tree
Average optimized depth
70
78-
Accuracy (%)
74-
Accuracy
74
72-
70-
68
68
Test
Training
66
84
Fixed tree depth
Grown tree depth
(a) Fixed trees
(b) Grown trees
Figure 9: Performance on ADNI as function of tree depth, for fixed and grown trees. Our growth optimisation procedure adapts model complexity to the task at hand to (a) facilitate interpretation and (b) avoid overfitting.
Finally, Table 12 highlights the superior efficiency of our algorithm training and runtime over its closest related work, an interpretable PO-BC-IL model. This can be understood from the expensive computation of Input-Output Hidden Markov Model (IOHMM) parameters to represent the POMDP environment in the latter (Bengio & Frasconi T995). Model optimisation was accelerated when initialising the tree structure with a sensible clinical policy as detailed in Section F
E BENCHMARKS IMPLEMENTATION DETAILS
All learned POMDP environment models are implemented as an Input-Output Hidden Markov Model (1OHMM) (Bengio & Frasconi 1995) and initialised uniformly at random. Benchmark im- plementation is similar to Hüyüük et al. (2021) as this work shares our goal of interpretable policy learning. Online-learning algorithms such as explainable DM-IL (Li al. 2017) could not be used for benchmarking, as our medical datasets do not allow interaction with the decision-making envi- ronment.
PO-BC-IL The recurrent neural network is implemented as an LSTM unit (Hochreiter & Schmid- huber 1997) and fully-connected layer, both of size 64; and trained through optimisation of the categorical cross-entropy over actions using the Adam optimiser (Kingma & Ba 2015) with learning rate 0.001 and batch size 32 until convergence.
PO-IRL 10HMM parameters are first trained (Bengio & Frasconi 1995) and freezed. The reward function is initialised as R(s, a) ~ (O, 0.0012) and is estimated as the average of 50 Markov Chain Monte Carlo (MCMC) samples, retaining every tenth drawn value after burning-in in 500 samples. In Offline PO-IRL, 10HMM parameters and reward signal are jointly learned by MCMC sampling. Q-values are then recovered with off-the-shelf POMDP solver
After training and freezing 10HMM parameters, policies are parametrised in terms of decision boundaries as in Hüyük et al. (202I), and initialised randomly. Training is carried out by maximising the likelihood of actions in the demonstrations dataset by Expectation Exppetation-Maximisttioo Maximisation. Maximisation is carried out with the Adam optimizer (Kingma Bal 2015) with learning rate 0.001 until convergence.
PO-MB-IL
Interpretability benchmarks Our fully-observable behavioural cloning benchmarks include lo- gistic regression and a static version of POETREE illustrated in Figure 2a Feature importance analysis extracts each input dimension's contribution to the predictions of the PO-BC-IL model through Shapley values (Lundberg et al. 2017)
httrass/www..omb.oon///do ndw.. https://scikit-leem..ooo
html 1
24

---

Published as a conference paper at ICLR 2022
F DATASET DETAILS
F.I PROOF-OF-CONCEPT POLICY
For the SYNTH dataset, treatment efficacy on diseased patients was set to 40%, with no spontaneous state changes and an initial 80% diseased population. Diagnostic tests were given a 99% precision and 5% false alarms.
F.2 THE ALZHEIMER'S DISEASE NEUROIMAGING INITIATIVE (ADNI)
Rare visits without CDR-SB measurement were discarded, as well as visits separated by more than 6 months from the previous one - leaving 1,626 patients with at least two visits and a median of three visits. Average MRI hippocampal volumes taken within half a standard deviation from the population mean.
Published guidelines for Alzheimer's disease investigation are reproduced in Figure 11 With the help of one of the surveyed physicians, this was distilled as a static tree policy in Figure [2 to initialise POETREE and assess the effect of inductive bias on optimisation performance. Gaussian noise of variance x² = 0.1 was added to initialisation parameters. Table 12 evidences the complexity gains afforded by this incorporation of prior knowledge, with action- matoonnngnhiig performance reaching 0.62 + 0.01 AUROC, 0.84 + 0.01 AUPRC, and 0.18 + 0.01 Brier score.
F.3 MEDICAL INFORMATION MART FOR INTENSIVE CARE (MIMIC-III)
Patients trajectories from the MIMIC dataset with missing values in one of the observation fea- tures, or with non-consecutive measurements, were eliminated. In total, 4,222 ICU patients were considered over up to 6 timesteps.
Policies recovered for the MIMIC antibiotic prediction task by our algorithm and the decision- boundary framework (Hüyük et al. (02T) are given in Figure [3] The latter again falls short of highlighting how patients are mapped onto the belief state, giving our decision tree greater expres- sive power. Our results are consistent with policy learning work the same task and dataset (Bical jet al., 202T), which identifies similar discriminative features: fever and abnormally high or low white blood cell counts known clinical criteria for infection and antibiotic treatment, if bacte- rial source is suspected (Masterton et al. 2008). Continuous variable thresholds evolve over time through their dependence on history: the temperature criterion, for instance, adjusts to whether the patient has a prior for infection.
G DISCUSSION WITH CLINICIANS
To evaluate the interpretability of our decision tree policies, we consulted five practising physicians for their feedback on our work in comparison to the otate-of-the-art. Each physician was given short presentation introducing (I) our goal of representing the decision-making behaviour in an intelligible way, (2) the ADNI dataset and the studied clinical environment, (3) how policies can be represented in terms of decision boundaries over belief simplices (Hüyüik al. (021). black- box mappings from beliefs to actions (Sun et al.) 2017). and direct mappings from observations to actions (as in work). All participants were unaware of what method proposed, to minimise any implicit bias in results.
Ethics Statement Our survey does not require formal ethics oversight as the identity of our clin- icians cannot be obtained from published information (see U.S Department Services 45 CFR 46. 104())(2) and sensitive private data was shared with nor obtained from participants. We followed the same procedure as in Hüyük et al. (22TT) to ensure a fair com- parison to this related work.
Health and Human
All five clinicians expressed the following preference in terms of interpretability of the decision- making process:
RNN < Decision Boundaries < Decision Trees
where the RNN policies, learned by PO-BC-IL (Sun et al. from observation to history, and from history to actions. This was illustrated in our survey through
2017 correspond to black-box mappings
Table [3] Decision-boundaal policies correspond to the INTERPOLE benchmark in Figure 3b (Hüyūk
ct al. 2021). Finally, decision tree policies learned through our POETREE framework were illustrated
by Fig Figure 3a
25

---

Published as a conference paper at ICLR 2022
Hoaldy
Neatsby
Vp low
Va low
CDR SD normal
CDR SB severe
CDR SD neemall
CDR SO severe
6+1
2+1
F+
F++1
MC7
Demontie
MC7
Demoniis
Va low
Vy low
CDR SB
CDR-SB
CDR BB quesicoable
CDR SBB severe
MIRI
CDR-SB SB questionable
CDR severe
MRI
SRRI
MRI
(a) Patient A: Healthy
(b) Patient B: Degrading
Heattyy
Va low
CIDR-BB
MRI
MRI
CDR-SB severe
CDR-SB qucationable
CDD-SB
[MR]
X
MRI
(c) Patient C: Dementia
Figure 10: Adaptive decision tree policy for three typical ADNI patients, with each path highlighted in blue.
No AD diagnosed
No AD
Diognantic mitt
AD diagnosed (faise posttive)
MMSE (if needed)
treatment (donepexi))
diagnosed ifolve negotive)
Mild AD
Diagnastic test
Patient consulting for symptoms
neededd)
treatment (donepezil)
No AD diagnosed negotive)
AD
Dag nogticiiic test
AD agnoseed
MMSE (I needed)
treatment (donspeait)
diggoosed
trezbment (msomantios)
Figure 11: Published clinical diagnostic policy for patient with MCI symptoms. Figure reproduced from Biasutti et al. 2012).
26

---

Published as a conference paper at ICLR 2022
CDR-SB severe
Yeu
CDR-SB questionable
Vh low
Vh low
No previous MRI
MRI
X
MRI
Vh above avg
MRI
MRI
Figure 12: Policy initialisation for inductive bias analysis, based on Figure 11
T> 38
Hematoerit < 36
WBC > 12
Antibiotics
WBC <
Antibiotics
Antibiotics
Antibiotics
Healthy
Infected
Healthy
Infected
(a) Decision- Decision-boundary policy (Häyūk et al. 202
T> 399C
> 37.8%C
no
Jow
WBC > 11
Antibiotics
Antibiotics
(b) Adaptive decision tree policy (only timesteps visualised).
Figure Recovered policies MIMIC. Units for white blood cell count (WBC) are 1000/mm3.
27

---

Published as a conference paper at ICLR 2022
Although more participants would be required for greater reliability, this survey useful firm end-user's s preference for our representation of behaviour through trees, and to validate the pri- orities of our work. Additional comments the belief space and decision-boundaty policy (Hüyükk) et al. 2021 are summarised below:
Classifying patients within discrete disease states aligns with the cognitive process followed in clinical practice (O' et al., 2008), but is challenging to achieve in practice, as patients often fall on a continuam of symptoms and illness severity.
Visual representations of belief states can misleading. particular, the triangular rep- resentation in Hüyük et al. (2021) does not evidence the linear progression from normal to MCI dementia, considered the traditional patient evolution (NICE) 2018). addition, higher-dimensional, hierarchical or overlapping belief states cannot be readily visualised.
The mapping from patient observations Z to belief space S, and the evolution of patient trajectories in this space as new information is acquired, unclear.
Feedback decision tree policies was also collected, highlighting important advantages and limitations:
The flow chart captured by the decision tree was described by clinicians most similar to their own approach to care, hierarchically eliminating possible diagnoses, treatments or investigations
agreement with cognitive research Zylberberg al.) 2017) and estab- lished guidelines illustrated in Figure
The concept of action value (for diagnostic tests, value of information; for treatments, patient improvement) captured in an easily understandable way in terms of expected patient evolution. In contrast, they noted, reward functions and belief updates proposed by related work (Makino & Takeuchi 2012 Hüyük et al. 2021) do not represent the medical thought-process faithfully.
Although the recurrent decision tree mapping from leaves to a subsequent tree policy was appreciated, there remain limitations in terms of the interpretability of the history embed- ding. particularly with complex leaf recurrence models. In further work, we could visu- alise history embeddings over a belief simplex following Hüyük et al. (202T), conditioning vertices to encode distinct beliefs about the patient condition (e.g., diagnoses). Different regions of belief space would thus be associated with different tree policies, rather than single most-likely action.
28

---

Published as a conference paper at ICLR 2022
Table 13: Patient trajectories under a policy learned by PO-BC-IL. History embeddings are obtained
through recurrent neural network.
Ohservation 24
Timestep
History h
Action ae
CDR-SB
Previous MRI
Normal
Average V,
+0...-0.5. 10,00000.000..
K
Normal
X
-0., 144-1110-007--0999--------100
X
Normal
X
00,0
X
Normal
X
40.
X
(a) Patient A (Hcalthy)
Observation 2
Timestep
History h
Action o,
CDR-SB
Previous MRT
Questionable
Low Vi
-0.7
Questionable
Low
+0.3-0..--22--6--02--111
MRI
Questionable
Low
MRI
Severe
Low V2
+0.8,
(b) Patient B (Degrading)
Observation zt
Timestep
History hu
Action d:
CDR-SB
Previous MRI
/
Severe
Low V2
-0.9, +0,, -0.9
X
[
2
Severe
X
-0.7 -0.
X
f
3
Severe
X
X
f
Severe
X
X
(c) Patient C (Dementia)
29


Reviews:
[{"review_id": "CQPeynFU0vj", "reviewer": "Reviewer_wNRz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I don't think the paper meets the requirement of ICLR in a few aspects:\nStrengths:\n- The motivation is strong, especially for clinicians\n- The paper is well written and easy to understand.\n- The experiments are completed and detailed , and  the results seem strong to me ( noted that I'm not a expert in clinical datasets)\n\nWeakness:\n- Some important classical machine learning methods are missing(e.g SVM, GBMs), which meet the needs of interpretability rather than deep learning methods. The policy learning methods(IL, AL, IRL) are not related to me as the reward function is not important in clinical settings. Some important works ([1][2]) for time-series data were also missed for discussion.\n\n1) Ke G, Meng Q, Finley T, et al. Lightgbm: A highly efficient gradient boosting decision tree[J]. Advances in neural information processing systems, 2017, 30: 3146-3154.\n2) Chen T, Guestrin C. Xgboost: A scalable tree boosting system[C]//Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 2016: 785-794.\n\n- Novelty: the technical contributions are marginal to me as the main aspects of the proposed method are the combination of RNN and Soft Decision Trees.\n\n", "summary_of_the_paper": "  The authors argued that many methods failed the merits of interpretability in some important areas, e.g. clinical decision-making. Thus, this paper proposed a (soft) tree-based method for synthetic clinical datasets in the matter of interpretability. The authors model the clinical decision process as a partially observable Markov Decision Process (POMDP), which naturally fits the assumption of medical diagnosis.", "summary_of_the_review": "I'd give a rejection based on the comments above. And I would like to suggest the authors revise the paper for re-submission to another conference.\n\n----Post Rebuttal---\nTo Reviewer kv5i. To be honest, I'm still not convinced that the issues raised in this paper are more artificially fabricated than they are actually present. Clearly, if the goal of this paper is to discover machine learning models that can be used to explain, then decision tree models and support vector machine models are superior choices when compared to other benchmark models, and in fact, the interpretability of decision tree models would not lag behind the methods posed by the article. It is possible, however, that the assumption made about the problem encountered in the clinical data (a Markov process with partial observability) is ill-posed, i.e., that the requirement to take into account the impact brought by the medical regimen is incorrect.\n\n---- Post Post-Rebuttal ---\nI appreciate the author's thorough and precise arguments. After several days of consideration, I reviewed the entire paper and decided to increase my score. Thanks for your effort.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ABJhRxSzQ_-", "reviewer": "Reviewer_kv5i", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am admittedly not an expert in imitation learning/behavioral cloning, policy learning, or soft/probabilistic decision tree models. Nevertheless, having more familiarity with offline reinforcement learning and healthcare broadly, I was excited to read this paper. \n\nThe central premise of the paper - distilling inherent clinician policies down to an easily interpretable (and easily followable) decision tree model - seems like a compelling and important task, and the authors motivate this well in their introduction, highlighting the unnecessary costs of medical practice variability.\n\nThe algorithm is explained reasonably well in Section 2 (\"Problem Formalism\") and Section 3 (\"Interpretable Policy Learning with Decision Trees\"). \n\nSome of the notation was a bit confusing, but this is more of a minor point. For example, $a_t$ is a one-hot encoded target (top of page 4), but then $a_{t, k}^l$ is also the output probability for action class $k$ in leaf $l$ - why not denote the predictions using $\\hat{a}_{t, k}^l$? \n\nThere were also a few confusing sentences. Consider this on in the first paragraph of Section 3.2: \"Finally, as third leaf output with parameters $\\theta_z^l \\in \\mathbb{R}^D$, our model also predicts \\emph{patient evolution}, or observations at the next timestep $\\tilde{z}_{t + 1}$; to formalise a consideration of expected treatment effects (Yau et al, 2020).\" Not only is the line after the semicolon not an independent clause (making the use of the semicolon incorrect), it's also unclear to me where this line about expected treatment effects came from. What does patient evolution have to do with expected treatment effects? There's another related line in the \"Action value quantification through counterfactual evolution\" section that reads, \"Overall, thanks to its integral part of model design, assessment of counterfactual evolution becomes intuitive\". I was very confused by this line. What is the counterfactual here? And what does this have to do with the \"integral part of model design\"?\n\nOn that note, it was unclear to me why a hyperbolic tangent function is used to estimate $\\tilde{z}_{t + 1}^{l_0}$. \n\nI understand the value of introducing differentiable nonlinearities for the goal of flexible function approximation, but I thought that $z_t$ was supposed to be in the original output space? Is $\\tilde{z}_{t + 1}^{l_0}$ not also supposed to be in that same space? But using the hyperbolic tangent would map everything to $[-1, 1]$. Some clarity here would be helpful.\n\nI thought the comparison to related work was done well, and despite not being intimately familiar with the field I was able to appreciate both the existing challenges and opportunities present at the time the work was carried out. Table 2 was especially helpful for contextualizing the paper's contributions. Admittedly, it does seem that this paper is more of a convex combination of a few different ideas (soft/probabilistic decision trees from Frosst & Hinton [2017], cascaded trees from Ding et al [2021] for recurrence) rather than a seminal and novel work in its own right. The authors highlight differences between their approach and vanilla CARTs/Stochastic Decision Trees (SDTs) in Appendix B.2, but I didn't see a comparable exposition of differences and explanation of novelty/contribution relative to Ding et al (2021). That being said, if you believe (as I do) that creativity is just synthesizing existing ideas in novel ways, then I suppose you could call this paper creative. The question, then, is how much more useful the proposed amalgamation of ideas is over each idea in isolation?\n\nI was impressed by the experimental results. It does seem that there's a tradeoff between interpretability and performance (e.g., AUROC), but this particular approach at least seems to be on the current pareto frontier of this tradeoff and provides a useful approach for practitioners.\n\nI thought ADNI dataset application was a compelling one, and the problem itself was explained adequately; however, I really had to sit and scratch my head to work through Figure 3. For a paper nominally dedicated to producing interpretable decision policies, I had a hard time interpreting the policy represented in Figure 3(a). I think the vignettes of Patient A/B/C is insightful, but would need some additional guideposts to add to the paper rather than detract from it. What if, for example, you were able to color and label each of the tree branches with a different color for patient A, B, and C? (Or at very least indicate the path through the tree for each patient separately in the appendix). Also, it was unclear to me whether CDR-SB was being treated as a categorical or an ordinal random variable. For example, in the \"MCI\" sub-tree there is a leaf for \"CDR-SB questionable\". If the answer to this is \"No\", then does that imply that the value could be either \"CDR-SB severe\" or \"CDR-SB normal\"? Is an MRI really warranted in either case?\n\nI was also a bit lost with regards to the \"Decision-making uncertainty\" and \"Anomalous behavior detection\" sections. I think most practitioners would interpret the extracted policy as a deterministic one (not a probabilistic one), yet the underlying construction of the tree is inherently probabilistic. How are we to reconcile those two in terms of interpretation? As a concrete example, the statement is made, \"Visits where an MRI is predicted with 90% certainty make up 8.4% of ADNI\". But this certainty is a reflection of both the entropy in the path to the leaf in the decision tree as well as \"uncertainty\" in the action conditioned on the leaf itself. Disambiguating the two seems important for being able to decide whether an action is truly \"anomalous\" (because taking action 1 in leaf L is inappropriate) or just unlikely because there are very few patients that are represented by a particular leaf node. Maybe I'm misunderstanding something more fundamental here, but I'd welcome any additional clarity here.\n\nOne other note: the authors state, \"We must highlight the similarity between our decision tree policy and published guidelines for Alzheimer's diagnosis, reproduced in Appendix F\". These two policies were not at all similar, in my reading. Figure 10 makes no explicit mention of MRI, hippocampal volume, or the CDR-SB, nor is it recurrent. Figure 11 exhibits all of the above.\n\n=====POST-REBUTTAL COMMENTS========\nThe authors have thoughtfully addressed most of my concerns. I particularly appreciate the update to Figure 3(a) and I think it makes the figure much easier to parse. With the revisions, I'm happy to increase my score. I'll note, though, that I believe a typo was introduced in Equation (2). Check the parentheses in the far right $D_{KL}$ term of the equation. Also, the term to the left of equality should take $h_{t+1}$ rather than $h_t$ as an argument, I believe. \n\nI've had a chance to read through the other reviewers' comments and the authors' responses to said comments. I agree with reviewer Kc79's original concerns and feel that those have been adequately addressed. The same with reviewer FrRV. Honestly (and this is more for the meta-reviewer) I was left scratching my head at the comments made by reviewer wNRz. The request to compare the proposed approach to SVMs and GBMs doesn't make any sense to me - SVMs and GBMs are inherently not as interpretable as decision trees, in my opinion, and the whole point of the paper was to learn a policy representation in decision tree form that closely matched clinician behavior. This is, contrary to wNRz's concerns, very well suited to tools from behavioral cloning and imitation learning. No reward is needed because the authors aren't taking an inverse reinforcement learning. This is all fine, reviewers can have disagreements. What was most concerning to me, though, was the combination of a very low score by the reviewer and a high confidence rating. The reviewer's comments reflected neither the depth nor understanding that I would expect from such a high confidence rating.", "summary_of_the_paper": "This paper proposes a novel approach for learning and representing human decision-making policies from observed behavioral data. The proposed approach emphasizes interpretability as a primary aim, while nevertheless seeking to maintain reasonable modeling accuracy. The decision tree model proposed extends canonical decision tree approaches to the probabilistic setting, allow for optimization of leaf-specific parameters via stochastic gradient descent. The proposed approach is evaluated both in terms of its interpretability (subjective measurements from a panel of licensed physicians) as well as its accuracy in recapitulating actions conditioned patient observations. The utility of the approach is demonstrated on both synthetic and real-world datasets.", "summary_of_the_review": "Overall, this is an interesting paper. The novelty and significance are positive, if not overwhelming. The utility for practitioners and potential impact in areas outside of machine learning, however, is nontrivial. The exposition of the method and its interpretation would benefit from additional refinement. Nevertheless, in its current state, I can recommend a marginal accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2CAAFVHpTSA", "reviewer": "Reviewer_FrRV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong points:**\n- Rather well written and clear paper.\n- The interpretability of the proposed approached is quantified by clinicians.\n- Interpretability of models is important, especially in healthcare.\n- Experiments on synthetic data and two real data sets.\n- Experiments include a comparison to what seems to be the most relevant existing method, Interpretable BC-IL (Huyuk et al., 2021). I like that at least one example clearly shows a situation where the proposed method is more interpretable than Interpretable BC-IL.\n- I liked the illustration of the model based on real data/examples. This is really nice.\n- Good job balancing the content between the main paper and the appendices. \n\n**Weak points:**\n- I did not understand the first term ($||z_t - \\tilde{z}_t||$) of equation 2. The input of the tree is $z_t$ and $h_t$, the observation at the current time step and history. Why is there a loss over these terms? What is $\\tilde{z}_t$ ? The observation at the next time step is defined as $\\tilde{z}_t+1$ so isn't that first term always 0?\n- The tree is first learned by considering all variables in a node and then simplified by only using one variable in a node. Even though a L1 regularization is used when learning each node, I am concerned about the impact this has and do not understand why a more direct approach such as learning a node directly with a single variable is not considered. See questions below.\n- I might be mistaken, but I think the impact of the different losses is not evaluated in the experiments.\n\n\n**Questions:**\n- Unless I am mistaken, the decision tree is modified after learning to make it interpretable by leaving only the variable with the highest weight in each node. I read appendix C and saw the reported loss of accuracy to be only 2%. Yet as I understand it this is a single empirical experiment. I am a bit uncomfortable, as I wonder whether this could lead to large changes in a policy when many variable have similar weights. I am concerned both with skewing the interpretation of the policy and with accumulations of errors within child branches. Could you please comment on that?\n- On a related note, is there any particular reason not to learn the tree by using directly a single variable in every node? This would guarantee that child branches are directly learned on the partition of the data that correspond to the inference setting.\n- Likewise, is there any reason to not use regular decision trees rather than probabilistic ones? Recurrent decisions trees could be used.\n\n**Details:**\n- I do not understand the purpose of the comparison to RNN in section 3.2.\n\n**Post-Rebuttal:**\nThank you for the detailed answers to my questions. This is an impressive amount of extra experiments. I found the answers very detailed and insightful. Most of my comments have been addressed beyond my expectation. I think the paper is well worth publishing at ICLR in its current state and have updated my score accordingly.\n\nOn the topic of learning an axis-aligned tree:\n- I think the novel approach mentioned in the rebuttal and learning multiple axis aligned threshold at every node is very interesting, but might be more difficult to interpret as many variables will be considered in each node. Nevertheless, it is impressive you could learn such a tree.\n- I now understand that this paper follows previous approaches to align the decision boundaries. I might well have missed it, but reading the paper initially, I thought that trees are not retrained after being axis aligned, which still seems strange to me. I could not see it said explicitly so perhaps I am wrong. Maybe this could be clarified.\n\nOn a final note, I share the sentiment of reviewer kv5i about other reviewers' comments. He said it better than I could.", "summary_of_the_paper": "POETREE aims to construct an interpretable model for a policy over a time series using decision trees. The healthcare domain is particularly targeted. As opposed to other works, the model directly maps observations of a POMDP to actions. POETREE creates a decision tree from time series data. The decision tree can be conditioned on the history, allowing the tree to be different at different time steps, allowing for example the tree to model that an exam done previously that is no longer informative is no longer likely. Each tree is a soft-probabilistic model first grown incrementally by developing, optimized globally (as it is differentiable), then pruned. Finally, the tree is simplified for interpretability by limiting each condition to a single variable. POETREE is then empirically evaluated and compared to baselines in terms of distribution modeling, interpretability and policy learning.", "summary_of_the_review": "*Initial summary*: I think this is a very interesting research on an important topic. Experiments are well done. My main concern is that the simplification of the model might have some bad impact. \n\n*After discussion*: Concerns have been mostly addressed. I think the paper is worth publishing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-VOGldLVgq", "reviewer": "Reviewer_Kc79", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and the problem well motivated. The key contribution seems to be extension of soft decision trees to the recurrent setting which is a nice and clinically useful contribution. The algorithm used to train is certainly a heuristic but seems reasonable. Empirical results are comparable although the algorithm does not seem to improve over state of the art in terms of interpretability always. \n\n\nWeaknesses:\n1. I am not sure how partial observability plays a role and the contributions from the partial observability perspective are unclear. Is the claim that the representations learned due to the recurrent setup is better for overcoming challenges of partial observability? If so I believe an additional evaluation is warranted for comparison. If not then I might have misunderstood and I believe for completeness authors should comment on partial observability. \n\n2. No additional discussion regarding the algorithm and its behavior is provided. I believe a discussion on potential failure cases and also choice of hyperparameters of the cost function will be helpful especially for experiments. \n\n3. I might have missed this but what exactly is the source of partial observability in all the empirical evaluation in the paper? Please add comments regarding the source and how the proposed method addresses it.", "summary_of_the_paper": "This paper proposes a new method to learn (stationary) interpretable policies using soft decision trees in partially observed settings. The soft decision tree structure is extended to allow for recursion over time, and account for policy decisions based on history of collected data. An algorithm is presented to optimize the parameters of the soft decision tree as well as the structure/topology of the tree. The algorithm mainly proceeds by splitting nodes and locally optimizing the parameters of the the associated probability representation of the soft node, and recursively split (if local optimization does not improve validation performance) and fixed as leaf otherwise. A global update step is then used after topology is fixed followed by pruning low probability paths in the trees. Experimental validation on surveys with clinicians demonstrate reasonable interpretability and improved prediction performance on imitating clinician policy. ", "summary_of_the_review": "Overall I believe this paper provides an interesting contribution for interpretable policy learning particularly for clinical decision-making. I do have a few followup questions that will make the contribution regarding some aspects clear. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a tree-based method for interpretable policy learning, for fully-offline and partially-observable clinical decision environments. The models are trained incrementally, as patient information becomes available. 

The method was overall deemed novel by the reviewers, and the interpretability of the model well validated by clinicians.

Numerous points of clarification were brought up by reviewers, related to the notation, learning process and result reporting. All of the concerns were responded to by the authors in great detail and the manuscript was appropriately revised. All the reviewers have raised their scores as a result of the updates.

Thus, the paper is ready for acceptance.
Decision: Accept (Spotlight)