Explaining by Imitating: Understanding Decisions by Interpretable Policy Learning
Authors: Alihan Hüyük
Daniel Jarrett
Cem Tekin
Mihaela van der Schaar
Venue: ICLR 2021 Poster (2021)
Keywords: interpretable policy learning
understanding decision-making

Abstract: Understanding human behavior from observed data is critical for transparency and accountability in decision-making. Consider real-world settings such as healthcare, in which modeling a decision-maker’s policy is challenging—with no access to underlying states, no knowledge of environment dynamics, and no allowance for live experimentation. We desire learning a data-driven representation of decision- making behavior that (1) inheres transparency by design, (2) accommodates partial observability, and (3) operates completely offline. To satisfy these key criteria, we propose a novel model-based Bayesian method for interpretable policy learning (“Interpole”) that jointly estimates an agent’s (possibly biased) belief-update process together with their (possibly suboptimal) belief-action mapping. Through experiments on both simulated and real-world data for the problem of Alzheimer’s disease diagnosis, we illustrate the potential of our approach as an investigative device for auditing, quantifying, and understanding human decision-making behavior.

Published as a conference paper at ICLR 2021
EXPLAINING BY IMITATING: UNDERSTANDING DECISIONS BY INTERPRETABLE POLICY LEARNING
Alihan Hüyük*
Daniel Jarrett"
University of Cambridge, UK ah20750cam cam.cm.....mmmmmmmmmmmmmmmm..mmm ac. uk
University of Cambridge, UK
daniel. jarzett @eaths cam. ac uk
Cem Tekin
Mihaela van der Schaar University of Cambridge, UK
Bilkent University, Ankara, Turkey cemtekin@es Sentekinéee.biikent edu.tr
Cambridge Centre for Al Medicine, UK The Alan Turing Institute, UK; UCLA, USA mv 72:4720a 720cam .k
ABSTRACT
Understanding human behavior from observed data is critical for transparency and accountability in decision-making Consider real-world settings such as healthcare, in which modeling decision-maker's ' policy is challenging- underlying states, knowledge of environment dynamics, and no allowance for live experimentation. We desire learning a data-driven representation of decision- making behavior that (1) inheres transparency by design, (2) accommodates partial observability, and (3) operates completely offline. To satisfy these key criteria, we propose a novel model-based Bayesian method for interpretable policy learning ("INTERPOLE") that jointly estimates agent's (possibly biased) belicf-update process together with their (possibly suboptimal) belicf-action mapping. Through experiments on both simulated and real-world data for the problem of Alzheimer's disease diagnosis, we illustrate the potential of our approach as an investigative de- vice for auditing, quantifying, and understanding human decision- makinn behavior.
no access to
INTRODUCTION
A principal challenge in modeling human behavior is in obtaining transparent understanding of decision-making. medical diagnosis, for instance, there often significant regional and institutional variation in clinical practice (1 much of it the leading cause of rising healthcare costs (2). The ability to quantify different decision processes is the first step towards a more systematic understanding of medical practice. Purely by observing demonstrated behavior, principal objective is to answer the question: Under any given state of affairs, what actions are (more/less) likely to be taken, and why?
We address this challenge by setting sights on three key criteria. First, desire method that is transparent by design. Specifically, a transparent description of behavior should locate the factors that contribute to individual decisions, language readily understood by domain experts 3 This will be clearer per our subsequent formalism, but we can already note some contrasts: Classical imitation by reduction to supervised classification- does not fit the bill, since black-bor hidden states of RNNs are rarely amenable to meaningful interpretation. Similarly. apprenticeship learning pooullllly through inverse reinforcement learning- not satisfy either, since the high-level nature of reward mappings is not informative to individual actions observed in the data. Rather than focusing purely on replicating actions (imitation learning) or on matching expert performance (apprenticeship learning), chief pursuit lies in understanding demonstrated behavior.
Second, real-world environments such as healthcare are often partially observable in nature. This requires modeling the accumulation of information from entire sequences of past observations an endeavor that is prima facie at odds with the goal of transparency. For instance, in a fully obsly-bbeeee setting, (model-free) behavioral cloning is arguably 'transparent' in providing simple mappings of states to actions; however, coping with partial observability using any form of recurrent function
Authhrrs contrihuted equally

---

Published as a conference paper at ICLR 2021
Table I: Comparison with Related Work. INTERPOLE satisfies our key criteria of (1) transparency by design. (2) partial observability, and (3) offline learning, and makes no assumptions w.r.t unbiasedness of belief's or optimality of policies. Observations, beliefs, (optimal) q-values, actions, and policies are denoted z, 6, q., a, and T; bold denotes learned quantities, italics are known (or queryable), and "1" denotes jointly-learned quantities.
Approach
Prototype
Overview
(1)))
Beliefs
Policy
BC
XXX V
(N/A)
No assumption
Imitation
Learning
GAIL
e(ala)
XEX X
(NIA)
No assumption
(iseered by dpsasis)
MB-IL
(e is)
X d
(NIA)
No assumption
(efieredd by endinommnn aadel)
Appremicessiip Leurring
IRL
rewerds dysomii
X X X
(N/A)
Optimal
Ixtafooctaanl lcaring:
PO-IRL
dusansyy
rewerd,, Aysaiic
X V X
Unbiased
Optimal
(ifovaac)
Off. PO-IRI.
10
Urazmic
recandal dysanio
x
Unhiased
Optimal
INTERPOLE
(Ours)
<(A(h)
bestalanes
/ No assumption
No assumption
(alovaa))
approximation immediately lands in black-box territory. Likewise, while (model-based) methods have been developed for robotic control, their transparency crucially hinges on fully-observable kinematics.
Finally, in realistic settings it is often impossible to experiment online- especially in high-stakes envi- ronments with real products and patients. The vast majority of recent work in (inverse) reinforcement learning has focused on games, simulations, and gym environments where access to live interaction unrestricted. By contrast, in healthcare settings the environment dynamics are neither known a priori, nor estimable by repeated exploration. We want data-driven representation of behavior that is learnable in a completely offline fashion, yet does not rely on knowing/modeling any true dynamics.
Contributions Our contributions are three-fold. First, we propose model for interpretable policy learning ("INTERPOLE" --here sequential observations are aggregated through a decision agent's s decision dynamics (viz. subjective update process), and sequential actions are determined by the agent's 's decision boundaries (viz. probabilistic belief-action mapping). Second, we suggest a Bayesian learning algorithm for estimating the model, simultaneously satisfying the key criteria of transparency, partial observability, and offline learning. Third, through experiments on both simulated and real-world data for Alzheimer's discase diagnosis, we illustrate the potential of our method as an investigative device for auditing, quantifying, and understanding human decision-making behavior.
2 RELATED WORK
We seek to learn interpretable parameterization of observed behavior to understand an agent's s actions. Fundamentally, this contrasts with imitation learning (which seeks to best replicate dem- onstrated policies) and apprenticeship learning (which seeks to match some notion of performance).
Imitation Learning In obsly-obe settings, behavior cloving (BC) readily reduces the imitation problem one of supervised classification (5] [3: i.e. actions are simply regressed obser- vations. While this can be extended to account for partial observability by parameterizing policies via recurrent function approximation [14], it immediately gives up on ease of interpretability per the black-box nature of RNN hidden states. A plethora of model-free techniques have recently been devel- oped, which account for information in the rollout dynamics of the environment during policy learning (see e.g. (15-20) most famously, generative adversarial imitation learning (GAIL) based on state- distribution matching 6 21 However, such methods require repeated online rollouts of intermediate policies during training, and also face the same black-box problem as BC in partially observable set- tings. Clearly model-free imitation, is difficult to admit both transparency and partial observability.
Specifically with an eye on explainability, Info-GAIL 12, 23 "interpretability" that hinges on clustering similar demonstrations to explain variations in behavior. However, as with GAIL it suffers from the need for live interaction for learning. Finally, several model- techniques for imitation learning (MB-IL) have been studied in the domain of robotics. 24 consider kinematic models designed for robot dynamics, while (25] and 77] consider (non-)linews autoregressive exogenous models. However, such approaches invariably operate in obbly-lbeer settings, and are restricted models hand-crafted for specific robotic applications under consideration.
proposes an orthogonal notion of
2

---

Published as a conference paper at ICLR 2021
Apprenticeship Learning In subtle distinction to imitation learning, methods in apprenticeship learning assume the observed behavior is optimal with respect to some underlying reward function. Apprenticeship thus proceeds indirectly-ofee to infer a reward function, which (with appropriate optimization) generates learned behavior that
through inverse reinforcement learning (IRL) in order
matches the performance of the original-as as measured by the rewards (see e.g. 8 (9)-29 These approaches have been variously extended to cope with partial observability (PO-IRL)
[9 30 to
offline settings through off-policy evaluation 33
well as learned environment models [0]
However, a shortcoming of such methods is the requirement that the demonstrated policy in fact be optimal with respect to a true reward function that lies within an (often limited) hypothesis class under consideration or is otherwise black-box in nature. Further, learning the true environment dynamics 10 corresponds to the requirement that policies restricted to the class of functions that map from unbiased beliefs (cf. exact inference) into actions. Notably though, 34 considers both a form of suboptimality caused by time-inconsistent agents as well as biased beliefs. However, perhaps most importantly, due to the indirect, task-level nature of reward functions, inverse reinforcement learning is essentially opposed to our central goal of transparency-ttie is, in providing direct, action-level descriptions of behavior. In Section we provide empirical evidence of this notion of interpretability.
Towards INTERPOLE In contrast, we avoid making any assumptions as to either unbiasedness of beliefs optimality of policies. After all, the former requires estimating (extemallyy) "true" envi- ronment dynamics, and the latter requires specifying (objectively) "true" classes of reward functions- neither of which are necessary per our goal of transparently describing individual actions. Instead, INTERPOLE simply seeks the most plausible explanation in terms of (internal) decision dynamics and (subjective) decision boundaries. To the best of our knowledge, our work is the first to tackle all three key criteria-while making no assumptions on the generative process behind behaviors. Table [ con- textualizes our work, showing typical incarnations of related approaches and their graphical models.
Before continuing, we note that the separation between the internal dynamics of an agent and the external dynamics of the environment has been considered in several other works, though often for entirely different problem formulations. Most notably, 355 tackles the same policy learning problem as we do in online, fully-observable environments but for agent's with internal states that cannot be observed. They propose agent Markov models (AMMs) to model such environment-agent interactions. For problems other than policy learning, [36-38] also consider the subproblem of inferring an agent's s internal dynamics; however, none of these works satisfy all three key criteria simultaneously do.
3 INTERPRETABLE POLICY LEARNING
We first introduce INTERPOLE's model of behavior, formalizing notions of decision dynamics and de- cision boundaries. In the next section, we suggest Bayesian algorithm for model-learning from data.
Problem Setup Consider partially-observable decision-making environment in discrete time. At each step t, the agent takes action a, A and observes outcome 2 E z [ We have at disposal an observed dataset of demonstrations D= {f at 24, )= by an agent, т, being the length of the i-th trajectory (we shall omit indices i unless required). Denote by h; = (a1, ø1, 11-1, 21_1) the observed history at the beginning of step t, where h1 = 0. Analogously, let He = (Ax Z) z)- indicate the set of all possible histories at the start of step t, where H₃ = (0). and let H = - H.
A proper policy is mapping π E A(A)" from observed histories action distributions, where π (a/h) is the probability of taking action a given h. We assume that D is generated by an agent acting according some behavioral policy Ta- The problem wish tackle, then, is precisely how to obtain an interpretable parameterization of Th- We proceed two steps: First, we describe a parsimonious belief-update process for accumulating histories- we term decision dynamics. Then, we take beliefs to actions via a probabilistic mapping- which gives rise to decision boundaries.
Decision Dynamics We model belief-updates by way of an input-output hidden Markov model (IOHMM) identified by the tuple (S, A, Z, T, O, bi), with S being the finite set of underlying states. T A(S)SxA denotes the transition function such that T(s (st, at) gives the probability of transitioning into state $t41 upon action a, in state st. and O E A(Z) denotes the observation function such that O( 2. (a,, 81+1) gives the probability of observing 2, after taking action d, and transitioning into state 11+1. Finally, let beliefs b, E A(S) indicate the probability (,(s) that the
While we take here that Z is finite, our method can casily be generalized allow continuous observations.
3

---

Published as a conference paper at ICLR 2021
environment exists in any state E S at time t. and let b1 give the initial state distribution. Note unlke in existing uses of the 10HMM formalism these "probabilities" are for representing the thought process of the human, and may freely diverge from the actual mechanics of the world. To aggregate observed histories as beliefs, we identify (8) with Ps = x s//,) ] interpretation that leads to the recursive belief-update process (where in our problem, quantities T, O, bi are unknown):
bs+ x ((s) (zz s',
(1)
key distinction bears emphasis: We do not require that this latter set of quantities correspond to (external) environment dynamics and we do not obligate ourselves to recover any such notion of "true" parameters. To do so would imply the assumption that the agent in fact performs exactly unbiased inference a perfectly known model of the environment, which is restrictive. It is also unnecessary, since our mandate is simply to model the (internal) mechanics of decision- which could well be generated from possibly biased belief's or imperfectly known models of the world. In other words, our objective (see Equation 3 of simultaneously determining the most likely beliefs (cf. decision dynamies) and policies (cf. decision boundaries) is fundamentally more parsimonious.
Decision Boundaries Given decision dynamics,
A
Af Decision Region (Fq
a policy is then equivalently a map æ E ( (5)) Now, what is interpretable parameterization? Consider the three-state example in Figure [ We argue that probabilistic parameterization that di- rectly induces "decision regions" (cf. panel lbb over the belief simplex uniquely interpretable. For instance, strong beliefs that a patient has under- lying mild cognitive impairment may map to the region where specific follow-up test is promptly prescribed; this parameterization allows clearly locating such regions- well as their boundaries. Precisely, we parameterize policies in terms of |A|- "mean" vectors that correspond to actions:
Belief Updute (Be
b
K br-1
L. K
(a) Decision Dynamics (b) Decision Boundaries
Figure 1: The INTERPOLE Model. Here, S= S=(K, T,, M) and A= (1, 2.3, 43. (a) Beliefs are updated rec- ursively (Equation M.. (b) Actions are chosen with res- pect to relative locations of mean vectors (Equation 2
T[alb) --ll--palll E''ese e " Eves Ha()
where 7) 2 is the inverse temperature, || the fg-norm, and pha E TR(S) the mean vector correspond- ing to action a E A. Intuitively, mean vectors induce decision boundaries (and decision regions) over the belief space ASS): At any time, the action whose corresponding mean is closest to the current belief is most likely to be chosen. In particular, lines that are equidistant to the means of any pair of actions form decision boundaries between them. The inverse temperature controls the transitions between such boundaries: A larger 7) captures more deterministic behavior (i.e. more "abrupt" transi- tions), whereas a smaller 1) captures more stochastic behavior (i.e. "smoother" transitions). Note that the case of 1) = 0 recovers policies that are uniformly random, and 1j - Do recovers argmax policies.
second distinction is due: The exponentiated form of Equation Saaiiinnnssoooooo should not be confused with typical Boltzmann (27 MaxEnt 33) policies common in RL: These are indirect parameterizations via op- timal/soft q- g-values, which themselves require approximate solutions to optimization problems; as we shall see in our experiments, the quality of learned policies suffers as result. Further, using g-values would imply the assumption that the agent in fact behaves optimally w.r.t. an (objectively) "true" class of reward functions e.g. finear-whieh is restrictive. is also unnecessary, as our mandate is simply to capture their (subjective) tendencies toward different cations-whichh are generated from possibly suboptimal policies. In contrast, by directly partitioning the belief simplex into probabilistic "decision regions", INTERPOLE'S 's mean-vector representation can be immediately explained and understood.
Learning Objective In a nutshell, our objective is to identify the most likely parameterizations T, O, by for decision dynamics as well as 17, {Ma JaEA for decision boundaries, given the observed data:
Decision Desamios (Ea
Dosision Roundaries (ba)
Z4
by
a2
T. 1,0,b
nh fMataEd
Given: D.S,A, A,7 Determine: T,O,b, 61.7) n,, {PalaEs
INTERPOLEE
Figure 2: The INTERPOLE Objective. Inputs (demon- stration data) are fed into INTERPOLE through dashed lines, and outputs (estimates) are issued in bold lines. (Belief' b, can then computed via forward pass).
Next, we propose Bayesian algorithm that finds the maximum a posteriori (MAP) estimate of these quantities. Figure illustrates the problem setup.
4

---

Published as a conference paper at ICLR 2021
4 BAYESIAN INTERPRETABLES POLICY LEARNING
Denote with 0 = (T, b,, {pa Jaga) the set of parameters to be determined, and let 0 be drawn from some prior P(O). In addition, denote with D = ((fi....;'8 +;,+))] the of underlying (unobserved) state trajectories, such that D D gives the complete (fully-observed) dataset. Then the complete likelihood of the unknown parameters with respect DUD LUD-in is given by the following:
PPD,D10) - < lb, (T, O, h 1)) x II b (s))
T(s (1) (2)0(zel St+1) (4)
II [=1
likelihoods
observation likeliboods
where (-.--) is described by y/ and Jae and each be[-] is a function of T. O, bb and ht-1
(Equation 1.. Since we do not have access to D, we propose expeccaion-maxxiizaaioo (EM)-like algorithm for maximizing the posterior P(O = (()))))(0)/ PPDD))) over the parameters:
Bayesian Learning Given initial estimate
Algorithm 1 Bayesian INTERPOLE
80, we iteratively improve the estimate by perf-
orming the following steps at each iteration k:
1: Parameters: learning rale w E R
2: Input: dataset D {h +1 P(O)
"E-step": Compute the expected log-likeli-
3: Sample 8 from P(Ø)
hood of the model parameters 0 given the pre- vious parameter estimate p as follows:
4: for k 1, 2...... do
5: Compute PDD (
AAppendix A.
-(::0= a (log) 10
6: Compute Vg VoQQ(B:6k)) at 1 Apppendix A.2
-Eo 1DD0(( (5)
7: 8h
0² k-1
+ (0-10 Vogog Po0
where we compute the necessary marginal-
8: while
izations of joint distribution PDD 8
9: 0
by way of forward-backward procedure
10: Output: MAP estim. 0 (f, 1. A
(detailed procedure given in Appendix [A.I].
"M-step": Compute new estimate 0% that improves the expected log-posterior- is, such that:
Q(0"; (0h1)) + log Q(0-=1; (k-1) 1P@"
(6)
subject to appropriate non-negativity and normalization constraints on parameters, which can be achieved via gradient-based methods (detailed procedure given in Appendix A.2p. We stop when it becomes no longer possible to obtain new estimate further improving the expected log-posterior -that is, when the "M-step" cannot be performed. Algorithm I summarizes this learning procedure.
Explaining by Imitating Recall our original mandate to give the most plausible explanation for behavior. Two questions can asked about our proposal to "explain by imitating" which we now have precise answers: One concerns explainability, and the other concerns directness of explanations.
First, what constitutes the "most plausible explanation" of behavior? Now, INTERPOLE identifies this as the most likely parameterization of that behavior using state- state-based model for beliefs and policies- but otherwise with no further assumptions. particular, we are only positing that modeling beliefs over states helps provide an interpretable description of how an agent reasons (which we do have ample evidence for but we are not assuming that the environment itself takes the form of a state-based model (which is entirely different claim). Mathematically, the complete likelihood (Equation 4 highlights the difference between decision dynamics (which help explain the agent's behavior) and "true" environment dynamics (which we do not care about). The latter independent of the agent, and learning them would have involved just the observation likelihoods alone. In contrast, by jointly estimating T, O, by with 7,, Jac according both the observation- and action- likelooook,lisodd,, we are learning the decision dynamics-whiet in general need not coincide with the environment, but which offer the most plausible explanation of how the agent effectively reasons.
The second question is about directness: Given the popularity of the IRL paradigm, could we have sim- ply used (indirect) reward parameterization, instead of our (direct) mean-vector parameterizaaionn? As it turns out, in addition to the "immediate" interpretability of direct, action-level representations,
In healthcare, discases are often modeled in terms of states, and beliefs over disease states are eminently transparent factors that medical practitioners (i.c. domain experts) readily comprehend and reason about 4
5

---

Published as a conference paper at ICLR 2021
it comes with an extra perk w.r.t. computability: While it is (mathematically) possible to formulate similar learning problem swapping out iز for rewards, in practice it is (computationally) intractable to perform in setting. The precise difficulty lies in differentiating through quantities (Ta, ( in turn depend on beliefs and dynamics-in the cetionnlikelihoods (proofs located in Appendix B
Proposition 1 (Differentiabiilty with (-Parammetrizzaioons Consider softmax, policies parameterized by g-values from a reward function, such that π (alb) = e (b.a) /in' e (b,a") in lieu of Equation 2 Then differentiating through log ग (aelbe) terms with respect to unknown parameters 0 is intractable.
In contrast, INTERPOLE avoids ever needing to solve any "forward" problem at all (and therefore does not require resorting costly-and
workarounds) for learning:
Proposition 2 (Differentiability with ju- (-Prrramterizatioon) Consider the mean-vector policy parame- terization proposed in Equation [ Differentiation through the log a(a terms with respect the unknown parameters 0 is easily and automatically performed using backpropagation through time.
5 ILLUSTRATIVE EXAMPLES
Three aspects of INTERPOLE deserve empirical demonstration, and we shall highlight them in turn:
Interpretability; First, illustrate the usefulness of our method in providing transparent
explanations of behavior. This is our primary objective here- of explaining by imitating.
Accuracy: Second, demonstrate that the faithfulness of learned policies is not given up for transparency. This shows that accuracy and interpretability are not necessarily opposed. Subjectivity: Third, show INTERPOLE correctly recovers underlying explanations for beh if the agent is biased. This sets us apart from other state-based algorithms.
In order to do so, show archetypical examples to exercise framework, using both simulated and real-world experiments in the context of disease diagnosis. State-based reasoning is prevalent in research and practice: three states in progressive clinical dementia 442 43 preterminal cancer screening 44 45,, or even-as recently shown-for cystic fibrosis
46 and pulmonary disease [47]
Decision Environments For our real-world setting, we consider the diagnostic patterns for 1,737 patients during sequences of 6-monthly visits in the Alzheimer's 's Disease Neuroimaging Initiative [48] database (ADNI). The state space consists of normal functioning ("NL"), mild cognitive impairment ("MCI"), and dementia. For the action space, we consider the decision problem of ordering vs. not ordering an MRT test, which-while often informative of Alzheimer's -- financially costly. MRI outcomes are categorized according to hippocampal volume: ("avg", "above avg". "below avg". "not ordered");; separately, the cognitive dementia rating-sum of boxes ("CDR-SB") result- which is always measured-is is categorized as: = {"normal", "questionable impairment", "mild/severe dementia") 42). In total, the observation space therefore consists of the 12 combinations of outcomes.
We also consider simulated setting to better validate performance. For this we employ diagnostic environment (DIAG) in the form of an 1OHMM with certain (true) parameters Turre Otrue birue Patients fall within diseased (s+) and healthy (s_) states, and vital-sign measurements available at every step are classified within positive (2+) ) and negative (z-) outcomes. For the action space, we consider the decision of continuing to monitor a patient (a=), or stopping and declaring a final diagnosis-ani and if so, a diseased (a4) ) or healthy (a_) declaration. If we assume agents have perfect knowledge of the true environment, then this setup is similar to the classic "tiger problem" for optimal stopping [49]. Lastly. we also consider a (more realistic) variant of DIAG where the agent's 's behavior instead generated by biased beliefs due to incorrect knowledge T. O, bj * Thrue Otrme bemue of the environment (BIAS). Importantly, this generates a testable version of real-life settings where decision-makers' (subjective) beliefs often fail to coincide with (objective) probabilities in the world.
Where appropriate, we compare INTERPOLE against the following bench- marks: imitation by behavioral cloning (5] using RNNs for partial observability (R-BC); Bayesian IRL on POMDPs [9] equipped with a learned environment model (PO-IRL); fully coun- terpart [10 of Bayesian IRL (Off. PO-IRL); and an adaptation of model- based imitation learning settings, with learned IOHMM as the model (PO-MB-IL). Algorithms requiring learned models for interaction are given 1OHMMs estimated using conventional methods 50 Further information on environments and benchmark implementations is found in Appendix C
Benchmark Algorithms
7 to partially-observabbe

---

Published as a conference paper at ICLR 2021
An MRI is more likely to be ordered.
MRI is ordered.
Final beliefs
A Belief' simplex Decision boundary
MRI is less likely to ordered. * An MRI is not ordered.
Belief updates
MC1
MC1
MC1
MC1
couNdfauuuu uydnn
NL
Dementia NL
Dementia
NL
Dementia
NL
Dementa
(a) Typical Normal Patient
(b) Typical Deteriorating Patient
(c) Patient is Diagnosed Late
(d) Patient with High-value Tests
Decision Trajectories. Examples of real patients, including: (a) A typical normally-functionn functioning patient, where the decision-maker's ' beliefs remain mostly on the decision boundary. (b) A typical patient who is believed to be deteriorating towards dementia. (c) A patient tatoo--pppretllyccooo than they actually were. (d) A patient with (seemingly redundant) MRI test that is actually highly informative.
Figure 3:
have been diagnosed much earlier
Interpretability First, we direct attention to the potential utility of INTERPOLE as an investigative device for auditing and quantifying individual decisions. Specifically, modeling the evolution of an agent's 's beliefs provides a concrete basis for analyzing the corresponding sequence of actions taken:
Explaining Trajectories. Figure 3 shows examples of such decision trajectories for four real ADNI patients. Each vertex of the belief simplex corresponds to one of the three stable diagnoses, and each point in the simplex corresponds to a unique belief (i.e. probability distribution). The closer the point is to vertex (i.e. state), the higher the probability assigned to that state. For instance, if the belief is located exactly in the middle of the simplex (i.e. equidistant from all vertices), then all states are believed to be equally likely. If the belief is located exactly
vertex (e.g. directly on top of MCI), then this corresponds to an absolutely certainty of MCI being the underlying state. Patients (a) and (b) are "typical" patients who fit well to the overall learned policy. The former is patient believed to remain around the decision boundary in all visits except the first; appropriately, they are ordered an MRI during approximately half of their visits. The latter is believed to be deteriorating from MCI towards dementia, hence prescribed an MRI in all visits.
Identifying Belated Diagnoses. many diseases, early diagnosis is paramount [5]. INTERPOLE allows detecting patients who appear to have been diagnosed significantly later than they should have. Patient (c) was ordered an MRI in neither of their first two visits-despite the fact that the "typical" policy would have strongly recommended one. At third visit, the MRI that was finally ordered led to near-certainty of cognitive impairment-buu this could have been known 12 months earlier! In fact, among all ADNI patients in the database, 6.5% were subject to this apparent pattern of "belatedness", where a late MRI is immediately followed by jump to near-certain deterioration.
Quantifying Value of Information. Patient (d) highlights how INTERPOLE can be used to quantify the value test in terms of its information gain. While the patient was ordered an MRI in all of their visits, it may appear (on the surface) that the third and final MRIs were redundant-since they had little apparent affect on beliefs. However, this is only true for the factual belief update that occurred according to the MRI outcome that was actually observed. Having access to an estimated model of how beliefs are updated in the form of decision dynamics, we can also compute counterfactual belief updates- that belief updates that could have occurred if the MRI outcome in question were to be different. In the particular case of patient (d), the tests were in fact highly informative, since (as it happened) the patient's s CDR-SB scores were suggestive of impairment, and (in the counterfactual) the doctor's beliefs could have potentially leapt drastically towards MCI. On the other hand, among MRIs ordered for ADNI patients, 19% may indeed have been unnecessary (i.c. triggering apparently insignificant belief-updates both factually well as counterfactually).
Evaluating Interpretability through Clinician Surveys
To cement the argument for interpretability, we evaluated INTERPOLE by consulting nine clini- cians from four different countries (United States, United Kingdom, the Netherlands, and China) for feedback. We focused on evaluating two aspects of interpretability regarding our method:
7

---

Published as a conference paper at ICLR 2021
Whether the proposed representation of (possibly subjective) belief trajectories are preferable to raw action-observation trajectories that is, whether decision dynamics are a transparent way of modeling how information is aggregated by decision makers,
Decision Dynamics:
Decision Boundaries: Whether the proposed representation of (possibly suboptimal) decision boudnaries are a more transparent way of describing policies, compared with the representation of reward functions (which the conventional approach in the policy learning literature).
For the first aspect, we presented to the participating clinicians the medical history of an example patient from ADNI represented in three ways using: only the most recent aetion-observation, the complete action-observation trajectory, as well as the belief trajectory recovered by INTERPOLEE Result: All nine clinicians preferred the belief trajectories over action-observation trajectories.
For the second aspect, showed them the policies learned from ADNI by both Off. PO-IRL and INTERPOLE, which parameterize policies in terms of reward functions and decision boundaries respectively. Result: Seven out of nine clinicians preferred the representation in terms of decision boundaries over that offered by reward functions. Further details can be found in Appendix [
Accuracy Now, a reasonable question is whether such explainability comes at cost: By learning interpretable policy, do we sacrifice any accuracy? To be precise,
Table 2: Performance Comparison in ADNI. INTER-
POLE is hest/second- hest for action-matching metries.
Calibration
Area under
Area under
ask the following questions:
Algorithm
(Brier Score)
ROC Curve
PR Curve
the relief-update process the same? For this, the appropriate metric is the discrepancy with respect to the sequence of beliefs-whichh we take to be E, PKL (bell (Belief Mismatch).
R-BC
0.18
0.61 +.05
0.81 + 0.08
PO-MB-IL.
0.07
0.07
0.79 0..1
PO-IRL
0.00
0.007
0.78 + 0.09
Off. PO-IRL
0.24
1.005
0.79 + 0.09
INTERPOLE 0.17 0.05
0.60 + 0.04
0.81 + 0.09
Is the belief-action mapping the same? Our metric is the discrepancy with respect to the policy distribution wthchh we take to be D,DLL (=6((16)]l((((Bb))) (Policy Mismatch).
Table 3: Performance Comparison in DIAG, INTER- POLE is best. Belief mismatch is n/a to R-BC. 1x100
Belief Mismatch
Policy Mismatch
Stopping Time Error
Is the effective behavior the same? Here, the metries are those measuring the discrepancy with respect to ground-truth actions observed (Action-Matching) for ADNI, and differences in stopping (Stopping Time Error) for DIAG.
Algorithm
R-BC
6.7 1.4
5.91 + 1.29
PO-MB-II.
42.7+334.0
47.9 18.9
6.34 + 1.48
PO-IRI.
42.7+334.
62.1 + 29.1
6.41 + 1.80
Off. PO-IRI.
26.1 5.6
2.1 + 0.2
5.42 + 0.69
Note that the action-matching and stopping time errors evaluate the quality of learned models in imitating per se, whereas belief mismatch and policy mismatch evaluate their quality in explaining
INTERPOLE
0.6 0.1
0.8 0.2 5.38 + 1.14
The results are revealing, if not necessarily surprising. To begin, observe for the ADNI setting in Table 2 that INTERPOLE performs first- or second-bes, across all three action-matchinn based metrics; where it comes second, it does so only by small margin to R-BC (bearing in mind that R-BC is specifically optimized for nothing but action- miatonng. Similarly for the DIAG setting, we observe in Table that INTERPOLE performs the best in terms of stopping-ting error. In other words, it appears that littleiif any iny-imitation accuracy is lost by using INTERPOLE as the model.
Perhaps more interestingly, we also in Table that the quality of internal explanations is superior- in terms of both belief mismatch and policy mismatch. In particular, even though the comparators PO-MB-IL, PO-IRL, and Off. PO-IRL are able to map decisions through belief',, they inherit the conventional approach of attempting to estimate true environment dynamics, which is unnecessary- and possibly detrimental if the goal is simply to find the most likely explanation of behavior. Notably, while the difference in imitation quality among the various benchmarks is not tremendous, with respect to explanation quality the gap is significant-
INTERPOLE has great advantage.
Subjectivity Most significantly, we now show that INTERPOL.E correctly recovers the underlying explanations, even or perhaps especially the agent is driven by subjective reasoning (i.e. with biased beliefs). This aspect sets INTERPOLE firmly apart from the alternative state-based techniques.
Belieffpolicy mismatch not applicable to ADNT since we have access ground-truth helicfs/palicies.
8

---

Published as a conference paper at ICLR 2021
Consider the BIAS environment: Here the true environment dynamics are unchanged from DIAG, but no longer coincide with the agent's 's decision dynamics. Specifically, we let the behavioral policy be generated using erroneous parameters ((z_la_,54) < (z_ 8+)); that is, the doctor now incorrectly believes the test have a smaller false- false-negative rate than it does in reality thus biasing their beliefs regarding patient states.
Ground- Truth &. PO-MB-II. J. INTERPOLE
Belief Trajoctories (ax Fig for consecutive observattons
0.5
0.6
0.2
Now suppose we wish to recover the decision boundary from the demonstrations- what confidence threshold does a doctor commit to healthy diagnosis? Figure I shows that both INTERPOLE and PO-MB-IL appear to recover the correct effective behavior: Starting from a neutral prior, doctors tend to stop and issue a "healthy" diagnosis if the first two observations return neg- ative signals. Importantly, INTERPOLE also cor- rectly locates the confidence target ~ 90%. On the first attempts
0.0
0.1
0.2
0.3 0.4
0.5
0.7
is, at
b(s-)
Decision Bounddates
(ox in Fig Lh
Figure 4: Explaining Subjective Behavior. Markers show the evolution of belief's that explain ground-truth and learned policies BIAS, for an example sce- nario where two consecutive negative (z-) observa- tions are made. While all policies display similar ef- fective behavior, only INTERPOLE correctly identifies the ground-truth decision boundary. This underscores the significance of distinguishing between decision dynamics (which help explain an agent's 's behavior) and "true" dynamics (which we do not care about).
other hand, FO-MB-IL-whiee estimate the enviromment's 's true parameters-ente up learning policy on the basis of miscalibrated beliefs, thereby incorrectly explaining the same effective behavior with a lower confidence target 70%. Finally, through similarly benchmarked metrics, Table g further confirms INTERPOLE's advantage in providing the (correct) explanations.
Table 4: POLE is best. Belief mismatch is n/a to R-BC. 1x 10
Performance Comparison in BIAS, INTER-
Belief Mismatch!
Policy Mismatch'
Stopping Time Error
Algorithm
R-BC PO-MB-IL -I--LL
191.8 1 78.8
2,22 -: 0.45
83.2
125.4
DISCUSSION
PO-IRI.
83.2+3.00
144.0 + 29.4
3.46+0.222
Off. PO-IRL
84.9
62.8 7.5
3.12
Three points deserve brief comment in closing. First, while we gave prominence to several exam- ples of how INTERPOLE may be used to audit and improve decision-making behavior, the potential applications are not limited to these use cases, Broadly, the chief proposition is that having quantita- tive descriptions of belief trajectories provides a concrete language that enables investigating observed actions, including outliers and variations in behavior: On that basis, different statistics and post-hoc analyses can be performed on top of INTERPOLE'S explanations (see Appendix C for examples).
INTERPOLE
0.34
188 0.6
2.14
Second, a reasonable question is whether or not it is reasonable to assume access to the state space. For this, allow us to reiterate a subtle distinction. There may be some "ground-trute truth"" external state space that is arbitrarily complex or even impossible to discover, but-as explained- we are not interested modeling this. Then, there is the internal state space that an agent uses to reason about decisions, which is what are interested in. In this sense, it is certainly reasonable to assume access to the state space. which is often very clear from medical literature [42 Since our goal is to obtain inter- pretable representations of decision, it is therefore reasonable to cater precisely to these accepted state spaces that doctors can most readily reason with. Describing behavior in terms of beliefs over these (already well-understood) states is one of the main contributors the interpretability of our method.
Finally, it is crucial to keep in mind that INTERPOLE does not claim to identify the real intentions of an agent: humans are complex, and rationality is-of course-bounded, What it does do, is to provide an interpretable explanation of how an agent is effectively behaving, which-aa aa we have seen for diagnosis of ADNI patients- offers a yardstick by which assess and compare trajectories and subgroups. In particular, INTERPOLE achieves this while adhering to our key criteria for healthcare settings, and without imposing assumptions of unbiasedness or optimality on behavioral policies.
ACKNOWLEDGMENTS
This work supported by the US Office of Naval Research (ONR) and Alzheimer's 's Research UK (ARUK). We thank the clinicians who participated our survey, the reviewers for their valuable feedback, and the Alzheimer's 's Disease Neuroimaging Initiative for providing the ADNI dataset.

---

Published as a conference paper at ICLR 2021
REFERENCES
[1] J. W. O'Sullivan, C. Heneghan, R. Perera, J. Oke, J. K. Aronson, B. Shine, and B. Goldacre, "Variation in diagnostic test requests and outcomes: a preliminary metric for OpenPathology.net" Sci. Reports, vol. 8, no. 1. pp. 1 1-6, 2018.
[2] M. Allen, "Unnecessary tests and treatment explain why health care costs so much," Sci. Amer.
2017.
[3] A. Holzinger, C. Biemann, C. Pattichis, and D. B. Constantinos, "What do we need to build explainable AI systems for the medical domain?" arXiv preprint arXtv:1712.009933, 2017.
[4] G. Montavon, W. Samek, and K.-R. Müller, "Methods for interpreting and understanding deep neural networks," Digit. Signal Process., vol. 73, pp. 1-15, 2018.
[5] B. Piot, M. Geist, and O. Pietquin, "Boosted and reward-regularizeed classification for ap- prenticeship learning," in Proc. 13th Int. Conf. Auton. Agents and Multiagent Syst., 2014, pp. 1249-1256,
[6] W. Jeon, S. Seo, and K.-R. Kim, "A Bayesian approach to generative adversarial imitation learning," " in Adv. Neural Inf. Process. Syst. 32, 2018.
[7] P. Englert, A. Paraschos, J. Peters, and M. P. Deisenroth, "Probabilistic model- based imitation learning," Adaptive Behav., vol. 21, pp.
2013.
[8] J. Choi and K.-E. Kim, "MAP inference for Bayesian inverse reinforcement learning," in Adv. Neural Inf. Process. Syst. 24, 2011, pp. 1989-1997.
[9] D. Jarrett and M. van der Schaar, "Inverse active sensing: modeling and understanding timely decision-making. in Proc. 37th Int. Conf. Mach. Learn., 2020.
[10] Makino and J. Takeuchi, "Apprenticeship learning for model parameters of partially observ- able environments," in Proc. 29th Int. Conf. Mach. Learn., 2012, pp. 891-898.
[11] M. Bain and C. Sammut, "A framework for behavioural cloning," Mach. Intell., vol. 15, pp. 103-129, 1996.
[12] S. Ross and D. Bagnell, "Efficient reductions for imitation learning," " in Proc. 13th Int. Conf. Artif. Intell. Statist., 2010, pp. 661-668.
[13] D. Jarrett, I. Bica, and M. van der Schaar, "Strictly batch imitation learning by energy-based distribution matching," arXiv preprint arXiv:2006. 141:: 2020.
[14] W. Sun, A. Venkatraman, G. J. Gordon, B. Boots, and J. A. Bagnell, "Deeply Aggre differentiable imitation learning for sequential prediction," " in Proc. 34th Int. Conf. Mach. Learn., 2017.
[15] L. Blonde and A. Kalousis, "Sample-efficient imitation learning via generative adversarial nets." in Proc. 22nd Int. Conf. Artif. Intell. Statist., 2019.
[16] I. Kostrikov, K. K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson, inssrrminnttor-actorooooo addressing sample inefficiency and reward bias in adversarial imitation learning." in Proc. 7th Int. Conf. Learn. Representations, 2019.
[17] A. H. Qureshi, B. Boots, and M. C. Yip, "Adversarial imitation via variational inverse reinforce- ment learning," in Proc. 7th Int. Conf. Learn. Representations, 2019.
[18] Sasaki, T. Yohira, and A. Kawaguchi, "Sample efficient imitation learning for continuous control," in Proc. 7th Int. Conf. Learn. Representations, 2019.
[19] K. Brantley, W. Sun, and M. Henaff, "Diagreemeet--eggllaiied imitation learning." in Proc. 8th Int. Conf. Learn. Representations, 2020.
[20] S. Reddy, A. D. Dragan, and S. Levine, "SQIL: imitation learning via regularized behavioral cloning," in Proc. 8th Int. Conf. Learn. Represantations, 2020.
10

---

Published as a conference paper at ICLR 2021
[21] J. Ho and S. Ermon, "Generative adversarial imitation learning," in Adv. Neural Inf. Process. Syst. 29, 2016, pp. 4565-4573.
[22] Y. Li, J. Song, and S. Ermon, "InfoGAIL: interpretable imitation learning from visual demon- strations," Adv. Neural Inf. Process. Syst. 30, 2017, pp. 3812-3822.
[23] M. Sharma, A. Sharma, N. Rhinehart, and K. M. Kitani, "Directed-Info GAIL: learning hierarchical policies from unsegmented demonstrations using directed information," " in Proc. 7th Ini. Conf. Learn. Representations, 2019.
[24] A. Ude, G. Atkeson, and M. Riley, "Programming full-body movements for humanoid robots by observation," Robot. Auton. Syst., vol. 47, no. 2-3, pp. 93-108 2004.
[25] van den Berg, S. Miller, D. Duckworth, H. Hu, A. Wan, X. Fu, K. Goldberg, and Abbeel, "Suueehuuann performance of surgical tasks by robots using iterative learning from buman-guided demonstrations," " in Proc. IEEE Int. Conf. Robot. Automat., 2010, pp. 2074-2081.
[26] P. Abbeel and A. Y. Ng, "Apprenticeship learning via inverse reinforcement learning," in Proc. 2 st Int. Conf. Mach. Learn., 2004.
[27] D. Ramachandran and E. Amir, "Bayesian inverse reinforcement learning"" in Proc. 20th Int. Joint Conf. Artif. Intell., 2007, pp. 2586-2591.
[28] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey, "Maximum entropy inverse reinforcement learning," in Proc. 23rd AAAI Conf. Artif. Intell., 2008, pp. 1433-1438.
[29] J. Fu, K. Luo, and S. Levine, "Learning robust rewards with adversarial inverse reinforcement learning," in Proc. 6th Int. Conf. Learn. Representations, 2018.
[30] Choi and K.-E. Kim, "Inverse reinforcement learning in partially observable environments," Mach. Learn. Res., vol. 12, pp. 691-730,2011
[31] E. Klein, M. Geist, B. Piot, and O. Pietquin, "Inverse reinforcement learning through structured classification," in Adv. in Neural Inf. Process. Syst. 25, 2012, pp. 1007-1015.
[32] A. C. Y. Tossou and C. Dimitrakakis, "Probabilistic inverse reinforcement learning in unknown environments," in Proc. 29th Conf. Uncertainty Artif. Intell., 2013, p. 635-643.
[33] V. Jain, P. Doshi, and B. Banerjee, "Model-free IRL using maximum likelihood estimation," in Proc. 33rd AAAI Conf. Artif. Intell., 2019, pp. 3951-3958.
[34] O. Evans, A. Stuhlmüller, and N. D. Goodman, "Learning the preferences of ignorant, inconsis- tent agents," in Proc. 13th AAAI Conf. Artif. Intell., 2016, pp. 323-329.
[35] V. V. Unhelkar and J. A. Shah, "Learning models of sequantial decision-making with partial specification of agent behavior," in Proc. 33rd AAAI Conf. Artif. Intell., 2019, pp. 2522-2530.
[36] F. Schmitt, H. J. Bieg, M. Herman, and C. A. Rothkopf, "I see what you see: inferring sensor and policy models of human real-world motor behavior," in Proc. 31st AAAI Conf. Artif. Intell., 2017, pp. 3797-3803.
observable continuous nonlinear dynamics," in Adv. Neural Inf. Process. Syst. 34, 2020. [37] M. Kwon, S. Daptardar, Schrater, and X. Pitkow, "Inverse rational control with partially
[38] S. Reddy, S. Levine, and A. D. Dragan, "Assisted perception: optimizing observations to communicate state," arXiv preprint arXiv:2008. 02840, 2020.
[39] C. Finn, P. Christiano, Abbeel, and S. Levine, "A connection between generative adver- sarial networks, inverse reinforcement learning, and energy-based models." NIPS Workshop Adversarial Training. 2016.
[40] F. A. Sonnenberg and J. R. Beck, "Markov models in medical decision making: a practical guide," " Health Econ., vol. 13, pp. 322-338, 1983.

---

Published as a conference paper at ICLR 2021
[41] C. H. Jackson, L. D. Sharples, S. G. Thompson. S. W. Duffy, and E. Couto, "Multistate Markov models for disease progression with classification error," Statistician, vol. 52, pp. 192-209, 2003.
[42] S. E. D'Bryant, S. C. Waring, C. M. Cullum, Hall, L. Lacritz, P. Massman, J. Lupo, Reisch, and R. Doody, "Staging dementia using Clinical Dementia Rating Scale Sum of Boxes scores: a Texas Alzheimer's 's research consortium study," Arch. of Neurology, vol. 65, no. 8, pp. 1091-1095, 2008.
[43] D. Jarrett, J. Yoon, and M. van der Schaar, "Dynamic prediction in clinical survival analysis using temporal convolutional networks," IEEE Biomed. Health Inform., vol. 24, no. 2, pp. 424 436, 2019.
[44] P. Petousis, A. Winter, W. Speier, D. R. Aberle, W. Hsu, and A. A. T. Bui, "Using sequential decision making to improve lung cancer screening performance," " IEEE Access, vol. 7, pp. 403-1199 419, 2019.
[45] Cardoso, S. Kyriakides, S. Ohno, Penault-Llorca, Poortmans, I. T. Rubio, S. Zackrisson, and E. Senkus, "Early breast cancer: ESMO Clinical Practice Guidelines for diagnosis, treatment and fol " Anna. Oncology, vol. 30, no. 8, pp. 1194-1220, 2019.
[46] A. M. Alaa and M. van der Schaar, "Attentive state-space modeling of disease progression," in Adv. Neural Inf. Process. Syst. 33, 2019, pp. 11 338-11 348.
[47] X. Wang, D. Sontag, and F. Wang, "Unsupervised learning of disease progression models," in Proc. 20th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2014, pp. 85-94.
[48] R. V. Marinescu, N. P. Oxtoby, A. L. Young. E. E. Bron, A. W. Toga, M. W. Winer et al., "Tadpole challenge: prediction of longitudinal evolution in Alzheimer's disease," arXiv preprint arXiv: Ia...18055.099.. 2018.
[49] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra, "Planning and acting in partially observable stochastic domains," Artif. Intell., vol. 101, no. 1 1-2, pp. 99-134, 1998.
[50] Y. Bengio and P. Frasconi, "An input output HMM architecture," in Adv. Neural Inf. Process. Syst. 7, 1995, pp. 427-434.
[51] C. Salvatore, P. Battista, and I. Castiglioni, "Frontiers for the early diagnosis of AD by means of MRI brain imaging and support vector machines," Current Algheimer Res., vol. 13, no. 5, pp. 509-533, 2016.
12

---

Published as a conference paper at ICLR 2021
A DETAILS OF THE ALGORITHM
A.1 FORWARD-BACKWAN PROCEDURE
We compute the necessary marginalizations of the joint distribution P(D)D, 0) using the forward- backward algorithm. Letting æur {Ie, ((+1:-.,,,ee) for any time-indexed quantity 2y, the forward messages are defined as ay(s) = PP(st = 8, 31a11-1: x1.-1 0), which can be computed dynamically
0++1 (5') = P( = s', a1.. 3 0)
=s,
Σ
P(s: = 8, au Zi 11))= = de, 21 8, a -1:t-1 0)
#ES
Σ (a (z)
sES
x Σ
a.(s) a, (1)
SES
with initial case ai (s) ((s s) (s). The backward messages are defined as Be(s)
P(aer Zto 18 = 8, 01:t-1 1, 0). which can also be computed dynamically as
Be(s) P(a Zur St = s, a (1.11,
Σ P(( S a, 21 01:-1 z1.t-1, 0))(cct Re41or|$641 = 5 Z1., 0)
s'E8
Σ ((a, (1 (8)2)(66))
s'ES
Σ
T(''ls a ( a,, 3(+1
with initial case (s) PPW0 8++1 8, a1 2i 0)
Then, the marginal probability of being in state s at time given the dataset D and the estimate can be computed as
(s() = [st = s/D, 0)
(P s|a1.r Zt.r 0) x (s a Z1:7 (0)
(s)
and similarly, the marginal probability of transitioning from state s to state s' the end of time given the dataset D and the estimate 0 can be computed as
(s ) IP(st 8,8++1 =)
x P((st 8,8441 = = Z1:T
P(s: s, 3:0 $141 s' a, 2e/5c 1..0--1..1::111
x PP(4411::4411// = a Z1,c,
=o.())taalb.))7(s1ss0000 8") (') <o
A.2 GRADIENT-ASCER PROCEDURE
Taking the gradient of the expected log likelihoodd Q(0: 0) in (5) with respect to the unknown parameters 0 (T,0, bi, i MaEA) first requires computing the Jacobian matrix Va, be for <
13

---

Published as a conference paper at ICLR 2021
t < T, where (Vbb))s (b)bb) for i,j E S. This can be achieved dynamically as Vo, by = Viti be b(b with initial case Vo, by = I, where
(Vo,ut+11s (1)
ab,(i)
0bi(j) [
LEvE b (tt (z)(ar))))))
Exes Ex'ss by (x) ((1),,))((2)))))
a (z)
Eres ( az) ( (,,
ExeS Exves li (2:) ( (z)
A.2.1 PARTIAL DERIVATIVES
The derivative of Q0;: 8) with respect to T(s'|s, a)) is
aQ(0;0) 0)
a
1a: a} Σ>> Ec log a) xES
a)
(T
log ग (acebr))
s's
Olog (aglb))
Ha a)
T()
(T
Ha: a) 5'(
T(8'(8,a)
Vo. log ((aelb))Vo. Ve V1
1==
where
(og((r(b))
(Vo, log (a.(b))));
Ob)
(-nllb- MMabl 2 -1og
log Σ c
nllb 1
abb(()
a'EA
= -2m(be(j) - Ma, (5)) + 2m Σ
Da'EA -- (be(s) mo(1))
-2n((b((i) Ma. (i)) + 2m Σ a(alb))(b:()) ())
aEA
and
1b)+
(T()')a,a)
0
by (1
DT(s'(s,a)
Eves Eves by (O(z lav, x²)
Tii= = (s)
= Ifar = =a}
by (2r
(EES Eves by (x) |r (2) x')1)
The derivative of Q(;; 0) with respect to O(2/a, 8') is
aQ(0:0) 9)
Ma = a, 21 = 2) Σ
(x11 O(2la,a')
80())a,8))
80(2
ar es
14

---

Published as a conference paper at ICLR 2021
+
log x (ailbi) (ba)
Mur = a, = 15')
r
d log x (ae
+
(()la, 8')
a0(zla,8))
Ha, = a, = 2} 10+((5))
O(2 s')
bb. log ( Ve V Votsla,s3bb41
where
- (i)
(Vol a0(2|a,5') 0
Eves by by av)
80())a,5))
EzEs Exes by Iz, ar ((2)))00
Iii 8' (r)) Σxes (=)
Ifrr = 2y =2}
The derivative of ((0; 0) with respect to b1()) is
aQ(0; 0)
a
Σ
(z) + Σ log log aaa (ba)
0b (8)
ab (8)
LEES
(s))
Σ
v, log (o
(1(s)
where (s) = (Vo,be
The derivative of Q(0: 0) with respect to is 0Q(0;0)
log (arlbe)
On
On
i=1
-nllb,
Maa 112 log Σ e - -
a' a'EA e nillb 22.1
Mor 112 -116e
a'EA E ullbb Har 1b, Mall?
acA
-1bbe Mor 112 +
xaabbo)lbe - Hall?
A
Finally, the derivative of Q(0:0) 6) with respect to pa (s) is
aQ(0;0)
θ T
log (a+lbe)
(s)
(s)(s)
-n/llbr Maal12 llog Σe E -olbe--aar
Dpa
a'EA
E e
2nI(a: = a}(b((s) - -a(ss)) - -2n
Eaee pp (b((s) Ha(s))
15

---

Published as a conference paper at ICLR 2021
(2n (b(((())) (s(s)) (1 Ma Ma(s))
2n(I(a; a} - (bal):))(b((s) -Ma(s))
B PROOFS OF PROPOSITIONS
B.1 PROOF OF PROPOSITION
First, denote with 92 E ( the optimal (belief-state)) g-value function with respect to the underlying (state-space) reward function R E SxA, and denote with vt E RR((5) the corresponding optimal value function (b(() (b) softmax.'e Qf (b, a'). Now, fix some component of parameters we wish to compute the derivative of log (a|b) with respect to 0y:
00,
log (alb)
=
30,
( (b, a) -2((1)
0
gb(b, a)
log Σ
(lk(aaa))
00;
o'EA
0
(i(b,a) E
-
elilhhaa)
00, (labb.))))
a
00,
Extes elilbaa
d'EA
8
2n (), - Σ (a'lb)
00, (
00,
d'EA
Q 00,
0
In (b, (1')
00, (b,
where make explicit here the dependence
but note that it itself a parameter; that R 0, for some j. We see that this in turn requires computing the partial derivative 04 (b, a)/00, Let be some appropriate discount rate, and denote with pn E RA(S)) the effective (belief-state) reward pn(b, a) E.es b(s) R(s, a) corresponding Further,
a)
Σ zEZ
F(zlb, P(1
ΣΣ (6)))()ss (v =EZ FE
Σ
E.es Eves (5)T ()(2aa,)
denote the (belief-state) transition probabilities induced by T and O, where the Dirac delta function such that 5(b') integrates to one if and only if = 0 is included in the integration region.
Then the partial (b, a)/00, is given as follows:
pR(b) + L a)vik,
00; (b, a)
00,
HEA(S)
pn(lb a) + JVEA(S) (b)
P(bllb, a)db
ae,
0e
(R.))h.))
2 50,
li ad) db
P(b'lb, a) Eu 69)
(EA(S)
from which observe that (b, (b)) Specifically, fix any function E ( then (b a)/20; is the fixed point of the operator
a)/00; is fixed point of certain Bellman-like operator.
16

---

Published as a conference paper at ICLR 2021
Th :RA(S)A (A((S)xAA defined as follows:
(TR,))(0,a)= - -E.)))).))=p(((...) +7 JVEAS))
P( a) Σ
क TTa'lV)) a' y'
which takes the form of a "generalized" Bellman operator on q-functions for POMDPs, where for brevity here we have written (R.((b,a) to denote the expression 00, PR( a)
2
Y frEA(S) (b)) 8e, P(b a)db', Mathematically, this means that a recursive procedure can in theory be defined cf. "Vq-iteration", analogous to q-iteration; see e.g. [52] that may converge on the gradient under appropriate conditions. Computationally. however, this also means that taking a single gradient is at least as hard as solving POMDPs in general.
Further, note that while typical POMDP solvers operate by taking advantage of the convexity property of pn(b, a)- see e.g.
a.-see
[53] here there is no such property to make use of: In general, it is not the case
that PR. (b, a) is convex. To see this, consider the following counterexample: Let S = {s_. 8+), A =
{a=}. J. Z = {=_, 2+ J. (ss a=) = T(s+|s+, a=) p 1, 1z_ 8-) O(z la_ la_,
1/4, bi (54) = 1/2, R(s__ad a=) = O, a=) = 1/2, and y = 1/2. For simplicity, we will simply
write instead of b(s+). Note that:
vi(b,a = b Σ r2
(1 (1 Σ 7' +((==,a))=
==0
1-
(b) = log Σ eli((ra)) = =log =loge (b(()aa= = (2, a=)==
{a
1
3
1
3
P(z+ 16, a=)==)==--))))))=0))))
4 (bp + (1 - b)(1 - -p)) [b(1 p) + (1 - b)p)
b
(1 b)
4
3
1
3
P(z_ 16, a=)
4 (b(1 p) + - (1)b)p)
(bp + (1 - b)(1 p))
(1
- 6) +
4
4
IP(s' 8 2+ 10, a,)
1b [11 b) - p)
b'|b, a=,
P(2+ b, a=)
P(2+ (b, a=)
1 + (1 -6- 6)
(Ps 84, 2- |b,
(11 (1 - p) + 50pp
P(2_ 6,, a=)
P(z bb, a=)
[1 b) + b
P(z+ 1b, 0=)
bf = b' b'|b, a= 2+
P(('(b,a=)-
P( 1b, )=)
if 6' = 6'16, a=
otherwise
Now, let the elements of be ordered such that is the i-th element, and consider PR,ilb, a)- evaluated at p= 1:
(b,aaa,) = 7p
Pn(bb a-) +
(b)) P(b'l (b, a_)db
NEA(S)
Op
θ
8 P(z b, P(z=(0,a=)) (,)=))
(b b, 0, 2+)
P(1 b, a=) + (/ a=,
Op
Op
t
1
3
= 2
bb + (1 b)
4
b
4
(1 - b)
4
-
b+ - (1 -1-bb) b)
(-1 +
1 (1 - b) 3 3
(1-b) )
4 4 -1- b1
[1 6) +
b
4
4
Clearly (R, am) (3/4, a=) > 0.
cannot be convex since pR. (1/2,
and pRi(lI,=)) 0 but
B.2 PROOF OF PROPOSITION 2
In contrast, unlike the indirect value parameterization above (which by itself requires approximate solutions to optimization problems), the mean-vector parameterization of INTERPOLIE maps beliefs
17

---

Published as a conference paper at ICLR 2021
directly to distributions over actions. Now, the derivatives of log (a(b) are given closed-fome expressions in Appendices A.I and A.2.
In particular, note that each by is computed through feed-forward structure, and therefore can easily be differentiated with respect to the unknown parameters 9 through backpropagation through time: Each time step leading up to action corresponds to a "hidden layer" in neural network, and the initial belief corresponds to the "features" that are fed into the network; the transition and observation functions correspond to the weights between layers, the beliefs each time step correspond the activations between layers, the actions themselves correspond to class labels, and the action likelihood corresponds to the loss function (see Appendices A. and A.2).
Finally, note that computing all of the forward- backward messages at and 3 in Appendix A. has complexity (nr computing all of the Jacobian matricies Va, by in Appendix A.2 has complexity O(rr² 5"), and computing all of the partial derivatives given in Appendix A.2 has complexity at most O(nr² AZ). Hence, fully differentiating the expected log- ligellklooodd Q 0) with respect to the unknown parameters has an overall (polynomial) complexity O(n7² 5 max maxx(,AA)) AZ}).
C EXPERIMENT PARTICULARS
C.1 DETAILS OF DECISION ENVIRONMENTS
ADNI We have filtered out visits without a CDR-SB measurement, which is almost always taken, and visits that do not occur immediately after the six-month period following the previous visit but instead occur after 12 months or later. This filtering leaves 1,626 patients with typically three consecutive visits each. For MRI outcomes, average is considered to be within half standard deviation of the population mean. Since there only two actions in this scenario, have set 17 1 and relied on the distance between the two means to adjust for the stochasticity of the estimated policy-closse means being somewhat equivalent to a smaller 17.
DIAG We set Tiree (true(s = (s = 1, meaning patients do not heal or contract
the diseases the diagnosis progresses, Otric a_. Otree (2 la_. s_) 0.4, meaning
measurements as a test have a false-negative and false-positive rates of 40%, and birue (s+) ) = 0.5.
Moreover, the behavior policy is given by T 7 O Otrue bi Dirue, v 10, 1 (8..)
0.5, and Ma_ (s_) Ma, (s+) = 1.3. Intuitively. doctors continue monitoring the patient until they are 90% confident in declaring final diagnosis. In this scenario. T and r7 are assumed be known.
The behavior dataset is generated as 100 demonstration trajectories.
BIAS We set all parameters exactly the same way we did in DIAG with one important exception: now O(s-(a= 2+) 0.2 while it is still the case that tt (z_|a=, $+) = 0.4, meaning O J Ofrue anymore. this scenario, is also assumed to be known (in addition to T and 17) to avoid any invariances between bj and O that we have encountered during training. The behavioral dataset is generated as 1000 demonstration trajectories.
C.2 DETAILS OF BENCHMARK ALGORITHMS
R-BC We train an RNN whose inputs are the observed histories by and whose outputs are the predicted probabilities म (alh;) of taking action given the observed history hg The network consists of an LSTM unit of size 64 and fully-connected hidden layer of size 64. We minimize the cross- entropy loss L - E Eaes Haa = a} log म (a]h; using Adam optimizer with learning rate 0.001 until convergence, that is when the cross crossy loss does not improve for 100 consecutive iterations.
PO-IRL The 10HMM parameters T, O, and bj are initialized by sampling them uniformly at random. Then, they are estimated and fixed using conventional 10HMM methods. The reward parameter R is initialized as (e a) Sa.a where Ss,a ~ (O, 0.0012). Then, it is estimated via Markov chain Monte Carlo (MCMC) sampling, during which new candidate samples are generated by adding Gaussian noise with standard deviation 0.001 to the last sample. A final estimate is formed by averaging every tenth sample among the second set of 500 samples, ignoring the first 500 samples. In order to compute optimal q valuues, we have used an thf-the--helf POMDP solver available at https: / //www pomop. org/ code/ /ndex.dee...///// html.
18

---

Published as a conference paper at ICLR 2021
Off. PO-IRL All parameters are initialized exactly the same way as in PO-IRL. Then, both the 10HMM parameters T. O, and b1, and the reward parameter R are estimated jointly via MCMC sampling. When generating new candidate samples, with equal probabilities, new T. O, and by from IOHMM posterior (without changing R) or obtained a new R the same way did in PO-IRL. (without changing T, O, and b1). A final estimate is formed the same way in PO-IRL.
have either sampled
PO-MB-IL The IOHMM parameters T. O, and are initialized by sampling them uniformly at random. Then, they are estimated and fixed using conventional IOHMM methods. Given the 10HMM parameters, we parameterized policies the same way we did in INTERPOLE, that is as described in (2). The policy parameters {Ma FaEA are initialized as (2((s) (1/IS) + sa,s)/ E where ca.s' ~ (O, 0.001 Then, they are estimated according solely to the action likelihoods in (4) using the EM algorithm. The expected pog-posterior is maximized using Adam optimizer with learning rate 0.001 until convergence, that is when the expected log-posterior does not improve for 100 consecutive iterations.
+ Ea,s))
INTERPOLE All parameters are initialized exactly the same way in PO-MB-IL Then, the IOHMM parameters T, O, and 61, and the policy parameters {Ma JoEA are estimated jointly according to both the action likelihoods and the observation likelihoods in (4). The expected log-posterior is again maximized using Adam optimizer with learning rate 0.001 until convergence.
C.3 FURTHER EXAMPLE: POST-IICC ANALYSES
Policy representations learned by INTEPOLE provide users with means to derive concrete criteria that describe observed behavior in objective terms. These criteria, in turn, enable the quantitative analyses the behavior using conventional statistical methods. For ADNI, we have considered two such criteria: belatedness of individual diagnoses and informativeness of individual tests. Both of these criteria are relevant to the discussion of early diagnosis, which is paramount for Alzheimer's disease [51 as we have already mentioned during the illustrative examples.
Formally. we consider the final diagnoses of a patient to be belated if (i) the patient was not ordered an MRI in of their visits despite the fact that MRI being ordered the most likely outcome according to the policy estimated by INTERPOLE and (ii) the patient was ordered an MRT in a later visit that led to a near-certain diagnosis with at least 90% confidence according to the underlying beliefs estimated by INTERPOLE. We consider An MRI to be uninformative if it neither (factually) caused nor could have (counterfactually) caused a significant change in the underlying belief-state of the patient, where an insignificant change is half standard deviation less than the mean factual change beliefs estimated by INTERPOLE.
Having defined belatedness and informativeness, one can investigate the frequency of belated diag- noses and uninformative MRIs in different cohorts patients to see how practice varies between one cohort to another. In Table 5 we do so for six cohorts: all of the patients, patients who are over 75 years old, patients with apoE4 risk factor for dementia, patients with signs of MCI or dementia since their very first visit, female patients, and male patients. Note that increasing age, apoE4 allele, and female gender are known to be associated with increased risk of Alzheimer's 's disease [54 57). For instance, see that uninformative MRIs much more prevalent among patients with signs of MCI or dementia since their first visit. This could potentially be because these patients are monitored much more closely than usual given their condition.
Table 5: Frequency of belated diagnoses and uninformative MRIs in various patient cohorts.
Frequency of belated diagnoses
Frequency, of uninformative MRIs
Cohort
All patients
6.52%
18.8%
Patients over 75 years old
9.29%
18.1% %
Patients with apoE4 risk factor
8.75%
19.3%
Patients with signs of MCI/dementia
9.26%
26. %
Female patients
7.19%
17.6%
Male patients
5.97%
19.8%
19

---

Published as a conference paper at ICLR 2021
Alternatively, divide patients into cohorts based whether they have belated diagnoses or an uninformative MRI to see which features these criteria correlate with more. We do so in Table For instance, see that a considerable percentage of belated diagnoses are seen among male patients.
Table 6: Features of patients with belated diagnoses and uninformative MRIs.
Patients with belated diagnoses
Patients with
Feature
All patients
uninformative MRTs
Mean age
73.9 7.1
75.8 + 7.5
73.0 :.3
Freq. of apoE4
45.7%
54.2%
49.0%
Freq, of MCI/dementia signs
68.4%
95.8%
98. %
Perc. of female patients
45.4%
39.0%
43.5%
Perc. of male patients
54.6%
61.0%
56.5%
C.4 FURTHER EXAMPLE: DECISION TREES
Clinical practice guidelines are often given in the form decision trees, which usually have vague elements that require the judgement of the practitioner [58] 59]. For example, the guideline could ask the practitioner to quantify risks, side effects, or improvements in subjective terms such as being significant, serious, potential. Using direct policy learning, how vague elements like these commonly resolved in practice can be learned in objective terms.
Formulating policies in terms of mOHMMS and decision boundaries is expressive enough to model decision trees. An 10HMM with deterministic observations, that is O(2la, s') for some x E Z and for all E A, S, essentially describes a finite-state machine, inputs of which are equivalent to the observations. Similarly, a deterministic decision tree can be defined as a finite-state machine with no looping sequence of transitions. The case where the observations are probabilistic rather than deterministic correspond to the case where the decision tree is traversed in a probabilistic way so that each path down the tree has probability associated with at each step of the traversal.
b (INI) 1 x (TSS-DDIS(b))
Patient arrives. Test for the discase.
ToRsios z8r-n8) XOC XOC-POT- (OST ons mnn
INI
Tiptiini 15r-1s) 20 (ijsssis ow)
No
Is there a significant risk of disease?
Yes
ba (DIS) =?
n (TST-TYP|(b) =
HLT
Trossjons Coossjon.ao rey-TW)
DIS
Tiosn/Dis EST-T19)
The patient healthy.
Test for the sub- type of the disease.
XOT- -Or-paT--T TVP, DSA
TOLISST-IE Daj)
The patient has Diseasc-A.
The patient has Disease-B.
DSA
DSB
(a) Decision Tree
(b) An Equivalent 1OHMM
Figure 5: Two Different Descriptions of the Same Policy: (a) the form decision tree and (b) in terms of equivalent IOHMM. For the 1OHMM in (b), arrows denote possible transitions, where the probability of transition is proportional to the quantity written above the corresponding arrow. Using direct policy learning, we can infer the risk of disease, by (DIS)), and the probability of testing for the sub-type based on the risk, क (TSST-TYP(b2) which are left vague in (a).
As a concrete example of modeling decision trees in terms of 1OHMMs, consider the scenario of diagnosing disease with two sub-types: D and Disease-B B.. Figure Ba depicts dalpictss the policy of the doctors in the form a decision tree. Each newly- arrivin patient is first tested for the disease in general sense without any distinction between the sub-types it has. The patient is then tested
20

---

Published as a conference paper at ICLR 2021
for specific sub-type of the disease only if the doctors deem there is significant risk that the patient diseased. Note that which exact level of confidence constitutes as a significant risk left vague in the decision tree. By modeling this scenario using framework, we can learn: (i) how the risk is determined based on initial test results and (ii) what amount risk is considered significant enough require subsequent test for the sub-type.
Let S = {INI, HLT, DIS, DSA, }SB}, where INI denotes that the patient has newly arrived, HLT denotes that the
denotes that the patient is healthy, DIS denotes that the patient diseased, patient has Disease-A, and DSB denotes that the patient has Disease-B. Figure [ depicts the state space S with all possible transitions. Note that the initial belief bN is such that bj (INI)
Let A = {TST-DIS, TST-TYP, STP-HLT, STP-DSA, STP-DSB), where TST-DIS denotes testing for the disease, TST-TYP denotes testing for the sub-type of the disease, and the remaining actions denote stopping and diagnosing the patient with one of the terminal states, namely states HI.T, DSA, and DSB.
After taking action a TST-DIS and observing some initial test result 2 Z. the risk of disease, which is the probability that the patient diseased, can be calculated with a simple belief update:
(DIS) x Σ bi (s) TST-DII))( DIS)
SES
= T(DIS|INN
Moreover, we can say that the doctors more likely for the sub-type of the disease as opposed to stopping and diagnosing the patient as healthy, that is no To(TST-TYPjb) > T0(STP-HLT) b2), when
by (DIS) (DSTT-VV(DII) MsTT-HIT(DIS)
assuming Prst-rve (DIS) Mste-mrr (DIS). Note that there only two possible actions at the second time step: actions TST-TYP and STP-HILT.
D DETAILS OF THE CLINICIAN SURVEYS
Each participant was provided short presentation explaining (1) the ADNI dataset and the decision- making problem consider, (2) what rewards and reward functions are, (3) what beliefs and belief simplices are, and (4) how policies can be represented in terms of reward functions as well as decision boundaries. Then, they were asked multiple-choice questions, one that is strictly about representing histories, and one that is strictly about representing policies. Importantly, the survey was conducted blindly-i.e they were given no context whatsoever as pertains this paper and our proposed method. The question slides can be found in Figures 6 and 7 Essentially, each question first states a hypothesis and shows two/three representations relevant to the hypothesis stated. Then, the participant is asked which of the representations shown most readily expresses the hypothesis. Here are the full responses that we have received, which includes some additional feedback:
Clinician
Question 1: C B>B
Question 2: B
Additional Feedback: The triangle was initially more confusing than not, but the first example (100% uncertainty) was helpful. isn't clear how the dots in the triangle are computed. Are these probabilities based on statistics? Diagram is always better than no diagram.
Clinician
Question 1: B>B
Question 2: B
Additional Feedback: always prefer pictures to tables, they much casier to understand.
Clinician
Question 1: C> A
Question 2: B
Additional Feedback: Of course the triangle is more concise and easier to look at. But how is the decision boundary obtained? Does the decision boundary always have to be parallel to one of the sides of the triangle?
21

---

Published as a conference paper at ICLR 2021
Clinician 4
Question 1: C> A>
Question 2: B
Additional Feedback: [Regarding Question 1.] representation and B do not show any interpreta- tion of the diagnostic test results, whereas representation C does. I think doctors most familiar with representation B, as it more closely resembles the EHR. Although representation C is visually pleasing, I'm sure how the scale of the sides of the triangle should be interpreted. [Regarding Question 2.] again I like the triangle, but it's hard to interpret what the scale of the sides of the triangle mean. think option A is again what doctors are more familiar with.
Clinician
Question 1: C> B
Question 2: A
Clinician
Question 1: C> B>A
Question 2: A
Clinician 7
Question I: C
Question 2: B
Additional Feedback: thought I'd share with you my thoughts on the medical aspects in your scenario first (although I realise you didn't ask for them). [...] The Cochrane review concludes that MRI provides low sensitivity and specificity and does not qualify it as an add on test for the early diagnosis due to dementia (Lombardi et. al. Cochrane database 2020). The reason for MRI imaging is (according to the international guidelines) to exclude non-degenerative or surgical causes cogmitive impairment. [...] In your example the condition became apparent when the CDR-SB score at Month 24 hit 3.0 (supported by the sequence of measurements over time showing worsening CDR-SB score). I imagine the MRI was triggered by slight worsening in the CDR-SB score (to exclude an alternative diagnosis). To answer your specific questions: Q1. The representation C describes your (false) hypothesis that it was the MRI that made the diagnosis of MCI more likely/apparent the best-l really like the triangles. Q2. I really like the decision boundary.
Clinician 8
Question I: C> B>BA
Question 2: B
Clinician Question 1: C
Question 2: B
Additional Feedback: Q1. Representation C gives the clearest illustration of the diagnostic change following MRI. However, the representation of beliefs on a continuous spectrum around discrete cognitive states could be potentially confusing given that cognitive function is itself continuum (and 'MCI. 'Dementia' and 'NL' are stations on spectrum rather than discrete states). Also, while representation C is the clearest illustration, it is the representation that conveys the least actual data and it isn't clear from the visualisation exactly what each shift in 2D space represents. Also, the triangulation in 'C' draws direct connection between NL and Dementia, implying that this is potential alternative route for disease progression, although this is more intuitively considered as linear progression from NL to MCI to Dementia. Q2. For me, the decision boundary representation best expresses the concept of the likelihood of ordering and MRI with the same caveats described above. Option B does best convey the likelihood ordering an MRI, but doesn't convey the information value provided by that investigation. However, my understanding is that this is not what you are aiming to convey here.
REFERENCES
[52] M. Herman, T. Gindele, J. Wagner, F. Schmitt, and W. Burgard, "Inverse reinforcement learning with simultaneous estimation of rewards and dynamics," in Proc. 19th Int. Conf. Artif. Intell. Statist,, 2016, pp. 102-110.
22

---

Published as a conference paper at ICLR 2021
Hypothesis:
This patient's condition (MCI) only became apparent after the first MRI
was ordered.
Question #1: Which of the following representations of the patient's medical history most readily expresses this hypothesis? (Please rank them from most to least accessible.)
MCT MiI ondeed. hippocarppl so uuus: 5080 rar?
3.0 /very mid dementia)
CER-35
25 Sceeen
1,5
Queetonn
155
Daeetion
2.
1.5
Caiecionn
3.0
An KSB orderec. Hippocarpel vtlume
Hippocrppt 5600
hippocampal volume:
sakmart
sakmart Chderes
(below avg-)
MWI
No.
Not
Ordaned
Orderd
Most ADAS11:
500
1900
Volue
(teow
(betww
(heiow
20g
2003
MRlls
Most recent ADAS13:
A36815
AD6511
21.00
18.00
18.37
15.03
21.00
10190007
Most recent whole brain volume: 1040240 mir"
NT.
Dementis
Representation Most Recent Measurement
Representation B:
Representation C: Sequence of Beliefs
Sequence of Measurements
Figure 6: Slide out of 9, which contains the first question regarding histories.
Hypothesis: The more "normal" patient appears, the less likely an MRI is ordered.
Question #2: Which of the following representations of the doctor's decision-making most readily expresses this hypothesis?
MC1
State
Reward of ordering MR)
Reward not ordering MRI
An mom. Ebely
NL
-0. 0.149
-0.120
MCI
0.475
-0. 10.115
Dementia
-0.96
An MEU Bobly to ordered
Dementia
Option A:
Option B: Decision Boundary
Rewards
Figure 7: Slide 9 out of 9, which contains the second question regarding policies.
23

---

Published as a conference paper at ICLR 2021
[53] M. Araya, O. Buffet, V. Thomas, and F. Charpillet, "A pomdp extension with belief-dependent rewards," " Adv. Neural Inf. Process. Syst. 24, 2010, pp. 64-72.
[54] S. Gao, H. C. Hendrie, K. S. Hall, and S. Hui, "The relationships between age, sex, and the incidence of dementia and Alzheimer disease: meta analysis." Arch. General Psychiatry, vol. 55, no. 9, pp. 809-815, 1998.
[55] L. J. Launer, K. Andersen, M. E. Dewey, L. Letenneur, A. Ott, L. A. Amaducci, C. Brayne, J. M. Copeland, J.-F. Dartigues, Kragh-Sorensen, A. Lobo, M. Martinez-Lage, T. Stijnen, and A. Hofman, "Rates and risk factors for dementia and Alzheimer's disease," Neurology, vol. 52, no. I, pp. 78-78, 1999.
[56] J. Lindsay, D. Laurin, R. Verreault, R. Hebert, B. Helliwell, G. B. Hill, and I. McDowell, "Risk factors for Alzheimer's disease: prospective analysis from the Canadian Study of Health and Aging," Amer. J. Epidemiology, vol. 156, no. 5, pp. 445-453, 2002.
[57] J.-H. Chen, K.-P. Lin, and Y.-C. Chen, "Risk factors for dementia," J. Formosan Med. Assoc., vol. 108, no. 10, pp. 754-764.
[58] R. Chou, A. Qaseem, Snow, D. Casey, J. T. Cross, Shekelle, and Owens, "Diagnosis and treatment of low back pain: a joint clinical practice guideline from the American College of Physicians and the American Pain Society," Ann. of Internal Medicine, vol. 147, no. 7, pp. 478 401, 2007.
[59] A. Qaseem, S. D. Fihn, P. Dallas, S. Williams, D. K. Owens, and P. Shekelle, "Manage- ment of stable ischemic heart disease: summary of a clinical practice guideline form the American College of Physicians/Amerriann College of Cardiology Foundation/AAmericnn Heart association/Ameercaan Association for Thoracic Surgery/Preventive Cardiovascular Nurses Association/Society of Thoracic Surgeons," Ann. of Internal Medicine, vol. 157, no. 10, pp. 735-743, 2012.
24


Reviews:
[{"review_id": "KfgMuSHoye", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis work proposes an approach for understanding and explaining decision-making behavior. The authors aim to make the method 1) transparent, 2) able to handle partial observability, and 3) work with offline data. To do this, they develop INTERPOLE, which uses Bayesian techniques to estimate decision dynamics as well as decision boundaries. Results on simulated and real-world domains show that their method explains the decisions in behavior data while still maintaining accuracy and focuses on explaining decision dynamics rather than the “true” dynamics of the world.\n\nStrengths:\n- This work tackles an interesting problem. The proposed setting that considers interpretability + partial observability + offline data is important and reflective of many real-world problems so an approach that works in this setting is very meaningful.\n- The paper is well-written and clear. The authors do a good job clearly stating the motivation for the work and differences from prior work.\n- The authors consider both simulated and real-world data. The application to healthcare is useful and interesting. Overall, the evaluation helps support the claims, except for the interpretability component described below.\n\nWeaknesses:\n- My biggest concern with the work is that it argues for interpretability but the best way to evaluate interpretability is through human evaluation. I appreciate that the authors include the visualizations and talk through what each point means, but it wasn’t 100% clear whether it was truly interpretable. The best way to judge this would be to compare non-interpretable and interpretable techniques and have humans blindly rate which made more sense.\n\nOther comments:\n- In Section 3 under Learning Objective, is it reasonable to assume access to the state space?\n- In Section 5, you talk about counterfactual updates. Can you describe this in more detail? This wasn’t super clear to me.\n- Adding to the weakness point above, the visualization in Figure 3 does not fully make sense. I understand the points and the arrows, but it’s not very clear whether the point’s exact location in the belief simplex is meaningful (e.g., in the middle vs on the line between MCI and dementia).\n- It looks like the action matching metric is used to evaluate the ability to imitate and the belief/policy mismatch is used for evaluating explainability. But belief/policy mismatch cannot be used for ADNI, so for this domain, there’s no proper evaluation of explainability other than the visualizations. I think this is a key piece that needs to be evaluated and shown.\n\nRecommendation: \nOverall, the work poses an interesting problem and solution, but the key motivation is to develop an interpretable approach, which I don’t think is sufficiently and correctly evaluated. Thus, I’m on the fence and would like to hear from the authors about this point.\n\n=========================\n\nResponse after author rebuttal: \nThe authors answered my concern about evaluating the interpretability of the approach. They evaluated the method with a few clinicians, and I'm glad to see that they preferred the authors' method.\n\nAdding these results + clarifying the points I included will definitely make the paper stronger. I increase my score as a result and recommend acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "C0t7b9ql_V", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for obtaining an interpretable representation of some behavioral policy based on partial observations and in an offline learning setting. The paper is well written and the approach is clearly explained. I just have few comments/questions:\n\nIn terms of alternative approaches to modeling agent/decision maker behavior, I can think of at least one other alternative and I wonder how your method can be compared to it: ‘Agent Markov Model’ introduced by Unhelkar and Shah (Learning Models of Sequential Decision-Making with Partial Specification of Agent Behavior AAAI 2019) is also modeling agent behavior using a state space model with partial observations. Similar to your approach, they are also interested in modeling the agent and not the actual mechanics of the world. As far as I understand AMM also satisfies three key criteria presented in the paper. AMM also uses a simpler model for the policy; they assume agent follows a stationary Markov policy: \\pi(a|s, z). Can you discuss advantages of your model over theirs? \n\nAnother advantage of AMM over INTERPOLE is that it can infer the number of latent states via a nonparametric Bayesian approach. In INTERPOLE, do you have any recommendation on how S can be chosen in a more general setting? How sensitive are the results w.r.t model misspecification (specifically w.r.t. the number of latent state)? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "4TrvsYEO-fM", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an algorithm for learning policies and internal models (\"decision dynamics\") from demonstrations. The key idea is to fit a distribution over policies, observation models, and transition models using an EM-like method. Offline experiments on a healthcare dataset show that the method learns interpretable decision dynamics, recovers biased internal models, and accurately predicts actions relative to prior methods.\n\nOverall, the paper is well-written, the experiments are convincing, and the proposed method, Interpole, makes a meaningful contribution to the literature on modeling decision-making.\n\nIt would be nice to evaluate Interpole on tasks in which the demonstrator's decision dynamics operate on high-dimensional latent belief states, to illustrate how Interpole scales with the complexity of the demonstrator's internal models. In such settings, would the mean-vector representation in Equation 2 become problematic due to the curse of dimensionality?\n\nThere is some missing related work on learning from demonstrations that also satisfies the three criteria of transparency by design, partial observability, and offline learning: assistive state estimation [1], learning sensor models from demonstrations in the LQG setting [2], learning from a demonstrator with false beliefs [3], and inverse rational control [4].\n\n1. https://arxiv.org/pdf/2008.02840.pdf \n2. https://fias.uni-frankfurt.de/~rothkopf/docs/Schmitt_et_al_2017.pdf\n3. https://arxiv.org/pdf/1512.05832.pdf \n4. https://arxiv.org/pdf/2009.12576.pdf \n\nTypos:\n - \"log-like-lihood\" -> log-likelihood (page 5)\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Explaining by Imitating: Understanding Decisions by Interpretable
Policy Learning

The topic is maximally timely and important: Understanding human
decision-making behaviour based on observational data. Any tangible
steps towards this challenging goal are bound to be significant, and
those this paper makes.

A Bayesian policy-learning method is introduced for this task, and
validated on both simulated data and user exeperiments in a real
decision-making task. The novel contribution is on learning
interpretable decision dynamics

The paper is written clearly enough..

The updated paper clarified most major concerns the reviewers had. In
particular, they added a user study.

The biggest remaining weaknesses are that

- relationship to the AMM model did not become completely clear yet

- the real user study has been carried out with only a small set of
users. But a large-cohort study would be too much work to ask for a
paper which has also a strong methodological contribution.

Decision: Accept (Poster)