Explaining by Imitating: Understanding Decisions by Interpretable Policy Learning

1 INTRODUCTION . A principal challenge in modeling human behavior is in obtaining a transparent understanding of decision-making . In medical diagnosis , for instance , there is often significant regional and institutional variation in clinical practice [ 1 ] , much of it the leading cause of rising healthcare costs [ 2 ] . The ability to quantify different decision processes is the first step towards a more systematic understanding of medical practice . Purely by observing demonstrated behavior , our principal objective is to answer the question : Under any given state of affairs , what actions are ( more/less ) likely to be taken , and why ? We address this challenge by setting our sights on three key criteria . First , we desire a method that is transparent by design . Specifically , a transparent description of behavior should locate the factors that contribute to individual decisions , in a language readily understood by domain experts [ 3 , 4 ] . This will be clearer per our subsequent formalism , but we can already note some contrasts : Classical imitation learning—popularly by reduction to supervised classification—does not fit the bill , since black-box hidden states of RNNs are rarely amenable to meaningful interpretation . Similarly , apprenticeship learning algorithms—popularly through inverse reinforcement learning—do not satisfy either , since the high-level nature of reward mappings is not informative as to individual actions observed in the data . Rather than focusing purely on replicating actions ( imitation learning ) or on matching expert performance ( apprenticeship learning ) , our chief pursuit lies in understanding demonstrated behavior . Second , real-world environments such as healthcare are often partially observable in nature . This requires modeling the accumulation of information from entire sequences of past observations—an endeavor that is prima facie at odds with the goal of transparency . For instance , in a fully-observable setting , ( model-free ) behavioral cloning is arguably ‘ transparent ’ in providing simple mappings of states to actions ; however , coping with partial observability using any form of recurrent function ∗Authors contributed equally approximation immediately lands in black-box territory . Likewise , while ( model-based ) methods have been developed for robotic control , their transparency crucially hinges on fully-observable kinematics . Finally , in realistic settings it is often impossible to experiment online—especially in high-stakes environments with real products and patients . The vast majority of recent work in ( inverse ) reinforcement learning has focused on games , simulations , and gym environments where access to live interaction is unrestricted . By contrast , in healthcare settings the environment dynamics are neither known a priori , nor estimable by repeated exploration . We want a data-driven representation of behavior that is learnable in a completely offline fashion , yet does not rely on knowing/modeling any true dynamics . Contributions Our contributions are three-fold . First , we propose a model for interpretable policy learning ( “ INTERPOLE ” ) —where sequential observations are aggregated through a decision agent ’ s decision dynamics ( viz . subjective belief-update process ) , and sequential actions are determined by the agent ’ s decision boundaries ( viz . probabilistic belief-action mapping ) . Second , we suggest a Bayesian learning algorithm for estimating the model , simultaneously satisfying the key criteria of transparency , partial observability , and offline learning . Third , through experiments on both simulated and real-world data for Alzheimer ’ s disease diagnosis , we illustrate the potential of our method as an investigative device for auditing , quantifying , and understanding human decision-making behavior . 2 RELATED WORK . We seek to learn an interpretable parameterization of observed behavior to understand an agent ’ s actions . Fundamentally , this contrasts with imitation learning ( which seeks to best replicate demonstrated policies ) and apprenticeship learning ( which seeks to match some notion of performance ) . Imitation Learning In fully-observable settings , behavior cloning ( BC ) readily reduces the imitation problem to one of supervised classification [ 5 , 11–13 ] ; i.e . actions are simply regressed on observations . While this can be extended to account for partial observability by parameterizing policies via recurrent function approximation [ 14 ] , it immediately gives up on ease of interpretability per the black-box nature of RNN hidden states . A plethora of model-free techniques have recently been developed , which account for information in the rollout dynamics of the environment during policy learning ( see e.g . [ 15–20 ] ) —most famously , generative adversarial imitation learning ( GAIL ) based on statedistribution matching [ 6 , 21 ] . However , such methods require repeated online rollouts of intermediate policies during training , and also face the same black-box problem as BC in partially observable settings . Clearly in model-free imitation , it is difficult to admit both transparency and partial observability . Specifically with an eye on explainability , Info-GAIL [ 22 , 23 ] proposes an orthogonal notion of “ interpretability ” that hinges on clustering similar demonstrations to explain variations in behavior . However , as with GAIL it suffers from the need for live interaction for learning . Finally , several model-based techniques for imitation learning ( MB-IL ) have been studied in the domain of robotics . [ 24 ] consider kinematic models designed for robot dynamics , while [ 25 ] and [ 7 ] consider ( non- ) linear autoregressive exogenous models . However , such approaches invariably operate in fully-observable settings , and are restricted models hand-crafted for specific robotic applications under consideration . Apprenticeship Learning In subtle distinction to imitation learning , methods in apprenticeship learning assume the observed behavior is optimal with respect to some underlying reward function . Apprenticeship thus proceeds indirectly—often through inverse reinforcement learning ( IRL ) in order to infer a reward function , which ( with appropriate optimization ) generates learned behavior that matches the performance of the original—as measured by the rewards ( see e.g . [ 8 , 26–29 ] ) . These approaches have been variously extended to cope with partial observability ( PO-IRL ) [ 9 , 30 ] , to offline settings through off-policy evaluation [ 31–33 ] , as well as to learned environment models [ 10 ] . However , a shortcoming of such methods is the requirement that the demonstrated policy in fact be optimal with respect to a true reward function that lies within an ( often limited ) hypothesis class under consideration—or is otherwise black-box in nature . Further , learning the true environment dynamics [ 10 ] corresponds to the requirement that policies be restricted to the class of functions that map from unbiased beliefs ( cf . exact inference ) into actions . Notably though , [ 34 ] considers both a form of suboptimality caused by time-inconsistent agents as well as biased beliefs . However , perhaps most importantly , due to the indirect , task-level nature of reward functions , inverse reinforcement learning is essentially opposed to our central goal of transparency—that is , in providing direct , action-level descriptions of behavior . In Section 5 , we provide empirical evidence of this notion of interpretability . Towards INTERPOLE In contrast , we avoid making any assumptions as to either unbiasedness of beliefs or optimality of policies . After all , the former requires estimating ( externally ) “ true ” environment dynamics , and the latter requires specifying ( objectively ) “ true ” classes of reward functions— neither of which are necessary per our goal of transparently describing individual actions . Instead , INTERPOLE simply seeks the most plausible explanation in terms of ( internal ) decision dynamics and ( subjective ) decision boundaries . To the best of our knowledge , our work is the first to tackle all three key criteria—while making no assumptions on the generative process behind behaviors . Table 1 contextualizes our work , showing typical incarnations of related approaches and their graphical models . Before continuing , we note that the separation between the internal dynamics of an agent and the external dynamics of the environment has been considered in several other works , though often for entirely different problem formulations . Most notably , [ 35 ] tackles the same policy learning problem as we do in online , fully-observable environments but for agent ’ s with internal states that can not be observed . They propose agent Markov models ( AMMs ) to model such environment-agent interactions . For problems other than policy learning , [ 36–38 ] also consider the subproblem of inferring an agent ’ s internal dynamics ; however , none of these works satisfy all three key criteria simultaneously as we do . 3 INTERPRETABLE POLICY LEARNING . We first introduce INTERPOLE ’ s model of behavior , formalizing notions of decision dynamics and decision boundaries . In the next section , we suggest a Bayesian algorithm for model-learning from data . Problem Setup Consider a partially-observable decision-making environment in discrete time . At each step t , the agent takes action at ∈ A and observes outcome zt ∈ Z.1 We have at our disposal an observed dataset of demonstrations D= { ( ai1 , zi1 , . . . , aiτi , z i τi ) } n i=1 by an agent , τi being the length of the i-th trajectory ( we shall omit indices i unless required ) . Denote by ht . = ( a1 , z1 , . . . , at−1 , zt−1 ) the observed history at the beginning of step t , where h1 . =∅ . Analogously , let Ht . = ( A × Z ) t−1 indicate the set of all possible histories at the start of step t , where H1 . = { ∅ } , and let H .= ∪∞t=1Ht . A proper policy π is a mapping π ∈ ∆ ( A ) H from observed histories to action distributions , where π ( a|h ) is the probability of taking action a given h. We assume that D is generated by an agent acting according to some behavioral policy πb . The problem we wish to tackle , then , is precisely how to obtain an interpretable parameterization of πb . We proceed in two steps : First , we describe a parsimonious belief-update process for accumulating histories—which we term decision dynamics . Then , we take beliefs to actions via a probabilistic mapping—which gives rise to decision boundaries . Decision Dynamics We model belief-updates by way of an input-output hidden Markov model ( IOHMM ) identified by the tuple ( S , A , Z , T , O , b1 ) , with S being the finite set of underlying states . T ∈ ∆ ( S ) S×A denotes the transition function such that T ( st+1|st , at ) gives the probability of transitioning into state st+1 upon action at in state st , and O ∈ ∆ ( Z ) A×S denotes the observation function such that O ( zt|at , st+1 ) gives the probability of observing zt after taking action at and transitioning into state st+1 . Finally , let beliefs bt ∈ ∆ ( S ) indicate the probability bt ( s ) that the 1While we take it here that Z is finite , our method can easily be generalized to allow continuous observations . environment exists in any state s ∈ S at time t , and let b1 give the initial state distribution . Note that—unlike in existing uses of the IOHMM formalism—these “ probabilities ” are for representing the thought process of the human , and may freely diverge from the actual mechanics of the world . To aggregate observed histories as beliefs , we identify bt ( s ) with P ( st = s|ht ) —an interpretation that leads to the recursive belief-update process ( where in our problem , quantities T , O , b1 are unknown ) : bt+1 ( s ′ ) ∝ ∑ s∈S bt ( s ) T ( s ′|s , at ) O ( zt|at , s′ ) ( 1 ) A key distinction bears emphasis : We do not require that this latter set of quantities correspond to ( external ) environment dynamics—and we do not obligate ourselves to recover any such notion of “ true ” parameters . To do so would imply the assumption that the agent in fact performs exactly unbiased inference on a perfectly known model of the environment , which is restrictive . It is also unnecessary , since our mandate is simply to model the ( internal ) mechanics of decision-making— which could well be generated from possibly biased beliefs or imperfectly known models of the world . In other words , our objective ( see Equation 3 ) of simultaneously determining the most likely beliefs ( cf . decision dynamics ) and policies ( cf . decision boundaries ) is fundamentally more parsimonious . Decision Boundaries Given decision dynamics , a policy is then equivalently a map π ∈ ∆ ( A ) ∆ ( S ) . Now , what is an interpretable parameterization ? Consider the three-state example in Figure 1 . We argue that a probabilistic parameterization that directly induces “ decision regions ” ( cf . panel 1b ) over the belief simplex is uniquely interpretable . For instance , strong beliefs that a patient has underlying mild cognitive impairment may map to the region where a specific follow-up test is promptly prescribed ; this parameterization allows clearly locating such regions—as well as their boundaries . Precisely , we parameterize policies in terms of |A|-many “ mean ” vectors that correspond to actions : π ( a|b ) = e−η‖b−µa‖ 2 / ∑ a′∈A e −η‖b−µa′‖ 2 , ∑ s∈S µa ( s ) = 1 ( 2 ) where η ≥ 0 is the inverse temperature , ‖ · ‖ the ` 2-norm , and µa ∈ R|S| the mean vector corresponding to action a ∈ A . Intuitively , mean vectors induce decision boundaries ( and decision regions ) over the belief space ∆ ( S ) : At any time , the action whose corresponding mean is closest to the current belief is most likely to be chosen . In particular , lines that are equidistant to the means of any pair of actions form decision boundaries between them . The inverse temperature controls the transitions between such boundaries : A larger η captures more deterministic behavior ( i.e . more “ abrupt ” transitions ) , whereas a smaller η captures more stochastic behavior ( i.e . “ smoother ” transitions ) . Note that the case of η = 0 recovers policies that are uniformly random , and η →∞ recovers argmax policies . A second distinction is due : The exponentiated form of Equation 2 should not be confused with typical Boltzmann [ 27 ] or MaxEnt [ 39 ] policies common in RL : These are indirect parameterizations via optimal/soft q-values , which themselves require approximate solutions to optimization problems ; as we shall see in our experiments , the quality of learned policies suffers as a result . Further , using q-values would imply the assumption that the agent in fact behaves optimally w.r.t . an ( objectively ) “ true ” class of reward functions—e.g . linear—which is restrictive . It is also unnecessary , as our mandate is simply to capture their ( subjective ) tendencies toward different actions—which are generated from possibly suboptimal policies . In contrast , by directly partitioning the belief simplex into probabilistic “ decision regions ” , INTERPOLE ’ s mean-vector representation can be immediately explained and understood . Learning Objective In a nutshell , our objective is to identify the most likely parameterizations T , O , b1 for decision dynamics as well as η , { µa } a∈A for decision boundaries , given the observed data : Given : D , S , A , Z Determine : T , O , b1 , η , { µa } a∈A ( 3 ) Next , we propose a Bayesian algorithm that finds the maximum a posteriori ( MAP ) estimate of these quantities . Figure 2 illustrates the problem setup .

Summary: The paper proposes an algorithm for learning policies and internal models ("decision dynamics") from demonstrations. The key idea is to fit a distribution over policies, observation models, and transition models using an EM-like method. Offline experiments on a healthcare dataset show that the method learns interpretable decision dynamics, recovers biased internal models, and accurately predicts actions relative to prior methods.