Explaining by Imitating: Understanding Decisions by Interpretable Policy Learning

1 INTRODUCTION . A principal challenge in modeling human behavior is in obtaining a transparent understanding of decision-making . In medical diagnosis , for instance , there is often significant regional and institutional variation in clinical practice [ 1 ] , much of it the leading cause of rising healthcare costs [ 2 ] . The ability to quantify different decision processes is the first step towards a more systematic understanding of medical practice . Purely by observing demonstrated behavior , our principal objective is to answer the question : Under any given state of affairs , what actions are ( more/less ) likely to be taken , and why ? We address this challenge by setting our sights on three key criteria . First , we desire a method that is transparent by design . Specifically , a transparent description of behavior should locate the factors that contribute to individual decisions , in a language readily understood by domain experts [ 3 , 4 ] . This will be clearer per our subsequent formalism , but we can already note some contrasts : Classical imitation learning—popularly by reduction to supervised classification—does not fit the bill , since black-box hidden states of RNNs are rarely amenable to meaningful interpretation . Similarly , apprenticeship learning algorithms—popularly through inverse reinforcement learning—do not satisfy either , since the high-level nature of reward mappings is not informative as to individual actions observed in the data . Rather than focusing purely on replicating actions ( imitation learning ) or on matching expert performance ( apprenticeship learning ) , our chief pursuit lies in understanding demonstrated behavior . Second , real-world environments such as healthcare are often partially observable in nature . This requires modeling the accumulation of information from entire sequences of past observations—an endeavor that is prima facie at odds with the goal of transparency . For instance , in a fully-observable setting , ( model-free ) behavioral cloning is arguably ‘ transparent ’ in providing simple mappings of states to actions ; however , coping with partial observability using any form of recurrent function ∗Authors contributed equally approximation immediately lands in black-box territory . Likewise , while ( model-based ) methods have been developed for robotic control , their transparency crucially hinges on fully-observable kinematics . Finally , in realistic settings it is often impossible to experiment online—especially in high-stakes environments with real products and patients . The vast majority of recent work in ( inverse ) reinforcement learning has focused on games , simulations , and gym environments where access to live interaction is unrestricted . By contrast , in healthcare settings the environment dynamics are neither known a priori , nor estimable by repeated exploration . We want a data-driven representation of behavior that is learnable in a completely offline fashion , yet does not rely on knowing/modeling any true dynamics . Contributions Our contributions are three-fold . First , we propose a model for interpretable policy learning ( “ INTERPOLE ” ) —where sequential observations are aggregated through a decision agent ’ s decision dynamics ( viz . subjective belief-update process ) , and sequential actions are determined by the agent ’ s decision boundaries ( viz . probabilistic belief-action mapping ) . Second , we suggest a Bayesian learning algorithm for estimating the model , simultaneously satisfying the key criteria of transparency , partial observability , and offline learning . Third , through experiments on both simulated and real-world data for Alzheimer ’ s disease diagnosis , we illustrate the potential of our method as an investigative device for auditing , quantifying , and understanding human decision-making behavior . 2 RELATED WORK . We seek to learn an interpretable parameterization of observed behavior to understand an agent ’ s actions . Fundamentally , this contrasts with imitation learning ( which seeks to best replicate demonstrated policies ) and apprenticeship learning ( which seeks to match some notion of performance ) . Imitation Learning In fully-observable settings , behavior cloning ( BC ) readily reduces the imitation problem to one of supervised classification [ 5 , 11–13 ] ; i.e . actions are simply regressed on observations . While this can be extended to account for partial observability by parameterizing policies via recurrent function approximation [ 14 ] , it immediately gives up on ease of interpretability per the black-box nature of RNN hidden states . A plethora of model-free techniques have recently been developed , which account for information in the rollout dynamics of the environment during policy learning ( see e.g . [ 15–20 ] ) —most famously , generative adversarial imitation learning ( GAIL ) based on statedistribution matching [ 6 , 21 ] . However , such methods require repeated online rollouts of intermediate policies during training , and also face the same black-box problem as BC in partially observable settings . Clearly in model-free imitation , it is difficult to admit both transparency and partial observability . Specifically with an eye on explainability , Info-GAIL [ 22 , 23 ] proposes an orthogonal notion of “ interpretability ” that hinges on clustering similar demonstrations to explain variations in behavior . However , as with GAIL it suffers from the need for live interaction for learning . Finally , several model-based techniques for imitation learning ( MB-IL ) have been studied in the domain of robotics . [ 24 ] consider kinematic models designed for robot dynamics , while [ 25 ] and [ 7 ] consider ( non- ) linear autoregressive exogenous models . However , such approaches invariably operate in fully-observable settings , and are restricted models hand-crafted for specific robotic applications under consideration . Apprenticeship Learning In subtle distinction to imitation learning , methods in apprenticeship learning assume the observed behavior is optimal with respect to some underlying reward function . Apprenticeship thus proceeds indirectly—often through inverse reinforcement learning ( IRL ) in order to infer a reward function , which ( with appropriate optimization ) generates learned behavior that matches the performance of the original—as measured by the rewards ( see e.g . [ 8 , 26–29 ] ) . These approaches have been variously extended to cope with partial observability ( PO-IRL ) [ 9 , 30 ] , to offline settings through off-policy evaluation [ 31–33 ] , as well as to learned environment models [ 10 ] . However , a shortcoming of such methods is the requirement that the demonstrated policy in fact be optimal with respect to a true reward function that lies within an ( often limited ) hypothesis class under consideration—or is otherwise black-box in nature . Further , learning the true environment dynamics [ 10 ] corresponds to the requirement that policies be restricted to the class of functions that map from unbiased beliefs ( cf . exact inference ) into actions . Notably though , [ 34 ] considers both a form of suboptimality caused by time-inconsistent agents as well as biased beliefs . However , perhaps most importantly , due to the indirect , task-level nature of reward functions , inverse reinforcement learning is essentially opposed to our central goal of transparency—that is , in providing direct , action-level descriptions of behavior . In Section 5 , we provide empirical evidence of this notion of interpretability . Towards INTERPOLE In contrast , we avoid making any assumptions as to either unbiasedness of beliefs or optimality of policies . After all , the former requires estimating ( externally ) “ true ” environment dynamics , and the latter requires specifying ( objectively ) “ true ” classes of reward functions— neither of which are necessary per our goal of transparently describing individual actions . Instead , INTERPOLE simply seeks the most plausible explanation in terms of ( internal ) decision dynamics and ( subjective ) decision boundaries . To the best of our knowledge , our work is the first to tackle all three key criteria—while making no assumptions on the generative process behind behaviors . Table 1 contextualizes our work , showing typical incarnations of related approaches and their graphical models . Before continuing , we note that the separation between the internal dynamics of an agent and the external dynamics of the environment has been considered in several other works , though often for entirely different problem formulations . Most notably , [ 35 ] tackles the same policy learning problem as we do in online , fully-observable environments but for agent ’ s with internal states that can not be observed . They propose agent Markov models ( AMMs ) to model such environment-agent interactions . For problems other than policy learning , [ 36–38 ] also consider the subproblem of inferring an agent ’ s internal dynamics ; however , none of these works satisfy all three key criteria simultaneously as we do . 3 INTERPRETABLE POLICY LEARNING . We first introduce INTERPOLE ’ s model of behavior , formalizing notions of decision dynamics and decision boundaries . In the next section , we suggest a Bayesian algorithm for model-learning from data . Problem Setup Consider a partially-observable decision-making environment in discrete time . At each step t , the agent takes action at ∈ A and observes outcome zt ∈ Z.1 We have at our disposal an observed dataset of demonstrations D= { ( ai1 , zi1 , . . . , aiτi , z i τi ) } n i=1 by an agent , τi being the length of the i-th trajectory ( we shall omit indices i unless required ) . Denote by ht . = ( a1 , z1 , . . . , at−1 , zt−1 ) the observed history at the beginning of step t , where h1 . =∅ . Analogously , let Ht . = ( A × Z ) t−1 indicate the set of all possible histories at the start of step t , where H1 . = { ∅ } , and let H .= ∪∞t=1Ht . A proper policy π is a mapping π ∈ ∆ ( A ) H from observed histories to action distributions , where π ( a|h ) is the probability of taking action a given h. We assume that D is generated by an agent acting according to some behavioral policy πb . The problem we wish to tackle , then , is precisely how to obtain an interpretable parameterization of πb . We proceed in two steps : First , we describe a parsimonious belief-update process for accumulating histories—which we term decision dynamics . Then , we take beliefs to actions via a probabilistic mapping—which gives rise to decision boundaries . Decision Dynamics We model belief-updates by way of an input-output hidden Markov model ( IOHMM ) identified by the tuple ( S , A , Z , T , O , b1 ) , with S being the finite set of underlying states . T ∈ ∆ ( S ) S×A denotes the transition function such that T ( st+1|st , at ) gives the probability of transitioning into state st+1 upon action at in state st , and O ∈ ∆ ( Z ) A×S denotes the observation function such that O ( zt|at , st+1 ) gives the probability of observing zt after taking action at and transitioning into state st+1 . Finally , let beliefs bt ∈ ∆ ( S ) indicate the probability bt ( s ) that the 1While we take it here that Z is finite , our method can easily be generalized to allow continuous observations . environment exists in any state s ∈ S at time t , and let b1 give the initial state distribution . Note that—unlike in existing uses of the IOHMM formalism—these “ probabilities ” are for representing the thought process of the human , and may freely diverge from the actual mechanics of the world . To aggregate observed histories as beliefs , we identify bt ( s ) with P ( st = s|ht ) —an interpretation that leads to the recursive belief-update process ( where in our problem , quantities T , O , b1 are unknown ) : bt+1 ( s ′ ) ∝ ∑ s∈S bt ( s ) T ( s ′|s , at ) O ( zt|at , s′ ) ( 1 ) A key distinction bears emphasis : We do not require that this latter set of quantities correspond to ( external ) environment dynamics—and we do not obligate ourselves to recover any such notion of “ true ” parameters . To do so would imply the assumption that the agent in fact performs exactly unbiased inference on a perfectly known model of the environment , which is restrictive . It is also unnecessary , since our mandate is simply to model the ( internal ) mechanics of decision-making— which could well be generated from possibly biased beliefs or imperfectly known models of the world . In other words , our objective ( see Equation 3 ) of simultaneously determining the most likely beliefs ( cf . decision dynamics ) and policies ( cf . decision boundaries ) is fundamentally more parsimonious . Decision Boundaries Given decision dynamics , a policy is then equivalently a map π ∈ ∆ ( A ) ∆ ( S ) . Now , what is an interpretable parameterization ? Consider the three-state example in Figure 1 . We argue that a probabilistic parameterization that directly induces “ decision regions ” ( cf . panel 1b ) over the belief simplex is uniquely interpretable . For instance , strong beliefs that a patient has underlying mild cognitive impairment may map to the region where a specific follow-up test is promptly prescribed ; this parameterization allows clearly locating such regions—as well as their boundaries . Precisely , we parameterize policies in terms of |A|-many “ mean ” vectors that correspond to actions : π ( a|b ) = e−η‖b−µa‖ 2 / ∑ a′∈A e −η‖b−µa′‖ 2 , ∑ s∈S µa ( s ) = 1 ( 2 ) where η ≥ 0 is the inverse temperature , ‖ · ‖ the ` 2-norm , and µa ∈ R|S| the mean vector corresponding to action a ∈ A . Intuitively , mean vectors induce decision boundaries ( and decision regions ) over the belief space ∆ ( S ) : At any time , the action whose corresponding mean is closest to the current belief is most likely to be chosen . In particular , lines that are equidistant to the means of any pair of actions form decision boundaries between them . The inverse temperature controls the transitions between such boundaries : A larger η captures more deterministic behavior ( i.e . more “ abrupt ” transitions ) , whereas a smaller η captures more stochastic behavior ( i.e . “ smoother ” transitions ) . Note that the case of η = 0 recovers policies that are uniformly random , and η →∞ recovers argmax policies . A second distinction is due : The exponentiated form of Equation 2 should not be confused with typical Boltzmann [ 27 ] or MaxEnt [ 39 ] policies common in RL : These are indirect parameterizations via optimal/soft q-values , which themselves require approximate solutions to optimization problems ; as we shall see in our experiments , the quality of learned policies suffers as a result . Further , using q-values would imply the assumption that the agent in fact behaves optimally w.r.t . an ( objectively ) “ true ” class of reward functions—e.g . linear—which is restrictive . It is also unnecessary , as our mandate is simply to capture their ( subjective ) tendencies toward different actions—which are generated from possibly suboptimal policies . In contrast , by directly partitioning the belief simplex into probabilistic “ decision regions ” , INTERPOLE ’ s mean-vector representation can be immediately explained and understood . Learning Objective In a nutshell , our objective is to identify the most likely parameterizations T , O , b1 for decision dynamics as well as η , { µa } a∈A for decision boundaries , given the observed data : Given : D , S , A , Z Determine : T , O , b1 , η , { µa } a∈A ( 3 ) Next , we propose a Bayesian algorithm that finds the maximum a posteriori ( MAP ) estimate of these quantities . Figure 2 illustrates the problem setup .

Summary: This work proposes an approach for understanding and explaining decision-making behavior. The authors aim to make the method 1) transparent, 2) able to handle partial observability, and 3) work with offline data. To do this, they develop INTERPOLE, which uses Bayesian techniques to estimate decision dynamics as well as decision boundaries. Results on simulated and real-world domains show that their method explains the decisions in behavior data while still maintaining accuracy and focuses on explaining decision dynamics rather than the “true” dynamics of the world.