Inverse Contextual Bandits: Learning How Behavior Evolves over Time

1 INTRODUCTION . Modeling decision-making policies is a central concern in computational and behavioral science , with key applications in healthcare [ 1 ] , economics [ 2 ] , and cognition [ 3 ] . The business of policy learning is to determine an agent ’ s decision-making policy given observations of their behavior . Typically , the objective is either to replicate the behavior of some demonstrator ( cf . “ imitation learning ” ) [ 4 , 5 ] , or to match their performance on the basis of some reward function ( cf . “ apprenticeship learning ” ) [ 6 , 7 ] . However , equally important is the pursuit of “ descriptive modeling ” [ 8 , 9 ] —that is , of learning interpretable parameterizations of behavior for auditing , quantifying , and understanding decision-making policies . For instance , recent work has studied representing behaviors in terms of rules [ 10 ] , goals [ 11 ] , intentions [ 12 ] , preferences [ 13 ] , subjective dynamics [ 14 ] , as well as counterfactual outcomes [ 15 ] . Evolving Behaviors In this work , we ask a novel descriptive question : How has the observed behavior changed over time ? While conventional approaches to policy learning almost invariably assume that decision-making agents are stationary , this is rarely the case in practice : In many settings , behaviors evolve constantly as decision-makers learn more about their environment and adjust their policies accordingly . In fact , disseminating new knowledge from medical research into actual clinical practice is itself a major endeavor in healthcare [ 16–18 ] . This research question is new : While capturing “ variation in practice ” in observed data has been studied in the context of demonstrations containing mixed policies [ 19 , 20 ] , multiple tasks [ 21 , 22 ] , and subgroup-/patient-specific preferences [ 8 ] , little work has attempted to capture variation in practice over time as an agent ’ s knowledge of the world evolves . Example ( Organ Allocations ) As our core application , consider organ allocation practices for liver transplantations : In the medical community , our understanding of organ transplantations has changed numerous times over past decades [ 23–25 ] . Thus an immediate question lies in how actual organ allocation practices have changed over the years . Having a data-driven , quantitative , and— importantly—interpretable description of how practices have evolved is crucial : It would enable policy-makers to evaluate if the policies they introduced have had the intended impact on practice ; this would in turn play a substantial role in designing better policies going forward [ 8 ] ( see Figure 1 ) . To tackle questions of this form , we desire a policy learning method that satisfies three key desiderata : • It should provide an ( i . ) interpretable description of observed behavior ( see Appendix C ) . Suppose that policy-makers introduce a new policy to promote Features A and B as main factors of consideration in decision-making ( whereas clinicians rely on Feature C at the time ) , and the policy succeeds at establishing Feature A as such but fails to do so for Feature B . Using ICB , we can infer the time-varying importances of these features and observe quantitatively that importance of Feature B stayed the same despite the intentions of the new policy . Having made this observation , the policymakers can now update their policy accordingly . • It should be able to capture an agent ’ s ( ii . ) nonstationary knowledge of the world . • It should operate in an ( iii . ) offline manner— since online experimentation is impossible in highstakes environments such as healthcare . Inverse Contextual Bandits To accomplish this , we first identify the organ allocation problem as a contextual bandits problem [ 26–28 ] : Given each arriving instance of patient and organ features ( i.e . the context ) , an agent makes an allocation decision ( i.e . the action ) , whence some measure of feedback is perceived ( i.e . internal reward ) . Crucially , the environment ( i.e . precisely , its reward dynamics ) is unknown to the agent and must be actively learned , so the agent maintains beliefs about their environment ( i.e . internal knowledge ) . Not only must an agent select actions that “ exploit ” the knowledge they have , but they must also select actions that “ explore ” the environment to update their knowledge . Thus the behavior of a learning agent is naturally modeled as a generalized bandit strategy—that is , how to take actions based on their knowledge , and how to update their knowledge based on the outcomes of their actions . Now , the forward contextual bandits problem asks the ( normative ) question : Given an unknown environment , what is an effective bandit strategy that minimizes some notion of regret ? By contrast , our focus here is instead on the opposite direction—that is , in formalizing and solving the problem of Inverse Contextual Bandits ( “ ICB ” ) : We ask the ( descriptive ) question : Given demonstrated behavior from a decision-making agent , how has the agent ’ s knowledge been evolving over time ? Precisely , we wish to learn interpretable representations of generalized bandit strategies from the observable context-action pairs generated by those strategies—regardless of whether those strategies are effective . Contributions Our contributions are three-fold . In the sequel , we first formalize the ICB problem , identifying it with the data-driven objective of inferring an agent ’ s internal reward function along with their internal trajectory of beliefs about the environment ( Section 2 ) . Second , we propose two concrete learning algorithms , imposing different specifications regarding the agent ’ s behavioral strategy : The first parameterizes the agent ’ s knowledge in terms of Bayesian updates , whereas the second makes the milder specification that the agent ’ s behavior evolves smoothly over time ( Section 3 ) . Third , through both simulated and real-world data for liver transplantations , we illustrate how ICB can be applied as an investigative device for recovering and explaining the evolution of organ allocation practices over the years , as well as benchmarking and validating the accuracy of our algorithms ( Section 5 ) . 2 INVERSE CONTEXTUAL BANDITS . Preliminaries Consider a Markov decision process of the form D : = ( X , A , R , T ) , where S indicates the state space , A the action space , R ∈ ∆ ( R ) X×A the reward dynamics , and T ∈ ∆ ( X ) X×A the transition dynamics . At each time t ∈ Z+ , the decision-making agent is presented with some state xt ∈ X and decides to take an action at ∈ A , whence an immediate reward rt ∼ R ( xt , at ) is received , and a subsequent state st+1 ∼ T ( xt , at ) is presented . Let the decision-making policy employed by the agent be denoted π ∈ ∆ ( A ) X , such that actions at any time are sampled according to at ∼ π ( xt ) . In the forward direction , given dynamicsR , T , the reinforcement learning problem ( “ RL ” ) deals with determining the optimal policy that maximizes some notion of expected cumulative rewards [ 29 ] : π∗R , T : = argmaxπ∈∆ ( A ) X Eπ , R , T ∑ t rt . In the opposite direction , given some observed behavior policy πb and transition dynamics T , the inverse reinforcement learning problem ( “ IRL ” ) deals with determining the reward dynamics R with respect to which πb appears optimal . For instance , the classic max-margin approach seeks [ 30 ] : R∗ : = argminR ( maxπ Eπ , R , T ∑ t rt − Eπb , R , T ∑ t rt ) . Having Dynamics Access to Having No Dynamics Access In conventional RL ( and IRL ) , the decision-maker has unrestricted access to environment dynamics—either explicitly ( i.e . R , T are simply known and used in computing the optimal policy ) , or implicitly ( i.e . the agent may interact freely with the environment during training ) . In contrast , in our setting the agent has no such luxuries—not only are the dynamics not known to the agent , but neither do they enjoy a distinct sandboxed training phase prior to live deployment . Without such access , the agent must consider both the information they may gain when taking actions ( cf . “ exploration ” ) and also the expected rewards due to those actions ( cf . “ exploitation ” ) . Note that this property results in much more difficult learning problems—in both the forward and inverse directions ( see Appendix B for a more detailed discussion ) . Consider environments parameterized by environment parameters ρ ∈ P , such that the reward dynamics of an environment are given byRρ , and the transition dynamics by Tρ . Let ρenv denote the true environment parameter , such that the actual rewards received by an agent are distributed as rt ∼ Rρenv ( xt , at ) , and the actual states encountered by the agent are distributed as xt+1 ∼ Tρenv ( xt , at ) . Since ρenv is unknown , an agent takes actions on the basis of their beliefs about it ; these beliefs are described by probability distributions Pβ ∈ ∆ ( P ) parameterized by belief parameters β ∈ B . For each time t , let βt capture the agent ’ s belief at the beginning of that step . Each time step , the agent first samples an environment parameter ρt ∼ Pβt according to their belief βt , then takes an action according to π∗Rρt , Tρt . Note that this ensures that each action is takenwith probability proportional to the probability with which the agent believes it to be optimal at time step t. Essentially , the agent ’ s policy at time step t is induced by their belief βt : πβt ( x ) [ a ] : = Eρ∼Pβt [ π ∗ Rρ , Tρ ( x ) [ a ] ] . After receiving a reward rt , the agent updates their current belief parameter according to a ( possibly-stochastic ) belief-update function f ∈ ∆ ( B ) B×X×A×R , such that βt+1 ∼ f ( βt , xt , at , rt ) . Together with the initial belief parameter β1 , the agent ’ s t-th step belief is a ( possibly-stochastic ) function of the history ht = { x1 : t−1 , a1 : t−1 , r1 : t−1 } defined recursively via f . 2.1 CONTEXTUAL BANDITS SETTING . In this work , we consider state transitions that occur independently of past states and actions . Due to this property , decisions can be made greedily without consideration of what future states may be , and yields a contextual bandits problem [ 26–28 ] . Note that this captures the organ allocation problem well : Distributions of newly arriving organs are largely independent of prior allocation decisions ; in fact , transplantation and allocation policies are often modeled in bandit-like settings [ 31 , 32 ] . Formally : Definition 1 ( Contextual Bandits ) Consider a decision-making problem D : = ( X , A , R , T ) , where R , T are unknown to the agent . Let T ( x , a ) = T ′ for some T ′ ∈ ∆ ( X ) , for all x ∈ X , a ∈ A , such that policies are greedy : supp ( π∗t ( xt ) ) = argmaxat∈A R̄ρt ( xt , at ) , ( 1 ) where R̄ρ ( x , a ) : = Er∼Rρ ( x , a ) [ r ] indicates the mean reward function , and ties are broken arbitrarily . Given a space of environment parameterizations P , the contextual bandits problem is to design a space of belief parameterizations B , and to determine the optimal belief-update function f∗ : = argmaxf∈∆ ( B ) B×X×A×R ∑ tEf , πβt , R , T [ rt ] . ( 2 ) Now , suppose that an agent follows a bandit-type policy for T time steps ; this would generate an observational dataset of contexts and actions D : = { x1 : T , a1 : T } ( adhering to the contextual bandits literature , we refer to states as “ contexts ” hereafter ) .1 But since rewards rt and beliefs βt are quantities internal to the decision-maker , we assume that they are not part of the observational dataset : The former represents the agent ’ s preferences over outcomes after observing contexts and taking actions , which is not explicitly observable ; likewise , the latter represents the agent ’ s beliefs about what kinds of outcomes their actions result in , which is also not explicitly observable . Thus we ask the novel question : From D , can we infer the true environment parameter ρenv , as well as the belief trajectory β1 : T ? β1 ρ1 x1 a1 r1 β2 ρ2 x2 a2 r2 β3 ρ∗ f f ··· ··· Figure 2 . Graphical Model for ICB . We aim to infer the ( dotted ) red quantities given the ( solid ) blue ones . Definition 2 ( Inverse Contextual Bandits ) Consider again a contextual bandit problem D : = ( X , A , R , T ) , and recall that dynamics R , T are unknown to the agent . Given an observational dataset D , and a family of environment parameterizations P and belief parameterizations B , the inverse contextual bandits problem ( “ ICB ” ) is to determine the true environment parameter ρenv and the belief parameters β1 : T ( see Figure 2 ) . 1We also use “ x∈X ” to denote contexts ( standard in contextual bandits ) instead of “ s∈S ” ( common in RL ) . It is important to observe that the novelty here is general—that is , in posing the “ inverse ” question of how knowledge evolves over time . Focusing our attention on contextual bandits simply represents a specific choice of problem setting : In this first work on tackling the inverse problem in modeling nonstationary agents , it presents a more tractable problem for analysis , and is moreover especially suited for our motivating application in organ allocations . Note that in the bandit setting , the fact that transition dynamics are unknown to the agent is ultimately inconsequential due to greedy policies ; we refer to ρ simply as “ reward parameter ” hereafter . We leave generalization to arbitrary transition dynamics T ∈ ∆ ( X ) X×A to future work . Also note that ICB itself is not a bandit problem ( see Appendix C ) . Remark 1 ( Environment vs. State Beliefs ) When speaking of inferring “ beliefs ” here in our setting , we are referring to environment beliefs—that is , an agent ’ s knowledge of the environment ’ s rewards . It is important to distinguish this from state beliefs in partially-observed environments [ 33 , 34 ] —that is , an agent ’ s estimate of which latent state they are in . State beliefs are computed by an agent using their ( known ) environment parameters , whereas environment beliefs concern the environment ’ s ( unknown ) parameters themselves . State beliefs can be easily inferred [ 35 ] : all factors that contribute to state-belief updates ( i.e . observations and actions ) are observable ; on the other hand , environment beliefs are more technically challenging due to latent factors ( i.e . internal rewards r1 : T ) that are never observable . Because of the same reason , tackling ICB with conventional IO-HMM inference methods is not possible either , even though the agent can be viewed as an IO-HMM ( see Appendix C ) . Remark 2 ( Subjective vs . Objective Reward ) When speaking of learning what “ rewards ” an agent is optimizing , the novelty here that our objective refers to the full belief trajectory β1 : T of the agent . It is important to distinguish this from simply learning the ground-truth parameter ρenv as in typical IRL . The ground-truth parameter is an objective quantity capturing the “ prescriptive ” notion of what the agent ought to be optimizing , whereas the belief trajectory is a subjective quantity capturing the “ descriptive ” notion of what the agent appears to be optimizing . Existing work in learning from nonstationary agents has focused solely on ρenv , which is all that is required for apprenticeship [ 36 , 37 ] . In contrast , for the goal of understanding behavior ( and how it evolves ) , Definition 2 brings β1 : T to focus .

Summary: This paper addresses the problem of Inverse Contextual Bandit.  They raise an important question: given demonstrated behavior from an agent, how has the agent’s knowledge been evolving over time? Formally, Given a contextual bandit problem $(X, A, R, T )$,  where $R, T$ are unknown to the agent. Given an observational dataset $D$, and a family of reward parameterizations $P$ and belief parameterizations $B$, the inverse contextual bandits problem is to determine the true environment parameter $\rho^*$ and the belief parameters $\beta_{1:T}$.   They propose two algorithms to learn these parameters. The first uses the agent’s knowledge in terms of Bayesian update. The second uses the Gaussian process. They demonstrate their algorithm through simulated and real-world data for liver transplantations.