Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies

1 INTRODUCTION . Decision modelling is often viewed through the lens of policy learning ( Bain & Sammut , 1995 ; Abbeel & Ng , 2004 ) , where the aim is to learn some imitation policy that captures the actions ( and thus decisions ) of an agent in some structured way ( Hüyük et al. , 2021 ) . Unfortunately , this usually implicitly relies on the assumption that the actions taken are already optimal with respect to some objective , and hence do not change over time . While this might be appropriate to approximate the stationary policy of an autonomous agent , such assumptions often break down when applying them to more flexible agents that might be learning on-the-fly . In particular , we may want to model the decision making process of a human decision-maker , whose actions we may see executed over time - this is clearly of great use in a range of fields including economics and medicine ( Hunink et al. , 2014 ; Zavadskas & Turskis , 2011 ) . In the context of machine learning , they would be considered to be undergoing a process of online learning ( Hoi et al. , 2018 ) ; training while interacting with the environment , where it is to be expected that beliefs and strategies are constantly adapted based on accumulating experience ( Elio & Pelletier , 1997 ) . Additionally , such belief updates are often imperfect ; for example , people are known to a priori over-weight the potential impact of rare events when they first start a task , before proceeding to neglect them when they do not occur in their experience ( Hertwig & Erev , 2009 ) . If our goal is to analyse human decision-making , it is thus important to be able to model non-stationary policies that are potentially imperfect in two senses : both marginally at each time-step , and in the way that they become adapted to new information . In this work , instead of relying on the assumption that an agent follows a stationary optimal policy , we consider their actions to be consistent within an online leaning framework where they learn from an incoming sequence of examples ( Hoi et al. , 2018 ) and update their policies accordingly , as in Figure 1 . Thus , instead of assuming that actions are globally optimal , we assume that at any exact time the agent thought that the action was better than the alternatives . As a natural consequence , we frame the problem of decision modelling as that of inverse online learning ; that is , we aim to uncover the process through which an agent is learning online . Note that this does not involve coming up with a new way to do online learning itself , rather we recover how it appears a given agent may have learnt . We do this by modelling the ( non-stationary ) policy of the agent using a deep state-space model ( Krishnan et al. , 2017 ) , allowing us to track the memory of the agent while allowing policies to evolve in a non-parametric and non-prescribed fashion . We construct an interpretable mapping from the accrued memory to policies by framing the decision-making process within a potential-outcome setting ( Rubin , 2005 ) ; letting actions correspond to interventions ( or treatments ) , and assuming that agents make decisions by choosing an action which they perceive to have the highest ( treatment ) effect . Contributions . In this work we make a number of contributions : First , we formalise the inverse online learning problem and a connection between decision modelling , online learning and potential outcomes , highlighting how they complement each other to result in an interpretable understanding of policies ( Section 2 ) ; Second , we introduce a practical method to estimate the non-stationary policy and update method of an agent given demonstrations ( Section 4 ) ; And third , we demonstrate how we can uncover useful practical insights into the decision making process of agents with a case study on the acceptance of liver donation offers ( Section 5 ) . 2 PROBLEM FORMALISATION . Preliminaries . Assume we observe a data trajectory1 of the form D = { ( Xt , At , Yt ) } Tt=1 . Here , Xt ∈ X ⊂ Rd , denotes a context vector of possible confounders ; At ∈ A = { 0 , 1 } , a binary action or intervention ; and Yt ∈ Y , a binary or continuous outcome of interest . The subscript t indicates a time-ordering of observed triplets ; any time-step t is generated by ( i ) arrival of context x , ( ii ) an intervention a being performed by an agent according to some policy πt ( x ) and ( iii ) a corresponding outcome y being observed . Goal . We are interested in recovering the agent ’ s non-stationary policy πt ( x ) = P ( A = 1|X = x , Ht = ht ) , which depends on the observed context x and can change over time due to the observed history Ht = { ( Xk , Ak , Yk ) } t−1k=1 . We make the key assumption that the agent acts with the intention to choose an intervention a leading to the best ( largest ) potential outcome Y ( a ) , but that the perception of the optimal intervention may change over time throughout the agent ’ s learning process about their environment . This problem formalisation naturally places us at the in- tersection of treatment effect inference within the Neyman-Rubin potential outcomes ( PO ) framework ( Rubin , 2005 ) with the classical ( inverse ) reinforcement learning ( RL ) paradigm ( Abbeel & Ng , 2004 ) . More specifically though the RL-subproblem of logged contextual bandits ( Swaminathan & Joachims , 2015 ; Atan et al. , 2018 ) , where the setting is offline and the contexts arrive independently . Perception Versus Reality . As is standard within the PO framework , we assume that any observation has two potential outcomes Y ( a ) , associated with each intervention ( treatment-level ) , which are 1We assume with a single trajectory here , though learning and inference can be applied over a dataset of multiple trajectories generated from some true distribution inherent to the environment , i.e . Y ( 0 ) , Y ( 1 ) ∼ Pe ( ·|X = x ) , yet only the potential outcome associated with the chosen treatment is revealed to the agent . If the agent had full knowledge of the environment , its optimal policy would be to choose an action as arg maxa µea ( x ) where µ e a ( x ) = EPe [ Y ( a ) |X = x ] , or conversely to choose to perform the intervention only when its expected conditional average treatment effect ( CATE ) τe ( x ) = µe1 ( x ) − µe0 ( x ) is positive . In general , we assume that such perfect knowledge of the environment is not available to the agent in many scenarios of practical interest . Instead , we assume that at any time t , the agent acts according to some perceived model P̃ of its environment , which induces the perceived expected potential outcomes µ̃a ( x ; t ) and associated treatment effect τ̃ ( x ; t ) . Here , µ̃a ( x ; t ) = EP̃ [ Y ( a ) |X = x , Ht = ht ] , is a function not only of the observed context but also the history , such that the agent can update its beliefs according to observed outcomes . Note that we allow the agent ’ s model of the environment to be misspecified ; in this case µ̃a ( x ; t ) would not converge to µea ( x ) even for very long trajectories . We consider this an important feature , as we assume that human decision-makers generally need to make simplifications ( Griffiths et al. , 2015 ) . Formally , we make the following assumption about the agent ’ s decision making process : Assumption 1 Mutual Observability : The observable space X contains only and all the information available to the agent at the point of assigning an action . Assumption 2 Perceived Optimality : An agent assigns the action they think will maximise the outcome . For a deterministic policy πt , the agent believes πt ( x ) = 1 ⇐⇒ µ̃1 ( x ; t ) > µ̃0 ( x ; t ) . Given a stochastic policy , πt ( x ) is a monotonic increasing function of the agent ’ s belief over the treatment effect τ̃ ( x ; t ) . Assumption 3 Continual Adaptation : The agent may continually adjust their strategy based on experience . The index in the dataset D represents a temporal ordering which π is non-stationary with respect to , i.e . πi ( x ) may not equal πj ( x ) , i 6= j ≥ 1 where πi represents the policy at time step i. Assumptions 1 and 2 are generally unverifiable in practice , yet , they are necessary : there would be little opportunity to learn anything meaningful about a non-stationary policy without them . Mutual observability is crucial , as assuming that different information streams are available to the observer and the agent would make it impossible to accurately describe the decision-making process . Assumption 2 , albeit untestable , appears very reasonable : in the medical setting , for example , a violation would imply malicious intent – i.e . that a clinician purposefully chooses the intervention with a sub-optimal expected outcome . Nonetheless , this does mean that we do not consider exploratory agents that are routinely willing to sacrifice outcomes for knowledge , a constraint we do not consider too restrictive within the medical applications we have in mind . Assumption 3 , on the other hand , is not at all restrictive ; on the contrary , it explicitly specifies a more flexible framework than is usual in imitation or inverse reinforcement learning as the set of policies it describes contains stationary policies . 3 RELATED WORK . Our problem setting is related to , but distinct from , a number of problems considered within related literature . Effectively , we assume that we observe logged data generated by a learning agent that acts according to its evolving belief over effects of its actions , and aim to solve the inverse problem by making inferences over said belief given the data . Below , we discuss our relationship to work studying both the forward and the inverse problem of decision modelling , with main points summarised in Table 1 . The Forward Problem : Inferring Prescriptive Models for Behaviour . The agent , whose ( nonstationary ) policy we aim to understand , actively learns online through repeated interaction with an environment and thus effectively solves an online learning problem ( Hoi et al. , 2018 ) . The most prominent example of such an agent would be a contextual bandit ( Agarwal et al. , 2014 ) , which learns to assign sequentially arriving contexts to ( treatment ) arms . Designing agents that update policies online is thus the inverse of the problem we consider here . Another seemingly related problem within the RL context is learning optimal policies from logged bandit data ( Swaminathan & Joachims , 2015 ) ; this is different from our setting as it is prescriptive rather than descriptive . If the goal was to infer the true τe ( x ) from the ( logged ) observational data D ( instead of the effect perceived by the agent ) , this would constitute a standard static CATE estimation problem , for which many solutions have been proposed in the recent ML literature ( Alaa & van der Schaar , 2018 ; Shalit et al. , 2017 ; Künzel et al. , 2019 ) . Note that within this literature , the treatmentassignment policy ( the so-called propensity score π ( x ) = P ( A = 1|X = x ) ) is usually assumed to be stationary and considered a nuisance parameter that is not of primary interest . Estimating CATE can also be a pretext task to the problem of developing optimal treatment rules ( OTR ) ( Zhang et al. , 2012a ; b ; Zhao et al. , 2012 ) , as g ( x ) = 1 ( τ ( x ) > 0 ) is such an optimal rule ( Zhang et al. , 2012b ) . The Backward Problem : Obtaining a Descriptive Summary of an Agent . When no outcomes are observed ( unlike in the bandit problem ) the task of learning a policy becomes imitation learning ( IL ) , itself a large field that aims to match the policy of a demonstrator ( Bain & Sammut , 1995 ; Ho & Ermon , 2016 ) . Note that , compared to the bandit problem , there is a subtle shift from obtaining a policy that is optimal with respect to some true notion of outcome to one that minimises some divergence from the demonstrator . While IL is often used in the forward problem to find an optimal policy – thereby implicitly assuming that the demonstrator is themselves acting optimally – , it can also , if done interpretably , be used in the backward problem to very effectively reason about the goals and preferences of the agent ( Hüyük et al. , 2021 ) . One way to achieve this is through inverse reinforcement learning ( IRL ) , which aims to recover the reward function that is seemingly maximised by the agent ( Ziebart et al. , 2008 ; Fu et al. , 2018 ; Chan & van der Schaar , 2021 ) . This need not be the true reward function , and can be interpreted as a potentially more compact way to describe a policy , which is also more easily portable given shifts in environment dynamics . Our problem formalisation is conceptually most closely related to the counterfactual inverse reinforcement learning ( CIRL ) work of Bica et al . ( 2021 ) . There the authors similarly aim to explain decisions based on the PO framework , specifically by augmenting a max-margin IRL ( Abbeel & Ng , 2004 ) approach to parameterise the learnt reward as a weighted sum of counterfactual outcomes . However , CIRL involves a pre-processing counterfactual estimation step that focuses on estimating the true treatment effects , and implicitly assumes that these are identical to the ones perceived by the agent ( i.e . it assumes that the agent has perfect knowledge of the environment dynamics generating the potential outcomes ) , marking a significant departure from our ( much weaker ) assumptions .

Summary: The objective of the paper is an interpretable descriptive model of online learning from trajectories of the form $(x_t,a_t,y_t)$, where $x_t$ is an i.i.d. context, $a_t$ is the action taken that depends on the context and history $(x,a,y)$ up to time $t-1$, and $y_t$ is the outcome resulting from the application of $a_t$ to $x_t$.  The authors' motivating example is liver donation acceptance decisions: a potential donor approaches with an organ they are offering to a particular recipient. The treatment center makes a binary decision to accept or reject the donation based on features of the donor and the recipient. The outcome is the survival time of the patient.  The structure of the authors' model is based on three components: 1. A memory summarization network that maps from the history up to time $t-1$ to the parameters of a multivariate normal distribution over memory vectors. The specific structure is that a new memory vector is produced from the previous memory vector and the observation from the previous time step. 2. An outcome estimation network. This network takes the memory vector as input and produces weights for each action. A dot product of these weights plus the (exogenous) action features yields the agent's estimate of the outcome of each action. 3. The treatment rule is just a logistic function applied to the predicted outcomes.  The model is trained using stochastic variation inference with the ELBO using an LSTM.  Empirically, the authors find that their approach outperforms action prediction baselines such as behavioral cloning, reward-regularised classification for apprenticeship learning and counterfactual inverse reinforcement learning. Additionally, they use their model to analyze shifts in the transplant policy over time, e.g., the declining importance of MELD score and how offers that are rejected make it less likely than a subsequent offer will be rejected.