Year: 2021

Abstract:
Human decision making is well known to be imperfect and the ability to analyse such processes individually is crucial when attempting to aid or improve a decisionmaker’s ability to perform a task, e.g. to alert them to potential biases or oversights on their part. To do so, it is necessary to develop interpretable representations of how agents make decisions and how this process changes over time as the agent learns online in reaction to the accrued experience. To then understand the decisionmaking processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem. By interpreting actions within a potential outcomes framework, we introduce a meaningful mapping based on agents choosing an action they believe to have the greatest treatment effect. We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them, using a novel architecture built upon an expressive family of deep state-space models. Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.

heading: 1 INTRODUCTION
text: Decision modelling is often viewed through the lens of policy learning (Bain & Sammut, 1995; Abbeel & Ng, 2004), where the aim is to learn some imitation policy that captures the actions (and thus decisions) of an agent in some structured way (Hüyük et al., 2021). Unfortunately, this usually implicitly relies on the assumption that the actions taken are already optimal with respect to some objective, and hence do not change over time. While this might be appropriate to approximate the stationary policy of an autonomous agent, such assumptions often break down when applying them to more flexible agents that might be learning on-the-fly. In particular, we may want to model the decision making process of a human decision-maker, whose actions we may see executed over time - this is clearly of great use in a range of fields including economics and medicine (Hunink et al., 2014; Zavadskas & Turskis, 2011). In the context of machine learning, they would be considered to be undergoing a process of online learning (Hoi et al., 2018); training while interacting with the environment, where it is to be expected that beliefs and strategies are constantly adapted based on accumulating experience (Elio & Pelletier, 1997). Additionally, such belief updates are often imperfect; for example, people are known to a priori over-weight the potential impact of rare events when they first start a task, before proceeding to neglect them when they do not occur in their experience (Hertwig & Erev, 2009). If our goal is to analyse human decision-making, it is thus important to be able to model non-stationary policies that are potentially imperfect in two senses: both marginally at each time-step, and in the way that they become adapted to new information.
In this work, instead of relying on the assumption that an agent follows a stationary optimal policy, we consider their actions to be consistent within an online leaning framework where they learn from an incoming sequence of examples (Hoi et al., 2018) and update their policies accordingly, as in Figure 1. Thus, instead of assuming that actions are globally optimal, we assume that at any exact time the agent thought that the action was better than the alternatives. As a natural consequence, we frame the problem of decision modelling as that of inverse online learning; that is, we aim to uncover the process through which an agent is learning online. Note that this does not involve coming up with a new way to do online learning itself, rather we recover how it appears a given agent may have learnt. We do this by modelling the (non-stationary) policy of the agent using a deep state-space
model (Krishnan et al., 2017), allowing us to track the memory of the agent while allowing policies to evolve in a non-parametric and non-prescribed fashion. We construct an interpretable mapping from the accrued memory to policies by framing the decision-making process within a potential-outcome setting (Rubin, 2005); letting actions correspond to interventions (or treatments), and assuming that agents make decisions by choosing an action which they perceive to have the highest (treatment) effect.
Contributions. In this work we make a number of contributions: First, we formalise the inverse online learning problem and a connection between decision modelling, online learning and potential outcomes, highlighting how they complement each other to result in an interpretable understanding of policies (Section 2); Second, we introduce a practical method to estimate the non-stationary policy and update method of an agent given demonstrations (Section 4); And third, we demonstrate how we can uncover useful practical insights into the decision making process of agents with a case study on the acceptance of liver donation offers (Section 5).

heading: 2 PROBLEM FORMALISATION
text: Preliminaries. Assume we observe a data trajectory1 of the form D = {(Xt, At, Yt)}Tt=1. Here, Xt ∈ X ⊂ Rd, denotes a context vector of possible confounders; At ∈ A = {0, 1}, a binary action or intervention; and Yt ∈ Y , a binary or continuous outcome of interest. The subscript t indicates a time-ordering of observed triplets; any time-step t is generated by (i) arrival of context x, (ii) an intervention a being performed by an agent according to some policy πt(x) and (iii) a corresponding outcome y being observed.
Goal. We are interested in recovering the agent’s non-stationary policy πt(x) = P (A = 1|X = x,Ht = ht), which depends on the observed context x and can change over time due to the observed history Ht = {(Xk, Ak, Yk)}t−1k=1. We make the key assumption that the agent acts with the intention to choose an intervention a leading to the best (largest) potential outcome Y (a), but that the perception of the optimal intervention may change over time throughout the agent’s learning process about their environment. This problem formalisation naturally places us at the in-
tersection of treatment effect inference within the Neyman-Rubin potential outcomes (PO) framework (Rubin, 2005) with the classical (inverse) reinforcement learning (RL) paradigm (Abbeel & Ng, 2004). More specifically though the RL-subproblem of logged contextual bandits (Swaminathan & Joachims, 2015; Atan et al., 2018), where the setting is offline and the contexts arrive independently.
Perception Versus Reality. As is standard within the PO framework, we assume that any observation has two potential outcomes Y (a), associated with each intervention (treatment-level), which are
1We assume with a single trajectory here, though learning and inference can be applied over a dataset of multiple trajectories
generated from some true distribution inherent to the environment, i.e. Y (0), Y (1) ∼ Pe(·|X = x), yet only the potential outcome associated with the chosen treatment is revealed to the agent. If the agent had full knowledge of the environment, its optimal policy would be to choose an action as arg maxa µea(x) where µ e a(x) = EPe [Y (a)|X = x], or conversely to choose to perform the intervention only when its expected conditional average treatment effect (CATE) τe(x) = µe1(x)− µe0(x) is positive. In general, we assume that such perfect knowledge of the environment is not available to the agent in many scenarios of practical interest. Instead, we assume that at any time t, the agent acts according to some perceived model P̃ of its environment, which induces the perceived expected potential outcomes µ̃a(x; t) and associated treatment effect τ̃(x; t). Here, µ̃a(x; t) = EP̃[Y (a)|X = x,Ht = ht], is a function not only of the observed context but also the history, such that the agent can update its beliefs according to observed outcomes. Note that we allow the agent’s model of the environment to be misspecified; in this case µ̃a(x; t) would not converge to µea(x) even for very long trajectories. We consider this an important feature, as we assume that human decision-makers generally need to make simplifications (Griffiths et al., 2015). Formally, we make the following assumption about the agent’s decision making process:
Assumption 1 Mutual Observability: The observable space X contains only and all the information available to the agent at the point of assigning an action.
Assumption 2 Perceived Optimality: An agent assigns the action they think will maximise the outcome. For a deterministic policy πt, the agent believes πt(x) = 1 ⇐⇒ µ̃1(x; t) > µ̃0(x; t). Given a stochastic policy, πt(x) is a monotonic increasing function of the agent’s belief over the treatment effect τ̃(x; t).
Assumption 3 Continual Adaptation: The agent may continually adjust their strategy based on experience. The index in the dataset D represents a temporal ordering which π is non-stationary with respect to, i.e. πi(x) may not equal πj(x), i 6= j ≥ 1 where πi represents the policy at time step i.
Assumptions 1 and 2 are generally unverifiable in practice, yet, they are necessary: there would be little opportunity to learn anything meaningful about a non-stationary policy without them. Mutual observability is crucial, as assuming that different information streams are available to the observer and the agent would make it impossible to accurately describe the decision-making process. Assumption 2, albeit untestable, appears very reasonable: in the medical setting, for example, a violation would imply malicious intent – i.e. that a clinician purposefully chooses the intervention with a sub-optimal expected outcome. Nonetheless, this does mean that we do not consider exploratory agents that are routinely willing to sacrifice outcomes for knowledge, a constraint we do not consider too restrictive within the medical applications we have in mind. Assumption 3, on the other hand, is not at all restrictive; on the contrary, it explicitly specifies a more flexible framework than is usual in imitation or inverse reinforcement learning as the set of policies it describes contains stationary policies.

heading: 3 RELATED WORK
text: Our problem setting is related to, but distinct from, a number of problems considered within related literature. Effectively, we assume that we observe logged data generated by a learning agent that acts according to its evolving belief over effects of its actions, and aim to solve the inverse problem by making inferences over said belief given the data. Below, we discuss our relationship to work studying both the forward and the inverse problem of decision modelling, with main points summarised in Table 1.
The Forward Problem: Inferring Prescriptive Models for Behaviour. The agent, whose (nonstationary) policy we aim to understand, actively learns online through repeated interaction with an environment and thus effectively solves an online learning problem (Hoi et al., 2018). The most prominent example of such an agent would be a contextual bandit (Agarwal et al., 2014), which learns to assign sequentially arriving contexts to (treatment) arms. Designing agents that update policies online is thus the inverse of the problem we consider here. Another seemingly related problem within the RL context is learning optimal policies from logged bandit data (Swaminathan & Joachims, 2015); this is different from our setting as it is prescriptive rather than descriptive.
If the goal was to infer the true τe(x) from the (logged) observational data D (instead of the effect perceived by the agent), this would constitute a standard static CATE estimation problem, for
which many solutions have been proposed in the recent ML literature (Alaa & van der Schaar, 2018; Shalit et al., 2017; Künzel et al., 2019). Note that within this literature, the treatmentassignment policy (the so-called propensity score π(x) = P(A = 1|X = x)) is usually assumed to be stationary and considered a nuisance parameter that is not of primary interest. Estimating CATE can also be a pretext task to the problem of developing optimal treatment rules (OTR) (Zhang et al., 2012a;b; Zhao et al., 2012), as g(x) = 1(τ(x) > 0) is such an optimal rule (Zhang et al., 2012b).
The Backward Problem: Obtaining a Descriptive Summary of an Agent. When no outcomes are observed (unlike in the bandit problem) the task of learning a policy becomes imitation learning (IL), itself a large field that aims to match the policy of a demonstrator (Bain & Sammut, 1995; Ho & Ermon, 2016). Note that, compared to the bandit problem, there is a subtle shift from obtaining a policy that is optimal with respect to some true notion of outcome to one that minimises some divergence from the demonstrator. While IL is often used in the forward problem to find an optimal policy – thereby implicitly assuming that the demonstrator is themselves acting optimally –, it can also, if done interpretably, be used in the backward problem to very effectively reason about the goals and preferences of the agent (Hüyük et al., 2021). One way to achieve this is through inverse reinforcement learning (IRL), which aims to recover the reward function that is seemingly maximised by the agent (Ziebart et al., 2008; Fu et al., 2018; Chan & van der Schaar, 2021). This need not be the true reward function, and can be interpreted as a potentially more compact way to describe a policy, which is also more easily portable given shifts in environment dynamics.
Our problem formalisation is conceptually most closely related to the counterfactual inverse reinforcement learning (CIRL) work of Bica et al. (2021). There the authors similarly aim to explain decisions based on the PO framework, specifically by augmenting a max-margin IRL (Abbeel & Ng, 2004) approach to parameterise the learnt reward as a weighted sum of counterfactual outcomes. However, CIRL involves a pre-processing counterfactual estimation step that focuses on estimating the true treatment effects, and implicitly assumes that these are identical to the ones perceived by the agent (i.e. it assumes that the agent has perfect knowledge of the environment dynamics generating the potential outcomes), marking a significant departure from our (much weaker) assumptions.

heading: 4 INFERRING THE ONLINE LEARNING PROCESS
text: Preliminaries: The Forward Problem. In the forward context, the problem we consider amounts to an agent attempting to, at each time-step, take the action they believe will maximise the outcome, without being aware of the true effect of their intervention. We assume that the agent believes the potential outcomes are a linear combination of the contextual features such that µ̃a(Xt; t) = 〈Xt, ωta〉, with ωta ∈ Ω a set of weights for the action a at time t, and 〈·, ·〉 denoting the inner product. Normally we may consider a linear model an oversimplification, but when modelling the thought process of an agent it appears much more reasonable - for example, does a doctor consider higher order terms when weighing up potential outcomes of a patient? It seems unlikely, especially as clinical guidelines are often given as linear cutoffs (Burgers et al., 2003). The considered forward problem thus proceeds as follows for every given time step t:
1. A context Xt ∼ PX arrives independently over time. We make no assumptions on the form of PX , in particular noting that there is no need for any level of stationarity.
2. Given Xt and the current set of weights {ωt1, ωt0}, the agent predicts their perceived potential outcomes µ̃1(Xt; t) and µ̃0(Xt; t).
3. Action At is assigned by some probabilistic optimal treatment rule π : Y2 7→ [0, 1] such that P(At = 1|Xt) = π(µ̃1(Xt; t), µ̃0(Xt; t)), before the associated true outcome is observed.
4. Based on this new information the agent (potentially stochastically) updates their belief over the response surface for the taken action according to some update function f : Ω2 ×X ×A×Y 7→ 4(Ω2) such that ωt+11 , ω t+1 0 ∼ f(ωt1, ωt0, Xt, At, Yt).
To maximise outcomes it is crucial to employ an update function f that maximally captures the available information at each step of the process. This then allows for an appropriate determination of π that aims to maximise the total (potentially future discounted) outcomes.

heading: 4.1 A MODEL OF INVERSE ONLINE LEARNING
text: Here, we focus on the inverse to the problem outlined above, which, to the best of our knowledge, has received little to no attention in related work. That is, after observing an agent’s actions, can we recover how they arrived at their decisions, and how this changed over time? In the following, we use vector notation ~xt:t′ = {xi}t ′
i=t with ~x = ~x1:T for all realised variables, with histories ~ht:t′ = {(xi, ai, yi)}t ′
i=t . Note that the previously described forward model assumes a factorisation of the full generative distribution of the observed data given by:
p(~x,~a, ~y) = T∏ t=1 p(yt|at, xt)× p(at|~x1:t−1,~a1:t−1, ~y1:t−1)× p(xt). (1)
We assume that p(xt) and p(yt|at, xt) are governed by some true environment dynamics, independent from the perception of the agent and thus not part of our modelling problem. This seems trivially true, as the perception of the agent will neither affect the contexts that arrive nor the true outcome conditional on the action being taken.
Our model for the evolving beliefs of an agent with respect to their interaction with the environment revolves around a specialised deep state-space model. Crucially, this takes the form of a latent random variable Mt ∈ Rd, which captures the memory of the agent at time t, and can evolve in a flexible way based on the observed history. We consider the memory to be an efficient summary of the history, which can be interpreted as a sufficient statistic for the beliefs of the agent at a given time. Naturally, this memory is unobserved and must be learnt in an unsupervised manner. The introduction of this latent variable leads to an extended probabilistic model with a more structured factorisation of the conditional distribution of the actions, and a model given by:
pθ(~a|~x, ~y) = T∏ t=1 πθ(at|τ̃t)︸ ︷︷ ︸ Treatment Rule × Oθ(τ̃t|xt,mt)︸ ︷︷ ︸ Outcome Estimation × Sθ(mt|~x1:t−1,~a1:t−1, ~y1:t−1)︸ ︷︷ ︸ Memory Aggregation . (2)
This model consists of three components: a memory summarisation network Sθ : H 7→ 4(M), which maps the history into a (distribution over) memory; a (perceived) potential outcome predictor network Oθ : X × M 7→ R2, that takes the memory and a context and predicts the perceived potential outcomes; and a treatment rule πθ : Y2 7→ [0, 1], that given the potential outcomes (summarised by the perceived treatment effect) outputs a distribution over actions. Throughout, θ denotes the full set of parameters of the generative model and notation is shared across components to reflect that they can all be optimised jointly. Below, we discuss each component in turn.
Memory Aggregation. We define the base structure of Sθ to be relatively simple and recursive, and assume the memory at a given time step to be distributed as:
Mt ∼ N ( µt,Σt ) with µt,Σt = MLP([Mt−1, Xt−1, At−1, Yt−1]), (3)
where µt and Σt are two output heads of the memory network S given the history ht−1. It is important to note that the memory network does not get the t-th step information (including the context) when predicting the memory at time t - this means the network cannot provide any specific predictive power for a given context and is forced to model the general response surfaces.
In many models of behaviour the update function f of the agent is modelled as perfect Bayesian inference (Kaelbling et al., 1998; Colombo & Hartmann, 2017). While this may seems natural on the surface given the logical consistency of Bayesian inference, it seems unlikely in practice that agents (and especially humans) will be capable of it. Not least because in all but the simplest cases it will become intractable for computational agents and humans themselves are well known to make simplifying approximations when dealing with past histories (Genewein et al., 2015). By employing a flexible neural architecture, we remove this relatively restrictive assumption on the memory update process. Our model then learns exactly how the agent appears to use previous information to update their predictions - allowing us to model common cognitive biases such as overly weighting recent or largely negative events (Hertwig & Erev, 2009).
Outcome Estimation. Having obtained a memory state mt, the second stage involves predicting the perceived potential outcomes for a context xt with the network Oθ(τ̃t|xt,mt). Given the forward model where the potential outcomes are considered a linear combination of the features this then proceeds again in three steps:
ωt1, ω t 0 = MLP(mt), (4)
µ̃1(xt; t) = 〈xt, ωt1〉, µ̃0(xt; t) = 〈xt, ωt0〉 (5) τ̃t = µ̃1(xt; t)− µ̃0(xt; t). (6)
First the memory mt is decoded into a set of weights for each action using some standard feedforward multi-layer perceptron (MLP), before the potential outcomes are predicted by taking the inner product with the context. This is preferable to, for example, concatenating and passing both xt and mt through a network since, while it may be potentially more expressive, it loses the connection to the forward model as well as the interpretability that is given by linearity. As it is, while our model of the potential outcomes is linear at each time step, the memory allows this to flexibly change throughout the course of the history. The perceived treatment effect is then calculated as the difference between potential outcomes. However, given the linear nature of the predictions, there is no non-degenerate solution for the ωs. Thus, to ensure identifiability, we set ωt0 = 0 as a baseline.
Treatment Rule. Under Assumption 2, the agent should be expected to take an action that maximises the outcome. There are different possible options for the exact treatment rule, the most obvious being π(at|τ̃t) = 1(τ̃t > 0); assuming that as long as the treatment effect is positive then the intervention will be taken. However, in order to maintain differentiability, as well as to allow for more modelling flexibility to capture the stochasitcity of agents, we parameterise the policy as a soft version of the indicator function:
π(at|τ̃t) = 1
1 + exp(−α(τ̃t − β)) . (7)
Assuming positive α, this is clearly monotonically increasing, satisfying Assumption 2. These parameters are learnt by the model and thus allow us to model a flexible threshold with β as well as aleatoric uncertainty in the actions of the agent with α.

heading: 4.2 LEARNING WITH AN INFERENCE NETWORK
text: As with all flavours of deep state space models, exact posterior inference is analytically intractable, making optimisation of the marginal likelihood equally so. As such, we aim to maximise an Evidence Lower BOund (ELBO) on this likelihood of the data following standardised variational principles (Blei et al., 2017; Hoffman et al., 2013). This is achieved by positing a parameterised approximate posterior distribution qφ(~m|~h), and with a simple application of Jensen’s inequality arrive at:
log pθ(~h) ≥ Eqφ [ log pθ(~m,~h)− log qφ(~m|~h) ] . (8)
Maximising this bound is not only useful in terms of the marginal likelihood but also equivalently minimises the Kullback-Leibler divergence between the approximate and true posterior DKL[qφ(~m|~h)||p(~m|~h)], giving us more confidence in our inferred posterior.
Factorising the Posterior for Efficient Inference. In order to accelerate learning and make inference quick, we factorise the approximate posterior in the same way as the true posterior:
qφ(~m|~h) = qφ(m1|~h) T∏ t=2 qφ(mt|mt−1,~ht−1:T )
(9)
Thus to estimate the approximate posterior,we use a backwards LSTM to create a summary bt = LSTM(ht:T ). This is then concatenated with the previous memory at time step t − 1 before being passed through a fully connected layer to estimate parameters. Given that the memory can be seen as a sufficient statistic of the past within the model,
it is well known that given this factorisation, conditioning on the memory leads to a RaoBlackwellization of the more general network that would be conditioned on all variables (Krishnan et al., 2017; Alaa & van der Schaar, 2019). This accelerates learning not only by parameter sharing between φ and θ but also by reducing the variance of the gradient estimates during training. With the specified inference network qφ, this leads to an optimisation objective:
F(φ, θ) = Eqφ(~m|~h) [ pθ(~a|~h) ] −DKL ( qφ(m1|~x)||pθ(m1) ) (10)
− T∑ t=2 Eqφ(mt−1|~h) [ DKL ( qφ(mt|mt−1,~ht−1:T )||pθ(mt|mt−1) )] This can optimised in a straightforward manner using stochastic variational inference and reparameterised Monte Carlo samples from the approximate posterior distribution. At each step a sample is drawn from the posterior through which gradients can flow and the ELBO is evaluated. This can be backpropagated through to get gradients with respect to φ and θ which can then be updated as appropriate using an optimiser of the practitioner’s choice.

heading: 5 CASE STUDY: LIVER TRANSPLANTATION ACCEPTANCE DECISIONS
text: In this section we explore the explainability benefits of our method for evaluating real decision making. Given space constraints, we relegate validation of the method on synthetic examples to the appendix, and focus here on a real medical example of accepting liver donation offers for transplantation. This is an important area where much has been done to improve decision support for clinicians (Volk et al., 2015; Berrevoets et al., 2020), but which often focuses on simply suggesting theoretically optimal actions. Here, by trying to understand better the decision making process, we hope to be able to provide more bespoke support in the future.
Organ transplantation is an extreme medical procedure, often considered the last resort for critically ill patients (Merion et al., 2005). Despite the significant risks associated with major surgery, transplantation can offer the chance of a relatively normal life (Starzl et al., 1982) and as a result demand
for donor organs far outstrips supply (Wynn & Alexander, 2011). Yet, perhaps surprisingly, a large proportion of organ offers are turned down by clinicians (in consultation with the receiving patients) for various reasons including donor quality or perceived poor compatibility with the recipient (Volk et al., 2015). This decision faced by clinicians can be (under some simplification) described by the process: 1) A donor organ becomes available and is offered to a patient; 2) The clinician considers the state of the organ, the state of the patient, as well as other information such as the previous reject history of the organ (this is the context X); 3) The clinician makes the binary decision to accept/reject the organ (the intervention A); 4) The patient receives/does not receive the organ and a measure of their outcome is recorded - we measure their survival time as the main outcome of interest (the outcome Y ). Full details of the dataset, preprocessing steps, and experimental setup can be found in the appendix. Note that in this example we are not trying to capture the policy of any individual doctor, but rather the general policy of a treatment centre over time and how what they find important changes - this is due to the form of the data available, but the method could equally be applied to individual clinicians if that information was made available. Additionally, while survival time will not be recorded immediately, the most powerful signal in this case is usually a death that follows very soon after the decision, which will be effectively captured in the data.
Evolving Relative Importances. First, we now examine exactly how our method can explain a nonstationary policy over time. In Figure 5 we plot the relative weights for covariates in terms of the treatment effect for accepting the organ offer as they change at two-year periods. The Model for EndStage Liver Disease (MELD) score (Kamath & Kim, 2007) was introduced in 2002 to give an overall rating of how sick a patient is. On introduction, it was the main consideration when making an acceptance decision, as can be seen clearly in Figure 5. However, over time this became seen as less important as clinicians started to not want to rely on a single score, with some clinicians even turning down offers for patients with high scores in hopes of a better match, with one study showing that 84
% of the patients who died while waiting for an organ had MELD scores ≥ 15 and had previously declined at least one offer (Lai et al., 2012). This can also be seen reflected in Figure 5 as the importance of the MELD score in accepting the offer decreases and the policy stabilises.
Reactionary Policies The most important aspect of our method is that we can model the reaction of the agent to their experience on the fly. For example we can see how observing various outcomes change the opinion of the agent moving forward. This is demonstrated in Figure 6 which plots density estimates of the policy shift (measured as the distance the decision boundary moves) after rejecting the offer and measuring a positive/negative outcome. Here it can be seen that when an offer is rejected then a negative outcome shifts the decision boundary downwards more than a positive outcome, making it less likely to reject an offer in the next time-step.
Matching Clinical Decisions. While it should be emphasised that our method is (to the best of our knowledge) the only method with the aim or capability to model the evolving decision process of the agent, as mentioned in Section 3 there are a variety of methods that are tangentially related. First, we consider a few standard approaches to imitation learning with behavioural cloning (BC) (Bain & Sammut, 1995), where we consider both a linear (BC-Linear) and deep (BC-Deep) functionapproximator, representing both ends of the interpretability spectrum for BC. Continuing in imitation learning, we also compare alongside reward-regularised classification for apprenticeship learning (RCAL) (Piot et al., 2014), a method that regularises the reward implied by the learnt Q-values. We additionally consider an adaptation of counterfactual inverse reinforcement learning (CIRL) (Bica et al., 2021) to the bandit setting where we first estimate the true CATE of the interventions and thus the true potential outcomes. We use these as the reward and apply an optimal policy, which in this one-step case is trivially achieved by choosing the action that maximises the potential outcome.
In Table 2 we compare the predictive power of our method and all benchmarks on the task of imitation learning – i.e. matching the actions of the real demonstrator in some held-out test set. This is the only task for which we can make meaningful comparison with the existing literature, given our divergent goals. Despite the fact that at each step our method only considers a linear decision boundary, we are still able to outperform the deep network architectures used in BC and RCAL in terms of AUC and APS. This can be put down to the flexible way in which the policy can change over time to allow for adaptation. Less surprisingly, we also outperform the stationary linear classifier as well as the version of CIRL, the poor performance of which is probably ex-
plained by the unrealistic nature of the causal assumptions in this setting. Further experimental results on action matching in on multiple additional datasets are provided in the appendix.

heading: 6 DISCUSSION
text: Limitations. By employing a deep architecture our method is able to learn a non-parametric and accurate form of both the non-stationary policy and update function, making far fewer assumptions on their true form than is normal in the literature. This does mean, however, that it works best when training data is abundant, and can otherwise be prone to over-fitting as is common with deep neural networks. Additionally, the assumption that agents are always taking the action they believe will maximise the outcome does mean our method is out-of-the-box unable to account for strategies that deliberately pick actions that could be sub-optimal but may yield useful information (i.e. exploratory strategies). An interesting future direction would include exploring this, potentially by augmenting the outcome to include a measure of expected information.
Societal Impact and Ethics. Being able to better understand decision making processes could be highly useful in combating biases or correcting mistakes made by human decisionmakers. Nonetheless, it should be noted that any method that aims to analyse and explain observed behaviour has the potential to be misused, for example to unfairly attribute incorrect intentions to someone. As such, it is important to emphasise that, as is the nature of inverse problems, we are only giving one plausible explanation for the behaviour of an agent and cannot suggest that this is exactly what has gone on in their mind.
Conclusions. In this paper we have tackled the problem of inverse online learning – that is descriptively modelling the process through which an observed agent adapts their policy over time – by interpreting actions as targeted interventions, or treatments. We drew together policy learning and treatment effect estimation to present a practical algorithm for solving this problem in an interpretable and meaningful manner, before demonstrating how this could be using in the medical domain to analyse the decisions of clinicians when choosing whether to accept offers for transplantation.

heading: A EXPERIMENTAL DETAILS
text: All experiments were performed on a 2016 MacBook Pro, using a 2.9 GHz Dual-Core Intel Core i5 with 8GB of LPDDR3 RAM and no GPU acceleration.
Code was written in PyTorch Paszke et al. (2019) and no external code was used in the implementation of either the model nor the benchmarks. Hyperparameters were selected through grid search over a validation fold of the training data.
All medical data used has undergone a de-identification process and consent was obtained by the relevant curators for their data to be publicly released.
A.1 UNOS DATA
Our data focuses on data from the United Network for Organ Sharing (UNOS), the US non-profit that manages their national waiting list and maintains a database and record of transplantations that occur.
We focus on liver-transplantations, where the donor is deceased. The data is made up of patient details from various centres around the United States. Of the multiple million records of individual transplantations available, once those with missing information on either the donor or recipient are removed we are left with around 500,000 patients spread across 31 centres. Available variables are listed in Table 3
In our action matching experiments we split the data into Train/Validation/Test folds by separating the data by centre. In particular we use centre CTR23901 with 114,314 patients for training; CTR124 with 21,067 patients for validation; and finally CTR279 with 14,295 patients for testing.
For the more exploratory experiments who’s focus is on inference over some data and not extrapolated prediction, we simply train on centre CTR23901, and examine what is inferred about the treatment of those specific patients

heading: B ADDITIONAL VALIDATION
text: While we focused on an important case study in the main paper, we appreciate the need to provide additional evidence that our model can work outside of one specific case example. As such, we now report results on some experiments focusing first on a toy (but relevant) synthetic example, followed by performance results on multiple further real datasets.
B.1 SYNTHETIC EXPERIMENT
We consider a toy example of an agent that is learning online about the treatment effect of the interventions they are performing.
We let X = R5,A = [0, 1], andY = R, with the true outcomes a linear combination of the features given the action (with weights stochastically samples). From a sampled prior set of weights in the agents mind, the agent then interacts, estimating the potential outcomes and choosing an action stochastically according to equation 7 with α = 1, β = 0. After every context, action, outcome triple (x, a, y), the agent adjusts it’s beliefs over the weights of the potential outcome functions using an online gradient descent method:
ωa = ωa − λ((ỹ − y)× x), (11)
with λ some learning rate. Clearly then the policy of the agent is nonstationary and changes over time in reaction to the observed outcomes.
Results. We simulate 10000 trajectories of length 50. When applying IOL to this simulated data, we can achieve an accuracy of 99%, meaning 99% of the time we can accurately determine which of the potential outcomes the agent thins will be higher. This is compared to a stationary estimate of the CATE which can only reach 52% given the fact that the policy is detached from the truth of the treatment effect. Clearly this is an extreme example designed to fail the existing approaches but it does serve to show how stark the contrast is.
B.2 FURTHER PERFORMANCE ON REAL DATA
Alongside the synthetic example above, we also provide further results on real medical data, demonstrating improved performance across a variety of domains. These datasets do not contain a recorded outcome, consequently so we do not include CIRL, and the next state is used as a substitute in IOL. Results are given in Table 2, and a description of the datasets follows:
Intensive Care Unit. The ICU data covers the treatment of 23,106 of patients in the intensive care unit from Amsterdam UMC Elbers (2019). These patients are in a more critical state than those on the general wards while suffering similarly from a variety of conditions and consequently are monitored more frequently, with the database containing around 1 billion clinical observations at varying timesteps down to minute by minute recordings. The data is aggregated into one hour timesteps and models the treatment of the prescription of antibiotics. Included are 24 series features focusing on vital signs including heart rate, blood pressure, and various chemical blood concentration levels.
Cystic Fibrosis. The CF data considers patients enrolled in the UK Cystic Fibrosis Registry TaylorRobinson et al. (2018), coviering around 5,800 patients over the course of multiple years. Measurements were taken on average every 6 months, and the 79 features cover common comorbidities, test results, and infections that arise. The intervention corresponds to the prescription of cortico steroids.
Hospital Wards. The Wards data is based on the care of 6,321 patients at the Ronald Reagan UCLA Medical Center in California who were treated on the general medicine floor between 2013-2016 Alaa et al. (2017). These patients were treated for a variety of conditions including pneumonia, sepsis, and fevers were in general stable and deterioration that required ICU care was rare. Measurements were taken roughly every 4 hours, with average stays lasting 9 days, and include common vital signs such as pulse and blood pressure alongside lab tests and results and 35 temporal features. The action space is taken as a choice of application of oxygen therapy.

Review summaries:
The objective of the paper is an interpretable descriptive model of online learning from trajectories of the form $(x_t,a_t,y_t)$, where $x_t$ is an i.i.d. context, $a_t$ is the action taken that depends on the context and history $(x,a,y)$ up to time $t-1$, and $y_t$ is the outcome resulting from the application of $a_t$ to $x_t$.  The authors' motivating example is liver donation acceptance decisions: a potential donor approaches with an organ they are offering to a particular recipient. The treatment center makes a binary decision to accept or reject the donation based on features of the donor and the recipient. The outcome is the survival time of the patient.  The structure of the authors' model is based on three components: 1. A memory summarization network that maps from the history up to time $t-1$ to the parameters of a multivariate normal distribution over memory vectors. The specific structure is that a new memory vector is produced from the previous memory vector and the observation from the previous time step. 2. An outcome estimation network. This network takes the memory vector as input and produces weights for each action. A dot product of these weights plus the (exogenous) action features yields the agent's estimate of the outcome of each action. 3. The treatment rule is just a logistic function applied to the predicted outcomes.  The model is trained using stochastic variation inference with the ELBO using an LSTM.  Empirically, the authors find that their approach outperforms action prediction baselines such as behavioral cloning, reward-regularised classification for apprenticeship learning and counterfactual inverse reinforcement learning. Additionally, they use their model to analyze shifts in the transplant policy over time, e.g., the declining importance of MELD score and how offers that are rejected make it less likely than a subsequent offer will be rejected.

The paper motivates an "inverse online learning" framework to learn non-stationary policy online. The generative assumption is a deep state-space model and it is assumed that the agent acts optimally according to the current belief in potential outcomes and updates its model based on the true outcomes observed. The critical assumption seems to be that of non-stationarity over the policy function as it responds/interacts with the environment based on updated estimates of its belief over the response function. The parameters of the deep state space model are estimated using the ELBO over the log likelihood of the observed trajectories. Synthetic and real-world validation demonstrates some benefit in terms of "identifying the optimal treatment" based on the true model when acting based off the estimates. 

 This paper proposes an approach for understanding the decision making of a black-box online learning agent from the a dataset of observed trajectories. Given a dataset of trajectories / decisions made by an agent along with initial context and resulting outcome, the paper proposes a method to perform inference over the generative model of the given dataset using variations inference. The results are used to interpret the decision making of an agent that is learning online. The paper demonstrates its usefulness by highlighting insights into factors that may govern decision on a organ donation acceptance decision dataset.