Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies
Authors: Alex Chan
Alicia Curth
Mihaela van der Schaar
Venue: ICLR 2022 Poster (2022)
Keywords: Decision Modelling
Imitation Learning
Inverse Online Learning

Abstract: Human decision making is well known to be imperfect and the ability to analyse such processes individually is crucial when attempting to aid or improve a decision-maker's ability to perform a task, e.g. to alert them to potential biases or oversights on their part. To do so, it is necessary to develop interpretable representations of how agents make decisions and how this process changes over time as the agent learns online in reaction to the accrued experience. To then understand the decision-making processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem. By interpreting actions within a potential outcomes framework, we introduce a meaningful mapping based on agents choosing an action they believe to have the greatest treatment effect. We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them, using a novel architecture built upon an expressive family of deep state-space models. Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time. 

Published as a conference paper at ICLR 2022
INVERSE ONLINE LEARNING: UNDERSTANDING NON- STATIONARY AND REACTIONARY POLICIES
Alex J. Chan, Alicia Curth, and Mihaela van der Schaar University of Cambridge Department of Applied Mathematics and Theoretical Physics Cambridge, UK [ajc340, amc253, mv472) @am. ac.uk
ABSTRACT
Human decision making is well known to be imperfect and the ability to analyse such processes individually crucial when attempting to aid or improve decision- maker's s ability to perform task, e.g. to alert them to potential biases or oversights on their part. To do so, is necessary to develop interpretable representations of how agents make decisions and how this process changes over time the agent learns online in reaction to the accrued experience. To then understand the decision- cast the policy making processes underlying set of observed trajectories, By interpreting inference problem as the inverse to this online learning problem. actions within a potential outcomes framework, we introduce meaningful mapping based on agents choosing an action they believe to have the greatest treatment effect. We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them, using a novel architecture built upon an expressive family of deep state-space models. Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.
INTRODUCTION
Decision modelling is often viewed through the lens of policy learning (Bain & Sammut 1995
Abbeel & Ng 2004), where the aim is to learn some imitation policy that captures the actions (and thus decisions) of an agent in some structured way (Huyük et al. 2021a). Unfortunately, this usually implicitly relies on the assumption that the actions taken are already optimal with respect to some objective, and hence do not change over time. While this might be appropriate to approximate the stationary policy of an autonomous agent, such assumptions often break down when applying them to more flexible agents that might be learning on- on-the-fly. In particular, decision making process of a human decision-maker, whose actions we may see executed over time this is clearly of great use in range of fields including economics and medicine (Hunink et al. 2014| Zavadskas & Turskis 2011). In the context of machine learning, they would be considered to be undergoing a process of online learning (Hoi et al., 2018); training while interacting with the environment, where it is to be expected that beliefs and strategies are constantly adapted based on accumulating experience (Elio & Pelletier 1997). Additionally, such belief updates are often imperfect; for example, people are known to a priori over- the potential impact of rare events when they first start a task, before proceeding to neglect them when they do not occur in their experience (Hertwig & Erev 2009). If our goal is to analyse human mecision-making, it is thus important to be able to model non-stationary policies that are potentially imperfect two senses: both marginally at each time-step, and in the way that they become adapted to new information.
may want to model the
In this work, instead of relying on the assumption that an agent follows a stationary optimal policy, we consider their actions to be consistent within an online leaning framework where they learn from an incoming sequence of examples (Hoi et al. 2018) and update their policies accordingly, as in Figure I Thus, instead of assuming that actions are globally optimal. we assume that at any exact time the agent thought that the action was better than the alternatives. As natural consequence, we frame the problem of decision modelling as that of inverse online learning; that is, we aim uncover

---

Published as a conference paper at ICLR 2022
((a)
Y(a)
Y(a)
t=
t=
Perceived
Treatment Effect
Observations
Truc Response
Surfaces
Perceived
X
Response Surfaces
Figure 1: Capturing online learning behaviour. We assume the agent models the response surfaces in their mind, and that their belief changes each time step in response to the actions they take and the consequent observations they make. The response surface they model need not converge on the true surface since the update method of the agent may be biased and not appropriately use the information available to them.
the process through which agent is learning online. Note that this does not involve coming up with new way to do online learning itself, rather we recover how it appears a given agent may have learnt. We do this by modelling the monn-tationnry) policy of the agent using deep state-space model (Krishnan et al.) 2017). allowing us to track the memory the agent while allowing policies to evolve in non- parametric and non-prescribed fashion. We construct an interpretable mapping from the accrued memory to policies by framing the decision-making process within potential-outcom setting (Rubin 2005); letting actions correspond to interventions (or trealments), and assuming that agents make decisions by choosing an action which they perceive to have the highest (treatment) effect.
Contributions. In this work we make number of contributions: First, we formalise the inverse online learning problem and connection between decision modelling, online learning and po- tential outcomes, highlighting how they complement each other to result in an interpretable un- derstanding of policies (Section 2) Second, we introduce a practical method to estimate the non- stationary policy and update method of an agent given demonstrations (Section 4) And third, we demonstrate how we can uncover useful practical insights into the decision making process of agents with case study on the acceptance of liver donation offers (Section 5) Code is
wwt. group codebase at htt.e:///ggtttn.oooo///eeeeeaaar
made available at http:////.ttto.oooomm//ee////ttt
along with the
2 PROBLEM FORMALISATION
Preliminaries. Assume observe data trajectory of the form D A, Here, X, E X' C R4, denotes a context vector of possible confounders; A, E A 10, binary action intervention; and Y, Y, binary continuous outcome of interest. The subscript indicates a time-ordering of observed triplets; any time-step t is generated by (i) arrival of context x, (ii) an intervention being performed by an agent according to some policy (z(2) and (iii) corresponding outcome y being observed.
Goal. We are interested in recovering the agent's non-stationary policy (π(x) P(A
Perceived Effect
V(a)
1|X x, H, hı), which depends on
the observed context x and can change over time due the observed history H, ((XE, As, Y=) We make the key assump- tion that the agent acts with the intention to choose intervention leading to the best
A =
x
A =
Time
(largest) potential outcome Y (a). but that the perception of the optimal intervention may change over time throughout the agent's ' learn- ing process about their environment. This prob- lem formalisation naturally places us at the in- tersection of treatment effect inference within the Neyman- Rubin potential outcomes (PO) framework
Figure Simplified inverse process. Our task is to infer the evolving perceived effect over time (the blue line) given only realisations of the agent's behaviour.
We assume with single trajectory here, though learning and inference can he applied over dataset of multiple trajectories
2

---

Published as a conference paper at ICLR 2022
(Rubin 2005) with the classical (inverse) reinforcement learning (RL) paradigm (Abbeel & Ng 2004). More specifically though the RL-subproblem of logged contextual bandits (Swaminathan & Joachims 2015 Atan et al. 2018), where the setting is offline and the contexts arrive independently.
Perception Versus Reality. As is standard within the PO framework, we assume that any observation has two potential outcomes (()), associated with each intervention (treatment-level), which are generated from some true distribution inherent to the environment, i.e. (0). (1)) ~ Io (-I 2), yet only the potential outcome associated with the chosen treatment is revealed to the agent. If the agent had full knowledge of the environment, its optimal policy would be to choose an action z], or conversely to choose to perform the
as arg max, jC, (z) where (z) Eg- (a)
intervention only when its expected conditional average treatment effect (CATE) (z)x) (z(()) (6(2)) is positive.
general, we assume that such perfect knowledge of the environment is not available to the agent in many scenarios of practical interest. Instead, we assume that at any time t, the agent acts according to some perceived model P of its environment, which induces the perceived expected potential outcomes jua (I;; 1) and associated treatment effect F(a; t). Here, (a; () Bp (Y (a))X
z, Hr h4], is function not only of the observed context but also the history, such that the agent can update its beliefs according to observed outcomes. Note that we allow the agent's model of the environment to be misspecified; in this case (2;;)) would not converge to n (x) even for very long trajectories. We consider this an important feature, as assume that human decision-maken generally need to make simplifications (Griffiths al. 2015). Formally, we make the following assumption about the agent's decision making process:
Assumption Mutual Observability The observable space X contains only and all the information available to the agent at the point of assigning action.
Assumption 2 Perceived Optimality: An agent assigns the action they think will maximise the outcome. For a deterministic policy Tr, the agent believes Ty (x())) = < (r(x;;) > -oo (z: t). Given stochastic agent, the policy Te (x)) P(A,= = 1 is monotonic increasing function of the agent's 's belief over the treatment effect 7 (1;
Assumption 3 Continual Adaptation: The agent may continually adjust their strategy based on experience. The index in the dataset D represents a temporal ordering which π is non-stationary with respect to, i.e. Ty (z) may not equal n, ( Z where π. represents the policy at time step i.
Assumptions I Tand 2 are generally unverifiable in practice, yet, they are necessary: there would be little opportunity to learn anything meaningful about a non-stationary policy without them. Mutual observability crucial, assuming that different information streams are available to the observer and the agent would make it impossible to accurately describe the decision-making process. Assumption [] albeit untestable, appears very reasonable: the medical setting, for example, violation would imply malicious intent - i.c. that a clinician purposefully chooses the intervention with a sub-optimal expected outcome. Nonetheless, this does mean that we do not consider exploratory agents that are routinely willing to sacrifice outcomes for knowledge, a constraint we do not consider too restrictive within the medical applications we have in mind. Assumption 3 on the other hand, not at all restrictive; on the contrary, it explicitly specifies more flexible framework than is usual in imitation or inverse reinforcement learning as the set of policies describes contains stationary policies.
3 RELATED WORK
Our problem setting is related to, but distinct from, number of problems considered within related literature. Effectively, we assume that we observe logged data generated by learning agent that acts according to its evolving belief over effects of its actions, and aim to solve the inverse problem by mak- ing inferences over said belief given the data. Below, we discuss our relationship to work studying both the forward and the inverse problem of decision modelling, with main points summarised in Table
The agent, whose (non- stationary) policy we aim to understand, actively learns online through repeated interaction with an environment and thus effectively solves online learning problem (Hoi et al.) 2018). The most prominent example of such an agent would be a contextual bandit (Agarwal et al., (2044) which learns assign sequentially arriving contexts to (treatment) arms. Designing agents that update
The Forward Problem: Inferring Prescriptive Models for Behaviour.
3

---

Published as a conference paper at ICLR 2022
policies online is thus the inverse of the problem consider here. Another seemingly related problem within the RL context is learning optimal policies from logged bandit data (Swaminathan 2015): this is different from our setting as it is prescriptive rather than descriptive.
& Joachims
If the goal was to infer the true r" from the (logged) observational data D (instead of the effect perceived by the agent), this would constitute standard static CATE estimation problem, for which many solutions have been proposed in the recent ML literature (Alaa & van der Schaar 2018; Shalit et al., 2017 Künzel et al. 2019). Note that within this literature, the treatment- assignment policy (the so-called propensity score π (2) = PPAA = 1|x = x)) is usually assumed to be stationary and considered a nuisance parameter that is not of primary interest. Estimating CATE can also be pretext task to the problem of developing optimal treatment rules (OTR) (Zhang et al. 2012a Zhao et al. 2012), as g(x)) = (r(x) > 0) is such an optimal rule (Zhang et al., 2012b).
Obtaining a Descriptive Summary of an Agent. When outcomes are observed (unlike in the bandit problem) the task of learning a policy becomes imitation learning (IL), itself large field that aims to match the policy of a demonstrator (Bain & Sammut 1995| Ho & Ermon 2016). Note that, compared to the bandit problem, there is a subtle shift from obtaining policy that is optimal with respect some frue notion of outcome to one that minimises some divergence from the demonstrator. While II. is often used in the forward problem to find an optimal policy thereby implicitly assuming that the demonstrator is themselves acting optimally -, it can also, if done interpretably, be used in the backward problem to very effectively reason about the goals and preferences of the agent (Hüyük et al. 2021a). One way achieve this is through inverse reinforcement learning (IRL). which aims to recover the reward function that is seemingly 2018 Chan & van der Schaar 202I]. This need not be the true reward function, and can be interpreted as potentially more compact way to describe a policy, which also more easily portable given shifts in environment dynamics.
The Backward Problem:
maximised by the agent (Ziebart et al., 2008 |uu et al.)
Our problem formalisation is conceptually closely related to the counterfactual inverse reinforcement learning (CIRL) work of Bica et al. (2021). There the authors similarly aim to explain decisions based on the PO framework, specifically by augmenting max-margin IRL (Abbeel & Ng 2004) approach to parameterise the learnt reward as a weighted sum of counterfactual outcomes. How- ever, CIRL involves pre-processing counterfactual estimation step that focuses on estimating the true treatment effects, and implicitly assumes that these are identical to the ones perceived by the agent (i.e. it assumes that the agent has perfect knowledge of the environment dynamics generating the potential outcomes), marking a significant departure from our (much weaker) as- sumptions. Without the focus on treatment effects, our method could be seen as generalised -on-maraett extension of the inverse contextual bandits of Hüyük et al. 2222 In general, the backward problem is hard to evaluate empirically since information about the true beliefs of agents is not normally available in the data, thus relying on simulation to validate (Chan et al.) 2021).
Table I: Problems considered in related work, noting the relationship between both input and output.
Problem
Input
Target quantity
Online Learning of Optimal Polices (Agarwal et al. 2014
(X,, YD
uout(o (m)
Learning Optimal Policy from Logged Bandits (Swaminsthan & Joachims
2015
(X,, A.,4.
x r (2)
Estimating Heterogeneous Treatment Effects (Alaa & van der Schaar 2018
(X,,,,
(z)
I Learning Imitator Policies via Imitation Learning (Bain & Sammur.
1995
(X,,
Learning Reward Functions via IRL (Abbeel & Ng 2004
(X)
Robs
Learning Reward Functions CIRI. Bica et al.
202
(X1, Y.)
Ro
Modelling Dynamic Policies via Inverse Online Learning
(X,, Ar, Y2)
T (x)
4 INFERRING THE ONLINE LEARNING PROCESS
The Forward Problem. In the forward context, the problem we consider amounts to agent attempting to, at each time-step, take the action they believe will maximise the out- come, without being aware of the true effect of their intervention. We assume that the agent believes the potential outcomes are linear combination of the contextual features such that Aa t), = w)), with wi e Ω a set of weights for the action a at time t, and (.--) de- noting the inner product. Normally we may consider linear model oversimplification, but
Preliminaries:
4

---

Published as a conference paper at ICLR 2022
M
5g
T
Agent
hig.
htg
Environmenu
X1
Y
X2
Y22
Xr
Ap
Figure 3: Model. Solid black lines indicate single time step dependencies. Given covariates X; and an observed history then treatment effects are estimated in the mind of the agent. An action A; is selected based on these effect and an outcome Y; is observed. The dashed grey lines then indicate information flow over time.
when modelling the thought process of agent it appears much more reasonable for example, does a doctor consider higher order terms when weighing up potential outcomes of a patient? seems unlikely, especially clinical guidelines often given linear cutoffs (Burgers al., 2003). The considered forward problem thus proceeds as follows for every given time step l:
context X; ~ Px arrives independently over time. We make no assumptions on the form of Px, in particular noting that there is no need for any level of stationarity.
2. Given X, and the current set of weights wh]. the agent predicts their perceived potential outcomes jan (Xe; I) and (Xx;t;)) l).
3. Action A, is assigned by some probabilistic optimal treatment rule y + [0, 1] such that IP(A; = 1(x,) = ((p1(X;;
()) before the associated true outcome is observed.
4. Based this new information the agent (potentially stochastically) updates their belief over the response surface for the taken action according to some update function f Ω x X X y A(D?)) such that w ,w6 f(wf, wh. X,, A,, Y.).
To maximise outcomes it is crucial to employ update function / that maximally captures the available information at each step of the process. This then allows for an appropriate determination of that aims to maximise the total (potentially future discounted) outcomes.
4.1 A MODEL OF INVERSE ONLINE LEARNING
Here, we focus on the inverse to the problem outlined above, which, to the best of our knowl- edge, has received little to no attention in related work. That is, after observing an agent's actions, can we recover how they arrived at their decisions, and how this changed over time? In the following, we use vector notation Fer {z, with F ëır for all realised vari- ables, with histories hert {(r ai, Mi,y;)) "=) Note that the previously described forward model assumes factorisation of the full generative distribution of the observed data given by:
a = II pluclar, 24) x p(ac( a 7 x p(zz).
(1)
We assume that p(x)) and p(yelag, 24) are governed by some true environment dynam- ics, independent from the perception of the agent and thus not part of our modelling problem. This seems trivially true, as the perception of the agent will neither af- the action being taken.
feet the contexts that arrive the true outcome conditional
Our model for the evolving beliefs of an agent with respect to their interaction with the environment revolves around specialised deep state-space model. Crucially, this takes the form of latent random variable M, E Rdd, which captures the memory of the agent at time t, and can evolve in flexible way based the observed history. We consider the memory to be efficient sum- mary of the history, which can be interpreted as a sufficient statistic for the beliefs of the agent at given time. Naturally, this memory is unobserved and must be learnt in unsupervised
5

---

Published as a conference paper at ICLR 2022
manner. The introduction of this latent variable leads to an extended probabilistic model with more structured factorisation of the conditional distribution of the actions, and a model given by:
T
po po(
= II To (t) x (o(felzs, m.) x So(ma|øin-1. u f1.
(2)
Treatment Rule
Outcome Estimation
Memory Aggregation
This model consists of three components: a memory summarisation network Sp H + A(M), which maps the history into (distribution over) memory; a (perceived) potential outcome predictor network Og X' x M + R2, that takes the memory and context and predicts the perceived potential outcomes; and a treatment rule To : )22 + [o, 1], that given the potential outcomes (summarised by the perceived treatment effect) outputs distribution over actions. Throughout, 0 denotes the full set of parameters of the generative model and notation is shared across compo- nents reflect that they can all be optimised jointly. Below, we discuss each component in turn.
Memory Aggregation. ple and recursive, and assume the memory at given time step be distributed as: M2 ~ ~N,, Σ.) with fr: I M A1-,,
We define the base structure of Sg to be relatively sim-
(3) where fe and Σ, are two output heads of the memory network S given the history 1 It is important note that the memory network does not get the t-th step information (including the context) when predicting the memory at time f - this means the network cannot provide any spe- eific predictive power for given context and is forced to model the general response surfaces.
In many models of behaviour the update function f of the agent is modelled as perfect Bayesian inference (Kaelbling et al., 1998; [olomboo & Hartmann 2017). While this may seems natural on the surface given the logical consistency of Bayesian inference, it seems unlikely in prac- tice that agents (and especially humans) will be capable of it. Not least because in all but the simplest cases it will become intractable for computational agents and humans themselves are well known to make simplifying approximations when dealing with past histories (Genewein et al. 2013). By employing flexible neural architecture, we remove this relatively restrictive assumption on the memory update process. Our model then learns exactly how the agent appears to use previous information to update their predictions allowing us to model common cogni- tive biases such as overly weighting recent or largely negative events (Hertwig & Erev 2009).
Outcome Estimation. Having obtained memory state My, the second stage in- volves predicting the perceived potential outcomes for a context He with the net- work Og(Felzz, mz). Given the forward model where the potential outcomes are con- sidered a linear combination of the features this then proceeds again in three steps: wf,ug MLP(m)),
(4)
(11))) (II, io(21; 1) (p),WW) 7 = (m - ()(1)))).
(5)
(6)
First the memory m, decoded into a set of weights for each action using some standard feed- forward multi-layer perceptron (MLP), before the potential outcomes are predicted by taking the inner product with the context. This is preferable to, for example, concatenating and passing both at and m, through a network since, while it may be potentially more expressive, it loses the connection to the forward model as well as the interpretability that is given by linearity. As it is, while our model of the potential outcomes is linear each time step, the memory allows this to flexibly change throughout the course of the history. The perceived treatment effect is then calculated as the difference between potential outcomes. However, given the linear nature of the predictions, there is no non-degenerate solution for the ws. Thus, to ensure identifiability, we set w/ = O as a baseline.
Treatment Rule. Under Assumption 2 the agent should be expected to take an action that maximises the outcome. There are different possible options for the exact treatment rule, the most obvious being ((a])tt) 1 > 0); assuming that long the treat- ment effect is positive then the intervention will be taken. However, in order to main- tain differentiabilityy as well as allow for more modelling flexibility capture the stoch- asitcity of agents, we parameterise the policy as soft version of the indicator function:
(aalft)) 1 1 +xxppp exp(-a(7t B))

---

Published as a conference paper at ICLR 2022
Assuming positive a, this is clearly monotonically increasing, satisfying Assumption 2. These parameters are learnt by the model and thus allow us to model a flexible threshold with B as well as alcatoric uncertainty in the actions of the agent with
4.2 LEARNING WITH AN INFERENCE NETWORK
As with all flavours of deep state space models, exact posterior inference is analytically intractable, making optimisation of the marginal likelihood equally so. As such, aim to maximise Evidence Lower BOund (ELBO) on this likelihood of the data following standardised variational principles Blle et al., 2017 Hoffman et al. 2013). This is achieved by positing parameterised approxi- mate posterior distribution do (m h). and with a simple application Jensen's 's inequality arrive at:
(8) Maximising this bound is not only useful in terms of the marginal likelihood but also cquivalently minimises the Kullback-Leible divergence between the approximate and true posterior ( (M]))]pp(()))) giving us more confidence in our inferred posterior.
log po (h) 2 Eag [o pa Ri) log (o(m))))
Factorising the Posterior for Efficient Inference. In order to accelerate learning and make infer- ence quick, we factorise the approximate pos- terior in the same way as the true posterior:
T
4s (mil)) (m (h) II
9s (MiiM Pr E,)
(=2
(9)
Thus to estimate the approximate posterior,we use backwards LSTM to create a summary 6, LS.M( This is then concatenated with the previous memory at time step passed through a fully connected layer to estimate pa- rameters. Given that the memory can be seen as sufficient statistic of the past within the model, is well known that given this factorisation, conditioning on the memory leads to a Rao- Blackwellization of the more general network that would be conditioned on all variables (Kr ishnan et al., 2017] Alaa & van der Schaar 2019). This accelerates learning not only by param- eter sharing between Φ and & but also by reducing the variance of the gradient estimates dur- ing training. With the specified inference network 1s. this leads to an optimisation objective:
before being
Figure 4: Structure of the approximate poste- rior inference network.
F( = (m|])) [p)[
Dx (m))m)))))ppl
(10)
(m [DKz (mm Re- (:--:))) (mm(mmmmm))))
This can optimised in straightforward manner using stochastic variational inference and re- parameterised Monte Carlo samples from the approximate posterior distribution. At each step a sample is drawn from the posterior through which gradients flow and the ELBO is evaluated. This can be backpropagated through to get gradients with respect to o and 0 which can then be updated as appropriate using an optimiser of the practitioner's choice.
CASE STUDY: LIVER TRANSPLANTATION ACCEPTANCE DECISIONS
In this section we explore the explainability benefits of our method for evaluating real deci- sion making. Given space constraints, we relegate validation of the method on synthetic ex- amples to the appendix, and focus here on real medical example of accepting liver donation offers for transplantation. This is an important area where much has been done to improve decision support for clinicians (Volk et al.) 2015; Berrevoets et al. 2020), but which often fo- cuses on simply suggesting theoretically optimal actions. Here, by trying to understand better the decision making process, we hope to be able to provide more bespoke support in the future.
7

---

Published as a conference paper at ICLR 2022
2003
2005
2007
Figure Changing Focuses. Predicted weights of different covariates two-year time intervals.
Organ transplantation is an extreme medical procedure, often considered the last resort for critically ill patients (Merion al. 2.05). Despite the significant risks associated with major surgery, trans- plantation can offer the chance of a relatively normal life (Starzl et al. 1982) and as a result demand for donor organs far outstrips supply (Wynn & Alexander 2011). Yet, perhaps surprisingly, large proportion of organ offers are turned down by climicians (in consultation with the receiving patients) for various reasons including donor quality or perceived poor compatibility with the recipient (Volk et al. 2013). This decision faced by clinicians can be (under some simplification) described by the process: I) A donor organ becomes available and offered to patient: 2) The clinician considers the state of the organ, the state of the patient, as well other information such as the previous reject history of the organ (this is the context X); 3) The clinician makes the binary decision to accept/reject the organ (the intervention A); 4) The patient receives/does not receive the organ and a measure of their outcome is recorded we measure their survival time as the main outcome of interest (the outcome Y). Full details of the dataset, preprocessing steps, and experimental setup can be found the appendix. Note that in this example we are not trying to capture the policy of any individual doctor, but rather the general policy of treatment centre they find important changes this is due to the form of the data available, but the method could equally be applied individual clinicians if that information while survival time will not be recorded immediately, the most powerful signal in this case is usually a death that follows very soon after the decision, which will be effectively captured the data.
time and how what
made available. Additionally,
Evolving Relative Importances. First, we now ex- amine exactly how our method can explain non- stationary policy over time. In Figure 5 we plot the relative weights for covariates treatment effect for accepting the organ offer as they change at two-year periods. The Model for End- Stage Liver Disease (MELD) score (Kamath & Kim 2007) was introduced in 2002 to give an overall rat- ing of how sick patient is. On introduction, it was the main consideration when making an accep- tance decision, as can be seen clearly Figure However, over time this became seen as less im- portant as clinicians started not want rely on single score, with some clinicians even turning down offers for patients with high scores hopes of a better match, with one study showing that 84 of the patients who died while waiting for an organ had MELD scores 2 15 and had previ- ously declined at least one offer (Lai et al. 2012). This can also be seen reflected in Figure 5 as the importance of the MELD score in accepting the offer decreases and the policy stabilises.
Outcome Effect
Y-Imc.6
T +.ich
terms of the
0.3
6.1
0.5
-3.0
2.5
Figure 6: One-step policy shift densities after observing an outcome.
Reactionary Policies The most important aspect of our method is that we can model the reac- tion of the agent to their experience on the fly. For example we can see how observing various outcomes change the opinion of the agent moving forward. This is demonstrated in Figure 6 which plots density estimates of the policy shift (measured as the distance the decision bound- ary moves) after rejecting the offer and measuring positive/negative outcome. Here it can be
8

---

Published as a conference paper at ICLR 2022
seen that when an offer is rejected then negative outcome shifts the decision boundary down- wards more than a positive outcome, making it less likely to reject an offer in the next time-step.
Matching Clinical Decisions. While it should be emphasised that our method is (to the best of our knowledge) the only method with the aim or capability to model the evolving decision process of the agent, mentioned in Section 3 there we consider a few standard approaches to imitation learning with behavioural cloning (BC) (Bain) & Sammuut 1995). where we consider both a linear (BC-Linear) and deep (BC-Deep) function- approximator, representing both ends of the interpretability spectrum for BC. Continuing in imitation learning, we also compare alongside reward- reaundddrgllriiedd classification for apprenticeship learning (RCAL) (Piot et al. 2014), a method that regularises the reward implied by the learnt Q-values. We additionally consider an adaptation of counterfactual inverse reinforcement learning (CIRL. (Bica) et al. 2021) to the bandit setting where we first estimate the true CATE of the interventions and thus the true potential outcomes. We use these as the reward and apply an optimal policy, which in this one-step case is trivially achieved by choosing the action that maximises the potential outcome.
variety of methods that are tangentially related. First,
In Table 2 we compare the predictive power of our method and all benchmarks on the task of imitation learning- - i.c. matching the actions of the real demonstrator in some held-out test set. This is the only task for which we can make meaningful comparison with the exist- ing literature, given our divergent goals. De- spite the fact that at each step our method only considers a linear decision boundary, we are still able to outperform the deep network ar- chitectures used in BC and RCAL in terms of AUC and APS. This can be put down to the flexible way in which the policy can change over time to allow for adaptation. Less sur- prisingly, we also outperform the stationary lin-
Table 2: Action prediction performance. Com-
parison of methods on transplant offer acceptances. Performance of the methods is evaluated on the quality of action matching against a held out test set of demonstrations. We report the area under the receiving operator characteristic curve (AUC) and the average precision score (APS).
Method
AUC 1
APS 1
BC-Linear
0.798 + 0.002
0.647 + 0.001
BC-Deep
0.803 = 0.002
0.655 + 0.001
RCAL
0.797 = 0.008
0.662 + 0.003
CIRL
0.553 + 0.008
0.510
ear classifier as well as the version of CIRL, the poor performance of which is probably ex-
IOL (Ours)
0.824 + 0.006
0.677
plained by the unrealistic nature of the causal assumptions in this setting. Further experimen- tal results on action matching in on multiple additional datasets are provided in the appendix.
6 DISCUSSION
Limitations. By employing a deep architecture our method is able to learn a non- non-parametric and accurate form of both the non-stationary policy and update function, making far fewer assump- tions on their true form than is normal in the literature. This does mean, however, that it works best when training data abundant, and can otherwise be prone to over-fitting as common with deep neural networks. Additionally, the assumption that agents always taking the action they believe will maximise the outcome does mean our method is out-of-the-box unable to ac- count for strategies that deliberately pick actions that could be sub-optimal but may yield useful information (i.e. exploratory strategies). An interesting future direction would include explor- ing this, potentially by augmenting the outcome to include a measure of expected information.
Societal Impact and Ethics. Being able to better understand decision making processes could be highly useful in combating biases or correcting mistakes made by human decision- makers. Nonetheless, it should be noted that any method that aims to analyse and ex- plain observed behaviour has the potential incorrect intentions to somcone. As such, it is important to emphasise that, as is the na- ture of inverse problems, we are only giving one plausible explanation for the behaviour of an agent and cannot suggest that this is exactly what has gone on in their mind.
be misused, for example to unfairly attribute
Conclusions. In this paper we have tackled the problem of inverse online learning that de- scriptively modelling the process through which an observed agent adapts their policy over time- by interpreting actions as targeted interventions, or treatments. We drew together policy learning and treatment effect estimation to present practical algorithm for solving this problem in an inter- pretable and meaningful manner, before demonstrating how this could be using in the medical domain to analyse the decisions of clinicians when choosing whether to accept offers for transplantation.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
AJC would like to acknowledge and thank Microsoft Research for its support through its PhD Scholarship Program with the EPSRC. AC gratefully acknowledges funding from As- traZeneca. This work was additionally supported by the Office of Naval Research (ONR) and the NSF (Grant number: 1722516). We would like thank all of the anonymous re- viewers on OpenReview, alongside the many members of the van der Schaar lab, for their in- put, comments, and suggestions various stages that have ultimately improved the manuscript.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learming. pp. 1, 2004.
Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. International Conference on Machine Learning, pp. 1638-1646. PMLR, 2014.
Ahmed Alaa and Mihaela van der Schaar. Limits of estimating heterogeneous treatment effects: Guidelines for practical algorithm design. In International Conference on Machine Learning, pp. 129-138, 2018.
Ahmed Alaa and Mihaela der Schaar. Attentive state-space modeling of disease progression. Proceedings of the 33rd Conference on Neural Information Processing Systems, 2019.
Onur Atan, James Jordon, and Mihaela van der Schaar. Deep-treat: Learning optimal personalized treatments from observational data using neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995.
Jeroen Berrevoets, James Jordon, Ioana Bica, Alexander Gimson, and Mihaela van der Schaar. Organite: Optimal transplant donor organ offering using an individual treatment effect. htps://proceedings. neurips. cc/puper/2020, 33, 2020.
Ioana Bica, Daniel Jarrett, Alihan Hüyük, and Mihaela van der Schaar. Learning "what-if" explana- tions for sequential decision-making, In International Conference on Learning Representations, 2021.
net http://oooummmnhhoeeeeeeeeee
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):55 8777 2017.
Jako S Burgers, Richard PTM Grol, Joost OM Zaat, Teun H Spies, Akke K van der Bij, and Henk GA Mokkink. Characteristics of effective clinical guidelines for general practice. British journal of general practice, 53(486): 2003.
Alex James Chan and Mihaela der Schaar. Scalable bayesian inverse reinforcement learning. In International Conference on Learning Representations, 2021. URL URLpsttt:///peeeeee
Alex James Chan, Ioana Bica, Alihan Hüyük, Daniel Jarrett, and Mihaela van der Schaar. The medkit-learn (ing) environment: Medical decision modelling through simulation. In Thirty-fith Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Matteo Colombo and Stephan Hartmann. Bayesian cognitive science unification, and explanation.
The British Journal for the Philosophy of Science, 68(2):551-444 2017.
Renée Elio and Francis Jeffry Pelletier. Belief change as propositional update. Cognitive Science, 21 (4):419-460) 1997.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforce- ment learning. In International Conference Learning Representations, 2018.
10

---

Published as a conference paper at ICLR 2022
Tim Genewein, Felix Leibfried, Jordi Grau-Moya, and Daniel Alexander Braun. Bounded rationality, abstraction, and hierarchical macision-making: An cnoommttiontthorrelie optimality principle. Frontiers in Roboties and AI, 2:27, 2015.
Thomas Griffiths, Falk Lieder, and Noah D Goodman. Rational use cognitive resources: Levels of analysis between the computational and the algorithmic. Topics in cognitive science, 7(2): 217-229, 2015.
Ralph Hertwig and Ido Erev. The description-exxeriinee gap in risky choice. Trends in cognitive sciences, 13(12)5177553, 2009.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 4572-4580, 2016.
Matthew Hoffman,, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference.
Journal of Machine Learning Research, 14(5), 2013.
Steven CH Hoi, Doyen Sahoo, Jing Lu, and Peilin Zhao. Online learning: A comprehensive survey. arXiv preprint 18X.vv8802.0287/7/ 2018
MG Myriam Hunink, Milton Weinstein, Eve Wittenberg, Michael F Drummond, Joseph Pliskin, John B Wong, and Paul P Glasziou. Decision making in health and medicine: integrafing evidence and values. Cambridge university press, 2014.
Alihan Hüyük, Daniel Jarrett, Cem Tekin, and Mihaela van der Schaar. Explaining by imitating: Understanding decisions by interpretable policy learning. In International Conference on Learning Representations, 2021a. URL
Jkk
Alihan Hüyük, Daniel Jarrett, and Mihacla van der Schaar. Inverse contextual bandits: Learning how behavior evolves over time. arXiv preprint arXiv:2107.06317 2021b.
Leslie Pack Kaelbling, Michael Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134, 1998.
Patrick S Kamath and W Ray Kim. The model for end-stage liver disease (meld). Hepatology, 45(3): 797-805, 2007.
Rahul Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Sören R Künzel, Jasject S Sekhon, Peter J Bickel, and Bin Yu. Metalearners for estimating heteroge- neous treatment effects using machine learning. Proceedings of the national academy of sciences, 116(10)44156-1165 2019.
Jennifer Cindy Lai, Sandy Feng, and John Paul Roberts, An examination of liver offers to candidates on the liver transplant wiit-list. Gastroenterology, 143(5):1261-1265 2012.
Robert M Merion, Douglas E Schaubel, Dawn M Dykstra, Richard B Freeman, Friedrich K Port, and Robert A Wolfe. The survival benefit of liver transplantation. American Journal of Transplantation, 5(2):307-313, 2005.
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted and reward-regularized classification for apprenticeship learning. Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pp. 1249-1256. International Foundation for Autonomous Agents and Multiagent Systems, 2014.
Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469):322-331 2005.
Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: general- ization bounds and algorithms. In International Conference on Machine Learning, pp. 3076 3085. PMLR, 2017.
11

---

Published as a conference paper at ICLR 2022
Thomas E Starzl, Shunzaburo Iwatsuki, David H Van Thiel, J Carlton Gartner, Basil J Zitelli, J Jeffrey Malatack, Robert R Schade, Byers Shaw Jr. Thomas R Hakala, J Thomas Rosenthal, et al. Evolution of liver transplantation. Hepatology. 2(5):6145-633) 1982.
Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. The Journal of Machine Learning Research, 16(1):1731-1555,
2015.
Michael L Volk, Nathan Goodrich, Jennifer C Lai, Christopher Sonnenday, and Kerby Shedden.
Decision support for organ offers in liver transplantation. Liver transplantation, 21(6):784-791, 2015.
James J Wynn and Charles E Alexander. Increasing organ donation and transplantation: the us experience over the past decade. Transplant International, 24(4):324 332, 2011.
Edmundas Kazimieras Zavadskas and Zenonas Turskis. Multiple criteria decision making (medm) methods in economics: an overview. Technological and economic development of economy, 17(2): 397-427, 2011.
Baqun Zhang, Anastasios Tsiatis, Marie Davidian, Min Zhang, and Eric Laber. Estimating optimal treatment regimes from classification perspective. Stat, 1(1):103-114, 2012a.
Baqun Zhang, Anastasios A Tsiatis, Eric B Laber, and Marie Davidian. A robust method for estimating optimal treatment regimes. Biometrics, 101041100)- 2012b.
Yingqi Zhao, Donglin Zeng, A John Rush, and Michael R Kosorok. Estimating individualized
treatment rules using outcome weighted learning. Journal of the American Statistical Association, 107(499):1106-11 2012.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008
12

---

Published as a conference paper at ICLR 2022
APPENDIX
Algorithm 1: Inverse Online Learning (IOL)
Result: Parameters variational distribution and 0 of generative model
Input: D,,,AJJ, learning rate
Initialise o,
while not converged do Evaluate po(mel(4-1))
Calculate forward model next step priors
Evaluate (me h):
> Inference network to get posterior Straight-through MC sample
Sample mn ~ (mi
NLLioss = (m)R) [ā]]]:
K Lioss = DKL a (m1 (T) (mp) + Et= B(DxL (m/ F(ó.0) 0) NL + K Liossi
(d'00) 0') - (0, 17 -0, F(,
D Gradient step for 6,0
0,0 + 6' H'
end
Return: o, 0
A EXPERIMENTAL DETAILS
All experiments were performed on 2016 MacBook Pro, using a 2.9 GHz Dual-Core Intel Core with 8GB of LPDDR3 RAM and no GPU acceleration.
Code was written in PyTorrch Paszke et al. (0199) and no external code was used in the implementation of either the model nor the benchmarks. ters were selected through grid search over a validation fold of the training data.
Hyperparame-
All medical data used has undergone a de-identification process and consent was obtained by the relevant curators for their data to be publicly released.
A.1 UNOS DATA
Our data focuses on data from the United Network for Organ Sharing (UNOS), the US non- non-profit that manages their national waiting list and maintains a database and record of transplantations that occur.
We focus tranattannslannattionn,, where the donor is deceased. The data is made up of patient details from various centres around the United States. Of the multiple million records of individual transplan- tations available, once those with missing information are left with around 500,000 patients spread across 31 centres. Available variables are listed in Table
either the donor recipient are removed we
In our action matching experiments split the data into Train/Validation/TT folds by separating the data by centre. In particular we use centre CTR23901 with 1 114.314 patients for training; CTR124 with 21,067 patients for validation; and finally CTR279 with 14,295 patients for testing.
For the more exploratory experiments who's focus is on inference over some data and not extrapolated prediction, we simply train on centre CTR23901, and examine what is inferred about the treatment of those specific patients
B ADDITIONAL VALIDATION
While we focused on an important case study in the main paper, we appreciate the need to provide additional evidence that model work outside of one specific case exam- ple. As such, we now report results on some experiments focusing first on a toy (but rel- evant) synthetic example, followed by performance results multiple further real datasets.
13

---

Published as a conference paper at ICLR 2022
Table UNOS Variables
AGE
Recipient's age transplant
AGE DON
Donor's s age at transplant
ALBUMIN TX
Recipient's albumin concentration at transplant
ASCITES_ TX
Recipient had ascites at the time of transplant Recipient's BMI
BMI BMI_CALC
COD COD_CAD CAD_DDDOO COLD_ISCH
Cause of death of the donor Cold ischemia time length of time from the donor being recovered until it is transplanted Donor's creatinine concentration
CREAT [REAT_DON DON
CREAT TX
Recipient's creatinine concentration at transplant Whether recipient was on dialysis before transplant Recipient's ethnicity
DIAL_TX ETECAT
ETHCAT_ EXC_HCO
Donor's ethnicity
Whether the recipient received hepatocellular carcinoma exception point
FUNC_STAT_INT FUNC STAT_TRA
Recipient's functional status the time of transplant Whether the recipient has hepatitis C Recipient's INR value at the time transplant
HCV_ -EV_STEOSSAAUS
INR_TX LIFE_SUP_TRRS _RR
Whether the recipient was on life support at the time of transplant
MED HED_COND TRR
Whether the recipient was in an icu, hospital bed, came from home before transplant
MELD PELD LAB SCORE LAB_SCORE
Recipient MELD Whether the donor was a DCD (donation after cardiac death) Number of previous liver transplants the recipient has had Whether the recipient was on a mechanical ventilator before transplant
NON_ H_T__ONTOO
WUM_PREV__LT TX ON_VENT_TRR
PORTAL VEIN TRR
Whether the recipient had portal vein thrombosis at the time of transplant
PREV_ TREV_AR_SURG_ TRR
Whether the recipient had prior abdominal surgery before the liver transplant Whether the recipient had any previous transplant Whether the donor is local, regional, or national Recipient's bilirubin at the time of transplant Whether the graft whole or segmental Donor mechanism of death Donor circumstance of death Recipient's primary cause liver disease Whether the recipient is status la Whether the recipient has hec Whether the recipient is status la
PREV_TX SHARE_TY
TBILI TX
TXLIV
death_mech_don_grrov
deathcirc
diagl
meldstat
statushoc
status1
14

---

Published as a conference paper at ICLR 2022
B.1 SYNTHETIC EXPERIMENT
We consider about the treatment effect
toy example of agent that is learning online the interventions they are performing. [0. 1]. andy
We let X' - = tion of the features given the action (with weights stochastically samples). From a sam- pled prior set of weights in the agents mind, the agent then interacts, estimating the po- tential outcomes and choosing an action stochastically according to equation / with a 3 O. After every context, action, outcome triple (a, a, y). the agent adjusts it's beliefs over the weights of the potential outcome functions using an online gradient descent method:
R. with the true outcomes linear combina-
Wa = wia - x(( y) x),
(1I)
with A some learning rate. stationary and changes over time in reaction to the observed outcomes.
Clearly then the policy of the agent is non-
Results. We simulate 10000 trajectories of length 50. When applying IOL to this simulated data, we can achieve an accuracy of 99%, meaning 99% of the time we can accurately de- termine which of the potential outcomes the agent thins will be higher. This is compared to a stationary estimate of the CATE which can only reach 52% given the fact that the pol- icy is detached from the truth of the treatment effect. Clearly this is an extreme example designed to fail the existing approaches but it does serve to show how stark the contrast is.
B.2 FURTHER PERFORMANCE ON REAL DATA
Alongside the synthetic example above, also provide further results real medical data, demonstrating improved performance across a variety of domains. These datasets do not con- tain recorded outcome, consequently as a substitute in IOL. Results are given in Table 2 and a description of the datasets follows:
do not include CIRL, and the next state is used
Intensive Care Unit. The ICU data covers the treatment of 23, 331066 of patients in the intensive care unit from Amsterdam UMC Elbers (2019). These patients are in more critical state than those on the general wards while suffering similarly from a variety of conditions and consequently more frequently, with the database containing around billion clinical observations at varying timesteps down minute by minute recordings. The data is aggregated into one hour timesteps and models the treatment of the prescription of antibiotics. Included are 24 series features focusing on vital signs including heart rate, blood pressure, and various chemical blood concentration levels.
monitored
Cystic Fibrosis. The CF data considers patients enrolled in the UK Cystic Fibrosis Registry Taylor Robinson et al. (2018), coviering around 5,800 patients over the course multiple years. Measure- ments were taken on average every 6 months, and the 79 features cover common comorbidities, test results, and infections that arise. The intervention corresponds to the prescription cortico steroids.
Hospital Wards. The Wards data is based on the care of 6,321 patients at the Ronald Reagan UCLA Medical Center in California who were treated on the general medicine floor between 2013-2016 Alaa et al. (2017). These patients were treated for variety of conditions including pneumonia, sepsis, and fevers were in general stable and deterioration that required ICU care was rare. Measurements were taken roughly every 4 hours, with average stays lasting days, and include common vital signs such as pulse and blood pressure alongside lab tests and results and 35 temporal features. The action space is taken choice of application of oxygen therapy.
15

---

Published as a conference paper at ICLR 2022
Table 4: Medical examples performance. Comparison of methods on three different medical datasets. Performance of the policy is evaluated on the quality of action matching against held out test of demonstrations. We report the area under the receiving operator characteristic curve (AUC) and average precision score (APS).
ICU
CF
Wards
Metric
AUC
APS
AUC
APS
AUC
APS
BC-L 0.681 0.001
0.655 1 0.000
0.830 + 0.010
0.797
0.009
0.896
0.006
0.853 0.002
BC-D 0.668 1 0.003
0.654 + 0.001
0.952 1 0.010
0.834 1: 0.003
0.845 1 0.010
0.811 1: 0.013
RCAL 0.396 + 0.018
0.563 + 0.007
0.689 + 0.022
0.745 + 0.008
0.942 1: 0.003
0.920 +: 0.004
IOL 0.686 + 0.092
0.631 + 0.065
0.942 + 0.031
0.943 + 0.026|0.9633
+ 0.031 0.941 + 0.055
REFERENCES
Alaa, A. M., Yoon, J., Hu, S., and Van der Schaar, M. (2017). Personalized risk scoring for critical care prognosis using mixtures of gaussian processes. IEEE Transactions on Biomedical Engineering, 65(1):207-218.
Elbers, P. W. G. (2019). AmsterdamUUMcd v1.0.2 ICU database.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga. L., et al. (2019). Pytorch: An imperative style, nigh-performance deep learning library. In Advances in neural informution processing systems, pages 8026-8037.
Raylor-Robinson, D., Archangelidi, O., Carr, S. B., Cosgriff, R., Gunn, E., Keogh, R. H., MacDougall, A., Newsome, S., Schlüter, D. K., Stanojevic, S., et al. (2018). Data resource profile: the uk cystic fibrosis registry. International journal of epidemiology. 47(1):9-106
16


Reviews:
[{"review_id": "vfXVTxvp0iY", "reviewer": "Reviewer_S6Ms", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The framework is a principled approach to the analysis the authors wish to perform in the organ exchange setting (where the linear restriction seems to fit well—maybe? see discussion).\n2. The performance of the model \"looks good\" in the sense that it produces plausible explanations and has a better AUC than the baselines.\n3. The authors seem to be the first to identify this problem as of interest (see discussion about novelty in weaknesses).\n4. The authors' linear decision-making approach with linear weights might make their approach compatible with linear statistical approaches that are compatible with the medical community.\n\nWeaknesses:\n1. In my opinion, the authors' claims about novelty are somewhat overstated. For example, for IRL:\n- Many IRL methods can produce an imitative policy as well as a reward estimate (e.g., Ho and Ermon (which the authors cite) and Fu, Luo and Levine). Particularly relevant to the authors' response is section 7.3 of Fu, Luo and Levine where they evaluate on pure imitation learning tasks, finding performance that is comparable with pure imitation learning methods.\n- IRL can produce non-stationary policies if the state includes the appropriate observations that would induce non-stationarity. In the authors' case, if the history of (X,A,Y) up to time t is included in the state at time T, the policy produced can be non-stationary in the way the authors desire. This is not that different from what the authors end up doing.\n- If the requirement is that a linear decision policy is learned, this can be accomplished by having the neural network produce the action weights as output and then using them as input to the dot product. This is essentially what the authors end up doing.\n2. There are many different design decisions about how this procedure could be carried out—from more macro-level decisions like whether the memory should be stochastic or deterministic, whether to use an RNN-type structure for memory (as the authors have done) vs. a GRU or LSTM, to micro-decisions like hyperparameter selection, whether to use regularization, etc. There isn't a lot of justification that appears in the paper, which makes it hard for subsequent authors to know whether the authors tried other ideas and they failed or if this was just the first pass. The authors perform no ablation testing on model features.\n3. I am somewhat concerned about the limitations of the linear model. There are many simple decision rules that cannot be captured in a linear model—for example, rejecting a patient if a particular feature is too high/low. I think the authors are a little glib about what linear models can cover.\n\nPost-review: I have lowered my review slightly after reading the other reviews. I thank the authors for their response. ", "summary_of_the_paper": "The objective of the paper is an interpretable descriptive model of online learning from trajectories of the form $(x_t,a_t,y_t)$, where $x_t$ is an i.i.d. context, $a_t$ is the action taken that depends on the context and history $(x,a,y)$ up to time $t-1$, and $y_t$ is the outcome resulting from the application of $a_t$ to $x_t$.\n\nThe authors' motivating example is liver donation acceptance decisions: a potential donor approaches with an organ they are offering to a particular recipient. The treatment center makes a binary decision to accept or reject the donation based on features of the donor and the recipient. The outcome is the survival time of the patient.\n\nThe structure of the authors' model is based on three components:\n1. A memory summarization network that maps from the history up to time $t-1$ to the parameters of a multivariate normal distribution over memory vectors. The specific structure is that a new memory vector is produced from the previous memory vector and the observation from the previous time step.\n2. An outcome estimation network. This network takes the memory vector as input and produces weights for each action. A dot product of these weights plus the (exogenous) action features yields the agent's estimate of the outcome of each action.\n3. The treatment rule is just a logistic function applied to the predicted outcomes.\n\nThe model is trained using stochastic variation inference with the ELBO using an LSTM.\n\nEmpirically, the authors find that their approach outperforms action prediction baselines such as behavioral cloning, reward-regularised classification for apprenticeship learning and counterfactual inverse reinforcement learning. Additionally, they use their model to analyze shifts in the transplant policy over time, e.g., the declining importance of MELD score and how offers that are rejected make it less likely than a subsequent offer will be rejected.", "summary_of_the_review": "The paper identifies an interesting problem that takes a different perspective than past work. It covers a lot of related literature but is a bit tunnel-visioned on the problem's perceived newness. The algorithmic approach introduced is a pretty basic tweak on existing methods, but the performance is promising.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rO76V9RRls", "reviewer": "Reviewer_vs7Z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The setting and description is quite confusing. The goal as stated in Sec 2 seems to suggest the problem is about \"inverse reinforcement learning\" while also stating the goal is to learn a non-stationary policy from trajectories as in imitation learning under a causal framework. \n\n2. The paper overall sounds like that of learning non-stationary policies under \"misspecification\" of the counterfactual response functions (linear in covariates/confounders), which is almost inevitable if we go by a parametric way of solving these problems, and that I believe is a much more transparent setup to motivate the problem. But the whole narrative about \"inverse learning\" is extremely confusing and in my opinion redundant. If I am missing something, I would ask authors to clarify. The narrative also has the danger of now confusing IRL as a task where the explicit goal is to summarize beliefs over the rewards and policy learning itself.\n\n3. The optimization and inference is not super well motivated. Why does maximizing likelihood the only and appropriate goal here? It makes sense, because the assumption on how the agent is acting based on the current estimates of response functions. In that sense I don't disagree but I believe the presentation needs to be significantly improved and the cost well motivated. Right now the motivation section and inference seems very disconnected.\n\n4. Also validation using synthetic example seems insufficient. Comparison to a stationary assumption is only one baseline. Isn't it important to also focus on gradient updates being noisy due to the model mis-specification of the response functions? Also the more interesting evaluation seems to be critical to show to over time. \n\n5. Table 2 results are confusing especially comparison to CIRL, what are the \"unrealistic\" assumptions CIRL is making that is being relaxed in this work? In fact it seems a bit weird to me that CIRL and IOL are not performing comparably modulo non-stationarity. If that is the main issue, (and I am not sure because i am not familiar with CIRL), then it doesn't seem to be about causal assumptions. Can authors please clarify their empirical evaluation more clearly?", "summary_of_the_paper": "The paper motivates an \"inverse online learning\" framework to learn non-stationary policy online. The generative assumption is a deep state-space model and it is assumed that the agent acts optimally according to the current belief in potential outcomes and updates its model based on the true outcomes observed. The critical assumption seems to be that of non-stationarity over the policy function as it responds/interacts with the environment based on updated estimates of its belief over the response function. The parameters of the deep state space model are estimated using the ELBO over the log likelihood of the observed trajectories. Synthetic and real-world validation demonstrates some benefit in terms of \"identifying the optimal treatment\" based on the true model when acting based off the estimates. ", "summary_of_the_review": "Overall I believe the presentation of the technical problem is confusing, could be significantly more clear, cost function/inference better connected to the problem and validation improved. I strongly believe the validation should include some form of real misspecification and then evaluate how the learned policy improves over time. \n\n\nPost-discussion update - Thank you to the authors for their clarifications. I have updated my score based on their response.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OFNNHqLcEm", "reviewer": "Reviewer_F1MZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting paper that presents a new and interesting problem. I think the use of generative modeling and inference to gain insight into decision making of an online learning agent is quite innovative. Overall, I think it is an interesting paper. However, I felt some part of the paper could be clearer. \nExample: CATE: conditional average treatment effect. It is not clear if the paper is introducing this terms or borrowing it from existing literature because there is no citation. \nAlso, it seems weird that the perceived utilities of an agent’s decision is given what seems to be a close to a medical term? \n\nAssumption 2 is not clear at all. The first part of assumption2 is clear but the second part \n“Given a stochastic policy,  t(x) is a monotonic increasing function of the agent’s belief over the\ntreatment effect ~  (x; t).\n”\ndoes not seem to be complete statement. What is monotonic? \n\nI felt the experiments in the paper were quite weak. The paper shows that the proposed algorithm outperforms existing imitation learning methods in terms of predictive power. However, what would also have been interesting is to see success/failure of existing imitation learning methods for gaining insight into decision making process of organ donation acceptance decision. Is that possible? If no, why not and if yes, what is the motivation behind the new method. \n\nOne of the claims of the paper is to formulate the problem of inverse online learning. While after reading the paper I can guess what the formulation might be. However, I dont think that the paper states the formulation explicitly. As in given the data, we would like to find X. \n\n\n\t\n\n  ", "summary_of_the_paper": "\nThis paper proposes an approach for understanding the decision making of a black-box online learning agent from the a dataset of observed trajectories. Given a dataset of trajectories / decisions made by an agent along with initial context and resulting outcome, the paper proposes a method to perform inference over the generative model of the given dataset using variations inference. The results are used to interpret the decision making of an agent that is learning online. The paper demonstrates its usefulness by highlighting insights into factors that may govern decision on a organ donation acceptance decision dataset.", "summary_of_the_review": "Overall, I like the paper but it can improve on clarity and experiments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers suggest acceptance of the paper. The authors study an interesting problem (understanding non-stationary and reactionary policies) and propose a solution to the problem which compares favorably to baselines in experiments. However, some of the reviewers also criticize unclarities in the presentation of the paper and the made assumptions. The authors clarified those points quite well in their rebuttal. Further concerns regarded design decisions and the comparison to failure cases of baselines. The authors addressed those in their rebuttal and promised to include corresponding material in their updated paper. Hence I am suggesting acceptance of the paper. Nevertheless, I would like to urge the authors to carefuly revise their problem presentation in the paper in order to improve clarity and add the promised additional insights to the final version of the paper.
Decision: Accept (Poster)