Inverse Contextual Bandits: Learning How Behavior Evolves over Time
Authors: Alihan Hüyük
Daniel Jarrett
Mihaela van der Schaar
Venue: ICLR 2022 Submitted (2022)
Keywords: inverse contextual bandits
understanding decision-making

Abstract: Understanding a decision-maker's priorities by observing their behavior is critical for transparency and accountability in decision processes—such as in healthcare. Though conventional approaches to policy learning almost invariably assume stationarity in behavior, this is hardly true in practice: Medical practice is constantly evolving as clinical professionals fine-tune their knowledge over time. For instance, as the medical community's understanding of organ transplantations has progressed over the years, a pertinent question is: How have actual organ allocation policies been evolving? To give an answer, we desire a policy learning method that provides interpretable representations of decision-making, in particular capturing an agent's non-stationary knowledge of the world, as well as operating in an offline manner. First, we model the evolving behavior of decision-makers in terms of contextual bandits, and formalize the problem of Inverse Contextual Bandits ("ICB''). Second, we propose two concrete algorithms as solutions, learning parametric and non-parametric representations of an agent's behavior. Finally, using both real and simulated data for liver transplantations, we illustrate the applicability and explainability of our method, as well as benchmarking and validating the accuracy of our algorithms.

Under review as a conference paper at ICLR 2022
INVERSE CONTEXTUAL BANDITS: LEARNING HOW BEHAVIOR EVOLVES OVER TIME
Anonymous authors Paper under double-blind review
ABSTRACT
Understanding decision decision-maker's priorities by observing their behavior is critical for transparency and accountability in decision processes-such as in healthcare. Though conventional approaches to policy learning almost invariably assume stationarity in behavior, this is hardly true in practice: Medical practice is constantly evolving as clinical professionals fine-tune their knowledge over time, For instance, as the medical community's understanding of organ transplantations has progressed over the years, pertinent question is: How have actual organ allocation policies been evolving? To give an answer, we desire policy learning method that provides interpretable representations non-stationary knowledge of the world, as well as operating in an offline manner. First, we model the evolving behavior of decision- makerrs in terms of contextual bandits, and formalize the problem of Inverse Contextual Bandits (ICB). Second, we propose two concrete algorithms as solutions, learning parametric and nonpara- metric representations of agent's 's behavior. Finally, using both real and simulated data for liver transplantations, we illustrate the applicability and explainability of our method, as well as benchmarking and validating the accuracy of our algorithms.
recision-making in particular capturing an agent's s
INTRODUCTION
Modeling decision-mane maaonnnnmmnnggga policies is central concern in computational and behavioral science, with key applications in healthcare [1) economics (2). and cognition 33. The business of policy learning to determine an agent's 's decision-making policy given observations of their behavior. Typically, the
objective is either to replicate the behavior of some demonstrator (cf. "imitation learning") 4 S or
to match their performance on the basis of some reward function (cf. "apprenticeship learning" ") 6 7
However, equally important is the pursuit of "descriptive modeling" [8][]] that is, of learning inter- pretable parameterizations of behavior for auditing, quantifying, and understanding decision-making policies. For instance, recent work has studied representing behaviors in terms of rules [10), goals [1].
intentions [12), preferences [3]. subjective dynamics [14), as well as counterfactual outcomes [1]
Evolving Behaviors In this work, ask novel descriptive question: How has the observed behav- ior changed over time? While conventional approaches to policy learning almost invariably assume that decision-making agents are stationary, this is rarely the case in practice: In many settings, behav- learn more about their environment and adjust their policies
iors evolve constantly as accordingly. In fact. disseminating new knowledge from medical research into actual clinical practice is itself a major endeavor in healthcare [16 This research question is new: While capturing "vari- ation in practice" in observed data has been studied in the context of demonstrations containing mixed policies [19 20). multiple tasks [21] (2), and aabgroup-ppatient--peese
preferences (8), little work has attempted to capture variation in practice over time as an agent's 's knowledge of the world evolves.
Example (Organ Allocations) As our core application, consider organ allocation practices for liver transplantations: In the medical community, our understanding of organ transplantations has changed numerous times over past decades (53 Thus immediate question lies in how actual organ allocation practices have changed over the years. Having a data-driven, quantitative, and- importantyyiimerppetaaall description of how practices have evolved is crucial: It would enable colicy-makers to evaluate if the policies they introduced have had the intended impact on practice; this would in turn play substantial role in designing better policies going forward [8] (see Figure [].
To tackle questions of this form, we desire a policy learning method that satisfies three key desiderata:
It should provide (i.) interpretable description of observed behavior (see Appendix

---

Under review as a conference paper at ICLR 2022
It should be able to capture an agent's (ii.) non- stationary knowledge of the world.
Famon
Feest an Importeees
It should operate in (ii..) offline manner- since online experimentation is impossible high- stakes environments such as healthcare.
Fastaa
Inverse Contextual Bandits To accomplish this, we first identify the organ allocation problem contextual bandits problem [26-28]: Given each ar- riving instance of patient and organ features (i.e. the context), an agent makes an allocation decision (i.e. the action), whence some measure of feedback is per- ceived (i.e. internal reward). Crucially, the environ- ment (i.e. precisely, its reward dynamics) is unknown to the agent and must be actively learned, so the agent maintains beliefs about their environment (i.e. inter- nal knowledge). Not only must an agent select actions that "exploit" the knowledge they have, but they must also select actions that "explore" the environment to update their knowledge. Thus the behavior of a learning agent is naturally modeled based on their knowledge, and how to update their knowledge based on the outcomes of their actions.
Time
Nov www.gobbiee//sssaaooo naoohaed to promota A und compptatnt badioa.
The guibifeecc not hare thanked inoecl
Figure Evaluating Impact of Policies via ICB. Suppose that policy- makers introduce a new policy to promote Features A and B as main factors of consideration in decision maaiimgaiinn (whereas clini- cians rely on Feature C at the time), and the policy succeeds establishing Feature A such but fails to do so for Feature B. Using ICB, we can infer the time-varying importances of these features and ob- serve quantitatively that importance of Feature B stayed the same despite the intentions of the new policy. Having made this observation, the policy- makers can now update their policy accordingly.
generalized bandit strategy-that is, how take actions
Now, the forward contextual bandits problem asks the (normative) question: Given an unknown environment, what is an effective bandit strategy that minimizes some notion of regret? By contrast, our focus here is instead on the opposite direction- that is, in formalizing and solving the problem of Inverse Contextual Bandits ("ICB"): We ask the (descriptive) question: Given demonstrated behavior from deciiion-maleee we wish to learn interpretable representations of generalized bandit strategies from the observable context-action pairs generated by those strategies regardless of whether those strategies are effective.
agent, how has the agent's ' knowledge been evolving over time? Precisely,
Contributions Our contributions are three-fold. In the sequel, we first formalize the ICB problem, identifying it with the data-driven objective of inferring an agent's internal reward function along with their internal trajectory of beliefs about the environment (Section (2. Second, we propose two concrete learning algorithms, imposing different specifications regarding the agent's s behavioral strategy: The first parameterizes the agent's s knowledge in terms of Bayesian updates, whereas the second makes the milder specification that the agent's behavior evolves smoothly over time (Section 3). Third, through both simulated and real-world data for liver transplantationss we illustrate how ICB can be applied as an investigative device for recovering and explaining the evolution of organ allocation practices over the years, as well as benchmarking and validating the accuracy of our algorithms (Section 5)
INVERSE CONTEXTUAL BANDITS
Preliminaries Consider Markov decision process of the form ID = (X, A, T). where S indi- cates the state space. A the action space, R E A(R) the reward dynamics, and T E A(X)XxA the transition dynamics. At each time t Z4. the Ie E X and decides to take an action E A, whence an immediate reward r, ~ R(x,, a.) is received, and a subsequent state $141 ~ T( a2) presented. Let the Recision-makinn policy employed by the agent be denoted π E ((A)* x such that actions at any time are sampled according to oe π (xz).
agent is presented with some state
In the forward direction, given dynamics R, T, the reinforcement learning problem ("RL") deals with determining the optimal policy that maximizes some notion of expected cumulative rewards (29] TR. == argmax (A( RR,, E, rt- In the opposite direction, given some observed behavior policy Ty and transition dynamics T. the inverse reinforcement learning problem ("IRL") deals with determining the reward dynamics R with respect to which Tr appears optimal. For instance, the classic max-margin approach seeks [30]: Rd ;= argming (max, E,,RT Σ, rt -E.,RRT Σ, rz).
Having Dynamics Access to Having No Dynamics Access In conventional RL (and IRL), the decision-maker has unrestricted access to environment dynamics-dee either explicitly (i.e. R, T are simply known and used in computing the optimal policy), or implicitly (i.e. the agent may interact freely with the environment during training). In contrast, in setting the agent has such
2

---

Under review as a conference paper at ICLR 2022
luxuries- -not only are the dynamics not known to the agent, but neither do they enjoy distinet sandboxed training phase prior to live deployment. Without such access, the agent must consider both the information they may gain when taking actions (cf. "exploration") and also the expected rewards due to those actions (cf. "exploitation"). Note that this property results in much more difficult learning problems- in both the forward and inverse directions (see Appendix [or more detailed discussion).
Consider environments parameterized by environment parameters E P. such that the reward dynamics of an environment are given by Ry and the transition dynamics by Tp- Let Penv denote the true environment parameter, such that the actual rewards received by an agent are distributed as r. ~ R Ram (2a, at), and the actual states encountered by the agent are distributed as It+1 Tam. (2t, at). Since Panv is unknown, an agent takes actions on the basis of their beliefs about it; these beliefs are described by probability distributions P3 E APP) parameterized by belief parameters B E B. For each time i, let Sr capture the agent's 's belief at the beginning of that step.
Each time step, the agent first samples an environment parameter Pt Ps. according to their belief 3,, then takes an action according to T.... T,, Note that this ensures that each action is taken with probability proportional to the probability with which the agent believes it to be optimal at time step I. Essentially, the agent's policy at time step is induced by their belief 3, TT, (1)) [a] == Ep-Ps [= (z)[a]]. After receiving a reward rt, the agent updates their current belief parameter according (possibly-stochastee belief- upiate function f E ABB BxXxXXXXXX such that 3++1 fB,,, It, Oz, rz). Together with the initial belief parameter 81, the agent's 1-th step belief is a poossibly-stochaste function of the history by = {211t-1, a1:-1, 1,:-1) defined recursively via
2.1 CONTEXTUAL BANDITS SETTING
In this work, we consider state transitions that occur independently of past states and actions. Due to this property, decisions can be made greedily without consideration of what future states may be, and yields a contextual bandits problem (26 28). Note that this captures the organ allocation problem well: Distributions of newly arriving organs are largely independent of prior allocation decisions; in fact, transplantation and allocation policies are often modeled in bandit-like settings 31 32
Formally:
Definition (Contextual Bandits) Consider
caaisiionmmainng problem D == (X, A. R. T), where R. T are unknown to the agent. Let ((x, a) = 'T' for some T E AXX), for all x E X, E A, such that policies are greedy:
(xpp(r;(xx)) arginax,, E. R,p (e).
(I)
where R2( = ER [r] indicates the mean reward function, and ties are broken arbitrarily. Given a space of environment parameterizations P. the contextual bandits problem is design a space of belief parameterizations B, and to determine the optimal belief-update function
E, E.Er,TR.R.T
(2)
Now, suppose that an agent follows a bandit-type policy for T time steps; this would generate an obser- vational dataset of contexts and actions D : {x117 1:7, a1:T) } (adhering to the contextual bandits litera- ture, we refer to states as "contexts" hereafter) ! But since rewards T; and beliefs 3, are quantities in- ternal to the decision makerr we assume that they are not part of the observational dataset: The former represents the agent's 's preferences over outcomes after observing contexts and taking actions, which is not explicitly observable; likewise, the latter represents the agent's 's beliefs about what kinds of out- comes their actions result in, which is also not explicitly observable. Thus we ask the novel question:
From D, can we infer the true environment parameter Pen. as well as the belief trajectory BuT
Definition 2 (Inverse Contextual Bandits) Consider again a contextual bandit problem D = (X,A, R. T), and recall that dynamics R, T are unknown to the agent. Given an observational dataset D. and family of environment parame- terizations P and belief parameterizations B, the inverse contextual bandits problem ("ICB") is to determine the true environment parameter Penv and the belief parameters BET (see Figure
Br
80
I
an
P1
I22
a2
Figure 2. Graphical Model for ICB. We aim to infer the (dotted) red quantities given the (solid) blue ones.
We also use "I E X" to denote contexts (standard in contextual bandits) instead of "s S" (common in RL).
3

---

Under review as a conference paper at ICLR 2022
It is important to observe that the novelty here is general that is, in posing the "inverse" question of how knowledge evolves over time. Focusing our attention on contextual bandits simply represents a specific choice of problem setting: In this first work on tackling the inverse problem in modeling non- stationary agents, it presents a more tractable problem for analysis, and is moreover especially suited for our motivating application in organ allocations. Note that in the bandit setting, the fact that transi- tion dynamics are unknown to the agent is ultimately inconsequential due to greedy policies; we refer to P simply as "reward parameter" hereafter. We leave generalization to arbitrary transition dynamics T E (XX) X to future work. Also note that ICB itself is not bandit problem (see Appendix (
Remark 1 (Environment vs. State Beliefs) When speaking of inferring "beliefs" here in our setting, we are referring to environment beliefss beliefs-thatti is, an agent's s knowledge of the environment's 's rewards. is important distinguish this from state beliefs in partially-observed environments 33 is, an agent's estimate of which latent state they are in. State beliefs are computed by an agent using their (known) environment parameters, whereas environment belief's concern the environment's (unknown) parameters themselves. State belief's can be easily inferred [35) : all factors that contribute to state-belicf updates (i.e. observations and actions) are observable; on the other hand, environment beliefs are more technically challenging due to latent factors (i.e. internal rewards T1:2) that are never observable. Because of the same reason, tackling ICB with conventional 10-HMM inference methods is not possible either, even though the agent can be viewed as an 10-HMM (see Appendix (
that
Remark 2 (Subjective vs. Objective Reward) When speaking of learning what "rewards" an agent is optimizing, the novelty here that our objective refers to the full belief trajectory BI: of the agent. It is important to distinguish this from simply learning the ground-truth parameter Pew as in typical IRL. The ground-truth parameter an objective quantity capturing the "prescriptive" notion of what the agent ought to be optimizing, whereas the belief trajectory is a subjective quantity capturing the "descriptive" notion of what the agent appears be optimizing. Existing work in learning from non- stationary agents has focused solely on pem.. which is all that is required for apprenticeship (7). In contrast, for the goal of understanding behavior (and how it evolves), Definition 2 brings B1. to focus.
3 BAYESIAN INVERSE CONTEXTUAL BANDITS
We operate in the standard contextual bandits setting 26) Consider a d-dimensional action space A, and suppose each time the agent observes k-dimensional context 2,[a] E R for each possible action, thus the context space is given by X == dx Let the space of reward parameters be P= R Rewards are assumed linear with respect to contexts Te, and normally-distribute with mean R2 (r, a) = pp.r(a) (al) and known variance o²: that is, (I, a) == =N((p, x (a)), o²) with (s denoting the inner product. (Note that these assumptions can be relaxed later in Section 3.1 below).
Belief Updates Before tackling a much more general case, we begin by modeling the agent's beliefs as Gaussian posteriors over Penv, and their belief-update function f Bayesian updates. Formally, beliefs are described by the family of distributions P := N (1, E). where 3 == {u, Σ} E IR* x R and A/ (H, Σ) is the multivariate Gaussian distribution with mean vector μ and covariance matrix Σ. At each time t, given the posterior 3, = {mr, Σt} such that Penv//gg ~ N NNi,, Σ,) and the observation (Ia, at, rt). the helief-update [S,, Tt, at, rt) is the Dirac delta centered at 8++1 (1441, E441). ). with:
Pt+1 == Eet1 (x, Me + aa mall) Σ = (I, (0=; + gz The =e [r]]]))))(]))
(3)
Note that the initial belief 31 = {MI E1} represents the agent's 's (unknown) prior over Peay. Finally, to facilitate Bayesian analysis, we consider soft-optimal policies as usual (see e.g. (38)) Formally, instead of picking actions uniformly from argmax.se. RRp, (Iz, at), they are chosen as follows, with E R4 adjusting the stochasticity of action selection (a -> oo recovers the hard-optimal case):
RR, (ara)ar) = Co Setetatt/Eeeee e (t..a)
Bayesian Learning Under this setup, we propose an that estimates the true reward parameter Penv and the initial belief parameter 31, and samples the belief trajectory Be. in alternating steps. First, the joint distribution of all quantities of interest is:
(EM)-like algorithm
P(pc(3) PP (Ie, pt) Pt) Pows It, at) P(Peax B1, TI:TT PI,T, D) = IP I1) P(xx)
(5)
Prior
Thme] Pa. led FR,, (=e[ar) Run (2..0))))))))
4

---

Under review as a conference paper at ICLR 2022
Note that since each belief parameter 3, is deterministic function of the initial belief parameter 31 and the history h, (cf. Equation 3) rewards T1:T-1 and belief parameters 32.r are essentially interchangeable given the initial befief and context-action pairs D (i.e. one is completely identified by the other). Then, starting with some initial estimates Pemv and 31 for the true reward parameter and the initial belief parameter, we iteratively obtain better estimates by performing the following two steps:
Expectation step: Compute the expected log- lillikeeihood of Penr: 91 given current estimates Penv B,:
Q s Bi Penv Bi) = Breer.Purrlnm Purllam. G, ,D llog (rii PLTT, D Plpenes Br)]
(6)
which we shall approximate by sampling reward values and reward parameters {r Pi:T: Pi:Jii1 from the distribution P(rI:T.PI:T (Pmm,31,D) using Markov chain Monte Carlo (MCMC) methods.
(1) AN
Maximization step: Compute new estimates Pe 31 that yield an improved expected lgg-posterior: e(pemm 3 Peav. 31) + logIP(pem. 81) Q(Peav. , B1; Peav. B1) + log PP B1),
(7)
which can be achieved using gradient-based methods.
Sampling Reward Values and Reward Parameters Observe that reward values Fırr are Gaussian- distributed when conditioned on reward parameters pı. (which means they can be sampled exactly):
Let T1:T [ra r.rr]] E Ie' and let X, (1/o²) [21[01] ... ..ffad (T el E R where Ogg is the i-by-j zero matrix. Then we have that F1.T | PILTT Penv, B1, D,,D. ~ N Ni, 2), where
Lemma 1
Σ - + XI
TT ET=xI
(8)
Proof. All proofs are found in Appendix D
Algorithm 1 Bayesian ICB
This motivates a Gibbs-like sampling procedure whereby samples from ((ri:T Pi:T/pew. 31, D) are approximated by sampling Ti.r and P1:r an alter-
Parameters:
variance of rewards o², stochasticity
of action selections o. leamingg rate 1) R, Input: datasct D. prior P P(pem, 91)
3: Initialize Penss 81 P(pen, (8))
nating fashion from their respective conditional distri-
4: loop
butions P(rI.TPPuTT Pemeee 31. D) and P(P T.TT Ponv; B1, D) instead. However, note that sampling reward parameters PIT from its conditional is not as easy to perform exactly as sampling reward val- ues TI: We achieve it by first observing that Pe 's are independent from each other conditioned on r1:r!
Initialize Tig Pi.T
for i E {1, ()) -- N3 do
7: Sample T1. P(rir (: Pcn. A.m. 3,, using Lemma
8:
for t E (1.
9:
Sample p',p" Pa. 1n
10: p&- min (1, Mialleeer
11:
(a) < p' w.p. p. and p" otherwise
P(P1: IITTT Penv: B1,D) X IT P(p (.)
12: end for
13:
end for
14:
(penc. N I 1o0_, log Pir Dlpem.
15: (pem. B.) (pem. 31)
Then, we sample each P: in turn by performing a single iteration of the Metropolis-Hor Haatopolis-Hastings algorithm using Pa. as the proposal distribution. Algorithm |
+n| (P)m..:)) Q(pam: (i) (e)m..0)) (dan.
+nll V (pe... logI(pem Bill lpen (deer
16: end loop
17: Output: Pewss (.9 18 are aurs 1. D.
(Bayesian ICB) summarizes the overall procedure.
3.1 NONPARAMETRIC BAYESIAN INVERSE CONTEXTUAL BANDITS
So far, we have modeled the learning procedure of the agent with Bayesian updates, and estimated the parameters that characterize their learning procedure (i.e. the true reward parameter Peav and the initial belief parameter 31). Clearly, this approach can be similarly applied to different parameterizations of the agent's 's behavior (e.g. using a this section, instead of imposing a particular type of belief-update, we take nonparametric approach and regard the agent's 's beliefs B more generally as random process. First, we establish prior P(G.r) over that belief process, then deseribe a procedure to sample from the posterior (P(@@rr(D).
model, or general function approximator for f). In
Gaussian Process Prior Different from above, now let the agent's beliefs be described by the family of distributions P3 = (8, Σp) for 3 E R where Ep k controls the variability of reward parameters sampled from beliefs. Let BI.7 [B.... ...8T] E R here we consider multivariate Gaussian process as prior over belief parameters BI.::
vec(Br.T) (O, Er EB).
(10)
5

---

Under review as a conference paper at ICLR 2022
where @ denotes the Kronecker product. Σn RTxT controls the covariances between different time steps, and Σg E Ri*k controls the covariances between different components of given belief 3 Although our methodology is applicable for any arbitrary Σr, we shall fix (Er)aj = min(i.j) so our prior becomes a multivariate Wiener process, and differences óe == Be - Br-1 between consecutive beliefs are independent and identically distributed according to (0, E is ((),) & xxp(-16/2pp) 5t). Intuitively, this means our prior favors belief trajectories that vary smoothly over time, where differences between consecutive beliefs are probabilistically bounded by Σg (since larger changes in consecutive 3 's are exponentially less likely in relation to Eg).
Sampling Beliefs from the Posterior Having established a Gaussian process prior P(G1:)) observe that the posterior for 31.T is still a Gaussian process when conditioned on the reward parameters PI:T:
Lemma 2 Let PI: = |p ..- -.n---l E BRXT
Then we have that vec( PI.T, D Ni, E) with i E(T @ Ep) vec(p Σ = ((Er Ea) (T kp)
(11)
Therefore, similar the parametric version of algorithm, we can approximate samples from the pos- terior I( (P) by sampling 31:T and P1:T in an
Algorithm Nonparametric Bayesian ICB
1: Parameters: covariance matrices Ep and En
Input: datasct D
alternating fashion from their respective conditional
3: Initialize p (0) 1/1
distributions P( PI:T: D) and |(:1r D).
4: for E {1, ...... N) do
and discarding the samples for P1:r- Likewise, we sample each Pr in turn by performing a single iter- algorithm. Algo-
5: Sample B P(
(6-1) .) using Lemma
6:
for C 2.. TT do
ation of the Nierropolis-Hasting rithm [2] (Nonparametrri Bayesian ICB) summarizes the overall sampling procedure. See Appendix[[[oon for a discussion of differences between Algorithm and 2.
7:
Sample p', Pat
8:
p min(1, Play 1
9:
P. + w.p. P. and p² otherwise
10:
end for
end
12: Output: (3 1N
4 RELATED WORK
Policy Learning In modeling an agent's decision-making from their observed behavior, the overar- ching goal is often in imitation learning (i.e. to replicate their actions) in apprenticeship learning (i.e to match their performance). The former directly parameterizes imitation policies as black- box functions, typically based on behavioral cloning [1 state-action distribution matching (42 The latter takes the indirect approach of first inferring some reward function for which the agent's demonstrations are assumed be optimal and the basis of which an apprentice policy is trained; this is often done via Bayesian [8 45 46 or max-ent 47 49 IRL. In contrast, our overarching goal is in descriptive modeling (i.e. to learn interpretable parameterizations for explaining observed behavior, see Appendix C for a detailed discussion on our interpretability criteria) For instance, recent work has represented policies in terms of human- unmrannnnnnerttaaalllle rules goals [TI], intentions [2], preferences [33 subjective dynamics [4,, and counterfactuals [15).
g
10
Learning from Variation Precisely, we wish to understand how behavior has changed over time, especially relevant in healthcare as knowledge ( and practices [16118] continuously evolve. While variation in practice has been captured policy learning from multi-modal [9 20), multi-task
21 22) and compound-task [1 demonstrations, little has sought to describe variation over time. Some research has explored IRL from non-stationary agents, using labeled data generated by agents performing specific policy updates over time [36] 37) or ranked post-hoc by expert [52] 53) However, all of these simply attempt to infer the optimal reward (viz. ground-trut Penv) ) for the (prescriptive) goal of apprenticeship. revealing nothing about how the actual behavior has evolved (viz. trajectories 3I:T) for the (descriptive) goal of understanding. The only close attempt has simply used a change-point detection intermediate step to allow inferring a sequence of optimal rewards usual (54)
Bandits Setting To the best of our knowledge, this is the first formal attempt at learning interpretable representations of non-stationary behavior. In formulating and solving this challenge of learning BIT, we have focused on the bandit setting (26-28). Now, the general notion of an "inverse" bandit problem had been independently proposed by (55) and [56] as the bandit-based counterpart to apprenticeship from learning agent (36] 37 52 53 However, they both ignore the presence of contexts (viz.
"non-comtextual" bandits), and more importanly-identee to the apprenticeship works above, they only attempt to infer the ground-truth reward Peny, without regard to the trajectory of evolution itself. Table contextualize ICB with respect to related works in learning from decision-makinn behavior.

---

Under review as a conference paper at ICLR 2022
Table Comparison with Related Work. We aim to provide [I] interpretable account (i.e. seward-base white-box) of [2] non-stationary behavior (i.e. evolving over time) that explicitly captures the [3] trajectory of changes itself (i.e. the agent's s knowledge) in a [4] stepwise fashion (i.e. versus batched intervals), and [5] shares information between consecutive estimates (i.c. 8r, Be+1 are not independent). All works shown operate offline.
Problem Formulation
Asis Variation
Learning Target
Interpretable Non-Station-
Outpat!
ary Agent?
Trajectory Changes
Stepwise Exolution
Information Sharing"
Examples
Polcy
Emitation Learning
48
Apprenticeship Learning
Interpeetable Policy Learinng
Varttton Vatuutinn
Medal Imitation
K
(
Compound- Task Learnnng
K sub-tasks
{m]]
in
Multi-Task Appremicoship
K clusters
(22)
Ranked Reward Extropalation
samples
o²
X
lnvense Soft Policy Ingrovement
inservals
K
Change Points Reward Leaming
M intervals
/
X
Inverse Contextual Bandits
T time steps
(3)
/
(Ours)
ILLUSTRATIVE EXAMPLES
Three aspects of our approach deserve empirical demonstration, and we shall highlight them in turn:
Explainability: ICB should help us understand how medical practice has changed over the years. This is our primary motivation-in providing interpretable representations of variation over time.
Belief Accuracy: ICB should recover accurate beliefs in robust manner across variety of learning agents- that is, it should not be sensitive to the learning procedure of the underlying agent.
Reward Accuracy: ICB should recover accurate ground-truth reward parameters in similarly robust manner- is, should be capable of "extrapolating beyond" suboptimal demonstrations.
Decision Environments We consider data from the Organ Procurement & Transplantation Network ("OPTN")) of December 4, 2020, which consists of patients registered for liver transplantation from 1995 to 2020 [58]. We are interested in the decision-making problem of matching organs that become available with patients who are waiting for a transplantation. For each decision, the action space A consists of patients who were in the waitlist at the time of an organ's 's arrival t, while the context [a]a] for each patient features: (ABO mismatch, age, creatinine, dialysis, INR, life support, bilirubin, weight difference).
A₁ includes the features of both the organ and the patient. We consider k
In addition to the OPTN dataset, to better validate the performance of our method in a more controlled manner, we also devise semi-synthetic decision environment, where the features {a, [a,,aa])ee [e]] A, of potential argan-patient pairs are taken from the OPTN dataset, but the organs are matched with a final patient de E A, for transplantation according to the policies of variety of simulated learning agents. We determine the ground-trus -truth reward parameter Penv of this semi-synthedt environment by perform- ing simple linear regression over the survival time of patients who actually underwent a tramsplantation.
Learning Agents For the semi-synthetic experiments, we generate observational datasets by employ- ing the following simulated agents, which includes stationary agent well six learning agents:
Stationary: The agent knows the ground-truth Pem and takes actions accordingly (i.e. Pt = Peny for all t).
Sampling: The agent employs the posterior sampling-based bandit strategy proposed 26 (cf. Equation 3)
Optimistic: The agent selects actions based on optimistic estimates of their rewards; this is LinUCB in 27]
Greedy: The agent only exploits their knowledge greedily and does not explore their environment effectively.
Stepping: The agent first explores the environment using uniform preferences until some time step L", whence they learn the ground- truth reward parameter Pen and immediately begin taking actions accord- ingly. In healtheare settings, stepping behavior like this may occur when guidelines are introduced.
Lineur: The agent learns the true reward parameter Per in a linear fashion, starting with uniform preferences.
Regressing: The agent first acquires the ground -ruth reward parameter Paw gradually until some time step t", at which point they begin to regress while retaining some amount of knowledge. Note that this is a particularly challenging setting because the regressing agent's behavior does not improve monotonically.
Benchmark Algorithms In addition our two proposed algorithms Bayesian ICB (B-ICB) and Nonparametric Bayesian ICB (NB-ICB), we consider all the applicable algorithms in the literature:
This is the classic Bayesian inverse reinforcement lcarning algorithm that has been widely applied to variety of problem settings 45 S9 60 As usual, it assumes pt Pow for all ( (I....,T).
B-IRL :
7

---

Under review as a conference paper at ICLR 2022
M-fold-IRL: This runs a copy of Bayesian IRL for each of . equal-sized intervals in the dataset. Note that setting M T is equivalent to assuming that beliefs over time are completely independent from each other.
CP-IRL. 54 This recently-proposed method runs M copies of Bayesian IRL well but it employs Shange-point detection algorithm learn best places to divide the dataset (hence referred to as "CP"-IRLA
I-SPI [36] This acently-propost method assumes the agent updates their behavioral policy via "soft policy improvements", which can he "inverted" to recover flo (hence referred to as inverse "SPr"). In our setting. this is equivalent to assuming the agent is less stochastic over time (i.e. larger values of a). However, I-SPI only recovers the ground-trule Pew. and does not provide any estimate for belief trajectories. This recently-proposee method learns Paw using rankings between context-action pairs; absent explicit rankings, it assumes the agent learns monotonically, such that pairs encountered later in time are pre- ferred over earlier ones (i.e. (22, az) < (2e,, ae) if < t'). Like I-SPI, T-REX only infers Paw. not beliefs.
T-REX 52
Marnia
Figure 4. Relative Feature Importances over Time for Creatinine, INR, and Bilirubin. Significant changes in behavior have generally coincided with the important events surrounding guidance on liver allocation policies [6]
Wongl
(a) Fenture Importances in 2000
Couta
In addition to these benchmark algorithms, as a baseline we also report the performance of simply estimating all preferences be uniform- that is, Pem. = Å. = -1 1/k (Baseline). Details regarding both the learn- ing agents and the benchmark algorithms can be found in Appendix
Daysk
ARO
Explainability First, we direct attention to the potential utility of ICB as an investigative device for auditing and quantifying behaviors as they evolve. We use NB-ICB to estimate belief parameters { [/ for liver transplantations in the OPTN dataset. Since the agent's 's re- wards linear combinations of features weighted per their belief parameters, we may naturally interpret the normalized belief param- eters (B)l Σ ==1 [B(( the relative importance of each feature Figure shows the relative importances of all eight
(b) Feature Importances 2010
Figure 3. Relative Feature Im- portances in 2000 and 2010. INR gains significant impor- tance (despite being the least important feature initially), which is consistent with the in- troduction of MELD 2002.
i E {I,...,k. .......,...... J. features in 2000 and 2010, and Figure shows the importance of crea- tinine, INR, and bilirubin-compone ("Model for End-Stage Liver Disease"), a scoring system for assessing the severity of chronic liver disease, [62]. Empirically, three observations immediately stand out: First, INR and creatinine appear to have gained significant importance over the 2000s, despite being the least important features in 2000. Second, their importances appear to have subsequently decreased towards the end the decade. Third, since 2015 their importances appear to have stcadily increased once again.
considered in the MELD score
Interestingly, we can actually verify that these findings perfectly consistent with the medical environments of their respective time periods. First, the MELD scoring system itself was introduced in 2002, which-usiie using INR and creatinine their most heavily weighted components explains the rise in importance of those features in the 2000s. Second, over time there was an increase in the usage of MELD exception points (i.e. patients getting prioritized for special conditions like hepatocellular carcinoma, which are not directly reflected in their laboratory MELD scores), which explains the decrease in relative importance for such MELD components. Third, 2015 saw the introduction of an official cap on the use of MELD exception points (e.g. limited at 34 for hepatocellular carcinoma), which is consistent with the subsequent increase relative importance of those features once again. Figure 4 also plots important historical events that happened regarding liver allocation policies [61]. Of course, ICB has no knowledge of these events during training, so any apparent changes behavior in the figure are discovered solely on the basis of organ- pataint matching data in the OPTN dataset. Intriguingly, the importance of bilirubin appears to have not increased until 2008 instead of earlier when the MELD score was first introduced. Now, there are possible clinical explanations for this: For instance, bilirubin is not weighted as heavily as other features when computing MELD scores, so their importance may not have been apparent until the later years, when patients generally became much sicker (with higher MELD scores overall). In any case, however, the point here is precisely that ICB is investigative device that allows introspectively describing how policies have changed in
8

---

Under review as a conference paper at ICLR 2022
Table 2. Mean Error of Belief Estimates. B-ICB and NB-ICB are best across all types of agents. Note that I-SPI and T-REX are not applicable as they do not estimate belief trajectories. Results show mean + std. error runs).
Learning Agent
Algorithm
Stationary
Sampling
Optimistic
Greedy
Stepping
Linear
Regressing
Baseline
0.477 + 0.000
0.395 0055+00.00 0.030
0.391 - 0.030
0.385
0.020
0.238 0.000
0.238 + 0.000
0.233 + 0.000
B-IRL
0,252 0.248
0.206
0.241 = =0.195
0.248 = 0:204
0.262
0.165 0.001
0.154 = 0.018
5-fold IRL
0.201
0.313=0.0 0.009
0.310 0.033
0.333
0.289
0.292
10-fold IRL.
0.424 12241:00 0.017
0.407
1.007-00.060
0.401 = -0.061
0.399 0.57
0.398 1 0.027
0.366 0.001
0.388 1: 0.043
T-fold IRI.
1,168 0.012
1.134+00.02
1.140 0.011
1.135 0.011
1.117 0.010
1.103 0.001
1.105 0.010
5-fold CP-IRL
0.272
0.000
0.334
0.250 0.029
0.277 1 0.029
0.294 1 0.046
0.282 0.029
0,278 = 0.010
10 fold CP IRL.
0.409
0.036
0.462
0.075
0.388 0.057
0.384
0.055
0.374 0.047
0.376 1: 0.049
0.401 a: 0.035
B-ICB NB-ICB
0.120 0.032
0.140
0.014
0.121
0.016
0.120
0.234 0.036
0.153
0.010
0.147 0.026
0.201 0.027
0.178 0.017
0.152 0.042
0.149 0.036
0.150 + 0.012
0.134 + 0.015
0,140 + 0.017
Table 3. Mean Error of Ground-Truth Reward Estimates. B-ICB is hest across all types of agents. M-fold IRI.,
CP-IRI, and NB-ICB not applicable as they do estimate Prws. Results show mean + std. error (5 runs).
Learning Agent
Algorithm
Stationary
Sampling
Optimistic
Greedy
Stepping
Linear
Regressing
Baseline
0.477 0.000
0.177 0.000
0.177 = =0.000
0.477 # 0.000
0.177 0,000
0.477 0.000
0.477 = 0.000
B-IRL
0,252 1: 0.248
0.224
0.258 0.168
0.266 1 0.182
0.237 1: 0.031
0.266 1: 0.022
0.251 1: 0.020
L.SPI
0.231 0.212
0:206
0.092
0.276 0.208
0.263
+ 0.183
0.237
0.031
0.266 + 0.021
0.250 + 0.020
T-REX
1.482 0 0.062
1,463 0.067
1,480 0.039
1.479 0.060
1.447 1.477:0.00 0.066
1,428 +: 0.089
1.438 = 0 0.983
B-ICB
0.121 0.032
0.149 0.050 0.148 0.036
0.041
0.161 0.027
0.225 0.026
0.246 0.029
this manner-Suce that notable phenomena may be duly investigated with a data-driven starting point (see Appendix C for a discussion on how to interpret behavior with ICB).
Belief Accuracy Since it is not possible to compare the belief parameters of different algorithms directly, we define [|Ep=P [p] EppP' s, [pll|a as the error of belief estimate 3, at time Then, Table the mean error of belief estimates learned by various algorithms in our semi semi-synthetic environment. As we would expect, B-ICB performs the best for the Sampling, Optimistic, and Greedy agents-which learn via Bayesian updates apdates--hile while NB-ICB- it models belief Interestingly, we also observe that both M-fold IRL and CP-IRL perform worse than vanilla IRL. This could be due to the fact that-in both algorithms -he estimates for each interval are trained with fewer data points and independently from the other intervals; that is, there is information sharing and potential similarities between adjacent intervals are disregarded. In contrast, both B-ICB and NB-ICB formalize some relationship between beliefs at different time steps (viz. Equations and T0): under either formulation, beliefs at different time steps are never independent from each other. Finally, is worth noting that NB-ICB is capable of capturing the sudden change in the Stepping agent's 'ss behavior (despite assuming "smoothly" evolving beliefs, see Appendix A for additional experiments).
is more flexible in terms how the best for the Stepping. Linear, and Regressing agents.
Reward Accuracy Defining IP Pemvll Il as the error in estimating the ground-truth parameter Penv. Table 3 shows the mean error for various algorithms. B-ICB performs the best; this shows that not only can it recover (subjective) descriptors of how the agent appears to be behaving over time, but can also infer the (objective) prescriptor of how the agent ought to be behaving ideally. That is, B-ICB can also "extrapolate beyond" suboptimal demonstrations to infer the ground-truth reward. Interestingly, T-REX completely fails: This is because in the contextual bandits setting, it effectively assumes later context-action pairs (i.e. generated by a better policy) on average carn larger rewards than earlier ones (i.e. generated by worse policy) volicy)-butte but this ignores the stochastic arrival of contexts themselves, which (in contextual bandits) is independent of the policy! For instance, sicker patients later time may always yield worse outcomes, matter how much better the allocation policy has become. Appendix A discusses in much more detail why existing methods fail in the ICB setting.
6 CONCLUSION
In this paper, we motivated the importance of learning interpretable representations of non-stationary behavior, formalized the problem of ICB, and proposed algorithms with advantages both in explain- ability and accuracy over any existing methods. Two points deserve brief comment: First, we focused on contextual bandits; while this encapsulates a large class of applications, future work may investigate generalizing this approach to environments with arbitraray T. Second, it is crucial to keep in mind that ICB does not claim to identify the real intentions of an agent: humans are complex, and rationality is bounded. What it does do, is to provide an interpretable explanation of how an agent is effectively behaving, offering a quantitative yardstick by which to investigate hypotheses and understand behavior.

---

Under review as a conference paper at ICLR 2022
REFERENCES
[1] A. Li. S. Jin, L. Zhang, and Y. Jia, "A sequential Decision-theoretic model for medical diagnostic system," Technol. Healthcare, vol. 23, no. 1, pp. 37 37-42, 2015.
[2] A. Clithero, "Response times in economics: Looking through the lens of sequential sampling models," J. Econ. Psychol. vol. 69, pp. 61-86, 2018.
[3] Drugowitsch, R. Moreno-Bote, and A. Pouget, "Relation between belief and performance in perceptual decision making," PLOO ONE, vol. 9, no. 5, 2014.
[4] M. Bain and C. Sammut, "A framework for behavioural cloning," Mach. Intell., vol. 15, pp. 103-129, 1996.
[5] J. Ho and S. Ermon, "Generative adversarial imitation learning," in Proc. 30th Conf. Neural Inf. Process. Syst., 2016.
[6] P. Abbeel and A. Y. Ng, "Apprenticeship learning via inverse reinforcement learning," in Proc. 2 Int. Conf. Mach. Learn., 2004.
[7] D. S. Brown, R. Coleman, R. Srinivasan, and S. Niekum, "Safe imitation learning via fast Bayesian reward inference from preferences," in Proc. 37th Int. Conf. Mach. Learn.), 2020.
[8] D. Jarrett, A. Hüyük, and M. Van Der Schaar, "Inverse decision modeling: Learning interpretable representations of behavior," in Proc. 38th Int. Conf. Mach. Learn., 2021.
[9] T. Bewley, "Am I building white box agent interpreting black box agent?" arXiv preprint arXiv:2007.01887 2020.
[10] T. Bewley, J. Lawry, and A. Richards, "Modelling agent policies with interpretable imitation learning," " in TAILOR Workshop at ECAI, 2020.
[11] T. Zhi-Xuun, J. L. Mann, T. Silver, J. B. Tenenbaum, and V. K. Mansinghka, "Online Bayesian goal inference for noundedly-rationn planning agents," in Proc. 24th Conf. Neural Inf. Process. Syst. 2020.
[12] H. Yau, C. Russell, and S. Hadfield, "What did you think would happen? Explaining agent behaviour through intended outcomes," in ICML Workshop on Extending Explainable Al, 2020.
[13] D. Jarrett and M. van der Schaar, "Inverse active sensing: Modeling and understanding timely decision-making." in Proc. 37th Int. Conf. Mach. Learn., 2020.
[14] A. Hüyük, D. Jarrett, C. Tekin, and M. van der Schaar, "Explaining by imitating: Understanding decisions by interpretable policy learning." Proc. 9th Int. Conf. Learn. Representations, 2021.
[15] I. Bica, D. Jarrett, A. Hüyük, and M. van der Schaar, "Learning what-if explanations for sequential decision-making making." Proc. 9th Int. Conf. Learn. Representations, 2021.
[16] M. Grimshaw and T. Russell, "Achieving health gain through clinical guidelines II: Ensuring guidelines change medical practice," Qual. Health Care, vol. 3, no. 1, pp. 45-52, 1994.
[17] R. Foy, G. MacL_enan, J. Grimshaw, G. Penney, M. Campbell, and R. Grol, "Attributes of clinical recommendations that influence change in practice following audit and feedback," J. Clin. Epidemiology, vol. 55, no. pp. 717-722, 2002.
[18) E. Bradley, M. Schlesinger, T. Webster, D. Baker, and S. K. Inouye, "Translating research into clinical practice: Making change happen." J. Amer. Geriatrics Soc., vol. 52, no. 11, pp. 1875-1882, 2004.
[19] Z. Wang, J. Merel, S. Reed, G. Wayne, N. de Freitas, and N. Heess, "Robust imitation of diverse behaviors," in Proc. s1 Conf. Neural Inf. Process. Syst., 2017.
[20] F.I. Hsiao, J. Kuo, and M. Sun, "Learning a multi-modal policy via imitating demonstrations with mixed behaviors." in Proc. 32nd Conf. Neural Inf. Process. Syst., 2018.
10

---

Under review as a conference paper at ICLR 2022
[21] M. Babes, V. Marivate, and M. L. Littman, "Apprenticeship learning about multiple intentions," in Proc. 28th Int. Conf. Mach. Learn., 2011.
[22] G. Ramponi, A. Likmeta, A. M. Metelli, A. Tirinzoni, and M. Restelli, "Truly batch model-free inverse reinforcement learning about multiple intentions," in Proc. 23rd Int. Conf. Artif. Intell. Statist., 2020.
[23] T. E. Starzl, S. Iwatsuki, D. H. van Thiel, J. C. Gartner, B. J. Zitelli, J. J. Malatack, R. R. Schade, B. W. Shaw Jr., T. R. Hakala, T. Rosenthal, and K. A. Porter, "Evolution of liver transplantation." Hepatology, vol. 2, no. 5, pp. 614S-636S, 1982.
[24] R. Adam, P. McMaster, J. G. O'Grady, D. Castaing, J. L. Klempnauer, N. Jamieson, P. Neuhaus, J. Lerut, M. Salizzoni, S. Pollard, and F. Muhlbacher, "Evolution of liver transplantation in Europe: Report of the European Liver Transplant Registry," Liver Transplantation, vol. 9, no. 12. pp. 1231-1243 2003.
[25] R. Adam, V. Karam, Delvart, O'Grady, D. Mirza, J. Klempnauer, D. Castaing, P. Neuhaus, N. Jamieson, M. Salizzoni, and S. Pollard, "Evolution of indications and results of liver transplantation in Europe. A report from the European Liver Transplant Registery (eltr)," Hepatology, vol. 57, no. 3, pp. 675-688, 2012.
[26] S. Agrawal and N. Goyal, "Thompson sampling for contextual bandits with linear payoffss"" in Proc. 30th Int. Conf. Mach. Learn., 2013, pp. 127-135.
[27] W. Chu, L. Li, L. Reyzin, and R. Schapire, "Contextual bandits with linear payoff functions," in Proc. 14th Conf. Artif. Intell. Statis., 2011.
[28] L. Li, W. Chu, J. Langford, and R. E. Schapire, "A contextual- approach to personalized news article recommendation." in Proc. 19th Int. Conf. World Wide Web, 2010.
MIT press, 2018. [30] A. Ng, J. Russell et al., "Algorithms for inverse reinforcement learning," " Proc. 17th Proc. Ini. Conf. Mach. Learn., 2000.
[29] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction.
[31] Y. S. Tang, A. A. Scheller- Wolf, and S. R. Tayur, "Generalized bandits with learning and queueing in split liver transplantation," Social Sci. Res. Netw,, 2021.
[32] J. Berrevoets, J. Jordon, I. Bica, A. Gimson, and M. van der Schaar, "OrganIT': Optimal transplant donor organ offering using an individual treatment effect," in Proc. 24th Conf. Neural Inf. Process. Syst., 2020.
[33] J. Pineau, G. Gordon, S. Thrun et al., "Point-based value iteration: An anytime algorithm for pomdps." in Proc. 18th Int. Joint Conf. Artif. Intell., 2003.
[34] H. Kurniawati, D. Hsu, and W. Lee, "Sarsop: Efficient point-based pomdp planning by approximating optimally reachable belief spaces," Robot.: Sci. Syst., 2008.
[35] J. Choi and K.-E. Kim, "Inverse reinforcement learning in partially observable environments," J. Mach. Learn. Res., vol. 12, pp. 691-730, 2011.
[36] A. Jacq, M. Geist, A. Paiva, and O. Pietquin, "Learning from learner," in Proc. 36th Int. Conf. Mach. Learn., 2019.
[37] G. Ramponi, G. Drappo, and M. Restelli, "Inverse reinforcement learning from gradient-based learner," arXiv preprint arXiv:2007.072 07,21 2020.
[38] D. Ramachandran and E. Amir, "Bayesian inverse reinforcement learning." " in Proc. 20th Int. Joint Conf. Artif. Intell., 2007.
[39] D. A. Pomerleau, "Efficient training of artificial neural networks for autonomous navigation." Neural Comput., vol. 3, no. 1, pp. 88-97, 1991.
[40] B. Piot, M. Geist, and O. Pietquin, "Boosted and reward reguaa--egularized classification for appren- ticeship learning." " in Proc. 13th Int. Conf. Auton. Agents Multi-agent Syst., 2014.

---

Under review as a conference paper at ICLR 2022
[41] D. Jarrett, I. Bica, and M. van der Schaar, "Strictly batch imitation learning by energy- based distribution matching." " in Proc. 34th Conf. Neural Inf. Process. Syst., 2020.
[42] N. Baram, O. Anschel, and S. Mannor, "Model-based adversarial imitation learning," in Proc. 34th Int. Conf. Mach. Learn., 2017.
[43] W. Jeon, S. Seo, and K.-R. Kim, "A Bayesian approach generative adversarial imitation learning," " in Proc. 32nd Conf. Neural Inf. Process. Syst., 2018.
[44] X. Zhang, Y. Li, Z. Zhang, and Z.-L. Zhang, "I-gail: Learning f-divergence for generative adversarial imitation learning," in Proc. 34th Conf. Neural Inf. Process. Syst., 2020.
[45] Choi and K.-E. Kim, "MAP inference for Bayesian inverse reinforcement learning," in Proc. 25th Conf. Neural Inf. Process. Syst., 2011.
[46] A. Balakrishna, B. Thananjeyan, J. Lee, F. Li, A. Zahed, J. E. Gonzalez, and K. Goldberg, "On-policy robot imitation learning from a converging supervisor," in Conf. Robot. Learn., 2020.
[47] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey, "Maximum entropy inverse reinforce- ment learning," in Proc. 23rd AAAI Conf. Artif. Intell., 2008.
[48] J. Fu, K. Luo, and S. Levine, "Learning robust rewards with adversarial inverse reinforcement learning." " Proc. 6th Int. Conf. Learn. Representations, 2018.
[49] A. H. Qureshi, B. Boots, and M. C. Yip, "Adversarial imitation via variational inverse reinforce- ment learning," Proc. 7th Int. Conf. Learn. Representations, 2019.
[50] M. Sharma, A. Sharma, N. Rhinehart, and K. M. Kitani, "Directed-info GAIL::
learning hierar- chical policies from unsegmented demonstrations using directed information," " in Proc. 7th Int. Conf. Learn. Representations, 2019.
[51] S.-H. Lee and S.-W. Seo, "Learning compound tasks without task-specific knowledge via imitation and self-supervised learning," " in Proc. 37th Int. Conf. Mach. Learn., 2020.
[52] D. S. Brown, W. Goo, P. Nagarajan, and S. Niekum, "Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations," in Proc. 36th Int. Conf. Mach. Learn., 2019.
[53] D. S. Brown and S. Niekum, "Deep Bayesian reward learning from preferences," " in NeurlPS Workshop on Safety and Robustness in Decision-Making, 2019.
[54] A. Likmeta, A. M. Metelli, G. Ramponi, A. Tirinzoni, M. Giuliani, and M. Restelli, "Dealing with multiple experts and non-stationarity in inverse reinforcement learning: An application to real-life problems," Mach. Learn., vol. 110, p. 2541-2576. 2021.
[55] L. Chan, D. in Proc. 14th Annu. ACM/IEEE Int. Conf. Human-RRbol Interact., 2019.
S. Srinivasa, and A. Dragan, "The assistive multi- bandit,"
[56] W. Guo, K. K. Agrawal, A. Grover, V. Muthukumar, and A. Pananjady, "Learning from an ex- ploring demonstrator: Optimal reward estimation for bandits," " arXiv preprint Xix::2006.14866 2021.
[57] D. Lee, S. Srinivasan, and F. Doshi-Velez, "Truly batch apprenticeship learning with deep successor features," " in Proc. 28th Int. Joint Conf. Artif. Intell., 2019.
[58] Organ Procurement and Transplantation Network and the Scientific Registry of Transplant Recipients. Department of Health and Human Services, Health Resources and Services Admin- istration, Healthcare Systems Bureau, Division of Transplantation, Rockville, MD. 2020.
[59] C. A. Rothkopf and C. Dimitrakakis, "Preference elicitation and inverse reinforcement learning." " in Proc. 11th Joint Eur. Conf. Mach. Learn. Knowl. Discovery Databases, 2011.
[60] S. Balakrishnan, Q. P. Nguyen, B. K. H. Low, and H. Soh, "Efficient exploration of reward functions in inverse reinforcement learning via Bayesian optimization," Proc. 34th Conf. Neural Inf. Process. Syst. 2020.
12

---

Under review as a conference paper at ICLR 2022
[61] Organ procurement and transplantation network: Timeline of evolution of liver allocation and distribution policy. Accessed on: Sep. 9, 2021. [Online]. Available: https://optn..ramssgtt hrsa
www.gvvemancceeyy-mmtttteeeeeeee timeleeee
[62] M. Bernardi, S. Gitto, and M. Biselli, "The MELD score in patients awaiting liver transplant: Strengths and weaknesses," Frontiers Liver Transplantation, vol. 54, no. 6, pp. 1297-1306, 2010.
13

---

Under review as a conference paper at ICLR 2022
A DISCUSSION ON EXPERIMENTS
Why existing methods should fail otherwise not apply in setting requires further exposition. First, note that since ICB is a novel problem (which the proposed B-ICB and NB-ICB algorithms are designed to solve directly), algorithms from prior works will in- herently be suboptimal, because they were simply not designed to solve the ICB problem. Concretely, the goal of capturing the evolution of non non-stationary behavior requires the following (see Table D
Applicability of Existing Methods
(a) Trajectory of Changes: Instead of learning single. static ground tround-truth reward parameter, we specifically wish to capture the entire trajectory of changes itself.
(b) Stepwise Evolution: If the agent's 's knowledge evolves continuously over time (i.e. in step-wise fashion), we wish to capture this (vs. a course, interval-wise appraoch).
(c) Information Sharing: For efficiency in learning, the method should be able to share informa- tion between consecutive estimate (i.e. B,, Br+1 are not independent).
No prior work has been designed with all of these consideration in mind. With respect to Table results (i.e. assessing how well evolving knowledge learned):
B-IRL: This only learns a single, static ground-truth reward parameter for the entire trajectory, failing all of (a), (b), and (c), hence would underfit.
M M-fold IRL and CP-IRL: These accomplish (a), but their interval- approach fails both (b) and (c), hence would underfit, and is data inefficient.
T T-fold IRL: This accomplishes (a) and (b), but fails (c) catastrophically by using only one datum to generate each estimate, hence is data inefficient.
With respect to Table (i.e. assessing how well the ground-truth reward is learned), the method should accoutn for non-stationary tehavior-that is, instead of assuming the agent optimal for the learned ground-truth reward the entire time:
B-IRI. This assumes the agent optimal for the learned ground-truth reward the entire time, which is model mismatch.
I-SPI: This accounts for non-stationarity, but makes the specific assumption that the behavioral policy is updated via soft policy improvement steps. In the contextual bandits setting, this is equivalent to the assumption that the agent behaves less stochastically as time passes. This is clearly unreasonably strong assumption, and leads to model mismatch.
T-REX: This makes the specific assumption that context-action pairs encountered later in time are preferred over earlier ones. In the contextual bandits setting, this is equivalent to the assumption that later context-action pairs (i.e. generated by a better policy) on average earn larger rewards than earlier ones (i.e. generated by worse policy). But this ignores the stochastic arrival of contexts themselves, which (in contextual bandits) is independent of the policy. For instance, sicker patients a later time may always yield worse outcomes, no matter how much better the allocation policy has become.
For these reasons, we do not expect existing methods work, for the simple reason that they were not designed to handle the ICB problem. (And the results corroborate our hypotheses).
Modeling Sudden Changes NB-ICB is capable of identifying sudden changes in behavior since it does not assume a particular mechanism with which the non-stationary behavior has came to be. In this section, we aim to demonstrate this capability with additional experiments. Such sudden changes might occur in clinical settings when new guidelines are introduced or existing guidelines are amended-hence is critical for our purposes to be able to model them.
Before the introduction of MELD score, the dominant factor of consideration in allocating organs was the waiting time of patients 163. In fact, MELD was partly introduced to promote the use of features that are more reliable indicators of patient's liver function (namely creatinine, INR, and bilirubin) than waiting time. Now, consider a hypothetical scenario where a MELD-bas is introduced at time t = [T/3] but later withdrawn time t |2T/3] for T 2500. Suppose this causes two sudden changes in clinical practice: (i) Before MELD is introduced, clinicians make decisions based solely waiting times but they start complying perfectly with the MELD-based
policy
14

---

Under review as a conference paper at ICLR 2022
Heams Impprtacns [swes
Grandoouh
Eutirots
Watiig tiwe
IM
Figure 5. Relative feature importances of Waiting time, Creatinine, INR, Bilirubin in the additonal organ allocation scenario as estimated by NB ICB together with ground-trute truth feature importances.
Algorithm
Mean Error
B-IRL
1.158 0.041
5-fold B-IRL
0,591 + 0.025
10-fold B-IRL
0,422 0.026
T-fold B-IRL
1.661 0.006
5-fold CP-IRL
0.310 0.032
10-fold CP-IRL
0.369 0.060
NB-ICB
0.308 0.024
Table 4. Mean error of belief estimates for the additional organ allocation scenario.
policy once it is introduced. (ii) Later when the MELD-based based policy withdrawn, clinicians revert back to following their original policy based on waiting times again.
In this hypothetical scenario, the context x r|a] for a potential organ- organ-patient match consists of four features: (waiting time, creatinine, INR, bilirubin)- Like in semi-synthetie experiments, sample these features from the real OPTN data for liver transplantations. For t E [T/3, 27 27/3) (i.e. when the MELD-based policy is in place), Pt. is such that R,, (z., a) 9.57 log(creatinine) 11.2 log (NR) + 3.78 (oglbibiuu),n) which matches with the real feature weights used in MELD score Otherwise, Pt is such that R,p, (Ir, a) (waiting time).
Figure 5 qualitatively shows that NB-ICB is able to identify the sudden changes caused by the introduction and later the withdrawal of the MELD-based policy. Quantitatively. Table show that NB-ICB is still the most accurate algorithm in estimating belief parameters.
B DISCUSSION ON NOVELTY AND SETTING
Problem Setup Consider a Markov decision process D = (X, R. as defined in Section 2 the most general case, the agent has access to environment dynamics R, T, so some degree of learning must occur. Let index the cumulative number interactions with the environment; if the environment is episodic, then time steps accumulate across resets.
In
Now, let f denote a probabilistic, online algorithm that learns some parameter of the MDP; for instance, consider this to be a parameterization of the environment dynamics (but this formalism also applies to learning value functions or black-box policies). And let 3 denote the agent's 's (-th step knowledge of the quantity being learned; for instance, this would be parameterization of their probabilistic belief about the environment dynamics. Note that each 3 naturally induces a policy Tpt
[x) == Ep--P, [T R, [a]]
(12)
where R, denotes the optimal policy that corresponds to point-val volut- knowledge that the envi- ronment dynamics exactly Rp, Tp- Finally, let T be the time step when the algorithm terminates (e.g. due to some measure of convergence). Broadly. T partitions the agent's behavior into two distinct phases:
15

---

Under review as a conference paper at ICLR 2022
Training phase: This is where f progressively updates the agent's knowledge by interacting with the environment. Within this phase, we may define the train-time time expected return:
train (J) = Σ Blr, R13,, S. T, m
(13)
Testing phase: Now after training, the agent ends up with final fr, and may then be deployed in the environment. Within this phase, we may define the test-time expected return:
VR
Σ
Elra
R| T, a])
(14)
T>T
Usually, if some knowledge 31 were learned in a prior training phase, the agent would naturally set π. = TTr in the testing phase.
Forward Problem In the forward direction, agent seeks to determine the optimal policy that maximizes some notion of expected return. Generally, there are two distinct classes of problems that of interest: The first assume that agent has access to environment dynamics (either explicitly, via direct knowledge of R. T: or implicitly, via interaction with the environment in an unassessed training phase). The second assumes the agent has no such access environment dynamics (therefore learning must occur via interaction with the environment in an assessed training phase).
Problem 1A (Agent has dynamics access): In this setting, either the dynamics are explicitly known, so no training phase is required (the agent can simply perform planning using dynamic programming, or simulation-based search, etc.), or the agent may freely interact with the envi- ronment in an unasses training phase. Either way, train is irrelevant; the agent simply seeks the following:
VR argmax (os((()) (TT)
(15)
Problem 1B (Agent has no dynamics access): In this setting, the training phase is assessed as well-thar is, not only does the agent seek the optimal policy for test-time deployment, but they also care about the return generated during the training phase. On the flip side, note that once
terminates, VR is trivially minimized for any T by setting π = T.r. Thus the agent seeks the following:
arginaxy ViR (f)
(16)
Crucially, in the former, the agent is optimizing कt" based on an "exploitation-only" metric. Whereas in the latter, the agent is required to select f that balances "exploitation" and "exploration".
Inverse Problem Now, suppose we-as as investigators- are given an observational dataset D = (œI (1ir); ) of states and actions generated by some agent. The inverse problem deals with deter- mining the true environment parameter Penv and/or the agent's 's belief parameters BI.r, with respect to which the agent's behavior appears most optimal. This often treated with Bayesian formulation:
Problem 2A (Agent has dynamics access): In this setting, we seek the following:
Pons == argmax, P(D|z, Tp argmax, V Rr (T))
(17)
Problem 2B (Agent has no dynamics access): In this setting, we seek the following:
Pene, Bur = argmax,,ss p, P(D) 11, ~ argmax, ()), ~ Tp, ar ~ BI.r).
(18)
Note that Problem 2A is equivalent to generalization of Bayesian inverse reinforcement learning 38 More broadly, as the "opposite" to Problem IA, different flavors inverse reinforcement learning all share the property that they aooolve-eppiccityy or implicitly- the quantity V. For instance, (arnaxx,VVei (ViS (TD) max, Vi (r))
max-ma inverse reinforcement learning 30 seeks Penv argmax,
On the other hand, Problem 2B is novel, and has not been studied before. Now, as an initial exploration of this problem setting, in this work we consider state transitions that occur independently of past states and actions (which is especially suited for our motivating application to organ allocations). In the forward sense, this corresponds special case of Problem 1B that yields the "contextual bandits"
16

---

Under review as a conference paper at ICLR 2022
Forward Problem
Inverse Problem
Agent has dynamics access
Problem 1A
Problem 2A
(i.c, agent optimizes for exploitation only)
(c.g. any optimal control)
(c.g. Bayesiaan IRL 88
Agent has no dynamics access
Problem 1B
Problem 2B
(i.c. agent balances exploration and exploitation)
(c.g. contextual bandits)
(e.g. Bayesian ICB (Ours)
Table 5. Classification of problems in sequential decision meakii.n-mmaiin.
setting; this means that decisions can be made greedily: s (x)) argmax, (2 a), with ties broken arbitrarily. In the inverse sense, this corresponds to a special case of Problem 2B that yields the "inverse contextual bandits" setting; this means that 7 being unknown is ultimately inconsequential, and may simply treat p the "reward parameter".
Importantly, inverse contextual bandits is nor-in any shape or form- furm-subsumee by inverse reinforce- ment learning. Note that B-IRL is an instantiation of Problem 2A, whereas B-ICB is instantiation of Problem 2B. Table summarizes the above discussion.
C FURTHER DISCUSSIONS
Interpretability In terms of interpretabilityy, we taking the standard view that interpretable description of behavior should locate the factors that contribute to individual decision, in language readily understood by domain experts [64]. With respect non-stationary behavior, this concretely manifests in three aspects:
(1) Feature Importance: As is often the case for interpreting supervised learning as well, in representing decision-making a basic desideratum is that the relative importances of inputs be quantified. We argue that linear weights-which which we are-are inherently interpretable.
(2) Task-level Description: Specifically for modeling decision-making behavior, however, clearly a task-level description (viz. reward function) more concisely interpretable account of the behavior than standard input-output feature sensitivities for a black-bor -box policy model.
Finally, the most important factor in our setting that we specifi- cally wish to describe non non-stationary behavior. Note that none of the usual approaches to post-hoc or ante-hoc machine learning interpretability can readily accommodate this unique characteristic.
(3) Non-stationary Behavior:
Given these considerations, let us look at standard approaches to modeling classifiers and/or policies:
Input-oupput feature importance (e.g. (65 This satisfies criterion (1), but not (2) or (3).
Counterfactual explanations (e.g. 166 ): This satisfies criterion (1), but not (2) or (3).
Inverse reinforcement learning (e.g. 67 ): This satisfies criteria (1) and (2), but not (3).
Compound-task imitation learning (e.g.
This satisfies criteria (3), but not (1) or (2).
Learning from learner: In addition, while I-SPI 36 and T-REX 152 superficially handle non-stationary behavior, they actually only seek to recover the ground-truth reward from such behavior (i.e. Penv. and do not attempt describe the evolution of such behavior over time we do (i.e. Br)).
In contrast, this is precisely why we propose ICB, which satisfies all three criteria: ICB explicitly seeks to describe how behavior changes over time by learning ......, 3r. which are beliefs over task-level reward functions that give insight into feature importances in the behavior.
It should be emphasized that the primary innovation here is not in proposing any new definition of interpretabilitty. Rather, the novelty is in proposing formulation and solution to ICB (more generally to Problem 2B in Appendix [ that retains all the advantages of conventional IRI, while importantly-geneeraiizz
to the case where we learn from the non-stationary behavior an agent with no access to environment dynamics. Of course, we can always naively adapt IRL methods to satisfy criterion (3)- that is, by conducting IRL within discrete intervals within the data: This is simply M-fold IRL, and CP-IRL (54), and does not take advantage of the fact that beliefs over time are likely correlated. As corroborated by our experiments, these methods do not appear effective, likely due the loss of data efficiency from lack of information sharing across time.
17

---

Under review as a conference paper at ICLR 2022
Interpreting Behavior with ICB Suppose we were policy-makee makers, auditors in any case, sup- pose we had a prior hypothesis about how behaviors may/should have changed over time. For instance, consider that we introduced a new policy at some time in the past, and would like to verify that it had the intended effect on actual clinical practice. As the OPTN experiment demonstrates, ICB gives a tool for estimating answer to this question.
In doing so external context (i.e. knoeledge of the times new policies were introduced, or otherwise knowledge of the times which we hypothesize there be behavior changes) is crucial. In partieular, they should be known beforehand. Simply running ICB on a behavioral dataset, and then trying to fish for and explain any trends found, would not be valid use of the method.
Finally, note that ICB does not claim to identify the real intentions of an agent: Humans are complex, and rationality is bounded. What it does do, to provide representation of how an agent is effectively behaving, offering a quantitative emthod to investigate hypotheses.
Algorithm vs. Algorithm 2 Algorithm models the learning procedure of the underlying agent with Bayesian updates and learns both an estimate of the ground-trute reward, as well as the trajectory of beliefs. Whereas, Algorithm models the agent's 's trajectory of beliefs more generally as a random process but only learns the trajectory of beliefs, without estimating the ground-truth reward. Hence, applying and interpreting the results of each approach depends on the assumptions and specific use case.
For instance, Algorithm I makes stronger assumptions, but gives an estimate of Penv. Since this estimate tells what the behavioral policy may appear to be optimizing (but ís not necessarily perfectly successful at optimizing yet), the estimate can potentially be used to set a new explicit if followed successfullyy- wouldd yield new policies that outperform the existing dataset. Conversely, Algorithm 2 makes weaker assumptions, but the tradeoff is that only a description of the evolutio nof knowledge is recovered, without an estimate of the ground-turth reward for downstream use.
Forward Agent as an 10-HMM The agent in ICB can be viewed as an Input-Oupput HMM (IO- HMM): At any point, the "hidden state" of the model is the agent's 's belief 3,, the "input" to the model the tuple (ze, ae. rt). the "transition function" is the belief-update function f. and the "output" of the model is the reward parameter Pt. In fact, this is depicted in Figure 2 (see the arrows on the top half of the figure).
However, there is a crucial difference: In this I0-HMM, we do not even observe its "outputs" (i.c. pt). and even its "inputs" are only partially-observed (i.e. r²). So, learning much more challenging- be it the 10-HMM parameter Penv. or the hidden state sequence $1,..., 3r. Thus even if we chose to cast this problem into a generic IO-HMM framework, would still be the case that conventional 10-HMM learning techniques are not applicable.
Now, the reason learning is still possible, is that we have additional prior knowledge that relates the "inputs" (Ie, dz. to the "outputs" p. such that can treat them as latent variables for inference. These relationships are also depicted in Figure 2 (see the arrows on the bottom half of the figure). And the algorithms we propose precisely leverage these prior relationships for learning.
Is ICB itself a bandit algorithm that acts in the same domain? The answer is no. There is only ever a single bandit D == (X, A, R, T) considered in setting, which the underlying agent interacts with through a single sequence of actions. There is also only ever a single environment (identified by the tuple R, T), which the agent starts off having no knowledge about. The agent employs a single belief-update function f to maintain their internal heliefs about the environment each time step. Each time step. the agent is presented with a new context I T, and takes an action a ~ (I) based on their policy Tg.
From the perspective of the agent, the forward problem of coming up with a good belief-update func- tion ff to maximize rewards in expectation over the entire (single) sequence of interactions is a bandit problem, precisely the "contextual bandit" problem (see Definition T). From the perspective of the investigator, the inverse problem is simply learning Penv, 31.7 from D. which is the (single) observed sequence of contexts and actions (i.e. "inverse contextual bandits") belongs to an entirely different class problems. Note that, unlike the agent, we as investigators do not interact the environment, observe sequential rewards, nor maintain and update beliefs of any form.
18

---

Under review as a conference paper at ICLR 2022
D PROOFS OF LEMMAS
Proof of Lemma First note that μr = Ex(2i MI + +X-11rr by definition. Then, we have
P(T1:T) PI:T Peaes D)
P(rIIT PLTT. Penu, B1, D)
P (r Penv; Mt, a4)
II (P(p: (3c)
t=2
T
II
Rew (2r, at) ar)( x II Pa..U lAn]
x
T NN(o>X 1) x II N(an Led
x
2=
1
N NNo Peem: 0² (T))rr))))
II exp
(pr ix)) 5T (p= (1x)
2
1=2
N N(22 X Teew:
a² (T())
T
1
II
- (pr (pt- --,, - in 11 T. 52
x exp
(p: ΣイΣ A1 E(
ON X Pem. [2][[r]]]]
x II exp
(-1 1
(-2 (FT.-I , ExX 2(p: - - X (
2=
N N Penv. 0² o²1)[r1:]]
T
FI:T - -x
x
exp
2
1=
.XT TT (FITT (X -X ST
N X Pens -² [[(rrr]]
T
II
- [Mir]]
x N XT 2 [[(TT]]
r
T
Σ
XL, Exx-1)
- (1)).
x
2
1=
IXI Exxx1 mral
1=2
NN ((1) I+ Σ XI x
1
(XT
XT (p: E
T
1
X 1xXe_1) [rIr]-
I+
1=2
Proof of Lemma We have
P(BI.TlPI:T:D D) x PP Pi:r. D)
PP(B.T)) PP D)
19

---

Under review as a conference paper at ICLR 2022
(0, Er @ Eg) [x)[ve((33:)) I o (occ))
x. N (0, @ (ee (
x N (
(ET 2 )B) (1 (10))))))) [vec(G1.r)]
E EXPERIMENTAL DETAILS
Decision Environments There are 308,912 patients in the OPTN dataset who either were waiting for liver transplantation underwent liver transplantation. Among these patients, we have filtered out the ones who never underwent a transplantation, the ones who were under the age of 18 or had donor who was under the age of 18, and the ones who had missing value for ABO Mismatch, Creatinine, Dialysis, INR, Life Support, Bilirubin, or Weight Difference. This filtering has left us with 31,059 patients, each corresponding we have sampled 2500 donor arrivals uniformly at random, and for each arrival, in addition to the patient who received the donor organ, sampled two more patients who were in the waitlist for a transplantation at the time of donor's s arrival in order to form an action space. For the simulated experiments, we have sampled 500 donor arrivals and two patients for each arrival uniformly at random.
different donor arrival. For the real-data experiments,
Learning Agents
Sampling: We set = 0.10.
Optimistic: The agent forms the same posteriors as Sampling but selects actions such that ag argmax,c 14 [2 + [a] [a]a"]E,, [,2,[a].
The agent forms the same posterior as Sampling but acts based on the mean reward parameters Pt. = Ep= [p].
Greedy:
Stepping: Formally, this agent is given by Pt = 11/h for t E (I....,t*) and Pt = Penv for E {t" 11.... 7).
We set = T/2.
Linear: Formally, this agent is given by Pt = 1/T Penv + (1 - C/T) ) (-11kk).
Regressing: Formally, this agent is given by Pt 1/2- Peny + (1 -/1x)) p for L (1.....""
and pt ((-t")/(T=-))>)) + ((rr)/((-()))) ) Penv for E {1. 11, T}, where p - 1/k and p² 7Pa + (1 - r)p Here, 7 E [0, 1] controls the amount of knowledge that is retained. We set t" T/2 and y =
All learning agents select actions stochastically as described in 5 with 20.
Benchmark Algorithms
B-IRI.: We have run the Metropolis-Hasttee algorithm for 10,000 iterations to obtain 1,000 sam- ples from PP 'D) with intervals of 10 iterations between each sample after 10,000 burn-in itera- tions. At each iteration, new candidate samples are generated by adding Gaussian noise with covari- ance matrix 0.00521 / to the last sample. The final estimate Penv is formed by averaging all samples.
Formally, this assumes Pt = for & E {1 + LGi 1)T/MJ,... LiT/MJ} and j E M}. We have used B-IRL as the IRL solver for each individual fold j E {l..... M).
M-fold-IRL:
CP-IRL: Formally, similar to M-fold IRL, this assumes Pj = =pD)) for E (16-1), {bع and j E {, where ((0) = 0 and ((M) TT However unlike M-fold IRL, it employs a change- point detection algorithm to learn the {()'s together with the pli''s. We have partitioned the trajectory (L...., T) into sub-trajectories of length 10 when detecting change points and used B-IRL as the IRL solver.
We have assumed that the agent performs soft policy improvement after each time step (starting with a uniformly random policy). When computing soft policy improvements, the transition probabilities T [>'='| = T[x'] were estimated using the empirical distribution of contexts in dataset D-that is expectations of the form E.r=T' [:] were approximated by computing 1/T Exepl instead. Similar to B-IRI, we have the Hastong algorithm for 10,000
I-SPI:
20

---

Under review as a conference paper at ICLR 2022
iterations to obtain 1,000 samples from P(penv (D) with intervals of 10 iterations between each sample after 10,000 burn-in iterations. The same proposal distribution was used, and again, the final estimate Peny formed by by averaging all samples.
T-REX: We have maximized the likelihood given in [52] using the Adam optimizer with learning rate of 0.001, 31 0.9 and B2 0.999 until convergence, that is when the likelihood stopped improving for 100 consecutive iterations.
20, and N 1000 (with additional 1000 samples as burn-in). When taking gradient steps, we have used the RMSprop optimizer with learning rate of 0.001 and discount factor of 0.9. We have run our algorithm for 100 iterations.
B-ICB: We have set 0.10,
NB-ICB: We have set Ep = 5 10-4. and ΣB = 5 10-5, We have taken 1,000 samples from I)) with an interval of 10 iterations between each sample after 10,000 burn-in iterations (i.c. N 20,000).
For the simulated experiments, the error bars are obtained by repeating each experiment five times.
SUPPLEMENTARY REFERENCES
[63] R. Freeman Jr, H. Wiesner, P. Roberts, McDiarmid, M. Dykstra, and R. M. Merion, "Improving liver allcoation: MELD and PELD," Amer. J. of Transplantation, vol. 4, pp. 114-131, 2004.
[64] A. Holzinger, Biemann, C. S. Pattichis, and B. Kell, "What do we need to build explainable ai systems for the medical domain?" arXiv preprint arXiv:1712.09922. 2017
[65] S. Lundberg and S. Lee, "A unified approach to interpreting model predictions," in Proc. 31 Int. Conf. Neural Inf. Process. Syst., 2017.
[66] Y. Goyal, Z. Wu, J. Ernst, D. Batra, D. Parikh, and S. Lee, "Counterfactual visual explanations," " Proc. 36th Int. Conf. Mach. Learn., 2019.
[67] Bica, D. Jarrett, A. Hüyük, and M. van der Schaar, "Learning what-if explanations for sequential decision decision-making, making." in Proc. 9th Int. Conf. Learn. Representations, 2011.
21


Reviews:
[{"review_id": "jVncPKAGZir", "reviewer": "Reviewer_pmos", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are many Strengths in the paper:\n- the paper tackles and formulates the important and relevant problem of ICB\n- the paper is very well-written and the motivation was easy to follow\n- related work is covered rigorously\n- simulation setups are clearly stated and seem comprehensive \n\nA weakness might be that the implementations used in the experiments are not shared at this moment. I would love to see an effort to ensure reproducibility (sharing the implementations as supplementary material). \n\nNote here that I’m not the expert in this sub-field, so I might miss some critical merits or weaknesses of the paper. \n", "summary_of_the_paper": "The paper formulates and studies the problem named “Inverse Contextual Bandits (ICB)”, which asks “Given demonstrated behavior from a decision-making agent, how has the agent’s knowledge been evolving over time?”  Then, the paper proposes two concrete learning algorithms, imposing different specifications regarding the agent’s behavioral strategy. Finally, some simulations using both simulated and real-world data are provided, showing how ICB can be applied as an investigative device for recovering and explaining the evolution of organ allocation practices over the years. ", "summary_of_the_review": "Overall, the paper is well-written; the motivation, related work, and the experimental part are all easy to follow. I do not find critical flaw in the paper at this moment, but it is very likely that I might miss something. I would recommend weak accept, but I will pay attention to the other reviewer's opinion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "arfo5TutVck", "reviewer": "Reviewer_G5nF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Problem definition and setup**:\n\nI found much of the content in section 2 to be unnecessarily general. Indeed, the preliminaries describe a reinforcement learning framework, but only the contextual bandits setting is considered in the paper. I found this additional notation to be distracting to the paper, and would suggest the authors introduce only background that is used in the paper. Additionally, this unneeded generality makes explanations in various parts of the paper more complicated than needed -- in particular, Remark 1 would not be needed if the RL framework were not introduced.\n\nAdditionally, I found the objectives of the paper to be rather vague. Specifically, the authors place significant emphasis on the interpretability of the output of an ICB algorithm, but do not seem to describe what interpretability means. I think it would be useful to describe some desired or defining characteristics of an interpretable output, and to discuss how the belief trajectory, as the authors have modelled it in their two algorithms, does or does not satisfy those characteristics.\n\n**Relation to related work**:\n\nI am a bit confused with how this work fits into/differs from some prior works mentioned in the paper. In particular, I was quite confused why one of the three main contributions of the paper was to formalize the ICB problem, but the IRL problem (which, as I understand it, is strictly more general) has already been formalized and studied. Could you elaborate more on how your problem setting and approaches differ from, e.g., [39] (Bayesian inverse reinforcement learning)? In what ways are you solving a different problem, or improving on previous works on IRL in your contextual bandits setting? Why should one expect the performance in Tables 3-4 of the algorithms you propose to be better than those from prior works (e.g., B-IRL)?\n\n**Experimental results**:\n\nWhile I find it interesting that, in the OPTN experiment, you are able to associate (external) policy decisions with changes in feature importances, I have a number of questions/comments about the interpretability of the algorithm output, and the applicability of the model considered in the paper.\n\nContextual bandits model: I think it would be useful to explain (i) what the rewards represent in the context of this experiment (I couldn’t find this explained anywhere, but I might have missed it) and (ii) why it is reasonable to assume that $\\rho^*$ does not vary over time. I was confused on point (ii) for a while, since I would expect most bandits algorithms to converge to a fixed $\\rho^*$. In the context of your experiments, this would correspond to the relative importance of the features converging to a single value. Could you explain why one should not expect this to happen in your experiments?\n\nExplainability of the results: I think that the paper could benefit from more explicitly outlining the desired properties of explainable results, how their method achieves this, and various ways that the method could be used. Indeed, in order for the results of Figures 3-4 to be “explainable” or “interpretable,” it seems that external context (such as that given in Figure 4) is crucial. Even then, only general trends can be seen from the figures displayed in this section. It is not clear to me how one might interpret such trends in this way when the number of features is large. Do the authors have more general ideas for interpreting these results? How might interpreting these results change when using Algorithm 1 vs Algorithm 2 in your paper? It may be useful to discuss these in the paper.\n", "summary_of_the_paper": "This paper studies an inverse (linear) contextual bandits (ICB) problem, where, given a $T$-round realization of a bandit policy’s actions and observed rewards, the goal is to design an algorithm to estimate the underlying environment parameter, along with the “belief trajectory” of the bandit policy. A particular emphasis is placed on the belief trajectory being “interpretable” and capturing changes in the policy’s “knowledge of the world” over time.\n\nThe paper’s main contributions are (i) formalizing the inverse contextual bandits problem, (ii) designing two algorithms for this problem based on two different ways of modelling beliefs of the bandit policy, and (iii) providing empirical illustrations of how their algorithm can be used to investigate and explain changes in medical decision-making over time\n", "summary_of_the_review": "While the problem setting seems interesting, I think that there are a number of issues indicating that this work is not yet ready for publication. Indeed, the problem setting should be more concisely described, and the relation of this work to prior works on IRL should be discussed more thoroughly. Additionally, I think that the authors should more clearly outline the goals of the interpretability of their method, and outline concrete solutions others could use to interpret the belief trajectories in practice.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "14e9er0qoPX", "reviewer": "Reviewer_NLig", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: the considered problem is significant. Their model gains many important properties compared to related works. As they mentioned,  this is the first formal attempt at learning interpretable representations of nonstationary behavior. Using Bayesian methods to compute posteriors is valid. In addition, the experiments for liver transplantations are interesting.\n\nWeaknesses: The presentation is hard to follow. It should be improved.  There is no new technique in this paper and there are no theoretical guarantees for their proposed algorithms. It would be interesting if the authors can upper bound the error $|\\rho^* - \\hat{\\rho}^*|$  to show that their algorithms converge.   ", "summary_of_the_paper": "This paper addresses the problem of Inverse Contextual Bandit.  They raise an important question: given demonstrated behavior\nfrom an agent, how has the agent’s knowledge been evolving over time? Formally, Given a contextual bandit problem\n$(X, A, R, T )$,  where $R, T$ are unknown to the agent. Given an observational dataset $D$, and a family of reward parameterizations $P$ and belief parameterizations $B$, the inverse contextual bandits problem is to determine the true environment parameter $\\rho^*$\nand the belief parameters $\\beta_{1:T}$. \n\nThey propose two algorithms to learn these parameters. The first uses the agent’s knowledge in terms of Bayesian update. The second uses the Gaussian process. They demonstrate their algorithm through simulated and real-world data for liver transplantations.\n", "summary_of_the_review": "Based on strengths and weaknesses as I mentioned above, I think this paper is on borderline. Now I am leaning towards acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-P9plnD_q0R", "reviewer": "Reviewer_79eV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Abstract\n\n- Style: you don't need to quote the ICB abbreviation; you introduced it for the first time so not only do you not need to quote it that's also not how abbreviations are defined.\n\n# Introduction\n\n- Style: pick quoting or emphasis, don't use both w.r.t. \"descriptive modeling\".\n- I find your core application interesting. I wonder w.r.t. introduced policies if you attach a particular reward to a particular organ allocation policy? Indeed one may find that a policy is not that good at all so then one would have to revert back to the previous observation regime and in so doing, have introduced an unnecessary non-stationary regime.\n- Style: please put your desiderata in bullets, it makes it easier to read.\n- What do you mean that the environment needs to be learned? I.e. which part are you referring to of the 3-tuple that is a contextual bandit: $\\langle A,S,R \\rangle$? Please be precise. It is currently not clear what you mean (I think you mean the action-indexed reward-distributions, but pretend that not everyone reading this is a bandit pro).\n- Figure 1: I cannot help but to think that this is an overly complex way of depicting non-stationarity. You may as well just show a Markov chain of rewards and simply say that the directed edges propagate through different regimes and ergo the model as stands is non-stationary. Effectively just an HMM wherein each state your agent takes actions. But alas, I may have misunderstood your intentions with that figure.\n- Isn't an ICB simply a structural (in time) bandit conditional on a previous bandit which acted in the same domain (though crucially not in the same environment)? Consider bandit one ($t=1$): $\\langle A_1,S_1,R_2 \\rangle$ and bandit two ($t=2$): $\\langle A_1,S_2,R_2 \\rangle$ it sounds as if you're suggesting updating the reward distribution at time two as: $p(R_2 \\mid A_1, A_2, R_1)$. One bandit is conditional on a past bandit which also took actions in the same system.\n- What does \"evolves smoothly\" mean? Do you mean that you're able to take derivatives and that the environment is Lipschitz continuous (the former is strictly a subset of the latter) or do you mean that there are no discontinuities?\n\n# Inverse contextual bandits\n\n- Notation: perhaps you should stick to convention w.r.t. to the state-space and denote it with $\\mathcal{S}$ as is common.\n- $\\mathbb{D}$ sounds awfully a lot like an MDP - what's the difference?\n- \"In this work, we consider state transitions that occur independently of past states and actions\" - that's an interesting assumption. What's the logic there? We assume that the organ allocation problem operates on the same dynamical system (your 'environment') no? If so, then tracking the non-stationary behaviour, without taking past allocations into account, is akin to saying that your bandits are all independent. But if that is the case why consider contextual bandits at all (I.e. why include $\\mathcal{S}$)? Your assumption alone allows you to simply stack up 'normal' bandits $\\langle A,R \\rangle$ Which it to say you are prescribing more complexity than you are actually using - why?\n- It is **not** ok for you to cite research nr [23] (by Y. Qin, F. Imrie etc) which is not yet publically available, not even on arXiv, for us (reviewers) to corroborate your claims.\n- You say \"Distributions of newly arriving organs are largely independent of prior allocation decisions; [...] [23, 32, 33]\" -- figure 2 and section 3 of [33] very much seem to refute your statement. It would be helpful for you to have a similar diagram and point to the d-separation that you claim impose the conditional independence. At present, it sounds implausible.\n- For definition 1, one can you please make use of equation environments to show your maths -- you have superscripts to subscripts and it is all becoming a bit too small.\n- You have now defined ICB about four times, once is enough.\n- Can you please clarify this statement: \"In this first work on modeling non-stationary agents,\" -- non-stationary MABs have been studied for decades so I do not think that's what you mean, hence, please clarify.\n- Table 1 is great, good job, very informative.\n\n# Illustrative examples\n\n- Figure 3 is great.\n- Figure 4 is great. I think it would be better if you had a similar figure or that figure closer to the top; it really captures the problem you are trying to solve. So placing it closer to the beginning of the paper may help the reader get the gist of your paper early on.", "summary_of_the_paper": "The authors present the ICB an offline method for providing an interpretable description of observed decision-making whilst capturing the agent's non-stationary understanding of the world.", "summary_of_the_review": "This is a good and interesting paper. But it is questionable if this could not be achieved with non-stationary MABs or indeed; if past organ allocations have no impact on present allocation, simply running one MAB after another (though it is not clear why we cannot do this). Simply, their novel question in the second paragraph of the introduction is not novel and has been asked many times before.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary: This paper studies an inverse (linear) contextual bandits (ICB) problem, where, given a $T$-round realization of a bandit policy’s actions and observed rewards, the goal is to design an algorithm to estimate the underlying environment parameter, along with the “belief trajectory” of the bandit policy. A particular emphasis is placed on the belief trajectory being “interpretable” and capturing changes in the policy’s “knowledge of the world” over time.

The paper’s main contributions are (i) formalizing the inverse contextual bandits problem, (ii) designing two algorithms for this problem based on two different ways of modelling beliefs of the bandit policy, and (iii) providing empirical illustrations of how their algorithm can be used to investigate and explain changes in medical decision-making over time

Discussion: This paper has received high quality, long and detailed reviews that highlighted some flaws, in particular in the well-posedness of the problem and the clarity of the writing. The authors' response was long and detailed as well, and its quality was recognized by the committee. 
However, the consensus is that this work would require a full pass allowing to include most of the feedback received in the main text rather than in appendices, to discuss related problems in the literature in more depth and perhaps to refocus the exposition on the problem considered.

Recommendation: Reject.
Decision: Reject