Learning "What-if" Explanations for Sequential Decision-Making
Authors: Ioana Bica
Daniel Jarrett
Alihan Hüyük
Mihaela van der Schaar
Venue: ICLR 2021 Poster (2021)
Keywords: counterfactuals
explaining decision-making
preference learning

Abstract: Building interpretable parameterizations of real-world decision-making on the basis of demonstrated behavior--i.e. trajectories of observations and actions made by an expert maximizing some unknown reward function--is essential for introspecting and auditing policies in different institutions. In this paper, we propose learning explanations of expert decisions by modeling their reward function in terms of preferences with respect to ``"what if'' outcomes: Given the current history of observations, what would happen if we took a particular action? To learn these cost-benefit tradeoffs associated with the expert's actions, we integrate counterfactual reasoning into batch inverse reinforcement learning. This offers a principled way of defining reward functions and explaining expert behavior, and also satisfies the constraints of real-world decision-making---where active experimentation is often impossible (e.g. in healthcare). Additionally, by estimating the effects of different actions, counterfactuals readily tackle the off-policy nature of policy evaluation in the batch setting, and can naturally accommodate settings where the expert policies depend on histories of observations rather than just current states. Through illustrative experiments in both real and simulated medical environments, we highlight the effectiveness of our batch, counterfactual inverse reinforcement learning approach in recovering accurate and interpretable descriptions of behavior.

Published as a conference paper at ICLR 2021
LEARNING "WHAT-IF" EXPLANATIONS FOR SEQUENTIAL DECISION-MAKING
Ioana Bica
Daniel Jarrett
University of Oxford, Oxford, UK The Alan Turing Institute, London, UK ioana bic.aenng.......... ww....c....oooooooo............. uk
University of Cambridge, Cambridge, UK
daniel. jarrett sesett@maths cam. ac uk
Alihan Hüyük
Mihaela van der Schaar
University of Cambridge, Cambridge, UK ah20750cam, ac. uk
University of Cambridge, Cambridge, UK
Cambridge Center for Al in Medicine, UK
University of California, Los Angeles, USA The Alan Turing Institute, London, UK mv4 722 cam ac uk
ABSTRACT
Building interpretable parameterizations of real-world decision-making the basis of demonstrated behavior-i.e. trajectories of observations and actions made by an expert maximizing some unknown reward function-is essential for introspecting and auditing policies in different institutions. In this paper, we propose learning explanations of expert decisions by modeling their reward function terms preferences with respect to "what if" outcomes: Given the current history of observations, what would happen we took a particular action? To learn these cost- benefit tradeoffs associated with the expert's s actions, we integrate counterfactual reasoning into batch inverse reinforcement learning. This offers a principled way of defining reward functions and explaining expert behavior, and also satisfies the constraints of real-world merriiion-maiig---- -wheee active experimentation is often impossible (e.g. in healthcare). Additionally, by estimating the effects of different actions, counterfactuals readily tackle the off-policy nature of policy evaluation in the batch setting, and can naturally accommodate settings where the expert policies depend on histories of observations rather than just current states. Through illustrative experiments both real and simulated medical environments, we highlight the effectiveness of our batch, counterfactual inverse reinforcement learning approach in recovering accurate and interpretable descriptions of behavior.
INTRODUCTION
Consider the problem of explaining sequential becision-making on the basis of demonstrated behavior. In healthcare, an important goal lies in being able obtain an interpretable parameterization of the ex- perts' behavior (e.g in terms of how they assign treatments) such that we can quantify and inspect poli- cies in different institutions and uncover the trade-off's and preferences associated with expert actions (James & Hammond 2000 Westert et al. 2018 Van Parys & Skinner 2016 Jarrett & van der Schaar 2020). Moreover. modeling the reward function different clinical practitioners can be revealing as to their tendencies to treat various diseases more/less aggressively (Rysavy et al.) 2015). which -in combination with patient outcomes has the potential to inform and update clinical guidelines.
In many settings, such as medicine, decision-makers can be modeled reasoning about "what-if" patient outcomes: Given the available information about the patient, what would happen we took particular action? (Djulbegovic et al. 2018 McGrath| 2009). As treatments often affect several patient covariates, by having both benefits and side-efiects, socision-makers often make choices based their preferences over these counterfactual outcomes. Thus, in case, interpretable explanation of a policy is one where the reward signal for (sequential) actions is parameterized on the basis of preferences over (sequential) counterfactuals (i.e. "what-if" patient outcomes).

---

Published as a conference paper at ICLR 2021
U
History ht)
Countertscluale
Hastory
Counterfactwols
Side dfees winznoliie
D|Z [ol
1/2--1001 hu]
[[U
Action (Trentment)
Action No Treatment)
Reward function:
R(
[u] Au]
Zunlol [
Pederns trgahoo
R(br,
101 [a
goat profer
Figure 1: Explaining decision decision-making behaviour in terms of preferences over "what if" outcomes. Consider the evolution of tumour volume (U) and side effects (Z) under binary action. E[Ux+1[ae] he] and 1 [ar] lte] are the counterfactuals for the patient features under action a given history ht of prior actions and covariates. Parameterizing the reward as the weighted sum of these counterfactuals: R(h,, az) = w,, [,U
hr]] + w ( | h,)], naturally allows us to model the preferences of experts: e.g. finding that |ww|| |wa| indicates that the expert is treating more aggressively, by placing more weight on reducing tumour volume than on minimizing side effects.
Given the observations and actions made by an expert, inverse reinforcement learning (IRL) offers principled way for modeling their behavior by recovering the (unknown) reward function being maximized (Ng et al.) 2000 Abbeel & Ng 2004 Choi & Kim, (0TT). Standard solutions operate by iterating candidate reward functions, solving the associated (forward) reinforcement learning problem at each step. In many real- world problems, however, we are specifically interested in the challenge of offline learning- that is, where further experimentation is not possible-suud medicine. In this batch setting, we only have access to trajectories sampled from the expert policy in the form of an observational dataset- Sunss as in electronic health records.
in
Batch IRL. By their nature, classic IRL algorithms require interactive access to the environment. or full knowledge of the environment's 's dynamics (Ng et al. 2000 Abbeel & Ng. 2004 Choi & Kim
2011). While batch IRL solutions have been proposed by way of off-policy eva evaluation (Klein et al.
2011 2012 Lee al., 2209)) they suffer from two disadvantages. First, they are limited by the assumption that state dynamics are fully-observable and Markovian. This is hardly true in medicine: treatment assignments generally depend on how patient covariates have evolved over time (Futomal et al. 2020). Second, rewards are often parameterized as uninterpretable representations of neural network hidden states and consequently cannot be used to explain sequential decision making.
"What-if" Explanations. To address these shortcomings and to obtain a parameterizable inter- pretation of the expert's ' behavior, we propose explicitly incorporating counterfactual reasoning into batch IRI. In particular, we focus on "what if" explanations for modeling decision-making, while simultaneously accounting for the partially-observabbe nature of patient histories. Under the max-margin apprenticeship framework (Abbeel & Ng 2004 Klein et al. 2011 Lee et al.) 2019), we learn a parameterized reward function R(h,, at) that is defined as a weighted sum over potential outcomes (Rubin) 2005) for taking action a given history ht
As highlighted in Figure consider the decision making process of assigning a binary action given
the tumour volume (U) and side effects (Z). Let E[U++1 [a.] | ht] and E[Z [a.] ht] be the counterfactual outcomes for the two covariates when action a₂ is taken given the history hy of covariates and previous actions. We define the reward as the weighted sum of these counterfactuals: R(h,, az) = w,, W,,EUU++1 [a:] hx] +w,BBZe++1b
he]. to take into account the effect of actions and to directly model the preferences of the expert. The ideal scenario is when both the tumour volume and the side effects are zero, so the reward weights of a doctor aiming for this are both negative. However, recovering that |w,] > |wz] it means that the doctor is treating more aggressively, as they are focusing more on reducing the tumour volume rather than on the side effects of treatments.
Alternatively, |ww,| < [wz] indicates that the side effects are more important and the expert is treating less aggressively. Our motivation for using counterfactuals to define the reward comes from the idea that rational decision making considers the potential effects of actions (Djulbegovic et al. 2018).
Contributions.
Exploring the synergy between counterfactual reasoning and batch IRL for un- derstanding sequential decision making confers multiple advantages. First, it offers a principled approach for parameterizing reward functions in terms of preferences over what-if patient outcomes, which enables us to explain the fost-benefin tradeoffs associated with an expert's 's actions. Second, by estimating the effects of different actions, counterfactuals readily tackle the off-policy nature of
2

---

Published as a conference paper at ICLR 2021
Method
Environment Batch
Feature map for reward
Policy
Feat, expectations
Abbeel & Ng (2004)
Model-based No ((8,) = basis functions for state St <[a st)
Model roll-outs
Choi Kim 2
Model-based No
2, by (s) ax) ] = basis for belief by wia.
bi)
Model roll-outs
Klein el all 2011
Model-free Yes o(e,) basis functions for state ir m(a
LSTD-Q
Lce et al. 2019
Model-free Yes
((22; az) concat(o(2)). 0c) Maa
xx)
DSFN
Counterfactual
Ours
Model-free
Yes
(h,, a+) = B[Yen[(o]]hh)
aT(a he)
ps-learning
Table 1: Comparison of our proposed method (batch, counterfactual IRL) with related works in IRL.
policy evaluation in the batch setting. Furthermore, we demonstrate that not only does this alleviate the cold-start problem typical of conventional batch IRL solutions, but also accommodates settings where the usual assumption of full observability fails to hold. Through experiments in both real and simulated medical environments, we illustrate the effectiveness of our batch, counterfactual inverse reinforcement learning approach in recovering accurate and interpretable descriptions of behavior.
2 RELATED WORKS
In our work, the aim is to explain decision-makes by recovering the preferences of experts with respect to the effects of their actions, denoted by the counterfactual outcomes. This goal is fundamen- tally different from the goal of IRI. methods which generally aim to match the performance of experts. We operate under the standard max-margin apprenticeship framework (Ng et al.) [000]] Abbeel & Ng. 2004), which searches for a reward function that minimizes the margin between feature expectations of the expert and candidate policies. However, approach to recovering and understanding decision policies is uniquely characterized by incorporating counterfactuals to obtain explainable reward functions. To tackle the challenges posed by worldw decision making, method also operates in an offline and model-free manner, and accommodates partially-observabbe environments.
Explainability. By using basis functions (Klein et al. 2012) or hidden layers of deep network (Lee et al. 2019) to define the feature map. the learned rewards of either approach are inherently uninterpretable, and cannot be used to explain differences in expert behavior. An alternative approach for recovering the expert policy (without reward functions) is imitation learning (Hussein et al. 2017] Osa al. 2018 Torabi al., 2019 Jarrett al. 2020 However, these methods do not allow us to fully model the decision-making process of experts and to uncover the trade-ofi offfs behind their actions.
Batch Learning. Klein al. (2011) propose an off-policy evaluation method based on least squares temporal difference (LSTD-Q)) () (Lagoudakis & Parr (203) for estimating feature expectations, and Klein al. (2012) use a linear score-based classifier to directly approximate the Q-function offline. However, both methods require the constraining assumptions that rewards are direct, linear functions of fully-observable states states- assumppionss we cannot afford to make in realistic settings such as medicine. Lee et al. (2019) propose a deep successor feature network (DSFN) based on Q-learning to estimate feature expectations. But their approach similarly assumes fully-observable states, and additionally suffers from the "cold-start" problem where off- policy evaluations are heavily biased unless the initial candidate policy is (already) close to the expert.
Partial Observability. No existing batch IRL method accommodates modeling expert policies that depend on patient histories. While Choi & Kim (201)) and Makino & Takeuchi (2012) extend the apprenticeship learning paradigm partially observable environments by considering policies beliefs over states, both need to interact with the environment (or a perfect simulator) during learning.
To the best of our knowledge, we are the first propose explaining sequential decisions through counterfactual reasoning and to tackle the batch IRL problem in partially-observable environments. Our use of the estimated counterfactuals yields inherently interpretable rewards and simultaneously addresses the cold-start problem in Lee et al. (001)). Table [ highlights the main differences between our method and the relevant related works. See Appendix for additional related works.
3 PROBLEM FORMULATION
Preliminaries. At timestep t, let random variable X, E X' denote the observed patient features and let A, E A denote the action (e.g. treatment) taken, where A finite set of actions. Let x, and a, denote realizations of these random variables. Let h, = (æp, ag...................... It-1, 1, a,-1, 22,) = (æo.t: a0:t-1) E H be a realization of the history H, E H of patient observations and actions until timestep t.
3

---

Published as a conference paper at ICLR 2021
A stationary stochastic policy represents mapping: 1/ A 10, 1], where T(a h) indicates the probability of choosing action a E AA given history h E H. and LeEA a(a h) = 1. Taking action os under history h, results in observing + and obtaining h441. The reward function is R: X + IR where R(h, a) represents the reward for taking action a E A given history h E H. The value function of a policy n, V H + IR is defined as: V (h)
E E ? R Hr A.) n, Ho = hi] where E [0, 1) is the discount factor and At ~ T[- H₂ for t 2 0. The action-value function Q: H A + I of policy is defined as (h, a) E - 7 R(,,, As) n, Ho = h, Ao = al where A, ~ T(- H₂) for 2 O. A higher Q-value indicates that action a will yield better long term returns if taken for history h. We assume we know the discount factor y which indicates the importance of future rewards for the current history and action pair.
Batch Let ( ;=1 be batch observational dataset consisting of N patient trajectories: The trajectory ç* for patient i consists of covariates a and
S¹ = (r). ag.... TTT- 1. actions of observed for 7 timesteps. For simplicity, we drop the superscript i unless explicitly needed. The actions E 'D are assigned according to some expert policy gg such that a ~ Tg( he).
We work in the apprenticeship learning set-up (Abbeel Ng 2004) and we consider a linear reward function R(,, ax) = w o(h,, az), where the weights w E Rd satisfy ||]|| S 1. The feature map : H X A - Rd also satisfies (|o(-)ll2 S 1 such that the reward is bounded. We assume that the expert policy Tg is attempting to optimize, without necessarily succeeding, some unknown reward function R (h,, ax) w² (h,, a,). where w" are the 'true' reward weights. Given R(he, dr), the value of policy can be re- re-written as: E[V (Ho)] = = &5 (Hr, Ar) T] w- EI y o (H, A.) T.. where the expectation is taken with respect to the sequence of histories and action pairs (Hr, Ae):20 obtained by acting according to x. The feature expectation of policy t, defined as the expected discounted cumulative feature vector obtained when choosing actions according to policy x is μr" E = HH,,,, A,) π] E RR' such that: [V (Ho) = w -px".
Our aim is to recover the expert weights w" as well as find a policy ' that is close to the policy of the expert Æę. We take the max-margin IRL approach and we measure the similarity between the feature expectations of the expert's policy and the feature expectations of candidate policy using || - 12. In this batch IRL setting, we do not have knowledge of transition dynamics and we cannot sample more trajectories from the environment. Note that in this context, we are the first to model expert policies that depend on patient histories and not just current observations.
Counterfactual reasoning. To explain the expert's 's behaviour in terms of their trade-off associated with "what if" outcomes, we use counterfactual reasoning to define the feature map o(h, at) part of the reward (((,,,a)= = w -ohhe, uz). We adopt the potential outcomes framework (Neyman) 1923; Rubin 1978. Robins & Hernán 2008). Let [(a) be the potential outcome, either factual or counterfactual, for treatment a E A. Using the dataset D we learn feature map @(he, ae) such that:
s(he at) = E[a+ Aa].
(1)
where [[a] he] is the potential outcome for taking action de at time t given the history he For the factual action at, assigned under policy .T( he), the factual outcome is Z2+1 and this is the same as the potential outcome E [a.] he]- The potential outcomes for the other actions a, E A are the counterfactual ones and they allow us to understand what would happen to the patient if they receive a different treatment a, To identify the potential outcomes from the batch data we make the standard assumptions of consistency, positivity and no hidden confounders as described in Appendix No hidden confounders means that we observe all variables affecting the action assignment and potential outcomes. Overlap means that at each timestep. every action has non non-zero probability and can be satisfied in this setting by having a stochastic expert policy. These assumptions are standard across methods for estimating counterfactual outcome (Robins et al., 22000; /Schulam & Saria 2017] BBia et al. 2020a). Note that these assumptions are needed to be able to reliably perform causal inference using observational data. However, they do not constrain the batch IRL set-up.
Estimating the potential outcomes from batch data poses additional challenges that need to be considered. The fact that the expert follows policies that consider the history of patient observations when deciding new actions, gives rise to dimendependent confounding bias. Standard supervised learning methods for learning E[Y + [a.] h,] from D will be biased by the expert policy used in the observational dataset and will not be able to correctly estimate the counterfactual outcomes under alternative policies (Schulam & Saria 2017). Methods for adjusting for the confounding bias involve
4

---

Published as a conference paper at ICLR 2021
using either inverse probability of treatment weighting (Robins al. 2000 Lim et al. 2018)
building balancing representations (Bica et al. 2020a). Refer to Appendix B for more details.
In the sequel, consider the model for estimating counterfactuals feature map o(h,, a4) represents the effect of taking action a, for history hT. The reward is then:
black box such that the
R(h, ae)
@b,, ar) E [ar] he]
Defining the reward function using counterfactuals gives an interpretable parameterization of doctor behavior: It allows to interpret their behavior with respect to the importance weights implicitly assigned to the effects of their actions. This enables describing the relative trade-offs in treatment decisions. Note that we are not assuming that the experts themselves actually compute these quantities (nor that they explicitly adopt the same causal inference assumptions); rather, we are simply providing a way to understand how decision-maker are effectively behaving (i.e. in terms of counterfactuals).
4 BATCH INVERSE REINFORCEMENT LEARNING USING COUNTERFACTUALS
Max- Max-margin IRL (Abbeel & Ng 2004) starts with an initial random policy л and iteratively performs the following three steps recover the expert policy and its reward weighs: (1) estimate feature expectations 1² of candidate policy т, (2) compute new reward weights w and (3) find new candidate policy x that is optimal for reward function R(h, at) w Φ(h, at). This approach finds a policy т that satisfies || μ ||2 < such that स has an expected value function close the expert policy.
The expert feature expectations can be estimated empirically from the dataset D using:
@(hr, oz) IE 1E [ap] hd]
Countarfactual module
Inverse reinforcement learning
p FF
π
Eveluae Pulley
In the batch setting, we cannot estimate the fea- ture expectations of candidate policies by taking the sample mean of on-policy roll-outs: n² *
f
Compute reward weights
W
1 N -T= o y'o(h), (h;)). To address this off-
Figure 2: Counterfactual inverse reinforcement learning (CIRL). Counterfactuals are used to define @(h, a). to estimate feature expectations μ= of candidate policy π in batch setting and to learn optimal policy for reward weights
policy nature of estimating feature expectations, introduce a new method that leverages the estimated counterfactuals. We also make of the counterfac- tuals to learn optimal policies for different reward weights. Figure illustrates how we integrate "what if" reasoning into batch IRL.
4.1 COUNTERFACTUAL μr LEARNING
Similar the approach proposed by Klein at.. 2012) Lee (209 consider history-action feature expectation defined as follows μt" ( a) E 1 66 HI, A. v. Ho = h, Ao a). where the first action a can be chosen randomly and for Z 1. A. T( H.). This can be re-written as:
1² (h, a) = @hh, a) Ba
> (H,,, A.) n, Hi = h =
(4)
o(h, a) 7 of ( (h', (N',a')),
(5)
where h the next history. Notice the analogy between μr' (h, a) and the valuue function:
(h, a) = R(h, a) + Ex
Y (B(H,,A) Hi = K4,A, =a]
(6)
=R(b, yEr a' (M", (h',a')).
that allows us to use temporal difference learning to estimate feature expectations (Sutton et al. 1998).
Existing methods for estimating feature expectations fall into two extremes: (1) madel-based (online)
IRL approaches learn a model of the world and then use the model as a simulator to obtain on on-policy roll-outs (bbeel & Ng 2004) and (2) batch IRI. approaches use Q-learning (or alternative methods)
5

---

Published as a conference paper at ICLR 2021
Algorithm 1 (Batch, Max-Margin) CIRL
Input: Batch dataset D, max iterations n, convergence threshold 6,
feature map o 0e) = E [ [a,]]A,]
2: μT k- compute T feature expectations (Equation
3: f- random initial reward weights, + compute optimal policy for Ro Wo
4: _0 f- compute To's 8 feature expectations
(counterfactual jı-lcamnig))
5: II { (u00) 1o 11
6:
for 1 to do
7:
P4-1, TA compute optimal policy for R& wy. p + compute T 's feature expectations
Wk -TT
8:
(counterfacual pteaeeuuullllleaaae je-learning)
9:
IT IT { A A U {1***)
10:
Orthogonally project 1 onto line through Pa R
(1** 1k-1)² (" Pg- PB-1) (NTH ( PE-1,
= 1100 -uul22
(NTTH Px-1) 12 (uta (i--1)
11:
if < then break
12:
end for
13: K arg ming," 1I 1T 12, R(h, WK alh
Output: R, A,
for off-policy evaluation (Lee et al. 2019). and can only be used to evaluate policies similar to the expert policy and require warm start. In our case, the counterfactual model allows us to compute h' (h, B [h]] for any h D and any arbitrary action Thus, we propose counterfactual y-learning, a novel method for estimating feature expectations that uses these counterfactuals as part of temporal difference learning with 1-step bootstrapping. This approach falls in-between (I) and (2) and allows us to estimate feature expectations for any candidate policy æ in the batch IRL setting.
The counterfactual ju-learning algorithm learns the y-values for policy iteratively by updating the current estimates of the p-values with the feature map plus the y-values obtained by following policy in the new counterfactual history h' (h, a, E [a] hl):
A²t (h, + pat (h, a) + ( a) yBa ( a')] p= (h)
(8)
where the learning rate. We use recurrent network with parameters to approximate fe (h, 0) and we train it by minimizing the sequence loss functions C; which changes for every iteration
(a.) Ea- Illv= f= (h, 0:)112]
0.t1 + 0: + (BC...)..
where the action can be chosen randomly from A and yı = (0, +/a' w=(-) (A" , a' 0,-1)) the target for iteration i. The parameters for the previous iteration 0;-1 are held fixed when optimiz- ing (O,). Referte to Appendix(C]oos full details of the counterfactual şиllaaniig algorithm. The feature expectations for the policy x are given by fMT ( [A- (Ho. (Ho.Ap)) Ao)). which can be estimated empirically from the observational dataset D using A t EN EMEA pe (his a)T(a kt).
4.2 FINDING OPTIMAL POLICY FOR GIVEN REWARD WEIGHTS
During each iteration of max-margin IRI., we obtain a candidate policy to evaluate by finding the optimal policy for given vector of reward weights. We use deep recurrent Q-learnin (Hausknecht Stone 2015), model-free approach for learning Q-values for reward function R(h, a) w o(h, a). The counterfactuals are used to compute o(h, a), and to estimate the next history for the temporal difference updates. See Appendix D for details. After estimating the Q-value, Q(h, a) for history and action a, a new candidate policy is obtained using: (a h) 1 1a arg m Q(haa))
4.3 COUNTERFACTUAL INVERSE REINFORCEMENT LEARNING ALGORITHM (CIRL)
Algorithm| [ describes our proposed counterfactual inverse reinforcement learning (CIRL) method for the batch setting. CIRL is based on the projection algorithm proposed by Abbeel & Ng 2004) and iteratively updates the reward weights to minimize the margin between the expert's 's feature expecta- tions and the feature expectations of intermediate policies. CIRL uses our proposed counterfactual leaennnn algorithm for estimating the feature expectations of intermediate policies in an off off-policy manner that is suitable for the batch setting. Compared to the algorithm for batch IRL. proposed by

---

Published as a conference paper at ICLR 2021
Lee et al. (2019) that requires the initial policy To to already be similar the expert policy, CIRL works for any initial policy x that is optimal for the randomly initialized reward weights wo-
Similarly to Choi & Kim 2011), the CIRL algorithm returns the reward function R(,, a) that results in a policy with feature expectations closest to the ones of the expert policy. We show experimentally that the reward that yields the closest feature expectations will be similar to the true underlying reward function of the expert. CIRL returns the set of policies tried TI and their feature expectations A, which allows us to compute a mixing policy that would yield similar performance to the expert policy (Abbeel & Ng, 2004). Let j be the closest point to 1º in the convex closure of J,, which can be computed by solving the quadratic programming problem:
A = {1000 1z)
min || Rh ull2 s.L. μ Σ A.p T A O. Σ =1
(10)
From the termination criteria of Algorithm 1. 1 is separated from the points p by a margin of at most Thus, the solution j will satisfy I| ,ill S e. To obtain a policy that is close to the performance of the expert policy, we mix together the policies {mo,... returned by Algorithm 1, where the probability of selecting π, is X
EXPERIMENTS
We evaluate the ability of CIRL to recover the preferences of experts over the "what if" outcomes of actions. These preferences are denoted by the magnitude of the recovered reward weights. Since we do not have access to the underlying reward weights of experts in real data, we first validate the method in simulated environment. To show the applicability of CIRL in healthcare, we also perform a case study on an ICU dataset from the MIMIC III database (Johnson et al.. 2016).
Benchmarks. For our proposed CIRL method, we uses the Counterfactual Recurrent Network, state-of-the art model for estimating counterfactual outcomes in a temporal setting (Bica et al. 2020a).
Note that other models for this task are also applicable (Lim et al. 2018). Reter to Appendix F
for details. We benchmark CIRL against MB-IRL: model- IRL iee. inverse reinforcement
learning with model-based policy evaluation (e.g. Yin et al. 6016); Nagabandi et al. (2018); Kaiser
jtt al. (2019); Buessing et al. (2018). We consider two versions this benchmark: (1)RI) which uses the patient history and MB(x)-IRL which only uses the current observations to define the policy, For both MB(x)-IRL and MB)))-IRL we define the reward as a weighted sum of counterfactual outcomes, but these methods use instead standard supervised methods estimate the next history h needed for the counterfactual μellaariigg algorithm. These benchmarks are aimed to highlight the need of handling the bias from the time-dependent confounders and using suitable counterfactual model, but also the importance of handling the patient history. We also compare against the deep successor feature networks (DSFN) proposed by Lee et al. (019),, which currently represents the th-the Batch IRL for the MDP setting. To show that their approach for estimating feature expectations the batch setting is suboptimal, we extend their method to also incorporate histories in the DSFN()) benchmark. Implementation details of the benchmarks can be found in Appendix G
5.1 EXTRACTING DIFFERENT TYPES OF EXPERT BEHAVIOUR
Simulated environment. We propose an environment that uses a general data simulation involving p-order regoosegressive processes. To analyze different types of expert behaviour (e.g. treating more/less aggressively) we simulate data for patient features representing disease progression (r),
e.g. tumour volume and side effects (z) and action (a) indicating the binary application of treatment. For time t, we model the evolution of patient covariates according to the treatments as follows:
Fi- - 2.5 Σ P 1=1
Σ 1=1
ai + +c 21 Σ 21 + +05 Σ Ge -a---pty (1I)
F1
where p 5 and N N0, 0.13) are sampled as follows: 2g -30330 5) and 2g (2, 1). We set Imaz 50 and Zmaz 15. The trajectory the patient terminates when either ae 0, 24 2 Iman= 2; 2 Zmav or 1. 2 20. The tumour volume xt, denoting the disease progression, decreases when we give treatment and increases otherwise. Conversely, the side effects 2,, increase when we give treatment and decrease otherwise. We define linear reward for taking action a, given history ht (Io.t> 20tts a0:--1) as follows:
are noise terms. The initial values for the features
R(he, 0t) = W2
J++1
+ W2
Z++1
(12)
Imas Tmnin
Zmaz Zmin
7

---

Published as a conference paper at ICLR 2021
where = [w1, Wz], ||w]| S and 2++1 and z++1 are simulated according to equations to take into account the effect of action a, for history h4. The features are normalized to [0. 1]. The best scenario for patient is when both the side effects and tumour volume are and doctor attempting to achieve this will have negative reward weights. However, different settings of the reward weights will result in different expert behaviours. For instance, for wi -0.7 and i22 -0.3, the expert policy will focus more on the disease progression and will consequently treat more aggressively, while this behaviour will be reversed for reward weights set w1 -0.3 and w2 -077. We used deep recurrent Q Q-learning (Hausknecht & Stone 2015) to find a stochastic expert policy that optimizes the reward function for different settings of the weights. The batch dataset consists of 10000 trajectories sampled from the expert policy. Refer to Appendix E for more details.
Recovering decision making preferences of experts. We first evaluate the benchmarks on their ability to re- cover the weights of the reward function optimized by the expert for the experimental setting with 0.99, w1 003 and Wy -0.7. Note that DSFN does not provide interpretable reward weights, and thus cannot be used for understanding the trade-offs in the expert behavior. We train each benchmark 10 times and we plot in Figureß reward weights obtained for the different iterations. We show that our proposed CIRL method performs best at recovering the preferences of the expert, which in this case is to treat less aggressively. While the
-2.
CRLL
1GST BiL by Mistt Ric Viice Rloarr Milce RL:
Figure 3: Reward weights recovered by benchmarks over 10 runs. The weights of the expert are W11 0.3 and W/s = -0.7. MB (B(h)-IRL method also recovers the correct trade-offs in the expert behavior, the computed weights have much higher variance. Conversely, the MB(z)-IRL method, which does not consider the patient history fails to recover the underlying weights of the expert policy.
Matching the expert policy. We evaluate the benchmarks' ability to recover policies that match the performance on the expert policy for two settings of the discount factor E (0.99, 0.5}. A lower indicates that the expert is optimizing for the immediate effect actions, while higher means they considered the long term effect of actions. For each 7 we learn expert policies for different reward weights and we use the expert policies to generate the batch datasets. We evaluate the policies learned by the benchmarks using two metrics: cumulative reward for running the policy in the simulated environment and accuracy on matching the expert policy (computed as described in Appendix (H). We report in Tables 2 and 3 the average results and their standard error over 1000 sampled trajectories from the environment. CIRL recovers policy that has the closest cumulative reward to the expert policy and that can best match the treatments assigned by the expert.
Table 2: Mean cumulative reward and standard deviation for running learnt policy in the environment.
7 = 0.99
0.5 11 -0.77
Reward weights
101 -0.3
Mi -0.7
W1 -0.5
Wh -033
Wi 00.5
102 -0.7
U2 -0.3
W/2 -0.5
U22 0.7
1/2 -0.3
We -0.5
-B(x))IRL
-3.78 + 0.02
-4.42 + 0.05
-4.90 + 0.05
0.05
-4.51 0.05
-4.53 + 0.05
-4.54 + 0.04
MB(h)-IRL
3.23
-4.10
0.02
-4.63 0.04
-4.43
-3.54 0.05
-4.35 + 0.03
0.03
DSFN
0.06
-432 +:220..00
-3.77 0.05
-4.11
-3.07 = 0.03
-4.67 + 0.07
DSFN(A)
0.07
-4.3
+ 0.07
-3.60 + 0.07
-3.55
-3.05 - 0.05
-4.61 + 0.05
CIRL.
2.89 0.02
3.92 + 0.03
3.41
0.05
2.79
2.91
0.02
-4.7 + 0.03
Expert
-272
3..1 + 0.02
-2.81 0.01
-2.65
2.36 0.01
33.77 + 0.03
Table 3: Average accuracy and standard deviation for matching the actions in the expert policy.
? = 0.99
0.5
Reward weights
W1 0.3
W1 0.7
W1 0.5
01 0.3
Wi 0.7
Wi 0.5
U/2
-0.7
00.3
00.5
-077
-0.3
00.5
MB(2)-IRL
62.5 + 0.41%
61.4 + 0.81%
54.6 = 0.56%
52.4 0.63%
60.1 - 0.39%
71.8 + 0.72%
MB(A)-ILL
77.8 0.31%
70.2 0.44%
71.4 1 0.69%
66.3 1 0.58%
70.2 1 0.71%
75.6 1 0.52%
DSFN
75.4 0.32%
0.21%
73.4 0.45%
80.2 + 0.37%
70.8 0.24%
69.8 + 0.44%
DSHN(h)
76.3 0 0.37%
67.5 + 0.32%
80.6 0.54%
80.4 0.56%
71.0 0.35%
70.2 1 0.47%
CIRL
81.8 0.42%
0.51%
83.7 0.76%
89.5 + 0.37%
73.2 = 0.43%
80.4 + 0.42%
8

---

Published as a conference paper at ICLR 2021
5.2 CASE STUDY ON REAL-WORLD DATASET: MIMIC III
Suppose we want to explain the decision-making process of doctors assigning antibiotics patients in the ICU. For this purpose, we consider a dataset with 6631 patients that have received antibiotics during their ICU stay, extracted from the Medical Information Mart for Intensive Care (MIMIC III) database (Johnson et al. (016).
Temparatur
We used CIRL, with 7 0.99. to recover the policy and the re- ward weights of doctors administering antibiotics to understand their preferences over the effect of antibiotics on the patient fea- tures. The relative magnitude of the reward weights for the counterfactual outcomes of the patient features considered are illustrated in Figure 4 CIRL found that reducing temperature had the highest weight in the reward function of the expert, fol- lowed by WBC. This corresponds to known medical guidelines (Marik 2000 Palmer et al.) 2008). Sepsis is a leading cause of morbidity and mortality in the ICU, and several studies have shown that early administratio of antibiotics is crucial to decrease the risk of adverse outcomes (Zahar al. 2011]. Fever and elevated WBC are among a small subset of variables that indicate a systemic inflammatory response, concerning for the development of sepsis (Neviere et al., 2017). While these findings are not specific to bacte- rial infection, the risk of failing to treat potentially serious infection often outweighs the risk of inappropriate antibiotic administration, thereby driving clinicians to prescribe antibiotics in the setting of these abnormal findings. Similarly, the decision to discontinue an- tibiotics is complex, but it is often supported by signs of a resolution of infection which included normalization of body temperature and downtrending of the WBC. As such, our finding that the two highest reward weights for the administration of antibiotics in the ICU is temperature and WBC is consistent with clinical practice. Moreover, note that the model is simply identifying the factors that are driving the meaiing of the clinician represented in this dataset.
Potassium
0.2
WBC
3.23
Creatinine
Heart
Blood pressure
Homatocht:
Heriaglibin
Figure 4: Radar plot of reward weights magnitude for assign- ing antibiotics.
Accuracy
MB(3)-LRL
70.1 + 0.11%
MB01)-IR
77.5 +: 0.15%
DSPN
73.5 1: 0.25%
DSPN(A)
75.3 + 091%
CIRL
83.4 + 0.17%
Table 4: Accuracy on match- ing expert actions.
In Table 4 to verify that explainability does not come at the cost of accuracy we also evaluate the benchmarks on matching the expert actions.
DISCUSSION
In this paper, we propose building interpretable parametrizations of sequential decision-making by explaining an expert's 's behaviour in terms of their preferences over "what-if" outcomes. To achieve this, we introduce CIRL, new method that incorporates counterfactual reasoning into batch IRL: counterfactuals are used to define the feature map part of the reward function, but also to tackle the off- off-policy nature of estimating feature expectations in the batch setting. The reward weights recovered by CIRL indicate the relative preferences of the expert over the counterfactual outcomes of their actions. Our aim is provide description of behavior, i.e. how the expert is effectively behaving under our interpretable parameterization of the reward based on counterfactuals. We are not assuming that the experts actually operate under this specific (linear, in our case) model or that they compute the exact counterfactuals. Instead, our purpose is to show that we can explain an agent's 's behavior on the basis of counterfactuals, which is useful in that it allows us to audit them, sanity-check their policies and find variation in practice. Further discussion can be found in Appendix
There are several limitations of your method and directions for future work. While our method considers reward functions that are linear in the features, one way of extending it to handle more complex reward functions is use domain knowledge to define the feature map as a non-linear function over the counterfactual outcomes. Nevertheless, these functions should be defined in way that still allows us to obtain interpretable explanations of the expert's 's behaviour. Moreover, although time-invariant reward weights are standard in the IRL literature (Abbeel & Ng 2004| Choi & Kim
2011 Lee et al. 2019), time-varinnt rewards/policies have been considered in the dynamic treatment regimes (reinforcement learning) literature (Chakrabortyy) (013) Zhang & Bareinboim 2019). Thus another direction for future work would be to extend our method to consider non-stationary policies and reward weights that can change over time.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We would like to thank the reviewers for their valuable feedback. The research presented in this paper was supported by The Alan Turing Institute, under the EPSRC grant EP/N510129/1 by Alzheimer's Research UK (ARUK), by the US Office of Naval Research (ONR), and by the National Science Foundation (NSF) under grant numbers 1407712, 12, 1462245, 1524417, 1533983, and 1722516. The authors are also grateful to Brent Ershoff for the insightful discussions and help with interpreting the medical results on MIMIC III.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp- 1. ACM, 2004.
Ioana Bica, Ahmed Alaa, James Jordon, and Mihaela van der Schaar. Estimating counterfactual treatment outcomes over time through adversarially balanced representations. In International Conference Learning Representations, 2020a.
Ioana Bica, Ahmed M Alaa, Craig Lambert, and Mihaela van der Schaar. From real-world patient data to individualized treatment effects using machine learning: Current and future methods to address underlying challenges. Clinical Pharmacology Therapeuties, 2020b.
Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp- 182-189, 2011.
Lars Buesing, Theophane Weber, Yori Zwols, Sebastien Racaniere, Arthur Guez, cean-Baptiste Lespiau, and Nicolas Heess. Woulda, coulda, shoulda: Counterfactually-gudde policy search. International Conference on Representation Learning (ICLR), 2018.
Benjamin Burchfiel, Carlo Tomasi, and Ronald Parr. Distance minimization for reward learning from scored trajectories. Thirtieth AAAI Conference Artificial Intelligence, 2016.
Bibhas Chakraborty. Statistical methods for dynamic treatment regimes. Springer, 2013.
Jaedeug Choi and Kee-Eung Kim. Inverse reinforcement learning in partially observable environments. Journal of Machine Learning Research, 12(Mar):691-730, 2011.
Benjamin Djulbegovic, Shira Elqayam, and William Dale. Rational decision making in medicine: implications for overuse and underuse. Journal of evaluation in clinical practice, 24(3):655 665, 2018.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pp. 49-58, 2016.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. arXiv preprint arXiv: 1710. 1248, 2017.
Joseph Futoma, Michael C Hughes, and Finale Doshi-Velez. Popcorn: Partially observed predic- tion constrained reinforcement learning. International Conference Artificial Intelligence and Statistics (AISTATS), 2020.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 AAAI Fall Symposium Series, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1788 1997.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
Brent C James and M Elizabeth H Hammond. The challenge of variation in medical practice. Archives of pathology & laboratory medicine, 120(7):0001-1003, 2000.
10

---

Published as a conference paper at ICLR 2021
Daniel Jarrett and Mihaela van der Schaar. Inverse active sensing: Modeling and understanding timely decision- decision-making. making. International Conference on Machine Learning (ICML), 2020.
Daniel Jarrett, Ioana Bica, and Mihaela van der Schaar. Strictly batch imitation learning by energy- based distribution matching. Advances in Neural Information Processing Systems, 2020.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3:160035, 2016.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv: MxXiv:9903.00333 2019.
Edouard Klein, Matthieu Geist, and Olivier Pietquin. Batch, off-policy and model-free apprenticeship learning. In European Workshop on Reinforcement Learning, pp. 285-296. Springer, 2011.
Edouard Klein, Matthieu Geist, Bilal Piot, and Olivier Pietquin. Inverse reinforcement learning through structured classification. In Advances in neural information processing systems, pp. 1007-1015, 2012.
Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. Journal of machine learning research, 4(Dec):1107-111) 2003.
Donghun Lee, Srivatsan Srinivasan, and Finale Doshi-Velez. Truly batch apprenticeship learning with deep successor features. In Proceedings of the Twenty-Eights International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 5909-5915 International Joint Conferences on Artificial Intelligence Organization, 2019. doi: 2244966/jcaii221998888 URL https:://///
10. 1ttp6/////caaa.1119//.
Bryan Lim, Ahmed Alaa, and Mihaela van der Schaar. Forecasting treatment responses over time using recurrent marginal structural networks. In Advances Neural Information Processing Systems, pp. 7493-7503, 2018.
Takaki Makino and Johane Takeuchi. Apprenticeship learning for model parameters of partially observable environments. arXiv preprint 1206. 2012.
Mohammad Ali Mansournia, Goodarz Danaei, Mohammad Hossein Forouzanfar, Mahmood Mah- moodi, Mohsen Jamali, Nasrin Mansournia, and Kazem Mohammad. Effect of physical activity on functional performance and knee pain in patients with osteoarthritis: analysis with marginal structural models. Epidemiology, pp. 631-640 2012.
Paul E Marik. Fever in the icu. Chest, 117(3):855-88)) 2000.
Brent M McGrath, How doctors think, 2009.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Toannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynam- ics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018.
Remi Neviere, Polly E Parsons, and Geraldine Finlay. Sepsis syndromes in adults: Epidemiology, definitions, clinical presentation, diagnosis, and prognosis. Monografia en Internet). Wolters Kluwer: UptoDate, 2017.
Jersey Neyman. Sur les applications de la théorie des probabilités aux experiences agricoles: Essai des principes. Roczniki Nauk Roiniczych, 10:1-51, 1923.
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume 1, pp. 663-670, 2000.
11

---

Published as a conference paper at ICLR 2021
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends(R) in Robotics, 7(1-2): 1-179, 2018.
Lucy B Palmer, Gerald C Smaldone, John J Chen, Daniel Baram, Tao Duan, Melinda Monteforte, Marie Varela, Ann K Tempone, Thomas O' O'Riordan, Feroza Daroowalla, et al. Aerosolized antibiotics and medicine, 36(7), 2008.
tracheobronchitis in the intensive care unit Critical care
Robert W Platt, Enrique F Schisterman, and Stephen R Cole. Time-modified confounding, American journal of epidemiology. 170(6))687-694, 2009.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI, volume 7, pp. 2586-2591, 2007.
James M Robins and Miguel A Hernán. Estimation of the causal effects of time-varying exposures. In Longitudinal data analysis, pp. 547-593, Chapman and Hall/CRC, 2008.
James M Robins, Miguel Angel Hernan, and Babette Brumback. Marginal structural models and causal inference epidemiology, 2000.
Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55, 1983.
Donald B Rubin. Bayesian inference for causal effects: The role of randomization. The Annals of statistics, pp. 34-58, 1978.
Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469) 2005.
Matthew A Rysavy, Lei Li, Edward F Bell, Abhik Das. Susan R Hintz, Barbara Stoll, Betty R Vohr, Waldemar A Carlo, Seetha Shankaran, Michele C Walsh, et al. Between-hospital variation in treatment and outcomes in extremely preterm infants. N Engl J Med, 372:1801-1811, 2015.
Peter Schulam and Suchi Saria. Reliable decision support using counterfactual models. In Advances in Neural Information Processing Systems, pp. 1697-1708, 2017.
Richard S Sutton, Andrew G Barto, et Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998.
Faraz Torabi, Garrett Wamell, and Peter Stone. Recent advances in imitation learning from observa- tion. arXiv preprint 1905. 13566, 2019.
Jessica Van Parys and Jonathan Skinner. Physician practice style variation- impiaiion-ipllicaaiinn for policy, JAMA internal medicine, 176(10):1549-1555 2016
Gert P Westert, Stef Groenewoud, John Wennberg, Catherine Gerard, Phil DaSilva, Femke Atsma, and David C Goodman. Medical practice variation: public reporting a first necessary step to spark change. International Journal for Quality in Health Care, 30(9):731-735, 2018
Hang Yin, Patrícia
Francisco S Melo, Aude Billard, and Ana Paiva. Synthesizing robotic handwriting motion by learning from human demonstrations. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, number CONF, 2016.
Jean-Ralph Zahar, -ran-Francois Timsit, Maïte Garrouste-Orgee Orgeas, Adrien Francais, Aurélien Vesim, Adrien Descorps-Dechee Declere, Yohann Dubois, Bertrand Souweine, Hakim Haouache, Dany Goldgran- Toledano, et al. Outcomes in severe sepsis and patients with septic shock: pathogen species and infection sites are not associated with mortality. Critical care medicine, 1886):188611888 201
Junzhe Zhang and Elias Bareinboim. Near-optimal reinforcement learning dynamic treatment regimes. In Advances in Neural Information Processing Systems, pp. 13401-13411, 2019.
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. 2008
12

---

Published as a conference paper at ICLR 2021
A ADDITIONAL RELATED WORKS
Alternative approaches for IRL require known MDP or POMDP model. In this context, several Bayesian approaches to IRL have also been proposed which make use of the know transition dynamics provided by the model of the environment (Ziebart et al. 2008, Ramachandran & Amir 2007). We do not assume access to a MDP or POMDP model. Makino & Takeuchi (2012) propose a method for estimating the unknown parameters of POMDP model of the environment; however their method is only applicable to discrete observation spaces. We do not make any assumptions about the observation space. While several model-free IRI. methods have also been developed, these also not applicable in the batch setting where we only have access to a fixed set of expert trajectories. Relative Entropy IRL (Boularias et al., 2011) requires non-expert trajectories following arbitrary policy; such experimentation to obtain non-expert trajectories is not possible in the healthcare setting. Distance Minimization IRL (DM-IRL) (Burchfiel et al. 2016) learns reward functions from scores assigned by experts to sub-optimal demonstrations. Moreover, guided cost learning proposed by FFin jet al. (2016) which performs model-free maximum entropy optimization also requires the ability to sample trajectories from the environment. Similarly, adversarial IRI., proposed by Fu et al. (2017) builds upon the maximum entropy framework in Ziebart et al. (2008). but also requires the ability to exccute trajectories from candidate policies.
B COUNTERFACTUAL ESTIMATION AND TIME-DEPENDENT CONFOUNDING
Observational patient data (e.g. electronic health records) contain information about how actions, such treatments are performed by doetors and how they affect the patient's 's trajectory. Such data can be used by causal inference methods to estimate counterfactual outcomes-whal would happen to the patient if the expert takes a particular action given a history of observations?
To identify the counterfactual outcomes from observational data we make the standard assumptions of consistency, positivity and no hidden confounders as described in Assumption I (Rosenbaum & Robins Rubin 1983 Robins et al. 2000p. Under Assumption I, [E((+++[aa, [a.] h,] = E[X++1 h, and this
can be estimated by training regression model on the batch observational data.
Assumption 1 (Consistencyy, Ignorability and Overlap). For any individual i, if action a, is taken at time I, we observe X++1 Y++1 [ux]. Moreover, we have sequential strong ignorabil- ity (Yz+1 [a]]acA at h4,, VL and sequential overlap Pr(At = a h4) > O, VI, E A.
The sequential strong ignorability assumption, also known as no hidden confounders, means that we observe all variables affecting the action assignment and potential outcomes. Sequential overlap means that at each timestep, every action has a non-zero probability, which can be satisfied by having stochastic expert policy, These assumptions are standard across causal inference methods (Robins) jet al. 2000; Schulam & Saria 2017: Lim et al. 2018 Bica et al. (022a).. We emphasize that these assumptions needed to be able reliably perform causal inference using observational data. Nevertheless, they do not constrain the batch IRL. set-up.
Using observational data estimate the counterfactual outcomes poses additional challenges in this set-up that need to be considered. In particular, direct estimation of the treatment effects biased by the presence of time-varying confounders (Mansournia et al. 2012)) which are patient covariates that influence the assignment of treatments and which, in turn, are affected by past actions.
For instance, consider that the doctor's ' policy of assigning treatments takes into account whether the patients' covariates denoting disease progression, such as tumor volume, has been increasing above normal thresholds for several timesteps. If these patients are also more likely to have severe side effects, without adjusting for the bias introduced by time-varying confounders, we may incorrectly conclude that the treatment harmful the patients (Platt et al. 2009). Using standard supervised learning methods to estimate the patients' response to treatments in this setting will be biased by the doctor's policy and will not be able to estimate the effect of treatments under different policies. This represents an issue for our batch IRL setting where we need to be able to evaluate candidate policies that are different from the expert policy. Thus, being able to correctly estimate counterfactual treatment outcomes is crucial for method. Methods for adjusting for the confounding bias use either inverse probability of treatment weighting (IPTW) or balancing representations. The first approach, IPTW, involves creating pseudo- population where the probability of taking action a
13

---

Published as a conference paper at ICLR 2021
does not depend on To.t (Robins al. 2000 Lim et al. 2018), while the second approach builds balancing representation of the history that is invariant to the treatment (Bica et al. (020a). Using either approach will result in an unbiased estimation of the counterfactual outcomes. See Robins et al.) (2000); Lim et al. (2018); [Bica et al. (2020b) for more details about time-dependent confounding as well as for more in-depth review of alternative methods for removing the bias from time-dependent confounders and estimating counterfactual outcomes.
C COUNTERFACTUAL J-LEARNING
Our counterfactual je-leaniig algorithm involves learning the μ-values for policy и iteratively by updating the current estimates of the p-values with the feature map plus the p-values obtained by following policy in the new counterfactual history h = (h, a, [[[[a][h]]
Att (h, a) + (h, a) + a(o(h, a) + 7 Ja (h' a')) - p (h, a));
(13)
where a is the learning rate and y the discount factor.
We approximate A=' d"(h, 0y) using an RNN with parameters 8 The RNN consists of an LSTM unit with linear output. To stabilize training we use target network ji' with parameters 07 that provide updates for the main network. Note that using a target network is standard in Q Qelanning (Mnih al. 2013] Hausknecht Stone, 2013). The target network f has the same architecture as the main network and its parameters of are updated to match 0y every M iterations.
The loss function for updating the p. network for estimating feature expectations at iteration is:
(0,) = Enwpllly.. -0 (h 0,,))1l2]
(14)
01.+1 + 8 + ((.(()))))
(15)
where a is the learning rate, is the batch observational dataset and
()h,a), ys. ((h,a) + + (N', d' (7)),
if patient trajectory terminates at history h otherwise
(16)
the target for iteration obtained using the target network yz. The action a chosen by following an e-greedy policy by selecting action T(. h) with probability 1 e and a random action with probability e. We perform M training iterations. Algorithm [ describes the full training procedure used.
Algorithm@desscrbee
Table 5 indicates the search ranges used for the various hyperparameters involved in the counter- factual pelearning algorithm. The hyperparameters were selected to ensure stability of training and convergence of the algorithm.
Hyperparameter
Hyperparameter search range
Hyperparameter value
LSTM size
64, 128, 256 128, 256. 512
128
Batch size
256
Learning rate
0.00001, 0.0001, 0.001 100, 200, 500
0.001
Target network update M
100
Min
0.0
Max E
0.9
E decay
0.00001
Number of training iterations M
20000
Optimizer
Adam
Table 5: Hyperparameters for training ju-network for estimating feature expectations.
14

---

Published as a conference paper at ICLR 2021
Algorithm 2 Counterfactuall /r learnnnn
1: Input: Batch dataset D - {xx), oy. 1 -
policy to evaluate æ, counter-
factual model o(h, a)
2: Initialize feature expectations network j with random weights 0
3: Initialize target feature expectations network n with random weights j 8
4: for i 1 to M do
8 Tb=1 5: Sample random minibatch B = { B do 7: With probability select random action a", otherwise select a² ~ a(. h²))
of histories h from D
6: for b
8: Compute counterfactual histories h² (h² E[y [a') (h') using the counterfactual
model: [[Y(a")]]] [a*]h]] = o(h", ,a)).
9: Set targets:
s/., = (o(h,,a)) (0h,, a²) + 7 t (h"" (h" 107))
trajectory terminates at history h otherwise
(17)
10:
end for
11: Perform gradient descent step on EB 1109,1- - (h,, ab 10,))lla with respect to parameters
12: i mod M = 0 then 0 = 0f,
13: end for
14: AM
Li=l EaEa ftt a)r(a) kb)
15:
Output: 11'
D FINDING OPTIMAL POLICY FOR GIVEN REWARD WEIGHTS
We use deep recurrent Q-learning Hausknecht & Stone 2015 to find the optimal policy for each setting of the reward weights R(h, = w ((h, a).
For history h, let the next history for taking action be h' = (h, (a) A]). where ({Y(a) h] estimated by the counterfactual model. Using the batch dataset D, we lear the Q-values iteratively by updating the current estimate of the Q- Q-values towards the reward plus the maximum Q-value over all possible actions the new history h/:
Q(h,a) a) Qh, + a(R(h,a) + max Q(h', Q(h,
(18)
where is the learning rate and y is the discount factor.
We approximate Q(h, algg)) using an RNN parameterized by 0g. The RNN consists of an LSTM unit with linear output. We use the standard practices for training Q networks (Mnih al., 2013 Hausknecht & Stone 2015) and we employ a target network Q with parameters O, to provide the updates for the main network and to stabilize training. The target network Q is the same architecture as the main network Q and parameters 0, are updated to match 0, every M iterations.
We use the following loss function for the Q learninn update at iteration i:
(0.) = En~ol(yg, QQh, a 9g
(19)
0g,441 0y (LC,
(20)
where a is the learning rate and
RRh, a), Gg. R(hha) + max, Q(V, dl)0,),
if patient trajectory terminates at history otherwise
(21)
the stale update target obtained from the target network Q. The action a is chosen by following an e-greedy policy by selecting action a = arg max, Q(h, (0,)) with probability 1 - E and a random action with probability We perform M training iterations. Table indicates indictts the search ranges used for the various hyperparameters involved in the deep recurrent Q learnign algorithm. The hyperparameters were selected to ensure stability of training and convergence of the algorithm.
15

---

Published as a conference paper at ICLR 2021
Hyperparameter
Hyperparameter search range
Hyperparameter value
LSTM size
64, 128, 256 128, 256, 512
128
Batch size
256
Learning rate
0.00001, 0.0001, 0.001 100, 200, 500
0.001
Target network update M
100
Min
0.0
Max
0.9
E decay
0.00001
Number of training iterations
20000
Optimizer
Adam
Table 6: Hyperparameters used for training Q-network to find optimal policy for setting of the reward weights.
E DATA SIMULATION
Using the dynamics of the simulated environment described in Equation 22, we trained a deep recurrent Q-learning agent (Hausknecht & Stone 2015) to find stochastic optimal policy that optimizes the reward function for different settings of the reward weights w1 and We in Equation 23.
Let h4 (Iort 20.1, 00:-11) contain the history of the simulated patient covariates representing disease progression x and side effects 2. Let h4+1 = (Foa++1 #0:++1, ag:t) be the new history after taking action a; given h4 and let r, R(h,, az) be the corresponding reward.
We again learn the Q Q- values iteratively by updating the current estimate of the Q Q-values for history he towards the reward plus the maximum Q-value
all possible actions in the next history he+1-
Qhh, a.) + Q(he, ae) + ox a(R(hee at) + 7 max Q(h++1,a') Q(b,, a+))
(22)
where a is the learning rate and y is the given discount factor.
We approximate the action-value function Qh+, ae(0e) using an RNN parameterized by 9.. We use an LSTM (Hochreiter & Schmidhuber 1997) unit part of the RNN with liner output activation for estimating the Q-values. We use the standard practices for training Q Q-networks (Mnih et al. 2013 Hausknecht & Stone 2015).
Let E = (he, ar. Ft, (e4 } be the experience replay memory of maximum capacity E obtained by simulating patient trajectories from the environment. During training, the Q-learning agent selects and executes actions in the environment using an é e-greedy policy that follows the greedy policy a = arg max,' (h, a d'(0.) with probability 1 and selects random action with probability The trajectories obtained from the agent's 's behaviour are added to the experience replay memory which then used to train the Q nettworr. At each training iteration, we sample a batch of examples from E.
Moreover, we use target network Q with parameters 0_ to provide the updates to the main network. The target network Q is the same architecture as the main network Q and its parameters 0- are updated to match 0,, every M iterations. The purpose of the target network to stabilize training.
We use the following loss function for the Q- Q-learning update at iteration i: (0e,s) (e (We.. - Q(he, ar (e,:))
(23)
Gelit1 + te, (LC...
(24)
where a is the learning rate and
ye.s = Tts r + y max a 0
if patient trajectory terminates at timestep otherwise
(25)
is the stale update target obtained from the target network Q We perform M training iterations.
Table/Zindicates the search ranges used for the various hyperparameters involved in the deep recurrent Q Q-learnign algorithm. We selected hyperparameters based on the cummulative reward obtained from the learnt optimal greedy policy in the simulated environment.
16

---

Published as a conference paper at ICLR 2021
Hyperparameter
Hyperparameter search range
Hyperparameter value
LSTM size
64, 128, 256
128
Experience replay memory capacity
10000, 50000 128, 256, 512
10000
Batch size
256
Learning rate
0.00001, 0.0001, 0.001 100, 200, 500
0,001
Target network update M
200
Min
0.0
Max
0,9
& decay
0.00005
Number of training iterations M
40000
Optimizer
Adam
Table 7: Hyperparameters used for training Q-ntwwork solve the simulated environment.
After training the Q network create a batch dataset D containing 10000 trajectories from the following stochastic expert policy:
T(a: he) Berouulii Bemoulli(sigmoid/aa (iig at))
(26)
where is parameter that introduces time-dependent confounding bias in the expert policy.
F IMPLEMENTATION DETAILS CIRL
The CIRL algorithm replies using counterfactual model to estimate the potential outcomes [a] hi] which are used to define the reward and to estimate the feature expectations. For this, we use the Counterfactual Recurrent Network (Bica al. 2020a) which removes the bias from the time-dependend counfounders by building a balancing representation that is invariant to the treatment received by the patient each timestep. Refer (Bica al., 2020a) for details about the model architecture. Note that alternative counterfactual models can be used for this purpose (Lim et al. 2018
For each simulated batch observational dataset, we use 9000 samples for training the Counterfactual Recurrent Network and 1000 for validation (hyperparameter optimization). We perform hyperpa- rameter optimization for the counterfactual model using the hyperparameter search ranges described Table We select the model that has the lowest error in estimating the factual outcomes on the validation dataset.
Table 8: Hyperparameter search range for Counterfactual Recurrent Network. Cs the size of the input and R is the size of the balancing representation built by the Counterfactual Recurrent Network.
Hyperparameter
Hyperparameter search range
Iterations of Hyperparameter Search
Leamiing rate
0.01, 0.001, 0.0001
Minibatch size
64, 128, 256
RNN hidden units
0.5C. 1C, 2C, 3C, 4C 0.5C, 1C, 2C, 3C, 4C
Balancing representation size
FC hidden units
0.SR, IR, 2R, 3R, 4R
RNN dropout probability
0.1, 0.2, 0.3, 0.4, 0.5
In the CIRL algorithm, we set the maximum number of iterations to 50 and the convergence threshold e to 0.001.
The experiments were run on system with 2 NVIDIA K80 Tesla GPUs, 12CPUs, and 112GB of RAM.
17

---

Published as a conference paper at ICLR 2021
G IMPLEMENTATION DETAILS BENCHMARKS
For benchmarks, we integrate as part of the proposed batch inverse reinforcement learning algorithm model-based policy evaluation reinforcement learning methods that use standard supervised learning approaches to estimate the next history (Nagabandi el al., 2018). These methods use standard supervised learning to estimate the next history h' needed for the counterfactual y-learning algorithm and for finding the optimal policy given vector of reward weights. Our MB benchmark uses a standard RNN to estimate the next history and define the policy, while MB(x) uses a multi- layer perceptron (MIP) that only considers the current observations for this purpose. The aim of these benchmarks to highlight the importance of handling the the bias from the time-depdendent confounders and using suitable counterfactual model, but also the importance of handling the patient history.
For each simulated batch observational datasets, we use 9000 of the samples for trainign and 1000 for validation. We chose hyperparameters according to the error of the models in estimating the next patient history the validation set.
The MB (h) benchmark receives as input the patient history he and current action or and estimates the next observations Fz+11 which are used to form the next history he+1- For this purpose, the MB (h) benchmark uses an LSTM unit, with a fully connected layer (FC) on top with ELU activation. The hyperparameter search ranges used for this model are described in Table 9
Table Hyperparameter search range for the RNN. C is the size of the input.
Hyperparameter
Hyperparameter search range
Iterations of Hyperparameter Search
20 0.01, 0.001, 0.0001 64, 128, 256
Learning rate
Minibatch size
RNN hidden units
IC, 2C, 3C, 4C, 5C
FC hidden units
IC, 2C, 3C, 4C, SC
Conversely, the MB(x)) benchmark receives input the current patient observation oa and current
action a and is trained to estimate the next observation I++1- The MB(x) uses a multi-layer perceptron with two fully connected layers and ELU activation. The hyperparameter search ranges used for this model are described in Table [T]
Table 10: Hyperparameter search range for MLP. C is the size of the input.
Hyperparameter
Hyperparameter search range
Iterations of Hyperparameter Search
20 0.01, 0.001, 0.0001
Learning rate
Minibatch size FC hidden units
64, 128, 256
2C, 3C, 4C, 5C
H EVALUATING ACCURACY ON MATCHING EXPERT ACTIONS
The accuracy matching the expert policy is computed follows. Consider running the expert policy ne in the environment and obtaining a test dataset with N trajectories: D, = {(' ( where each Ç¹ (=gr- a ... 27,--1, afr. 2tp)) has length T'. Let å be the action selected by the policy recovered by CIRL for history h4 (2ra: a0a-1)- 1). The accuracy for matching the expert policy is equal to 1 N Ei + EE I/a, Gzl)).
18

---

Published as a conference paper at ICLR 2021
I LIMITATIONS AND INTERACTION BETWEEN COUNTERFACTUAL MODULE AND CIRL
We illustrate in Figure how the different components in method interact and which their
limitations.
h4 = (Io, Dt 0t-1,
Possible limitations:
handling large actions spaces handling high dimensional patient covariates
(a) Counterfactual model
@(h,, at) = E Y [at] ht]
|
Limitations:
(b) CIRL
linear rewards
time-invariant
R(hr, a1) w @hh,, a1)
Preference magnitude
W1
W2
Figure 5: The two different components part of our method for obtaining interpretable parametriza- tions of decision making: (a) Counterfactual model which estimates the potential outcomes for taking action a, given patient history h, Note that we do not propose a new counterfactual estimation algorithm, but instead use already existing methods which could have limitations in terms of how they handle large action spaces and large patient covariates. (b) CIRL algorithm which represents the novelty of this work. We propose incorporating the estimated counterfactuals as part of a method for batch IRL that recover the preferences of the expert over the "what if" patient outcomes. As part of the limitations of CIRL are the use of linear rewards and time-invariant reward weights w.
19


Reviews:
[{"review_id": "yNMLfZnnc30", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of Inverse Reinforcement Learning in a batch setting. In this problem, we are given several trajectories from the expert policy e.g. electronic health records, and our goal is to recover the particular reward function the expert is maximizing without any possibility to solve the forward RL problem. One of the main motivations of this paper is to provide explanations behind the expert’s decision-making.\n\nIn order to solve the batch IRL, first the paper assumes a linear reward function i.e. $R(h_t, a_t) = <w, \\phi(h_t, a_t)>$. This in turn implies that the expected reward under a policy $\\pi$ is the inner product between weight vector $w$ and feature expectation $\\mu^{\\pi}$. With this setup, the goal is to come up with a candidate policy $\\pi$ which is close to the expert's policy $\\pi_E$ in terms of feature expectation.\n\nThe authors propose a new algorithm to learn a policy given a dataset of trajectories from the expert policy. The main component of the algorithm is to learn the feature expectations given a candidate policy, which the authors show can be obtained through a TD learning-based algorithm. With this procedure, the main algorithm is an iterative algorithm, which chooses the current weight vector to be the residual of the observed feature expectation and the convex hull of the feature expectation of the policies found so far. The final policy is the best convex combination of the feature expectations of the candidate policies found in the algorithm.\n\nI think the paper makes interesting contributions to the literature on IRL in the batch setting. However, I do have several questions about the experiments and some of the claims made in the paper.\n\n1.  I don't understand what value the counterfactuals contribute to the paper. I understand that asking about outcomes for a different chosen action leads to thinking about counterfactuals, but the paper isn't using any counterfactual estimation algorithm explicitly. Furthermore, the authors should discuss when the counterfactuals $E[Y_{t+1}(a_t) | h_t]$ are well-defined. Notice that $Y_{t+1}(\\cdot)$ depend on counterfactuals $Y_{t'}(\\cdot)$ through history $h_t$. Are they well-defined? What if $h_t$ is continuous because some $x_{t'}$ is a continuous random variable?\n2.  The authors claim that this work can handle batch IRL setting with history-dependent reward functions. But that comes with a lot of intrinsic assumptions. First of all, $w$ does not change with time, and $\\phi(\\cdot)$ is always $d$-dimensional. This avoids the curse of dimensionality problem as the complexity of the problem does not increase with the length of the history. This also shows up in the experiments section where the reward just depends on the history of length $5$, \n3. The assumption of linear rewards and time-invariant $w$ is very critical in this paper. This should be discussed in more detail as it lets the expected reward to be written in the form of the dot product between w and feature expectation. \n4.  The first simulation uses linear features and also assumes that reward just depends on the current x and v. This makes the feature expectations computations really simple. Ideally, it would have been nice to see high dimensional w, non-linear feature, and dependence of reward on the history of length > 1. \n5. Finally, the authors argue that the use of counterfactuals gives an interpretable parametrization of the expert policy. I don't agree with this statement completely. Isn't it completely driven by the low-dimensional parameter $w$? The use of counterfactuals here is also different than the counterfactuals introduced by Hérnan, and Robins, 2000. Why not first pose counterfactuals for the reward functions $R(h_t,a_t)$ and then consider some restricted low-dimensional marginal models for interpretation?\n\nIn summary, I think the paper chose a great problem as finding interpretable policies in a batch IRL setting would be really useful. However, I do think the ideas are not carried out fully, and some parts of the paper can benefit significantly from better design choices.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YGwj2n-Mdt-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes how to learn cost benefit tradeoffs associated with the expert’s actions. The proposed approach integrates counterfactual reasoning into batch inverse reinforcement learning and offers a sensble framework for defining reward fuctions and tentatively explain how domain expert think and act. The framework is developed for those cases where active interaction with the system under study, i.e., experimentation, is not possible, which is very often the case in healtcare.\nThe paper estimates the effects of different decisions by exploiting the concept of counterfactual to accommodate settings where the policy applied by the expert depends on histories of observations rather than just current states of the system under consideration.\n----------------------------------------------------------------------------------------------------------------------\nReason for Score:\nOverall, I vote for accepting. I like the methodological framework, it is well structured and convincing, even if the basic assumprtions required to make the proposed approach working are strict. Indeed, in many situations it is not clear whether these assumptions are satisfied or not. Furthermore, assumptions can not be tested. The problem tackled is extremely challenging from the theorectical point of view and to the best of my knowledge this is the first paper which tries to explain sequential decisions through counterfactual reasoning and to tackle the batch IRL problem in partially-observable environments.\n----------------------------------------------------------------------------------------------------------------------\nPros.\n1) tackles a very relevant problem, both theoretically and practically.\n2) well structured and written.\n3) methods sound and convincing.\n3) numerial experiments are well designed and results seem to confirm the effectiveness of the proposed appproach.\n----------------------------------------------------------------------------------------------------------------------\nCons.\n1) at page 3 when introducing the value function of a policy, the choice of V is not that clever because early in the paper you let V to be the volume of tumor.\n2) please provide quantitative description of how accuracy is computed in numerical experiments\n3) the explanation components of the proposed approach should be further developed, I see potential there but at the current stage it limits to the weights and not for example tries to investigate whether these weights change during time\n4) I would like to see how the proposed approach scales with the number of covariates of patients.\n----------------------------------------------------------------------------------------------------------------------\nQuestions during rebuttal period: \nPlease address and clarify the cons above \n----------------------------------------------------------------------------------------------------------------------\nI found no typos.\n----------------------------------------------------------------------------------------------------------------------", "extra_scores": {}, "extra_text": {}}, {"review_id": "VvmiBePAuM7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary \n\nThe authors address the problem of comparing the effects of different treatments in medical decision making.  They formulate this as learning a reward function via inverse reinforcement learning in a batch setting, where the only admissible data is in the form of medical records.  Their goal is to develop a framework where they estimate the reward function as well as an optimal policy, and subsequently to compare alternatives (different treatments or abstaining from treatment) for a given history.  They propose to integrate counterfactual reasoning into the process of learning optimal policies and reward functions, which they argue is critical for off-policy evaluation in this setting, the results of which is their method dubbed CIRL.  They present results of two simulation studies, as well as an experiment on the MIMIC III data set.  \n\n## Strong points\n\n1. The authors frame the value of the ability to interrogate preferences of different policies in terms of the value it would bring to comparing data-driven clinical guidelines of when & how to treat. (cf. page 1, Introduction : *Moreover, modeling the reward function of different clinical practitioners can be revealing as to their tendencies to treat various diseases more/less aggressively (Rysavy et al., 2015), which —in combination with patient outcomes—has the potential to inform and update clinical guidelines.*\n\n2. By re-framing batch IRL as counterfactual reasoning, they cast the feature embedding component of the reward function to be learned as a function of possible actions, which yields interpretable results (cf. page 2, Contributions).\n\n\n## Weak points\n\n1. Reading over the paper, it is not clear they make their case for interpretable decision making.  The one strong piece of evidence they rely upon is a simulation study for recovering the parameters of the data generating process in a tumour volume vs harmful side-effects (cf. Figure 3), which shows that CIRL recovers the generating weights of the reward function with lower variance than comparable methods, but this may be due to the agreement of the assumptions CIRL makes on the reward function with that of the function used in their simulation.  It would be more convincing had they also tried more complex reward functions.  \n\n2. I grant that real data (cf. the MIMIC III case study of section 5.2) does not provide reward weights for comparison, but would like to see more in-depth evaluation than the scenario provided.  In this example, the highest proportional weight is given to Temperature and WBC (White Blood-Cell count, a marker of lymphocytic activity), which is consistent with medical guidelines.  But analgesics are also indicated to reduce temperature, and WBC may be high due to viral rather than bacterial infection.  To be more convincing, I would want to see more experiments for other interventions, to see how frequently CIRL is able to recover the desired outcome(s).\n\n\n## Recommendation\n\nIRL is a bit beyond my expertise, so I'll speak mostly to the advances in interpretability.\n\nThe premise in the introduction, that different reward functions of different clinical practitioners (or even different institutions) is not tested, and so it is difficult to assay how useful this would be in a real setting.  To the extent that CIRL can recover accurate estimates of the reward function weights, these are interpretable in the same manner that weights in a GLM are interpretable.  This seems sufficient for comparing the relative frequency of actions chosen by experts given the same history, but I am not convinced it provides an explanation for why experts make their decision (cf. arxiv.org/abs/1606.03490)\n\nThe ability of the method's claim as explainable AI notwithstanding, the blending of counterfactual reasoning with batch IRL is promising, and I recommend the paper be accepted.\n\n\n## Questions for the authors\n\n- The problem formulation in section 3 is a bit light on details with respect to how counterfactual reasoning is present in the definition of the feature representation (cf. Equation 1).  \n\n- I felt as if there was some switches in nomenclature that might help people from adjacent fields better read and understand the paper.  For example, the $\\mu^{\\pi}(h,a)$ are described as \"history action feature expectation\".  However, a friend of mine who read the paper with me recognized the definition of successor representations.  Are these two names for the same quantity? \n\n- Section 4.1 equation (5) describes how the parameters of the recurrent network are used to approximate the feature expectations, but I wonder how efficient this is as a size of the action space A?  The experiments run in the paper feature very small (binary?) actions.\n\n- In the experiments described in Section 5 the weights in the experiments are constrained to have 1-norm < 1, but when the space of possible actions given a history is large, I wonder how can you characterize if a difference in weights between two potential actions is meaningful (i.e statistically significant)?  \n\n## Suggestions for improving the paper\n\n- The definition for feature expectations at the end of section 4.1 are sufficiently similar to the on-policy roll-out estimation of $\\mu$ hat that it bears taking some care here to explain the difference.\n\n- The introduction to Section 4 of the paper lays out the core of Algorithm 1 very clearly, but it would help readability of the paper if the subsection titles for section 4 followed the description of the numbered list in the introduction (e.g 4.1 is entitled \"Counterfactual mu-learning\" rather than \"Estimating feature expectations\" as in the list)", "extra_scores": {}, "extra_text": {}}, {"review_id": "9Xny4R-QbAD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposed a way to learn explanations of expert decisions by modeling their reward function. Different from standard IRL, this main focus is on modeling the preference in the logged decision to answer the \"what-if\" (aka counterfactual reasoning) question. The proposed method considered the expert can take decisions based the history instead of one observation. The proposed approach is run on a simulated and a real medical dataset, showing the effectiveness of the approach.\n\nDetailed comments:\nPros:\n1. This paper is studied an important question that is related to many key applications. Learning explainable reward signals is very useful in practice, for downstream tasks such as learning an explainable AI system and for better incorporating the system with the human expert. Partial observability or the non-Markov decision policy in the data-set is very common in real-world applications but the mainstream of work in RL/IRL focuses on the MDP settings.\n2. Counterfactuals are part of the nature of RL/IRL from a fixed dataset. This paper leverage the idea from counterfactual reasoning and tried to ask and answer the counterfactual in IRL. I think it is a good attempt in an important direction and also help to bring the \nthe gap between causal inference and the RL community.\n\nCons:\n1. Eq 4 gives an updating rule of counterfactual $\\mu$: $\\hat{\\mu}$. As $\\mu$ and $Q$ function can be defined as an expected feature/rewards. Can the definition of $\\hat{\\mu}$ first be written in the expected form and then in a recursive form which gives the (DP) updating rule? In general, it's better to give a more clear definition and explanation of $\\hat{\\mu}$ here.\n2. The empirical study in the MIMIC III dataset is interesting but it seems unclear what the conclusion can be drawn from there. Except for the accuracy of matching expert actions, is there any other further explanation of the learned reward by CIRL, and how does that compare with other IRL methods?\n\nMinor comments:\nIt might worth mentioning that [1] also tried to learn the model to explain the decision in (expert) data and considered partial observability. Maybe this work is more IRL-like and [1] is more like model-based RL.\n\n[1] POPCORN: Partially Observed Prediction Constrained Reinforcement Learning, AISTATS 2020", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a counterfactual approach to interpret aspects within a  sequential decision-making setup. The reviewers have reacted to each others' comments as well as the authors' response to their views. I am recommending acceptance of this paper, as it targets an interesting problem and presents an intriguing approach. I think the community would appreciate further discussing this paper at the conference.
Decision: Accept (Poster)