Learning-to-defer for sequential medical decision-making under uncertainty
Authors: Shalmali Joshi
Sonali Parbhoo
Finale Doshi-Velez
Venue: Accepted by TMLR ()

Abstract: Learning-to-defer is a framework to automatically defer decision-making to a human expert when ML-based decisions are deemed unreliable. Existing learning-to-defer frameworks are not designed for sequential settings. That is, they defer at every instance independently, based on immediate predictions, while ignoring the potential long-term impact of these interventions. As a result, existing frameworks are myopic. Further, they do not defer adaptively, which is crucial when human interventions are costly. In this work, we propose Sequential Learning-to-Defer (SLTD), a framework for learning-to-defer to a domain expert in sequential decision-making settings. Contrary to existing literature, we pose the problem of learning-to-defer as model-based reinforcement learning (RL) to i) account for long-term consequences of ML-based actions using RL and ii) adaptively defer based on the dynamics (model-based). Our proposed framework determines whether to defer (at each time step) by quantifying whether a deferral now will improve the value compared to delaying deferral to the next time step. To quantify the improvement, we account for potential future deferrals. As a result, we learn a pre-emptive deferral policy (i.e. a policy that defers early if using the ML-based policy could worsen long-term outcomes). Our deferral policy is adaptive to the non-stationarity in the dynamics. We demonstrate that adaptive deferral via SLTD provides an improved trade-off between long-term outcomes and deferral frequency on synthetic, semi-synthetic, and real-world data with non-stationary dynamics. Finally, we interpret the deferral decision by decomposing the propagated (long-term) uncertainty around the outcome, to justify the deferral decision.

Published in Transactions on Machine Learning Research (07/2023)
Learning-to-defer for sequential medical decision-making un- der uncertainty
Shalmali Joshit* Columbia University
cuddmali.jostiiBcolumbbi edu
Sonali Parbhoo" Imperiol College London
parthhoo@@mperii.ccm
Finale Doshi- Velez Horvard University
finale Aseas
harvard. Finae@@sas.harvard. edu
Reviewed OpenReview: https: oppenrereeer...iieu. net/ forum? Opn3KnbRSF
Abstract
Learning-to-defer framework to automatically defer decision-making to a human expert when ML-based decisions are deemed unreliable. Existing learning-to-dee to-defer frameworks are not designed for sequential settings. That is, they defer every instance independently, based immediate predictions, while ignoring the potential long-term impact of these interventions. As a result, existing frameworks are myopic. Further, they do not defer adaptively, which is crucial when human interventions are costly. In this work, we propose Sequential Learning-ter to- oo-Defer (SLTD), a framework for learning-to-dee to-defer to a domain expert soquential docision-making settings. Contrary existing literature, we pose the problem of learning- to-defer as model-based reinforcement lcarning (RL) to i) account for long-term consequences of MT--based actions using RL and ii) adaptively defer based on the dynamics mmodel-basedy) Our proposed framework determines whether to defer (at each time step) by quantifying whether deferral now will improve the value compared to delaying deferral to the next. lime step. To quantify the improvement, we account for potential future deferrals. As a result, we learn a pre-emptive deferral policy (i.e. a policy that defers early if using the ML-based policy could worsen long-term outcomes). Our deferral policy is adaptive to the non-stationarity in the dynamics. We demonstrate that adaptive deferral via SLTD provides improved trade-off between long-tern outcomes and deferral frequency on synthetic, semi-synthetic, and real-world data with non-stationary dynamics. Finally, we interpret the deferral decision by decomposing the propagaled (long-term) uncertainly around the outcome, to justify the deferral decision.
1 Introduction
Machine learning (ML) has the potential to be deployed for decision-making in complex domains such healthcare, lending, and legal systems. In many cases, ML-based policy may not generalize situations not encountered during training. In practice, it may be safer to defer to a human expert when using the ML policy may not improve outcomes or cause active harm. Automatically deferring to human expert is called Tearning-to-defer: Earlier works have considered the problem of learning-to-defer in non-sequential settings Mozannar and Sontag 2020 Madras et al. 2017)..
In sins such as managing health, however, two key challenges remain. First, deferral decisions can significantly alter long-teri term outcomes. Thus modeling the long-term outcome is critical to decide when to
"Equal Contribution 'Corresponding Author

---

Published in Transactions on Machine Learning Research (07/2023)
defer expert. Deferring too late may lead to unintended and irreversible harm. Deferring too early may increase the burden on the human expert. Second, when human interventions (after deferral) are costly, learning-to-defer adaptively and only when critical is crucial. To defer adaptively, we need a well-characterized model of the environment, a challenging estimation issue, especially under non-stationarity, i.c., when the dynamics of the environment change over time.
Existing learning-to-defer methods defer based on immediate outcomes e-g. Mozannar and Sontag (2020) Madras et al. (0177; Gennatas et al. (2020), and are therefore myopic. Further, the objective to defer is to improve the performance of some prediction tasks (such as the ability to predict a patient outcome).
These frameworks either defer based on the probability of correct short-term prediction or characterizing the trade-off of paying cost (to defer). Instead, interventions based on an ML system can have long-term consequences that are crucial the model. Further, in many cases, merely deferring to optimize for decision/ prediction accuracy in supervised learning setting does not suffice to improve long-term outcomes. Existing approaches also do not leverage the potential of modeling the environment to defer adaptiuely, especially beneficial if the environment is non-stationary.
Tear suboptimal compared to
Pre-emptively defer expert (SLTD)
Healthy
Monitar
Outcome
Myopic deferral to expert
Adverse
deferral
Eafly defor
Pro pre-emptie defer
Late detor
Figure 1: A conceptual overview of deferral strategies in a medical setting. We show potential patient outcome trajectories over time (x-axis). The outcome is shown the y-axis. A higher outcome is better. A discretized reward is available to us: 'Healthy', 'Continue Monitoring', and 'Adverse.. The white regions indicale regions where Ttar will provide similar recommendations to To (specific actions are not shown). The shaded gray region indicates that at these times, the patient is in states (not shown) where Taar provides rocommendations that do not improve long-term outcomes compared to Ty. The patient follows the gray trajectory from time L = 0 to
when Star's recommendations are used. The red trajectory shows the final outcome we continue to use Ttar till the end of the horizon. One potential deferral strategy is to defer when the observed outcome deteriorates to 'Adverse' (orange trajectory). This deferral is late/mu ate//moopic and the expert policy is unable to significantly improve the patient's outcome in a single time step. This baseline is akin to model where the expert takes over from Wtar when they see that the patient is deteriorating and is not an automated deferral. A netter-automated deferral strategy is to defer when the predicted outcome using Taar is 'Adverse' (yellow trajectory). This deferral strategy corresponds to supervised learning-toddcte methods such as those of Mozannar et. al. and Madras et. al. and is also hite/myyiic as i) it based on immediate predicted outcomes, and ii) does not characterize the regions of the state-space where Taa is worse than mo (gray region) in terms of the long-term patient outcome. On the other hand, the green trajectories indicate long-term outcomes if we defer in the gray region (corresponding to what SLTD and ST.DDSSttioonayy will do), where immediate predicted outcomes are not adverse, but long-term consequences of continuing to use Terr significantly deteriorate the patient. Note that here we do not show the distinction between behaviors due to the non-stationarily of the dynamics. Thus, ideally, a pre-emplive deferral policy should characterize all regions of the state space where long-term outcomes using Taar are worse than no- and defer accordingly. Also note that here, both green trajectorics are pro-emptive according to our sctup though SLTD will defer earlier at t = as well as at t == and then rely on Taar for t 2 In regions markod "Farly defer", we would like to avoid deferrals since mtar is as good as better than an and deferring is unnecessarily costly. SLTD can avoid such premature deferrals by incorporating cost for deferrals.
2

---

Published in Transactions on Machine Learning Research (07/2023)
Algorithmic Motivation. To address these challenges, deviate significantly from existing learning- to-defer methods, which use supervised learning framework. Instead, we model the learning-to-defer problem for sequential settings as offline model-based reinforcement learning (RL). SLTD is the first RL-based learning-to-defe framework. We focus on scttings where online experimentation is prohibitive for safcty reasons, such as healthcare.
We assume access to batch data collected by human experts (such as clinicians) using behavior policy. Our goal is to learn a deferral policy with respect to fixed ML-based policy (called the target policy). SLTD decides whether or not to defer (to expert policy) at each instance by modeling the impact of delaying defernal (by one time step) on the long-term outcomes. SLTD defers if delaying deferral does not improve outcomes compared to deferring in the current instance. To quantify long-term outcomes, we also account for all future deferrals. In doing so, SLTD precisely identifies the regions of the state space where the target MT.-basod policy will not improve outcomes. As a result, our method pre-emptive, i.c., it defers in all regions where the ML policy is unlikely to improve long-term outcomes. Soe Figure for a conceptual overview of SLTD.
Human expert interventions are often costly. Hence deferring too often is not desirable. To defer adaptively, we propose to leverage an estimate of the environment dynamics and the associated uncertainty. Modeling the dynamics allows us to reliably quantify the impact of delaying deferral on long-term outcomes, which is particularly beneficial in non-stationary settings. We show that modeling the non-stationarity provides a better trade-off of improving outcomes versus the frequency of deferrals. In a myopic environment (i.e., when the effect of interventions are observed in the near future), it may seem unnecessary to model the dynamics. However, we demonstrate that deferral methods that defer myopically, based immediate outcomes still benefit from modeling the dynamics, and consequently the impact of potential future myopic deferrals. When SLTD defers, human experts can benefit from an additional justification of the deferral decision to determine potential interventions. Hence, we also interpret SLTD's decision to defer at any given time by quantifying the long-term uncertainty in the outcome and decomposing the sources of uncertainty. We justify how the decomposition can guide experts to potential interventions.
Clinical Motivation. We are motivated by clinical settings where a target policy is learned from batch data to work well across multiple institutions. Such policy may perform well on average, but when deployed to new enviromment, encounter a different or evolving patient. population. An evolving patient physiology can often result in non-stationary dynamics on which the target policy is not uniformly better, necessitating deferral. Clinicians might also follow a slightly different treatment protocol than this learned policy, which could be an auxiliary reason to defer. Most importantly, regulatory constraints may prevent significantly adapting our target policy completely to the new site. Tn this case, it is safer to leverage batch data from the new site to quantify when it is reliable to deploy the learned target policy. In situations where the policy does not improve outcomes compared to the human expert, exacerbated by challenges like non-stationarity, it is safer to defer to human experts. We further propose model-based method motivated by the fact that in many clinical settings, mechanistic models of patient physiology are available. Beyond healthcare, our work is applicable in many safety-focused, data-scarce, non-stationary settings where online policy improvement is not allowed due to ethical or practical constraints.
2 Related Work & Background
Mixture-of-Experre (MoE). Many methods focus on deciding to deploy two or more policies. For example, Jacobs et al.] (1991); Jordan and Jacobs ( (1994) switch between different policies decisionnmmkinn by partitioning the input space into regions assigned to different specialized sub-models. Variants of this framework enforce an explicit preference for a specific expert, e.g., a human expert, and train other experts to complement the human expert (Pradier et al. 2021). In sequential settings, Parbhoo et al. (2017); Gottesman ct al. 2019b; Parbhoo et al. (2018) combine parametric and conpaaraettric experts to learn more accurate estimates of the value function. On the other hand, we focus on deferral to humman experts when future outcomes using the current ML-based policy are potentially undesirable. Further, we defer based on explicitly quantifying the impact of delayed deferral to decide when to defer.
Policy Improvement with Expert Supervision. Sonabend et al. (020)) use hypothesis testing to assess whether, at each state, policy from human expert would improve value estimates over a target policy
3

---

Published in Transactions on Machine Learning Research (07/2023)
during training to improve the target policy. In contrast, work identifies the value of delaging deferral to human expert al lest time. Improvements using expert supervision are unlikely to be always feasible due to safety and regulatory constraints. tearneng-to-defer with respect to a fixed target policy is crucial as a safeguard. Some works focus on safe policy improvement in a non-stationary MDP setting (Chandak et al. 2020blal a)- Chandak et al. (2020a) assume that the non-stalionarity governed by an exogenous process, and so past actions do not impact the underlying non-stationarity. Our work differs in two ways: first, we argue that model misspecification, specifically ignoring non-stationarity induced by (deferral) actions, affects the likelihood of future deferrals. Accounting for this non-stationarity is crucial to avoid costly deferrals. Second, we incorporate human expertise by explicitly measuring the impact of delaying deferral.
Learning-to-dcfer to Human Expertise. Madras et al. (2017 Mozannar and Sontag 2020 proposc supervised models to defer to the expert. Here, the classifiers are trained on the samples of an expert's s decisions. Madras et al. (2017) train separate rejection and prediction function, while Mozannar and Sontag (2020) learn a joint predictor for all targets and deferral. [Mdraas et al. (2017) is conceptually closer to our work but in a non-sequential setting. Other approaches such as Raghu et al. (2019); Wilder et al. (2020) first train a standard classifier on the data and then compute uncertainty estimates for this classifier and the human expert. The models defer to the expert the model is highly uncertain or can significantly benefit. from deferral. Liu et al.] (2021) | incorporate uncertainly in Learning-lo-Defer algorithms for classification tasks. Instead, we focus on tonninn-to-defer in non-stationary, sequential settings. Our work highlights the role that the dynamics of the environment can have on our ability to defer preemptively.
Learning-to-defer as Causal OPE. We pose our Learning-to-defer problem as an offline model-basee reinforcement learning problem. Besides learning, evaluating the utility of policy (on data collected from behavior policy) in an offline manner is called Off-policy Evaluation (OPE) (Precup) 2000). OPE is a challenging problem as the utility of a policy needs to be determined without exploration. Literature on OPE is extensive, summarized in a seminal review of (Uchara et al. 2022). In model-based settings, Importance Sampling (IS), and its variants, and a Direct Method (DM) that directly estimates the Q-function, are the most foundational methods for OPE. Offline learning refers to learning an improved policy in offline scttings. Q-function estimation requires making parametric assumptions of the Action-value functions, while highly flexible parametrizations
prone to overfitting that may not generalize. We strongly believe this is less grounded for healtheare settings. On the other hand, mechanistic models of discase and physiology are often available in clinical settings, motivating our moodel-based approach to offline learning. For evaluating the quality of our proposed method, we rely on either the knowledge of the true dynamics or IS in this work. IS is asymptotically unbiased but can suffer from variance challenges in finite-sample settings. For real-world data, we use the self-normalized variant of importance sampling from Uchara et al. (2022); Precup (2000): Robins et al. (2007) as one of the evaluation metrics. Nonetheless, overlap assumptions are critical to obtaining unbiased OPE estimates using IS, a crucial assumption that may be violated in practice, especially under non-stationarily. Quantifying overlap in data-driven manner is challenging (Oberst et al., 2020). in this work, we assume sufficient overlap between clinician/ /expert policy and behavior policy during evaluation.
Implicit in the framework of OPE, including our work, are assumptions about no hiddon confounding, which are surfaced by a causal framing of the OPE problem (Uchara et al. 2022 Gottesman et al. 2018). This view poses OPE as a causal inference problem given observations from a causal system. The nidhed confounding assumption is primarily because most OPE solutions assume that the data is generated from an MDP which does not allow for potential latent factors to drive decisions. This assumption can have profound consequences on the quality of OPE estimates. In our setup, we also assume that our oflline data is generated from an MDP, thus making the no unobserved bobbservee/hhdden hidden confounding assumption.
Efforts to relax this assumption allow for the presence of latent factors that influcnce docision-making in off-line data and provide OPE estimates that are robust to the variability of this influence (Kallus and Zhou 2018 Tennenholtz et al., 2020: Oberst and Sontag 2019). Often these works focus on parametric assumptions of how much the propensity or the probability of a particular treatment is allowed to deviate, known as the Marginal Structural Model (MSM) assumption (Robins et al.) (2000) provide conservative OPE estimates under worst-case deviations under the MSM assumption. The utility of MSM assumptions
4

---

Published in Transactions on Machine Learning Research (07/2023)
has been significant in epidemiological settings, though relevance in chronic condition management, which is the focus of our clinical setting, is less clear.
Decomposing Uncertainty for Interpreting Policies. Uncertainty, if well calibrated can help decision-
makers understand the failure modes of model (Bhatt et al. 2020 Tomsett et al. 2020 Zhang et al.
2020
Several methods estimate predictive uncertainty in ML (Gal and Ghahramani] 2016: Guo et al. 2017) Here, we focus on capturing the propagated uncertainty in sequential settings to interpret deferral decisions. We interpret the (different) sources of propagated uncertainty when SLTD defers to the expert. Decomposing the sources of uncerfainty into modeling and irreducible uncertainty over predictions has been explored
in classification and prodiction scttings Yao ct al. 2019: Depoweg ct al. 2018) but remains significantly under-explored for sequential settings.
Background and Notation. We consider our enviromment to be a finite horizon MDP defined by where S indicates the state-space, A indicates the action-space, P the transition
M = dynamics, r: 8 a R2+ the reward function, Po the initial state dist.ribttio. The action-space is assumed to be discrete, while the state space can be discrete or continuous. Any intervention policy (usually stochastic in our case) given by π =: S x A + [0, 1]. We consider a non-stationary cnvironment such that the dynamics at any time are governed by a specific MDP M₁ = Pc(s"(s.a) Thus the environment is a sequence of MDPs. We assume the existence of true set of non-stationary dynamies governing all episodes and denote it by M : {Mī In the rest of the draft, M [Me} denotes estimate of the true dynamics M'. Let T be the episode-length. The value of policy x at is given by V-M(s) = EM r a)ls = T. The action value is given by Q,,1(s, a) = r(s,,) a) + Eves Pr(s'|s, (((5'18,0)) VV)
3 Sequential
teerninggto-Deffie
Problem Setup. Assume we are given a policy taar that may be learned from batch data from one more environments. Thar is intended to be deployed in a new environment. We have access to batch data, denoted by D = {s ar,0, "..0,-..,,,,, T,,T, Ji 1/v from some (potentially non-stationary) expert policy To- Note that we assume that Tu is given. For example, it may be the behavior policy from which we have data samples in the target environment. Here N denotes the number of episodes. Our goal is learn deferral policy 9 (s S T (where 1 corresponds to defer or L) with respect to Ttar to defer to the expert policy To. In practice, experts may deviate from To in some cases. In our experiments, we account for this by using an e-greedy version of the behavior policy as to, which serves as a proxy model for such deviation. In addition, a clinician may override a treatment recommendation even when a model does not defer. We account for this using e-greedy version of Tear- For ease of exposition, we still refer to them as To and Tear-
collected in the new non-stationary environment M [M,}]
Deferral to the expert is denoted by the action L. That is, we will augment the action space of existing MDP M to include a new deferral action A := AU L. At every step, the agent decides whether or not to defer. Tf the agent defers, mo will be deployed for that. time step. We describe the formulation assuming strict adherence to To deferral to emphasize other aspects of our contribution such the impact of non-stationarity and how to account for relevant sources of uncertainty to compare outcomes. SLTD can easily account for the uncertainty of expert actions in the framework.
In practice, the target policy Taar may not uniformly improve over To for all states. That is guarantecing that (s) 2 VAM (s) for all s E S, is challenging. Even when fine-tuning is allowed, it is challenging to ensure that the target policy is indeed better than mo all regions of the state space. Hence, we would like to get, the best of both worlds. We can deploy Tear, to reduce the costs of relying on human expertise, and learn to automatically defer to the costlier policy To (i.e. human expert) when relying on Ttar docs not improve outcomes. In regions of the stale-space where the value of Ttar is lower than no, it is better to defer to the human as a "safety protocol". Formally, we only assume that V (s) > V= (s) for some states S.
Type of non-stationarity: Assuming that the non-stationary dynamics are represented by a sequence of MDPs allows the SLTD framework to be general and not restricted to specific forms of non-stationary environments. The main difference between each component in the MDP sequence is that they can be arbitrarily different state- paatetransition dynamics within the same family of distributions (e.g. gaussian or multinomial distributions). As result, this sequence will not share the optimal policy, and hence the optimal is a non-stationary
5

---

Published in Transactions on Machine Learning Research (07/2023)
deterministic policy. When mechanistic models the specific type of non-stationarity are available, they can be incorporated into our framework to provide less conservative deferral policies.
SLTD. To determine whether to defer at each time step, we quantily whether deferring (relying on mo) or not deferring (using * at the current time step improves the long-term outcome. Long-term outcomes are affected by potential future deferrals. Thus comparing the consequences of deferring versus relying the ML policy at the current instance is equivalent to comparing the impact of deferring now versus delaying deferral by one time step.
Future deferrals imply that some unknown mixture of Ttar and To is used in the future. We denote such a mixture policy as Tmix- To minimize cumbersome notation, we denote a policy where Ttar is deployed at instance and miix in the future as: Tear(t), Similarly, if we defer now, then the policy that is deployed at time is To and miix in the future. We denote this mixture as mi(()) Thus, at any instance I, we want to defer if V. is accounted through a constant cost. c > 0 in terms of the value. That. is, deferral incurs cost e and the resulting value is: VM
mixe))) (8) < V.M (oct), (s). Note that we consider deferral to mn as a costly one. This
(s) c. We can now formalize our stochastic deferral policy:
mixt)))
Definition 1. Let Tiar, be such that there erists s² S S E {0, :
, where P(V
mirl)) (s)
VM mi))) (8) c) > for constant cost of deferral e > 0 and threshold T > 0, Vs E s.. Then the deferral
policy I (s. a I[P(VMM marlt. (s) < ViM mii (s) c) > -] - 1 > r].
Corollary 1. By Definition Freee t), includes the earliest time in the episode where Ir. (s, t) P(VM ma(tt))
V.M
mo(s)) > Thus, I (s,t) is pre-emplive deferral policy.
The cost e determines how conservative SLTD is and trades-off frequency of deferral to the value attained. This parameter should be tuned by domain experts aware of the trade-off and risks involved. For instance, in a critical care setting, we may be more conservative and use a smaller c than in a chronic care situation. T is a safety threshold on the probability of worse outcome beyond which we deem that deferral is necessary.
! indicates that to reliably learn the deferral policy, we need to estimate Tra. (s, & P(VM P(VA mix(()) (s) < V mort), (s) 93)-0) c). To estimate this probability, we should model all sources of uncertainty in the system, including the non-stationary dynamics, and the uncertainty associated with our modeling assumptions. We use a Bayesian RL approach to account for all sources of uncertainty. We motivate this by first describing our dynamic programming approach to learn-to-defer.
Definition
Our dynamic programming procedure maintains an estimate of the deferral probability Fr... (s, t) and refines it as we train on the batch data. Given an estimate of @ (s, we outline the procedure to i) estimate the value under mixture policies corresponding to deferral (and delayed deferral), ii) modeling the probability of improvement under various sources of uncortainty, and finally iii) obtaining a new estimate of the deferral probability I=... (s, l)Y's S at the given time using i) and ii). We then bootstrap this procedure over our batch data to refine our deferral probabilities. We describe the procedure for estimating the dynamics in the discrete setting.
Estimating Value function. At any instance we defer based on current estimates of Drw. (s. (or equivalently gr (s, We sample actions from Ttar if g (s, = and To otherwise (equivalent to 1). Note that the current estimate of I (s, t) determines the future mixture policy well. We now estimate the value of the mixture policics using the Bellman Equation of the state and action value functions. For the mixture policy n. 2 mii(()) (corresponding to no deferral at f), the Q-unccton is:
(s, a) = r(s,a) + Σ (8'18, (41 EEES
(1)
and the Value function is:
Ko (8) = Σ
Teunco (am
(2)
aEA
Similarly for the mixture policy if defer at. t.

---

Published in Transactions on Machine Learning Research (07/2023)
Algorithm 1 Sequential Learning to Defer
Input: D", expert policy No, target policy Star-
Estimate Posterior Distributions {M₁ - (1(1 (posteriors over rewards not shown here)
Initialization: Deferral function IT (8, = 0 for all s E S and E (1,2, m. .
for n BOOTSTRAPS(D") do
Sample M₁ _ [Ma, ~ P(([D))]Vt E {1.2, ... (1.2,...,.7 V f1.2, IL.2,...TTT.Vvk
K
for E 1T, ,1 do for E S do
Compute VMM mixrt))
VM mox()))
CAM
Fee (8, t= R EMintbu(1D)?
[1(VM
(arlt))
Va mixc))) c)]
end for end for end for
return Irew (8, (
Estimating the probability of improving outcomes by delaying deferral. At each instance t, for all states s, we can estimate the indicator function I Startt) (s) < V-M moxt)) (s) - c| given an estimate of Ir.r as described above. However, do not have access to the true dynamics M In batch settings, such as ours, oflen estimate the dynamics using maximun-likelihowe estimation. Such methods make specific assumptions about the distribution governing the dynamics. Our assumptions about the dynamics may be incorrect resulting in potential missecification of our dynamics model. This increases the uncertainty in the outcome and potentially over-estimates the probability that relying on the model may improve outcomes. To account for this additional source of uncertainty, we use a Bayesian RL approach. We describe the procedure for the dynamics. The procedure for rewards follows an analogous process.
Suppose the parameters of the distributions governing the dynamics are denoted by 0, Vt E {0...., T). We denote the full set of parameters by 0 = {. We assume prior distribution over the parameters of the distribution governing the dynamics PAM (s'ls, a) and the rewards r(s, a). Given batch samples D", we can estimate the posterior distribution over the non-stationary MDPs and rewards using Bayesian inference:
p(0 p(D')0)p(()
More specifically, we assume conjugate priors for our parameters O. By relying on conjugate priors in our inference, the parameters of posterior distributions over the dynamics and rewards are obtained in closed form. For discrete state dynamics (and rewards), assume Dirichlet prior distribution and model the observations p(D* (0) using Multinomial distribution. For continuous states, p(D" is assumed to be normally distributed with 0 being the mean and variance parameters. The prior distributions over the mean and precision finverse of the variance) is the Normal-gamma prior. This is a domain-dependent choice and SLTD is agnostic so long as we can sample from the posterior distributions of the learned model dynamics. A detailed derivation of how the data leveraged to estimate the postcrior distributions over the dynamics are provided in Appendix A By allowing flexibility of modeling the dynamics via Bayesian RL, we can account. for uncertainty over our modeling assumptions.
Finally, based on our assumption that the non-stationary environment is governed by a sequence of MDPs, estimate the MDP for cach time step independently from batch data. This allows us to make fewer assumptions about the type of non-stationarity. Any additional domain knowledge about the nature of non-stationarity can be leveraged for data efficiency. We can now estimate the impact of delayed deferral by sampling non-stationary MDPs from our posterior distributions and averaging to obtain our final probability:
Foaet (5) < VMM EM M
(s A P(V-
moxe)) (s) c)
mixce) (5) < V-M
Ve mixi))) (s) cl]
(3)
Σ
1[V
seneet) (s)
V-Ma
Vioee).mixcey) (8) c]
K
7

---

Published in Transactions on Machine Learning Research (07/2023)
where the second line comes from the definition due to the randomness over the dynamics, and the last term comes from approximating the expectation using K samples from the posterior distribution of the dynamics p(-1D") Thus, for every instant i, in a given state our deferral policy Im (s, t) is given by, Ircar (s,t) t) = 1 (
(s,t) > T|.
Dynamic Programming to estimate I (s, Our dynamic programming procedure is summarized in Algorithm 1 We initialize Irt (s, t) = for all s E S. We estimate VM
mrar(t)) (s), VMM (oct),meett)) (s) for a given t using Bellman Equations and 2 Following that, we can update our estimate of 9rua (s, using our posterior MDPs, i.e. Equation 3 We repeat (over t) using the updated estimates of Ir (8,(). Note further that To is stochastic. Thus, we do not make explicit assumptions on the specific actions an expert will take in the futher. More specifically, SLTD accounts for the added uncertainty in human's 's actions by explicitly taking expectations over actions a ~ Tо- In our experiments, we use an e-greedy versions of both To and Teari which will allow for further deviations from the expert policy or allowing for overrides, to reflect realistic settings.
Optimality of Learned Deferral Policy. Note that the optimal policy in environment is deter- ministic non-stationary policy. More specifically, the optimal policy is sequence of policies where each component in the sequence is optimal with respect to the specific dynamics at the corresponding time instance. It may not be possible to always reach such an optimal via deferral. Our goal is thus to significantly improve over our target policy Tear by deferring to the expert policy. This could be envisioned as a setup where are restricted to a policy class of offline RL where we are only allowed to learn from families that defer to the expert policy and use taar otherwisc.
4 Decomposing the uncertainty at deferral
SLTD defers at time t because the probability that relying on Tiar improves the outcome is below our safety threshold, i.c. SLTD is uncertain of an improved outcome. Conveying this uncertainty can help the domain expert take over decision-making. We interpret this deferral decision in terms of the total and decomposed uncertainty on long-term outcomes. We convey two different sources of uncertainty at deferral. First, we consider ceiistmic/moddelin uncertainty and the aleatoric uncertainty which mainly results from the stochasticity of the environment itself. A high relative value of the former suggests that adding more data to train SLTD can improve the confidence of the model. High aleatoric uncertainty suggests that the environment itself is highly variable leading the lack of confidence in relying on taar
uncertainty, which captures whether our model specification has resulted in high
Concretely, let to be a time when SLTD defers. The agent is in state Sta" We are interested in the reward (and uncertainty over the reward) at time due to deferral at taz i.c., Birr|sta: pha mottt)))ma((a+))- We denote the posterior MDP samples for any state-action pair by Pt. The variability in these samples captures modeling uncertainty. The dynamics parameters are denoted by 0c(s, a) for each state-action pair. First, sample the parameters of the dynamics from posterior distribution p(O, followed by sampling the MDPs μe ~ p( ((s )). Once we defer, we sample actions from To at time = to and Tmix for t' > ta where the mixture probability is determined by the learned Freee for future deferrals. The expected long-term outcome is given by:
PT
Meal Merl J
r(sT,ar) x
II Pe (evllll)pp (0'(s, aT (av (oooll))pp (0v (D)dddddddd e'ta+11
2ag Pia J9
Integrands are written in short-hand: s = ( "., ST) maintain one estimate of parameter Or and sample K MDPs pe from this distribution. Thus, the epistemic uncertainty we capture is due to the uncertainty over dynamics under fixed parameters. The total uncertainty can now be decomposed using the law of total variance:
St,
(analogously for other quantities). We
Var(rrisur (8tg: D) -eri (Da (Pt 8e, D) + Var Varme (D) (B[(r |Pea Stgp D))
Total Uncertainty
Trreducible/ Aleatorii Uncertainty
Uncartainty
The second term is the variance conditioned on knowledge of the model Mt.. This is the propagated uncertainty due to modeling uncertainty at La and can be reduced by data collection. The first term averages over the
8

---

Published in Transactions on Machine Learning Research (07/2023)
variance due to Heg and captures propagated uncertainty due to aleatoric uncertainty at ta. which conveys stochasticity of the environment itself. This uncertainty can only be reduced by careful interventions at l.- We estimate these using Moote-CCrloo sampling. Additional details on the derivation are provided in Appendix[A] As suggested before, a high propagated epistemic uncertainty conveys that the current uncertainty of model prediction (of the dynamics) is high but could be improved additional data could be collected. High propagated aleatoric uncertainty indicates high variability in the dynamics that can only be reduced with careful interventions and is otherwise not manageable.
5 Experiments
We evaluate SLTD's ability to defer adaptively in sequential settings with respect to a known and fixed Trar to the expert policy Tg- We test the utility of i) deferring basod on long-term outcomes, ii) adaptively deferring by quantifying the impact of delaying deferral, i.e. in regions where delayed deferral can worsen outcomes, iii) modeling the non-stationarity on deferral frequency, iv) quantifying multiple sources of uncertainty to estimate the probability of different outcomes under delayed deferral. We test method on synthetic data, non-stationary diabetes simulatofmoode modified from Chandak et al. (2020b), and real-world HIV data.
Synthetic Data. In this syuthetic simulation, the region of deferral is known apriori by careful design of Wtar- This environment has 8 discrete states and binary actions (1), All samples start at state 0 and progress toward sink state 7. The episode length is 15. State has a low reward (-5) while all other states have a reward of +1. The initial dynamics are set up such that action ag reduces the probability of landing in stage 6, and action increases the probability of reaching state 6. Ttar increases the chances to reach state 6 unfavourably by taking action aj in states 2, 3, 4 when << 5 or t >> 12. We expect to defer in states 2, 3, 4 even though rewards are favorable if a method is pre-emptive. When 5 < t S 12, the dynamics flip such that ag becomes an unfavorable action that increases the probability of landing in 6, while aj reduces this probability. Here, Tear again increases the chances of landing in 6, by taking ap more often in states 2, 3. 4. By flipping the better action to ag in this region, it becomes crucial to estimate the dynamics over predicting the best action. The dynamics are non-stationary and the probability of landing in state 6 progressively increases when 5 S t < 12. For 2 13, the dynamics reset to noise levels at t << 5 adding non-stationarity to the dynamics. Note that the optimal policy is ts, = It's E S,5 < t < 12 and 0 otherwisc. The expert policy is such that:
0.9 i 12, E (2,3, 4]
0.7 if 1 (2,3,4) no(s,t) = p(a1)
0.1 t<5 or >> 12 ES
for all states 8 E S. In this case, pre-emptive deferral will allow us to reach close to the optimal by deferring in states {2, 3, 4].
Real-world simulator: Diabctes Data. We use an open-source implementation of the FDA-approved Type-1 Diabetes Mellitus simulator (TIDMS) for modeling the treatment of Type-1 diabetes. We sample 10 adolescent patient trajectories (episodes) over 24 hours (aggregated at 15 minute intervals). Glucose levels are discretized into 13 states. Combination interventions of insulin and bolus are discretized to generate a total of 25 actions. We introduce non-stationarity in each episode by increasingly changing the adolescent patient's properties to an alternative patient. We enable this by smoothly varying the weighting of the patient parameters over the horizon. While this does not reflect a realistic patient scenario but will nonetheless evaluate the utility of all methods for smoothly transitioning non-stationary environment. The non-stationarity significantly affects the utility of the initial target policy which is learned the dynamics of the original patient, thus necessitating deferral as the patient properties change over time. The non-stationary target policy Ttar for this task is estimated using Q-learning, An e-greedy version of this Q-learned policy is used in our experiments. We defer to a clinician policy, here simulated by learning an e-greedy version of policy learned using Q-learning under (estimated) non-stationary dynamics on the target data. For ovaluation of value post lcarning, we estimate the dynamics on N 1000 patients to remove estimation bias for evaluation purposes. We further provide IS estimates as we discuss in the metrics below.
Jinyn Xie. Singlucose v0.2.1 (2018) [Online]. Available:
Accessed on: 07-24-2011
9

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Real-world: HIV Data. We identified individuals between 18 72 years of age from the EuResist database (Zazzi et al. 2012) comprising of genolype, phenotype, and clinical information of over 65, 000 individuals in response to antiretroviral therapy administered between 1983 2018. We focus on subset of 32, 960 patients' genotype, treatment response, CD4+, and viral load measurements, gender, age, risk group, number of past treatments collected over on average 14 years (aggregated 4 month intervals). Our action space consists of the 25 most frequently occurring drug combinations, while our state space consists of 100 continuous states of cell counts and viral loads. Since the virus evolves in response to drug pressure, the problem is inherently non-stationary. Our data is collected using a standard first-line therapy provided by clinicians. For our first. case study (Case-T), we use candidate clinician-provided policy as data from the first-line therapy, we investigate whether deferring to second-line therapy (To)): as proposed by standard medical guidelines in response to polential drug resistance[Sang et al., 2020), improves long-term outcomes. The non-stationary behavior policy the first line therapy estimated using Q-learning, For our second case study (Case-II), data is collected from a non-stationary behavior policy which corresponds first-line therapy typically used for treating patients of sublype C. Using the same candidate Taar in Case I, we then examine whether deferring to first-line therapy, given by clinical collaborators, for patients of subtype M (due to potential drug resistance) improves long-term outcomes.
Given
SUTD
SLTO Stat.
SLTD-One Step
Figure 2: Learned deferral probabilities for SLTD (top row), SLTD stationary (second row), SLTD one-step (third row), and Augmented-MDP (red dotted line) for = 0. Each row is method (Row is SLTD, Row 2 is SLTD-Stationary, and Row 3 is SLTD-One Step). We are plotting Oreer (s, which is function of states and time t. Since this is function of two dimensions, we represent the variation along in a row and then for each fixed f, we show the different values this function can take for different states s. According to Synthetic data design, target policy always takes suboptimal actions the yellow region. That is, optimal deferral policy is defer for states 2,3, for all 0 15. That is, (8, t) 1A' (2,3,4] 4} and 0 ft$ 15 and 0 otherwise. Further, the dynamics change time that the optimal action flips, as well the noise increases when S S 12 requires deferral more often. Thus, shaded yellow regions are the region of pre-emptive deferral. SLTD, which models non-stationarity defers adaptively and early in the shaded yellow region (top row) and increases the deferral probability when dynamics change. SL.DDsttationayy does not learn calibrated probabilities in the yellow region over time and only defers when the average dynamics the environment require deferral. SLTD-one-step and Augmented MDP (dotted red line) only defer state when the reward is negative and is not pre-emptive.
Baselines. We compare to the following basclines.
Mozannar and Sontag 2020 This is a supervised method using consistent loss function to learn-to-defer. It learns an augmented regressor to defer or recommend treatment myopically (independently at every time step). When the model defers, the clinician's treatment recommendation is used.
Mozannar et. al.
Madras ct. al. Madras al.) 2017): This an alternative supervised learning-to-defer method. This baseline learns separate regressors defer and recommend treatments. We modify to Tuar to recommend and learn the rejection function to defer to mo. Note that both the supervised learning-ton tearing-to-ddfer to-defer methods are trained to predict action targets.
A Augmented-MDP: A conceptual contribution of SLTD is to defer by comparing outcomes by delaying deferral with some knowledge of the expert policy. The deferral action itself is considered to augment the MDP action space. We explore bascline that uses Value Iteration in this augmented MDP. Comparing with this baseline helps evaluate the utility of deferring based on outcomes of delayed versus immediate deferral. This baseline will defer permanently to the expert, and knowledge of expert policy is not assumed. This
10

---

Published in Transactions on Machine Learning Research (07/2023)
augmented MDP has action-space is AJ I, augmented state-space S U Sdefer (Sdefer is the deferred state), and defers based on the cost This baseline models non-stationary dynamics, and is designed for sequential settings. However since this method defers permanently to the expert, it incurs a larger deferral cost. In our experiments, all values are plotted without the cost to reflect actual environment outcomes.
SLTD-Stationary: To assess the impact of misspecifying the non-stationarity, we compare to a variant of SLTD that assumes the dynamies (and rewards) are stationary while allowing the method the flexibility of learning a non-stationary deferral policy.
We compare to a myopic version of SLTD that defers based on the immediate reward. The key difference with the myopic Madras et. al., Mozannar et. al. baselines is that SLTD-One Step models the dynamics and the uncertainty on the immediate reward. Thus, this baseline accounts for future deferrals while deferring myopically.
SLTD-One Step:
Ablations for Uncertainty Modeling: For all SLTD variants, evaluate the utility of accounting for different sources of uncertainty, more specifically the modeling uncertainly to estimate the probability of improving outcomes via delayed deferral. In SLTD, modeling uncertainty is accounted for by sampling multiple (K) MDPs (Equation 3) from the posterior dynamics distribution, over which our outcomes are averagod. Higher variability across K indicates higher modeling uncertainty. Hence, in Equation B using K = assumes a perfect estimate of the dynamics model and only accounts for the irreducible stochasticity of the environment. A larger K accounts for potential variability in estimation (original SLTD formulation). If our modcling uncertainty in the environment is indeed large, we anticipate the choice of K to have larger effect on SLTD's performance. Modeling uncertainty can be large when there is insufficient data to fit the target function class of the dynamics.
5.1 Evaluation Metrics.
We use the following evaluation metrics for assessing our proposed models.
Value Estimation. Once a deferral policy is learned, evaluating its utility using offine data is the problem of off-policy evaluation (OPE). We use the following estimation procedure for evaluating the Value of a deferral policy:
1. Rollout with true dynamics: For Synthetic data and Diabctes, we have access to the true dynamics, although Diabetes data may be prone lo discretization error. Here can roll out the trajectories under the true dynamics using deferral and collect the true value estimates. In our experiments, we collect 10000 trajectorics for all datasots.
Rollout with estimated dynamics: For HIV data, we do not have access to the truc dynamics. Hence we use the same procedure above but with estimated dynamics, which are estimated using Maximum-Likellhood. This evaluation can be used when there is sufficient domain knowledge to rely on the utility of MLE estimates of dynamics. In our HIV data, the MLE estimates are considered state-of-the the-art in clinical knowledge of HIV treatment Parbhoo et al. (0177). However, in general, relying on an ML. estimate is prone to a biased value estimate.
3. Self-normalized Importance Sampling (TS): For real-world data where true dynamics are unavailable and relying on MLE estimates of the dynamics may lead to bias, we provide an IS estimate of the value. This metric will likely be used in most real-world scenarios while choosing the best deferral policy. Note that IS is only asymptotically unbiased and suffers from large variance issues. Hence we use a Self-normalized IS estimate. Note that IS can only be used when the assumption of overlap well as no unobserved confounding holds.
Deferral Frequency. In order to assess the trade-off of deferral frequeney and value al tained, we use the true dynamics for Synthetic data and Diabetes data to roll out trajectories and estimate deferral frequency. For HIV data, we use the Maximu--Likelihodd dynamics to roll out trajectorics to estimate deferral frequency based on the consistency of value estimates of Self normaazor IS and MLE dynamics, which then allows us use the ML.F. dynamics to roll out trajectories. Please see further justification of this choice in Section D
11

---

Published in Transactions on Machine Learning Research (07/2023)
Method
Syrthete
Dinbetes
Caue-1
HIV Chawsl
Value
Sell-Nendiiasl 18 kx.)
Value extizaile
Sell-Nurmaliad 18 [suma
Value socimate
SelfNornadiedd 2
Value oviimate using dyyeemeee (mean
Sul--oenndien IS
dynamics
true dytammic
dynamiis (meen Ram)]
2
91/11
Dott
0.073
16.157
51/11>-tat
0.133
1,120
0.371
7-50++113
17.196 0.120
17,261
17,859
0.597
S1,10- Stop
6,224 4: 00020
1,225 0.015
18 0.105
L0ee (05
9.90
10,703 0.301
7.303 0,001
13,227 0,374
51/11>
(K-1)
03377
3.819 0.125
1888+000
0:50++01
11:150 0.317
12.15
0.001
SLTTD-Stut
(K=1)
0.155 + 0.000
1.005 +: 0.265
3.054 1.861
17e+000
1.06++05
6.527 0.017
8.600
0.182
18.551
0.236
17.877 + 0.157
S1.T1>-Dn Step (K-1)
6,333 +: 00030
-1.17U 0.003
16 15.061 +: 9.830
-BI 000++00 4 4.tte+Ut
6.173 0.215
6:208 4: 0.530
7:336 0.142
4:178 3: 0.131
Angnenedd-MTP
3.611 + 0.038
9.781 + 0.000
-3.712 0.009
-2:66e+000 0.000
NJA
NJA
N/A
N/A
Mononnat 0s, al.
7,828 4: 0.163
4.113 +: 10073
-1,553 = 10:811000
-2:37e (06 1.300 (04
6.857 +: 0.205
2.121 0.381
8,259 0.123
3.36s #: 0:251
Mudeas oi. sl.
3.315 1: 0.202
4.745 0.065
35.955 =: 0.488
14.48 0.026
9.288 4: 0.301
0.520 0.116
4.881 0.117
4.773 +: 0.027
Fox
1,325 4: (1005
-1.397 = 0,000
.0.771 = 0.003
-2.44 4ie (06 0,000
4.122 s. 0.237
3.120 0.138
3.186 0.310
3.200 #: 1:212
0.522 +: 0.044
11,564 4: 0.000
$6.106 + 0.11d
3.086 4: 0.000
8.815 #: 0.027
0.168 4: 6.0.5
7,807 0.130
4,541 +: 0.010
Table 1: Expected Value of SLTD compared with basclines for Synthetic data, Diabetes, and HIV data. i) True dynamics (first column), ii) Self-normalized Importance Sampling estimate (second column). For Synthetic data, IS estimate is biased relative to the true dynamics suggesting IS estimates may not be reliable due to finite samples. The best value according to all methods for Diabetes is Madras et. al. suggest.ing benefits to myopic deferral for this data. However, the frequency trade-off shown in Table shows that Madras ct. al. achioves this value when it defers (always) to the clinician. SLTD is competitive for Synthotic data and outperforms all other methods for HIV data. According to IS estimates, Madras et. al. provides the best value for Synthetic data. For Diabetes, severe overlap issues result in unreliable IS estimates, except for Madras et. al. which matches To as it completely defers to the clinician. IS-estimates are consistent with MLE-based estimates for HIV where SLTD consistently outperforms all basclines. Figure 3 shows the deferral frequency versus value trade-off for all datasets.
6 Results
Optimizing for long-term outcomes learns qualitatively different deferral policies. Our deferral policy is non-stationary stochastic function In... (s. which threshold. Visualizing (s, enables us to understand the utility of various modeling choices of SLTD. Figure 2 shows the histograms of Zn... for SLTD and its Stationary and One-Step variant when the cost c = 0 for Synthetic data. Visualizing without deferral cost allows us to see how adaptive SLTD is without a penalty. Each row corresponds to a method; the x-axis corresponds to time over the horizon T. Each box in a row corresponds to a single time point. For a fixed 1, Ir... is a stochastic function of the states, shown as a histogram.
The yellow shaded region indicates the state space where Tear takes unfavorable actions. Over time, the dynamics change so that the favorable action flips 5 S 1 S 12 and the stochasticity in the dynamics increases requiring more frequent deferrals. Deferring in the yellow region desirable to pre-emptively avoid landing in state of 6. SLTD is highly adaptive, and pro-emptively defers in the yellow region. As the stochasticity increases, the probability of deferrals appropriately increases. The stationary variant significantly underestimates the need to defer in states 2,3,4 4 when t < 5. It is only able to pre-emptively defer in regions where the average stochasticity of the estimated dynamics aligns with the environment. The One-Step variant defers only in state and is therefore not preemptive. Augmented-MDP (red vertical line) deterministically defers in state 6. Thus deferring based on the probability of improved outcomes of immediate and delayed deferrals is desirable over alternatives (see also Figures and in Appendix for Diabetes and HIV data).
It is important to note that SLTD lowers deferral frequency for 2 12. The reason for this is two-fold. First, it is an artifact of the synthetic data that we use to evaluate SLTD. More specifically, there is an additional stop adding noise to our dynamics, which resets
form of non-stationarity that kicks in at t > 12 where the dynamics to where they were at < 5. As a result, the deferral frequency is expected to be lower than at 5 S 1 < 12. Second, since SLTD is designed in an oflline learning environment, the support of the data matters (for any offline learning algorithm). For t >> 12, the offline data collected does not have significant support over the "deferral" states [2, 3, 4), which induces an estimation issuc. While we expect this to be compensated by uncertainty modeling that we incorporate, lack of data support is a much more fundamental problem in offline learning, and we anticipate requiring better prior knowledge of the dynamics to completely
12

---

Published in Transactions on Machine Learning Research (07/2023)
overcome the bias that is introduced in the deferral frequency. This bias further lowers deferral frequencies even more compared to I < 5. Nonetheless, the learned deferral policy improves as can be seen from Table [ and noting that the value estimate we have from the optimal policy for Synthetic data was estimated to be v 8. Mozannar et. al. is able to achieve close to optimal by deferring over 80% of the time.
Method
SLTD
AATD-Stationary
SLTD-On Step
Augmented-MDP
Mozannar
Madras et
Clinician
Behavior
Hy
Ooter Frequency
Preqpeecy
The trade-off of deferral frequency and value attained. Synthetic data and Diabetes use true dynamies, while HIV uses Maximuu-Likelihood estimates for value estimates. The plot shows the expected value (higher is better) for a sweep over deferral cost (corresponding to best performing hyperparameters for each baseline) to the deferral frequency (lower is better). SLTD achieves the best trade-off and better value by deferring pro-emptively. The stationary variant is unable to attain high values for the same deferral frequency. The One-Step variant. cannot, improve over Ttar in MDPs where the effect of interventions is not, myopic (Syuthetic and HIV) and achieves good performance by modeling the dynamics in Diabetes, compared to Mozannar ct. al.. Madras et. al. baseline performs well on Diabetes data suggesting this data is myopic, though non-stationary (considering deterioration of SLTD -aaD--Stti,,,ry,, and the high deferral frequency of Madras et. al. baseline. Augmented-MIPP is unable to achieve good performance indicating the benefits of deferring by explicitly quantifying the impact of delaying deferral as in SLTD over Augmented-MDDE Value Iteration method. SLTD and the stationary variant achieve the best trade-offs in HIV data. Deferral trade-offs when relying on Self-normalized IS for model-selection are provided in Tables and in the Appendix.
Figure 3:
SLTD improves long-term outcomes. Table shows the value (higher better) for all baselines using the different value estimation methods outlined above. We have shown the highest value for each value estimate independently. For HIV data, we do not see significant differences between MLE dynamics and IS evaluation metrics. SLTD unanimously outperforms all basclines. Table D shows significant biases in the IS estimate compared to the true dynamics for Synthetic data data. Based on IS estimates, Madras et. al. performs the best for Synthetic data. Table S in Appendix shows the frequency of deferrals and the corresponding cost hyperparameter, IS value estimates are used to select the best deferral methods. This performance is obtained when there is no cost to deferring (c 0) resulting in deferral frequency of over 50%. For e >>0 O, we see a drop in value to less than for Madras et. al. (see Figure 3). This suggests that, SLTD is still efficient and able to reach a high value for fewer deferrals in Synthetic data data. For Diabetes data, Madras ct. al. performs the best according both nelf-normalized IS and using the true dynamics. However, Figure B suggests that the Madras et. al. baseline always defers to clinician and never has a deferral frequency less than 1. Due to significant overlap issues, TS-estimates for Diabetes are unreliable except for Madras et. al. which matches clinician policy (due to permanent deferral). As such there are untestable overlap issues using IS that need further attention in such non-stationary environments.
Based on the true dynamics, Augmented-MDP baseline is not preemptive despite modeling the non-stationary dynamics. Hence, deferring by comparing outcomes of delayed deferral is a better alternative, specifically Value Iteration used in Augmented-MDP. Additional benefits of modeling the dynamics are clear from the improved performance of all SLTD variants compared to the myopic Mozannar et. al. baseline when > 0. Mis-specification of dynamics (SLTD-Stat.) results in worse performance. SLTD-Stat. defers more often to achieve comparable performance. Figure 3 demonstrates this trade-off is general, for all choices of deferral costs and other parameters. In Figure 3 the x-axis corresponds to deferral frequency (lower is better) and y-axis the value attained (higher is better). For Synthotic data and Diabctes value estimates using the true dynamics are shown. For HIV data, since we do not have access to the true dynamies, the deferral frequency is estimated using MLE istimate of the dynamics. From Table 1 we can see that all value estimation
13

---

Published in Transactions on Machine Learning Research (07/2023)
procedurs provide comparable estimates for HIV. SLTD achieves the best trade-off for Synthetic data and HIV.
Further, pre-emptive deferral allows SLTD to reach close to optimal (value average obtained is 7.0) in Synthetic data, though Mozannar et. al. achieves higher value when deferral cost is 0. SLTD--Oe Step only relies immediate rewards failing to improve long-term outcomes for HIV. However, as long as model the dynamics appropriately, even myopic deferral using SLTD-One Step is beneficial for Diabeles compared to Mozannar et. al.. This is possible when the effect of interventions is observed myopically, as is the case in Diabetes data since modcling the dynamics and impact of future deferral is beneficial to charactorize. Madras et. al. performs well on Diabeles data suggesting optimal actions don't 't significantly deviate in larget data and that its design of training a rejection function worked better than the loss function design of Mozannar et. al.
Defer Time
Total Uncertainty
Modeling Uneertainty
Mewn
Outcome
Synthotic data Diabetes
3
26,190
0.233
3.42
3
3118.17
31.160
73.669
Table 2: Interpreting first time of deferral for a sample trajectory. Modeling uncertainty remains low in all cases whereas in comparison, total variance is high. This indicates irreducible stochasticily of the dynamics is the primary source of uncertainty. Additional results are in Appendix.
Ablations for uncertainty modeling. We study the utility of accounting for modeling uncertainty in our framework. As described in Section 4 multiple sources of propagated uncertainty contribute to variability in estimated outcomes. Modeling uncertainty is crucial to account for in model-based framework. Here we evaluate the impact of not accounting for this uncertainty on SLTD's performance.
If modcling uncertainty is high, variability of the sampled MDPs used to estimate Equation B will be higher. Evaluating for K =1,, will evaluate the impact of ignoring this uncertainty. In Table 1 (see also Figure g in Appendix), demonstrate the results with K = 1 for all SLTD variants. We do not observe significant differences for Synthetic data and Diabetes indicating that our modeling uncertainty is low in these data. The difference is higher in HIV suggesting the importance of accounting for this uncertainly for real-world HIV data. Such analysis is crucial to understanding whether our modeling assumptions are reasonable.
Decomposing uncertainty in SLTD can help interpret deferral. Conveying the type of uncertainty to domain expert can help identify the dominant source of uncertainty that resulted in deferral to their standard practice (expert policy). Table 2 shows this decompooition for one limepoint for discrete data. In each case, the modeling uncertainty is a small fraction of the total uncertainty. This suggests that systematic non-stationarity is the dominant source of uncortainty which generally cannot be reduced by collecting data and may require careful interventions beyond the standard policy. Knowledge of the amount of model uncertainty can enable users to further improve decision-making through data collection or improving model assumptions.
7 Discussion
We proposed SLTD, a learning-to-dedt framework for sequential settings using offline model-based RL. We learn a deferral policy by quantifying the impact of delaying deferral to the future. SLTD can defer based on long-term outcomes and learns a pre-emptive deferral policy. Further, we emphasize a model-based RL method that captures the dynamics of the environment, particularly non-stationarity. Modeling the non-stationarity of the environment allows deferring adaptively. Misspecifying non-stationarity leads to significantly more deferrals to improve long-term outcomes. We demonstrate that cxisting learning-to-dcfer frameworks are myopic. That is, these methods do not. learn a pre-emptive policy even in sequential set ings as they focus on the immediate consequences actions. We further demonstrate the utility of accounting for all potential sources of stochasticity to quantify the impact of delayed deforral. Explicit charactcrization the probability of improving outcomes beneficial to prevent over-estimation of the benefits of delaying deferral. We further interpret deferral decisions of SLTD by decomposing the long-term propagated uncertainty.
14

---

Published in Transactions on Machine Learning Research (07/2023)
Limitations and Future Work. While quantifying the uncertainty is useful, especially to compensate for the fundamental challenge of data support in offline reinforcement learning, modeling uncertainty through non- stationarity is costly. Developing a model-free framework is an important aspect of future work. Uncertainty quantification may not be able to compensate for cases of severe data support issues, which may result in biases in the learned SLTD policy. In this case, better prior knowledge of the dynamics is necessary. SLTD assumes no hidden/ /nobserved confounding by relying on an MDP data-generating assumption. Thus our current. uncertainty quantification does not account for the added epistemic uncertainty due to potential hidden confounding. Incorporating clinically motivated sensitivity models to account for unobserved confounding is active area of our future work. SLTD can account for some deviations from the expert policy, as well as the potential for its recommendations to be overridden, though significant deviations should be modeled as an online human-in-the-looy system. In this case, SLTD can serve as a reliable warm-start policy that could be further improved using (ouline) human input. We also posit that our current e-greedy version will prove to be more consorvative comparod to such an online framework, as we expect real human dccisions to be more informed (modulo their own biases) than noise model that is e-greedy. Nonetheless, rigorously testing this in-the-loop Bearning-to-dffe framework is left to future work. Finally, note that if the type of non-stationarity we would like to defer against is not observed in the offline data, the deferral policy may over werrunderstimate the need to defer.
Ethical considerations. SLTD is a technical proof-of-concept to defer to an expert by accounting for long-term effects, assuming that the expert is better at increasing value over the current policy in certain regions. In practice, an expert policy may not be bias free. Thus, deferral may result in biased decisions if the expert is biased. Such bias may be exacerbated due to potential sources of hidden confounding (Gottesman) ct al. 2018). While introspection. Nonetheless, deferring is better when an automated decision may be harmful.
not focused addressing bias, exposing uncortaintics may encourage expert
8 Acknowledgements
This material is based upon work supported by the Center for Research on Computation and Society (CRCS) at Harvard University, the National Science Foundation under Grant No. IIS-2007076. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not nocossarily refloct the views of the funding agencies.
References
Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Gauthier Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, et al. Uncertainty as a form of transparency: Measuring, communicating, and using uncertainty. arXiv preprint arXiv:201..07566, 2020.
Yash Chandak, Scott M Jordan, Georgios Theocharous, Martha While, and Philip Thomas. Towards sale policy improvement for non-stationary mdps. arXiv preprint arXiv:2010.126/5 2020a.
Yash Chandak, Georgios Theocharous, Shiv Shankar, Martha White, Sridhar Mahadevan, and Philip Thomas. Optimizing for the future in non-stationary mdps. In International Conference on Machine Learning, pages 1414-1425. PMLR, 2020b.
Stefan Depeweg, Jose-Miguel Hornandoz-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pages 1184 1193. PMLR, 2018
Daniel Fink. A compendium of conjugate priors. edu/ http///ge////////ON//NG RONJIN 20TEX. pdf, 46, 1997.
See http://www. people. cornell.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1059, New York, New York, USA, 20 Jun 2016. PMLR. URL http::/proceediug mir, air.press/ #48/gal16.cm
15

---

Published in Transactions on Machine Learning Research (07/2023)
Efstathios D. Gennatas, Jerome H. Friedman, Lyle H. Ungar, Romain Pirracchio, Eric Eaton, Lara G. Reichmann, Yannet Interian, José Marcio Luna, Charles B. Simone, Andrew Auerbach, Flier Delgado, Mark J. van der Laan, Timothy D. Solberg, and Gilmer Valdes. Expert-augmented machine learning. Proceedings of the National Academy of Sciences, 117, 2020.
Omer Gottesman, Fredrik Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, Yi Ding, David Wihl, Xuefeng Peng, et al. Evaluating reinforcement learning algorithms in observational health settings. arXiv preprint arXiv: 180..1298, 2018.
Omer Gottesman, Yao Liu, Scott Sussex, Emma Brunskill, and Finale Doshi- Velez. Combining parametric and nonparametric models for off- off-policy evaluation. In International Conference on Machine Learning, pages 2366-2375. PMLR, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321-1331 PMLR, 2017.
Robert A Jacobs, Michael I Jordan, Steven Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79 87, 1991.
Michael 1 Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Newral compulation, 6(2):181 214, 1994.
Nathan Kallus and Angela Zhou. ronfounding-robust policy improvement. Advonces in neural information processing systems, 31, 2018.
Jessie Liu, Blanca Gallego, and Sebastiano Barbieri. Incorporating uncertainty in learning to defer algorithms for safe computer-aided diagnosis. Xi preprint arXim:2108. 08.0999, 2021.
David Madras, Toniann Pitassi, and Richard Zemel. Predict responsibly: improving fairness and accuracy by learning to defer. arXiv preprint arXio:1711.0660, 2017.
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. ar preprint arXiv:2006.00 01862, 2020.
Michael Oberst and David Sontag, Counterfactual off-policy evaluation with gumbel-max structural causal models. In International Conference on Machine Learning, pages 4881-4890. PMLR, 2019.
Michael Oberst, Fredrik Johansson, Dennis Wei, Tian Gao, Gabriel Brat, David Sontag, and Kush Varshney. Characterization of overlap in observational studies. In International Conference on Artificial Intelligence and Statistics, pages 788-798. PMLR, 2020.
Sonali Parbhoo, Jasmina Bogojeska, Maurizio Zazzi, Volker Roth, and Finale Delei-Velez. Combining kernel and model based learning for hiv therapy selection. AMIA Summits on Translational Science Proceedings, 2017:239, 2017.
Sonali Parbhoo, Omer Gottesman, Andrew Slavin Ross, Matthieu Komorowski, Aldo Faisal, Isabella Bon, Volker Roth, and Finale Velei-Velez. Improving counterfactual reasoning with kernelised dynamic mixing models. PloS one, 13(11)=e0205839 2018.
Mclanic Pradicr, Javier Zazo, Sonali Parbhoo, Roy Perlis, Maurizio Zazzi, and Finale Velci.Vllzz. Preferential mixture-of-experts Interpretable models that rely on human expertise as much as possible. arXiv preprint arxiv:2101.1 0130660, 2021.
Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000.
Maithra Raghu, Katy Blumer, Greg Corrado, Jon Kleinberg, Ziad Obermeyer, and Sendhil Mullainathan. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903. 1900.1229....... 2019.
16

---

Published in Transactions on Machine Learning Research (07/2023)
James Robins, Mariela Sued, Quanhong Lei-Gomee and Andrea Rotnitzky. Comment: Performance of double-robust estimators when' inverse probability" weights are highly variable. Stabistical Science, 22(4): 544 559, 2007.
James M Robins, Miguel Angel Hernan, and Babette Brumback. Marginal structural models and causal inference in epidemiology. Epidemiology, pages 550-560, 2000.
Michael Saag, Rajesh T Gandhi, Jennifer Hoy, Raphael J Landovitz, Melanie A Thompson, Paul Sax, Davey M Smith, Constance A Benson, Susan P Buchbinder, Carlos Del Rio, Antiretroviral drugs for treatment and prevention of hiv infection in adults: 2020 recommendations of the international antiviral socicty-usa pancl. Jama, (24
2020.
Aaron Sonabend, Junwei Lu, Leo Authony Celi, Tianxi Cai, and Peter Szolovits. Expert-superrised reinforcement learning for offline policy learning and evaluation. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balean, and H. Lin, editors, Advonces in Neural Information Processing Systems, volume 33, pages 18967-18977. Curran Associates, Inc., 2020.
Guy Tennenholtz, Uri Shalit, and Shie Mannor. Off-policy evaluation in partially observable environments. AAAT, pages 10276-10283, 2020.
Richard Tomsett, Alun Preece, Dave Braines, Federico Cerutti, Supriyo Chakraborty, Mani Srivastava, Gavin Pearson, and Lance Kaplan. Rapid trust calibration through interpretable and uncertainty-aware ai. Patterns, 14):10004,, 2020.
Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. A review of off-policy evaluation in reinforcement learning. arXiv preprint ar 200.2255,, 2022.
Bryan Wilder, Eric Horvitz, and Ece Kamar. Learning to complement humans. arXiv preprint 20X55:2005.00582, 2020.
Jiayu Yao, Weiwei Pan, Soumya Ghosh, and Finale Doshi Velez. Quality of uncertainty quantification for bayesian neural network inference. arXie preprint or Xiv:11100 09686, 2019.
Maurizio Zazzi, Francesca Incardona, Michal Rosen-Zvi, Mattia Prosperi, Thomas Lengauer, Andre Altmann, Anders Sonnerborg, Tamar Lavee, Eugen Schülter, and Rolf Kaiser. Predicting response to antiretroviral treatment by machine learning: the euresist project. Intervirology. 55(2):1233 2012.
Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. Effect of confidence and explanation accuracy and trust calibration in ai-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 295 305, 2020.
17


Reviews:
[{"review_id": "zc2EfxGxKe", "reviewer": "Reviewer_nvhL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths: \n\n- Research on making the deployment of new clinical policy more practical is important, and the learning-to-defer framework fits broadly with this intention (although the precise implementation of it may need more work). \n\n- The contributions are mostly well-presented and easy to follow. \n\n## Weaknesses: \n\n- The intended application is not convincing to me, since it forces a clinician to alternate between giving advice to a machine and letting the machine decide what to do. This seems infeasible to implement in practice, which is the main motivation behind this work. In case the clinician can override the machine's decision (in case it didn't defer), the machine's model is incorrect (since this action is not part of the model) and the value of the future mixed policy is inaccurate. This is a big difference between the present work and the baseline models used for comparison: a) these aim only to do prediction, not decision-making and b) once the prediction is made, the expert/user can do whatever they want with it. \n\n- The paper mostly ignores the huge body of work on (off-policy) policy evaluation (OPPE), see for example \"A Review of Off-Policy Evaluation in Reinforcement Learning\" (Uehara et al., 2022). This is unfortunate since the present method requires solving the OPPE problem in Algorithm 1, or alternatively, the off-policy policy optimization problem. In particular, success of Algorithm 1 relies on a highly accurate model M of system dynamics and responses to decisions. Finding such a model is no trivial feat and relies on strong assumptions such as causal sufficiency/exchangeability/ignorability, common support for actions, etc. In fact, the topic of causality is completely absent in the present work, which renders the method invalid unless additional assumptions (such as the above) are made. Ways of recovering from violations of such assumptions in the context of policy improvement have already been studied, see e.g., \"Confounding-Robust Policy Improvement\" (Kallus & Zhou, 2018). Bayesian modelling does not remove selection bias due to unobserved confounding and any measures of model uncertainty should take that into account.\n\n- The paper claims to be well suited for \"data-scarce\" applications, but given the variance issues of OPPE, I am not convinced by this claim. \n\n- There are many experimental details missing in the main paper. For example, \n  - In the simulator examples, how many samples used to fit the transition model in the synthetic data? Is it enough to fit the model M with essentially no error?\n  - In the HIV data, the paper does not divulge how the value was computed. In the appendix, it says that it was based on MLE estimates of the system dynamics. Clearly, this biases the value in favour of other model-based estimators of the value. Moreover, this evaluation approach assumes that a perfect model can be found---which renders the present paper superfluous. In other words: this method of evaluation is circular---even computing the metric requires solving the problem the metric is meant to evaluate solutions for. \n  - It is unclear how the baselines for Mozannar and Madras are used since these are not developed for decision-making, but for prediction. Are they used to predict the reward? The optimal action?\n\n- Algorithm 1 is only given for the discrete case and the appendix contains no algorithm for the continuous case. As such, it is unclear how the dynamic programming step is performed for continuous states. \n\n- It is not clear if the policy \\pi_0 is known or estimated. If it is estimated, it is very unlikely that the estimate is a perfect model of clinical decision-makers since these will have access to more information in any real-world application. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a framework for learning to defer sequential medical decisions based on Bayesian model-based reinforcement learning in Markov decision processes. The problem is posed as learning a policy that chooses adaptively between executing a given target policy and deferring to an expert. The policy is learned by optimizing the cumulative reward given every time the target policy outperforms the behavior policy by a certain margin, as estimated by a learned model. This approach is evaluated in a series of experiments based on simulated and real-world data. \n\nClaimed contributions: \n- Defining learning-to-defer problem for sequential settings as offline model-based reinforcement learning (RL)\n- Show that modeling non-stationarity in environments provides a better trade-off of improving outcomes versus the frequency of deferrals\n- Demonstrate that deferral methods that defer myopically, based on the immediate outcome, still benefit from modeling system dynamics\n- Justify how the decomposition can guide experts to potential interventions\n- The work is applicable in many safety-focused, data-scarce, non-stationary settings where online policy improvement is not allowed due to ethical or practical constraints", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The paper would benefit from taking its motivation from a real-world clinical task, instead of a general clinical environment. This would add credibility to the framing of learning-to-defer, and perhaps change the setting slightly. \n\n- The work should be better contextualised in the literature of off-policy evaluation and causal estimation. \n\n- The experiments should add missing details on comparisons and evaluation metrics (there is plenty of space until the 12-page limit is reached). \n\n- Typo in Section 3: \"In practice, the target policy π0 may not uniformly improve over πtar for all states.\" It should be the reverse, no?", "broader_impact_concerns": "- Ignoring the impact of selection bias in offline evaluation of clinical decision making policies is dangerous and must be avoided. See e.g., \"Guidelines for reinforcement learning in healthcare\" by Gottesman et al.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PFmfTS8gZZ", "reviewer": "Reviewer_w4hA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The problem setting seems well motivated, and is a nice extension to existing deferral-based frameworks\n* The proposed method appears to account for uncertainty in the deferral process, and significantly outperforms myopic approaches\n\nWeaknesses:\n* I found a few parts difficult to understand, and would appreciate clarity:\n    * Figure 2 is unclear to me; while SLTD appears to learn to defer earlier than SLTD-Stat, it appears to stop deferring at t=13? Is this result expected? My understanding is we should still defer in states 2,3,4 after t=12 as the target policy still tries to pick sub-optimal actions\n    * In the diabetes data, non-stationarity is created by swapping the adolescent patient property with an alternative pateint over time. Is this intended to model a realistic scenario of how patient properties may change? Wouldn't a different process be more realistic (e.g., a slowly increasing/decreasing value, such as body weight)? I ask as this design choice seems to significantly favour approaches that can model sudden changes in dynamics.\n    * Does SLTD require knowledge a-priori of what an expert will perform? From my understanding, to calculate the expected return under the mixture policy, one must have access to the decision that the expert may perform in the future? Is this a realistic assumption given an actual expert could be a human? I believe the expert policies considered here are still synthetic.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors introduce a new framework to better model decision making problems involving deferrals to a human expert. This framework, called Sequential Learning to Defer (SLTD) treats interventions as a multi-step decision making process, and aims to model long-term consequences of interventions. To this end, they introduce a model-based method that factors in not only long-term outcomes, but also non-stationarity that might exist in the dynamics of the environments. Fundamentally, they learn a 'deferral function' that estimates the probability that a costly 'expert' policy $\\pi_0$ (i.e., a human) improves over an alternative 'target' policy $\\pi_{\\text{tar}}$ (i.e., an automated system).  This deferral function itself constitutes a policy that determines whether, for a given timestep, deferral to a human expert should take place. By accounting for future uncertainty, they are able to defer pre-emptively, and improve overall outcomes. Moreover, by taking a fully Bayesian approach, they are able to model key sources of uncertainty, such as dynamics, which allows them to handle non-stationarities that may cause maximum-likelihood approaches to be over-confident/miscalibrated in their estimations.\n\nThey show that their approach to their introduced problem setting performs significantly better than previous myopic approaches (that don't take into account future outcomes), and perform a set of ablations to show that key modelling decisions, such as modelling future values, are key to strong performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would appreciate clarity on the points raised above in the weaknesses (and potential rewriting to make it clearer in the manuscript), and will base my recommendation on the answers. ", "broader_impact_concerns": "None, I believe the authors list these in the paper already.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PJKTdJ3zrA", "reviewer": "Reviewer_yj16", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- It is an interesting setup and somewhat novel: This work advances learn-to-defer algorithms by considering them in non-stationary, sequential settings and by identifying the value of delaying deferral, although some of its ideas overlap with that of safety and robustness in reinforcement learning (RL).\n- Resources and Reproducibility: Simulation data are open-source and code was made available.\n\nWeaknesses:\n- The presentation of the paper could be improved and sometimes it's hard to follow. For example, it claims that this work focuses on offline model-based RL. According to the paper, the target policy is learned from the offline data whereas the behavior policy is provided by the domain expert. What are the differences between target policy and behavior policy? How is this behavior policy provided, i.e., online or offline? Is the target policy expected to learn or to evaluate? I am confused since it looks to me that this work should concern more about the setting of deploying a learned offline policy into an online changing environment. But if it's a totally offline setting, I am wondering how does dynamics change. In figure 1, it is unclear what are those symbols, i.e., pills, syringes. Are they available actions at each time point? What are the arrows with different colors? What is the expectation of outcomes here? When dynamics are changing, how's the optimal policies for them? Will the optimal policies change as well or they are the same? Which one is SLTD? I didn't see the green arrow. In figure 2, what does each plot in the row mean? I don't understand what is the difference. What's y-axis? It looks like it should be the value of \\hat{g}_{{\\pi}_{tar}}. But there is no value of 1 (defer) in those plots, how does this \\hat{g}_{{\\pi}_{tar}} do? What's the optimal policy or ground truth here?\n- Since health care setting is quite different from the general setting, it's necessary to clarify which kind of medical decision-making task this work focuses on? Besides, the setups, assumptions, and patient outcomes could be different a lot under different health care situations, e.g., critical care, chronic care, etc. What are they in this work? How to define the state space, action space, and reward function under this health care setting? How to obtain the target policy and behavior policy in each dataset?\n- Since this paper claims its applicability in a data-scarce setting, it would be more convincing if the number of data samples used for training can be provided. This is important, particularly in the health care setting.\n- Why the non-stationary dynamics are represented by a sequence of MDP? What are the differences among those MDPs? Do they share the same optimal policy?\n- What's the ground truth for the value of the deferred policy?\n\n\nMinors:\n- Typo \"the target policy $\\pi_0$ may not uniformly improve over $\\pi_{tar}$ for all states\"\n- Typo “such that such that…”.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "\nThis paper aims at learning a policy in sequential medical decision-making where the system may decide to defer to a domain expert in case the outcome is expected to be worse than deferring. Furthermore, the paper focuses on non-stationary situations where uncertainty is particularly high, and where deferring may be especially relevant. Finally, the proposed methods are evaluated on synthetic, semi-synthetic, and real-world data, in order to validate their effectiveness. \n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It looks to me that the contribution of this work is more on the algorithmic side since there lack justifications or supporting details for the clinical practice. On the methodology side, there are lot of clarity issues regarding problem setups/ technical details. For evaluation, the sample efficiency should be reported as well. On the clinical side, I am not completely sure about its broad applicability. Although it is valuable, it could discuss more about its assumptions, limitations, and ethical concerns.\n- See my comments in the main weaknesses above. \n- How to select the conjugate priors?\n- What's the difference between long-term outcomes and short-term outcomes in health setting in this paper?", "broader_impact_concerns": "- The model is motivated for a health care setting. It is hard to see how it is possible to quantify the value of the deferred policy. I can see that it is not an issue in the experiments in this paper. But I have doubts about its practical usefulness. See questions to the authors above.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a borderline paper, and the consensus leaned more towards rejection due to the limitation in applicability. However, I consider the TMLR criteria to be satisfied in the sense that the paper is sound, and that there could be interesting follow-up work.
Decision: Accept (poster)