Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding
Authors: Alizée Pace
Hugo Yèche
Bernhard Schölkopf
Gunnar Ratsch
Guy Tennenholtz
Venue: ICLR 2024 poster (2024)
Keywords: offline reinforcement learning
hidden confounding
uncertainty quantification
causal inference
healthcare
vasopressor and fluid administration
Primary area: reinforcement learning
TL;DR: The paper tackles nonidentifiable hidden confounding in offline RL by introducing and leveraging delphic uncertainty -- a novel uncertainty estimation method for confounding bias, improving performance on simulated and real-world confounded data.

Abstract: A prominent challenge of offline reinforcement learning (RL) is the issue of hidden confounding: unobserved variables may influence both the actions taken by the agent and the observed outcomes. Hidden confounding can compromise the validity of any causal conclusion drawn from data and presents a major obstacle to effective offline RL. In the present paper, we tackle the problem of hidden confounding in the nonidentifiable setting. We propose a definition of uncertainty due to hidden confounding bias, termed delphic uncertainty, which uses variation over world models compatible with the observations, and differentiate it from the well-known epistemic and aleatoric uncertainties. We derive a practical method for estimating the three types of uncertainties, and construct a pessimistic offline RL algorithm to account for them. Our method does not assume identifiability of the unobserved confounders, and attempts to reduce the amount of confounding bias. We demonstrate through extensive experiments and ablations the efficacy of our approach on a sepsis management benchmark, as well as on electronic health records. Our results suggest that nonidentifiable hidden confounding bias can be mitigated to improve offline RL solutions in practice.

Published as a conference paper at ICLR 2024
DELPHIC OFFLINE REINFORCEMENT LEARNING UNDER NONIDENTIFIABLE HIDDEN CONFOUNDING
Alizée Pace 1,2,3
Hugo Yèche
Bernhard Schölkopf 3
Gunnar Rätsch 1,2
Guy Tennenholtz 4
2 *Department of Computer Science, ETH Zürich 3 Max Planck Institute for Intelligent Systems, Tübingen
1 ETH AI Center
Google Research
zee. wwweeepaceBaeeeee .tta..ccee...c
ABSTRACT
A prominent challenge of offline reinforcement learning (RL) is the issue of hidden confounding: unobserved variables may influence both the actions taken by the agent and the observed outcomes. Hidden confounding can compromise the validity of any causal conclusion drawn from data and presents a major obstacle to effective offlin RI.. In the present paper, we tackle the problem of hidden confounding in the nonidentifiable setting. We propose definition of uncertainty due to hidden confounding bias, termed delphic uncertainty, which uses variation over world models compatible with the observations, and differentiate it from the well-known epistemic and aleatoric uncertainties. We derive a practical method for estimating the three types of uncertainties, and construct pessimistic offline RL algorithm to account for them. Our method does not assume identifiability of the unobserved confounders, and attempts to reduce the amount of confounding bias. We demonstrate through extensive experiments and ablations the efficacy of our approach on a sepsis management benchmark, as well as on electronic health records. Our results suggest that nonidentifiable hidden confounding bias be mitigated to improve offline RL solutions in practice.
INTRODUCTION
Large observational datasets for decision-making open the possibility of learning expert policies with minimal environment interaction. This holds promise for contexts where exploration is impractical, unethical or even impossible, such as optimising marketing, educational or clinical decisions based on relevant historical datasets (Gottesman et al., 2018 Singla et al. 2021 Thomas et al.) 2017). Recent years have thus seen the emergence of offline reinforcement learning (RL) literature (Levine et al.) 2020), which proposes to adapt RI. methods to overcome estimation biases induced by learning from finite, fully offline data.
Aside from estimation biases, confounding variables are common in offline data (Gottesman et al. 2018). The problem of hidden confounding, where outcome and decisions are both dependent on an unobserved factor, is widely overlooked in many of the concurrent offline RL methods. Nevertheless, it may induce significant errors, even for the simplest of bandit problems, and is especially aggravated in the sequential setting (Chakraborty and Murphy] 2014] "Tennenholtz et al.] 2022, Zhang and Bareinboim] 2019). Hidden confounding exists in numerous applications. In autonomous driving, for example, the observational policy may behave according to unobserved factors (c.g. road conditions (Haan al. (211)), which also affect environment dynamics and rewards. Alternatively, in the medical context, unrecorded patient state information such as socio-economic factors or visual appearance may have been taken into account by the acting physician (Gottesman al. 2018).
In this work, we focus on nonidentifiable hidden confounding in offline RL. While prior work has mostly addressed the problem in the identifiable setup (Kumor et al. 2021 Lu et al. 2018a Wang et al. 1 Zhang and Bareinboim, 2020), we show that significant improvement in policy learning be achieved even in the realistic nomidentifiable setting. We propose approach to estimate uncertainty due to confounding bias and to account for the degree of confoundedness while learning. In turn, this leads to improved downstream performance for offline learning algorithms.

---

Published as a conference paper at ICLR 2024
Our main contributions follows. (1) To the best of our knowledge, we are the first to address nonidentifiable confounding bias in deep offline RL. (2) We achieve this by introducing a novel uncertainty quantification method from observational data, which we term delphic uncertainty. (3) We propose an offline RI. algorithm that leverages this uncertainty to obtain confounding-avenn policies, and (4) we demonstrate its performance on both synthetic and real-world medical data.
2
PRELIMINARIES
$6-1
81
S1+1
We consider the contextual Markov Decision Pro- cess (MDP) (Hallak et al. 2015), defined by the tu- ple M = (S, 2, A, T, r, Po: M; 7), where S the state space, A is the action space, Z is the context space, T S x 2 x A + AS is the transition function, r S 2 A + [0, 1] is the reward function, and 7 E [0, 1) is the discount factor. We assume an initial state distribution Po Z + AS and a context distribution V, such that each inter- action episode has a fixed context 2 ~ V, and the environment initialises at state so Pu( -22). At time t, the environment at state 8, S and an agent selects an action a, E A. The agents receives a reward r; = r(11, ax, =) and the environment then transitions to state 8++1 ~ T( 18,, 0t, 2). A causal graph of the process depicted in Figure
G1-1
Ft
Figure I: Contextual MDP. Black arrows show the transition dynamics, blues ones the reward function, and red ones the policy. Confounding arises when both behavioural policy ne and environment returns depend on hidden context variable 2 (dashed lines).
We define a context-aware policy π as a mapping π :S x Z + AA, such that T ss 2) is the probability of taking action in state s and context Likewise, we define context context-independenn policy म : S + AA. We denote the set of all such policies II and II, respectively
We assume access to a dataset of N trajectories D r' = (si, ab. ro:r..... ro.... ......... azp r'r) are trajectories induced by an unknown, context-aware behavioural policy my E such that of ~ TT( -s)..=). The decision makinn context 2 for each trajectory is not included in the observational dataset. In the following, drop index i unless explicitly needed.
- where the sequences
Finally, define the offline RL task, which consists of finding an optimal context- context-independent policy म* E il one which maximises the expected discounted returns. Specifically, we define the state- action value function of policy मे E īī by Q* (s, a) Es (Ee= 7'r(st, at, =8000 = 8, ag a). where Eş denotes the expectation induced by following policy ă E II. We also define the value of i by V (s) = En-a [Q* a)]. An optimal policy is then defined by म *( (s) arg max [v*(
3 SOURCES OF ERROR IN OFFLINE RL
Optimising a policy from observational data is prone various sources of error, which many RL works propose to decompose, estimate, and bound (Levine et al. 20200 Tennenholtz and Mannor 2022). First, the process is prone statistical error in correctly estimating a value model from the observed data (Jin et al. 202T). Inherent environment stochasticity (aleatoric uncertainty) can result in imprecise models, whereas finite data quantities (epistemic uncertainty) can lead to poor model approximation. Improper handling of estimation errors causes covariate shift and overestimation problems, as evident in behaviour cloning (Ross et al. 2011) and offline RL (Kumar et al., 2020). Offline RL approaches typically mitigate these errors through pessimism, penalising areas where error is expected to be large (Jin et al. 2021 Levine et al. 2020).
Another source of error, which is often overlooked in the RL literature, is structural bias. Independent of data quantity, such bias oceur when the state-action space coverage is incomplete (Uehara and Sun 2022). or when the expressivity of the model class considered is inappropriate (Lu et al. 2018b). Our work considers confounding bias critical type of structural bias, which often disregarded despite many data collection environments being prone to its occurrence (Gottesman
al. 2018 Haan et al. 2019, Kallus and Zhou 2018 2020).
'One can also consider depenre-d policies. Nevertheless, Markov policies sufficiently illustrate the challenges of task, and can be easily generalised to history-denendent ones.
2

---

Published as a conference paper at ICLR 2024
(c)
ma 0)
(ri(t
PA 0,2 0)
Pr(ris, ==
(Hla)
Pr (rla)
02
(a) Observational data
(b) World (0(
(c) World 2: v<(0 0.5.
Figure 2: Confounding Bias Example. World and are two models for the binary confounding variable that are compatible with the marginalised observational data (a), composing models for v(z). ma(alz) 2 and
P(r a, 2). Under an alternative policy, such as a context-independem uniform policy Func(-) (-) = 1 A/, these
two worlds give different values each action.
Confounding Bias. Confounding bias arises when the observational policy depends on unobserved
factors which also affect the chosen action and the reward transition function (Tennenholtz et al., 2022). To better understand how confounding bias may affect offline RL algorithms, consider the process detailed in Section 2
and depicted in Figure [ The offline data was generated by sampling trajectories from the behavioural policy distribution T ~ (T), which is marginalised over v(z) and factorises as follows:
H Par. (T) E EENN Po(Sula) (1) II (adss Pr(relsi, 2) ((
(I)
where P. is the probability of sampling reward from r(st, dr, z). Any offline reinforcement learning objective can be written as an expectation over this trajectory distribution (Levine et al. 2020). Confounding arises when one learns models trajectories following Pa. (T). but estimates the value of policies π that change the probability of taking an action a given state and context (s, 2) as is necessarily the case when considering context-independent policies. Since all model terms Equation (I) are unknown and nonidentifiable due to their dependence on 2, there may exist several "worlds" that could induce the same observational distribution Ps, (T). This is known the identifiability problem in the causal inference literature (Kallus and Zhou 2018 Namkoong et al. 2020), which has been studied extensively, providing methods for analyzing when counterfactual estimates can be obtained. Particularly, without additional assumptions about the causal structure of the environment such as using environment interventions (Lu al. [00 2018a| /Zang and Bareinboim 2020 or the existence of observable back- or front-door variables ( (Kumor et al. 2021 Wang et al. 2021 the context acting as confounder nonidentifiable and cannot be estimated. Below, we illustrate through simple example, how two equally plausible models can correctly construct the same observational distribution, yet induce two different values for another policy.
An Illustrative Example. Suppose access to the bandit data in Figure 2a induced by an unknown context-awane policy Tr with marginal distribution P., (a. Simplifying Equation to this setup, we obtain:
Pa, (a, r) E (m
2)).
We can therefore change v. Th, and P, to induce the same marginalised distribution Pn,, with a significant difference in reward for counterfactual policy. Indeed, in Figure 26 World 1 assumes a singleton context with a uniform behavioural policy, whereas World 2 (Figure 2c assumes two contexts and behavioural policy which changes its distribution w.r.t. the sampled context. In both of these worlds, the observational distribution Pr, (a, r) remains the same, whereas the uniform policy a.n(()) = 1/|A| results in different reward distributions and optimal actions. Without explicit access to the ground-truth context, modelling an alternative policy to the privileged data setttaannnrainnggg one will therefore prone to spurious correlations and estimation biases.
In the next section, we propose to address the general nonidentifiable confounding problem in offline RL by estimating the amount of possible confounding bias within the observational dataset and correcting for it during learning. We compare our approach to related work in Table T and more detail in Section
3

---

Published as a conference paper at ICLR 2024
Table 1: Comparison of related work addressing confounding bias in decision-making further discussed in Section 7 Our formalism can be built upon with different parametric assumptions. For instance, the sensitivity analysis assumption could be applied to restrict the set of the compatible worlds considered in delphic uncertainty. The identifiable proxy variable case results in singlcton set of compatible worlds, [W] = 1.
Related work
Modelling assumptions
Pessimism striteyy
Offine RI. algorithm
Offine RL [Levine al. 2020
No confounding
Penalised Q
Enviroument inter action Zhang and Bareinboim 2020
P. (r) accessible
X
Proxy variables (Wang et
M(2) identifiable
Sensitivity analysis [Kalius and Zhou 2020
st s, a) /allale, s') <
Worst-case Q
x
Delphic offline RI. (Ours)
Set W of possible v(=)
Penalised Q
4 MEASURING CONFOUNDING BIAS THROUGH DELPHIC UNCERTAINTY
In this section, we formulate a method for estimating uncertainty arising from confounding bias in offline RL, which we term delphic uncertainty 2 While aleatoric and epistemic uncertainty can be expressed as probability distributions over model outputs and parameters, respectively (Hüllermeier and Waegeman 2021). delphic uncertainty is a distribution over counterfactual values. We propose a general approach to decouple aleatoric, epistemic, and delphic uncertainties, which we leverage to overcome confounding bias in Section 5
We first formalise the notion of world models compatible with the observational distribution Pa, (T): Definition 4.1. A compatible world for Ps.. is a tuple w = (Zw, Pwr Po,w. , Prows Tw: Tb w) which satisfies Px, (T) Rezuu (0, (8ol=) I Tb. [de] =) (relot, 0e, 2)T Th (Se+1 18c, Oe, =) for any trajectory T. We denote by W the set of all compatible worlds.
We focus uncertainty estimates of value functions. Let W (i.e., is some compatible world for ). We use ow to denote the parameters of a Q-value function in world w. For fixed w and 6w. we assume each value model Qa_ is defined by some stochastic model, e.g., normal distribution Indeed, here Te,, accounts for aleatoric uncertainty, capturing the
1e,, ~ ((eee. intrinsic stochasticity of the environment (Kendall and Gal 2017). Additional statistical uncertainty arises from the distribution over model parameters O.. in the fixed world E W. Starting from prior over Pu., evidence from the data leads to a posterior estimate over the correct model parameters P(O../w,, which captures epistemic uncertainty (Hüllermeier and Waegeman 2021). We refer the interested reader to Appendix A for an overview of statistical uncertainty estimation methods.
We are now ready to define the uncertainty induced by confounding variables, which we term delphic uncertainty. To do this, we leverage Definition [4.1 and define delphic uncertainty by varying over compatible world models. Following on prior work separating epistemic and aleatoric uncertainty (Kendall and Gal 2017). we can decompose the variance the value function estimate between the three types of uncertainties. Particularly, let w be a compatible world for PM,. and let P(w) be some distribution over worlds in W. This posits the following distribution for Qs..: P(Qo...) = [ P(Qo.,. P(O. P d0 We have the following result. Its proof, based on the law of total variance (Weiss et al. 2006), is given in Appendix
Theorem 4.2 (Variance Decomposition). For any II, we have
Var(QT.) = Eun Be.. Var(QU 10,,, w|lw| + Vars., (E1Q7_ 18 w wlw
Var (Bo.. DE [E[2, 10 w|ww])
aleatoric uncertainty
epistemic uncertainty
deiphic uncertainty
To gain further intuition of this result, consider the case of normal distributions. We can rewrite Theorem 4.2
Var(Q Eng [B... [To |w]3 + Vara,. (po_ (wc) + Varo,, (ao,, luc]] + (Fa, (w)) (2)
The first three terms, calculated through statistics of predicted means and standard deviations, capture alcatoric and epistemic uncertainties, whereas the final term, calculated through variation over compatible world models, captures delphic uncertainty. Indeed, the latter form of uncertainty remains even in deterministic environments (oo.. + O. no aleatoric uncertainty) and infinite data ([D) - oc, no epistemic uncertainty). We refer the reader to Appendix B.2 for further discussion.
TThe word "delphic" characterises quantities that ambiguous and opaque, relating to the hidden confound- ing variables and their elusive effect model predictions.
4

---

Published as a conference paper at ICLR 2024
5 OFFLINE RL UNDER DELPHIC UNCERTAINTY
In the previous section, we defined delphic uncertainty through variation over compatible world models. We propose method to measure it in practice, then leverage uncertainty estimate within an offline reinforcement learning framework to mitigate confounding bias.
5.1 MEASURING DELPHIC UNCERTAINTY
Following the estimation approach outlined in Theorem 4.2 delphic uncertainty can be measured through the variation over value functions Q" from different world models w. We propose one method to find set of world models; note that alternative partial identification assumptions (Kallus) and Zhou 2018 Wang et al. 2021) could be adopted.
Modelling Compatible Worlds using Variational Inference. Mirroring the illustrative example in Section 3 compatible world w E W must capture the following key components, using observational data, and grounded by the estimation of the counterfactual action-value function Q":
1. A model for the behavioural policy (als, z) s.t. the world can be adjusted m
2. A model for the action-value function Q% (s, a, under the behavioural policy;
3. A confounder distribution (2) within the world
With these elements, the value function of policy π can be estimated over one step using importance sampling and marginalising over (s, a) ggg- (2) (a]ss) uo 2) Q (s,a, a)). We
discuss alternative counterfactual estimation approaches in Appendix C
We propose to train all elements jointly by maximising the likelihood of the first two terms over the behavioural data, marginalising over latent variable model for (z). In fact, starting from plausible prior, we can learn a data-driven posterior for the confounder distribution (2(T) which leverages information about gained over the course an observed trajectory. This allows the world model to converge to the ground-truth y (and to the true dependence of QT and z) in the identifiable setting.
The compatible world is therefore trained through vari- ational inference, using state-actions (s, a) E 7 and sampling & ~ (2(T) through the reparametrisation trick for each trajectory E D. Optimal parameters are obtained by maximising the Evidence Lower Bound (ELBO, Kingma and Welling (2014) over D; namely,
r(a)
E
Pu(str)
Ro.Po...... 4g,4 4g.
(e,a,,
E( awnuls [log (alss, a log Q (s, a, a.)
Figure 3: World model architecture w (v, Tb, QT*), under prior p(z) for the con- founder distribution. Variance Q" over multi- plc worlds captures delphic uncertainty.
-DDx (p (#IT) p(z),,
where fa, 3}} are hyperparameters (Higgins et al. 2017), p(z) is prior over the confounder distribu- tion and DEL r, is the Kullback-Lebler divergence. Observed action probabilities and Monte-Carlo estimates are used as targets for Tb,w and Q
Counterfactual Variation Across Worlds. The set of compatible worlds W for the observational training dataset be approximated by ensemble of W world models, each trained using different confounder space dimensionalities |2), prior distributions p(z) and model architectures we propose specific implementation details in Appendix C Epistemic and aleatoric uncertainty can be separately captured by implementing each world model element as an ensemble of probabilistic models. Following our definition of confounding bias (Theorem 4.2). we measure delphic uncertainty through the variance in Q (s, a) across worlds: w5 a) Var (QT (s, a)) for policy at state- action (s, a), taking the mean Q" output, averaged over the world model ensemble.
When no confounding exists, all models in W should identify similar v, QT and Ty, returning a similar value of QT, On the other hand, confounding with ambiguous returns would lead different values across world models. While one could theoretically consider all possible world models in Definition 4.1 found that, in practice, varying over subset of compatible models enough to show improved offline RL efficiency. We refer the reader to Appendix C for an exhaustive overview of the training procedure.
5

---

Published as a conference paper at ICLR 2024
Algorithm 1: Delphic Offline Reinforcement Learning
Input: Observational dataset 'D, Offline RI. algorithm.
Learn compatible world models {2w, Vu, Po,w:; Pr,.. Twy uey Tho New that all factorise to Pro (T).
Obtain counterfactual predictions Q for each w E -W.
Define local delphic uncertainty: uă a) Var (s,
Apply pessimism using ue in Offline RL algorithm (see Section 5.2)
5.2
DELPHIC ORL: OFFLINE REINFORCEMENT LEARNING UNDER DELPHIC UNCERTAINTY
Inspired by pessimistic approaches in offline RL (Fujimoto al. 2019 Tin al. 2021: Kumar et al., 2019 2020 Levine et al. 2020), we propose to penalise the value of states and actions where delphic uncertainty high, such that the learned policy is less likely to rely on spurious correlations between actions, states and rewards. This pessimistic approach, which enables the agent to account for and mitigate confounding bias when making decisions, is summarised in Algorithm []]
In this paper, we incorporate pessimism with respect to delphic uncertainty by modifying the target Qtarget for the Bellman update in model-free offline RL algorithm to
Qearget a) = Qtorget a) Xu (M,(s))).
where т is the latest learned policy, (s, a) is tuple sampled for the update and hyperparameter À controls the penalty strength. This approach enjoys the performance guarantees established in prior offline RL work penalising reward of value functions with oracle error functions (Levine et al. 2020
Yu et al. 2020). We apply our penalty to Conservative Q-Learning (Kumar et al.) 2020), but could also implement within any model-free offline RL algorithm which already induces pessimism with respect to epistemic uncertainty (Fujimoto and Gu 202 Kumar et al. 2020
Levine et al.] 2020).
Similar to epistemic uncertainty penalties in offline RL (Fujimoto and Gu
2021 p, we conjecture that
ours may act as a form of regularisation towards the observed policy.
Note that various other methods can be adopted to drive pessimism against delphic uncertainty within existing offline RL algorithms (Fujimoto and Gu 2021; Kumar et al.] 2020 Yu et al., 2020), depending on the task at hand. A penalty can be subtracted from the reward function in model- based methods (Yu et al.] (2020). The uncertainty measure can also be used to identify a subset of
possible actions (Fujimoto et al. 2019), or to weigh samples in the objective function, prioritising unconfounded data. We refer the reader to Appendix C for implementation details.
6 EXPERIMENTS
In this section, we study the benefits of our proposed delphic uncertainty estimation method and its application in offline RL. We validate two principal claims: (1) Our delphic uncertainty measure captures bias due to hidden confounders. (2) Algorithm Ieadd to improved offline RL performance in both simulated and real-world confounded decision-makinn problems, compared to skate-of-the-nn but biased approaches. As baselines compatible with the discrete action spaces of environments studied here, we consider Conservative Q-Learning (CQL) (Kumar al., 2020). Catch-Constrained
Q-Learning (BCQ) (Fujimoto et al. 2019) and behaviour cloning (BC) (Bain and Sammut 1996). Implementation and dataset details are provided in Appendices Cand anspeectiiviveyy In the following. we measure and vary confounding strength through the dependence of the behavioural policy on the hidden confounders, I e a. [Ta 2)/ mT (als, z/)] (Rosenbaum, 2002). where
also affects the transition dynamics or reward function.
6.1 SEPSIS SIMULATION
We explore a simulation of patient evolution in the intensive care unit adapted from Oberst and Sontag (2019). The diabetic status of a patient, accessible to the near-optimal behavioural policy but absent from the observational dataset, acts as hidden confounder 2.
Uncertainty Measures. First, we study the relationship between our uncertainty estimates and the decision-making setup. In Figure we find that epistemic uncertainty reduces with greater data quantities and increases out of the training set distribution, whereas aleatoric uncertainty increases with environment stochasticity, in agreement with prior work (Kendall and Gal 2017). Our delphic uncertainty estimate, the other hand, cannot be reduced with more data and increases with greater confounding. Moreover, we found that delphic uncertainty relates to meaningful regions of state- action space, as it is highest under vasopressor administration the only treatment for which patient

---

Published as a conference paper at ICLR 2024
Delphic
0.0003
Epistemic Aleatoric
Uncertainty
0.0003
0.0003
0.0002
0.0002
0.0002
0,0001
0.0001
0.0001
0.0000
0.0000
200
400
600
800
0.0
0.2 0.4 0
20
40
60
80
100
N
(a) Number of trajectories N
(b) Environment stochasticity of
(c) Confounding strength T
Figure 4: Uncertainty measures as function of data properties, averaged over state-action pairs in the sepsis dataset. Epistemic uncertainty reduces most with more data, alcatoric uncertainty increases most with environment stochasticity (reward variance), and delphic uncertainty increases most with confounding strength.
evolution is confounded by the hidden diabetic status, and which is therefore penalised more strongly as confounding strength I' increases. We refer the reader Appendix B for an exhaustive overview and further experiments.
Onilne AL (bserved)
COL
Online RL unobserved;
BOQ
Offline RL Performance. In Figure we com-
Environment Returss
80
Delphic ORL (Ours)
BC
pare environment returns obtained through offline RL, imitation learning, and our proposed approach. Our results reveal the susceptibility of offline RL confounding bias: the presence of unobserved fac- tors z that influence both the behaviour policy and transition dynamics leads to inaccurate value function estimates. Behaviour cloning appears be less prone to this bias but still faces challenges in dealing with missing information mance gap the online context-awar policy in the unconfounded case (T = 1), and with the distribution shift in observed histories (Ortega et al. 2021). In contrast, our approach to penalising delphic uncer- tainty leads superior performance, especially as confounding strength increases. In Appendix E also compare different approaches to implementing pessimism w.r.t. delphic uncertainty, as detailed in Section 5.2 measure the effects of incorporating partial identification assumptions (Kallus and Zhou 201 8) or dependence on history, and provide an ablation over performance as function of pessimism hyperparameter X.
60
40
20
20
40
60
80
100
2, evidenced by the perfor-
Figure 5: Performance Results as a function of confounding strength T. Normalised environment returns (mean and shaded 95% Cls) over 10 runs.
6.2 REAL-WORLD DATA
We demonstrate the added value of our algorithm in optimising decision-making policies from real- world medical data. Our clinical policies are trained using a publicly available dataset electronic health records, with over 33 thousand patient stays in intensive HiRID, Hyland et al. (2020). We consider the problem of optimising the treatment policy for vasopressor and fluid administratior/) and design the reward function to avoid states of circulatory failure. Significant information about patients' conditions is not available in the dataset, despite being critical the treatment choices of attending physicians, such cocio-economic factors or medical history (Yang and Lok, 2018). For ease of evaluation, we introduce additional, artificial confounders by excluding them from the observational dataset, focusing on diagnostic indicator variables, age and weight (T E [1. 200).. Disease severity is measured through the SOFA score system (Vincent et al. 1996). Recent RL methods for clinical data could improve performance (Killian et al., [023]] Shirali et al. 2023 Tang et al., 2022), but we focus on showing the limitations of offline RL under hidden confounding, and how our simple but effective penalty recovers good performance.
and over 200 measured variables
Confounding Medical Dataset. As the aforementioned variables affect both the probability of treatment assignment and downstream patient evolution, they act as confounders over outcome models when excluded from the data. In Figure we highlight how our delphic uncertainty measure captures confounded state-action pairs in concordance with the introduced confounders. Delphic uncertainty
Thheee therapeutic agents commonly given to overcome shock in intensive care (Benham-Hermetz al. Their administration strategy has already heen studied RL. task (Raghu et al. 2017)
2012
7

---

Published as a conference paper at ICLR 2024
2 Cortho dag.
--- 2 (age)
2- (age)
2 Eneuro digg
Deiphic uncoraimmy
(neuro, diag.?
mapp
10 15 Disease Scverity
Severitys
Disease Severty
(a) {age]. r 18.2
(b) [neuro. diag.). 29.1
(c) Uncertainty increase under Z.
Figure 6: Delphic uncertainty as function (s, 2) real world medical data, {vasopressors) In (a, b), we note the dependence of the behavioural policy m. (top) and/or state distribution P(s) (bottom) on confounders z. In (c), delphic uncertainty increases most in confounded states and under factors with greater confounding strength, compared to orthopaedic diagnosis (T 3.4).
Confounders 2
BCQ
BC
COL
Delphic ORL
Delphic ORL (Ours)
CQL
BC
54.6
70
[age]
48.2
58.8 =: 0.8
64.7 + 0.5
64.4 +
66.5
0.9
(ecuro diag.)
29.1
55.0
1.3
0.9
59.6 1.7
65,7
Expert preference (3)
gaszro. diag.)
59.8
60
trauma)
16.3
56.3
0.8
63.2 1.1
63.5 1
65.7 1.0
cardio. diag.)
13.2
56.2 + 1.0
60.6 + 0.7
58.6 0.9
62.7 1.1
diag.)
50
weight)
60.1 D8
68.4
solation)
6.8
61.2 = 0.8
64,5 : 0.6
6488 0.9
65.3
1.2
endo. diag.)
60.1 1.1
63.1 0.6
65.5 1
65.7
resp. diag. }
onha. ding.) }
3.4
62.3 =: 0.8
64.6 0.6
65.8
:: 0.7
65.9 1.0
surgical status)
3.2
62.2 + 1.1
64.3 +.
67.4 0.7
66.8 1.1
sepais)
30
15
20
100
200
intoxication)
62.3 0.9
63.4 0.5
65.2 0.6
(ell)
62.6 d: 0.8
65.4
d: 0.5
68.2 a:
0.7
67.6 1.1
r
Table Off-Policy Evaluation (OPE) on the real-world medi- cal datasct. Delphic ORL yields improvements when strongly confounds treatment decisions (large I'), Mcan and 95% Cls over 10 runs. Best and overlapping results in bold.
Figure Expert Clinician Evaluation of treat- ment policies, supporting the conclusion that Delphic ORL improves leaming in confounded settings.
is generally highest for high disease severity, where important factors such as age or comorbidity may affect the choice of treatment intensity (Azoulay al. 2009). Indeed, delphic uncertainty increased to greater extent under important confounders (e.g., age or patients' neurological diagnosis) than less critical factors (e.g., orthopaedic diagnosis).
Confounding-Avees Averse Policies. We investigate the efficacy of our penalisation approach in learning policies that optimise patient outcomes in the presence of confounding bias. To evaluate the perfor- mance of these policies, we employ doubly-robust off- off-policy evaluation (OPE) (Jiang and Li 2016
Le al. 2019). which provides a confounding fefouundnn--nddeeendeen estimate of treatment success by leverag- ing access to We refer the reader to Appendix D.2 for an exhaustive overview of this evaluation method. Table shows our approach maintains improved performance even as the confounding level increases, while offline RL methods suffer from bias and yield suboptimal policies. As ablation, we also studied the discrepancy of our trained policy with that in the data. We compared actions taken by our policy and those in the data and found that policy more different than behaviour cloning, but more similar than offline RL. baselines. This suggests our learned policy was indeed able to exploit the reward signal in the data, identifying observed treatment strategies that may be more robust to confounding biases. We refer the reader to Appendix for an overview of this ablation.
Expert Clinician Evaluation. Motivated by the observed success of our method, we evaluated our algorithm using expert clinicians. Figure/[show the evaluation results of six human expert clinicians, who ranked pairs of different policies based on their observed patient outcomes. More specifically, the human experts were shown simulated patient trajectories and were asked to blindly compare the expected value actions from either our policy or the CQL policy to those the behaviour cloning policy. The results provide additional validation for the performance improvements of our method in confounded settings. We refer the reader to Appendix D.2 for an exhaustive overview of the clinician evaluation experiment.
8

---

Published as a conference paper at ICLR 2024
7 RELATED WORK
Online-RL. methods rely on environment interaction for training. limiting their applicability in many
real-world domains such as healthcare (Gottesman et al. 2018). This has fueled research efforts in
offline methods to optimise policies through pessimism (Buckman et al. | Cheng et al., [022] Jin
et al. [021] Levine et al. 2020: Uehara and Sun 2022 Xie et al. 2021). Recent practical algorithmic
developments in offline RL focus on addressing stafistical errors induced by epistemic and aleatoric
uncertainty (Fujimoto and Gu 2021 Kidambi et al. 2020: Kostrikov et al. 2021 Kumar et al. 2020
Yu et al. 2020).
Structural errors such as confounding bias are also pervasive in offline RL (Lu et al. 201 8a), but cannot be captured by epistemic or aleatoric uncertainty quantification methods, as they do not depend on data quantity, Confounding bias cannot be reduced to the missing information problem in partially-observat environments either (Hausknecht and Stone 2015). History-dependen policies, for example, are equally prone to this source of error: while long- -eng-termmmm information can recover latent environment information, it exacerbates distribution shifts between behavioural and learned policies when learning from observational data (Ortega al., 202 Swamy et al. (022).
Several approaches have been proposed to address confounding bias in offline RL, summarised in Table T Most make assumptions to estimate the confounding variables, including access to the environment (Lu et al. [2018a) Zhang and Bareinboim 2020) or to observable back- or front-door proxy variables (Kumor et al. 2021; Lu et al.] [022] Shi et al. 2022 Wang et al.) (02T). This allows algorithms to apply covariate adjustment methods (Pearl 2009) to correct for confounding when modelling alternative policies (interventional probabilities and counterfactuals) Variational approaches modelling confounding distributions have been adopted in prior work addressing confounding bias (Lu et al. 2018a| 2022 Shi et al. (022). These methods are sensitive to modelling assumptions in the non-identifaale settings (Rissanen and Marttinen, 202T), which further supports our motivation to not focus on estimating the true confounder distribution, but rather obtain plausible se/ of confounder models compatible with the observational data.
Extensive work also discusses confounding bias and makes use of the aforementioned solutions in off- off-policy evaluation (Bennett and Kallus 2021 Bennett et al. [02II) and bandits (Chen et al. 2023 Sen et al. 2017: Tennenholtz et al. [222]) but the proposed approaches remain poorly translatable to the practical optimisation challenge of learning offline RL policies.
Our work is also closely related research on sensitivity analysis for treatment effect estimation under hidden confounding (Jesson et al. |022 Kallus et al. [2019; Oprescu et al. 2023 Rosenbaum 2002). These works propose partial identification bounds for confounded heterogeneous treatment effect estimation or bandit ceciisionmaking problems (Kallus and Zhou 2018) by assuming a bound on the dependence of the behavioural policy on hidden confounders. In this context, Jesson al.) (202T) also distinguish sources of aleatoric and epistemic uncertainty from confounding biases. Other work has proposed sensitivity analysis bounds for off- off-policy evaluation, formulating uncertainty sets over policy returns (Kallus and Zhou 20200 Namkoong et al. 2020 Zhang and Bareinboim] (019)). Still, regret bounds from sensitivity analysis remain wide and often ill-adapted to high-ddmensional state and action spaces sequential decision-makinn makinn problems. Our approach thus complements these theoretical frameworks with a practical solution to addressing confounding bias in offline RL..
8 CONCLUSION
We proposed practical solution address the challenge of learning from confounded data, specifi- cally in situations where confounders are unobserved and cannot be identified. Delphic ORL captures uncertainty by modelling world models compatible with the observational distribution, achieving improved performance across both simulated and woal-world confounded offline RL tasks. Our results demonstrate that Delphic ORL can learn useful policies in cases where traditional algorithms fail due to excessive confounding. Overall, we believe research into tackling hidden confounding in offline RL will lead to more reliable and effective decision- -aaiing tools various critical fields.
As limitations of our work, note that our evaluation focuses on medically redicaaly-motivateed motivated scenarios, on the decision-making
hypothesis that these are representative of other confounded decision making contexts. Second, the modelling cost of compatible worlds W may be expensive for large-scale problems. We refer the reader Appendix A.3 for further discussion the limitations and possible impact of our work.

---

Published as a conference paper at ICLR 2024
Ethics Statement We propose in-depth discussion the limitations and possible societal impact of our work in Appendix A.33 We also provide more details on our approach to evaluating our algorithm using expert climicians in Appendix D.2
Reproducibility Statement Implementation details for approach and for baselines considered are presented in Appendix C
Dataset details and processing steps are included in Appendix D.I We
include source code as supplementary material.
Acknowledgments We thank the anonymous reviewers for their thoughtful feedback. This project was supported by grant #2022-278 of the Strategic Focus Area "Personalized Health and Related Technologies (PHRT)" of the ETH Domain (Swiss Federal Institutes of Technology) and by ETH core funding (to Gunnar Rätsch). This publication was made possible by an ETH Al Center doctoral fellowship to Alizée Pace.
REFERENCES
Élie Azoulay, Barbara Metnitz, Charles Sprung, Jean-François Timsit, François Lemaire, Peter Bauer, Benoît Schlemmer, Rui Moreno, Philipp Metnitz, and SAPS 3 investigators. 2009. End-of- life practices in 282 intensive care units: data from the SAPS database. Intensive care medicine 35 (2009), 623-630.
Michael Bain and Claude Sammut. 1996. A Framework for Behavioural Cloning. MACHINE
INTELLIGENCE 15 (1996), 103-129. http://citeseerrx wss.e:..ssspssueeuuuuuu www.wiewdvvvvi.ooo
S sunmazy2doi=10.. 1...1.11.220
Julia Benham-Hermetz, Mark Lambert, and Robert CM Stephens. 2012. Cardiovascular failure, inotropes and vasopressors. British Journal of Hospital Medicine 73, Sup5 (2012), C74-C77.
Andrew Bennett and Nathan Kallus. 2021. Proximal reinforcement learning: Efficient off-policy evaluation in partially observed markov decision processes. arXiv preprint arXiv:2110. 10..33320 (2021).
Andrew Bennett, Nathan Kallus, Lihong Li, and Ali Mousavi. 2021.
Off-policy Evaluation in Infinite-Horizon Reinforeement Learning with Latent Confounders. In Proceedings The 24th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol. 130), Arindam Banerjee and Kenji Fukumizu (Eds.). PMLR, 1999-2007. //proceedings mlr press / v1 30/110/beemii btt.//ennettla....mmmm. html
https
Jacob Buckman, Carles Gelada, and Marc G Bellemare. 2021. The Importance of Pessimism in Fixed-Dataset Policy Optimization. International Conference on Learning Representations.
Bibhas Chakraborty and Susan A Murphy. 2014. Dynamic treatment regimes. Annual review of statistics and its application (2014), 447-464.
Siyu Chen, Yitan Wang, Zhaoran Wang, and Zhuoran Yang. 2023. Unified Framework of Policy Learning for Contextual Bandit with Confounding Bias and Missing Observations. arXiv preprint arXiv:2303.11187 (2023).
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. 2022. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning. PMLR. 3852-3878.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv: /406. /078 (2014).
Francesco D' Angelo, Vincent Fortuin, and Florian Wenzel. 2021. On stein variational neural network ensembles. arXiv preprint arXiv:2106 10760 (2021).
Bradley Efron. 1982. The Jackknife, the Bootstrap and Other Resampling Plans. Society for Industrial and Applied Mathematics. http://////11..ooo11.1111..11111 9tt.//11777811119711111
10

---

Published as a conference paper at ICLR 2024
Scott Fujimoto and Shixiang Shane Gu. 2021. A minimalist approach to offline reinforcement learning. Advances neural information processing systems 34 (2021), 20132-20143
Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function approximation error in actor-critic methods. In International conference on machine learning. PMLR, 1587-1596.
Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-policy deep reinforcement learning without exploration. In International conference on machine learning. PMLR, 2052-2062.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as bayesian approximation: Representing model uncertainty in deep learning. international conference on machine learning. PMLR, 1050-1059.
Caroline Gans-Combe 2022. Automated Justice: Issues, Benefits and Risks in the Use of Artificial Intellignnce and Its Algorithms in Access to Justice and Law Enforcement. Ethics, Integrity and Policymaking: The Value of the Case Study (2022), 175-194.
Omer Gottesman, Fredrik Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, Yi Ding, David Wihl, Xuefeng Peng, et al. 2018. Evaluating reinforcement learning algorithms in observational health settings. arXiv preprint arXiv: 12298 (2018).
Pim De Haan, Dinesh Jayaraman, and Sergey Levine, 2019, Causal Confusion in Imitation Learning. Neur/PS (2019).
Assaf Hallak, Dotan Di Castro, and Shie Mannor. 2015. Contextual Markov Decision Processes. CoRR abs/1 502.02259 (2015). http: http:///2a.x..oorrr/////51.1... 02259
Matthew Hausknecht and Peter Stone. 2015. Deep recurrent q-learning for partially observable mdps. In 2015 vaai fall symposium series.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. heta-VAEE Learning Basic Visual Concepts with Constrained Variational Framework. In International Conference on Learning Representations.
htttps://openrew// htt.://wwww..etteoooruuuoddod//-
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, (1997), 1735-1780.
Eyke Hüllermeier and Willem Waegeman. 2021. Aleatoric and epistemic uncertainty in machine learning: An introduction concepts and methods. Machine Learning 110 (2021), 457-506.
Stephanie L. Hyland, Martin Faltys, et al. 2020. Early prediction of circulatory failure in the intensive care unit using machine learning. Nature Medicine 26 (3 2020), 364-373. Issue
https:://oi.org:: http///oo.or/////////
Andrew Jesson, Sören Mindermann, Yarin Gal, and Uri Shalit. 2021. Quantifying Ignorance in Individual-Level Causal-Efffec Estimates under Hidden Confounding. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 4829-4838. attps://proceedings
ml www.prresslll. wttps////1////////eeee /esoon/essnnsle. 2.a. html
Andrew Jesson, Sören Mindermann, Uri Shalit, and Yarin Gal. 2020. Identifying Causal-Effect Inference Failure with Uncertainty-Awar Models. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran
Associates, Inc., 11637-11649. https: htttps://poceeeetteee http:///occccccccoceoo ngs neur imaripss.. wwurips.cc/pape. www.cc/ppppp...m @atpaprrffles__ Pww.dd-apppppp...
Nan Jiang and Lihong Li. 2016. Doubly robust off-policy value evaluation for reinforcement learning.
In International Conference on Machine Learning. PMLR, 652-661.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. 2021. Is pessimism provably efficient for offline rl?. In
International Conference Machine Learning. PMLR, 5084-5096.

---

Published as a conference paper at ICLR 2024
Nathan Kallus, Xiaojie Mao, and Angela Zhou. 2019. Interval Estimation of Individual-Lever Causal Effects Under Unobserved Confounding. In Proceedings of the Twenty-Secone International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol. 89), Kamalika Chaudhuri and Masashi Sugiyama (Eds.). PMLR, 2281-2290 //procedd igs. ml press / vw9/va19llll.. lusl 9a ht.m
https
Nathan Kallus and Angela Zhou. 2018. Confounding-robust policy improvement. Advances in neural information processing systems 31 (2018).
Nathan Kallus and Angela Zhou. 2020. Confounding-robust policy evaluation in infinite-horizon reinforcement learning. Advances in Neural Information Processing Systems 33 (2020). 22293- 22304.
Alex Kendall and Yarin Gal. 2017. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems 30 (2017).
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. 2020. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems 33 (2020), 21810-21823.
Taylor W Killian, Sonali Parbhoo, and Marzyeh Ghassemi. 2023. Risk Sensitive Dead-end Iden- tification in Safety-Critical Offline Reinforcement Learning. arXiv preprint arXiv:2301.0560 (2023).
Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In 2nd International Conference on Learming Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.).
1312.6112.11111
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. 2021. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning. PMLR, 5774-5783.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. 2019. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. NeurlPS 32 (2019).
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conservative Q-Learning for Offline Reinforcement Learning. 34th Conference on Neural Information Processing Systems (NeurIPS 2020) (2020).
Daniel Kumor, Junzhe Zhang, and Elias Bareinboim. 2021. Sequential causal imitation learning with unobserved confounders. Advances in Neural Information Processing Systems 34 (2021), 14669- 14680.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances neural information processing systems 30 (2017).
Hoang Le, Cameron Voloshin, and Yisong Yue. 2019. Batch policy learning under constraints. In International Conference on Machine Learning. PMLR, 3703-3712.
Sergey Levine, Aviral Kumar, et al. 2020. Offlline Reinforvemeet Learning Tutorial, Review, and Perspectives on Open Problems. (5 2020). httres//aaaii..or//b////0......
Qiang Liu and Dilin Wang. 2016. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems 29 (2016).
Chaochao Lu, Bernhard Schölkopf, and José Miguel Hernández-Lobato. 2018a. Deconfounding reinforcement learning in observational settings. arXiv preprint arXiv:1812.10577 (2018).
Miao Lu, Yifei Min, Zhaoran Wang, and Zhuoran Yang. 2022. Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes. arXiv preprint arXiv:2205. 13589 (2022).
12

---

Published as a conference paper at ICLR 2024
Tyler Lu, Dale Schuurmans, and Craig Boutilier. 2018b. Non-delusional Q-learning and value- iteration. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran As- sociates, Inc. https::/prooeedi new.rriissnnnnnnnu.sss..o...s ccpppppppppppppppppppppppppppp fi files/////eee.. htslleep/ppe/111// #1c/5500046666dd00feff666665666-
Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. 2020. Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confound- ing. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ran- zato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 18819-18831. d222aa8822221e222115677722222222
https://proceeding: wwwp:/rwoo..edinn.............e. neurips cc pappr/2020/ pdf
Radford M Neal. 2012. Bayesian learning for neural networks, Vol. 118. Springer Science & Business Media.
Michael Oberst and David Sontag, 2019. Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models. ICML (2019).
Miruna Oprescu, Jacob Dorn, Marah Ghoummaid, Andrew Jesson, Nathan Kallus, and Uri Shalit. 2023. B-Learner: Quasi-Oracle Bounds on Heterogeneous Causal Effects Under Hidden Con- founding. International Conference on Machine Learning (2023).
Pedro A Ortega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, et al. 2021. Shaking the foundations: delusions in sequence models for interaction and control. arXiv preprini (rXiv:2110.100 10819 (2021).
Ian Osband, John Aslanides, and Albin Cassirer. 2018. Randomized prior functions for deep reinforcement learning. Advances Neural Information Processing Systems 31 (2018).
Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. 2022. Epistemic neural networks. arXiv preprint arXiv:2107.08924 (2022).
Alizée Pace, Alex Chan, and Mihaela der Schaar. 2022. POETREE: Interpretable Policy Learning with Adaptive Decision Trees. In International Conference on Learning Representations.
Judea Pearl. 2009. Causality. Cambridge university press.
Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo A. Celi, Peter Szolovits, and Marzyeh Ghassemi. 2017. Deep Reinforeement Learning for Sepsis Treatment. NIPS (2017).
Severi Rissanen and Pekka Marttinen. 2021. A critical look at the consistency of causal estimation with deep latent variable models. Advances in Neural Information Processing Systems 34 (2021), 4217-4217.
Paul R Rosenbaum. 2002. Observational Studies. In Observational Studies. Springer, 1-17.
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol. 15), Geoffrey Gordon, David Dunson, and Miroslav Dudík (Eds.). PMLR, Fort Lauderdale, FL, USA, 627-633755 httpp::/pproeeddngg mlr. www.sprsspppe....o
wtt.l/pppeeesv11//ssse1 http:///1//ooosssss html
Sorawit Saengkyongam, Nikolaj Thams, Jonas Peters, and Niklas Pfister. 2023. Invariant policy learning: A causal perspective. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
Javier Moneeero,, Lina Dencik, and Lilian Edwards. 2020. What does it mean to 'olve' 'tee problem of discrimination in hiring? Social, technical and legal perspectives from the UK on automated hiring systems. In Proceedings of the 2020 conference on fairness, accountability, and fransparency. 458-
13

---

Published as a conference paper at ICLR 2024
Rajat Sen, Karthikeyan Shanmugam, Murat Kocaoglu, Alex Dimakis, and Sanjay Shakkottai. 2017. Contextual Bandits with Latent Confounders: An NMF Approach. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol. 54), Aarti Singh and Jerry Zhu (Eds.). PMIR, 518-527.
ml wwwpsr/ss/511 sen///en111 html
Takuma Seno and Michita Imai. 2022. d3rlpy: An Offline Deep Reinforcement Learnnn Learning Library. Journal of Machine Learning Research 23, 315 (2022), 1-20.
v23/22-0011 7tmmll
Chengchun Shi, Jin Zhu, Shen Ye, Shikai Luo, Hongtu Zhu, and Rui Song. 2022. Off-policy confidence interval estimation with confounded markov decision process. J. Amer. Statist. Assoc. (2022), 1-12.
Ali Shirali, Alexander Schubert, and Ahmed Alaa. 2023. Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care. arXiv preprint arXiv:2306. 08044 (2023).
Adish Singla, Anna N Rafferty, Goran Radanovic, and Neil T Heffeman. 2021. Reinforcement learning for education: Opportunities and challenges. arXiv preprint arXiv:2107.0888 (2021).
Gokul Swamy, Sanjiban Choudhury, J Andrew Bagnell, and Zhiwei Steven Wu. 2022. Sequence Model Imitation Learning with Unobserved Contexts. arXiv preprint arXiv:2208. 02225 (2022).
Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi-Veles and Jenna Wiens. 2022. Lever- aging factored action spaces for efficient offlin reinforcement learning in healthcare. Advances in Neural Information Processing Systems 35 (2022), 34272-34286.
Guy Tennenholtz, Assaf Hallak, Gal Dalal, Shie Mannor, Gal Chechik, and Uri Shalit. 2022. On Covariate Shift of Latent Confounders Imitation and Reinforcement Learning. ICLR. https
//openreview. net http/oormmmmmm www.:ddwwwwwwwwww
Guy Tennenholtz and Shie Mannor. 2022. Uncertainty Estimation Using Riemannian Model Dynam- ics for Offline Reinforcement Learning. Advances in Neural Information Processing Systems 35 (2022), 19008-19021.
Guy Tennenholtz, Uri Shalit, Shie Mannor, and Yonathan Efroni. 2021. Bandits with partially observable confounded data. In Uncertainty in Artificial Intelligence. PMLR, 430-439.
Philip Thomas, Georgios Theocharous, Mohammad Ghavamzadeh, Ishan Durugkar, and Emma Brunskill. 2017. Predictive off off-policy policy evaluation for nonstationary decision problems, with applications to digital marketing. In Proceedings of the AAAI Conference on Artificial Intelligncce Vol. 31. 4740-4745.
Masatoshi Uchara and Wen Sun. 2022. Pessimistic Model-based Offline Reinforvement Learning under Partial Coverage. In International Conference on Learning Representations.
Ashish Vaswani, Google Brain, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. Ist Conference on Neural Information Processing Systems (NIPS 2017), (2017).
J L Vincent, Rui Moreno, Jukka Takala, Sheila Willatts, Arnaldo De Mendonça, Hajo Bruining, CK Reinhart, PeterM Suter, and Lambertius G Thijs. 1996. The SOFA (Sepsis-related Organ Failure Assessment) score describe organ dysfunction/failure: On behalf of the Working Group Sepsis-Related Related Problems of the European Society of Intensive Care Medicine (see contributors to the project in the appendix).
Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. 2021. Provably efficient causal reinforcement learning with confounded observational data. Advances in Neural Information Processing Systems 34 (2021), 21 1164-21175.
Yixin Wang and David M Blei. 2019. The blessings of multiple causes. J. Amer. Statist. Assoc. 114, 528 (2019), 1574-1596.
14

---

Published as a conference paper at ICLR 2024
N.A. Weiss, P.T. Holmes, and M. Hardy, 2006. Course in Probability. Pearson Addison Wesley, https://bboks cttp:///....oooooo...c..cccccc http:l/e..ccoooociadeeeeeeeee
Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. 2021. Bellman-
consistent pessimism for offline reinforcement learning. Advances neural information processing
systems 34 (2021), 6683-6694.
Shu Yang and Judith J. Lok. 2018. SENSITIVITY ANALYSIS FOR UNMEASURED CONFOUND- ING IN COARSE STRUCTURAL NESTED MEAN MODELS. Statistica Sinica 28, (2018), 1703-1723. https://wwws jstor org/ stable/2661111
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. 2020. MOPO: Model-based Offline Policy Optimization. In NeurIPS, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 14129-14142.
wttperoeeeeiinnnnnue...e..eeee.e neurips cctpppppp///2/////////ppp /ttpp//eeeeesaaeerorooooooooop// ffle//8222222000f222228066611613 /ttp://////ooobbe0aa0aaaaaaa0 Paper...
Hugo Yèche, Rita Kuznetsova, Mare Zimmermann, Matthias Hüser, Xinrui Lyu, Martin Faltys, and Gunnar Ratsch. 2021. MHRID-ICUUBenccmarkt - A Comprehensive Machine Learning Benchmark on rishuuesolttion ICU Data. NeurIPS (6 2021). htttps:/phyyionnet www.:ooettttgrggoooeeeeeeeeeeeee crg/cntten/ hirid/1.1.17
Junzhe Zhang and Elias Bareinboim. 2016. Markov decision processes with unobserved confounders: causal approach. Technical Report. Technical report, Technical Report R-23, Purdue Al Lab.
Junzhe Zhang and Elias Barcinboim. 2019. Near-Optimal Reinforcement Learning in Dynamic Treatment Regimes. NeurIPS 32 (2019).
Junzhe Zhang and Elias Bareinboim. 2020. Designing Optimal Dynamic Treatment Regimes: Causal Reinforcement Learning Approach. ICML (2020), 119.
15

---

Published as a conference paper at ICLR 2024
A ADDITIONAL RELATED WORK
A.1 STATISTICAL UNCERTAINTY ESTIMATION
Uncertainty estimation is a crucial aspect of machine learning models, as it provides valuable insights into the reliability and confidence of model predictions and can be used to guide policy optimisation in reinforcement learning. Statistical sources of error can be estimated through aleatoric and epistemic uncertainty, which have been widely studied in the machine learning literature (Hüllermeier and Waegeman 2021). In this section, we review existing methodologies for capturing and quantifying these two types of uncertainty.
Aleatoric Uncertainty. Aleatoric uncertainty, also known as data uncertainty or irreducible noise, stems from the inherent variability and randomness in the observed data (Hüllermeier and Waegeman 2021 This form of statistical uncertainty cannot be reduced even with infinite data quantities.
The most common approach to modelling alcatoric uncertainty is to set probability distribution over model outputs and to learn its parameters (Kendall and Gal 2017). Outputs can for instance be assumed be normally distributed with either fixed variance (introducing single parameter be estimated through maximum likelihood), or a variance that depends on the input. In this latter case of heteroscedastic aleatoric uncertainty, separate neural network branch can be trained to predict the variance (Kendall and Gal (0177).
Epistemic Uncertainty. Epistemic uncertainty arises from the lack of knowledge or ambiguity in the model parameters (Hüllermeier and Waegeman 2021), which can be reduced with additional data. Capturing epistemic uncertainty is particularly important to approximate model error in out-of- distribution scenarios (Tin et al.) 2021).
Bayesian neural networks (BNNs) offer a principled approach to capturing epistemic uncer- tainty (Neal 2012). By placing prior distributions over the model weights and using Bayesian inference, BNNs can provide posterior distributions over the weights, which represent the uncertainty in the model parameters. This uncertainty can then be propagated through the network to obtain predictive distributions that quantify epistemic uncertainty.
Bootstrap ensemble methods are another effective epistemic uncertainty estimation technique (Efron 1982). These methods rely on creating multiple subsets, or bootstrapped samples, from the original dataset by randomly sampling with replacement. Each bootstrapped sample then used to train separate model, resulting in an ensemble of models with slightly different parameter configurations. By aggregating the predictions from these diverse models, the epistemic uncertainty can be estimated through measures such as variance or entropy. Bootstrap ensemble methods provide practical and scalable approach to capturing model uncertainty, particularly when Bayesian methods are computationally expensive or infeasible (Lakshminarayanan et al., 2017).
Monte Carlo dropout sampling (Gal and Ghahramani 2016) can also be used estimate epistemic uncertainty by performing multiple forward passes with dropout enabled at test time. The distribution of predictions from these multiple samples gives an estimate of the predictive uncertainty. Finally, more recent efforts in epistemic uncertainty estimation include randomised priors (Osband et al. 2018), epistemic neural networks (Osband et al. 2022) and deep ensembles trained with Stein variational gradient descent D'Angelo et al. 202 Liu and Wang 2016).
A.2 NONIDENTIFIABLE CONFOUNDING BIAS
Comparison to Sensitivity Analysis. In the causal inference literature, sensitivity analysis studies the robustness of treatment effect estimation to hidden confounding. This framework assumes a bound on the ratio between treatment propensities between any two confounder values (Rosenbaum) 2002) or on the ratio between treatment propensities when accounting for and marginalising confounders
(Jesson et al. 2020 Kallus et al., 2019: Oprescu al. 2023).
In contrast, the main assumptions our delphic uncertainty estimate determine which "world models' compatible with the observational data are considered to construct uncertainty sets over given outcome model (Conditional Average Treatment Effect or, in the sequential setting, Q-value function). In particular, consider set of possible Z and prior distributions p(2), and specify model architecture for the dependence of the behavioural policy and transition, reward or value function on (which is then trained to fit the marginalised observational trajectory distribution).
16

---

Published as a conference paper at ICLR 2024
Importantly, sensitivity analysis approaches require domain expertise to set maximum propensity-ran rratoo parameter I (Kallus et al. 2019). from which uncertainty sets over the modelled outcome are derived. Delphic ORL does come with its set of hyperparameters (number of world models considered in odd pessimism hyperparameter A), which can be determined through practical, quantitative means with arguably less domain expertise.
Time- Vany-na Confounders. The Contextual Markov Decision Process (Hallak et al. 2015) and associated problem described in Section 2 describes confounders as sampled from a context distribution v(z) and fixed over the course of an episode. We note that this framework does not exclude the existence of time-varying confounders. Consider the Markov Decision Process with Unmeasured Confounding (MDPUC) (Zhang and Barcinboim] 2016). in which a new i..d. hidden confounder variable 2, affects the transition at each timestep t. This framework can be framed as a CMDP where the overall episode context 2 = {zI...., 211) includes all confounder variables. Although do not focus on this specific framework in experimental setting, this would form an interesting avenue for further work. An important distinction between MDPUC and Partially- Observable Markov Decision Processes (POMDPs) is the assumption that confounder variables are sampled i.i.d at each timestep. While POMDPs can therefore be viewed as the generalisation of this maaiing setup, note that confounding biases are only induced in this setup if the behavioural policy has access to some missing information about the state variable.
Finally, note that [Saenkkyongam et al. (2023); Tennenholtz et al. (2022) also study confounding in offline environments, but are more concerned with the complementary challenge of covariate shift- with the latter work even assuming access to the contextual information.
A.3 BROADER IMPACT, LIMITATIONS AND FUTURE WORK
Addressing hidden confounding in offline reinforcement learning has the potential to significantly impact the development and deployment of decision support systems in real-worke applications. In real-world data, the behaviour policy is typically that of a human's ss who make use of a rich density of information to take decisions, with important factors likely not all recorded into our dataset. By improving the validity of causal conclusions drawn from data, Delphic ORL can improve the effectiveness and safety of RL-based decision decision-making in critical fields. For example, learning to drive autonomously from observational data could benefit from delphic uncertainty (Haan et al. 2019), as human drivers rely on many unobserved sources of information, such as road conditions, that affect the safety of their behaviour. In digital marketing, another common use of reinforcement learning in industry (Thomas et al.) 2017), learning from observational campaigns without A/B testing can also be prone to confounding. Finally, emerging decision support systems in justice Sians-Combe [002)). hiring (Sánchez-Monee) hidden confounding biases if unaccounted for.
et al. 2020)) education (Singla et al. 2021) are likely suffer fromm
While our results demonstrate the efficacy of Delphic ORL in learning useful policies in the presence of confounding, it is important to acknowledge the limitations and potential unintended consequences associated with RL algorithms, especially in high-stakes applications such as healthcare. Collabo- ration with domain experts is crucial to ensure thorough evaluation of RL algorithms (Gottesman) et al. 2018). In clinical settings, predictive or recommendation models derived from Delphic ORL should not be solely relied upon, and mitigation strategies must be implemented to minimise negative consequences during deployment.
An important consideration in the application of Delphic ORL is the trade-off between confounding bias and estimation variance in Q-function estimation, as noted in other work addressing confounding bias (Wang and Bleil 2019). This emphasizes the significance of large, high-quality training datasets to leverage the benefits of Delphic ORL and ensure sufficient predictive power.
As our experiments primarily focused on medically-motivatedd confounding scenarios, future work should investigate the applicability and generalisation of Delphic ORL to other domains. Although our framework does not in theory exclude dynamic environments where confounding factors change over time (see Appendix(A.2) an empirical study of the behaviour of delphic uncertainty estimates and pessimism penalties may reveal new challenges in this context.
Finally, the question of how to best approximate the set of compatible worlds W in Definition 4. remains open. In Section 5 and Appendix C we detail our approach which efficiently captures variability across counterfactual fulue-funetion, but further theoretical or practical work on how
17

---

Published as a conference paper at ICLR 2024
best to model W would likely improve the calibration of delphic uncertainty estimates. Better approximation algorithms may also improve the efficiency, scalability. and modelling power of our method for very nigh-dimensional, highly confounded problems although our real-world data analysis forms a promising first proof-of-concepe
B THEORETICAL DETAILS
B.1 PROOF OF THEOREM 4.2
We start by considering the decomposition of variance in Q caused by random variable 0. In the following, we drop superscript for clarity.
First, we decompose Var(Qe
0):
Var(Qo 0) E[Q 1
[ar 0)] = Es [B Eg [B[20
E[Qa1 Eg
[ 0]
where the last line results from the law of iterated expectations: Eg [[ BJ] E[A) for random variables A, B.
Next, we study (ar 0)):
Vara (E(Q 0)) = Eg[E[Qu 0 Eo 811
= [B[0 0] E[Qa]²
(4)
again using iterated expectations.
Summing equations and we obtain:
Eg Var(Qe 0)] + Vara (E) 0)) = LE(Q)1 - E[Q+]
Var(Qe)
(5)
This result is known as the law of total variance (Weiss al. 2006). which can be interpreted as decomposition of epistemic and alcatoric uncertainty (Kendall and Gal 2017).
We can rewrite the above result within given world model denoting 0 as 0w. Now conditioning on the world model w, we have:
Var Qa ww) = Eg Var(Qu., Owe w)]w]] + Varo,, (B[Qu... Owe w]]w
We also write equation 5 such that the conditioning random variable is now which induces variation in Qa_ if we consider counterfactual trajectory distribution. Combined with Equation (6), we obtain:
Var(Qe..) = En [Var[ (a.. [w)] + Var,, (E(Qe.. wr])
Ew [ew Var(Qo_ Ow ww| + Varg.. (E(Qo. I Ow, w]]w)]
+ Var Es... [[[Q... 0w w||l]]
using iterated expectations. This concludes the proof of Theorem 4.2
Note that if we assume Qa has Gaussian distribution for fixed {0w., w). parameterised as N (Hoo. ) we have Var(Qu.. Ow, w) = o% and IE[Q._ | Ow, w| = pea. We obtain re- sults in Equation (2 by expanding the first term in the variance decomposition, Ea_ 10 |w), as follows:
Eg_ [0 |w] = Eog 103 /w] - Egg,, [0a_. [w] + En, [ao_ |om_|ww] Varo,, (0o.. (w) Ro_ [00., (000, 1w]2.
18

---

Published as a conference paper at ICLR 2024
B.2 ASYMPTOTIC INTERPRETATION OF THEOREM 4.2
We consider three extreme cases of Theorem 4.2 to clarify its decomposition. First, we consider the limit of infinite-date with confounding (e.g., s,,, dependence on 2). In this case, 0w and w converge to a single ground-truth Any remaining statistical error will come from the intrinsic environment stochasticity or the behavioural policy, and therefore has aleatoric nature. Indeed, only the first term in Theorem 4.2 would remain.
Next, consider the setting in which the value is a deterministic mapping of states, with only one compatible world model. Learning from finite data quantities leads to statistical error in optimising the parameters 8w.. and is known as epistemic uncertainty. Indeed, deterministic environments with only one compatible world model will reduce Theorem 4.2 4.2to to the second term.
Finally, we consider the case of infinite data in a deterministic setting. In this case, multiple compatible world models may exist which induce the same observational distribution (as demonstrated in Section 3). The source of error remaining is delphic uncertainty, and arises if multiple models assign high likelihood to the observational data, but return different estimates of the value. In this paper we propose to estimate this final form of uncertainty by learning an ensemble of compatible world models, in similar fashion to the bootstrap method for quantifying epistemic uncertainty.
C IMPLEMENTATION DETAILS
C.I STATISTICAL & DELPHIC SOURCES OF UNCERTAINTY
World Model Training. We implement world models as variational models for estimating the confounder distribution, jointly with a model for the behaviour policy T. and for the action-value function QTh, both dependent on a z sampled from the posterior. As the environments we consider have discrete action spaces, learn the behaviour policy by minimising its cross-entropy on the training data, as in behaviour cloning. Training is carried out for 50 epochs or until loss on the validation subset (10% of training data) increases for more than 5 consecutive epochs. Within a world model w, hyperparameters {a, }3} can be tuned based on prediction performance on the validation set.
Model QTh corresponds to an on-policy action-value function approximation. We compute targets through Monte Carlo updates (for the sepsis environment with sparse episodic rewards) or Temporal Difference learning (for the real-world ICU dataset) based on samples from the observational training data with discount factor of y 0.99. The Q-function is trained
classifier over 200 quantiles.
Between world models w, the confounder space dimensionality is randomly varied over |2| {1, 2, 4,8, 8, 16). and the prior for p(2) = ((2;0, Σ²) is randomly varied through the variance for each z-dimension, 23 {1.0, 0.013. For the sepsis simulation, the encoder architecture for the confounder distribution v(=[T)) consists of a multi-layer perceptron with hidden layer dimensions (128, 64, 32) and ReLU activation before the final layer mapping to dimension Z For the real dataset, the encoder architecture is implemented as a transformer (Vaswani et al., 2017) with 2 layers, heads, and embedding dimension 32, considering a maximum history length The behavioural policy ma(ass, 2) and oetion-value function Q** 2) are both implemented as multilayer perceptrons with hidden layer dimensions (32, 64, 128) and ReLU activation.
tokens.
Uncertainty Estimates. Additional inductive biases can be incorporated to capture epistemic and aleatoric uncertainty within single world model w, as these relate to statistical sources of uncertainty. Following prior work (Kendall and Gal 2017; Yu et al. 2020). we capture alcatoric uncertainty by modelling normal probability distribution over outputs (mo )T We then measure epistemic uncertainty within each world model by training on different data bootstraps, returning an ensemble of parameters (0),, .......) for each w.
Recalling Equation 2 the delphic uncertainty term Var,:. (Ee[pe,,)) is estimated by measuring the variance between predictions 148_. (averaged over model parameters 0..), across across
multiple generative models w. Epistemic uncertainty can be estimated as the variance of outputs over different model parameters 9u. averaged across worlds w Z. Finally, aleatoric uncertainty is measured through the fitted probability distribution over model outputs Q averaged over all 8., in given world, and over all worlds W.
19

---

Published as a conference paper at ICLR 2024
Algorithm 2: Delphic Offline Reinforcement Learning: Bellman Penalty in Offline Q-Learning Algorithm.
Input: Obxervational dataset DD Model-free Offline RI. algorithm (e.g. CQL. Kumar et al. 20200) Penalty hyperparameter X. Learn set of compatible world models (Zw, Vus Po,e Pr Paas Tw, Tb New that all factorise
(T).
Obtain counterfactual predictions QT, for each w W.
Define local delphic uncertainty: ( a)) Var, (Q a)).
Initialise Q-function parameters @).
for each iteration do
Sample (s, a, r, s') ~ D.
Compute penalised Bellman target: Qtarget r + maxo'E D')ss,a'---o a), where <<als)) argmax, Q,(s,a).
Perform gradient descent w.r.t. & on [Qo(s, a) Chargerle (s. a)] + flmee (Ø), where regularisation term fiine depends on the choice offline learning algorithm.
10
end
The number of world models W was varied between 5 and 20 for both datasets and was chosen as the smallest number converging to an average delphic uncertainty comparable to the largest W. An ablation of delphic uncertainty as a function of the number of world models given in Appendix E This resulted in 10 and 15 world models for the sepsis and real-world datasets respectively. Finally, each world model was trained over 5 different data bootstraps to estimate epistemic uncertainty. Overall, compared to sensitivity analysis where parameter T needs to be fixed through domain expertise (Oprescu et al. 2002), we found delphic uncertainty to be less dependent on expert input in determining model parameters.
Counterfactual Estimates. Our approach is to change the policy term in Pao to ob- tain counterfactual estimates, using the importance sampling estimator Q (s,
slale) Er-plE (=1>) Rh. [c]s,3] 2 (s.a,2) Note that other factors in the world model (e.g. Vu- Qw) could be varied to obtain general counterfactual predictions in this world model. As an example, we also found promising results by measuring delphic uncertainty through variation across w over the following counterfactual quantity: (((.a))DE z-pp Eg... 1Q7 (s, a, =)]. where is sampled from the model prior (z) instead of the learned posterior Ve (z]r). In this case, the resulting delphic uncertainty estimate, capturing variation over the counterfactual quantity across world models, be- comes independent of given policy and dependent the new quantity introduced (in the previous example, on prior Prr (z)).
C.2 DELPHIC OFFLINE REINFORCEMEN LEARNING
We detail our learning procedure in Algorithm As our base offline RL algorithm is CQL (Kumar et al. 2020), our regularisation term Raf flime()) is the CQL penalty: (tie(())))
log Eaea exp Q as (s. à) Qo(s, (.). We base our algorithm on an existing implementation for CQL (Seno and Imai 2022), which includes additional training details for stability, such as target networks, double Q-networks and delayed updates (Fujimoto et al. 22018). For architecture details, see the baseline implementation of CQL in Appendix C.3 As for all baseline algorithms, we train for 100 epochs, using 500 (sepsis dataset) or 10a (ICU dataset) timesteps per epoch. In practice, the policy π considered for uncertainty estimation and the target network are updated every 8000 timesteps, to improve stability in training.
Note that actor-critic variant of Algorithm is also feasible, setting in ui to be the actor policy, as well as other offline learning paradigms in Rof (Ø), such as BC regularisation (Fujimoto and Gu 2021
Alternative Forms of Pessimism. Following the discussion on alternative forms of pessimism in propose practical alternatives to the Delphic ORL penalty in Line 8 Algorithm 2 which substracts a factor of u, from the Q-function Bellman target based. In the following. note that Wd can also be independent of if varying over different factors in Px, as detailed above.
Section 5.2
20

---

Published as a conference paper at ICLR 2024
Delphic ORL via Uncertainty Threshold: One approach, inspired by Batch Constrained Q-Learning (Fujimoto et al., 2019), is to constrain value function updates to only consider actions falling below certainty uncertainty threshold. For tuple (s, a,ar, s'), the Q-function Bellman target can be computed as: Qiarget = r + maxa':u (o" <')<< Q (s', a'), where À is threshold controlling the maximum delphic uncertainty accepted for a given action choice.
In model-based methods, penalty proportional to the uncer- tainty wa(s, a) can be substracted from the reward function r(s, a), as in Yu et al. (0200).
Model-Based Delphic ORL:
The effective reward function becomes: f(s, a) = r(s,a) -u,,))))_sss)
Delphic ORL via Weighting: The uncertainty measure can also be used to weight samples in the objective function, prioritising unconfounded states and actions during training:
A
(o,a.))
((s,a,))
ud(ss,)
where L can be the Q-function Bellman update the supervised learning objective for behaviour cloning.
We compare the performance of different implementations of pessimism on the simulated sepsis enviromment in Appendix [E] Finally, as delphic ORL benefits from full trajectories in world model training, we also measure whether introducing history-dependence can help our offline RI framework. We achieve this by modifying our implementation of CQL. to incorporate representation of history in the Q-function and/or the policy networks, extracted through transformer.
Hyperparameter Tuning. There is no natural validation criterion in Offline RL, and the best approach to choose hyperparameters in this context remains open question (Levine et al. 20200). In practice, we run our algorithm for different values of À E {10 10-2, 10 11-1,1) and choose the final policy giving the best off off-pc evaluation performance on the validation set (using the Fitted Q-Evaluation implementation available in the codebase, Le et al. (2019)). As noted in related works, expert input may be useful at this stage to also determine how strong penalty again potential hidden confounding would be desirable or how much confounding could be expected (Rosenbaum) 2002). Other hyperparameters specific to offline RL algorithms tuned in the same way and given in the following section.
C.3 BASELINE METHODS & TRAINING DETAILS
All reinforcement learning algorithms and baselines are implemented based the open access d3rlpy library (Seno and Imail 2022). The discount factor used is 7 = 0.99, and state and actions normalised to mean O and variance (Fujimoto and Gu 202I) for all algorithms. Training is carried out on NVIDIA RTX2080Ti GPUs on our local cluster, using the Adam optimiser with default learning rate and a batch size of 32. Models are trained for 100 epochs with 500 (sepsis dataset) or 10 (ICU dataset) timesteps per epoch. Model Model-specific hyperparameters are tuned as in Delphic ORL.
Behaviour Cloning (BC). Behaviour cloning (Ross et al. 2011 a supervised learning model of the behaviour policy, mapping states to actions observed in the dataset. After considering the following architectures: multi-layer perceptron (MLP), Long Short Term Memory (LSTM) network (Hochreiter and Schmidhuber 1997), Gated Recurrent Unit (GRU) (Cho et al.) (0144) and Transformer ( (Vaswani et al. 2017), GRU was found to give the best validation performance on both the simulated and real datasets. Implementation details for the GRU BC models include two hidden layers of dimension (64, 32) and ReLU activation. The last layer is passed through a softmax layer to produce action probability outputs, and the model is trained by minimising action cross-entropy over the observational dataset, with L2 regularisation of weight 0.01.
Conservative Q-Learning (CQL). Discrete CQL (Kumar et al. 2019) is implemented with penalty hyperparameter a of 1.0 (sepsis environment) and 0.5 (ICU dataset), tuned over the following distributional model with
values: {0.1, 0.5, 1.0, 2.0, 5.0). The Q-funtioon is implemented standard MLP architecture (two linear layers with 256 hidden units) and 200 quantile regression outputs.
21

---

Published as a conference paper at ICLR 2024
Batch Constrained Q-Learning (BCQ). Discrete BCQ (Fujimoto et al. 2019) is implemented with a threshold for action flexibility set to 0.5 for both environments, tuned over the following values: {0.1, 0.3, 0.5, 1.0, 2.0, 5.0). The Q-function is implemented as distributional model with standard MLP architecture (two linear layers with 256 hidden units) and 200 quantile regression outputs.
D EXPERIMENTAL DETAILS
D.1 DECISION-MAKIN ENVIRONMENTS
Sepsis Environment. Introduced by Oberst and Sontag (2019). this environment simulates the trajectory of patients in the intensive care. Based on the authors' publicly available code our state space S consists of 4-dimensional observation vectors (measures for heart rate, systolic blood pressure, oxgenation and blood glucose levels) which we normalise to mean and variance (0, 1). The discrete action space A comprises the combination of three binary treatments (antibiotic, vasopressor or ventilation administration) for total dimension of 8. An unobserved binary variable encodes the diabetic status of patients, with 20% of trajectories having positive status. The agent obtains a reward of +1 if the patient reaches healthy state (and is thus ready for discharge) and negative reward of 1 the patient reaches a death state.
The observational dataset D is generated by rolling out the optimal (diabetes-aware) policy in the environment for 10,000 environment interaction steps, taking a random action with probability é = 0.1 to ensure sufficient state-action coverage for offline learning. The maximum episode length set to 20 timesteps. The resulting dataset has confounding strength of If 100.
Environment stochasticity can be varied by changing the variance around the originally deterministic reward obtained at the end of a trajectory, between o² = O as in the original environment and o² = 0.4. Datasets of varying confounding strength I E [1, 100) are obtained by setting the behaviour policy for as weighted average of the policies for different values: (1 (1-p) =)T [== = 0) prs(= = 1),
where p depends on I and e. Environment transition and reward functions and their dependence on are kept fixed. Finally, vary the dimension of the confounder space Z by introducing more binary indicators with the same effect on the transition dynamics as the diabetes indicator.
Electronic Health Records Dataset. Our real-world data experiment is based on the publicly available HiRID dataset (Hyland et al. (020). This dataset counts over 33 thousand patient admissions at an intensive care unit in Bern University Hospital, Switzerland (Hyland et al., 2020) and can be
pre-processed using open access code from the HiRID benchmark (Yèche et al. 2021). Patient stays
were downsampled to hourly measurements and truncated to a maximum length of 20 hours and default training, validation and test sets were used.
We consider the task of optimising fluid and vasopressor administration (AA is the combination of two binary choices). The reward function is designed to penalise circulatory failure events (r = -1 for all timepoints in the duration of the event) and to reward timepoints where the patient is not in such a critical state (r = 1, and r = 2 in the timepoint following recovery from circulatory failure). Circulatory failure events for each patient are labelled following internationally accepted criteria (Yéche et al. (2021). This short-term reward function is dense, unlike previous RL work on optimising intravenous fluid and vasopressor administration (Raghu et al. 2017). J, making off- off-policy evaluation more reliable (Gottesman et al. 2018).
The state space S consists of all variables in the electronic health records which are not considered treatment for the organ system considered, based on the variable categorisation released with the dataset (Hyland et al. 2020p. This results in state space dimensionality of 203. The list of variables excluded for each task in given in Table At each timepoint within a patient stay, we also compute the Sequential Organ Failure Assessment (SOFA) score (Vincent et al. 1996) which is used to quantify the severity of a patient's illness in the intensive care unit. A higher score indicates greater severity of illness.
Selected confounders are obtained by excluding some state dimensions from the observational dataset (up to |2| = 14). These variables do not constitute the entire confounder space, as much exogenous, unrecorded information affects patient evolution and is taken into account in medical treatment
https:://githubb.com//////
/gmm///m -max- scm
22

---

Published as a conference paper at ICLR 2024
Table 3: Offline reinforcement learning task on real-world medical dataset.
Task
Circulatory treatment
Action space A = {0, 1}2
Fluids
Vasopressors
Organ failure avoided by R
Circulatory failure
State space S (selected variables, |S| 204)
Heart rate
Respiratory rate
Body temperature
Urinary output
Blood pressure
GCS score
Cardiac output
Central venous pressure
Oxygen saturation
Base
Lactate
Arterial pH
PaO2
Creatinine
Serum sodium
Serum potassium
Haemoglobin
Glucose
Other lab values
Ventilator settings
Antibiotics
Steroids
Diuretics
Insulin
Cerebrospinal fluid drain
Anticoagulants
Other treatment variables excluded
Blood product infusions
Vasodilators Antiarrhymic agents Antihypertensive agents
Cristalloid infusion
Colloid infusion
Confounder variables z
Age
Cardiovascular diagnosis
Weight
Pulmonary diagnosis
Gastrointestinal diagnosis
Orthopaedic diagnosis
Neurological diagnosis
Metabolic/endocrine diagnosis
Hematology diagnosis
Trauma diagnosis
Sedation
Intoxication
Emergency status
Surgical status
23

---

Published as a conference paper at ICLR 2024
decisions (Yang and Lok 2018). We ignore this in analysis cannot evaluate with respect to this missing information, but we note that this is precisely the motivation behind our work.
The confounding strength for each confounding space considered was estimated as follows. Each point in the training dataset was binned into a (s, 0, 2) category, depending on its discrete action and context values (a, 2) and on its SOFA score as a summary variable for s. We discretise the SOFA score into quantiles. Finally, we compute the mean policy value for each (s, a, 2) bin through Ta(als, 2) P(a, 2) Ps,(s 2), and we take I as the ratio max is, (Tt. [al(a 2)/ mT(aass =a)s,))). =')]-
D.2 ANALYSIS DETAILS
In this section, we provide additional details pertaining to the analysis of our experimental results. All results reported in this work include 95% confidence intervals around the mean, computed over ten training runs unless otherwise stated. Environment returns and off-policy evaluation results are normalised scale of 0 to 100. Figure { was obtained by varying the dimension the x-axis, while keeping the other variables fixed to N 864 trajectories, confounding strength I = 15 and reward function variance of 0.0. To generate Figure 6 patients with the relevant confounder (=) value were binned by disease severity and the probability of vasopressor prescription (top) and the overall density (bottom) in each group were computed. Figure (60 was then obtained by computing the relative increase in delphic uncertainty when including the relevant 2-dimension to the hidden context space 2 (in other words, removing this dimension from the visible state space).
Off-Policy Evaluation (OPE). Doubly robust methods trade off bias of an approximate reward model and of weighted methods with the high variance of importance sampling approaches (Jiang and Li 2016). Assuming is accessible for each trajectory evaluation time to overcome confounding, doubly-robust oof-policy evaluation estimates the value of policy i as follows:
#(als) (r Q(s a(s,a,))) + Q(s, (6))
VD (f) (e....))
Q(s,+(s)))))
(8)
Aa(a)s,
where it is a model for the behavioural policy and Q for expected returns under म, learned on the dataset with observable
Fitted O-Evlluation is an established value estimation method (Le et al. 2019). The algorithm iteratively applies the Bellman equation to compute bootstrapping targets for Q-function updates: QK+1 f arg ming [(......) [(r Q(s, a, =) + 7Qu #(s'),)) 2}}²] which can be solved as a supervised learning problem. This results in learned Q-value for the evaluated policy Q* (s, a, =)
which can be used in the weighted doubly-robust estimate in Equation to provide return estimates in Table 2
Both the Q-function and the behaviour policy in Equation S are parametrised as fully-connected neural network dimension with 3 layers of hidden dimension (64, 32, 16) and ReLU activation. The former is trained by minimising the mean squared error with the Q-function update above, the latter by minimising the cross-entropy with respect to action choices in D.
Human Policy Evaluation. Off-policy evaluation has limitations, being itself prone to its own set of statistical errors and data-related concerns (Gottesman et al., 2018). We aim to confirm conclusions drawn over OPE returns through human expert evaluation of treatment policies.
Synthetic patient trajectories are first generated by randomly sampling from the ICU dataset along each state dimension, with varying amounts of contextual information as detailed in Table [ Action choices at the end of the trajectories are computed for the Delphic ORL, CQL and BC policies, trained on the observational dataset with the same degree of confounding. Trajectories are selected if they induced a disagreement between these methods, to shed light on potential improvements or harmful behaviour learned by the offline RL models. Trajectories are then simplified into 12 critical variables (as shown in Figure D). and shown to physicians, who are asked to rank two treatment options in terms of expected patient outcomes. Unknown to the physicians, and in random order, one of the options was predicted by the Delphic ORL or CQL policy, and the other by the BC baseline. Overall, we consulted six clinicians with different degrees of expertise in intensive care (from junior assistant doctors to department heads) from Switzerland and the United Kingdom, collecting their treatment preferences over 45 such trajectories.
24

---

Published as a conference paper at ICLR 2024
Table 4: Data settings considered during expert clinician evaluation. Physicians asked to rank action choices based on only state information 200), or with varving amounts of observed contextual information ((All 14) refers to all possible Z variables outlined in 3
r
1
15
20
100
200
2
0
10
11
13
14
Observed
(All 14)
(Age.
(Age.
(Age)
0
Neuro. diag..
Neuro. diag.
Trauma diag..
Surgery)
Surgery)
We contacted our local institution's 's ethics committee to enquire about the possible necessity of ethics approval for this experimental framework. We were informed that this not considered necessary as the experts contribute to the validation of algorithms and are thus not themselves the subject of the research, and the undertaking comes with minimal risks those experts (anonymous data collection). Best practice was nonetheless observed, by providing participants with an information and consent letter (Figure 8) to inform them of their rights and obligations, and of how their data is collected and used. Participants were asked to read and sign this letter before collecting their anonymous expert opinion.
Results in Figure report the preference of clinicians for actions from either Delphic ORL or CQL or from behaviour cloning. We note their overall preference for the Delphic ORL policy in the confounded settings (high [). As more contextual information about the patient becomes available, however, and confounding is less marked (small I'), physicians favour the behaviour cloning policy closer to expected clinical practice.
E ABLATIONS AND ADDITIONAL RESULTS E. CONTEXTUAL BANDIT ENVIRONMENT
We also conduct an experiment based on the illustrative example provided in Section 3 We define the confounded simulation environment as in World 2 (Figure 2c) and generate 5k pairs as observational data. We present our results in Table 5
Table 5:
Comparison of offline and delphic offfine RL on contextual bandit experiment (Figure
Reward function
41
022
Pr. a): Ground truth observational reward
0.6
0.5
0.6
Pa (rla): Estimation + epistemic uncertainty
0.59 + 0.02
0.47 + 0.02
0.59 + 0.02
E. [.[P (aaa) =)]: Ground truth interventional reward
0.41
0.5
0.41
E. [.[P(rla [a =)]: Estimation + delphic uncertainty
0.46 + 0.29 0.53 + 0.02 0.59 + 0.28
The first two rows capture the ground truth and estimated reward function, based on the observational data. The latter is what would be optimised in RL or offline RL (with penalty on epistemic uncertainty), resulting in policy which chooses ag or as optimal.
The second two rows capture the interventional reward distribution, without the confounding effect of Te. Delphic ORL provides estimate of this, correctly identifying high delphic uncertainty in confounded states ag or 02. The resulting policy chooses an as the preferable action.
The ground-truth interventional reward (row 3) determines the expected reward for each action, showing that the delphic ORL policy (a1) outperforms the offline RL one. Overall, this experiment further supports our method and helps understand its mechanisms.
E.2 SEPSIS ENVIRONMENT
Ablation Study: Delphic Uncertainty. In Figure 10a we find that delphic uncertainty is highest on the sepsis dataset when treatment involves vasopressors. By design of the simulation (Oberst) and Sontag 2019), this treatment is the only one for which patient evolution is confounded by the hidden diabetic status, which further supports the conclusion that delphic uncertainty captures model
25

---

Published as a conference paper at ICLR 2024
have read and understord the zudy informat on. My quessiors have been answered
camply witn the inckss messonaae exci usion tenta partectpat fon descrbbed sbrve.. am compleide and sodivactiion
parto in this study voluntarly By and consert my personal be as deseribed above.
Signature partelpent
Signattre condurding
understand thst can stop parti pattppaingggggooo at any moment.
have enough decicle my perticp
woule like to be infor informes about the rosuts of works
1 Yos, Name and Phono Numbor or Emali: D No
the paricponn confim my signature that:
Ful name of partidiprt
Consent Form
Plack Date
Plack Date
Evaluation of new reinforcement learning algorithm applied to treatment optimisation
partkippate project person
validdate
not enough
treatron rostmmeen cppltoro for difieennt patjunt cheutss bas collected No perionelly b diiin cless will be col leddedd and no third--pe third-ppprr per poroonal cisld.
sprrttyy providiers will hand to any
We wit to publicly release anorymmann preference resulte together with paper and code to ereblee rewoeeeeeeeeeeeeeeeeeeeeeee recron peroonuctt bubiitt Aneny Anenymbedd preference dels will also the etored on a secure compuiing tnsuuurruuuree (REDACTE 'ED).
Your partiiiption stucy is volumbiry. moy whdrrww your partle pation any with specilying
verying amount informaioon each Cholize belaveen iestrtet cettrtttooooos antidpated boneit
mnsouuringg nemment gredg eeeeroeeere pardommeed gynthelic pellent
A potantial bursift perticipating to faciitato viledistin of novel algarttee, coull be useld future trossiiee recoo innerdatiion systemm..
aroee
Please dhocse the stoalmenn opton you believe would lesd to optimal outcomes esam ing patient charts, You can dhoose not to answer if you
There no known introcieeed parte jasting In this study colleced entinely monoymous.
ixperiment late voaldaaiion
strdeegios
emperttios intersaie care.
infrrmmennt sariing algonihmm pous on
and cnty dieaao craggge
What data le collection ted from mee and how ls used??
REDACTED REDACTED REDACTED
am suppo do partic ipatt??
Information consent form
panticipaation??
What riaks berwits can inppett?
inventignn weeeeigggedd devallope ped alporimmm learn
Connuuting (ulrrame)
Who funds this study??
Partoppant name).
Who can participate??
Annnymiaad prdiineeio
Einks commisiion
echiove thes we
Information to make
What are my rights
Comploints office REDACTED TED
REDMC TED
Figure 8: Full text of information and consent letter given to participants.
26

---

Published as a conference paper at ICLR 2024
Contextual information:
Patient age: 30
Neurological Irauma patient surgery
At the end of this patient trajectory, would
A. Administer fluids?
B. Not administer fluids?
Figure 9: Illustration of action ranking by medical experts. Synthetic patient trajectories and varying degree of contextual information (varying ||| and I' are given clinicians, who must rank the trcatment options in terms of expected patient outcomes.
0.0011
0.0304
Delphic uncertal uncertainty Ud
1s]
Delphic uncortaitty Ug
0.0203
0.0010
0.0009
0.0303
0.0309
Vorc
Vont Vao
Mnd Vor-
[35
And
Ant Vent
(a) We function of a.
(b) 2g as function of [W].
Figure 10: Ablation Study: Delphic Uncertainty. (a) Delphic uncertainty is highest under vasopressors in the sepsis environment, correctly identifying their confounded effect (Abbreviations: Vaso Vasopressors, Anti Antibiotics, Vent Ventilation). (b) Empirically, only small number of compatible worlds (for sepsis, [W] Rs 10) is necessary to obtain an asymptotic estimate of us-
bias due to hidden confounding. As a result, delphic ORL penalises more strongly the action of taking vasopressors only (which induces high uncertainty with respect to possible confounders) than other actions with a more certain value estimate, including combinations of vasopressors with other treatments. Naturally, the performance of the resulting policy weaker than baseline with access but is stronger than a policy based on a value function with hidden confounding biases.
in Figure 5
In Figure 10b we note that only small number of world models (for sepsis, [W] = 10) is necessary to obtain an estimate of delphic uncertainty consistent with large number of world models. This motivates our practical choice to only consider small set of world models to obtain reasonable estimate of uncertainty for Delphic ORL, but warrants further theoretical work establishing guarantees and probability of correctness.
Delphic ORL. In Figure I1 we study the performance of Delphic ORL as function of hyperparameter interpolating between naive implementation of Offline RL for very low values of 7 (virtually no penalty) and an excessively pessimistic algorithm, where the confounding penalty overcomes any possible high-reward behaviour.
Ablation Study:
confound
Next, Table compares the performance of different approaches to implement pessimism with respect to delphic uncertainty. We find that our approach proposed the main paper, based on penalising the target for the Bellman update, performs best in this experimental setting (sepsis dataset with I = 46). Weighting- based approaches also show promising performance (either matching or improving the performance of BC and CQL., respectively). which may be avenue for further work
27

---

Published as a conference paper at ICLR 2024
Algorithm
Environment Returns
Online RL.
67.8 1.1
Environment Returns
80
BC
38.5 1 4,5
BCQ
18.5 2.4
60
CQL
31.1 + 3.5
CQL with history-dependemt Q
26.9 = 3.1
CQL with history-dependent Q.
23.5 2.0
Delphic ORL with [ 46
32.7 = 4.8
Delphic ORL (worst-case Q
18.1 = 2.0
Delphic ORL Tureshold)
24.6 3.4
Delphic BC (Weighting)
39.6 4.1
10-1
10-2
10-1
10P
Delphic ORL (RRi.WWeightig)
A
Delphic ORL (Algo. 2
$4,9 4.6
Figure 11: Performance results as a function of hy- perparameter À on the sepsis environment (T 16).
Table 6:
Performance of different pessimism
methods on the sepsis environment (T 16).
and fine-tuning. Modifying the Bellman target only include actions below certain uncertainty threshold was however found to be excessively pessimistic, and degraded performance compared to the base CQL algorithm. Model-basee based Offline RL and Delphic ORL were not included their performance was never found to improve over a random baseline policy. Finally, setting the Bellman target as the lowest Q-function within the set of compatible worlds models is the closest setting to related work on robustness (Kallus and Zhou 2018 Wang et al. 2021) | and is also too pessmistic..
We compare to the sensitivity analysis framework for estimating the set of compatible worlds (Kallus and Zhou| 2018). We implement this through our variational approach. discarding compatible world models that do not satisfy the r = 46 assumption (all but 2 out of 10 trained compatible world models). This results in a poorer estimate of delphic uncertainty as the resulting delphic ORL performance is negatively affected (Delphic ORL with r = 46 in Table(6).
We hope this ablation study will motivate further work into the best possible approach to implement pessimism with respect to delphic uncertainty, to learn offline RL policies that are robust to hidden confounding bias.
We also measure whether introducing history dependence in training helps performance, as our algorithm indirectly uses history when computing delphic uncertainty (although both Q-function and policy are only state-dependent). In fact, we find that direct history dependence harms performance. which can be explained by the shift in history distribution between training (history distribution of the behavioural policy) and execution time (history distribution of the acting policy), as discussed in Ortega et al. (2021); y Swamy et al. (2022). Overall, this result further motivates our focus on policies where confounding biases only affect the value function. delphic ORL, only the delphic uncertainty term is used as penalty for Q, with limited possible utilisation of history as ua is computed over the entire dataset.
E.3 WORLD CLINICAL DATASET
In this section, we provide additional evaluation metrics and investigations to understand the treatment strategies identified by the different algorithms considered, and in particular how Delphic ORL determines aanfounding-robuse policies.
Interpreting expert preferences. Figure 7 shows the preference of clinicians for offline RL policies (Delphic ORL or CQL) over the BC policy. A score of 0% means clinicians always prefer the BC policy, whereas 100% means they always prefer the offline RL one. Overall, found the preferences of the clinicians to be noisy (66% agreement in action preferences on average), which explains why preferences remain close to 50%. Under no confounding both offline RL ones (preference <50%). as it should accurately capture doctors' eecision-making in this conconnfudded setting. Optimising a reward signal (necessarily distinct from doctors' internal objective) for offline RL therefore gives treatment policies that do not match the doctor's 's preference as well.
0), clinicians prefer the BC policy to
As confounding is increased by removing critical deciiiion-maiing factors from the data (2 = {all] } however, doctors prefer the delphic ORL policy to the BC one ~58% of the time. This highlights both that BC degrades in performance due to missing information, and that delphic ORL becomes more favouralle as it takes conservative actions in the face of unknown possible confounding (see
28

---

Published as a conference paper at ICLR 2024
below). On the other hand, doctors now prefer the CQL policy to the BC one only ~41% of the time. This suggests that CQL, picking up on spurious correlations, proposes an even worse treatment policy than BC with missing information.
Understanding the delphic ORL policy. Table 7 provides a quantitative analysis of the disparities in action choices between different algorithms and the doctors' policy. As expected, behaviour cloning exhibits the closest resemblance to the doctors' treatment policy, which aligns with the characteristics of observational datasets. However, our proposed method outperforms behaviour cloning in terms of learning distinet policy that deviates from the doctors' actions. These findings highlight the unique capabilities of our method in capturing important features and patterns beyond the direct imitation of doctors, enabling the model to make informed decisions that may differ from the observational data and potentially lead to improved treatment outcomes.
Table Difference in action choices from Drest across different algorithms (%). Our method learns distinct policy from the doctors', Mcan and 95% Cls over runs. Highest and overlapping values in bold.
Confounders z
BCQ
BC
CQL
Delphic ORL
All helow
32,2 + 1.3
19.7
+ 1.1
33.4 + 0.9
35.2 + 1.5
(age)
31,5 + 1.3
12.8
0.4
27.3
0.3
32.3 + 0.5
[neuro. diag.)
31.1 +: 1.3
16.3
1.0
34.3
+ 1.3
30.6 f 1.1
gastro. diag.)
27.1 +: 1.1
14.3
0.9
28.9 1.1
29.4 1.3
trauma]
10.1::1
1.4
0.4
22.2
cardio. diag.}
28.7 1.3
18.8 +8.8
36.2
+..2+1.3
29.6 1.6
endo. diag.)
27.4 1.3
0.8
0.9
23.1 0.9
hemato. diag.)
30.1 1.5
1.84
1.14
23.6
weight)
28.9 1.3
13.2
0.64
23.6 1.2
sedation]
30.5 1.5
14.5
2.11
25,8 + 1.0
resp. diag.)
27.7 1: 1.3
0.
2.15
25.2 1 1.2
intoxication
25.7
+: 1.1
02.6
0.6
23.1 + 0.9
surgical status)
27.3
1.3
0.6
2.89
22.1 1 1.2
ortho. diag.)
25.6 + 1.1
0.6
2.11+1111
22.3 + 1.0
scpsis]
26.1 + 1.1
23.5 0..6
21.9 + 1.2
25,3
1: 0.9
1.22
23.1
0.8
21.7 1: 0.9
Following published recommendations on evaluating RL models in observational settings (Gottesman) al. 2018). we also analyse where policies differ most from the action choices in the observational dataset, and find that the policy learned by Delphic ORL diverges most at high disease severity (SOFA scores 15-20). these cases, our policy appears to prescribe less fluids and vasopressors than the data which may be reasonable if unsure about possible adverse effects of an intervention. This relates to comment received from one of the expert clinicians interviewed: "If I lack information about a patient [e.g. age, medical background and deliberately excluded variables], I would probably be more conservative with my treatment".. Finally, we note a closer match to actions in the observational data very high disease severity (SOFA score 20), where negative rewards for not taking therapeutic action outweighs potential confounding bias. Beyond this analysis, further insights could be gained by comparing interpretable representations of the trained policies (Pace et al. 2022), but we leave this as further work.
29


Reviews:
[{"review_id": "i32Fx1PUjq", "reviewer": "Reviewer_3wVu", "summary": "* The authors address the unobserved confounding problem in offline reinforcement learning with pessimism over possible \"worlds\" (confounder values) compatible with the observation (distribution of trajectory).\n* They define a new type of uncertainty \"Delphic uncertainty\" as the variance of Q value over the compatible (thus unidentifiable) worlds with theoretical decomposition with other types of uncertainties.\n* Simulated evaluation and evaluation by experts clearly indicated that their method outperformed existing methods that do not address the Delphic uncertainty such as CQL and BC when strongly confounded.", "questions": "1. Intuitively, it seems that estimating $z$ from the trajectory and using it as $\\pi(a|s,z)$ as in POMDP methods would improve performance for later steps $t$, but is such an extension possible? Also, is the proposed method still superior when such a POMDP method is used as a baseline? I'm wondering if the online identification of the world is possible within an episode through such a formulation.\n1. The existing approaches for a similar setting are discussed (e.g. using partial identification) but not compared. Isn't it possible to compare them?\n1. Is the $\\max$ taken w.r.t. not only $z,z'$ but also $s,a$? If not, how $\\Gamma(s,a)$ is summarized for an environment?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* The unobserved confounding is a major issue in offline reinforcement learning.\n* They investigate a minimal problem setting (contextual MDP) to reproduce it and propose a simple and intuitive method that models the uncertainty related to the confounding.\n* The theory that decomposes the variance into several types of uncertainties motivates the approach well.\n* Empirical evidence including evaluation by experts clearly supports their claim.", "weaknesses": "1. Baselines and environments tested are relatively limited (see also Question 1 and 2).\n1. Not being a major concern, it would be more intuitively superior if an end-to-end formulation was possible, as in the CQL. The proposed method is divided into a step of learning multiple possible worlds and a step of pessimism using them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "weVMUVdZVb", "reviewer": "Reviewer_x1vz", "summary": "This paper studies the problem of offline learning an optimal independent policy. To achieve this goal, the authors propose the Delphic Offline RL algorithm that:\n1. identifies the compatible world model for confounded MDP and learn the world-dependent value function $Q_w^\\pi$;\n2. incorporates pessimistic policy optimization using the estimated delphic uncertainty;\nThe experimental results show that Delphic ORL method achieves better performance under large confounding strength.", "questions": "It is assumed that the offline policy is known. What if $\\pi_b$ is unavailable so that the importance sampling method has biased in estimating the value function? Is it possible to incorporate other unbiased estimation method in confounded POMDP setting like (Shi. et al, 2022)?\n\n\n### Reference:\n\nShi, Chengchun, et al. \"A minimax learning approach to off-policy evaluation in confounded partially observable markov decision processes.\" International Conference on Machine Learning. PMLR, 2022.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well motivated and the author proposed an interesting idea to incorporate the delphic uncertainty in pessimistic offline policy optimization. The writing is generally clear and easy to follow.", "weaknesses": "1. It seems that the estimator $\\mathbb{Var}_w (Q^π_w (s, a))$ is not an unbiased estimator for the delphic uncertainty as the other two forms of uncertainty can still enter the estimation (noting that $Q_w^\\pi$ is not the conditional expectation given by Theorem 4.2). I didn't see the author making effort to justify this point. \n\n2. Is it necessary to evaluate the delphic uncertainty on a state-action level rather than evaluating the same thing for the total reward of the whole trajectory under  policy $\\pi$? I'm not convinced of the soundness of the method here, as there might be some correlations between different state-action pairs in the Q function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NBlfr4LHx0", "reviewer": "Reviewer_qrjW", "summary": "The paper uses the contextual Markov Decision Process to model unobserved confounders in offline reinforcement learning. First, the authors define the class of contextual MDP that are compatible with the dataset. Then, based on a variance decomposition formula, the authors introduce the delphic uncertainty. Delphic uncertainty means the variance of policy performance across all compatible worlds. Then based on the delphic uncertainty term, the authors propose a penalty term for offline RL.", "questions": "1. On page 4, what does the notation $P_w \\mapsto \\Delta W$ mean? Do you mean $ P_w \\in \\Delta W$?\n\n2. Right after Def 4.1, the author proposes to model the Q function as a random element \"value model $Q_{\\theta w}$ is defined by some stochastic model\". Where is this randomness coming from? Isn't the value function of a policy just a deterministic function?\n\n3. Related to the previous remark, the meaning of Theorem 4.2 is unclear to me. Could the authors detail the probability setup for this theorem? From my understanding, there is a measure over the space of compatible worlds (by the notation $E_w$), and then there is a measure over Q-value functions (by the notation $E_{\\theta_w}$). What is the relationship between these measures?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The method is applied to two real/semi-real-world medical datasets, which is very nice.", "weaknesses": "The probability setup of the paper is a bit unclear to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N3esIAtYKJ", "reviewer": "Reviewer_Lkaz", "summary": "The paper addresses the problem of nonidentifiable confounding in RL. In this regard, a new notion of so-called delphic uncertainty is introduced in addition to aleatoric and epistemic uncertainties. An offline RL algorithm is proposed that penalizes taking actions with high delphic uncertainty. The performance is reported on both synthetic and real data.", "questions": "Please refer to the Weaknesses. I'm happy to update my scores after hearing your thoughts.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The presentation is excellent with adequate illustrations that helped my understanding.\n- The new notion of uncertainty is insightful and important in the real application of RL.\n- The algorithm and employed strategies sound reasonable to me.", "weaknesses": "I generally enjoyed reading this paper, but there are a few things that I wish were discussed in more depth or clarified:\n1. I'm a little confused about how should I decide whether a world is compatible or not. Apparently, I can start with any confounder space dimensionality, prior $p(z)$ and model architectures and then estimate the parameters using ELBO, and then I have a compatible world? How far I can go here or what should I see to say a world is not compatible here?\n2. The behavior policy is in general a context-aware policy. So, I'd expect enforcing similarity to the behavior policy might result in some sort of context awareness. Is this the case? For instance, in the illustrative bandit example, an optimal context-independent policy only explores $a_0$ and $a_2$, which in World 2 is very different from the behavioral policy and seems to be suboptimal compared to a uniform policy. So, if I get it right, the similarity to behavioral policy is encouraged in this setting with unobserved confounders.\n3. As a similar question to the previous question, don't we expect avoiding actions $(s,a)$ with high delphic similarity to result in a policy more similar to the behavioral policy? It seems to be the opposite: page 8 \"... we also studied the discrepancy of our trained policy with that in the data. Particularly, we compared the actions taken by our policy and the policy in the data and found that ... our policy was significantly different.\"\n4. Could you elaborate on how counterfactual $Q_w^\\pi$ is estimated using importance sampling in Section 5.1? Also, I'm not sure where in appendix C is referred to.\n5. I'm not sure how to think about a reasonable $\\Gamma$. Isn't Figure 7 concerning?\n\nRecommendations:\nI do not see the name of real data or the details explicitly mentioned in the main text. But I can see, as you have cited, Raghu et al. have pioneered using RL in the setting of sepsis treatment using a publicly available MIMIC dataset. If this is not your data, I recommend reporting performance on MIMIC for further reproducibility. Also, looking at the recent citations of Raghu et al., it seems CQL might not be the best baseline here. For instance, Shirali, Schubert, and Alaa have included medical guidelines as potential contexts for RL formulation, with better performance and higher similarity to behavioral policy. You may want to consider more recent works as a baseline or use their observations in favor of context awareness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)