When to Act and When to Ask: Policy Learning With Deferral Under Hidden Confounding
Authors: Marah Ghoummaid
Uri Shalit
Venue: NeurIPS 2024 poster (2024)
Keywords: policy learning
causal inference
sensitivity analysis
human-algorithm collaboration
Primary area: causal_inference
TL;DR: Learning a treatment recommendation policy under hidden confounding, with the option of deferring the decision to an expert

Abstract: We consider the task of learning how to act in collaboration with a human expert based on observational data. The task is motivated by high-stake scenarios such as healthcare and welfare where algorithmic action recommendations are made to a human expert, opening the option of deferring making a recommendation in cases where the human might act better on their own.
    This task is especially challenging when dealing with observational data, as using such data runs the risk of hidden confounders whose existence can lead to biased and harmful policies. However, unlike standard policy learning, the presence of a human expert can mitigate some of these risks. We build on the work of Mozannar and Sontag (2020) on consistent surrogate loss for learning with the option of deferral to an expert, where they solve a cost-sensitive supervised classification problem. Since we are solving a causal problem, where labels don’t exist, we use a causal model to learn costs which are robust to a bounded degree of hidden confounding.
    We prove that our approach can take advantage of the strengths of both the model and the expert to obtain a better policy than either. We demonstrate our results by conducting experiments on synthetic and semi-synthetic data and show the advantages of our method compared to baselines.

When to Act and When to Ask: Policy Learning With Deferral Under Hidden Confounding
Marah Ghoummaid Faculty of Data and Decision Sciences, Technion marahghoummaiddgmail. com
Uri Shalit
Faculty of Data and Decision Sciences, Technion urishalitStauex tau. ac.il
Abstract
We consider the task of learning how to act in collaboration with human expert based on observational data. The task is motivated by high-stake scenarios such as healtheare and welfare, where algorithmic action recommendations are made to human expert, opening the option of deferring recommendation in cases where the human might act better on their own. This task is especially challenging when dealing with observational data, as using such data runs the risk of hidden confounders whose existence can lead to biased and harmful policies. However, unlike standard policy learning, the presence of a human expert can mitigate some of these risks. We build the work of Mozannar and Sontag (2020) consistent surrogate loss for learning with the option of deferral to an expert, where they solve cost-sensitive supervised classification problem. Since we are solving a causal problem, where labels do not exist, we use a causal model to learn costs which are robust to bounded degree of hidden confounding. We prove that our approach can take advantage of the strengths of both the model and the expert to obtain better policy than either. We demonstrate results by conducting experiments on synthetic and semi-synthetic data and show the advantages our method compared to baselines.
Introduction
Machine learning models are increasingly being developed to perform tasks performed by human decision-makee in high-stakes settings such as clinical decision making [Adams et al. 2022 Ra jpurkar et al. 2022], criminal justice [Stevenson and Doleac [022] and social services [Behncke et al., 2009, McBrien et al., 2022]. Some of these tasks involve recommending actions such as medical treatment, releasing on bail, or receiving benefits. Typically the data used to train such oction-recommendation models is based on past human decisions and their outcomes. For example, we might observe how patients were treated for diabetes and their subsequent health outcomes.
Learning to act better in the future based on past observed actions is a causal problem. For such problems we face the fundamental problem of causal inference (Holland] 1986), i.c. the unknowability of counterfactual outcomes: "Would this patient have been better had they been treated differently?" Estimating causal quantities based on observational data such hospital records is risky, as there always the possibility of hidden confounding. Roughly speaking. this means there exist factors that affected the human decision maker and the outcome, but are unavailable to the model during training.
Learning models from data which has hidden confounding can lead to biased and harmful treatment assignment policies. However, human experts' decisions also be biased, sub-optimal, wrong. Thus, a setting in which both the human expert and the model can complement each other might be the best choice, mitigating the potential weaknesses and leveraging the strengths of both the model and the human expert [Bansal et al. 2020 Charusaae et al., 2022]. We believe this is especially pertinent in causal inference, as the human expert typically has access the hidden confounders.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this paper, we design a framework for learning a causal action recommendation model that can jointly work with a human decision-makes while using observational data where hidden confounders might exist. We assume that the effect of the hidden confounders is limited (in way we specify below), as without any assumption about the nature of hidden confounding, learning would be impossible. Our goal is to design a system consisting of a machine learning model estimating causal effects and a human expert working in a complementary setting. In addition to learning treatment assignment, the model learns the weaknesses and strengths of the human expert and will decide in each case whether to recommend a certain treatment to the patient, or whether it is better to defer the decision to the expert. The ultimate goal is to learn better policies and to reduce the burden human experts. We call our method Causal Action Recommendation with Expert Deferral (CARED).
CARED follows in the footsteps of Mozannar and Sontag (2020), who developed method for learning a classifier with the option of deferral to an expert. They propose a reduction of the problem to cost-sensitive learning problem, where costs are based the true labels of data samples. They give a surrogate loss for the cost-sensitive learning problem which generalizes the cross-entropy loss, and prove this loss is consistent, i.e., converges to the optimal solution of the original problem. Our problem can similarly be viewed as a classification problem with the option of deferring to an expert. However, the causal case is more difficult: the correct label for each sample is the best treatment to be prescribed to this sample based on its features, and due to the fundamental problem of causal inference we cannot know this label. Thus, unlike Mozannar and Sontag 20200 we do not have the true labels for our classification problem.
To overcome the challenges of the causal setting, we propose set of costs that instead of being based on the true labels, are based on estimated bounds on counterfactual outcomes. These costs guide our model towards learning what is the right treatment for each patient while acknowledging that the human expert has access to additional information the model cannot access. We prove a generalization bound on the loss of the joint machine-ent expert system, and further prove that under certain assumptions about the model used to estimate the bounds mentioned above, the joint machine- expert system outperforms both the human expert and a pure machine learning model. Finally, we evaluate CARED on synthetic and semi-sy semi-synthetic data to demonstrate how we can learn policies that outperform both pure machine learning policies and human experts. We further show that CARED outperforms a recently proposed method by Gao and Yin [2023] that addresses the same problem with an mnvers--rropensiy weighted approach.
2 Related work
Many works focus on solving the problem of policy learning for action recommendations from observational data. Some notable approaches include reweighting by inverse propensity weighting
(IPW) and other weighting techniques (Swaminathan and Joachims 2015 Kallus 2017 Beygelzimer
and Langford 2009]. and the approach of using doubly robust scores to determine the optimal
treatment assignment policy for binary treatments (Dudík et al. 2014 Athey and Wager 2021?
Kallus and Zhou 2020]. Other methods predict the Conditional Average Treatment Effect (CATE)
and use it the guideline for treatment assignment for each sample, such as Jesson et al. 2021]
and Kallus et al. 2019). Most of these works assume ignorability, i.e., that there are no hidden confounders that a affect both treatment assignment and the outcome in the data. As mentioned earlier, this assumption rarely holds when observational data is in use, and its presence, if not accounted for,
can lead to biased and harmful policies. Our work builds on previous work for learning supervised classification problems with the deferral option by Mozannar and Sontag 2020), and is inspired by Athey and Wager [ who derive costs for learning policies from observational data under the assumption of no hidden confounders.
Gao and Yin 2023 present a framework for collaborative human- All policy learning from obser- vational data with deferral, building on carlier work (Gao et al. [021] which did not allows for hidden confounding. To the best of our knowledge, theirs is the only existing method that learns a policy with the deferral option under hidden confounding. Their method minimizes an inverse propensity weighted estimator of the worst-case risk over a class of differentiable policies, and over an uncertainty set around the observed propensities. The uncertainty set is determined by constraints motivated by the Marginal Sensitivity Model TTnn 2006). While our method employs both out- come models and propensity scores. Gao and Yin 2023 's approach focuses on propensity score re- re-weighting. The re- weighted objective implies that only cases where the proposed policy agrees to a high degree with the observed policy are taken into account. As we show in the experimental section

---

below, this (along with the lack of outcome model) might lead to under peleerrreffommccc. especially in the (common) case where the constraint set is not accurately known. [ao and Yin [2023] also explore the case where there are multiple specific human experts the model is optimizing for, which we plan to explore in future work. We also compare our method to Kallus and Zhou [202)]. who take a similar approach as Gao and Yin (2023) for learning a policy while allowing the violation of the unconfoundedness assumption. However, they do not have the option of deferral.
In other related work, [Stensrud et al. [2024] consider the case where the expert's action can be used as input the method, motivated by the fact that the expert typically has access to unobserved confounders. This is distinct from our use case but has interesting implications in identifying the so-called "superoptimal regime" where the expert's is action can strictly improve over a policy derived purely from the observables. Finally, Yin et al. [2024) offer novel approach towards learning to defer in the non-causal setting, which could be adapted to our use case in the future.
3 Setup
We work under an observational data setting with the Seyman-Rubin potential outcomes framework [Rubin] 2005). Let (X, A, (1), (O), U) be a sample drawn from the unobservable distribution Piul.. where A E = (0. } is binary treatment, X E X C Rdd is set of baseline covariates, (1) and (0) are the real-valued treated and untreated potential outcomes, respectively, and U IR is an unobserved confounder. We face the fundamental problem of causal inference and only observe n draws from the coarsened distribution P over the observed variables Z (X, Y), where we assume that Y (A),, i.e. (causal) consistency. We generally follow the convention that higher outcomes are better, we mention when assumed differently. We use the Marginal sensitivity Model (MSM) Tan] 2006) as a way to model a limited degree of unobserved confounding. We are interested in learning policy with the option of deferral, such that given the patient's 's covariates it either assigns a treatment or defers the decision to an expert.
Let e(x) P(A= X = xx) and e(x, u) (A - U--U-U =) u) be the observed and full propensity scores, under and Prun. the observed and the full unobserved distributions, respectively.
Assumption 1 (MSM Assumption). We assume e(x),, u) E (0. 1) and that the ratio between the full odds of treatment e(x, w)/(1 - e(x,, u)) and the observed odds of treatment e(x)/(1 e(x)) is bounded by at most factor of A 2 1 almost surely under Pran:
e(x, u)
e(x) 1- e-e(z) SA
A <
e(z,, u)
Note that When A = Assumption 1 equivalent to the classic assumption of unconfoundedness with respect to the observed X. As A increases, the MSM allows for greater levels of unobserved confounding. Setting is matter of ongoing research, and is typically done by calibrating with respect to observed confounders [MCClann et al. 20224]. In our case, it is also tied to the fraction of deferrals to the human expert and can be tuned to achieve a desired deferral rate.
For a treatment a E {0, 1} and a covariate vector x E X' we define the Conditional Average Potential Outcome (CAPO) as E (a) X z). The Conditional Average Treatment Effect (CATE) is the difference r(x) E (1) - Y()) = æ]. Let M(AA be the set of distributions Prull that agree with the observed P on (X, A, Y) and that also agree with Assumption Then y+ a) = M(A) E (a) - xr] is the so-called sharp upper bound on the CAPO E (a)) æs. and
similarly for Y- (zx a), taking infimumm instead. We further let Ý (z. y (z, a) be estimated upper and lower bound for the corresponding CAPO functions.
A policy with deferral is a function क X - f0, 1} that maps covariates to possible action, or defers the decision to an expert, where 1 denotes deferral. A policy æ can be assessed by its where higher policy values are related to better policies unless stated
policy value V(T) E otherwise. The action when the algorithm chooses to defer is the action the human expert has taken, see discussion of this in the limitations subsection (8) The learned policies will be assessed relative
to baseline policy value, such as the policy value based solely on the human expert. Our goal is for CARED to learn policy that does worse than the baseline policy and hopefully outperforms it.
3

---

4 CAPO-Based Policies
We now present two baseline policies that can be defined based on the CAPO bounds.
The first is Bounds Policy Equation ( This policy assigns treatment if the upper and lower bounds on the CATE have the same sign, and otherwise it defers, i.e. it defers if the CATE interval crosses This approach was used in previous work, e.g. [Jesson et al. 2021 Oprescu et al. 2023]. The second the Pessimistic Policy Equation 2 is pessimistic the sense that it does not trust the human expert, so it does not allow deferral to an expert. It agrees with the Bounds Policy in the cases where that policy would not defer, and in the cases where the Bounds Policy would defer it decides based on the lower bounds, which (assuming higher outcomes are better) are indeed pessimistic.
Let = (z, 0), (x, 0), (x, 1)), Y (æ, 1)) be the CAPO bounds estimates for a sample (X =x=2,4 -
a. y). We further assume that the expert policy is reflected in the training data in the sense that their policy is Texp (2²) = ag- Then, given model that supplies CAPO bounds Q, we define the following policies, which we consider as baselines henceforth:
Bounds Policy:
1
if 1 >
bounds (x)
0
if -1,1))--((,0) <0 otherwise.
(I)
Pessimistic Policy:
1
if (z, -
0 if (z, (2, (z.0) <0
y
pessimistic (x)
(2)
1 otherwise, Y (x, (2, >0
0 otherwise
The above policies, while accounting for hidden confounding, even when allowing deferral to an expert, do not learn the strengths and weaknesses of the expert, so they might not be the optimal policies. We show this theoretically in Section 6.2 and empirically in Section 7
5
Method
In this section, we present our method in which we design a machine- expert system for learning policy with the option of deferral an expert under hidden confounding. In [5.]] we describe the joint machine-expert objective we are interested in optimizing, the challenges imposes, and how to optimize a consistent surrogate cost- sensitive loss that converges to the optimal solution of the original objective, based on Mozannar and Sontag [2020]. Then in 5.2 we introduce our proposed set of costs based on the CAPO bounds for solving this cost-sensitive problem. Lastly, we explain the step of CAPO estimation in 5.3 Our algorithm is summarized below in Algorithm
5.1
Joint machine-expert objective function
We design a joint machine-expert system, where we aim to learn a policy т : X' + (U We denote by m A the expert's action that is assumed be drawn from the distribution M x, U = u. Note that the expert might have access to additional information - the hidden confounder
U. which is unavailable the model. We cast our problem cost-sensitive optimization objective
L(TT) = Px.yg-PXXYY).mmma u [C(,, ग (I)) + C1 (z, m, v) 1.
(3)
where C'x,, a) is the cost incurred by the system if for a sample with covariates x, an action/treatmen was chosen by the model, and C'_ (z, m, 8) the cost incurred when action m was chosen for sample æ by the expert. We explain below why C²1 depends on y but C' does not.
In the case of no unobserved confounding and no deferral, (Ahhe and Wager (2021 have shown how the costs above can be set such that the cost minimizer is the policy with optimal policy value. We describe how set costs that address both hidden confounding and deferral in the next subsection.
The above objective is non-convex and difficult to optimize. We deal with this challenge by building on the approach of Mozannar and Sontag (0200 for learning classifiers with the option of deferral to

---

an expert, where they have similar objective as in Equation (3) surrogate loss for the cost-sensitive learning problem, which is a weighted cross-entropy loss. where the weights are based on set of costs they build using the true labels for the classification problem.
They give a convex and consistent
As explained earlier, our problem can also be viewed as classification problem with the option of deferring to an expert. However, due to the fundamental problem of causal inference (Holland 1986] we do not have the true labels, which are in our case, the best treatment to be prescribed to this sample based on its features, which is challenge that we deal with in Section 5.2
Following is the surrogate loss we optimize, in Section 5.2 we present our proposed set of costs, and in Section 6 we give consistency and generalization guarantees for this surrogate loss.
Let mi X + be the raw output of the policy æ corresponding to a class i E (0,1, 1, and define xαx) = arg (0. (z()). Let 2 = (x, a, y) be a sample, and Q(T) =
(z, 0), 8- (z, 0), y 1)), Y (z, 1)) the CAPO bounds. These bounds are then used to define the scores c(O) = C(r,0), c(1) - C(z, 1), and c(L) = C1 (I, m, m,y) as we show in Section 5.2
below. Deñine w/ (z, ((x)) = max c(k) c(j). Then the surrogate loss function for Equation (3) given by:
(cE(T,Z;())
E 5E(U.1.1)
-w (z, log
exp(kx(x)) Eke
S exp(TT<(x))
(4)
The method we propose uses the above system loss for learning a policy from observational data with the presence of limited degree of hidden confounding and the ability to defer. The difficulty in this case is constructing the costs Czz, 0), C(z, 1) and C' (I, m, y) which are used to derive the weights w/, since ground truth labels are never available - a challenge we address now.
5.2 Action Costs
As mentioned earlier, our classification problem does not have the true labels, as opposed to Mozannar and Sontag 2020). We therefore propose set of costs based estimated bounds counterfactual outcomes rather than on the true labels, guiding our model in learning for which cases it can safely recommend actions, and which to defer to the human expert. These costs are then used to derive the weights in the objective Equation
For i E [n] let (æi, Vi, ) be the i-th observed sample, and let ( a), (xx, a) be the estimates of the upper and lower bounds of the CAPO for an action A, and covariates x E X. We propose setting the cost of assigning the action a to the i-th sample to be:
Clr,, a) Y + (x,,, 1 a) Y (1x
This cost encourages the model to choose the action with the highest outcome. When this is not the case, i.e. the model chooses another action than the one with the highest estimated outcome, then it will incur a cost which the difference between the lower bound of the outcome corresponding the action chosen, and the upper bound of the outcome corresponding to the action which was supposed be chosen based on the outcomes. This cost can be thought of the worst-case regret of the model relative to the optimal action. It is important to note that this is only one option among many options for building a set of costs for this purpose. For instance, we can switch the roles of the upper and lower bounds of the CAPOs to obtain various costs expressing differing levels of risk-aversion.
As for the cost of deferring to an expert, following the same logic, we propose two alternatives. Given covariates z;, expert action a; and outcome y: the conservative deferral cost is:
C (Ii, ai y:) = Y (a;, a;) y/::
and similarly the optimistic deferral cost is: C1 (I;, ag y4) = Y + (a;,, 1 - a;) ys. These costs represent to what extent are we willing to take an action different from the one the expert took which resulted in the outcome actually seen in the data, i.e. m a, which lead to outcome ys- In the optimistic case, deferring to an expert results in incurring a greater cost in comparison to its conservative counterpart, encouraging the model to make fewer deferrals and focusing on samples is highly uncertain about. It is optimistic in the sense of assuming the model's 's estimates are likely
5

---

Choosing between these alternatives should to be done based on the characteristics of the specific policy and use case, In Appendix A.3 we show comparative analysis of the two alternatives.
correct.
To summarize present here the costs corresponding to the conservative approach:
Cl 1) (x, 0) Y (x;
Clz 0) (x 1) y (I,,
Y (2,, 0) yt if a, =1
Ci ai, Ut)
Y (2i, 1) yi+ otherwise.
Taken together, these costs encourage the model to classify a sample as = 1 or a 0 in cases where it certain that particular action would be best, and to defer cases where the expert seems to have made the correct decision (in the conservative case). An illustrative example of the costs choice can be found in Appendix A Additionally, we demonstrate costs' coherence in Theorem) |
5.3 CAPO Estimation
From an algorithmic point of view, any method that yields upper and lower bounds on the CAPO given some degree of hidden confounding can used to obtain the costs Section 5.2 As we will see in the next section, our theoretical results require bounds with certain generalization and validity properties to hold. In this work, we use the B-learner [Opreseu et al. 2023 for estimating the upper and lower bounds of the CAPO, as it has the properties needed for the theoretical analysis and shows good performance in practice, and being a meta-learner can accommodate various base learners, including random forests and neural networks.
Algorithm 1 CARED Policy Learner- The Conservative Approach
input Data {T; as yt) : í {1...., n
I: Use data [(r;, 0z, 16) : (1...., n to learn the CAPO bounds:
((x:) = (Y+ (z,, 0), Y (z,, 0). y (z,, 1)), Y (Iz, 1))
//e.g. using B-learner (Oprescu et al.) J 2023]
2: Construct the conservative costs based on the CAPO bounds:
(1) = C(z;, 1) = p- 0) - y (x;;,;11 1)
(0) C(z,, 0) - (x 1) - y (2,, 8 /-
(m,, 0) Vt-
if az =1
a ((L) = (x1, ai 9:)
8 (2 1) Vé: otherwise
3: Calculate weights w/ (0x1 (k) - (i(j)
4: Learn policy: arg mnn Σi=1 Ese(o...) -we log output π
exp( (x:))
Exe(o...) (x,))
6 Theoretical Guarantees
We present theoretical guarantees for CARED as follows: In In we show that the optimum of the surrogate loss function LOE (eq. 4 agrees with the optimum of the machine-expert loss function L (eq. (]. Then, in 6.2 we show that the costs we use LeE are coherent, in the sense that minimizing them leads to decision that is non-inferior to the decision either the expert or the machine would have made on their own. Finally, in 6.3 we give a generalization bound for the loss LCE-
6.1 Consistency
Corollary 1. LCE is convex in and is consistent loss function for L: Let ă arg inf, E[LCE((T Q)), then î arg inf, L(x).
This result is straightforward adaptation of Proposition 1 of Mozannar and Sontag 2020 their result not sensitive to the particular choice of costs (B.0.T). motivates using surrogate loss function LCE which is much more amenable to optimization than the original machine-expert loss L.

---

6.2 Costs Are Coherent
We now show that the costs C(,, 0), C(a, 1), Ca a, v) are coherent, the sense that they indeed work as intended: a policy that minimizes them locally is always at least as good as (and indeed, in general better than) both the expert on their own and the machine on its own.
Definition 1. [Bound Validity] For a sample with corresponding potential outcomes ((0) and (1),, let (x) (Y++ (zz 0), Y (x, 0), (a, 1)), Y 1)) be the estimated CAPO bounds. Then Q is valid for (Y), (1)) if ((a) E ( (x, Y+ (2r, a)] for a E {0,
Validity means that bounds indeed contain their respective potential outcomes. For example, Oprescu et al. 2023) prove that their bounds are valid on average.
Theorem 1 (Costs are coherent). Let ((xe)) E arg min LOE(T, 24; (ax))) Tep the expert's 's policy, and
Q Tbounds(a)) the CAPO-based policy defined in I If (zz,) is valid for ((()),, then
((f(x,)) 2 max {y (Teep(zz)), Y ( (ax)m
Furthermore, under certain technical conditions on the distribution of Y(0), and Q, the inequality is strong with non-zero probability for each sample.
Theorem shows that whenever the bounds include the true potential outcome, the action that Q minimizes our proposed loss function is at least as good as the action implied by the baseline π bounds policy, as well as the human expert policy.
6.3 Generalization Bound
We now show the generalization bound for our machine-expert system loss. We start with assumptions, then we state the main theorem.
Assumption 2 (Policy learners). Let IT be the class of policies over which we optimize Equation (4) We assume the class IT is with restricted complexity, specifically (IT) = O ( where (,,((T is the Rademacher Complexity of the policy class II. Classes that have this property include linear functions, logistic functions, decision trees with a bounded depth, and neural networks with weight decay or dropout Kallus and Zhou 2020].
Assumption 3 (Policy Learner with Bounded Outputs). Let II and NT : + R be the raw output of the classifier π corresponding to a class/action j E {0, 1, 1.. We assume there exists a constant C₂ such that | S C2 for all j-
Assumption (Boundedness of the Outcomes). Y is bounded, i.e. |Y| S Cy for Cr > >0.
Assumption 5. [Rates for ERM CAPO Bounds Estimators] The CAPO bounds estimators (0, a) for a E {0, 1} and o {+, } satisfy: [| a) - y a)ll E On -1/(2r)) with < r < 2.
Assumption 5 implies that the convergence rate of the L2 norm of the estimation error of the CAPO bounds converges not too slowly (in probability). Nottaly,,, the B-Learner bound estimates satisfy Assumption Junder some assumption on the class of policy leamnrrs and the nuisance estimators, according to Corollary from Oprescu et al. 2023)..
We now give generalization bound on the loss of the joint machine-expert system:
Theorem 2 (Generalization Bound), Given policy class 11 satisfying Assumption 2 Assumption 3 with a constant C+ Assumption f with a constant Cy, and Assumpttonn 0 r 2.
Let ι II. be a policy, and let Q Q be the CAPO bounds and the estimated CAPO bounds respectively. Then there exists a constant C' >> 0 such that with probability of at least 1 - 8 the following holds:
(om))
2l1(4/8) + C. (n-V(ett)) (5)
LD(T; Q) - Ls Q) < (1) + 24.
24-C.. Cr
T
-. =-= LCE(T, Zúi (z+)) is the training loss, and LD(T; Q) = E Exmp(z) (LOE(T, z; Qx))) is the expected loss for the combined machine-expert system loss in and Rn, the Rademacher Complexity of the policy class II.
where Ls(T;; Q2) = -
7

---

7 Experiments
Here we examine the utility of CARED by conducting experiments on synthetic and semi-synthetic data in Section [.. and Section [22] respectively. Further details about the experiments, datasets, models, and nyper-parameters can be found in Appendix C
Our main comparison is to Gao and Yin 2023 's method (ConfHAT that allows for bounded degree of confounding in addition to allowing the option of deferral to a human expert. They learn policy by optimizing minimax reweighting-baser risk estimate over uncertainty set around the observed propensities. The uncertainty sets are determined by the MSM assumption, which is the same assumption we employ to bound the degree of hidden confounding. Another bascline we consider is Kallus and Zhou [2020]'s 's method (CRLogit) which proceeds under a similar approach and allows for a bounded degree of confounding, but does not allow deferral to human. Additionally, we evaluate all methods against the following baselines: Oracle Policy which is the best policy that assigns for each patient the true best treatment. In both experiments, this policy is available to us, as we have the true potential outcomes. Current Expert/ Baseline Policy this is the default policy that we refer to in cases of deferral. Typically, is the policy of the current expert, but can be any other policy of our choice. Pessimistic Policy: Using the Equation 2 policy with the
bounds Q given by the B-learner Oprescu et al. [2023]. B-Learner Policy: Using the " bounds policy of Equation (1) with the bounds Q given by the B-learner (Oprescu et al. 2023). We use the exact same Q in Algorithm [ and in the B-learner and pessimistic policies. Random Deferral Policy: this is a variant of the B-Learner policy which introduces a different approach to deferral, where samples are deferred randomly based specified deferral rate.
7.1 Synthetic data
In this experiment, we replicate the study from Gao and Yin 2023 using synthetic data, comparing our method to ConfHAl and CRLogit. Since this experiment is measuting regret, we follow Gao (et al. [2021] and compare
Baseline Policy that assigns treatment a = 0 for all patients and is denoted by To. In this experiment lower outcomes are better.
Data The data is generated according to the following data generation process:
ş ~ (0rm.. X ~ ~N((22 1)) U = (1) < <(O),,
Y(A) = 3J r + I[A = 1] + 0.5 = 1| + + we +
where 90 = [0. 0.5, -0.5, 0, Btreat = [-1.5, 1, 1.5, 1, 0.5,, Ps =
2.5, x) with 00,0.75, -0.5, 0, 1, Oj. The confounder is denoted by U, and the true propensity score is given by: e(X,U) =
a = -2, w. = 1.5, and ~ (0, 1). The nominal propensity is logistic by X, xxe(X) =
with the true Ao. such that log(Ao) = 2.5.
Experiment We replicate the experiment from Gao and Yin 2023 and run 10 trials, with different instances of the above data, a train data size of 2000, and test data of size 10000. For each trial, we vary the sensitivity parameter A in (0.01, 0.5, 1, 1.5, 2, 2.5, 3.5, 4}, corresponding to various levels of assumed hidden confounding. We compare the policy regref for the returned policy for each method relative to the Baseline Policy. In addition, we report the policy regret for the human expert in the dataset (Human's Policy) which is reflected by the variable A in the dataset. As for the CARED policy, we obtain it by applying Algorithm D with a logistic policy implemented as a single-layer MLP network.
Results In Figure I we see that CARED outperforms all other methods, specifically improving over the expert for all A values. In contrast, the ConfHAI and CRLogit improve over the expert for a limited range of values of A. which is for range around the true value of the sensitivity parameter Ao showed the plot, and perform noticeably worse than the expert for many A values when the is mis-specified. In contrast, CARED shows robustness to all A values, making it a safer choice, as correctly specifying A is challenging [McClean al. 2024..
7.2 IHDP Hidden Confounding
In this experiment, we aim to demonstrate how our method can adapt to the human expert using a semi-synthetic dataset: the IHDP hidden confounding datasct.
8

---

130
125
1.0
Lookah Paley
Pulicy Ragrss
-3.25
Possenitic
2.50
Ouade Maliy
-1.00
0.5
1.2 2.5 2.0
2.5
8.0
kighAl uncertainty paramater
Figure Synthetic Data: Policy regret, lower policy regret better. x-axis is levels of hidden confounding according to the MSM model. The true Ag is reported as a black vertical line. Human's 's Policy is the human expert's choices (A) as observed in the data. CRLogit Policy [Kallus and Zhou 2020] learn a policy with an IPW approach under hidden confounding, without deferral. ConfHAl Policy [ao and Yin (023) similarly learns a policy with IPW approach under hidden confounding with deferral. CARED: our proposed method Pessimistic Policy and B-Learner Policy are based on CAPO bounds from the B-Learner [Oprescu et al.) 2023] and are defined in Equation and
Equation respectively. Oracle Policy assigns the best true treatment each patient.
Dataset The condun-confounndigg version the IHDP dataset (Hill 2011 was introduced by Jesson et al. 2021 The Infant Health and Development Program (IHDP) dataset [Hill 2011] is dataset consisting of real covariates and a treatment that were collected from an RCT that targeted low-birth- weight, premature infants. The treatment was providing both intensive high-quality child care and home visits from a trained provider, while the outcomes were simulated according to the response surface described by Hill (011). Jesson al. [202T] induced hidden confounding onto the dataset by hiding the vi covariate; however, the response surface B from Hill 2011) is still used to generate the observed outcomes. In this dataset higher outcomes are better, thus build the loss function using the appropriate set of costs for this assumption.
Semi-Synthetic Expert (Current Expert) We design semi-synthetic expert, based on the original observed expert's policy, and the Oracle policy. The new semi-synthetic expert is designed as follows: when the feature 217 "wooked during pregnancy" receives the value 1, then the new expert is identical to the Oracle policy, and otherwise, it is identical to the original expert. This way, the new expert is perfect when T17, which happens in probability 0.59. See details in Appendix C.2.1
Experiment We conduct this experiment on the modified IHDP Hidden Confounding consisting of tuples (X, A', Y"), where the A' is the new expert, and the Y' is the outcome corresponding to the expert A. We generate 1000 realizations of the dataset. For each realization, we train the policy model for different values of the causal uncertainty parameter A and calculate: the rate of the samples deferred to the expert, and the policy value of the learned policy. We then plot the average policy value per causal uncertainty level the parameter A in Figure 2a and the average policy value per average deferral rate over all trials in Figure 2b As for our method, we obtain our policy by applying Algorithm with logistic policy implemented as single-layer MLP network.
Results In Figure 2a We observe that CARED consistently improves upon the expert's policy, yielding a higher average policy value across all levels of the causal uncertainty parameter, A. Notably, it outperforms the baseline methods for reasonable values of A, and achieves policy value closest to that of the optimal Oracle Policy. As in the previous experiment, CARED demonstrates greater robustness to variations in the assumed level of confounding. For high values, CARED's performance trends toward that of the expert policy, which is expected, as larger A values result in less informative and thus less reliable CAPO intervals. In contrast, ConfHAl only begins to improve when the assumed confounding level closely matches the true sensitivity parameter.
These results highlight the robustness and reliability of our policy, showing that performs well even when the sensitivity parameter is misspecified by the data analyst. Moreover, when the sensitivity parameter reaches very high levels, CARED avoids unnecessary risk by converging towards the expert policy, reflecting its conservative response to high causal uncertainty.

---

In Figure 2b we compare our method only to baselines that incorporate deferral. Here, instead of regret, we assess the policies by their average policy value across all trials, and we plot the performance according to the average deferral rate over all trials. Our results show that our policy consistently outperforms the other methods at equivalent deferral rates.
The peak of the plot of the CARED Policy is when both the model and the expert work in collaboration and take advantage of the strengths of both the ML model and the expert. This happens at a deferral rate that is close to ~ 0.6, which is almost precisely the percentage of the cases where the expert does as well as the oracle policy according to how the expert was designed. This indicates that our model learns the strength of the expert, and knows when to defer to the expert. While ConfHAI defers only a small fraction of samples on average, which might indicate that this method does not learn the weaknesses and strengths of the expert.
142
L555
Shapp Fally
Policy 154
SAIED
Valles
Pay lualyy
Cale heley
Oracle hrlls
Rakdon Dals Polcy
341
135
0.0
lgght unceriariyy pai cincier
Deferral
(a) Policy Value for different values of A
(b) Policy Value for different levels of deferral
Figure 2: IHDP Hidden Confounding: Figure 2a shows policy value for different levels of allowed hidden confounding in the data according the MSM model Assumption [T] The x-axis represents different values of the uncertainty parameter A, and the true An is reported as a black vertical line. Figure (egureeb/showw shoshows policy value for different rates of deferral. The x-axis represents different levels of practitioner caution by varying the percentage of recommendations deferred. The methods shown here the same in Figure I in addition Random Deferral Policy that defers randomly chosen fraction of samples to the expert at each deferral rate.
8 Discussion
In this paper we proposed CARED: a method for learning policies from observational data where the model can recommend a treatment or defer to an expert. When learning to act from observational data which includes experts' actions, hidden confounders are by necessity factors that influenced the experts' decisions, and are thus available to them even though they are unavailable to the model. This makes our setting pertinent to the problem of safely learning to recommend actions based on observational data where the actions were taken by human experts, as the case in many medical and legal settings, for example. CARED thus mitigates some of the risk of learning causal models from observational data. A further advantage is that we might not need to know the true A for the system to be useful: instead we might wish to calibrate the rate of deferral instead, as that might be the more practical constraint the system faces, in terms of human labor vs. the joint system's policy value.
We showed both theoretically and by experiments in synthetic and semi-synthetic data that our method outperforms relevant baselines for this task, and can combine synergistic manner the expert's 's and machine learning model's capabilities.
Limitations and future work Our current method requires access to the experts actions in the observational data; it cannot accommodate directly a different expert, since we cannot know the potential outcomes corresponding to how that new expert would have acted, further limitation is the assumption that at test time the experts would behave the same for the deferred cases as they would have before system deployment. Realistically, deploying an action recommendation system might change the experts' behavior more broadly. Accounting for this would require testing and modeling the experts behavior such conditions. Future work will explore this more dynamic setting, taking into account the ongoing interactions and learning between the human expert and the system.
10

---

Acknowledgments
We would like to express our gratitude to colleague, Rom Gutman, for his valuable insights and support throughout this work. We also thank Angela Zhou and Ruijiang Gao whose methods we replicated and compared in our evaluations, for their their helpful responses to our questions about implementing their methods. Additionally, we extend our appreciation to Hussein Mozannar for his generous assistance in clarifying aspects of his work, which served the basis for our approach. We would like to thank the anonymous reviewers for useful discussion and feedback. MG and US were supported by ISF grant 2456/23.
References
R. Adams, K. E. Henry, A. Sridharan, H. Solcimani, A. Zhan, N. Rawat, L. Johnson, D. N. Hager, S. E. Cosgrove, A. Markowski, et al. Prospective, multi-site study of patient outcomes after implementation of the trews machine learning-based early warning system for sepsis. Nature medicine, 28(7):1455- 1460, 2022.
Athey and S. Wager. Policy learning with observational data. Econometrica, 89(1):133-161, 2021.
G. Bansal, B. Nushi, E. Kamar, E. Horvitz, and D. S. Weld. Optimizing Al for teamwork. arXiv preprint arXiv:2004. 13102, 2020.
S. Behncke, M. Frölich, and M. Lechner. Targeting labour market programmes results from a randomized experiment. Swiss Journal of Economics and Statistics, 145(3):221-266 2009.
A. Beygelzimer and J. Langford. The offset tree for learning with partial labels. In Proceedings of the 15th ACM SIGKDD international conference Knowledge discovery and data mining, pages 129-138, 2009.
J. Brooks-Gume F-r. Liaw, and P. K. Klebanov. Effects of carly intervention on cognitive function of low birth weight preterm infants. The Journal of pediatrics, 1 120(3):350-359 1992.
M.-A. Charusaie, H. Mozannar, D. Sontag, and S. Samadi. Sample efficient learning of predictors that complement humans, In International Conference on Machine Learning, pages 2972- 270030 PMLR, 2022.
A. Curth, D. Svensson, J. Weatherall, and van der Schaar. Really doing great at estimating cate? a critical look at ml benchmarking practices in treatment effect estimation. In Thirty-fftth conference on nearal information processing systems datasets and benchmarks track (round 2), 2021.
M. Dudik, Erhan, J. Langford, and L.. Li. Doubly robust policy evaluation and optimization. 2014.
R. Gao and M. Yin. Confounding robust policy improvement with human-ai teams. arXiv:2310.08822 2023.
arXiv preprint
R. Gao, M. Saar-T Tsechanscy,,, M. De-Arteaga, L. Han, M. K. Lee, and M. Lease. Human-ai collaboration with bandit feedback. In Z. Zhou, editor, Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI-21, pages 1722 1728. International Joint Conferences Artificial Intelligence Organization, 8 2021.
J. L. Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphicul Statisticy, 20(1):217-240, 2011.
P. W. Holland. Statistics and causal inference. Journal of the American statistical Association, 8 945-996, 1986.
A. Jesson, S. Mindermann, Y. Gal, and U. Shalit. Quantifying ignorance in individual-lever causal- effeect estimates under hidden confounding. In International Conference Machine Learning, pages 4829 4838. PMIR, 2021.
N. Kallus. Recursive partitioning for personalization using observational data. In International conference on machine learning, pages 1789-177 1798. PMLR, 2017.
N. Kallus and A. Zhou. Minimax-optiman policy lcarning under unobserved confounding. Management Science 2775):2870 2020.
N. Kallus, X. Mao, and A. Zhou. Interval estimation of individual causal effects under unobserved confounding, In Proceedings of the 22nd International Conference on Aritificial Intelligence and Statistics (AISTATS) 2019, volume 89, 2019.
T. McBrien, B. Winters, E. Zhou, and V. Eubanks. Screened & scored in the District of Columbia, November 2022. URL https://epic. org wttg//creeneddseee -asrreendd--ooree-eeeeoee in-dc/ hhttps://ppc..o org/ screened
soreened-scored-ooo scored- ic-dc/ posted November-2022].

---

A. McClean, Z. Branson, and E. H. Kennedy, Calibrated sensitivity models, arXiv preprint arXiv:2405.08738 2024.
H. Mozannar and D. Sontag. Consistent estimators for learning to defer to an expert. In International Conference Machine Learning, pages 7076-7087. PMLR, 2020.
M. Oprescu, J. Dorn, M. Ghoummaid, A. Jesson, N. Kallus, and U. Shalit. B-learner: Quasi-oracle bounds on heterogeneous causal effects under hidden confounding. In Proceedings of the 40th International Conference on Machine Learning, pages 26599 26618. PMLR, 2023.
Rajpurkar, E. Chen, O. Banerjee, and E. J. Topol. Al in health and medicine. Nature medicine, 28(1):31-38. 2022.
D. B. Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100X469):222 331. 2005.
Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory algorithms. Cambridge university press, 2014.
M. J. Stensrud, J. Laurendeau, and A. T.. Sarvet. Optimal regimes for algorithm-assiste human decision-making. Biometrika, page asae016, 2024.
M. T. Stevenson and L. Doleac. Algorithmic risk assessment in the hands of humans. Available SSRN 3489440, 2022.
A. Swaminathan and T. Joachims, Counterfactual risk minimization: Learning from logged bandit feedback. In International Conference on Machine Learning, pages 814-823. PMLR, 2015.
Z. Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 101(476):61191666 2006.
Yin, J.FF. Ton, R. Guo, Y. Yao, M. Liu, and Y. Liu. Fair classifiers that abstain without harm. In The Twelfth International Conference on Learning Representations, 2024. httt:/tttp///peeeeeeettt net/forum? id=jvveCAb&Vz
12

---

A Action Costs
In this section, present examples that illustrate our choice of the Then prove the consistency of costs some base cases, that is, we show that the lowest cost corresponds the right action for our choice for the set of costs.
Let v) be sample, and assume Y()) > (() w.l.og. that is, the right treatment is A 1. y*(x, (x, 0), Y (I, 0). y (z, 1), y (I, 1) are the CAPOs corresponding to this sample.
We recall our proposed costs:
C(x, 1) (z, (x,
C(x, 0)
(2 (x,
(x 0) - yi, if o =
CI (z, a.
8 (x1,1) - ye: otherwise.
(x, 0) - My of (+ 1x - otherwise.
CP a,,,,))
where C1 (zz, v), and CI (z, v) correspond to the conservative and optimistic deferral costs respectively.
A.1 No overlap between CAPOs intervals
Example In this example, show for this example in Figure 3a where we let Y (xz 0) = 4, Y (x, 0) = 1, (z, = 9, Y (z, 1) = 5, and (0) 2, (1) 6. Then we have that:
where the CAPOs intervals don't 't overlap, we provide visual diagram
C(æ, 1) = -1
C(.,0) 0) = 8
As for the deferral cost, there possible cases: if the expert is right, meaning 1, Y()), the deferral costs are:
CT a, = - C (x, = -2
for both alternatives, have the lowest costs corresponding to the deferral cost, which means that will guide the model to choose the expert's decision, which this case, the right treatment.
On other hand, when expert is wrong, i.e. = =0, Y (0). the deferral costs are:
CL (x,, v, = 3
CT (a,, a, v) =
Where for both alternatives, the lowest cost is the cost of the treatment which is the right treatment.
Proof of the general case show visualization the general case where the CAPOs intervals overlap Figure 36 In the general case we have that:
C(a,1) = -d
C(x, 0) = la d
As for the deferral cost, when expert is right, meaning = 1, = (l). the deferral costs are:
Ocens a) = (lo (lo +
(2,0), = (d
for both alternatives, we have the lowest costs corresponding to the deferral cost, which means that we will guide the model to choose the expert's decision, which is in this case, the right treatment.
13

---

On the other hand, when the expert is wrong, ie. 0, (0). the deferral costs are:
C1 ax, = 6 + C (r, = + d 4
Where for both alternatives, the lowest cost is the cost of the treatment which is the right treatment.
T1,11-5
M (x,3)-
6
T0 =1
(a) Example
yt (6.4)
Y(1)
(7
(*,
6
(b) The general case
Figure 3: No overlap between CAPOs intervals visualization
A.2 CAPOs intervals overlap
We now focus on the cases where the CAPOs intervals overlap. In Figure 4 we present two examples for cases where the CAPOs intervals overlap and show how this affects the costs.
Example
example presented in Figure where
Y (
(zx,) 0) 5, 8 (2, 1, 1,,8+(x,1
y (z, 1) and Y()) Y()) Then we have that:
C(r, 1) =
C(s, 0) =
As deferral cost, there are two possible cases: the expert right, meaning 1, Y(1), deferral costs are:
CI, (x, - CT (2,0,)) = -1
for both alternatives, we have the lowest costs corresponding to deferral cost, which means that we will guide the model to choose the expert's decision, which in this case, the right treatment.
On the other hand, when the expert is wrong, i.c. a; 0. y1 = Y(0), the deferral costs are:
Clena (x,, a, 4) 2 C (x, a, y)
Where for both alternatives, the lowest cost is the cost of the treatment = 1, which is the right treatment.
Example 2. the example presented in Figure 4b we show an interesting case, where although Y(1) > (0), the value (1) lies into the intersection of the two intervals. We y (ax, 0) 5, (2,0)
+ (ar, 1) 9, y- (ar, 1) and Y(0) Y(1) 4.5. Then we have that:
C Car, 1) =
C(x, 0) 8
As deferral cost, there are two possible cases: the expert right, meaning 1. Y(1), deferral costs are:
Coani (z,, v) = 335
C (I, a. = 0.5
14

---

for both alternatives, we have the lowest costs corresponding to the deferral cost, which means that we will guide the model to choose the expert's decision, which in this case, the right treatment.
On the other hand, when the expert is wrong, i.e. ai 84 (0), the deferral costs are:
C1 (xz, a, y) = 2 C (x, v) =
Where for both alternatives, the lowest cost is the cost of the treatment = 1, which is the right treatment.
yi (x,1))
Y00
]
(',,0)
V(0
(a)
Y (4
((t) P
4.5
(* 0) -1
(' -5
v(0)
(b)
Figure 4: CAPOs intervals overlap visualization
A.3 Comparative Analysis of the Conservative and Optimistic Costs
We provide an analysis that highlights the scenarios where each of the conservative and optimistic approaches proves superior to the other.
For sample (Ir, a, y, (0),, Y(1)) Pun, where w.l.o.g it holds that Y(1) Y()), that is, the right treatment for this sample 1.
We distinguish between two main cases:
The expert right:
When the expert is right, i.e. (e))(x) = 1, and Y (Tew( (x+)) = (1). then,a policy with this set of costs will predict the wrong treatment #(x) = O when the following condition holds:
C(æ,, 0) < min(C(z, C), (x,
1, Y(1)))
(6)
For the conservative approuch, this holds when:
(+ y ( Y(1)
(7)
As for the optimistic approach, holds when:
f+ 1) - fz 0) < 8 (xx 0) (1)
(8)
When Equation 7 holds and Equation g does not hold, these are the cases where optimistic approach outperforms the conservative approach, i.e.:
y (r, (z, (x,
28 (s, 0)
We note that Equation never holds, and thus when the expert is right, the optimistic approach cannot outperform its conservative counterpart.
On the other hand, when Equation ( holds and Equation does not hold, these are the cases where the conservative approach outperforms the optimistic approach, i.c.:
2 (x, 0) < < (z. 1) Y(1) < & (a, (x,,0)
(10)
The expert wrong
When the expert is wrong, ie. Resp (a) = O, and Y (T=p((:)) = (0). then, a policy with this set of costs will predict the wrong treatment ((x) = O when the following condition holds:
min(C(ax 0), C1 (x, = 0, y (())) < C(x,1)
(11)
For the conservative approach, this holds when:
Y (z, 1) Y(0) < (x) (x,
(12)
15

---

As for the the optimistic approach, this holds when:
(z. 1) - ( < g (x, (x,
(13)
When Equation 12 holds and Equation 13p does not hold, these are the cases where the optimistic approach outperforms the conservative approach, i.e.:
2 (a, 1) (z. (0) (z, (1,,1)
(14)
On the other hand, when Equation conservative approach outperforms the optimistic approach, i.e.:
13 holds and Equation [22 does not hold, these are the cases where the
y (z, 8 (z, < (a, +) (() < 2Y 1x,
(15)
We note that Equation (14 never holds, and thus when the expert is wrong, the conservative approach cannot outperform its optimistic counterpart.
B Proof of Main Theorems
this section, we provide proof our main theorems.
B.0.1 Consistency
Proof of Corollary we can apply Proposition from Mozannar and Sontag 2020 directly to our setting yielding the statement of our Corollary.
Note that used slightly different statement of Proposition I from Mozannar and Sontag 2020 readability considerations and consistency with our setup and problem formulation.
from
formulation of the original statement using our notation as follows:
Corollary 2. Lee is convex and is consistent loss function for L: Let arginf, B[Loe(L, Q)), then arg (0, E. arg min, B[c(i) 2).
with c(0) C(x, 0), c(1) (zz, 1), and c(1) C1 (ar, a, v) defined in Section
We recall that the surrogate loss Lee 4 defined for policies n, R be the raw output of the policy и corresponding to a class E 10, 1, such that (x) = arg (0,11, (x).. Therefore, this holds
especially for = arg inf B[ , 2; Q)), ie.
म arg max
(16)
1)
As for optimizing the original loss function L(T) 3 the optimization problem is given by,
arg inf [[(()')
(17)
according to Mozannar and Sontag 2020 this problem can be solved as a cost-sensitive problem with the costs
c(0) (zx, 0), c(1) = C(x, 1). and c(L) = C1(x, a, y) for each sample (æ, a, y). i.e. the solution of this
optimal optimization problem satisfies the following for each sample 2: # arg min,g (0.1..1 E[c(t) 7 =
Thus, adaptation of the results equivalent the original results from Mozannar and Sontag 2020
B.1 Costs Are Coherent
this section, we prove the coherency our costs as defined Section 6.2 and show the improvement of the CARED policy over policies that depend solely on either a human expert or machine.
Proof of Theorem 7 is, the right treatment for this sample is 1. We assume D holds with probability 1 5. Then prove this theorem steps: Comparison against the human expert policy (Texp), and comparison against the bounds policy(Teownde).
For sample (x, a, y, (0). Y(1) ~ Puun. where w.l.o.g it holds that Y(1) > Y(0), that
Comparison Against the Expert:
We distinguish between two main cases:
The expert right:
When the expert is right, ie. (x) = 1, and Y (a:)();;)) = Y()), then our policy will predict the wrong treatment स (xx) = 0 when the following condition holds:
C(z, 0) < min(C(x, C1 Y(I))
(18)
16

---

That is:
(x1 1) - Y (2,
< (x,
(.) - Y())
(19)
In simple words, we recall that our method assigns the treatment with the minimal cost among all other treatments. Thus, when the cost of the wrong treatment is the minimal cost among the other costs, method will make mistakes.
We note that it holds that:
Y- (x,0)- (1)
S (x,,0) (x, S
p* 0) 8 V= 1)
(+(x,0)28-(xx0)
(x, (z, 1),1),888 (x, Y (1)) -x,
Thus, # (x) is wrong when:
8+ -
-
(20)
We note that the condition in 20 never holds when CAPO bounds are valid, and therefore, with probability 1 5. it holds that (()) 1, and Y(m (2)) = (1), That is, (+)))) Y (T))
The expert wrong:
When the expert wrong, i.c. Tepp(>)) (x) 0, and Y (Fep(z)) (0), then, from the same considcrations above, our policy is wrong when:
[C (r,, Y(0)) < C(=,1)
(21)
That is:
min
(,,,0)) (x,
(22)
We note that holds that:
y (x, 1) (0)
80xx,
<
<
1) - -x-)x))
8- (=
(((.))28-
( (x, (,. (x,1)-Y(())
Thus, (xx) is wrong when:
(=)0 8
(23)
When this happens, we have that # (2x) = O, and V(#(z)) Y(0), that (#(x,)) = Y (Tepp (x:)), otherwise, when this condition does not hold, we have that ((f(z)) > Y (Teap(=))
Comparison Against the Bounds Policy
We analyze the cases where cach policy makes mistakes. We note that स is always right when CAPO intervals don't 't overlap as proved Appendix A This holds for the Thounds policy well, as this implics directly from its definition. On the other hand, when the CAPO bounds overlap, both policies can make mistakes
Thus, we distinguish between two cases:
CAPO bounds don't overlap In this case, both policies recommend the right action, thus it holds that, Y (f(x) = Y (Thannnts = Y (1).
CAPO Bounds don't Overlap In this case, Thounds defer the decision to the expert. As for म, as shown above, the policy स is wrong when the expert is wrong and the condition holds. Therefore, if condition 23 holds, and the expert is wrong, then both policies will go wrong, and thus have that (=(x)) (TToundd
Y (0).
On the other hand, if CAPO intervals overlap, but Condition 23 't hold, then # give the right recommenda- tion with Y (=(x)) (L), but Whounds still give the wrong recommendation with Y (Toonnds (x)) Y (0). that (T)()) Y (Tthonnds (2)).
17

---

B.2 Generalization Bound
We now prove Theorem
Corollary (Bounded log-Softmax). Let II be learner that satisfies Assumption Then, the
explry(())
is bounded.
(0, (x))
π a policy satisfying Assumption 3 that is for each j {0, 1, 1} we have that [Ty()= & Cx. where K is the number of classes. That is, it holds that
Proof of Corollary 3
-Cu (-)())) S C.
exp(-Cx) S exp((T)) & exp(Cx)
As a result, we have,
exp(TT(())
exp(C,)
E Ene(o.1.)) exp(TT(<))
Eie(o...y expl (F)
exp(C+)
3 exp(-Ca)
1 I exp(2 : C.)
3
Similarly, we show that
exp(TT(xx)
1 exp(-22 C+)
Exetaray exp(Tx(<))
3
Lemma 1 (Bounded loss function). Let l1 be class of policy learners satisfying Assumption 3 with constant Cs. and suppose Assumption 4 holds with constant Cv. Then LCE(TT Q) < C. Cy.
Proof of Lemma 7
Let IT be class of policy learners satisfying Assumption then the loss function LOE(T, 26; ((:)) is bounded:
exp(πz(())
+c =
Σ
- (z, (2)))
Exe(0.1.)) exp(TT(<z))
)E(O.1.1)
Σ jE(0.1.1)
w (²,22((z)) log
exp(TT,(x))
Σ ))e(o.1.)) exp((Te(x))
Σ (E(0.1.1)
w (z, Q(x))())
3 exp(2 (+))
Σ )E(0,1.1
w((2(((x)) C.C.
2 Cr
Σ
w (z, Q(x))
1E(0.1.1)
2 Ca |1 Cyl
- Ca Cy
Where step (a) follows from Corollary 3 and (b) follows from the boundedness of Y in Assumption 4
Lemma (L2 Consistency of the max(:.) -) Estimator), Let estimators An, B., of A, B based samples respectively, where there exists a function (n) such that:
A| All O (f(m)
|B,. Bl| S O (f())
18

---

Then,
max(An. (5))-max(A B)ll & O₂ (f(n))
We prove this Lemma in two steps: first we show convergence in probability of max( An, B..) to max(A, B). Then show the convergence rate.
Proof.
Convergence An and B, converge probability to A and B respectively as n tends to infinity, that is:
An P
A
n-000
B, P
71000
Then,
max(AA; An) = (An + Bn + An Bu) p,
1 n-c (A+ B B -- BI) = max max(A, B)
Convergence Rate convergence rate max( (An, Br,) ) is determined by the slower convergence rate Å.. and Bn, which is (m)) f(m)) f(m)
Corollary 4 (Bounded weights). The weights w² (2, ((x)) for E 1, of the weighted surrogate loss
function 4 based on the costs define in Section 5.2 satisfy:
(i) w/ (z, Q()) - 4.460 Cr (it) ||0 (z, Q(=)) (1, (2, (())( 1 1.0, (n 1/(44)))))
(n-
Proef. We recall that for E 1), a cost e(i) is of the form c(i) A B, where A, B (Ý+ (z, O), (0, 8 (x, 1)), y (z, 1),+ As for maxte(o.1.) c(k) is also of the form max c(k) = C D. where C D E (I+ (a, Y) (z. (z, 1)), y (z, 1), Then,
① w (z. Q(x)) = (44 B) (C D)I | + |B] CI
[D| 4 Cr.
where (a) follows from the boundedness outcomes (Assumption f with a constant Cy).
(ii) For A, B, C. E (x+ 0), Y (r, Y+ (x, 1)), estimators, respectively. Then,
Y- (r, 1), w.). we denote Á, B, Ć, D to be their
(2, (2, (()))) -B- (C- D)I
< || Áll 1- B|1 - OII ||D III
O- 1/(4+))
where (a) follows from Assumption and Lemma
Proof of Theorem Let II be policy where the policy class II satisfies Assumption 2 for policy E II. and Q. Q the CAPOs bounds and the estimated CAPOs bounds respectively. Given our loss function (, Q) which satisfies Assumption 3 and Lemma We interested in bounding the term
Ln(=:Q) Q): (Ln(T:Q) Q)) (Lo(m: Q) Ls( Q))
(24)
We get the upper bound for Term B by applying Theorem 26.5 from SoomSShlev-Shware and Devid (014), that is, with probability at least 5 we have:
(1(8)))
(T( Q) Ls(T;: Q) S 2RI + 21-4 C. Cr
n
where R., (1I) is the Rademacher Complexity of the policy class 11 we have from Assumption 2 C., and Cy the constants have from Assumption and Assumption respectively.
19

---

We now show the upper bound for Term A:
LD(T; Q) LD(x;; Q) (e., 2; (x..))))))
LOEETT, ((x)))
exp((T((z))
(=
Σ (x))(x)) -log
Exe(ua..)) eXp(T<<<))
JE(O,111)
exp(1T((z))
Σ (E(0.1.1)
w/ (2,2)(2z) - -log Exe(o,111) exp(T<(x))
= =E==(z..w)>p(c)))
Σ (w'(x.Q(=)) w w2 (=(x)((x))) JE(0,1,1)
exp(TT((r))
llog log
(EEEOA,1)) exp(Tx(x))
< E==(snnirots)
Σ JE(O
((=() w' (())))))) -22 (2)
(0)
22 Cr (E=((rnn)iit))
(z,(),((x)) - w Q(x)))
(E(U.1.1)
C.
Σ
E_ (==(((aw) (2, w (z, (x))
(6)
jE(O,1,
22.Cr Cr
Σ
(2.aw))))))))))) (=. w (z. (()))
(0)
JE(0.1,1)
C.
E (x)
1E(0.1.1)
< Cr.
Σ
B |B=(nna))
[wwde, Q(x)) - w/ (z, [(x)))
(
je(0.1.1)
$2 C. 3-4 Op (2
(n- -1/(2-))))
< 21 OM. c. (n-1/(2tr))
(
where:
(a) Corollary
(b) Linearity of Expectation.
(c) [[X] < [[]]
(d) [[X] / E[X]
(e) Jensen's 's Inequality: E[X]² & B[X²).
(f): Assumption
(g) Corollary and Definition of O,(()) with constant C > 0.
Putting it all together, we have:
Lo(T; Q) - Ls(=;(Q= = (T: Ln(T:Q1) + (Ln(=;Q) Ls(T: Q1)
5 2R. 21 C. Cr 21m(1/8)
+ (n (n-1/(2t))
20

---

C Additional Experimental Detail
The experiments in this paper were conducted using PowerEdge R750XA Server with CPUs and 4 NVIDIA A40 GPUs. Here we provide all the details required replicate the paper results. In addition, we provide replication code at wtthtts://ggithut wwt./ooom///aaeeceee http///aaaahh/CAARRR
C.1 Synthetic Data
In this experiment, We used the synthetic dataset from Gao and Yin 2 The CAPO bounds were estimated using XGBRegressor from xgboost as the base learners for the B-Larrner (Oprescu et al. 2023) estimator, and a wengisttiReggess.. hyper-parameter choices for each model. As for the policy model, use logistic regression model imple- mented as one-layer MI.P using several functions from pytorch, The policy model was implemented using pytorch_1i barghhiigg model as a wrapper model that can receive any policy model as the base model for learning the policy. Moreover, the hyper-parameter we use for the policy model are follows: learning rate= 0.001, optimizer= Adam, patience= 3, and max_ epochs 100.
from acikit-learn
the propensity score estimator. We show Table the
We replicate the experiment from Gao and Yin (2023) where they generate 10 instances of the synthetic dataset according to the Data Generation Process they mention in their paper, where lower outcomes are assumed to be better in this dataset. For each instance, they vary the level of allowed hidden confounding and compare the methods by their regret related to the baseline no-treat policy, i.e. (2) 0.
Table 1: Hyper paremrrrrsseterss for model choices in the synthetic data experiment
Model
parametrr
Value
XGERegressor (xgbooat)
learning_rat rate
0.1
min child weight
max depth
5
estimators
200
Logistic Regression (scikit-learn)
C
penalty
elasticnet
solver
saga
11_ 11_ratio
0.7
max_iter
100
C.2 IHDP Dataset
201 which was introduced by Jesson et al. 202T]. The CAPO bounds were estimated using XGBRegressor from xgboost as the base learners for the from scikit-less
In this experiment, we use the hidden confounding version of IHDP Hill
the propensity parpemerr choices for each model, As for the policy model, we use a logistic regression model implemented as a single-layer ML. using several functions from pytorch, The policy model was implemented using pytorch_lightning model as a wrapper model that can receive any policy model as the base model for learning the policy. Moreover, the hyper-parameters were tuned for each uncertainty level A using the ray. tune over the search space of the hyper parameters: The search spaces used is Learning_rate E [Le 4, 0.1] joptimizer [SGD, Adam, AdamaW] weight_decayy [le 10, -3-3. 3]. patience E [5, 20). and max_ EpochsE [30, 50).
B-Learner Oprreeu et al.
2023 estimator, and
score estimator. We show in Table 2 see
2023 method, train logistic policy with the byper-paramcterd learning_res rate 0.01, batch_size 32, feax_epochs 100. The syper-parameters were tuned using grid search over a set possible values for each parameter, and the set with the most reasonable loss was chosen. In addition, we set C(X)= = 0 which represents the additional cost of deferral used in Gao and Yin (2023)'s 's objective for cases where the outcome of deferral is Y + C(X). In this experiment, the outcome of deferral to the expert is solely based on the result of the treatment preseribed by the expert, which is Y.
As for Gao and Yin
We generate 1111 instances of the Hidden Confounding IHDP [Jesson et al. 2021 each consists of training (n = 470), validation (n = 202), and test (n = 75) subsets, where for each instance the seed runs over 0, 1,..., which is the number of the trial/instance.
We filter the 1111 instances by value oCATE = Vev/aaaCCat (ar(CAATE..x()) by excluding trials with high OCATE, as instances with high deaTE are unrealistic and does not match the results of the original study Srrooks-Guma et al.) [192]. as explained in Curth ct al.] [202T]. In our case, based on the histogram of dcate for the 1111 instances of the dataset, exclude those with oCATE > 15 and remain with 1000 instances.
Table 2: Hyper-parameter for model choices in IHDP experiment
21

---

Model
Hyper- parameter
Value
XGBRegressor (xgboost)
learning_rai rate
0.05
min_child__a weight
5
depth
estimators
500
Logistic Regression (scikit-learp)
C
penalty
elasticnet
solver
saga
11 ratio
0.7
max max_iter
10000
C.2.1 Semi-Synthetie Expert
In this experiment, design semi-synthetic expert, based the original observed expert's policy, and the outcomes observed under this policy. The actions and outcomes under this policy denoted and Y, respectively, as they are (trivially) exactly the observed actions and outcomes. Let A' and y" denote the new actions by the new expert's policy and the observed outcomes under this new policy respectively. We recall that the actions and the outcomes generated by the Oracle policy are denoted by A', and Y" respectively. Then, for i [re]. the expert policy given by:
A = AN if x =1
Ai, otherwise
Y()), = Y(1),, otherwise.
Y
The expert A' is defined based the covariate 217 "worked during pregnancy", which is a binary covariate that indicates whether the mother worked during the pregnancy or not. This covariate receives value of with a probability 0.59. We huild the new expert to equal to oracle policy when #17 = 1, and equal to the original policy A otherwise. Thus this expert is perfect when #17 = 1. This is an aspect of the expert that ideally we would want the model to learn. The choice of covariate shown in Jesson et al. (202T). where they show analysis of the relationship between each covariate and both the treatment and outcome. choose the covariate 2117 as it one the features that is correlated with both the treatment and the outcome, and thus, an important feature that we are interested in testing our model on its ability to learn it.
was according to the covariate table
22

---

NeurIPS Paper Checklist
Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Justification:
Guidelines:
The answer means that abstract and introduction do not include the claims made in paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Section g
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are violations of these assumptions (c.g., independence assumptions, noiseless settings, model well specificaaion, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to penalize honesty concerning limitations.
Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
23

---

Justification: We state all the assumptions and theoretical guarantees in Section 6 All the proof' are provided in Appendix B
Guidelines:
The answer NA means that the paper does not include theorctical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, if they appear the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We explain the main details of the experiments in Section 7 Appendix C
More details are given in
Guidelines:
The answer NA means that the paper does not include experiments,
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reprodueible important, regardless whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, may be necessary to either make possible others to replicate the model with same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results. access to hosted model (e.g.. in the of large language model), releasing of model checkpoint, or other means that are appropriate to research performed.
While NeurlPS does not require releasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproducc that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) contribution a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soue source models, may be that access registered users), hut should be possible for other researchers have some path to repraducing or verifying the results.
model is limited some way (e.g..
5. Open access to data and code
Question: Does the paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, described supplemental material?

---

Answer: [Yes]
Justification: We include in Appendix the link for results replication. We provide instructions for running and replicating the results well.
Guidelines:
The answer NA mcans that paper does include experiments requiring code.
Please see NeurIPS code and data submission guidelines (lttpp::/niiss.ccppubii for more details.
While we encourage the release of code and data, we understand that this might not be possible,
so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g., for
open-source benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurlPS code and data submission guidelines htttpp::/nips.cc///e guides/CodeSubnin www.osuuubssssoPPooooooo
reproduce
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand results?
Answer: [Yes]
Justification: the relevant details are given in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: Each experiment this paper was conducted on several trials, all the results were reported with error bars and confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should he explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, one should it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality crrors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric emror bars that would yield results that are of range (e.g. negative rates).
error bars reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer:[Yen
Justification: we mention the compute resources information the machine we used running all the experiments in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our work does not violate any aspect NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (c.g., if there is special consideration duc to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussed briefly the introduction and discussion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact why the paper does address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there a direct path to any negative applications, the authors should point out. For example, the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
legitimate to point out that an improvement in
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that been put in place responsible release data or models that have a high risk for misuse (e.g. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelincs:
The answer NA means that the paper poses such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the ereators or original owners assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
All assets used in the paper. properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, license, copyright information, and terms of use the package should be provided. For popular datasets, wwppppaipphhaooooo thcode. cww/co////aa/oo comm/datseetelll has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
27

---

Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: No assets.
Guidelines:
The answer NA means that the paper does not release assets.
Rescarchers should communicate the details of the dataset/code/mooel as part of their sub- missions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing.
Guidelines:
The answer NA that the paper does not involve crowdsourcing research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: IRB.
Guidelines:
The answer NA means that paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
initial submissions, not include any information that would break anonymity applica- ble), such as the institution conducting the review.
28


Reviews:
[{"review_id": "zpMR8YYCbI", "reviewer": "Reviewer_Vchf", "summary": "This paper extends the work of [Mozannar & Sontag 2020] on \"learning to defer\" to the causal inference setting with (bounded) hidden confounding. Compared to the original supervised learning setting, the causal setting does not have ground truth labels. Therefore this paper proposes to use estimated bounds on the potential outcomes to drive the deferral decisions, and formulates it as an optimization problem. \n\nExperiments are conducted on synthetic and semi-synthetic (IHDP) datasets.", "questions": "Major:\n- L227 Eqn (3) B-learner policy: this is used as a baseline approach in the experiments. Currently this is presented in the theory section, and for a moment I thought this is your proposed approach. It is such a straightforward definition, a simple approach that also outperforms the baselines. I think this needs to be presented earlier in the text, and you need to make it clear why it's not good enough and how your approach is different. \n\nOthers:\n- L37: \"human expert typically has access to the hidden confounders\" - in what form? Do you mean human decisions are more likely to account for hidden confounders. \n- L181-182: in the definition of pessimistic and optimistic deferral costs, what is $\\max_{a' \\neq a_i}$, aren't there only two actions? L190 I assume it's defining the same thing but different notation is used. \n- L202 Algorithm 1 is never referenced in the methods section. Also it seems to simply repeat the expressions already presented in the text above, not sure this algorithm is necessary.\n- L224 what does bounds are valid *on average* mean? How does that affect the theoretical results if some bounds are invalid? \n- L227 \"This policy defers if and only if the CATE upper bound is positive and the CATE lower bound is negative.\" You may elaborate that intuitively if bounds are both positive then the CATE is positive (and vice versa) so it's clear what to do, and the deferral situation is when it's unclear the true CATE is >0 or <0 and unclear what to do. \n- L256 Theorem 2the generalization bound compares the training loss and the expected loss which are both L_CE, but how is the result of this learning process related to Eqn (1) the true optimization objective? \n- Theorem 2: What does the generalization mean and how does it compare to other things? Does it tell us about what we should do in practice when using your approach? \n\n- experiments should compare to a random deferral strategy as a baseline. \n- Fig 2a \"proposed method outperforms both baselines\" -> this is not true, for large log(Λ) the proposed approach is overtaken by B-learner. \n- Fig 2b as deferral rate increases beyond 60% performance drops, I think it's worth commenting on if for example we allow for *up to* 80% deferral, can the policy learning figure out it only needs to defer 60% of the time. Alternatively, what are practical ways of selecting this deferral ratio? \n\nMinor issues: \n- Citation style: there are places where the reference in text should be parenthesized but are not. \n- Typo: There's a slash on L13 of the abstract. \n- L122-124, L138-139L deferral policy defined twice\n- L132 \"ho\" -> \"how\"\n- L204 \"we see than\" -> we see that\n- L249 says $Y^{+}_1(x) - Y^{+}_1(x)$. I think there is a typo. \n- L271 says three variants, but only two are listed \n- L347 \"We note there when learning to act\" -> we note that", "limitations": "Main limitations noted include access to expert actions in observational data, and that experts behavior do not change when they receive machine's deferral.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Studies an interesting and relevant problem\n- Writing is generally clear.", "weaknesses": "- Although writing is generally clear, the flow could be improved to better explain some of the baselines / obvious / naive approaches, and justify why the proposed approach (arguably more complex) is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Baselines Presenting:**\nThank you for this useful suggestion, we will make sure to better explain the baselines and justify the merits of our approach See also our reply to the major question. \n\n**Clarification of baseline and differentiation from proposed method:** \nThank you for this important comment - we indeed should have explained this point better. We will revise and explain in more detail the baseline policy we denote by $\\pi_{bounds}^Q$, which defers if the estimated CATE interval crosses 0. This approach is indeed natural and has been used for deferral in many previous papers (including papers which only deal with statistical uncertainty). It was also used in the B-Learner paper.\n\nWe will also give a more detailed discussion of the ramifications of Theorem 1: in particular, the Theorem shows that whenever the bounds include the true potential outcome, the action that minimizes our proposed loss function is at least as good as the action implied by the baseline “bounds” policy, as well as the human expert policy. We further show that under very reasonable conditions there will be a non-trivial proportion of cases where the action minimizing our loss function will be strictly better than the one chosen by the bounds policy and expert policy. Finally, our experiments show empirically the added value of our approach compared to these baselines. The reasoning for why our approach might perform better is that it directly optimizes for the deferral classification, as opposed to classifying a sample as “deferred” solely based on some uncertainty measure. This last point has already been made by Mozannar & Sontag 2022 for the non-causal case.\n\nWe will make sure to clarify these important points in the revised text. \n\n**Human expert access to hidden confounders clarification:**\nWhen learning how to act based on historical data where the actions in the data were made by human experts, the confounders are by definition factors that affected both the action choice and the outcome. Thus, if at test time we have a human decision maker drawn from the same distribution as those in the historical data, they will by necessity have access to all the confounders. Notably, this is true even if those confounders only affect the decision maker unconsciously: the point is that we have access to actions taken by the “true” policy which generated the historical data. Consider for example the case where a human clinician’s decisions are affected by the demeanor of the patient: this would typically be considered a hidden confounder, as it is not recorded in health records. However, at test time the clinicians can still see the patients’ demeanor and be affected by it - in that sense they have access to a confounder which is hidden from the eyes of any system which is merely based on recorded data. \n\n**Deferral costs notation:**\nIndeed, while our approach can be extended to multiple actions using the max operator as in L181, in our paper we focus on the binary case. We will correct this and use the notation for binary actions as in L190. \n\n**Algorithm 1:**\nThank you for your note, we will consider whether we can do without the algorithm, saving the corresponding space. \n\n**Bounds validity:**\nThis sentence refers to Corollary 2 of Oprescu et al. 2023 [2], which  gives a precise statement. Informally it means that with high probability, the average of the estimated upper bounds over the dataset is larger than the average of the true upper bounds given by the MSM. A symmetric result is true for the lower bounds. \n\n[2] Oprescu, Miruna, et al. \"B-learner: Quasi-oracle bounds on heterogeneous causal effects under hidden confounding.\" International Conference on Machine Learning. PMLR, 2023.\n\n**CATE bounds' deferral policy explanation:**\nThank you for the comment. You are correct in pointing this out and we will make sure this point is explained more clearly. \n\n**Theorem 2 - Optimization objective:**\nL is a non-convex loss function that is hard to optimize, so we propose optimizing L_CE which is convex, and easier for optimization. We show that L_CE converges to the optimal solution of L. Then, since L_CE is easier to work with, and converges to the desired solution, we show the theoretical guarantees on L_CE\n\n**Theorem 2 - Generalization meaning**\nTheorem 2 implies that our optimization objective is reasonable in the sense that attaining good performance on the train set by minimizing it can indeed lead to good performance over unseen samples. In principle one could have had an objective which cannot be learned, or where the errors cannot be controlled even for arbitrarily large samples. \n\nThe reason this result is not completely standard is the fact that we optimize a cost sensitive objective where the costs themselves are estimated from data. This is similar in spirit (though not at all in the specifics) to cases in causal inference where objectives weighted by the inverse propensity score are considered, and one has to account for the error in estimating the propensity scores themselves. Our approach using estimated costs turns out to be tractable, but still requires some care in its derivation and proof. As is usually the case in generalization bounds, the result does not translate immediately into a practical guide, but it does imply that the method is expected to respond “well”  (i.e., similar to standard supervised learning) to changes in sample size and the complexity of the function class.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bzRbLkR50v", "reviewer": "Reviewer_GgdC", "summary": "The paper combines two frameworks: optimizing surrogate losses for learning to defer, and bounds on CATE under unobserved confounding under the marginal sensitivity model. The challenge with applying the surrogate loss from Mazumdar and Sontag to the causal setting is that the costs of various classification outcomes are unknown, and need to be estimated. The paper proposes essentially deferring when the causal effects are not (partially) identified to be strictly positive or negative and develops conservative/optimistic characterizations of these losses. Deferral comprises of reporting the recorded $a_i$. When the unobserved confounding is exactly correlated with potential outcomes (as is the case with simulation specifications of unobserved confounding that are \"favorable\" to robust bounds methodology), therefore deferral can be adaptive and improve upon both the expert policy and valid bounds.", "questions": "- Could the authors please describe if there is some relationship between implicit assumptions on regimes where this approach \"does well\", i.e. can _strictly_ improve upon robust PI bounds policies, and recent work on \"superoptimal\" treatment regimes? \n\nsuperoptimal treatment regimes: Optimal regimes for algorithm-assisted human decision-making\nMJ Stensrud, JD Laurendeau, AL Sarvet - Biometrika, 2024\n\n-", "limitations": "sufficiently although studying more explicitly 1) when the policy improvement is strict and 2) how do assumptions on the expert-ness of the underlying confounded behavior policy relate to the performance of the conservative/optimistic losses given in the paper", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "significance: \n\nWhile the paper connects two prior areas (surrogate losses for learning to defer, and PI bounds), there is some work needed to establish the connection and the paper does a good job of doing so and showing that such an approach can obtain improvements upon current conservativism of robust approaches only (under implicit assumptions on expert-ness of the underlying behavior policy). This is well-illustrated in the empirics.", "weaknesses": "There seems to be an implicit assumption that the confounded behavior policy", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Implicit assumption on the confounded behavior policy:**\nWe would be grateful for a further clarification of this comment, as it seems to have been cut short. \n\n**Relationship between implicit assumptions on regimes where this approach \"does well\":**\nThis is a fascinating connection, as both our work and the work cited consider explicitly the case of human experts with access to information which the model does not. The work by Stensrud et al. considers a different scenario, where using an instrumental variable and some clever identification results one can “override” the human expert. The main difference is the reliance on instrumental variables for identification, which is a serious drawback as such variables do not always exist. However, in future work we are considering merging their approach with MSM-type bounds. We include a mention of their work and this connection in the revised version.\n\n**Exploring policy improvement conditions and expert-ness assumptions impact on loss performance:**\n1) In the proof of Theorem 1 we outline conditions for when the policy improvement is strict, which relies on whether inequality (12) holds or does not hold. Unfortunately this condition, which can easily occur in practice (as we see in the experiments), does not have a straightforward interpretation in terms of CATE bounds. Our best attempt of explaining it is as follows:\nAssuming w.l.o.g.that Y(1)>Y(0), then this event occurs when the negative of a “narrow” CATE bound (narrow in the sense of taking the lower bound for the higher potential outcome minus the upper bound for the lower potential outcome) is larger than the lower bound minus the actual potential outcome. We will refer to this condition in the main paper.\n2) In general we cannot know for sure whether the experts are mostly correct or not, due to the fundamental problem of causal inference. However, if we assume they are generally correct, then one might wish to use conservative approach and defer more often to the experts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mfHMa0BdV1", "reviewer": "Reviewer_ukAA", "summary": "This work learns a policy that can abstain from predicting an action in the case where actions are binary. Their idea follows previous work in the learning to defer literature for supervised learning. Their proposed surrogate can recover the optimal policy, for which they design the cost functions in a way to satisfy this condition. Their experiments show the effectiveness of the proposed method.", "questions": "- The cost function $C(x_i,a)$ is the worst-case regret. I wonder if what would the results be if an alternative such as mean regret or best-case regret is used.\n- L 240 I wonder if the Rademacher Complexity varies with NN architecture as the paper claims any NN with weight decay or dropout has the same Rademacher Complexity.\n- Did the authors consider the fact that the number of sample the expert can process is limited?", "limitations": "- The cost function relies on bounds of potential outcomes, which is often unknown in real-world applications. This eventually reduces to the dependence on the B-Learner. I wonder if the authors considered alternatives of B-learner to estimate the bounds.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- They show the surrogate loss has the same optimal solution as the original expert-machine loss.\n- Their experiments on synthetic data and IHDP hidden confounding show their method CARED outperforms baselines consistently in terms of policy value.", "weaknesses": "- The authors missed some references on learning to abstain such as [1]\n\n[1] Yin, Tongxin, Jean-Francois Ton, Ruocheng Guo, Yuanshun Yao, Mingyan Liu, and Yang Liu. \"Fair Classifiers that Abstain without Harm.\" In _The Twelfth International Conference on Learning Representations_.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**References on Learning to Abstain:**\nThank you for bringing this paper to our attention. While our paper has been inspired by the work of Mozannar & Sontag on deferral which adopts an end-to-end approach, this work presents an alternative, post-hoc approach to the problem of (non-causal) deferral. We leave it to future work to consider how the approach of Yin et al. above can be adapted to the causal case, where no ground-truth labels are available.\n\n**Alternatives of the cost function:**\nThis is a very important research direction which can be further extended to a general question - how to choose the best set of costs for a specific problem? Each choice (example average cost) would imply different trade-offs which would be interesting to explore.\n\n**Rademacher complexity of NN architectures:**\nIndeed the specific complexity would depend on the architecture. Our point was merely to state that this condition is not vacuous and is fulfilled by a range of widely used approaches. We refer the reader to the following papers which discuss this subject with far more nuance. We will mention this point in the paper.\n* Bartlett, Peter L., Dylan J. Foster, and Matus J. Telgarsky. \"Spectrally-normalized margin bounds for neural networks.\" Advances in neural information processing systems 30 (2017).\n* Neyshabur, Behnam, et al. \"Towards understanding the role of over-parametrization in generalization of neural networks.\" arXiv preprint arXiv:1805.12076 (2018).\n* Golowich, Noah, Alexander Rakhlin, and Ohad Shamir. \"Size-independent sample complexity of neural networks.\" Conference On Learning Theory. PMLR, 2018\n\n**Impact of limited expert sample processing:**\nThis is an important point. We believe that in many cases the deferral rate would be set apriori according to such limits on the experts’ time and capabilities. This motivates the comparison in Figure 2b which asks “given a fixed deferral rate, which policy would perform the best”: for example, assuming we know that experts can only deal with 30% of the samples, we can fix the deferral rate accordingly. \n\n**Reliance on B-Learner for estimating bounds:**\nWe assume we are working with observational data, which is the case with real-world applications. In this case, for each sample, we have the observed treatment, and the observed corresponding outcome. That is, for each sample, we only observe one treatment, and will need to estimate the other potential outcomes in order to know what is the right treatment for each sample.\nOur theoretical guarantees hold for any model that provides bounds on the CAPO and for which Assumptions 2-5 hold. In practice, the algorithm would work with any model that gives bounds on the CAPO even if these assumptions do not hold, though for such a model our theoretical guarantees might not carry over. \nThus, the B-Learner is just one possible model that our method can use. However, our choice of the B-Learner is due to the strong guarantees it provides such as validity, sharpness, robustness, and the ability to perform well on moderate amounts of data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v2enBbrg3O", "reviewer": "Reviewer_r89L", "summary": "This paper studies deferral policy learning when there is unmeasured hidden confounding observed by human experts but not recorded in data. The paper formulates it as a cost-minimization problem and derives a feasible surrogate loss. The method is shown to achieve better policy value on synthetic and semi-synthetic data.", "questions": "- In Sec. 4.1, the expert’s action space $\\mathcal{Y}$ is not defined. Is it the same as the action space $\\mathcal{A}$?\n\n- Are the observed data $(X_i, A_i,Y_i)$ only generated by human experts? Can it be generated by an algorithm? \n\n- In Figs 1 and 2, why the value of CARED policy does not change with a wide range of $\\Gamma$ values? And why does it not peak at the true $\\Gamma$? It would be better to plot the true $\\Gamma$ as a vertical line in these figures.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is generally well-written.\n- Theoretical properties are explored.\n- The proposed method is easy to implement and the performance is validated by the empirical studies.", "weaknesses": "The performance of the proposed human-AI collaboration system relies on the properly specified cost function. For the expert deferring cost and action assigning costs, the current algorithm sets them in a conservative way. As the author mentioned, there are multiple ways to set these costs. Is the conservative cost always the best? If so, please provide theoretical and empirical support. If not, it would be helpful to clarify the situations when the conservative cost is inferior and evaluate other cost functions in the empirical studies. \n\nFor the theoretical analysis, the main condition in Definition 1 requires coverage for all $Y(a)$, which is a strong and maybe infeasible requirement. In practice, the guarantee usually only holds for the expected coverage or the probability of coverage. How would this more practical validity definition influence the Theorem 1 claim? \n\nLastly, the paper states \"there are no previous works that learn a policy with the option of deferral under hidden confounding,\" but it seems [1] considers a similar scenario of unmeasured confounding in the human-AI system. If so, it would be important to discuss the differences and use this method as a baseline in the simulation studies. \n\nI look forward to the author's responses. \n\n[1] Confounding-Robust Policy Improvement with Human-AI Teams (https://arxiv.org/abs/2310.08824)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Cost function alternatives - conservative vs. optimistic costs:** This is an excellent question. Under the validity assumption, the conservative approach is guaranteed to be correct whenever the expert is correct, while the optimistic costs do not have this guarantee. Thus, intuitively the conservative costs would be better in cases where the experts are generally correct. More generally, neither the conservative nor the optimistic case are always superior to each other, as we show now:\nLet $(x,a,y)$ be a sample where w.l.o.g. $Y(1)>Y(0)$. We have the following sufficient conditions:\n1. When the expert is wrong, the conservative costs are strictly better than the optimistic costs when the following holds:\n$2\\hat{Y}^{-}(x, 0) < \\hat{Y}^{+}(x, 1)+Y(1) < \\hat{Y}^{-}(x, 0)+\\hat{Y}^{+}(x, 0)$.\n\n2. When the expert is wrong, the optimistic costs are strictly better than the conservative costs, when the following holds:\n$2\\hat{Y}^{-}(x, 1) < \\hat{Y}^{+}(x, 0)+Y(0) < \\hat{Y}^{-}(x, 1)+\\hat{Y}^{+}(x, 1)$.\n\nIf none of these conditions hold we might have that both are wrong or both are correct.\n\n**Bounds validity assumption:** \nThe Theorem’s claims hold for any $x$ for which the bounds are valid. Thus, if the bounds are valid for a fraction of $1-\\delta$ of the $x$’s, the Theorem will hold for the same fraction of cases. The goal of the theorem is to show that the costs we derived indeed make sense and would lead a model to choose the best course of action.\n\n**Related work:**\nThank you for pointing this paper out! As this very relevant paper has also been brought up by another reviewer, we choose to comment on it in detail in the “general” comment at the top. We should definitely have caught on to it, and we will refer to it, revise our claims, and discuss it in the revised version of our paper, as detailed in the aforementioned comment.\n\n**Expert’s action space:**\nYou are right, this is a typo. The action space is the same as the action space $\\mathcal{A}$.\n\n**The source of observed data:**\nIn principal the only requirement we have is that the process generating the actions in the historical data is the same one  generating them at test time; and our method is geared toward the case where the process generating these actions has access to extra information that is not reflected in $X$, but which is correlated with the outcome $Y$. Thus, while we are motivated by human experts, the method is not limited to that case.\n\n**CARED policy performance across $\\Lambda$ Values:**\nIn Figure 2b we see that for large enough $\\Lambda$ the CARED policy does indeed change and its value decreases; this can also be seen to a lesser degree in Figure 2a. We will add a line corresponding to the true $\\Lambda$ value in Figures 1 and 2, following your suggestion. Notavelyt for the IHDP hidden confounding dataset this $\\Lambda$ value will be estimated from the propensity scores, as this dataset was created by hiding a confounder from the original IHDP dataset.\n\n[3] Jesson, Andrew, et al. \"Quantifying ignorance in individual-level causal-effect estimates under hidden confounding.\" International Conference on Machine Learning. PMLR, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o2okLqWKyl", "reviewer": "Reviewer_fb21", "summary": "The paper proposes a learning to defer method for policy learning problem with binary actions under unobserved confounding. The proposed method uses MSM to bound the confounding strength and compare the pessimistic bound between Y1, Y0 and identifiable human reward. The final optimization uses the surrogate loss proposed in Mozannar and Sontag [2020]. Overall, the method seems sound. However, the authors overlooked some existing literatures in learning to defer for policy learning problems. The authors compare the proposed method  with a self-proposed rejection-inference style baseline, and policy learning method considering MSM without humans.", "questions": "See weakness.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper proposes a sound solution to policy learning with unobserved confounding by conservatively comparing the lower bound of Y1, Y0 and identifiable human reward. The method has a strong performance compared to baselines used in the paper by leveraging a stronger direct method class. It is interesting to see how direct method can be used in the UC case for learning to defer since with unconfoundness, both Y1 and Y0 are identifiable and humans may not be needed unless the policy class is limited.", "weaknesses": "Missing related work and inaccurate claims: For the related work, it seems that the learning to defer for policy learning problem is first proposed in [1], and the learning to defer for policy learning with unobserved confounders is first proposed in [2]. These two papers adopts the Inverse-Propensity-Score framework to address the problem with similar experimental setup and the same problem setting. The authors should compare with existing works and it seems the claim that it is \"the first to learn a policy with the option of deferral to an expert while allowing for hidden confounders\" is not very accurate. Also the assumptions in this paper is stronger since it requires an additional assumption in Definition 1. \n\nBinary action: [1][2] can work with multiple actions, it also seems the proposed method  should also be extended to multiple actions, so why authors only restrict to binary actions?\n\nTheoretical result: For the bound validity assumption/definition 1, this seems to suggest Y is deterministic, which is rather strong. For example, a simple Gaussian distribution would violate this assumption. What if Y is a random variable, how can theorem 1 be adapted? I would expect a regret bound for the total reward in the paper (theorem 2 is a bound for the approximated loss instead of the reward if I understand correctly). Similarly, for a general direct method, when would we expect assumption 1 holds and not holds?\n\nExperiments - baseline: For baselines, I would expect a purely pessimistic AI baseline for a fair comparison. For example, the eq 3 can also use the pessimistic principle (compare Y1_lower and Y0_lower in the otherwise condition) authors used for comparing human and pessimistic bounds of the counterfactuals.  The authors should also compare with existing works using IPS method and discuss the trade-offs. \n\nExperiments: the experimental results are relatively weak with a synthetic dataset and the IHDP dataset. It would make the paper stronger if the authors can experiment with real human responses and real-world datasets. \n\nPresentation: I feel the authors should expand on the introduction of Mozannar and Sontag [2020] for the paper to be self-contained. Currently, it seems hard to understand unless readers have read Mozannar and Sontag [2020] separately. \n\n[1] Gao R, Saar-Tsechansky M, De-Arteaga M, et al. Human-AI collaboration with bandit feedback, IJCAI-21. \n[2] Gao R, Yin M. Confounding-robust policy improvement with human-ai teams[J]. arXiv preprint arXiv:2310.08824, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Related work:**\nThank you for pointing this paper out! As this very relevant paper has also been brought up by another reviewer, we choose to comment on it in detail in the “general” comment at the top. We should definitely have caught on to it, and we will refer to it, revise our claims, and discuss it in the revised version of our paper as detailed in the aforementioned comment. We will also make sure to refer and discuss the earlier work by Gao et al. (2021) which deals with the unconfounded case.\n\n**Binary action:**\nIndeed our method can be readily extended to multiple actions. We found the presentation to be simpler (including visualizing the intervals as we do in the appendix) for the binary case. \n\n**Theoretical results - the bounds validity assumption:**\nThe Theorem’s claims hold for any $x$ for which the bounds include the true potential. Thus, if the bounds obey this for a fraction of $1-\\delta$ of the $x$’s, the Theorem will hold for the same fraction of cases. Bounds that include quantile components (as the B-learner bounds do) could be robust to a fair degree of noise in the random variable. Oprescu et al. show (in their Corollary 2) that the B-learner bounds are valid on average. More generally, the goal of the theorem is to show that the costs we derived make sense and would lead a model to choose the best course of action.\n\n**Experiments - baselines and data:**\nThank you for the suggestions. We wish to point out that we did compare with an IPS based method: the method of Kallus & Zhou 2020, denoted CRLogit  and CRLogit L1 in our experiments. We further attempted to compare with the method of Gao & Yin; however, as we detail in the general comment to all reviewers at the top, we unfortunately could not obtain an implementation of their code or experimental setup, and could not replicate it fully based on the details in the arxiv paper.\n\nWe wish to point out that experimenting with real human responses would require an “active” experiment: since this is a causal problem we cannot merely use historical human decisions, but rather we would need to recruit real humans that act as decision makers, in a setup where we know the causal ground truth. While we completely agree that this would be an ideal experiment, we believe that most causal inference papers in the community are not required to attain such a high standard, and we humbly ask the reviewer to take this into account. For example, in the Gau & Yin paper, while data of real human responses is used, they synthesize the risks and the hidden confounding aspects of the data. Similarly, we use the IHDP dataset which includes real human subjects, and induce hidden confounding in it. \n\n**Related work presentation - Expanding on Mozannar et al.'s work:**\nThank you for this important comment. We will make sure to improve our presentation of Mozannar and Sontag [2020] and ensure the paper is self-contained.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all of our reviewers for their insightful comments and constructive feedback, and we are encouraged by your support. Your comments have helped us clarify and strengthen this work, and we are grateful for them. We will address here a major comment regarding the paper of Gao & Yin, and address additional comments in the individual responses to the reviewers. If there are remaining or new questions please let us know and we will do our best to address them. 


We wish to thank reviewers fb21and r89L for bringing to our attention the paper “Confounding-Robust Policy Improvement with Human-AI Teams” by Gao and Yin, from October 2023 [1]. We were unaware of this paper, which has the same motivation as ours: learning to act in conjunction with a human expert, when the data used for learning has a bounded degree of hidden confounding. Thus, our claim in the Discussion of being the first to study the problem is incorrect, and we apologize for our oversight. We will add a discussion of this very relevant paper, and the similarities and differences of their approach compared to ours. 
The main differences between the papers are:


1. Gao & Yin optimize an inverse propensity weighted objective, while we optimize a cost-sensitive objective. This means that unlike our approach, their approach does not model the outcome directly. Moreover, the weighted objective they employ leads to a situation where only cases where the proposed policy agrees to a high degree with the observed policy would be taken into account. Our approach can be more efficient as we can make use of bounds that merge in a near-optimal manner the propensity score and outcome models such as the B-learner. 

2. Gao & Yin provide a bound on the regret of the policy. Their bound assumes a well-specified model and scales with the square of the smallest inverse propensity score in the historical policy, which could be quite small. Our result does not bound the regret directly: instead, it shows that we can obtain low values of the weighted cost function while taking into account the error in estimating the CAPO bounds. We further show that the pointwise minimum of the loss function is better than both the human and the bounds-policy. Our theoretical results are thus not directly comparable. 

3. Gao & Yin also address the case where there are multiple human experts each with their own policy. This implies that the specific experts who were used to generate the historical data to also be the experts that use the system upon deployment. This is not a case we have examined and we will look into its ramifications in future work.

Regrettably no code is available online for replicating the method of Gao & Yin, and no code is available for replicating their experimental setup. We have contacted them but unfortunately they have said that the code cannot be available at this time due to constraints on their side. We have attempted to replicate their simulation, which is similar to ours, but several crucial details are not reported in the paper and we could not replicate their numbers.

We believe both papers have their merits and drawbacks, as we approach the same challenge using quite different algorithmic approaches. We hope that in the future a direct experimental comparison will be possible.

In the attached PDF we add, following the suggestion of reviewer Vchf, a version of Figure 2b of our paper with an added random deferral baseline.


[1] Gao, Ruijiang, and Mingzhang Yin. "Confounding-robust policy improvement with human-ai teams." arXiv preprint arXiv:2310.08824 (2023).

Meta-review: This paper proposes a framework for learning to defer under hidden confounding by relying on partial identification of causal effects and deferring (in a cost-sensitive objective setting) when the bounds cover 0 (effect not guaranteed essentially). This is an improvement over existing work in attempting to account for the presence of hidden confounding. Authors demonstrate their designed objective has has the same optimal as the original human-expert loss. Empirical evaluation on IHDP data demonstrate utility of their approach.

All reviewers found the work important. 

An important clarification raised was the comparison with Gao & Yin, which attempt to address a similar problem. However, the worst-case hidden confounding assumption under the MSM model is used on a IPW loss on the causal effect (as opposed to direct partial identification with a cost-sensitive loss). The rebuttal has provided a fairly detailed description of the similarities and differences. Authors have suggested they have not been able to reproduce the experimental claims on Gao & Yin and that the code is not available. 

Based on reviewer suggestions, authors have added a random baseline, and improved and clarified some specific issues. All things considered I am recommending an accept considering that reviewers found the work of value despite not being the first in addressing the problem, and the following updates for camera-ready:

1. Most importantly, authors should include all noted differences between Gao & Yin in the literature review, including modifying the claim made in the paper on novelty and highlight the complementary aspects of their approach. 
2. I strongly encourage the authors to include a concrete comparison with Gao & Yin on a simpler empirical setup where reproducing results is possible, and more importantly, attempt to reproduce their results in the actual setup used in the paper for empirical comparison. This is necessary for a fair comparison even if the authors were not aware of the work before the reviewers pointed it out. 
3. Reviewers have additionally pointed out several typos as well as some general presentation issues that I would like the authors to deliberately address for the camera-ready. 
4. Please also check the simulation setup and why a policy (CARED specifically) should not be sensitive to the strength of confounding and provide justification of the behavior in discussion of these empirical results.
Decision: Accept (poster)