Title: When to Act and When to Ask: Policy Learning With Deferral Under Hidden Confounding
Venue: NeurIPS 2024
Accepted: yes
Keywords: policy learning

causal inference

sensitivity analysis

human-algorithm collaboration
Abstract: We consider the task of learning how to act in collaboration with a human expert based on observational data. The task is motivated by high-stake scenarios such as healthcare and welfare where algorithmic action recommendations are made to a human expert, opening the option of deferring making a recommendation in cases where the human might act better on their own.
    This task is especially challenging when dealing with observational data, as using such data runs the risk of hidden confounders whose existence can lead to biased and harmful policies. However, unlike standard policy learning, the presence of a human expert can mitigate some of these risks. We build on the work of Mozannar and Sontag (2020) on consistent surrogate loss for learning with the option of deferral to an expert, where they solve a cost-sensitive supervised classification problem. Since we are solving a causal problem, where labels don’t exist, we use a causal model to learn costs which are robust to a bounded degree of hidden confounding.
    We prove that our approach can take advantage of the strengths of both the model and the expert to obtain a better policy than either. We demonstrate our results by conducting experiments on synthetic and semi-synthetic data and show the advantages of our method compared to baselines.

When to Act and When to Ask: Policy Learning With Deferral Under Hidden Confounding

Abstract

We consider the task of learning how to act in collaboration with a human expert based on observational data. The task is motivated by high-stake scenarios such as healthcare and welfare, where algorithmic action recommendations are made to a human expert, opening the option of deferring recommendation in cases where the human might act better on their own. This task is especially challenging when dealing with observational data, as using such data runs the risk of hidden confounders whose existence can lead to biased and harmful policies. However, unlike standard policy learning, the presence of a human expert can mitigate some of these risks. We build on the work of Mozannar and Sontag [2020] on consistent surrogate loss for learning with the option of deferral to an expert, where they solve a cost-sensitive supervised classification problem. Since we are solving a causal problem, where labels do not exist, we use a causal model to learn costs which are robust to a bounded degree of hidden confounding. We prove that our approach can take advantage of the strengths of both the model and the expert to obtain a better policy than either. We demonstrate our results by conducting experiments on synthetic and semi-synthetic data and show the advantages of our method compared to baselines.

1 Introduction

Machine learning models are increasingly being developed to perform tasks performed by human decision-makers in high-stakes settings such as clinical decision making [Adams et al., 2022, Rajpurkar et al., 2022], criminal justice [Stevenson and Doleac, 2022] and social services [Behncke et al., 2009, McBrien et al., 2022]. Some of these tasks involve recommending actions such as medical treatment, releasing on bail, or receiving benefits. Typically the data used to train such action-recommendation models is based on past human decisions and their outcomes. For example, we might observe how patients were treated for diabetes and their subsequent health outcomes.

Learning to act better in the future based on past observed actions is a causal problem. For such problems we face the fundamental problem of causal inference [Holland, 1986], i.e. the unknowability of counterfactual outcomes: 'Would this patient have been better had they been treated differently?'. Estimating causal quantities based on observational data such as hospital records is risky, as there is always the possibility of hidden confounding. Roughly speaking, this means there exist factors that affected the human decision maker and the outcome, but are unavailable to the model during training.

Learning models from data which has hidden confounding can lead to biased and harmful treatment assignment policies. However, human experts' decisions can also be biased, sub-optimal, or wrong. Thus, a setting in which both the human expert and the model can complement each other might be the best choice, mitigating the potential weaknesses and leveraging the strengths of both the model and the human expert [Bansal et al., 2020, Charusaie et al., 2022]. We believe this is especially pertinent in causal inference, as the human expert typically has access to the hidden confounders.

In this paper, we design a framework for learning a causal action recommendation model that can jointly work with a human decision-maker, while using observational data where hidden confounders might exist. We assume that the effect of the hidden confounders is limited (in a way we specify below), as without any assumption about the nature of hidden confounding, learning would be impossible. Our goal is to design a system consisting of a machine learning model estimating causal effects and a human expert working in a complementary setting. In addition to learning treatment assignment, the model learns the weaknesses and strengths of the human expert and will decide in each case whether to recommend a certain treatment to the patient, or whether it is better to defer the decision to the expert. The ultimate goal is to learn better policies and to reduce the burden on human experts. We call our method Causal Action Recommendation with Expert Deferral (CARED).

CARED follows in the footsteps of Mozannar and Sontag [2020], who developed a method for learning a classifier with the option of deferral to an expert. They propose a reduction of the problem to a cost-sensitive learning problem, where costs are based on the true labels of data samples. They give a surrogate loss for the cost-sensitive learning problem which generalizes the cross-entropy loss, and prove this loss is consistent, i.e., converges to the optimal solution of the original problem. Our problem can similarly be viewed as a classification problem with the option of deferring to an expert. However, the causal case is more difficult: the correct label for each sample is the best treatment to be prescribed to this sample based on its features, and due to the fundamental problem of causal inference we cannot know this label. Thus, unlike Mozannar and Sontag [2020] we do not have the true labels for our classification problem.

To overcome the challenges of the causal setting, we propose a set of costs that instead of being based on the true labels, are based on estimated bounds on counterfactual outcomes. These costs guide our model towards learning what is the right treatment for each patient while acknowledging that the human expert has access to additional information the model cannot access. We prove a generalization bound on the loss of the joint machine-expert system, and further prove that under certain assumptions about the model used to estimate the bounds mentioned above, the joint machineexpert system outperforms both the human expert and a pure machine learning model. Finally, we evaluate CARED on synthetic and semi-synthetic data to demonstrate how we can learn policies that outperform both pure machine learning policies and human experts. We further show that CARED outperforms a recently proposed method by Gao and Yin [2023] that addresses the same problem with an inverse-propensity weighted approach.

2 Related work

Many works focus on solving the problem of policy learning for action recommendations from observational data. Some notable approaches include reweighting by inverse propensity weighting (IPW) and other weighting techniques [Swaminathan and Joachims, 2015, Kallus, 2017, Beygelzimer and Langford, 2009], and the approach of using doubly robust scores to determine the optimal treatment assignment policy for binary treatments [Dudík et al., 2014, Athey and Wager, 2021,?, Kallus and Zhou, 2020]. Other methods predict the Conditional Average Treatment Effect (CATE) and use it as the guideline for treatment assignment for each sample, such as Jesson et al. [2021], and Kallus et al. [2019]. Most of these works assume ignorability, i.e., that there are no hidden confounders that affect both treatment assignment and the outcome in the data. As mentioned earlier, this assumption rarely holds when observational data is in use, and its presence, if not accounted for, can lead to biased and harmful policies. Our work builds on previous work for learning supervised classification problems with the deferral option by Mozannar and Sontag [2020], and is inspired by Athey and Wager [2021] who derive costs for learning policies from observational data under the assumption of no hidden confounders.

Gao and Yin [2023] present a framework for collaborative human-AI policy learning from observational data with deferral, building on earlier work [Gao et al., 2021] which did not allows for hidden confounding. To the best of our knowledge, theirs is the only existing method that learns a policy with the deferral option under hidden confounding. Their method minimizes an inverse propensity weighted estimator of the worst-case risk over a class of differentiable policies, and over an uncertainty set around the observed propensities. The uncertainty set is determined by constraints motivated by the Marginal Sensitivity Model [Tan, 2006]. While our method employs both outcome models and propensity scores, Gao and Yin [2023]'s approach focuses on propensity score re-weighting. The re-weighted objective implies that only cases where the proposed policy agrees to a high degree with the observed policy are taken into account. As we show in the experimental section

below, this (along with the lack of outcome model) might lead to under-performance, especially in the (common) case where the constraint set is not accurately known. Gao and Yin [2023] also explore the case where there are multiple specific human experts the model is optimizing for, which we plan to explore in future work. We also compare our method to Kallus and Zhou [2020], who take a similar approach as Gao and Yin [2023] for learning a policy while allowing the violation of the unconfoundedness assumption. However, they do not have the option of deferral.

In other related work, Stensrud et al. [2024] consider the case where the expert's action can be used as input to the method, motivated by the fact that the expert typically has access to unobserved confounders. This is distinct from our use case but has interesting implications in identifying the so-called 'superoptimal regime' where the expert's action can strictly improve over a policy derived purely from the observables. Finally, Yin et al. [2024] offer a novel approach towards learning to defer in the non-causal setting, which could be adapted to our use case in the future.

3 Setup

We work under an observational data setting with the Neyman-Rubin potential outcomes framework [Rubin, 2005]. Let ( X,A,Y (1) , Y (0) , U ) be a sample drawn from the unobservable distribution P full, where A ∈ A = { 0 , 1 } is a binary treatment, X ∈ X ⊂ R d is a set of baseline covariates, Y (1) and Y (0) are the real-valued treated and untreated potential outcomes, respectively, and U ∈ R k is an unobserved confounder. We face the fundamental problem of causal inference and only observe n draws from the coarsened distribution P over the observed variables Z = ( X,A,Y ) , where we assume that Y = Y ( A ) , i.e. (causal) consistency. We generally follow the convention that higher outcomes are better, we mention when assumed differently. We use the Marginal sensitivity Model (MSM) [Tan, 2006] as a way to model a limited degree of unobserved confounding. We are interested in learning a policy with the option of deferral, such that given the patient's covariates it either assigns a treatment or defers the decision to an expert.

Let e ( x ) = P ( A = 1 | X = x ) and e ( x, u ) = P full ( A = 1 | X = x, U = u ) be the observed and full propensity scores, under P and P full, the observed and the full unobserved distributions, respectively.

Assumption 1 (MSM Assumption) . We assume e ( x ) , e ( x, u ) ∈ (0 , 1) and that the ratio between the full odds of treatment e ( x, u ) / (1 -e ( x, u )) and the observed odds of treatment e ( x ) / (1 -e ( x )) is bounded by at most a factor of Λ ≥ 1 almost surely under P full :

Note that When Λ = 1 , Assumption 1 is equivalent to the classic assumption of unconfoundedness with respect to the observed X . As Λ increases, the MSM allows for greater levels of unobserved confounding. Setting Λ is a matter of ongoing research, and is typically done by calibrating with respect to observed confounders [McClean et al., 2024]. In our case, it is also tied to the fraction of deferrals to the human expert and can be tuned to achieve a desired deferral rate.

For a treatment a ∈ { 0 , 1 } and a covariate vector x ∈ X we define the Conditional Average Potential Outcome (CAPO) as E [ Y ( a ) | X = x ] . The Conditional Average Treatment Effect (CATE) is the difference τ ( x ) = E [ Y (1) -Y (0) | X = x ] . Let M (Λ) be the set of distributions ˜ P full that agree with the observed P on ( X,A,Y ) and that also agree with Assumption 1. Then Y + ( x, a ) = sup ˜ P full ∈M (Λ) E [ Y ( a ) | X = x ] is the so-called sharp upper bound on the CAPO E [ Y ( a ) | X = x ] , and similarly for Y -( x, a ) , taking an infimum instead. We further let ˆ Y + ( x, a ) , ˆ Y -( x, a ) be estimated upper and lower bound for the corresponding CAPO functions.

A policy with deferral is a function π : X → { 0 , 1 , ⊥} that maps covariates to a possible action, or defers the decision to an expert, where ⊥ denotes deferral. A policy π can be assessed by its policy value V ( π ) = E [ Y ( π )] , where higher policy values are related to better policies unless stated otherwise. The action when the algorithm chooses to defer is the action the human expert has taken, see discussion of this in the limitations subsection (8). The learned policies will be assessed relative to a baseline policy value, such as the policy value based solely on the human expert. Our goal is for CARED to learn a policy that does no worse than the baseline policy and hopefully outperforms it.

4 CAPO-Based Policies

We now present two baseline policies that can be defined based on the CAPO bounds.

The first is Bounds Policy Equation (1). This policy assigns treatment if the upper and lower bounds on the CATE have the same sign, and otherwise it defers, i.e. it defers if the CATE interval crosses 0. This approach was used in previous work, e.g. [Jesson et al., 2021, Oprescu et al., 2023]. The second is the Pessimistic Policy Equation (2). It is pessimistic in the sense that it does not trust the human expert, so it does not allow deferral to an expert. It agrees with the Bounds Policy in the cases where that policy would not defer, and in the cases where the Bounds Policy would defer it decides based on the lower bounds, which (assuming higher outcomes are better) are indeed pessimistic.

Let ˆ Q ( x ) = ( ˆ Y + ( x, 0) , ˆ Y -( x, 0) , ˆ Y + ( x, 1)) , ˆ Y -( x, 1)) be the CAPO bounds estimates for a sample ( X = x, A = a, Y = y ) . We further assume that the expert policy is reflected in the training data in the sense that their policy is π exp ( x i ) = a i . Then, given a model that supplies CAPO bounds ˆ Q , we define the following policies, which we consider as baselines henceforth:

Bounds Policy:

Pessimistic Policy:

The above policies, while accounting for hidden confounding, even when allowing deferral to an expert, do not learn the strengths and weaknesses of the expert, so they might not be the optimal policies. We show this theoretically in Section 6.2, and empirically in Section 7.

5 Method

In this section, we present our method in which we design a machine-expert system for learning a policy with the option of deferral to an expert under hidden confounding. In 5.1 we describe the joint machine-expert objective we are interested in optimizing, the challenges it imposes, and how to optimize a consistent surrogate cost-sensitive loss that converges to the optimal solution of the original objective, based on Mozannar and Sontag [2020]. Then in 5.2 we introduce our proposed set of costs based on the CAPO bounds for solving this cost-sensitive problem. Lastly, we explain the step of CAPO estimation in 5.3. Our algorithm is summarized below in Algorithm 1.

5.1 Joint machine-expert objective function

We design a joint machine-expert system, where we aim to learn a policy π : X → A ∪ {⊥} . We denote by m ∈ A the expert's action that is assumed to be drawn from the distribution M | X = x, U = u . Note that the expert might have access to additional information - the hidden confounder U , which is unavailable to the model. We cast our problem as a cost-sensitive optimization objective

̸

where C ( x, a ) is the cost incurred by the system if for a sample with covariates x , an action/treatment a was chosen by the model, and C ⊥ ( x, m, y ) is the cost incurred when action m was chosen for sample x by the expert. We explain below why C ⊥ depends on y but C does not.

In the case of no unobserved confounding and no deferral, Athey and Wager [2021] have shown how the costs above can be set such that the cost minimizer is the policy with optimal policy value. We describe how we set costs that address both hidden confounding and deferral in the next subsection.

The above objective is non-convex and difficult to optimize. We deal with this challenge by building on the approach of Mozannar and Sontag [2020] for learning classifiers with the option of deferral to

an expert, where they have a similar objective as in Equation (3). They give a convex and consistent surrogate loss for the cost-sensitive learning problem, which is a weighted cross-entropy loss, where the weights are based on a set of costs they build using the true labels for the classification problem.

As explained earlier, our problem can also be viewed as a classification problem with the option of deferring to an expert. However, due to the fundamental problem of causal inference [Holland, 1986] we do not have the true labels, which are in our case, the best treatment to be prescribed to this sample based on its features, which is a challenge that we deal with in Section 5.2.

Following is the surrogate loss we optimize, in Section 5.2 we present our proposed set of costs, and in Section 6 we give consistency and generalization guarantees for this surrogate loss.

Let π i : X → R be the raw output of the policy π corresponding to a class i ∈ { 0 , 1 , ⊥} , and define π ( x ) = argmax i ∈{ 0 , 1 , ⊥} π i ( x ) . Let z = ( x, a, y ) be a sample, and ˆ Q ( x ) = ( ˆ Y + ( x, 0) , ˆ Y -( x, 0) , ˆ Y + ( x, 1)) , ˆ Y -( x, 1)) the CAPO bounds. These bounds are then used to define the scores c (0) = C ( x, 0) , c (1) = C ( x, 1) , and c ( ⊥ ) = C ⊥ ( x, m, y ) as we show in Section 5.2 below. Define w j ( z, ˆ Q ( x )) = max k ∈{ 0 , 1 , ⊥} c ( k ) -c ( j ) . Then the surrogate loss function for Equation (3) is given by:

The method we propose uses the above system loss for learning a policy from observational data with the presence of a limited degree of hidden confounding and the ability to defer. The difficulty in this case is constructing the costs C ( x, 0) , C ( x, 1) and C ⊥ ( x, m, y ) which are used to derive the weights w j , since ground truth labels are never available - a challenge we address now.

5.2 Action Costs

As mentioned earlier, our classification problem does not have the true labels, as opposed to Mozannar and Sontag [2020]. We therefore propose a set of costs based on estimated bounds on counterfactual outcomes rather than on the true labels, guiding our model in learning for which cases it can safely recommend actions, and which to defer to the human expert. These costs are then used to derive the weights in the objective Equation (4).

For i ∈ [ n ] let ( x i , y i , a i ) be the i -th observed sample, and let ˆ Y + ( x i , a ) , ˆ Y -( x i , a ) be the estimates of the upper and lower bounds of the CAPO for an action a ∈ A , and a covariates x ∈ X . We propose setting the cost of assigning the action a to the i -th sample to be:

This cost encourages the model to choose the action with the highest outcome. When this is not the case, i.e. the model chooses another action than the one with the highest estimated outcome, then it will incur a cost which is the difference between the lower bound of the outcome corresponding to the action chosen, and the upper bound of the outcome corresponding to the action which was supposed to be chosen based on the outcomes. This cost can be thought of as the worst-case regret of the model relative to the optimal action. It is important to note that this is only one option among many options for building a set of costs for this purpose. For instance, we can switch the roles of the upper and lower bounds of the CAPOs to obtain various costs expressing differing levels of risk-aversion.

As for the cost of deferring to an expert, following the same logic, we propose two alternatives. Given covariates x i , expert action m = a i and outcome y i the conservative deferral cost is:

and similarly the optimistic deferral cost is: C ⊥ ( x i , a i , y i ) = ˆ Y + ( x i , 1 -a i ) -y i . These costs represent to what extent are we willing to take an action different from the one the expert took which resulted in the outcome actually seen in the data, i.e. m = a i which lead to outcome y i . In the optimistic case, deferring to an expert results in incurring a greater cost in comparison to its conservative counterpart, encouraging the model to make fewer deferrals and focusing on samples it is highly uncertain about. It is optimistic in the sense of assuming the model's estimates are likely

correct. Choosing between these alternatives should to be done based on the characteristics of the specific policy and use case. In Appendix A.3 we show a comparative analysis of the two alternatives.

To summarize we present here the costs corresponding to the conservative approach:

Taken together, these costs encourage the model to classify a sample as a = 1 or a = 0 in cases where it is certain that particular action would be best, and to defer cases where the expert seems to have made the correct decision (in the conservative case). An illustrative example of the costs choice can be found in Appendix A. Additionally, we demonstrate costs' coherence in Theorem 1.

5.3 CAPO Estimation

From an algorithmic point of view, any method that yields upper and lower bounds on the CAPO given some degree of hidden confounding can be used to obtain the costs Section 5.2. As we will see in the next section, our theoretical results require bounds with certain generalization and validity properties to hold. In this work, we use the B-learner [Oprescu et al., 2023] for estimating the upper and lower bounds of the CAPO, as it has the properties needed for the theoretical analysis and shows good performance in practice, and being a meta-learner it can accommodate various base learners, including random forests and neural networks.

Algorithm 1 CARED Policy Learner - The Conservative Approach

input Data { ( x i , a i , y i ) : i ∈ { 1 , ..., n }}

1: Use data { ( x i , a i , y i ) : i ∈ { 1 , ..., n }} to learn the CAPO bounds:

//e.g.

using B-learner

2023]

2: Construct the conservative costs based on the CAPO bounds:

3: Calculate weights w j i = max k ∈{ 0 , 1 , ⊥} c i ( k ) -c i ( j )

4: Learn a policy: π ∈ arg min π ′ ∈ Π 1 n ∑ n i =1 ∑ j ∈{ 0 , 1 , ⊥} -w j i log ( exp( π ′ j ( x i )) ∑ k ∈{ 0 , 1 , ⊥} exp( π ′ k ( x i )) ) . output π

6 Theoretical Guarantees

We present theoretical guarantees for CARED as follows: In 6.1 we show that the optimum of the surrogate loss function L CE (eq. 4) agrees with the optimum of the machine-expert loss function L (eq. 3). Then, in 6.2 we show that the costs we use in L CE are coherent, in the sense that minimizing them leads to a decision that is non-inferior to the decision either the expert or the machine would have made on their own. Finally, in 6.3 we give a generalization bound for the loss L CE .

6.1 Consistency

Corollary 1. L is convex in π and is a consistent loss function for L

CE : ; ˆ .

This result is a straightforward adaptation of Proposition 1 of Mozannar and Sontag [2020], as their result is not sensitive to the particular choice of costs (B.0.1). It motivates using the surrogate loss function L CE which is much more amenable to optimization than the original machine-expert loss L .

6.2 Costs Are Coherent

We now show that the costs C ( x, 0) , C ( x, 1) , C ⊥ ( x, a, y ) are coherent, in the sense that they indeed work as intended: a policy that minimizes them locally is always at least as good as (and indeed, in general better than) both the expert on their own and the machine on its own.

Definition 1. [Bound Validity] For a sample x with corresponding potential outcomes Y (0) and Y (1) , let ˆ Q ( x ) = ( ˆ Y + ( x, 0) , ˆ Y -( x, 0) , ˆ Y + ( x, 1)) , ˆ Y -( x, 1)) be the estimated CAPO bounds. Then ˆ Q is valid for ( Y (0) , Y (1)) if Y ( a ) ∈ [ ˆ Y -( x, a ) , ˆ Y + ( x, a )] for a ∈ { 0 , 1 } .

Validity means that bounds indeed contain their respective potential outcomes. For example, Oprescu et al. [2023] prove that their bounds are valid on average.

Theorem 1 (Costs are coherent) . Let ˜ π ( x i ) ∈ arg min L CE ( π, z i ; ˆ Q ( x i )) , π exp the expert's policy, and π ˆ Q bounds ( x i ) the CAPO-based policy defined in 1. If ˆ Q ( x i ) is valid for ( Y i (0) , Y i (1)) , then Y (˜ π ( x i )) ≥ max { Y ( π exp ( x i )) , Y ( π ˆ Q bounds ( x i ) )} .

Furthermore, under certain technical conditions on the distribution of Y (0) , Y (1) and ˆ Q , the inequality is strong with non-zero probability for each sample.

Theorem 1 shows that whenever the bounds include the true potential outcome, the action that minimizes our proposed loss function is at least as good as the action implied by the baseline π ˆ Q bounds policy, as well as the human expert policy.

6.3 Generalization Bound

Wenowshow the generalization bound for our machine-expert system loss. We start with assumptions, then we state the main theorem.

Assumption 2 (Policy learners) . Let Π be the class of policies over which we optimize Equation (4). We assume the class Π is with restricted complexity, specifically R n (Π) = O ( 1 √ n ) , where R n (Π) is the Rademacher Complexity of the policy class Π . Classes that have this property include linear functions, logistic functions, decision trees with a bounded depth, and neural networks with weight decay or dropout Kallus and Zhou [2020].

Assumption 3 (Policy Learner with Bounded Outputs) . Let π ∈ Π and π j : X → R be the raw output of the classifier π corresponding to a class/action j ∈ { 0 , 1 , ⊥} . We assume there exists a constant C π such that | π j | ≤ C π for all j .

Assumption 4 (Boundedness of the Outcomes) . Y is bounded, i.e. | Y | ≤ C Y for C Y > 0 .

Assumption 5. [Rates for ERM CAPO Bounds Estimators] The CAPO bounds estimators ˆ Y ⋄ ( x, a ) for a ∈ { 0 , 1 } and ⋄ ∈ { + , -} satisfy: ∥ ˆ Y ⋄ ( x, a ) -Y ⋄ ( x, a ) ∥ ≲ O p ( n -1 / (2+ r ) ) with 0 < r < 2 .

Assumption 5 implies that the convergence rate of the L 2 norm of the estimation error of the CAPO bounds converges not too slowly (in probability).Notably, the B-Learner bound estimates satisfy Assumption 5 under some assumption on the class of policy learners and the nuisance estimators, according to Corollary 1 from Oprescu et al. [2023].

We now give a generalization bound on the loss of the joint machine-expert system:

Theorem 2 (Generalization Bound) . Given a policy class Π satisfying Assumption 2, Assumption 3 with a constant C π , Assumption 4 with a constant C Y , and Assumption 5 with 0 < r < 2 .

Let π ∈ Π , be a policy, and let Q, ˆ Q be the CAPO bounds and the estimated CAPO bounds respectively. Then there exists a constant C > 0 such that with a probability of at least 1 -δ the following holds:

where L S ( π ; ˆ Q ) = 1 n ∑ n i =1 L CE ( π, z i ; ˆ Q ( x i )) is the training loss, and L D ( π ; Q ) = E z ∼ p ( z ) [ L CE ( π, z ; Q ( x ))] is the expected loss for the combined machine-expert system loss in 4, and R n (Π) is the Rademacher Complexity of the policy class Π .

7 Experiments

Here we examine the utility of CARED by conducting experiments on synthetic and semi-synthetic data in Section 7.1 and Section 7.2, respectively. Further details about the experiments, datasets, models, and hyper-parameters can be found in Appendix C.

Our main comparison is to Gao and Yin [2023]'s method (ConfHAI) that allows for a bounded degree of confounding in addition to allowing the option of deferral to a human expert. They learn a policy by optimizing a minimax reweighting-based risk estimate over an uncertainty set around the observed propensities. The uncertainty sets are determined by the MSM assumption, which is the same assumption we employ to bound the degree of hidden confounding. Another baseline we consider is Kallus and Zhou [2020]'s method (CRLogit) which proceeds under a similar approach and allows for a bounded degree of confounding, but does not allow deferral to a human. Additionally, we evaluate all methods against the following baselines: Oracle Policy which is the best policy that assigns for each patient the true best treatment. In both experiments, this policy is available to us, as we have the true potential outcomes. Current Expert/ Baseline Policy this is the default policy that we refer to in cases of deferral. Typically, it is the policy of the current expert, but it can be any other policy of our choice. Pessimistic Policy : Using the Equation (2) policy with the ˆ ˆ Q policy bounds Q given by the B-learner Oprescu et al. [2023]. B-Learner Policy : Using the π bounds of Equation (1) with the bounds ˆ Q given by the B-learner [Oprescu et al., 2023]. We use the exact same ˆ Q in Algorithm 1, and in the B-learner and pessimistic policies. Random Deferral Policy : this is a variant of the B-Learner policy which introduces a different approach to deferral, where samples are deferred randomly based on a specified deferral rate.

7.1 Synthetic data

In this experiment, we replicate the study from Gao and Yin [2023] using synthetic data, comparing our method to ConfHAI and CRLogit . Since this experiment is measuting regret, we follow Gao et al. [2021] and compare to a Baseline Policy that assigns treatment a = 0 for all patients and is denoted by π 0 . In this experiment lower outcomes are better.

Data The data is generated according to the following data generation process:

where β 0 = [0 , 0 . 5 , -0 . 5 , 0 , 0] , β treat = [ -1 . 5 , 1 , -1 . 5 , 1 , 0 . 5] , µ x = [ -1 , 0 . 5 , -1 , 0 , -1] , η = 2 . 5 , α = -2 , ω = 1 . 5 , and ϵ ∼ N (0 , 1) . The nominal propensity is logistic by X , e ( X ) = σ ( β ⊤ X ) with β = [0 , 0 . 75 , -0 . 5 , 0 , -1 , 0] . The confounder is denoted by U , and the true propensity score is given by: e ( X,U ) = (Λ 0 U +1 -U ) e ( X ) [1+2(Λ 0 -1) e ( X ) -Λ 0 ] U +Λ 0 +(1 -Λ 0 ) e ( X ) , with the true Λ 0 , such that log (Λ 0 ) = 2 . 5 .

Experiment We replicate the experiment from Gao and Yin [2023] and run 10 trials, with different instances of the above data, a train data size of 2000 , and test data of size 10000 . For each trial, we vary the sensitivity parameter Λ in { 0 . 01 , 0 . 5 , 1 , 1 . 5 , 2 , 2 . 5 , 3 , 3 . 5 , 4 } , corresponding to various levels of assumed hidden confounding. We compare the policy regret for the returned policy for each method relative to the Baseline Policy . In addition, we report the policy regret for the human expert in the dataset (Human's Policy) which is reflected by the variable A in the dataset. As for the CARED policy, we obtain it by applying Algorithm 1 with a logistic policy implemented as a single-layer MLP network.

Results In Figure 1 we see that CARED outperforms all other methods, specifically improving over the expert for all Λ values. In contrast, the ConfHAI and CRLogit improve over the expert for a limited range of values of Λ , which is for a range around the true value of the sensitivity parameter Λ 0 showed in the plot, and perform noticeably worse than the expert for many Λ values when the Λ is mis-specified. In contrast, CARED shows robustness to all Λ values, making it a safer choice, as correctly specifying Λ is challenging [McClean et al., 2024].

7.2 IHDP Hidden Confounding

In this experiment, we aim to demonstrate how our method can adapt to the human expert using a semi-synthetic dataset: the IHDP hidden confounding dataset.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

log(

) uncertainty parameter

1.25

1.00

0.75

0.50

0.25

0.00

0.25

0.50

Policy Regret

Human's Policy

CRLogit Policy

ConfHAI Policy

CARED Policy

Pessimistic Policy

B-Learner Policy

Oracle Policy

True log(

)

Dataset The hidden-confounding version of the IHDP dataset [Hill, 2011] was introduced by Jesson et al. [2021]. The Infant Health and Development Program (IHDP) dataset [Hill, 2011] is a dataset consisting of real covariates and a treatment that were collected from an RCT that targeted low-birthweight, premature infants. The treatment was providing both intensive high-quality child care and home visits from a trained provider, while the outcomes were simulated according to the response surface B described by Hill [2011]. Jesson et al. [2021] induced hidden confounding onto the dataset by hiding the x 9 covariate; however, the response surface B from Hill [2011] is still used to generate the observed outcomes. In this dataset higher outcomes are better, thus we build the loss function using the appropriate set of costs for this assumption.

Semi-Synthetic Expert (Current Expert) We design a semi-synthetic expert, based on the original observed expert's policy, and the Oracle policy. The new semi-synthetic expert is designed as follows: when the feature x 17 -'worked during pregnancy' receives the value 1 , then the new expert is identical to the Oracle policy, and otherwise, it is identical to the original expert. This way, the new expert is perfect when x 17 , which happens in probability 0 . 59 . See details in Appendix C.2.1.

Experiment We conduct this experiment on the modified IHDP Hidden Confounding consisting of tuples ( X,A ′ , Y ′ ) , where the A ′ is the new expert, and the Y ′ is the outcome corresponding to the expert A . We generate 1000 realizations of the dataset. For each realization, we train the policy model for different values of the causal uncertainty parameter Λ and calculate: the rate of the samples deferred to the expert, and the policy value of the learned policy. We then plot the average policy value per causal uncertainty level - the parameter Λ in Figure 2a, and the average policy value per average deferral rate over all trials in Figure 2b. As for our method, we obtain our policy by applying Algorithm 1 with a logistic policy implemented as a single-layer MLP network.

Results In Figure 2a We observe that CARED consistently improves upon the expert's policy, yielding a higher average policy value across all levels of the causal uncertainty parameter, Λ . Notably, it outperforms the baseline methods for reasonable values of Λ , and achieves a policy value closest to that of the optimal Oracle Policy. As in the previous experiment, CARED demonstrates greater robustness to variations in the assumed level of confounding. For high Λ values, CARED's performance trends toward that of the expert policy, which is expected, as larger Λ values result in less informative and thus less reliable CAPO intervals. In contrast, ConfHAI only begins to improve when the assumed confounding level closely matches the true sensitivity parameter.

These results highlight the robustness and reliability of our policy, showing that it performs well even when the sensitivity parameter is misspecified by the data analyst. Moreover, when the sensitivity parameter reaches very high levels, CARED avoids unnecessary risk by converging towards the expert policy, reflecting its conservative response to high causal uncertainty.

In Figure 2b we compare our method only to baselines that incorporate deferral. Here, instead of regret, we assess the policies by their average policy value across all trials, and we plot the performance according to the average deferral rate over all trials. Our results show that our policy consistently outperforms the other methods at equivalent deferral rates.

The peak of the plot of the CARED Policy is when both the model and the expert work in collaboration and take advantage of the strengths of both the ML model and the expert. This happens at a deferral rate that is close to ∼ 0 . 6 , which is almost precisely the percentage of the cases where the expert does as well as the oracle policy according to how the expert was designed. This indicates that our model learns the strength of the expert, and knows when to defer to the expert. While ConfHAI defers only a small fraction of samples on average, which might indicate that this method does not learn the weaknesses and strengths of the expert.

10

1

10 0

log(

) uncertainty parameter

13.5

14.0

14.5

15.0

15.5

16.0

Policy Value

Expert Policy

CRLogit Policy

CONfHAI Policy

CARED Policy

Pessimistic Policy

B-Learner Policy

Oracle Policy

True log(

)

0.0

0.2

0.4

0.6

0.8

1.0

Deferral Rate

13.5

14.0

14.5

15.0

15.5

16.0

Policy Value

Expert Policy

CONFHAI Policy

CARED Policy

B-Learner Policy

Random Deferral Policy

Oracle Policy

(a) Policy Value for different values of

Λ

(b) Policy Value for different levels of deferral

8 Discussion

In this paper we proposed CARED: a method for learning policies from observational data where the model can recommend a treatment or defer to an expert. When learning to act from observational data which includes experts' actions, hidden confounders are by necessity factors that influenced the experts' decisions, and are thus available to them even though they are unavailable to the model. This makes our setting pertinent to the problem of safely learning to recommend actions based on observational data where the actions were taken by human experts, as is the case in many medical and legal settings, for example. CARED thus mitigates some of the risk of learning causal models from observational data. A further advantage is that we might not need to know the true Λ for the system to be useful: instead we might wish to calibrate the rate of deferral instead, as that might be the more practical constraint the system faces, in terms of human labor vs. the joint system's policy value.

We showed both theoretically and by experiments in synthetic and semi-synthetic data that our method outperforms relevant baselines for this task, and can combine in a synergistic manner the expert's and machine learning model's capabilities.

Limitations and future work Our current method requires access to the experts actions in the observational data; it cannot accommodate directly a different expert, since we cannot know the potential outcomes corresponding to how that new expert would have acted. A further limitation is the assumption that at test time the experts would behave the same for the deferred cases as they would have before system deployment. Realistically, deploying an action recommendation system might change the experts' behavior more broadly. Accounting for this would require testing and modeling the experts behavior in such conditions. Future work will explore this more dynamic setting, taking into account the ongoing interactions and learning between the human expert and the system.

9 Acknowledgments

We would like to express our gratitude to our colleague, Rom Gutman, for his valuable insights and support throughout this work. We also thank Angela Zhou and Ruijiang Gao whose methods we replicated and compared in our evaluations, for their their helpful responses to our questions about implementing their methods. Additionally, we extend our appreciation to Hussein Mozannar for his generous assistance in clarifying aspects of his work, which served as the basis for our approach. We would like to thank the anonymous reviewers for useful discussion and feedback. MG and US were supported by ISF grant 2456/23.

References

Adams, K. E. Henry, A. Sridharan, H. Soleimani, A. Zhan, N. Rawat, L. Johnson, D. N. Hager, S. E. Cosgrove, A. Markowski, et al. Prospective, multi-site study of patient outcomes after implementation of the trews machine learning-based early warning system for sepsis. Nature medicine , 28(7):1455-1460, 2022.

Athey and S. Wager. Policy learning with observational data. Econometrica , 89(1):133-161, 2021.

Bansal, B. Nushi, E. Kamar, E. Horvitz, and D. S. Weld. Optimizing AI for teamwork. arXiv preprint arXiv:2004.13102 , 2020.

Behncke, M. Frölich, and M. Lechner. Targeting labour market programmes-results from a randomized experiment. Swiss Journal of Economics and Statistics , 145(3):221-268, 2009.

Beygelzimer and J. Langford. The offset tree for learning with partial labels. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 129-138, 2009.

Brooks-Gunn, F.-r. Liaw, and P. K. Klebanov. Effects of early intervention on cognitive function of low birth weight preterm infants. The Journal of pediatrics , 120(3):350-359, 1992.

M.-A. Charusaie, H. Mozannar, D. Sontag, and S. Samadi. Sample efficient learning of predictors that complement humans. In International Conference on Machine Learning , pages 2972-3005. PMLR, 2022.

Curth, D. Svensson, J. Weatherall, and M. van der Schaar. Really doing great at estimating cate? a critical look at ml benchmarking practices in treatment effect estimation. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 2) , 2021.

Dudík, D. Erhan, J. Langford, and L. Li. Doubly robust policy evaluation and optimization. 2014.

Gao and M. Yin. Confounding-robust policy improvement with human-ai teams. arXiv preprint arXiv:2310.08824 , 2023.

Gao, M. Saar-Tsechansky, M. De-Arteaga, L. Han, M. K. Lee, and M. Lease. Human-ai collaboration with bandit feedback. In Z.-H. Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 1722-1728. International Joint Conferences on Artificial Intelligence Organization, 8 2021.

L. Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics , 20(1):217-240, 2011.

W. Holland. Statistics and causal inference. Journal of the American statistical Association , 81(396):945-960, 1986.

Jesson, S. Mindermann, Y. Gal, and U. Shalit. Quantifying ignorance in individual-level causal-effect estimates under hidden confounding. In International Conference on Machine Learning , pages 4829-4838. PMLR, 2021.

Kallus. Recursive partitioning for personalization using observational data. In International conference on machine learning , pages 1789-1798. PMLR, 2017.

Kallus and A. Zhou. Minimax-optimal policy learning under unobserved confounding. Management Science 67(5):2870-2890 , 2020.

Kallus, X. Mao, and A. Zhou. Interval estimation of individual-level causal effects under unobserved confounding. In Proceedings of the 22 nd International Conference on Aritificial Intelligence and Statistics (AISTATS) 2019 , volume 89, 2019.

McBrien, B. Winters, E. Zhou, and V. Eubanks. Screened & scored in the District of Columbia, November 2022. URL https://epic.org/screened-scored-in-dc/ . [ https://epic.org/ screened-scored-in-dc/ ; posted November-2022].

McClean, Z. Branson, and E. H. Kennedy. Calibrated sensitivity models. arXiv preprint arXiv:2405.08738 , 2024.

Mozannar and D. Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning , pages 7076-7087. PMLR, 2020.

Oprescu, J. Dorn, M. Ghoummaid, A. Jesson, N. Kallus, and U. Shalit. B-learner: Quasi-oracle bounds on heterogeneous causal effects under hidden confounding. In Proceedings of the 40th International Conference on Machine Learning , pages 26599-26618. PMLR, 2023.

Rajpurkar, E. Chen, O. Banerjee, and E. J. Topol. AI in health and medicine. Nature medicine , 28(1):31-38, 2022.

B. Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association , 100(469):322-331, 2005.

Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

J. Stensrud, J. Laurendeau, and A. L. Sarvet. Optimal regimes for algorithm-assisted human decision-making. Biometrika , page asae016, 2024.

T. Stevenson and J. L. Doleac. Algorithmic risk assessment in the hands of humans. Available at SSRN 3489440 , 2022.

Swaminathan and T. Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In International Conference on Machine Learning , pages 814-823. PMLR, 2015.

Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association , 101(476):1619-1637, 2006.

Yin, J.-F. Ton, R. Guo, Y. Yao, M. Liu, and Y. Liu. Fair classifiers that abstain without harm. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum? id=jvveGAbkVx .

A Action Costs

In this section, we present examples that illustrate our choice of the costs. Then we prove the consistency of our costs for some base cases, that is, we show that the lowest cost corresponds to the right action for our choice for the set of costs.

Let z = ( x, a, y ) be a sample, and assume Y (0) > Y (1) w.l.o.g, that is, the right treatment is A = 1 . ˆ Y + ( x, 0) , ˆ Y -( x, 0) , ˆ Y + ( x, 1) , ˆ Y -( x, 1) are the CAPOs corresponding to this sample.

We recall our proposed costs:

where C cons ⊥ ( x, a, y ) , and C opt ⊥ ( x, a, y ) correspond to the conservative and optimistic deferral costs respectively.

A.1 No overlap between CAPOs intervals

Example In this example, we show a case where the CAPOs intervals don't overlap, we provide a visual diagram for this example in Figure 3a, where we let ˆ Y + ( x, 0) = 4 , ˆ Y -( x, 0) = 1 , ˆ Y + ( x, 1) = 9 , ˆ Y -( x, 1) = 5 , and Y (0) = 2 , Y (1) = 6 . Then we have that:

As for the deferral cost, there are two possible cases: if the expert is right, meaning a = 1 , y = Y (1) , the deferral costs are:

for both alternatives, we have the lowest costs corresponding to the deferral cost, which means that we will guide the model to choose the expert's decision, which is in this case, the right treatment.

On the other hand, when the expert is wrong, i.e. a = 0 , y = Y (0) , the deferral costs are:

Where for both alternatives, the lowest cost is the cost of the treatment a = 1 , which is the right treatment.

Proof of the general case We show a visualization for the general case where the CAPOs intervals don't overlap in Figure 3b. In the general case we have that:

As for the deferral cost, when expert is right, meaning a = 1 , y = Y (1) , the deferral costs are:

for both alternatives, we have the lowest costs corresponding to the deferral cost, which means that we will guide the model to choose the expert's decision, which is in this case, the right treatment.

On the other hand, when the expert is wrong, i.e. a = 0 , y = Y (0) , the deferral costs are:

Where for both alternatives, the lowest cost is the cost of the treatment a = 1 , which is the right treatment.

(a) Example

Y-(x,1) =5

Y+(x,1)=9

（Y(1)=6

Y-(x,0)=1

Y+(x,0)=4

Y(0)=2

5

Y-(x,1)

y+ (x,1)

Y(1)

Y-(x,0)

r+(x,0)

Y(0)

(b) The general case

A.2 CAPOs intervals overlap

We now focus on the cases where the CAPOs intervals overlap. In Figure 4 we present two examples for cases where the CAPOs intervals overlap and show how this affects the costs.

Example 1. In the example presented in Figure 4a, where we let ˆ Y + ( x, 0) = 5 , ˆ Y -( x, 0) = 1 , ˆ Y + ( x, 1) = 9 , ˆ Y -( x, 1) = 4 , and Y (0) = 2 , Y (1) = 6 . Then we have that:

As for the deferral cost, there are two possible cases: if the expert is right, meaning a = 1 , y = Y (1) , the deferral costs are:

for both alternatives, we have the lowest costs corresponding to the deferral cost, which means that we will guide the model to choose the expert's decision, which is in this case, the right treatment.

On the other hand, when the expert is wrong, i.e. a i = 0 , y i = Y (0) , the deferral costs are:

Where for both alternatives, the lowest cost is the cost of the treatment a = 1 , which is the right treatment.

Example 2. In the example presented in Figure 4b, we show an interesting case, where although Y (1) > Y (0) , the value of Y (1) lies into the intersection of the two intervals. We let ˆ Y + ( x, 0) = 5 , ˆ Y -( x, 0) = 1 , ˆ Y + ( x, 1) = 9 , ˆ Y -( x, 1) = 4 , and Y (0) = 2 , Y (1) = 4 . 5 . Then we have that:

As for the deferral cost, there are two possible cases: if the expert is right, meaning a = 1 , y = Y (1) , the deferral costs are:

for both alternatives, we have the lowest costs corresponding to the deferral cost, which means that we will guide the model to choose the expert's decision, which is in this case, the right treatment.

On the other hand, when the expert is wrong, i.e. a i = 0 , y i = Y (0) , the deferral costs are:

Where for both alternatives, the lowest cost is the cost of the treatment a = 1 , which is the right treatment.

(a)

(b)

Y-(x,1)= 4

Y(1) =6

Y-(x,0) =1

y+(x,0)=5

Y(0)=2

Y-(x,1) = 4

Y+(x1)=9

cY(1)=4.5

Y-(x,0)=1

Y+(x,0)=5

Y(0)=2

A.3 A Comparative Analysis of the Conservative and Optimistic Costs

We provide an analysis that highlights the scenarios where each of the conservative and optimistic approaches proves superior to the other.

For a sample ( x, a, y, Y (0) , Y (1)) ∼ P full, where w.l.o.g it holds that Y (1) > Y (0) , that is, the right treatment for this sample is 1 .

We distinguish between two main cases:

The expert is right :

When the expert is right, i.e. π exp ( x ) = 1 , and Y ( π exp ( x i )) = Y (1) , then,a policy with this set of costs will predict the wrong treatment ˜ π ( x ) = 0 when the following condition holds:

For the conservative approach , this holds when:

As for the the optimistic approach , this holds when:

When Equation (7) holds and Equation (8) does not hold, these are the cases where the optimistic approach outperforms the conservative approach, i.e.:

We note that Equation (14) never holds, and thus when the expert is right, the optimistic approach cannot outperform its conservative counterpart.

On the other hand, when Equation (8) holds and Equation (7) does not hold, these are the cases where the conservative approach outperforms the optimistic approach, i.e.:

The expert is wrong

When the expert is wrong, i.e. π exp ( x ) = 0 , and Y ( π exp ( x i )) = Y (0) , then, a policy with this set of costs will predict the wrong treatment ˜ π ( x ) = 0 when the following condition holds:

For the conservative approach , this holds when:

As for the the optimistic approach , this holds when:

When Equation (12) holds and Equation (13) does not hold, these are the cases where the optimistic approach outperforms the conservative approach, i.e.:

On the other hand, when Equation (13) holds and Equation (12) does not hold, these are the cases where the conservative approach outperforms the optimistic approach, i.e.:

We note that Equation (14) never holds, and thus when the expert is wrong, the conservative approach cannot outperform its optimistic counterpart.

B Proof of Main Theorems

In this section, we provide the proof of our main theorems.

B.0.1 Consistency

Proof of Corollary 1. we can apply Proposition 1 from Mozannar and Sontag [2020] directly to our setting yielding the statement of our Corollary.

Note that we used a slightly different statement of Proposition 1 from Mozannar and Sontag [2020] from readability considerations and consistency with our setup and problem formulation.

The formulation of the original statement using our notation is as follows:

Corollary 2. L CE is convex in π and is a consistent loss function for L : Let ˜ π = arg inf π E [ L CE ( π, z ; ˆ Q )] , then arg max i ∈{ 0 , 1 , ⊥} ˜ π i = arg min i ∈{ 0 , 1 , ⊥} E [ c ( i ) | Z = z ] .

We recall that the surrogate loss L CE [4] is defined for policies π i : X → R be the raw output of the policy π corresponding to a class i ∈ { 0 , 1 , ⊥} , such that π ( x ) = arg max i ∈{ 0 , 1 , ⊥} π i ( x ) . Therefore, this holds especially for ˜ π = arg inf π E [ L CE ( π, z ; ˆ Q )] , i.e.

As for optimizing the original loss function L ( π )[ 3], the optimization problem is given by,

according to Mozannar and Sontag [2020], this problem can be solved as a cost-sensitive problem with the costs c (0) = C ( x, 0) , c (1) = C ( x, 1) , and c ( ⊥ ) = C ⊥ ( x, a, y ) for each sample ( x, a, y ) , i.e., the solution of this optimal optimization problem satisfies the following for each sample z : ˙ π = arg min i ∈{ 0 , 1 , ⊥} E [ c ( i ) | Z = z ] Thus, our adaptation of the results is equivalent to the original results from Mozannar and Sontag [2020].

B.1 Costs Are Coherent

In this section, we prove the coherency of our costs as defined in Section 6.2, and show the improvement of the CARED policy over policies that depend solely on either a human expert or a machine.

Proof of Theorem 1. For a sample ( x, a, y, Y (0) , Y (1)) ∼ P full, where w.l.o.g it holds that Y (1) > Y (0) , that is, the right treatment for this sample is 1 . We assume Definition 1 holds with probability 1 -δ . Then we prove this theorem in two steps: Comparison against the human expert policy ( π exp ), and comparison against the bounds policy( π bounds ).

Comparison Against the Expert :

We distinguish between two main cases:

The expert is right :

When the expert is right, i.e. π exp ( x ) = 1 , and Y ( π exp ( x i )) = Y (1) , then our policy will predict the wrong treatment ˜ π ( x ) = 0 when the following condition holds:

That is:

In simple words, we recall that our method assigns the treatment with the minimal cost among all other treatments. Thus, when the cost of the wrong treatment is the minimal cost among the other costs, our method will make mistakes.

We note that it holds that:

Thus, ˜ π ( x ) is wrong when:

That is:

We note that it holds that:

Thus, ˜ π ( x ) is wrong when:

We note that the condition in 20 never holds when CAPO bounds are valid, and therefore, with probability 1 -δ , it holds that ˜ π ( x ) = 1 , and Y (˜ π = ( x )) = Y (1) , That is, Y (˜ π ( x )) = Y ( π exp ( x ))

The expert is wrong :

When the expert is wrong, i.e. π exp ( x ) = 0 , and Y ( π exp ( x )) = Y (0) , then, from the same considerations above, our policy is wrong when:

When this happens, we have that ˜ π ( x ) = 0 , and Y (˜ π ( x )) = Y (0) , that is, Y (˜ π ( x i )) = Y ( π exp ( x i )) , otherwise, when this condition does not hold, we have that Y (˜ π ( x )) > Y ( π exp ( x ) )

Comparison Against the Bounds Policy

We analyze the cases where each policy makes mistakes. We note that ˜ π is always right when CAPO intervals don't overlap as proved in Appendix A. This holds for the π bounds policy as well, as this implies directly from its definition. On the other hand, when the CAPO bounds overlap, both policies can make mistakes

Thus, we distinguish between two cases:

CAPO bounds don't overlap In this case, both policies recommend the right action, thus it holds that, Y (˜ π ( x )) = Y ( π bounds ( x )) = Y (1) .

CAPO Bounds don't Overlap In this case, π bounds defer the decision to the expert. As for ˜ π , as shown above, the policy ˜ π is wrong when the expert is wrong and the condition 23 holds. Therefore, if condition 23 holds, and the expert is wrong, then both policies will go wrong, and thus we have that Y (˜ π ( x )) = Y ( π bounds ( x )) = Y (0) .

On the other hand, if CAPO intervals overlap, but Condition 23 doesn't hold, then ˜ π give the right recommendation with Y (˜ π ( x )) = Y (1) , but π bounds still give the wrong recommendation with Y ( π bounds ( x )) = Y (0) , that is Y (˜ π ( x )) > Y ( π bounds ( x )) .

B.2 Generalization Bound

We now prove Theorem 2.

Corollary 3 (Bounded log-Softmax) . Let π ∈ Π be a learner that satisfies Assumption 3. Then, the term exp( π j ( x )) ∑ k ∈{ 0 , 1 , ⊥} exp( π k ( x )) is bounded.

Proof of Corollary 3. π is a policy satisfying Assumption 3, that is for each j ∈ { 0 , 1 , ⊥} we have that | π j ( · ) | ≤ C π , where K is the number of classes. That is, it holds that

Lemma 1 (Bounded loss function) . Let Π be a class of policy learners satisfying Assumption 3 with a constant C π , and suppose Assumption 4 holds with a constant C Y . Then ∣ ∣ ∣ L CE ( π, z ; ˆ Q ) ∣ ∣ ∣ ≤ 8 · C π · C Y .

Proof of Lemma 1. Let Π be a class of policy learners satisfying Assumption 3, then the loss function L CE ( π, z i ; ˆ Q ( x i )) is bounded:

Where step ( a ) follows from Corollary 3, and ( b ) follows from the boundedness of Y in Assumption 4.

Lemma 2 (L2 Consistency of the max( · , · ) Estimator) . Let For estimators ˆ A n , ˆ B n of A,B based on n samples respectively, where there exists a function f ( n ) such that:

As a result, we have,

Similarly, we can show that

Then,

Then,

Convergence Rate The convergence rate of max( ˆ A n , ˆ B n ) is determined by the slower convergence rate of ˆ A n and ˆ B n , which is max( f ( n ) , f ( n )) = f ( n )

Corollary 4 (Bounded weights) . The weights w j ( z, ˆ Q ( x )) for j ∈ { 0 , 1 , ⊥} of the weighted surrogate loss function 4 based on the costs we define in Section 5.2 satisfy:

Proof. We recall that for j ∈ { 0 , 1 , ⊥} , a cost c ( j ) is of the form c ( j ) = A -B , where A,B ∈ { ˆ Y + ( x, 0) , ˆ Y -( x, 0) , ˆ Y + ( x, 1)) , ˆ Y -( x, 1) , y i } .

As for max k ∈{ 0 , 1 , ⊥} c ( k ) is also of the form max k ∈ [ K +1] c ( k ) = C -D , where C, D ∈ { ˆ Y + ( x, 0) , ˆ Y -( x, 0) , ˆ Y + ( x, 1)) , ˆ Y -( x, 1) , y i } . Then,

where ( a ) follows from the boundedness of the outcomes (Assumption 4 with a constant C Y ).

(ii) For A,B,C,D ∈ { Y + ( x, 0) , Y -( x, 0) , Y + ( x, 1)) , Y -( x, 1) , y i } , we denote ˆ A, ˆ B, ˆ C, ˆ D to be their estimators, respectively. Then,

where ( a ) follows from Assumption 5, and Lemma 2.

Proof of Theorem 2. Let π ∈ Π be a policy where the policy class Π satisfies Assumption 2, for a policy π ∈ Π , and Q, ˆ Q the CAPOs bounds and the estimated CAPOs bounds respectively. Given our loss function L CE ( π, z ; Q ) which satisfies Assumption 3, and Lemma 1. We are interested in bounding the term

We get the upper bound for Term B by applying Theorem 26.5 from Shalev-Shwartz and Ben-David [2014], that is, with probability at least 1 -δ we have:

where R n (Π) is the Rademacher Complexity of the policy class Π we have from Assumption 2, C π , and C Y are the constants we have from Assumption 3, and Assumption 4 respectively.

Proof. We prove this Lemma in two steps: first we show convergence in probability of max( ˆ A n , ˆ B n ) to max( A,B ) . Then we show the convergence rate.

Convergence ˆ A n and ˆ B n converge in probability to A and B respectively as n tends to infinity, that is:

We now show the upper bound for Term A :

where:

( a ) : Corollary 3 .

( b ) : Linearity of Expectation .

( c ) : E [ X ] ≤ | E [ X ] | .

( d ) : | E [ X ] | = » E [ X ] 2 .

( e ) : Jensen's Inequality: E [ X ] 2 ≤ E [ X 2 ] .

( f ) : Assumption 5 .

( g ) : Corollary 4 , and Definition of O p ( · ) with constant C > 0 .

Putting it all together, we have:

C Additional Experimental Detail

The experiments in this paper were conducted using a PowerEdge R750XA Server with 2 CPUs and 4 NVIDIA A40 GPUs. Here we provide all the details required to replicate the paper results. In addition, we provide replication code at https://github.com/marahgh/CARED .

C.1 Synthetic Data

In this experiment, We used the synthetic dataset from Gao and Yin [2023]The CAPO bounds were estimated using XGBRegressor from xgboost as the base learners for the B-Learner [Oprescu et al., 2023] estimator, and a LogisticRegression from scikit-learn as the propensity score estimator. We show in Table 1 the hyper-parameter choices for each model. As for the policy model, we use a logistic regression model implemented as a one-layer MLP using several functions from pytorch , The policy model was implemented using pytorch_lightning model as a wrapper model that can receive any policy model as the base model for learning the policy. Moreover, the hyper-parameters we use for the policy model are as follows: learning_rate = 0 . 001 , optimizer = Adam , patience = 3 , and max_epochs = 100 .

We replicate the experiment from Gao and Yin [2023] where they generate 10 instances of the synthetic dataset according to the Data Generation Process they mention in their paper, where lower outcomes are assumed to be better in this dataset. For each instance, they vary the level of allowed hidden confounding and compare the methods by their regret related to the baseline no-treat policy, i.e. π 0 ( x ) = 0 .

C.2 IHDP Dataset

In this experiment, we use the hidden confounding version of IHDP [Hill, 2011] which was introduced by Jesson et al. [2021]. The CAPO bounds were estimated using XGBRegressor from xgboost as the base learners for the B-Learner [Oprescu et al., 2023] estimator, and a LogisticRegression from scikit-learn as the propensity score estimator. We show in Table 2 see the hyper-parameter choices for each model. As for the policy model, we use a logistic regression model implemented as a single-layer MLP using several functions from pytorch , The policy model was implemented using pytorch_lightning model as a wrapper model that can receive any policy model as the base model for learning the policy. Moreover, the hyper-parameters were tuned for each uncertainty level Λ using the ray.tune over the search space of the hyper-parameters: The search spaces used is learning_rate ∈ [1 e -4 , 0 . 1] , optimizer ∈ [ SGD,Adam,AdamW ] , weight_decay ∈ [1 e -10 , 1 e -3] , patience ∈ [5 , 20] , and max_epochs ∈ [30 , 50] .

As for Gao and Yin [2023]'s method, we train a logistic policy with the hyper-parameters: learning_rate = 0 . 01 , batch_size = 32 , max_epochs = 100 . The hyper-parameters were tuned using a grid search over a set of possible values for each parameter, and the set with the most reasonable loss was chosen. In addition, we set C ( X ) = 0 which represents the additional cost of deferral used in Gao and Yin [2023]'s objective for cases where the outcome of deferral is Y + C ( X ) . In this experiment, the outcome of deferral to the expert is solely based on the result of the treatment prescribed by the expert, which is Y .

We generate 1111 instances of the Hidden Confounding IHDP [Jesson et al., 2021], each consists of training ( n = 470 ), validation ( n = 202 ), and test ( n = 75 ) subsets, where for each instance the seed runs over 0 , 1 , ... , which is the number of the trial/instance.

We filter the 1111 instances by the value σ CATE = √ V ar ( CATE test ( X )) by excluding trials with high σ CATE , as instances with high σ CATE are unrealistic and does not match the results of the original study [Brooks-Gunn et al., 1992], as explained in Curth et al. [2021]. In our case, based on the histogram of σ CATE for the 1111 instances of the dataset, we exclude those with σ CATE > 15 and remain with 1000 instances.

C.2.1 Semi-Synthetic Expert

In this experiment, we design a semi-synthetic expert, based on the original observed expert's policy, and the outcomes observed under this policy. The actions and outcomes under this policy are denoted A and Y , respectively, as they are (trivially) exactly the observed actions and outcomes. Let A ′ and Y ′ denote the new actions by the new expert's policy and the observed outcomes under this new policy respectively. We recall that the actions and the outcomes generated by the Oracle policy are denoted by A ∗ , and Y ∗ respectively. Then, for i ∈ [ n ] , the new expert policy is given by:

The new expert A ′ is defined based on the covariate x 17 - 'worked during pregnancy', which is a binary covariate that indicates whether the mother worked during the pregnancy or not. This covariate receives a value of 1 with a probability of 0 . 59 . We build the new expert A ′ to be equal to the oracle policy when x 17 = 1 , and equal to the original policy A otherwise. Thus this expert is perfect when x 17 = 1 . This is an aspect of the expert that ideally we would want the model to learn. The choice of covariate x 17 was according to the covariate table shown in Jesson et al. [2021], where they show an analysis of the relationship between each covariate and both the treatment and the outcome. We choose the covariate x 17 as it is one of the features that is correlated with both the treatment and the outcome, and thus, an important feature that we are interested in testing our model on its ability to learn it.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work in Section 8.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We state all the assumptions and theoretical guarantees in Section 6. All the proofs are provided in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We explain the main details of the experiments in Section 7. More details are given in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include in Appendix C the link for results replication. We provide instructions for running and replicating the results as well.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the relevant details are given in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Each experiment in this paper was conducted on several trials, and all the results were reported with error bars and confidence intervals.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer:[Yes]

Justification: we mention the compute resources information of the machine we used for running all the experiments in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our work does not violate any aspect of the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Discussed briefly in the introduction and discussion.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All assets used in the paper, properly credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No IRB.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a learning to defer method for policy learning problem with binary actions under unobserved confounding. The proposed method uses MSM to bound the confounding strength and compare the pessimistic bound between Y1, Y0 and identifiable human reward. The final optimization uses the surrogate loss proposed in Mozannar and Sontag [2020]. Overall, the method seems sound. However, the authors overlooked some existing literatures in learning to defer for policy learning problems. The authors compare the proposed method  with a self-proposed rejection-inference style baseline, and policy learning method considering MSM without humans.\n\nSTRENGTHS:\nThe paper proposes a sound solution to policy learning with unobserved confounding by conservatively comparing the lower bound of Y1, Y0 and identifiable human reward. The method has a strong performance compared to baselines used in the paper by leveraging a stronger direct method class. It is interesting to see how direct method can be used in the UC case for learning to defer since with unconfoundness, both Y1 and Y0 are identifiable and humans may not be needed unless the policy class is limited.\n\nWEAKNESSES:\nMissing related work and inaccurate claims: For the related work, it seems that the learning to defer for policy learning problem is first proposed in [1], and the learning to defer for policy learning with unobserved confounders is first proposed in [2]. These two papers adopts the Inverse-Propensity-Score framework to address the problem with similar experimental setup and the same problem setting. The authors should compare with existing works and it seems the claim that it is \"the first to learn a policy with the option of deferral to an expert while allowing for hidden confounders\" is not very accurate. Also the assumptions in this paper is stronger since it requires an additional assumption in Definition 1. \n\nBinary action: [1][2] can work with multiple actions, it also seems the proposed method  should also be extended to multiple actions, so why authors only restrict to binary actions?\n\nTheoretical result: For the bound validity assumption/definition 1, this seems to suggest Y is deterministic, which is rather strong. For example, a simple Gaussian distribution would violate this assumption. What if Y is a random variable, how can theorem 1 be adapted? I would expect a regret bound for the total reward in the paper (theorem 2 is a bound for the approximated loss instead of the reward if I understand correctly). Similarly, for a general direct method, when would we expect assumption 1 holds and not holds?\n\nExperiments - baseline: For baselines, I would expect a purely pessimistic AI baseline for a fair comparison. For example, the eq 3 can also use the pessimistic principle (compare Y1_lower and Y0_lower in the otherwise condition) authors used for comparing human and pessimistic bounds of the counterfactuals.  The authors should also compare with existing works using IPS method and discuss the trade-offs. \n\nExperiments: the experimental results are relatively weak with a synthetic dataset and the IHDP dataset. It would make the paper stronger if the authors can experiment with real human responses and real-world datasets. \n\nPresentation: I feel the authors should expand on the introduction of Mozannar and Sontag [2020] for the paper to be self-contained. Currently, it seems hard to understand unless readers have read Mozannar and Sontag [2020] separately. \n\n[1] Gao R, Saar-Tsechansky M, De-Arteaga M, et al. Human-AI collaboration with bandit feedback, IJCAI-21. \n[2] Gao R, Yin M. Confounding-robust policy improvement with human-ai teams[J]. arXiv preprint arXiv:2310.08824, 2023.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies deferral policy learning when there is unmeasured hidden confounding observed by human experts but not recorded in data. The paper formulates it as a cost-minimization problem and derives a feasible surrogate loss. The method is shown to achieve better policy value on synthetic and semi-synthetic data.\n\nSTRENGTHS:\n- The paper is generally well-written.\n- Theoretical properties are explored.\n- The proposed method is easy to implement and the performance is validated by the empirical studies.\n\nWEAKNESSES:\nThe performance of the proposed human-AI collaboration system relies on the properly specified cost function. For the expert deferring cost and action assigning costs, the current algorithm sets them in a conservative way. As the author mentioned, there are multiple ways to set these costs. Is the conservative cost always the best? If so, please provide theoretical and empirical support. If not, it would be helpful to clarify the situations when the conservative cost is inferior and evaluate other cost functions in the empirical studies. \n\nFor the theoretical analysis, the main condition in Definition 1 requires coverage for all $Y(a)$, which is a strong and maybe infeasible requirement. In practice, the guarantee usually only holds for the expected coverage or the probability of coverage. How would this more practical validity definition influence the Theorem 1 claim? \n\nLastly, the paper states \"there are no previous works that learn a policy with the option of deferral under hidden confounding,\" but it seems [1] considers a similar scenario of unmeasured confounding in the human-AI system. If so, it would be important to discuss the differences and use this method as a baseline in the simulation studies. \n\nI look forward to the author's responses. \n\n[1] Confounding-Robust Policy Improvement with Human-AI Teams (https://arxiv.org/abs/2310.08824)\n\nQUESTIONS:\n- In Sec. 4.1, the expert’s action space $\\mathcal{Y}$ is not defined. Is it the same as the action space $\\mathcal{A}$?\n\n- Are the observed data $(X_i, A_i,Y_i)$ only generated by human experts? Can it be generated by an algorithm? \n\n- In Figs 1 and 2, why the value of CARED policy does not change with a wide range of $\\Gamma$ values? And why does it not peak at the true $\\Gamma$? It would be better to plot the true $\\Gamma$ as a vertical line in these figures.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

comments: SUMMARY:
This work learns a policy that can abstain from predicting an action in the case where actions are binary. Their idea follows previous work in the learning to defer literature for supervised learning. Their proposed surrogate can recover the optimal policy, for which they design the cost functions in a way to satisfy this condition. Their experiments show the effectiveness of the proposed method.

STRENGTHS:
- They show the surrogate loss has the same optimal solution as the original expert-machine loss.
- Their experiments on synthetic data and IHDP hidden confounding show their method CARED outperforms baselines consistently in terms of policy value.

WEAKNESSES:
- The authors missed some references on learning to abstain such as [1]

[1] Yin, Tongxin, Jean-Francois Ton, Ruocheng Guo, Yuanshun Yao, Mingyan Liu, and Yang Liu. "Fair Classifiers that Abstain without Harm." In _The Twelfth International Conference on Learning Representations_.

QUESTIONS:
- The cost function $C(x_i,a)$ is the worst-case regret. I wonder if what would the results be if an alternative such as mean regret or best-case regret is used.
- L 240 I wonder if the Rademacher Complexity varies with NN architecture as the paper claims any NN with weight decay or dropout has the same Rademacher Complexity.
- Did the authors consider the fact that the number of sample the expert can process is limited?

LIMITATIONS:
- The cost function relies on bounds of potential outcomes, which is often unknown in real-world applications. This eventually reduces to the dependence on the B-Learner. I wonder if the authors considered alternatives of B-learner to estimate the bounds.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper combines two frameworks: optimizing surrogate losses for learning to defer, and bounds on CATE under unobserved confounding under the marginal sensitivity model. The challenge with applying the surrogate loss from Mazumdar and Sontag to the causal setting is that the costs of various classification outcomes are unknown, and need to be estimated. The paper proposes essentially deferring when the causal effects are not (partially) identified to be strictly positive or negative and develops conservative/optimistic characterizations of these losses. Deferral comprises of reporting the recorded $a_i$. When the unobserved confounding is exactly correlated with potential outcomes (as is the case with simulation specifications of unobserved confounding that are \"favorable\" to robust bounds methodology), therefore deferral can be adaptive and improve upon both the expert policy and valid bounds.\n\nSTRENGTHS:\nsignificance: \n\nWhile the paper connects two prior areas (surrogate losses for learning to defer, and PI bounds), there is some work needed to establish the connection and the paper does a good job of doing so and showing that such an approach can obtain improvements upon current conservativism of robust approaches only (under implicit assumptions on expert-ness of the underlying behavior policy). This is well-illustrated in the empirics.\n\nWEAKNESSES:\nThere seems to be an implicit assumption that the confounded behavior policy\n\nQUESTIONS:\n- Could the authors please describe if there is some relationship between implicit assumptions on regimes where this approach \"does well\", i.e. can _strictly_ improve upon robust PI bounds policies, and recent work on \"superoptimal\" treatment regimes? \n\nsuperoptimal treatment regimes: Optimal regimes for algorithm-assisted human decision-making\nMJ Stensrud, JD Laurendeau, AL Sarvet - Biometrika, 2024\n\n-\n\nLIMITATIONS:\nsufficiently although studying more explicitly 1) when the policy improvement is strict and 2) how do assumptions on the expert-ness of the underlying confounded behavior policy relate to the performance of the conservative/optimistic losses given in the paper", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the work of [Mozannar & Sontag 2020] on \"learning to defer\" to the causal inference setting with (bounded) hidden confounding. Compared to the original supervised learning setting, the causal setting does not have ground truth labels. Therefore this paper proposes to use estimated bounds on the potential outcomes to drive the deferral decisions, and formulates it as an optimization problem. \n\nExperiments are conducted on synthetic and semi-synthetic (IHDP) datasets.\n\nSTRENGTHS:\n- Studies an interesting and relevant problem\n- Writing is generally clear.\n\nWEAKNESSES:\n- Although writing is generally clear, the flow could be improved to better explain some of the baselines / obvious / naive approaches, and justify why the proposed approach (arguably more complex) is necessary.\n\nQUESTIONS:\nMajor:\n- L227 Eqn (3) B-learner policy: this is used as a baseline approach in the experiments. Currently this is presented in the theory section, and for a moment I thought this is your proposed approach. It is such a straightforward definition, a simple approach that also outperforms the baselines. I think this needs to be presented earlier in the text, and you need to make it clear why it's not good enough and how your approach is different. \n\nOthers:\n- L37: \"human expert typically has access to the hidden confounders\" - in what form? Do you mean human decisions are more likely to account for hidden confounders. \n- L181-182: in the definition of pessimistic and optimistic deferral costs, what is $\\max_{a' \\neq a_i}$, aren't there only two actions? L190 I assume it's defining the same thing but different notation is used. \n- L202 Algorithm 1 is never referenced in the methods section. Also it seems to simply repeat the expressions already presented in the text above, not sure this algorithm is necessary.\n- L224 what does bounds are valid *on average* mean? How does that affect the theoretical results if some bounds are invalid? \n- L227 \"This policy defers if and only if the CATE upper bound is positive and the CATE lower bound is negative.\" You may elaborate that intuitively if bounds are both positive then the CATE is positive (and vice versa) so it's clear what to do, and the deferral situation is when it's unclear the true CATE is >0 or <0 and unclear what to do. \n- L256 Theorem 2the generalization bound compares the training loss and the expected loss which are both L_CE, but how is the result of this learning process related to Eqn (1) the true optimization objective? \n- Theorem 2: What does the generalization mean and how does it compare to other things? Does it tell us about what we should do in practice when using your approach? \n\n- experiments should compare to a random deferral strategy as a baseline. \n- Fig 2a \"proposed method outperforms both baselines\" -> this is not true, for large log(Λ) the proposed approach is overtaken by B-learner. \n- Fig 2b as deferral rate increases beyond 60% performance drops, I think it's worth commenting on if for example we allow for *up to* 80% deferral, can the policy learning figure out it only needs to defer 60% of the time. Alternatively, what are practical ways of selecting this deferral ratio? \n\nMinor issues: \n- Citation style: there are places where the reference in text should be parenthesized but are not. \n- Typo: There's a slash on L13 of the abstract. \n- L122-124, L138-139L deferral policy defined twice\n- L132 \"ho\" -> \"how\"\n- L204 \"we see than\" -> we see that\n- L249 says $Y^{+}_1(x) - Y^{+}_1(x)$. I think there is a typo. \n- L271 says three variants, but only two are listed \n- L347 \"We note there when learning to act\" -> we note that\n\nLIMITATIONS:\nMain limitations noted include access to expert actions in observational data, and that experts behavior do not change when they receive machine's deferral.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}