Title: Towards Estimating Bounds on the Effect of Policies under Unobserved Confounding
Venue: NeurIPS 2024
Accepted: yes
Keywords: causal inference

estimation

bounding

unobserved confounding
Abstract: As many practical fields transition to provide personalized decisions, data is increasingly relevant to support the evaluation of candidate plans and policies (e.g., guidelines for the treatment of disease, government directives, etc.). In the machine learning literature, significant efforts have been put into developing machinery to predict the effectiveness of policies efficiently. The challenge is that, in practice, the effectiveness of a candidate policy is not always identifiable, i.e., not uniquely estimable from the combination of the available data and assumptions about the domain at hand (e.g., encoded in a causal graph). In this paper, we develop graphical characterizations and estimation tools to bound the effect of policies given a causal graph and observational data collected in non-identifiable settings. Specifically, our contributions are two-fold: (1) we derive analytical bounds for general probabilistic and conditional policies that are tighter than existing results, (2) we develop an estimation framework to estimate bounds from finite samples, applicable in higher-dimensional spaces and continuously-valued data. We further show that the resulting estimators have favourable statistical properties such as fast convergence and robustness to model misspecification.

Towards Estimating Bounds on the Effect of Policies under Unobserved Confounding

Abstract

As many practical fields transition to provide personalized decisions, data is increasingly relevant to support the evaluation of candidate plans and policies ( e.g. , guidelines for the treatment of disease, government directives, etc.). In the machine learning literature, significant efforts have been put into developing machinery to predict the effectiveness of policies efficiently. The challenge is that, in practice, the effectiveness of a candidate policy is not always identifiable, i.e. , not uniquely estimable from the combination of the available data and assumptions about the domain at hand ( e.g. , encoded in a causal graph). In this paper, we develop graphical characterizations and estimation tools to bound the effect of policies given a causal graph and observational data collected in non-identifiable settings. Specifically, our contributions are two-fold: (1) we derive analytical bounds for general probabilistic and conditional policies that are tighter than existing results, (2) we develop an estimation framework to estimate bounds from finite samples, applicable in higher-dimensional spaces and continuously-valued data. We further show that the resulting estimators have favourable statistical properties such as fast convergence and robustness to model misspecification.

1 Introduction

Understanding how to act upon the world around us requires humans and artificial systems to thoughtfully evaluate the effect of different plans, policies, and interventions one might consider. Data and methods that facilitate this process are increasingly relevant for high-stakes decision-making, notably in medical care with the rise of precision medicine [41], but also in education [25], law enforcement [13], public policy [44], and economics [2]. The interventions that a practitioner might consider, could consist of complex policies where a variable is set to follow a conditional or stochastic relationship depending on other variables in the system. For example, policy-makers might contemplate higher taxes on processed foods and stronger campaigns on its health risks targeted to overweight individuals. A sensible question in this context could be 'what is the effect of a policy that reduces the consumption of processed foods by 50% in people with a body mass index above 30?'. Contrary to atomic (also called hard) interventions that force a particular value, this policy suggests a softer intervention.

The identifiability and estimation of policies from data is a widely studied problem in the reinforcement learning [37, 39] and causal inference [2, 12, 15, 31, 32, 40] literatures. Despite the generality entailed by many common approaches in these fields, they often rely on an impractical condition: the assumption that the effectiveness of the policy is uniquely computable from the observed data and

assumptions about the data generating process. This can be violated in real-world settings subject to unobserved confounding, and lead to the non-identifiability of the effectiveness of a given policy. This holds irrespective of the number of samples collected so that in reality multiple answers for the effect of a policy may be equally plausible. In their seminal work in the early 1990's, Manski and Robins [27, 33] showed that, nevertheless, the effect of an atomic intervention may always be bounded in a non-trivial interval ( i.e. , probabilities strictly contained in r 0 , 1 s ), irrespective of unobserved confounders or the causal structure underlying the variables involved. Causal effects are therefore in general said to be partially identifiable , i.e. , one can derive bounds that shrink the range of a priori possible values for a given effect and potentially serve as a useful support for decision-making.

Starting from this insight, the problem of partial identification has been gaining growing attention in the literature. To improve upon these bounds, one common restriction on the data generating mechanism is to assume knowledge of a causal graph describing the phenomenon of interest. Several results have derived progressively tighter bounds by exploiting the independencies in observational and interventional distributions implied by the causal graph [4, 5, 14, 46, 47, 48]. For instance, Balke and Pearl [4] (and subsequent refinements, e.g. , [14, 48]) defined polynomial optimization programs to derive bounds that are provably optimal. Tighter analytical bounds have also been derived in selected settings, such as for the "instrumental variable" graph [47] and discrete systems with general graphs or equivalence classes [5, 46]. These results, however, are almost exclusively concerned with atomic interventions and small systems of discretely-valued variables. Despite their generality, there is still a gap towards making the partial-identification and estimation of the effectiveness of policies practical. In particular, making scalable inferences from finite samples with continuous outcomes and higher-dimensional covariates is not possible with existing tools.

This paper aims to provide a novel estimation framework to support decision-making in nonidentifiable settings, overcoming some of these challenges. We introduce several new results for the partial identification and estimation of the effect of stochastic and conditional policies from a combination of observational data and assumptions on the domain, encoded in a causal graph. Our contributions may be summarized as follows. We introduce several graphical criteria to derive new analytical bounds on the effect of policies with continuous outcomes and covariates, that improve upon the non-parametric bounds of [27, 33] and [47]. Given these analytical bounds, we then construct estimators leveraging the double machine learning [10] toolkit for scalable inference, and demonstrate that estimators exhibit favourable statistical properties such as robustness to noise and fast convergence. These results are applicable to arbitrary stochastic or conditional policies that an investigator might design, and high-dimensional covariate spaces. Recent work has made progress is developing powerful estimators for identifiable causal effects [6, 8, 19, 20, 36, 49]. However, these estimators are not applicable in non-identifiable settings. To our knowledge, the proposed estimation machinery is the first result for bounding the effect of policies given a causal diagram.

Preliminaries. We use capital letters to denote variables ( X ), small letters for their values ( x ), bold letters for sets of variables ( X ) and their values ( x ), and use supp to denote their domains of definition ( x P supp X ). ✶ x p X q is the indicator function that equals 1 if the statement t X ' x u is true, and equal to 0 otherwise.

The framework we use to underpin the estimation of the effect of policies rests on structural causal models (SCMs) [31, Def. 7.1.1]. An SCM M is a tuple x V , U , F , P p U qy , where V is a set of endogenous (observed) variables, U is a set of exogenous latent variables, and F ' t f V u V P V is a set of functions such that f V determines values of V taking as argument variables Pa V Ď V and U V Ď U , i.e. V Ð f V p Pa V , U V q . Values of U are drawn from an exogenous distribution P p u q . We assume the model to be recursive, i.e. that there are no cyclic dependencies among the variables. Graphically, each SCM M is associated with a causal diagram G over V , where V Ñ W if V appears as an argument of f W in M , and V glyph[arrowaxisleft]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] W if U V X U W ‰ H , i.e. V and W share an unobserved confounder.We will use graph-theoretic family abbreviations to represent graphical relations, e.g. an, de, etc. Moreover, for a causal diagram G over V , the X -lower-manipulation of G deletes all those edges that are out of variables in X , and otherwise keeps G as it is. The resulting graph is denoted as G X . The X -upper-manipulation of G deletes all those edges that are into variables in X , and otherwise keeps G as it is. The resulting graph is denoted as G X . We use | ù d to denote d -separation in causal diagrams [31, Def. 1.2.3].

For a sample set D : ' t v p i q u i ' 1 ,...,n ' P , we use E D r f p V qs : ' p 1 { n q ř n i ' 1 f p v p i q q . We use } f } P : ' a E P rt f p V qu 2 s . p f ´ f ' o P p r n q denotes that a function p f is a consistent estimator of f at a rate r n , p f ´ f ' O P p r n q denotes that it is bounded in probability at rate r n .

2 Partial Identification of the Effect of Policies

A policy π over a subset X Ă V is a sequence of decision rules or plans π : ' t π X u X P X for determining the assignment of variables X . In its most general form, every π X : supp X ˆ supp C X Ñ r 0 , 1 s is a probability mapping from domain of C X Ă C Ă V to the domain of X .

Qualitatively different types of interventions may be modelled with π X . Specifically, a deterministic intervention setting X : ' g p c X q based on the values of C X can be encoded as π X p x | c X q : ' ✶ g p c X q p x q while a probabilistic intervention may be written π X p x | c X q : ' P induced by π X p X ' x | c x q . The intervened model M π represents a different regime in which the assignment t f X u X P X is replaced with the assignment induced by π . For an outcome Y P V , the interventional

X

Y

C

(a)

G

X

Y

C

(b)

G

π

distribution P M π p Y q , equivalently P π p Y q , is defined as the distribution over Y in M π . It will be useful to adopt the notation ¯ π : ' ś X P X π X p x | c X q to denote the product of policy assignments on individual variables subject to intervention in π .

Example 1 (Illustration of policies) . In the context of a public health program, let X be a measure of an individual's weekly physical exercise routine, C an individual's family history of metabolic diseases, and Y hospital admissions related to heart disease. Consider the causal diagram Fig. 1a; currently, an individuals exercise voluntarily, which depends on unobserved factors that could be associated with risk of heart disease. The government is considering an incentive plan aimed at increasing the frequency of exercise depending individual's family history of metabolic diseases (with some probability). The proposed intervention can be encoded as π ' t π X u and sets the new assignment ˜ f X such that the variable X follows the pre-specified distribution. Graphically, this policy is represented by Fig. 1b (with the new edge corresponding to the implementation of the policy highlighted in blue) and may be evaluated with the following quantity,

The inferential challenge is that this quantity is not uniquely computable from P p x, y, c q and G , requiring more complex and nuanced notions of policy evaluation.

Formally, we are interested in the evaluation of the effectiveness of a plan or policy π , acting on a (potentially multivariate) discrete action X based on values of observed covariates C P R d , on an outcome of interest Y P r 0 , 1 s 2 .

Definition 1 (Average treatment effect) . The effectiveness of a policy π on Y is E P π r Y s .

From the investigator's perspective, only the causal diagram G of the environment M is available. No assumptions about the form or shape of P p U q and F are made, but for the structural knowledge encoded in G . In general, there might exist multiple SCMs M that induce the same causal diagram G and entail P p V q but result in different values of E P π r Y s . The identification of a set of possible solutions that contain the true value of E P π r Y s leads to the notion of partial identification .

Definition 2 (Partial Identification) . The effectiveness of a policy π is said to be partially identifiable from G and P p v q if

where p ψ ℓ , ψ u q are functionals of P that are bounded away from 0 and 1 , respectively.

The earliest bound on the effect of policies, the so-called natural bounds , were developed considering discrete atomic interventions, i.e. of the form π X ' x p X q , X P t 1 , . . . , d X u .

X

1

Y

X

2

C

(a)

X

1

Y

X

2

C

(b)

X

2

Y

X

1

S

Z

(c)

A

X

1

Y

X

3

X

2

C

W

(d)

Proposition 1 (Natural Bounds) . For any x , E P r Y ✶ x p X qs ď E P x r Y s ď E P rp Y ´ 1 q ✶ x p X qs ` 1 . Remarkably, these inequalities hold irrespective of the causal graph of the system G ; a result that dates back to [27, 33]. Similarly, we could adapt the underlying proof strategy to derive functionals of P that bound the effect of more general probabilistic or conditional policies π , given as follows.

Proposition 2 (Natural Policy Bounds (NPB)) . For any π , E P r Y ¯ π s ď E P π r Y s ď E P rp Y ´ 1 q ¯ π s` 1 .

These inequalities also hold irrespective of the causal graph of the system. The natural bounds from [27, 33] are a special case of the natural policy bounds (NPBs) by setting π : ' t π X u X P X where π X : ' ✶ x p X q , and therefore ¯ π ' ✶ x p X q . We could show that the NPBs are tight in some cases. For example, in Example 1, provably, no better bounds for E P π r Y s than those defined by Prop. 2 could be derived (a result given in Prop. 10 in Appendix B). For other systems that involve variables that are 'separated' in G , however, better bounds may be derived by exploiting the implications of the causal graph and definition of the policy on the induced observational and interventional distributions. Consider the following example as a first illustration of this idea.

Example 2 (Tighter bounds with causal diagram) . We are interested in evaluating the effect of a policy π : ' t π X 1 p x 1 | c q , ✶ x 2 p X 2 qu from P p x 1 , c, x 2 , y q and G in Fig. 2a. In particular, this problem involves a stochastic conditional intervention on X 1 given C and a deterministic intervention on X 2 ; both variables having differing dependencies onto the rest of the system. We could show that,

The inequality gives an expression for the bound in terms of P that exploits the fact that intervening on X 2 does not influence Y , and is tighter that the natural policy lower bound as ψ ℓ p P q ě E P r Y π X 1 s P p X 2 ' x 2 q ' E P r Y π X 1 ✶ x 2 p X 2 qs p' NPB q . For the upper bound we could similarly establish that, E P π r Y s ď E P rp Y ´ 1 q π X 1 s ` 1 ' ψ u p P q . (4)

which is smaller than the natural policy bound as ψ u p P q ď E P rp Y ´ 1 q π X 1 ✶ x 2 p X 2 qs ` 1 p' NPB q . This example, although relatively straightforward, serves to illustrate the potential of causal diagrams (and the constraints they imply on the effect of policies) for defining tighter bounds.

3 Graphical Criteria for Partial Identification

This section aims to consider more general separation statements between variables encoded in a causal diagram and the decomposition they imply to provide a systematic algorithm to bound the effectiveness of policies. We start by introducing the notion of partial adjustment sets (Def. 3) that is applicable with multiple intervention variables.

Definition 3 (Partial adjustment set) . Let π : ' t π X 1 , π X 2 u be a policy on t X 1 , X 2 u with a conditioning set C . A set W Ď V zp X 1 Ť X 2 Ť C Ť Y q is said to be a partial adjustment set for π X 2 in G if p Y | ù d X 2 | W , C , X 1 q G π X 1 X 2 .

Proposition 3. Let π : ' t π X 1 , π X 2 u be a policy mapping a set of covariates C to a set of treatment variables t X 1 , X 2 u . Let W be a partial adjustment set for π X 2 in G . Then,

In words, partial adjustment sets are designed to exploit the unconfounded status of some intervention variables with respect to the outcome, and could be leveraged to derive tighter bounds when available.

Example 3 (Tighter bounds with partial adjustment sets) . Consider the problem of evaluating the effect of a policy π : ' t π X 1 p x 1 | c q , π X 2 p x 2 | c qu from P p x 1 , c, x 2 , w, y q and G in Fig. 2b. Following Def. 3, we could establish that W ' H is a valid partial adjustment set for π X 2 since we can verify that p Y | ù d X 2 | C, X 2 q G π X 1 X 2 . Prop. 3 then gives us a valid expression for bounding the effect of the policy,

For the upper bound we similarly find that,

Next, we define a second useful notion, so-called partial conditional instrumental variables sets , that can be exploited to evaluate bounds in z -specific distributions P p ✶ z p Z q{ P p Z qq instead of in P as described in Prop. 4.

Definition 4 (Partial conditional instrumental set) . A set Z Ď V is said to be a partial instrumental set conditional on R for a policy π in G if p Y | ù d Z | R q G π .

For illustration, we give a simple criterion below to show how this subgroup structure could be exploited to derive tighter bounds.

Proposition 4. Let Z be an unconditional partial instrumental set with respect to a policy π , i.e. p Y | ù d Z q G π . Then,

Input: Graph G , policy π , outcome Y . Output: Bounds for E P r Y s .

✶ The following example shows that partial adjustment sets and partial conditional instrumental sets may be usefully combined to derive tighter bounds than would be available had each proposition (Props. 3 and 4) been applied in isolation.

Example 4 (Tighter bounds with partial adjustment and instrumental sets) . For this example, consider the evaluation of an atomic intervention π : ' t ✶ x 1 p X 1 q , ✶ x 2 p X 2 qu in the causal diagram G given in Fig. 2c. Note that ¯ π ' ✶ x p X q . Following Def. 3, tHu is a valid partial adjustment set for π X 1 since we can verify that p Y | ù d X 2 | X 1 q G π X 1 X 2 . Further, we could verify that t Z u is a partial instrumental set conditional on t X 2 u with respect to π X 1 since p Y | ù d Z | X 2 q G π X 1 . These two separation statements in (manipulated versions of) G could be leveraged to derive a tighter bound than previously considered:

To derive bounds in a more systematic fashion, combining the notions developed so far, we present Alg. 1 that recursively seeks to find partial adjustment and partial instrumental sets in an efficient and automatic manner.

Intuitively, Alg. 1 seeks to recursively simplify the query. First by omitting the intervened variables that have no effect on the outcome; second by finding the set of intervened variables for which a partial adjustment set could be used to tighten the bound, and third by finding the set of variables that act as valid partial instrumental sets to evaluate bounds on the most favorable conditional distributions rather than on the joint distribution.

Proposition 5. Alg. 1 is sound.

In words, Prop. 5 says that for a given causal diagram, policy π , and joint distribution P p v q , the average treatment effect E P π r Y s is contained in the bounds produced by Alg. 1.

Proposition 6. Let k be the number of variables and m be the number of edges in G . The run time of Alg. 1 is O p k p k 2 ` m qq .

Example 5 (Steps of Alg. 1) . For this example we consider evaluating a policy π : ' t π X 1 p¨ | C q , π X 2 p¨ | C q , π X 3 p¨ | C qu from observational data P p x 1 , c, x 2 , x 3 , a, w, y q compatible with G in Fig. 2d by explicitly following Alg. 1. On line 1, we define C ' t C u , K ' t A,W u . We start by omitting potentially irrelevant intervention variables by evaluating R ' t X 1 , X 2 u and noticing that E P π r Y s ' E P π R r Y s . To find partial adjustment sets in line 4, we evaluate the for loop iterating over R . For R ' X 1 , we find that S ' X 2 , W X 1 ' t A,W u but the if condition fails as Y is not d -separated from X 1 given t X 2 , C, A, W u . We turn onto R ' X 2 , where S ' X 1 , W X 2 ' t A,W u . The if condition is triggered as p Y | ù d X 2 | X 1 , C, A, W q in G π X 1 ,X 2 and therefore we update W ' W X 2 ' t A,W u (giving a partial adjustment set). We continue with line 12, set T ' t X 2 u , U ' t X 1 u , and iterate over K ' t A,W u in the for loop. For Z ' A , we have that p Y | ù d A | W,X 2 q in G π X 1 , the if condition is triggered and therefore we update Z ' t A u . For Z ' W , we find that p Y ­ K K d W | A,X 2 q in G π X 1 and therefore terminate the for loop. Finally putting together the pieces, we evaluate the bounds to be,

X

1

Y

X

2

W

1

W

2

Remark (Multiple bound expressions). Alg. 1 is designed to make use of partial adjustment and conditional instrumental variable sets but it will return a single bounding expression. In general multiple different expressions could be derived for a given causal diagram and data distribution. For example, given G in Fig. 3 and a policy π on t X 1 , X 2 u , E P r ¯ π Y { P p X 2 | W 1 qs and E P r ¯ π Y { P p X 2 | W 2 qs give valid, but numerically different lower bounds for E P π r Y s . The superiority of one bound over another might depend on the associations in the underlying distribution P p v q . In this case, we could construct two different SCMs compatible with G that entail different orders in the numerical values for the two bounds, such that neither is always optimal. In general, we conjecture that finding an optimal analytical bound (using the proposed procedure) is undecidable from the graph alone.

The observation in this remark motivates to develop a procedure to automatically enumerate all partial adjustment sets to facilitate the search for tighter bounds. The following proposition adapts the ListSep algorithm by [43] to enumerate all partial adjustment sets.

Proposition 7 (Enumerating partial adjustment sets) . Let π : ' t π X 1 , π X 2 u be a policy on t X 1 , X 2 u with a conditioning set C . All partial adjustment sets may be enumerated in time O p k p k ` m qq where k are the number of variables and m be the number of edges in G .

Remark (Related methods for policy evaluation). In parallel to the graphical approach for encoding structural assumptions about the domain at hand, a number of works have adopted sensitivity assumptions that quantify the degree of unobserved confounding through various data statistics, such as odds ratios, propensity scores, etc. A rich literature on sensitivity assumptions exists, including Tan's sensitivity model [38] and Rosenbaum's sensitivity model [34]. Under these models, [17, 21, 28, 45], among others, present methods that achieve validity and rate properties for the resulting estimators. These approaches start with estimators that optimize the average or conditional treatment effect bounds subject to constraints implied by the sensitivity model. Several of these works leverage Neyman orthogonality techniques to obtain rate guarantees and doubly-robustness properties. For instance, [17, 21, 28] study the estimation of bounds under Tan's sensitivity model which quantifies the degree of unobserved confounding through odds ratios, and propose estimators with various validity, sharpness, and favourable convergence rate guarantees. In contrast, [45] study the estimation of bounds under Rosenbaum's sensitivity model also deriving estimators with sharpness and fast convergence guarantees. Some works under these assumptions have also considered policy evaluation (as opposed to the evaluation of atomic interventions) under various sensitivity models in the context of Reinforcement Learning, e.g. , [7, 22, 23]. We interpret this line of work as complementary (applicable under different assumptions, i.e., sensitivity models rather than causal diagrams) to the techniques proposed in this paper.

4 Estimation of the Effect of Policies

This section aims to develop an estimation framework for the effect of a policy E P π r Y s given finite samples from P that partially identifies its value. The key observation of this section is that multiple characterizations for estimation could be derived for a given bound returned by lines 20-21 in Alg. 1.

In a first instance, parameterized by the probability ratio given in Alg. 1 defined by γ ' p γ 1 , γ 2 q ,

T PW ,ℓ : ' E P r γ 2 Y s p' ψ ℓ z q , γ 2 : ' ¯ π U γ 1 , γ 1 : ' ¯ π T ✶ z p Z q{ P p T , Z | W z Z , C q . (10) In a second instance, parameterized by a collection of regression parameters µ ' p µ 0 , µ 1 , ˜ µ 1 , µ 2 q where

T REG ,ℓ : ' E P r µ 0 p C , W , z qs could be shown to equal ψ ℓ z .

Both formulations define equivalent but different estimation targets for the lower bound 3 and may be combined leveraging the double machine learning (DML) toolkit for more efficient and robust inferences [10].

The following procedure is the main contribution of this section. It defines a DML estimator for bounding the effectiveness E P π r Y s of a policy π .

Definition 5 (DML Estimator) . Given π and G , let t R , T , W , C , Z , Y u be defined as in Alg. 1. Consider a finite sample of data D ' P , randomly split into K folds. The k 'th partition of the sample is denoted D p k q and D p´ k q : ' D z D p k q . For each k , learn approximate nuisances p ˆ γ k , ˆ µ k q with D p´ k q . Then, define

to be an estimate for the bounds on E P π r Y s where,

To analyse the error of the DML estimator, we consider the case that nuisances can be estimated consistently. Thus requirement is relatively mild in practice as accurate probability estimation employing off-the-shelf classification and regression methods is feasible in general. Its error with respect to the true bounds is given by the following proposition.

Proposition 8 (Error rates) . Suppose the nuisance estimates p ˆ µ , ˆ γ q are L 2 -consistent and bounded. Then, the error of the DML estimator ˆ T DML P t ˆ T DML ,ℓ , ˆ T DML ,u u in Def. 5 is given as follows

where R k is a random variable that converges to zero at a rate O P p 1 { ? n q .

In words, the DML estimator exhibits a robustness property since the error of ˆ T DML is bounded in probability at n ´ 1 { 2 rate whenever the nuisances converge at a rate n ´ 1 { 4 . Note that the term

Setting 1:

Setting 2:

Setting 3:

Setting 4:

5

No misspecification

Misspecification of

Misspecification of μ

Slow nuisance convergence

0.15

PW

REG

0.15

0.15

±

0.4

一

0.10

DML

0.10

0.10

工

0.05

0.00

500

1000

2000

0.00

0.05

500

1000

2000

0.00

0.05

500

1000

2000

0.1

0.2

500

1000

2000

宁

Absolute Average Error

0.3

0.15

工

0.5

0.4

0.5

0.4

0.2

0.10

0.3

T

0.3

0.1

0.05

0.2

0.2

0.0

500

工

1000

2000

0.00

500

1000

2000

0.1

0.0

500

1000

+

2000

0.1

500

1000

2000

0.3

0.15

T

0.5

0.3

0.4

0.2

0.10

0.2

0.3

0.1

0.05

0.1

0.2

0.06

0°0

0'0

500

1000

2000

++

0.00

0.10

0.08

0.06

500

1000

2000

工

0.20

0.15

0.0

500

1000

2000

0.1

0.3

500

1000

2000

0.02

0.04

0.10

0.00

500

1000

2000

0.00

0.02

500

1000

2000

0.05

0.00

工

宁

500

1000

2000

0.1

0.0

500

1000

2000

Sample size (n)

} ˆ ˜ µ 1 ,k ´ ˆ µ 1 ,k } quantifies the error in approximating the conditional expectation in Eq. (13) for a given ˆ ˜ µ 1 estimated at that stage, and that the estimation error of ˆ γ 2 and ˆ γ 2 are equivalent since they are deterministic transformations of each other. The following proposition is a corolloray that shows that the DML estimator is unbiased under misspecification.

Proposition 9 (Bias under misspecification) . Suppose either ˆ γ 1 ' γ 1 or ˆ µ 2 ' µ 2 and that either ˆ γ 1 ' γ 1 or ˆ ˜ µ 1 ' ˆ µ 1 . Then, ˆ T DML P t ˆ T DML ,ℓ , ˆ T DML ,u u is an unbiased estimator of the corresponding bound defined in Alg. 1.

5 Experiments

This section evaluates the quality of policy effect estimation from finite samples. Our goal is to illustrate the computation of bounds and provide empirical evidence of the fast convergence and robustness to misspecification of estimators.

Finite sample bounds are estimated with gradient boosting classification and regression models (for conditional expectations) or by taking sample averages (for unconditional expectations). We truncated estimates of probability mass functions in the interval r 0 . 01 , 0 . 99 s to ensure positivity. We assess the quality of an estimator ˆ T by computing the absolute average error (AAE) with respect to (a proxy for) the true bounds (estimated in practice with larger sample sizes), i.e. , AAE = | ˆ T u ´ min z ψ u z | ` | ˆ T ℓ ´ max z ψ ℓ z | . Throughout, we report various statistics: 25th, 50th, 75th percentile, etc., across evaluation runs with ten different random seeds. Further details of the simulations are provided in Appendix C.

5.1 Synthetic Simulations

The synthetic simulations consider 4 data generating mechanisms constructed according to the graphs in Fig. 4. They highlight the use of partial instrumental sets, partial adjustment sets, high-dimensional partial adjustment sets, and combinations of partial instrumental and adjustment sets of varying dimensionality. The task is to estimate bounds on the effectiveness of a policy π : ' t π X 1 , π X 2 u defined as follows.

That is the policy assigns X 1 P t 0 , 1 u randomly with probability 0 . 5 , and assigns X 2 P t 0 , 1 u as a function of C with the probability of X 2 ' 1 increasing with the value of C .

To estimate bounds on E P π r Y s , we consider the proposed estimators, labelled: T PW , estimated with the nuisances in Eq. (10), T REG , estimated with the nuisances in Eq. (11), and T DML estimated with the procedure in Def. 5. Our evaluations test performance across 4 different settings designed to highlight various properties.

Setting 1 : All nuisances estimated correctly. This setting aims to show that all estimators converge to the bound of interest.

Setting 2 : Nuisances ˆ γ are sampled from a uniform distribution to induce misspecification in the estimation of γ .

Setting 3 : Nuisances ˆ µ are sampled from a uniform distribution to induce misspecification in the estimation of µ . Settings 2 and 3 aim to demonstrate the doubly-robustness property of the DML estimator.

Setting 4 : Noise ϵ is introduced in the estimation of all nuisances p γ , µ q to emphasize error due to finite sample variation. Specifically, noise ϵ ' Normal p n ´ α , n ´ α q , α ' 1 { 4 , that induces a slower rate of convergence as a function of sample size, inspired by [24, 20]. This setting aims to show that the fast convergence behavior of the DML estimator compared to competing estimators.

The results are given in Fig. 4 4 . We observe that across all data generating mechanisms, estimators improve with the size of the dataset and converge under no misspecification (Setting 1) to the underlying bounds. It is interesting to note also the differing accuracy of estimators in the small sample regime. T PW , based the estimation of a ratio of probabilities, can be unstable with low sample sizes if the ratio denominator is estimated to be close to zero while T REG , based on a sequence of regression tasks, tends to be better behaved. T DML in contrast is constructed as a combination of elements of T PW and T REG . In particular, note in Def. 5 the use of nuisances µ and γ . As a result, T DML has quite a different performance profile. Settings 2 and 3 in Fig. 4 show that the DML estimator T DML is robust to misspecification in either the nuisances µ or γ that highlights the robustness property. Further, when decaying noise is introduced in the estimation of nuisances (Setting 4), the DML estimator outperforms in general with a faster convergence rate. We also observe that performance remains close to optimal with high-dimensional variables W , demonstrating that the DML estimator provides a practical toolkit for bounding in practice.

5.1.1 Width of Bounds According to Different Graphical Criteria

This section evaluates the width of the bounds returned by exploiting the different graphical criteria provided in Sec. 3. The simulations are based on the data generating mechanism described by causal diagram illustrated in the fourth row of Fig. 4. We consider evaluating the policy in Eq. (16) and compute the bounds obtained by applying Prop. 2 (most conservative), Prop. 3 (using the partial adjustment set W only), Prop. 4 (using the partial instrumental set Z only), and finally Alg. 1 (that combines all propositions and is the proposed approach). Fig. 5 gives the results over 10 seeds of the data and across multiple data sizes, highlighting the gain achieved by exploiting the causal structure using the proposed approaches.

Remark (Actual width of bounds in practice). The majority of our empirical evaluations are spent on evaluating the accuracy of different methods at estimating bounds, without addressing whether the returned bounds are actually informative. In practice, the graph structure can play an important role in tightening bounds but the actual width of the bounds in a particular problem are primarily driven by the distribution of data. Here is an example to make this more concrete.

For a given policy π , Prop. 2 defines tight bounds (under some circumstances) on E P π r Y s . The width of this bound is given by 1 ´ E P r ¯ π s that is ultimately driven by P p x q . This term may therefore evaluate to anything between 0 and 1 depending on P p x q and π .

T

S

F

H

Y

E

M

A

(a)

G

Obesity

(b)

For more complex causal structures, the bounds proposed in Props. 3 and 4 (and Alg. 1) reduce the width of the interval above. Loosely written, from 1 ´ E P r ¯ π s to 1 ´ E P r ¯ π s ˆ α for some α that is a function of the joint distribution P and the structure of the graph. But again, the actual width of the interval is ultimately determined by the values of P and π that may be large or small depending on the value probabilities involved.

5.2 Evaluating Health Campaigns

This section illustrates the evaluation of lifestyle recommendations for the mitigation of obesity in individuals from Colombia, Peru and Mexico [30]. The data was collected from anonymous users using a web platform, and includes reported obesity levels measured according to BMI p Y q , age p A q , smoking status p S q , frequency of consumption of high caloric food p H q , whether individuals monitored their calorie intake p M q , family history being overweight p F q , exercise frequency p E q , and time using technology devices p T q . Obesity is a multi-factored medical condition for which several causes have been acknowledged in the literature; causal diagrams relating the variables above have been curated in several related studies [1, 9]. We considered these findings to construct the causal diagram in Fig. 6a that we assumed for this example.

We aim to study the effect of a health campaign designed to lower the intake of high caloric food p H q and increase the frequency of exercise p E q on obesity levels p Y q . For instance, we could hypothesize that the campaign leads to an increase in the observed proportion of individuals rarely consuming of high caloric food ( H ) from 0 . 12 to 0 . 5 and that of individuals doing exercise ( E ) regularly from 0 . 05 to 0 . 5 . These statements could be formulated as a stochastic policy π α : ' t π α H , π α E u acting on H and E , with new assignments,

Weconsider the evaluation of expected BMI levels E P π α r Y s that range from 12 to 50 in the population, with a mean of 29 . 3 . First note that this or other policies acting on p H,E q are not identifiable due to the bi-directed edge t H glyph[arrowaxisleft]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] Y u , but may nevertheless be bounded using Alg. 1. We find that

To illustrate the inference of. bounds with the DML estimator, we consider evaluating policies with α ' 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 . Fig. 6b gives the results. The end-points of the intervals denote estimated lower and upper bounds. We see that policies that promote a healthier lifestyle (larger values of α ) are expected to reduce obesity levels on average but substantial uncertainty is still expected.

6 Conclusions

The evaluation of policies is arguably one of the critical ingredients enabling more personalized decision-making systems. When the effect of policies is not identifiable, bounds can provide an effective support for making informed decisions. In this paper we developed partial identification and estimation tools for bounding the effect of a (stochastic or conditional) policy given data and assumptions encoded in a causal graph. We introduced several graphical characterizations that induce tighter bounds, and developed an estimation framework that exhibit robustness to noise and fast convergence. The results of this paper were illustrated through synthetic simulations and a real-world health campaign example for the reduction of obesity levels.

References

Steven Allender, Brynle Owen, Jill Kuhlberg, Janette Lowe, Phoebe Nagorcka-Smith, Jill Whelan, and Colin Bell. A community based systems diagram of obesity causes. PloS one , 10(7):e0129683, 2015.

Susan Athey and Guido W Imbens. The state of applied econometrics: Causality and policy evaluation. Journal of Economic perspectives , 31(2):3-32, 2017.

Vahid Balazadeh Meresht, Vasilis Syrgkanis, and Rahul G Krishnan. Partial identification of treatment effects with implicit generative models. Advances in Neural Information Processing Systems , 35:22816-22829, 2022.

Alexander Balke and Judea Pearl. Bounds on treatment effects from studies with imperfect compliance. Journal of the American Statistical Association , 92(439):1171-1176, 1997.

Alexis Bellot. Towards bounding causal effects under Markov equivalence. In The 40th Conference on Uncertainty in Artificial Intelligence . PMLR, 2024.

Alexis Bellot, Anish Dhir, and Giulia Prando. Generalization bounds and algorithms for estimating conditional average treatment effect of dosage. arXiv preprint arXiv:2205.14692 , 2022.

Andrew Bennett and Nathan Kallus. Policy evaluation with latent confounders via optimal balance. Advances in neural information processing systems , 32, 2019.

Rohit Bhattacharya, Razieh Nabi, and Ilya Shpitser. Semiparametric inference for causal effects in graphical models with hidden variables. The Journal of Machine Learning Research , 23(1):13325-13400, 2022.

Bryony Butland, Susan Jebb, Peter Kopelman, Klim McPherson, Sandy Thomas, Jane Mardell, Vivienne Parry, et al. Tackling obesities: future choices-project report , volume 10. Citeseer, 2007.

Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters, 2018.

David Maxwell Chickering and Judea Pearl. A clinician's tool for analyzing non-compliance. In Proceedings of the National Conference on Artificial Intelligence , pages 1269-1276, 1996.

Juan Correa and Elias Bareinboim. A calculus for stochastic interventions: Causal effect identification and surrogate experiments. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 10093-10100, 2020.

Elizabeth K Drake, Steve Aos, and Marna G Miller. Evidence-based public policy options to reduce crime and criminal justice costs: Implications in washington state. Victims and offenders , 4(2):170-196, 2009.

Noam Finkelstein and Ilya Shpitser. Deriving bounds and inequality constraints using logical relations among counterfactuals. In Conference on Uncertainty in Artificial Intelligence , pages 1348-1357. PMLR, 2020.

Limor Gultchin, Virginia Aglietti, Alexis Bellot, and Silvia Chiappa. Functional causal Bayesian optimization. In Uncertainty in Artificial Intelligence , pages 756-765. PMLR, 2023.

Yaowei Hu, Yongkai Wu, Lu Zhang, and Xintao Wu. A generative adversarial framework for bounding confounded causal effects. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 12104-12112, 2021.

Andrew Jesson, Sören Mindermann, Yarin Gal, and Uri Shalit. Quantifying ignorance in individual-level causal-effect estimates under hidden confounding. In International Conference on Machine Learning , pages 4829-4838. PMLR, 2021.

Shalmali Joshi, Junzhe Zhang, and Elias Bareinboim. Towards safe policy learning under partial identifiability: A causal approach. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 13004-13012, 2024.

Yonghan Jung, Jin Tian, and Elias Bareinboim. Estimating identifiable causal effects on Markov equivalence class through double machine learning. In International Conference on Machine Learning , pages 5168-5179. PMLR, 2021.

Yonghan Jung, Jin Tian, and Elias Bareinboim. Estimating joint treatment effects by combining multiple experiments. In International Conference on Machine Learning , pages 15451-15527. PMLR, 2023.

Nathan Kallus, Xiaojie Mao, and Angela Zhou. Interval estimation of individual-level causal effects under unobserved confounding. In The 22nd international conference on artificial intelligence and statistics , pages 2281-2290. PMLR, 2019.

Nathan Kallus and Angela Zhou. Confounding-robust policy improvement. Advances in neural information processing systems , 31, 2018.

Nathan Kallus and Angela Zhou. Confounding-robust policy evaluation in infinite-horizon reinforcement learning. Advances in neural information processing systems , 33:22293-22304, 2020.

Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. arXiv preprint arXiv:2004.14497 , 2020.

Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications to educational games. In AAMAS , volume 1077, 2014.

Henry B Mann and Abraham Wald. On stochastic limit and order relationships. The Annals of Mathematical Statistics , 14(3):217-226, 1943.

Charles F Manski. Nonparametric bounds on treatment effects. The American Economic Review , 80(2):319-323, 1990.

Miruna Oprescu, Jacob Dorn, Marah Ghoummaid, Andrew Jesson, Nathan Kallus, and Uri Shalit. B-learner: Quasi-oracle bounds on heterogeneous causal effects under hidden confounding. arXiv preprint arXiv:2304.10577 , 2023.

Kirtan Padh, Jakob Zeitler, David Watson, Matt Kusner, Ricardo Silva, and Niki Kilbertus. Stochastic causal programming for bounding treatment effects. arXiv preprint arXiv:2202.10806 , 2022.

Fabio Mendoza Palechor and Alexis de la Hoz Manotas. Dataset for estimation of obesity levels based on eating habits and physical condition in individuals from colombia, peru and mexico. Data in brief , 25:104344, 2019.

Judea Pearl. Causality . Cambridge university press, 2009.

Judea Pearl and James M Robins. Probabilistic evaluation of sequential plans from causal models with hidden variables. In UAI , volume 95, pages 444-453. Citeseer, 1995.

James M Robins. The analysis of randomized and non-randomized aids treatment trials using a new approach to causal inference in longitudinal studies. Health service research methodology: a focus on AIDS , pages 113-159, 1989.

Paul R Rosenbaum, P Briskman Rosenbaum, and Briskman. Design of observational studies , volume 10. Springer, 2010.

Andrea Rotnitzky, James Robins, and Lucia Babino. On the multiply robust estimation of the mean of the g-functional. arXiv preprint arXiv:1705.08582 , 2017.

Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In International conference on machine learning , pages 3076-3085. PMLR, 2017.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Zhiqiang Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association , 101(476):1619-1637, 2006.

Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning , pages 2139-2148. PMLR, 2016.

Jin Tian. Identifying dynamic sequential plans. arXiv preprint arXiv:1206.3292 , 2012.

Anastasios A Tsiatis, Marie Davidian, Shannon T Holloway, and Eric B Laber. Dynamic treatment regimes: Statistical methods for precision medicine . CRC press, 2019.

Mark J van der Laan and Susan Gruber. Targeted minimum loss based estimation of causal effects of multiple time point interventions. The international journal of biostatistics , 8(1), 2012.

Benito van der Zander. Algorithmics of Identifying Causal Effects in Graphical Models . PhD thesis, Universität zu Lübeck, 2020.

Evert Vedung. Public policy and program evaluation . Routledge, 2017.

Steve Yadlowsky, Hongseok Namkoong, Sanjay Basu, John Duchi, and Lu Tian. Bounds on the conditional and average treatment effect with unobserved confounding factors. arXiv preprint arXiv:1808.09521 , 2018.

Junzhe Zhang. Designing optimal dynamic treatment regimes: A causal reinforcement learning approach. In International Conference on Machine Learning , pages 11012-11022. PMLR, 2020.

Junzhe Zhang and Elias Bareinboim. Bounding causal effects on continuous outcome. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 12207-12215, 2021.

Junzhe Zhang, Jin Tian, and Elias Bareinboim. Partial counterfactual identification from observational and experimental data. arXiv preprint arXiv:2110.05690 , 2021.

Yao Zhang, Alexis Bellot, and Mihaela Schaar. Learning overlapping representations for the estimation of individualized treatment effects. In International Conference on Artificial Intelligence and Statistics , pages 1005-1014. PMLR, 2020.

Appendix

Table of Contents

A Preliminaries, Related Work, and Impact Statement

A.1 Preliminaries

For the derivation of results, we use the notion of counterfactuals defined as follows.

For an SCM M , arbitrary subsets of endogenous variables X , Y , the potential outcome of Y to intervention do p x q , denoted by Y x p u q , is the solution for Y with U ' u in the sub-model M x . It can be read as the counterfactual sentence 'the value that Y would have obtained in situation U ' u , had X been x .'. Statistically, averaging u over the distribution P p u q leads to the counterfactual variables Y x . The distribution of the variable Y x is denoted P p Y x q . With this formalism, we can express all the quantities in the main body of this paper in the language of counterfactuals. For instance, the distribution of Y in sub-model M x can alternatively be written P p Y x q ' P x p Y q . Moreover, E P r f p Y x q | z s , also written E P p¨| z q r f p Y x qs , denotes the conditional expectation of f p Y q over P p y x | z q . Similarly, the distribution over Y in the sub-model M π can be written P M p Y π q , or P p Y π q ' P π p Y q for short. See [31, Chapter 7] for further context on counterfactuals.

Definition 6 (The Axioms of Counterfactuals, Chapter 7.3.1 [31]) . For any three sets of endogenous variables X , Y , W in a causal model and x , w in the domains of X and W , the following holds:

Composition: W x ' w implies that Y x , w ' Y x .

Effectiveness: X w , x ' x .

Reversibility: Y x , w ' y and W x , y ' w imply that Y x ' y .

Theorem 1 (Soundness and Completeness of the Axioms Theorems 7.3.3, 7.3.6 [31]) . The Axioms of counterfactuals are sound and complete for all causal models.

The following rules to manipulate experimental distributions produced by policies extend the docalculus and will be used for the proof of several theoretical statements [12].

Theorem 2 (Inference Rules σ -calculus [12]) . Let G be a causal diagram compatible with an SCM M , with endogenous variables V . For any disjoint subsets X , Y , Z Ď V , two disjoint subsets T , W Ď V zp Z Ť Y q (i.e., possibly including X ), the following rules are valid for any intervention strategies π X , π Z , and π 1 Z such that G π X π Z , G π X π 1 Z have no cycles:

Rule 1 (Insertion/Deletion of observations):

Rule 2 (Change of regimes under observation):

Rule 3 (Change of regimes without observation):

where Z p W q is the set of elements in Z that are not ancestors of W in G π X .

A.2 Related Work

Active experimentation by physically manipulating reality is generally not feasible for many consequential applications. This motivates the study of causal effect identification and estimation. The former studies the problem of inferring whether a unique expression or bound of the desired causal effect could be derived from the available data and assumptions. The latter studies the problem of providing efficient estimators from finite samples to compute causal effects or bounds on causal effects in practice. In the following, we review these two lines of work to better contextualize our contribution.

Partial identification . The natural bounds were derived to demonstrate that useful inference about causal effects could be drawn without making identifying assumptions beyond the observed data [27]. Relatedly, an analysis of bounds of causal effects was also provided for studies with imperfect

compliance under a set of instrumental variable assumptions that are not sufficient however to identify the causal effect of interest [33]. These bounds may be written in closed form and have recently been extended as a more general strategy for bounding causal effects given assumptions encoded in causal diagrams in discrete systems [5, 46]. Recent work has also considered bounds in closed form with access to both observational and interventional distributions [18]. These techniques were developed alongside a second line of research that employs a polynomial optimization program to compute causal bounds given a causal diagram [4]. They proposed a family of canonical models parameterized according to the causal diagram, reducing the bounding problem to a series of equivalent linear programs. [11] further used Bayesian techniques to investigate the sharpness of these bounds with regard to the observational sample size. Recently, [14, 48] describe a polynomial programming approach to solve the partial identification for general causal graphs. More recent proposals consider parameterizations in the space of linear combinations of a set of fixed basis functions [29] and neural networks [3, 16].

In parallel, a number of works have adopted sensitivity assumptions (as an alternative or in combination with a causal diagram) that quantify the degree of unobserved confounding through various data statistics, such as odds ratios, propensity scores, etc. A rich literature on sensitivity assumptions exists, including Tan's sensitivity model [38] and Rosenbaum's sensitivity model [34]. Under these models, [17, 21, 28, 45], among others, present methods that achieve validity and rate properties for the resulting estimators. These approaches start with estimators that optimize the average or conditional treatment effect bounds subject to constraints implied by the sensitivity model. Several of these works leverage Neyman orthogonality techniques to obtain rate guarantees and doublyrobustness properties. For instance, [17, 21, 28] study the estimation of bounds under Tan's sensitivity model which quantifies the degree of unobserved confounding through odds ratios, and propose estimators with various validity, sharpness, and favourable convergence rate guarantees. In contrast, [45] study the estimation of bounds under Rosenbaum's sensitivity model also deriving estimators with sharpness and fast convergence guarantees.

While some works have considered policy evaluation under various sensitivity models in the context of Reinforcement Learning, e.g., [7, 22, 23], most works target estimation of bounds on the effect of atomic interventions. We interpret this line of work as complementary (applicable under different assumptions, i.e., sensitivity models rather than causal diagrams) to the techniques proposed in this paper.

Causal effect estimation. With a causal diagram as input, causal effect estimation has traditionally focused on a subset of identifiable scenarios, relying on assumptions such as backdoor criterion or the availability of adjustment sets. Identification expressions in those cases are given by (sequential) covariate adjustments that have lead to statistically appealing estimators from observational data [32]. Notable examples are doubly robust estimators [10, 35, 42]. Recently these techniques have been extended to settings identifiable from multiple experimental distributions [8, 19, 20]. To our knowledge, no estimation framework specific to the estimation of analytical expressions derived for bounds on the effect of policies or atomic interventions has been developed.

A.3 Broader Impact Statement

Our work investigates the conditions under which policies may be bounded from observational data. Issues of policy evaluation are central to fields of application involved in decision-making as well as AI and ML. We believe that a tool to bound the effect of stochastic and conditional policies in systems with unobserved confounding is an important addition to the scientific toolbox. Reasoning instead without acknowledging for the potential of unobserved confounding may lead researchers to operate on a more heuristical basis. For instance, the hypothetical effect of public policies might be misrepresented if informed by the evaluation of an idealized atomic intervention that is difficult to implement in practice. And, once implemented, might have unintended consequences. With this background, we believe that research on the partial identification and estimation of policies based on stochastic or conditional interventions can help scientists and individuals make more informed decisions.

In this work, we start from the assumption that a causal diagram that is consistent with the underlying data generating system of interest is available. In general, this requires domain knowledge. While some of the bounds provided do not require full knowledge of the causal graph, whenever a d -separation is assumed its truth value should be justified by prior knowledge or experiment. It is

important also to make the distinction between the task of partial identification, that is inferring an expression for bounds on causal effects, and that of estimation, that is providing efficient estimators from finite samples to compute bounds in practice. This set of results concerns both of these tasks. The first objective of our procedure is to provide an expression for lower and upper bounds, irrespective of the accuracy with which one is able to approximate P p V q from finite samples. The second objective is introduce efficient estimators for bounds on the effect of policies using finite samples from P p V q . In higher-dimensional systems, the computational complexity of estimating the conditional distributions that define lower and upper bounds on causal effects is a substantial challenge. Consequently, practitioners must exercise caution when deploying the proposed method in small sample scenarios where estimators may be inaccurate. Moreover, we have stated our convergence guarantees in the infinite sample limit, without quantifying the finite-sample estimation uncertainty. Finite-sample properties could be explored similarly to [24] given a particular choice of function class to extend our results with high-probability bounds. Finally, we emphasize that simulations on real and synthetic data are provided for illustration purposes only. These results do not recommend or advocate for the implementation of a particular policy, and should be considered in practice in combination with other aspects of the decision-making process.

B Proofs

This section presents proofs for theoretical statements in the main body of this paper.

B.1 Proofs of statements in Sec. 2

Prop. 1 restated. For any x , E P r Y ✶ x p X qs ď E P x r Y s ď E P rp Y ´ 1 q ✶ x p X qs ` 1 . Proof. Consider the derivation of the lower bound.

Consider the derivation of the upper bound. It holds that,

✶ (1) follows by the composition axiom of counterfactuals.

✶ (1) follows by the boundedness of Y , here assumed bounded by 1 for simplifying the derivation.

Proposition 10. The natural bounds are tight in general.

Proof. We show this statement by providing a pair of SCMs that agree on the input observational distribution but evaluate to the lower and upperbounds, respectively, specified by the natural bounds. Let M p G q denote the space of SCMs that induce the causal diagram G in Fig. 1a. We introduce a pair of SCMs compatible with the causal diagram G that evaluate to lower and upper bounds respectively. Let X be binary, Y P r 0 , 1 s , and C P R , and consider M 1 , M 2 P M p G q defined as follows,

and,

For a P M 1 p u q ' P M 2 p u q , both SCMs agree on observational distributions P M 1 p x, y, c q ' P M 2 p x, y, c q . However the following derivations show that the interventional expectation E P M 1 r Y | do p x ' 1 qs differs across models: for M 1 equal to the analytical lower bound, and for M 2 equal to the analytical upper bound demonstrating that (in this case) the bound is tight. In particular,

Proof. Consider the derivation of the lower bound. Let π denote a policy from an arbitrary set of covariates C to X . By marginalizing over X , C we find that

C X denotes the subset of C that is used to inform the intervention on X . Moreover,

(1) holds by Rule 2 in Thm. 2 by swapping the policy π by the do intervention do p X ' x q : given that the policy acts on X taking as inputs C it is always true that Y | ù d X | C in G πX and G XX in any graph G . The following equality follows by marginalizing over the domain of X , which is assumed discrete. (2) follows by the composition axiom of counterfactuals. By combining these two expressions we get

Consider the derivation of the upper bound. It holds that,

(1) follows by the boundedness of Y , here assumed bounded by 1 for simplifying the derivation. As above by combining this inequality with the decomposition of the policy effect we get

Prop. 3 restated . Let π : ' t π X 1 , π X 2 u be a policy mapping a set of covariates C to a set of treatment variables t X 1 , X 2 u . Let W be a partial adjustment set for π X 2 in G . Then,

Proof. Consider the derivation of the lower bound. By marginalizing over X , C we find that

C X denotes the subset of C that is used to inform the intervention on X . Denote ¯ π : ' ś X P X π X p x | c X q . Let X 1 , X 2 be a partition of X such that W Ď V zp X Ť C Ť Y q is partial adjustment set for π X 2 . Then,

(1) follows by the definition of X 2 as a partial adjustment set: it holds that p Y | ù d X 2 | W , C , X 1 q in G X 1 X 2 which induces the equality E P r Y x 1 , x 2 | c , w s ' E P r Y x 1 | x 2 , w , c s . The last equality follows by the composition axiom of counterfactuals. Combining this expression with the decomposition of the policy effect we get,

For the upper bound, consider the following derivation,

The inequality follows from the boundedness of Y and the rest of the arguments are analogous to the lower bound derivation. Combining this expression with the decomposition of the policy effect implies that,

Prop. 4 restated . Let Z be an unconditional partial instrumental set with respect to a policy π , i.e. p Y | ù d Z q G π . Then,

Proof. Consider the derivation of the lower bound. Let π denote a policy from an arbitrary set of covariates C to X . Given that p Y | ù d Z q G π , by marginalizing over X , C we find that

C X denotes the subset of C that is used to inform the intervention on X . Moreover,

(1) holds by Rule 2 in Thm. 2 by swapping the policy π by the do intervention do p X ' x q : given that the policy acts on X taking as inputs C it is always true that Y | ù d X | C in G πX and G XX in any graph G . The following equality follows by marginalizing over the domain of X , which is assumed discrete. (2) follows by the composition axiom of counterfactuals. By combining these two

expressions we get

✶ And since the l.h.s. does not depend on Z we can further tighten the bound by writing,

Consider the derivation of the upper bound. It holds that,

(1) follows by the boundedness of Y , here assumed bounded by 1 for simplifying the derivation. As above by combining this inequality with the decomposition of the policy effect we get

Prop. 5 restated. Alg. 1 is sound.

Proof. For the soundness of Alg. 1, we will consider each operation in turn and show that recovered adjustment sets and conditional instrumental sets lead to a valid bound.

1. Omit redundant intervention variables. For a policy π : ' t π X p c X qu X P X denote π R : ' t π X p c X qu X P R , R Ď X . Let R ' X Ş An p Y q in G π . By Rule 3 of the σ -calculus we have that,

since Y | ù d X z R in G π , X z R and G π R , X z R as there are no directed paths from X z R to Y by definition of R . The first two lines therefore reduce the number of intervention variables. We proceed to bound the equivalent E P π R r Y s .

Find partial adjustment sets W . Line 4-11 consider finding partial adjustment sets by traversing the set of treatment variables R P R . Recall that

Denote ¯ π R : ' ś X P R π X p x | c X q . In words lines 4-11 iteratively considers the existence of separators sets between R P R and Y to reduce the scope of the policy on Y , i.e. Y π R . By [43, Lemma 3.18.], if there exists a separator W between two sets of variables X and Y such that S Ď W Ď K in a graph G , i.e. X | ù d Y | W in G , then W ' An p X Ť Y Ť S q Ş K is also a valid separator. The definition of W R in line 7 aims to leverage this result to check for the existence of separators in our case.

Consider a R P R , let S ' R z R , K ' V zp X Ť C Ť Y q . In particular, if there exist a set t C , R u Ă W Ă K such that p Y | ù d R | W q in G π S ,R then W R ' An p R Ť Y Ť C q Ş K ' An p Y Ť C q Ş K in G π S satisfies p Y | ù d R | W R q in G π S ,R , i.e. W R is a separator if one exists. Note that W R does not include any member of t C , R , Y u . Assume this separation holds and that the if statement in line 8 is triggered. Then,

The last equality holds by assumption p Y | ù d R | W R q in G S ,R .

Now considering the term E P r Y s | r, c , w R s , for the second pass through the for loop, we can see that if the d -separation statement in the if condition is fulfilled we can further reduce the scope of the intervention on S . In particular, for R 1 P R z R , assuming that the if statement is triggered for W R 1 we have that,

Upon reaching the end of the for loop we have for W defined in line 9, T ' t R : W R P W u , U ' R z T that,

3. Find partial instrumental sets Z . Starting line 12, we look for partial conditional instrumental sets. Consider first the case that a potential instruments Z P W is evaluated in the for loop. On line 15, assume that the if statement is triggered, i.e. p Y | ù d Z | W z Z, T q in G U It then holds that,

In turn, for the case that some Z 1 R W is evaluated in the for loop, if p Y | ù d Z 1 | W , T q in G U ,

For the Z obtained in line 15, we derive that,

4. Return bounds. We now proceed to use the expression above to lower and upper bound the effect of interest. In particular, for the lower bound we have that,

By combining the inequality above with the decomposition of the policy effect we get

where γ : ' ¯ π R ✶ z p Z q{ P p T , Z | W z Z , C q . Note that we have used the definition R ' T Ť U . In turn, for the upper bound we have that,

By combining the inequality above with the decomposition of the policy effect we get

where γ : ' ¯ π R ✶ z p Z q{ P p T , Z | W z Z , C q . Note that we have used the definition R ' T Ť U . Prop. 6 restated. Let k be the number of variables and m be the number of edges in G . The run time of Alg. 1 is O p k p k 2 ` m qq .

Proof. Let k be the number of variables and m be the number of edges in G . Operations in Alg. 1, such as computing ancestors (e.g. lines 2,7), could be done in O p k 2 q time, e.g. with a Breadth-First Search algorithm. Checking for d -separation is commonly done with the Bayes-Ball algorithm that can be implemented with a reachability search method, e.g. [43, Sec. 3.2.1], and could be done in O p k ` m q time by [43, Prop. 3.17]. The sets R and Z have size at most k , so the for loops in lines 5 and 14 are executed at most k times each. Combining these we get that Alg. 1 requires O p k p k 2 ` m qq time to return the bounds.

Prop. 7 restated. Let π : ' t π X 1 , π X 2 u be a policy on t X 1 , X 2 u with a conditioning set C . All partial adjustment sets may be enumerated in time O p k p k ` m qq where k are the number of variables and m be the number of edges in G .

Proof. This proposition is adapts Proposition 3.20 in [43] to find partial adjustment sets using the ListSep algorithm. ListSep performs backtracking to enumerate all separator sets Z between X and Y such that I Ď Z Ď R , aborting branches that will not find a valid separator. In particular, it calls the TestSep and FindSep algorithms recursively.

The TestSep algorithm takes as input a graph, two sets of nodes to be separated, and a candidate separator set. By setting the graph to be G π X 1 X 2 , the two sets to be separated to be Y and X 2 , and the separator set Z , TestSep will provably return whether W ' Z zp C , X 1 q is a partial adjustment set.

The FindSep algorithm uses the observation that if there exists a separator Z between X and Y such that I Ď Z Ď R then Z : ' An p X Y Y Y I q X R is a separator. For each possible partition p X 1 , X 2 q of X , we can therefore find a separator by testing (using TestSep ) whether An p X Y Y Y C q X p V zp X Y Y qq in G π X 1 X 2 is a separator. If it is then we have found a partial adjustment set.

To find all partial adjustment sets, we then proceed as follows. For each possible partition p X 1 , X 2 q of X , apply the ListSep algorithm with graph G π X 1 X 2 , sets to be separated Y and X 2 , and possible separator sets Z constrained as C , X 1 Ď Z Ď V zp X Y Y q . Then, for every separator set Z found, output partial adjustment sets W ' Z zp C , X 1 q .

This procedure finds all possible partial adjustment sets in time O p k p k ` m qq where k are the number of variables and m be the number of edges in G by Proposition 3.20 [43].

B.2 Proofs of statements in Sec. 4

We make use of two auxiliary lemmas from the literature.

Lemma 1 (Continuous Mapping Theorem, [26]) . Let t X n u n P N , X be random elements defined on a metric space S . Consider a continuous function g : S Ñ S 1 (where S 1 is another metric space). Then,

Lemma 2 (Lemma 2, [24]) . Let f η : ' f p V ; η q denote a finite and continuous functional and η denote its nuisances. For n independent samples of P , D : ' t v p i q u i ' 1 ,...,n ' P , let ˆ T ' E D r f ˆ η s and T : ' E P r f η s for some η . Let E D ´ P r f η s : ' E D r f η s ´ E P r f η s . Then, the following decomposition holds:

Suppose further that samples used for estimating η are independent and separate; and the nuisances are consistent. Then, R ' A ` B is a random variable that converges to zero at a rate O P p 1 { ? n q .

Prop. 8 restated. Suppose the nuisance estimates p ˆ µ , ˆ γ q are L 2 -consistent and bounded. Then, the error of the DML estimator ˆ T DML P t ˆ T DML ,ℓ , ˆ T DML ,u u in Def. 5 is given as follows

where R k is a random variable that converges to zero at a rate O P p 1 { ? n q .

Proof. In this proof we consider the estimation of the lower bund without loss of generality. Recall the definition of nuisance functions and estimators:

We first verify that the population level value of the regression estimator coincides with the lower bound ψ ℓ z . This can be seen with the following derivation,

The DML estimator is similarly unbiased as,

Consider the estimated value of T DML ,ℓ following the procedure in Def. 5,

We could then write, where,

By the construction in Def. 5, the samples used for estimating p γ 1 , γ 2 , µ 0 , µ 1 , ˜ µ 1 , µ 2 q and evaluating the outer expectation are independent and separate. Under the assumption that nuisance parameters p ˆ γ 1 , ˆ γ 2 , ˆ µ 0 , ˆ µ 1 , ˆ ˜ µ 1 , ˆ µ 2 q are consistent, R ' A ` B converges to zero at a rate O P p 1 { a | D p k q |q by Lem. 2.

Before manipulating C and deriving its large sample behaviour, consider the following intermediate results:

and similarly E P r ˆ µ 0 s ' E P r γ 1 ˆ µ 1 s . Note further that,

and similarly E P r γ 2 ˆ µ 2 s ' E P r γ 1 ˆ ˜ µ 1 s .

Now consider C ' C 1 ` C 2 ` C 3 where,

where the last equality follows from the result above. Therefore

For the upper bound, consider the following definition of nuisance functions and estimators:

We first verify that the population level value of the regression estimator coincides with the lower bound ψ ℓ z . This can be seen with the following derivation,

The arguments of the proof are now analogous to that of the lower bound. We can conclude therefore that,

Prop. 9 restated. Suppose either ˆ γ 1 ' γ 1 or ˆ µ 2 ' µ 2 and that either ˆ γ 1 ' γ 1 or ˆ ˜ µ 1 ' ˆ µ 1 . Then, ˆ T DML P t ˆ T DML ,ℓ , ˆ T DML ,u u is an unbiased estimator of the corresponding bound defined in Alg. 1.

Proof. Consider the estimated value of T DML ,ℓ following the procedure in Def. 5,

Under the assumption that,

The bias of the estimator is given by,

The last equality follows from the derivation of (C) in the proof of Prop. 8.

If either ˆ γ 1 ' γ 1 or ˆ µ 2 ' µ 2 and that either ˆ γ 1 ' γ 1 or ˆ ˜ µ 1 ' ˜ µ 1 then this expression equals 0 and we have that,

An analogous result holds for the upper bound.

C Details on experiments

This section provides details of the data generating mechanisms used for synthetic experiments, and implementations.

As described in the experimental section, for estimating nuisances p γ , µ q we used Gradient Boosting models for classification and regression where appropriate. We implemented the models using Python using the commands GradientBoostingClassifier() and GradientBoostingRegressor() using default hyperparameters. We systematically bound probability values in the interval r 0 . 01 , 0 . 99 s to avoid propagating potentially large approximating errors.

C.1 Simulations

We make use of the following 4 data generating systems, summarized by the graphs in Fig. 4. In this section, we use the notation t X u that equals 1 if the statement t X u is true, and equal to 0 otherwise.

✶ Example 1. The data generating mechanism for the first scenario is given by:

where,

The outcome Y is then re-scaled to lie in the r 0 , 1 s interval. The target for estimation can be derived with Alg. 1 and equals, for the lower and upper bound respectively,

where,

The outcome Y is then re-scaled to lie in the r 0 , 1 s interval. The target for estimation can be derived with Alg. 1 and equals, for the lower and upper bound respectively,

Example 3. The data generating mechanism for the third scenario is given by:

where,

The outcome Y is then re-scaled to lie in the r 0 , 1 s interval. The target for estimation can be derived with Alg. 1 and equals, for the lower and upper bound respectively,

Example 4. The data generating mechanism for the fourth scenario is given by:

where,

The outcome Y is then re-scaled to lie in the r 0 , 1 s interval. The target for estimation can be derived with Alg. 1 and equals, for the lower and upper bound respectively,

C.2 Health Campaign Evaluation

The data was curated from anonymous from Colombia, Peru and Mexico, using a web platform [30]. The exact questions considered in the survey can be found in [30]. The authors made the data available under a Creative Commons license 5 and is currently hosted by Kaggle as a c.s.v file, accessible through the following link: kaggle.com/code/mpwolke/obesity-levels-life-style/ .

T

S

F

H

Y

E

M

A

(a)

G

T

S

F

H

Y

E

M

A

(b)

G

π

T

S

F

H

Y

E

M

A

(c)

G

π

H

E

We considered a number of pre-processing steps. In particular, we created a BMI variable from weight and height data that were a posteriori removed from the dataset, imputed (with median or mode among observed instances) missing values, scaled continuously-valued covariates by subtracting the mean and dividing by the standard deviation, and scaled the outcome BMI to lie in the interval r 0 , 1 s . Several covariates were ignored, including data on the consumption of water, consumption of vegetables, consumption of food between meals, consumption of alcohol, and number of main meals. These may be interpreted as unobserved variables, potentially confounding other relationships in the data as specified by the assumed causal diagram. The data from a total of 2111 individuals was recorded in this study.

The bounds on the effect of a policy π α : ' t π α H , π α E u acting on H and E , with new assignments defined as t P new p H ' rarely q ' α u , P new p E ' regularly q ' α u are approximated from the expression in Prop. 4. Following this proposition we find that Age p A q , Smoking p S q , calories consumption Monitoring p M q , and Family history with overweight p F q , that is W ' t A,S,M,F u , form a partial adjustment set for π α E given that p Y | ù d E | W , H q G π H E . We can verify also that Z ' t T u is a partial instrumental set for π since p Y | ù d T q G π . For illustration, the corresponding mutilated diagrams are shown in Fig. 7. Finally, we partition the data equally to obtain two sets of samples for training first and second stage classifiers and regressors, respectively, obtaining a first estimate of bounds, before then switching the role of the two data samples and averaging over the resulting estimates.

D NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made match the theoretical and experimental results presented in the paper. The impact statement in the Appendix reflects how much the results can be expected to generalize to other settings.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include several remarks in the paper to qualify our contribution, e.g. the remark in Sec. 3, and include an Impact Statement in the Appendix to more thoroughly describe the limitations of our analysis, assumptions, and applicability in real-world settings.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theoretical statement are quoted in full in the paper. We have attempted to provide an example to illustrate the significance of each theoretical statement and highlight its implications. The formal proof of all statements is given in the Appendix.

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the data generating mechanisms, details of the target of estimation and information as to what python functions and libraries can be used to fit the proposed estimators. We do not, however, disclose an open source implementation of the proposed methods at this moment.

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] and [No]

Justification: The data is publicly available and we have provided full details as to where to access the data and how to run the synthetic data generation pipeline. The code will not be open sourced at this moment but we believe to have provided sufficient details to reproduce our results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provided details where applicable. In our case, data splits, hyperparameters, optimizers, etc., are not significant for the implementation the method.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report our results as box plots that include several statistics, e.g. median, 25th, 75th, percentiles, etc., to summarize our results for different seeds of the data and algorithm.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: Experiments were run on a single CPU in under a few minutes of wall time.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the guidelines and we do not think that our work presents any notable concerns.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We include a broader impact statement in the Appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing asset

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The authors develop graphical characterizations and estimation tools to bound the effect of policies given a causal graph and observational data collected in non-identifiable settings.

STRENGTHS:
Strengths and contributions: (1) derive analytical bounds for general probabilistic and conditional policies that are tighter than existing results, (2) develop an estimation framework to estimate bounds from finite samples, applicable in higher-dimensional spaces and continuously-valued data. They further show that the resulting estimators have favourable statistical properties such as fast convergence and robustness to model misspecification.

WEAKNESSES:
Please refer to Questions.

QUESTIONS:
Can the authors highlight the main challenge/novelty of the proposed bound for policy compared to bounds for treatment effect?

LIMITATIONS:
Please refer to Questions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a partial identification method for the value of policies that intervene on a potentially multivariate set of variables in a known causal graph. It searches for adjustment sets or IVs that apply to one set of variables intervened on conditioned on others.\n\nSTRENGTHS:\nExpanding the reach of partial identification methods to account for new kinds of action/covariate spaces is a valuable contribution. The paper is well-executed, with clean algorithmic strategies and statistical guarantees for the double ML estimator. I also appreciate the application to evaluating health campaigns -- although the bounds obtained in this case are not incredibly tight, they show that the method could provide suggestive evidence in a setting of practical interest.\n\nWEAKNESSES:\n(1) It would be valuable to have more intuition about what the method is doing. My heuristic understanding right now is that it is combining bounds for policies that intervene on two sets of variables, one of which can be made to satisfy a no-unobserved-confounders type assumption given the graph, and one of which cannot (and hence we get fairly trivial bounds for that portion of the treatment variables). What would the authors say is the right way to understand the strategy? Similarly, it would be useful to contrast explicitly with, e.g., previous strategies for partial identification with discrete variables. \n\n(2) The majority of the experiments are spent on evaluating accuracy of the DML method at estimating the bounds. This is valuable, but for partial identification methods the more important question is likely to be whether the bounds are informative. It would be more useful to have experiments that assess what kinds of graph structures lead to more or less informative bounds, to guide potential users about when the method is likely to be useful, even if it meant pushing some of the current experiments content to an appendix\n\nQUESTIONS:\nSee (1) above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper proposes a new method to partially identify policy effects using a known causal graph and observational data. In theory, the paper derives general bounds for probabilistic and conditional policies, generalizing the existing results on discrete atomic policies. To estimate the bounds, the authors extend the well-known double machine learning framework to their setup. Finally, simulations are provided to demonstrate the convergence rate of the estimators. Then, the estimators are applied to estimate bounds for a range of policies in an obesity study.

STRENGTHS:
The paper is overall well-written and organized. The contributions of the paper are clear in contrast with previous works. The authors not only derive new theoretical bounds but also provide an efficient double machine learning method to estimate the bounds. Finally, experiments are provided to demonstrate the robustness against the issue of model misspecification.

WEAKNESSES:
1. I think the paper can discuss a bit more about the cases in which the policy effect is identifiable, given a known graph and observational data, before moving to partial identification. Generally, partial identification is valuable when the assumption required for point identification may not hold in practice. The authors can potentially discuss the assumption and highlight the value of their method.
2. I didn't see any assumption required to derive the natural bound and its extension proposed by the authors. If the bound is assumption-free, would it be too conservative to use in practice? The authors should clarify this. 
3. The estimation strategies in the paper are quite standard. The authors should explain why this part is novel compared to the existing literature.

QUESTIONS:
N.A.

LIMITATIONS:
N.A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new framework that uses structural causal graphs for estimating the effect of policies under unobserved confounding. Their approach first derives tighter analytical bounds on the estimates of the effects of the policies under the structural causal graph setting and the paper uses the results to develop estimators of these bounds using the double machine learning toolkit. The provide numerical experiments to show their approach produces good estimates of the effects of the policies under different settings such as misspecification.\n\nSTRENGTHS:\nThe paper tackles an interesting problem of estimating the effect of policies under unobserved confounding and develops an effective structural causal models approach. The paper provides good discussion on the analytical bounds and is able to study multiple settings in the numerics.\n\nWEAKNESSES:\nThe paper is somewhat hard to follow as the authors of the paper assume readers are familiar with existing notation for structural causal graphs. For example, they assume readers know the notation \"an, de\" for such graphs. Much of the notation is defined ahead of time, but without context for how and where they are used throughout the paper. Additionally, some new notation appear without explanation and are left to be interpreted by the reader. For example, in the partial identification section, $\\bf{X}_1, \\bf{X}_2$ are introduced, but the paper provides minimal intuition for what each set of variables means or should mean. As a result, the paper in general is hard to follow as it is overloaded with notation but limited intuition. Potentially, more concrete examples could be provided. The partial identification section in particular could benefit from figures of graphs that satisfy definition 3. \n\nIn the numerics, it may be helpful to develop some sort of baseline to demonstrate how the new estimation framework out performs existing approaches. \n\nThe paper also seems to have various typos and broken references. There are missing citations at line 67 and 311 and a missing proposition reference on line 316.\n\nQUESTIONS:\n1. Can more detailed be given on differences between $T^{TW}$ and $T^{REG}$ that can explain the differences in the numerics? Is $T^{DML}$ derived from one of the two estimators combined with the DML toolkit and if so which one is used? \n2. Since the approaches provide an upper and lower bound on the effect of the policies, how do you select an estimate if the difference between the bounds is large?\n\nLIMITATIONS:\nLimitations have been addressed by authors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}