Towards Estimating Bounds on the Effect of Policies under Unobserved Confounding
Authors: Alexis Bellot
Silvia Chiappa
Venue: NeurIPS 2024 poster (2024)
Keywords: causal inference
estimation
bounding
unobserved confounding
Primary area: causal_inference
TL;DR: We introduce bounds and estimators for the effect of policies given observational data and a causal graph of the system.

Abstract: As many practical fields transition to provide personalized decisions, data is increasingly relevant to support the evaluation of candidate plans and policies (e.g., guidelines for the treatment of disease, government directives, etc.). In the machine learning literature, significant efforts have been put into developing machinery to predict the effectiveness of policies efficiently. The challenge is that, in practice, the effectiveness of a candidate policy is not always identifiable, i.e., not uniquely estimable from the combination of the available data and assumptions about the domain at hand (e.g., encoded in a causal graph). In this paper, we develop graphical characterizations and estimation tools to bound the effect of policies given a causal graph and observational data collected in non-identifiable settings. Specifically, our contributions are two-fold: (1) we derive analytical bounds for general probabilistic and conditional policies that are tighter than existing results, (2) we develop an estimation framework to estimate bounds from finite samples, applicable in higher-dimensional spaces and continuously-valued data. We further show that the resulting estimators have favourable statistical properties such as fast convergence and robustness to model misspecification.

Towards Estimating Bounds on the Effect of Policies under Unobserved Confounding
Alexis Bellot*
Silvia Chiappa
Google DeepMind London, UK
Abstract
As many practical fields transition provide personalized decisions. data in- creasingly relevant to support the evaluation of candidate plans and policies (e.g. guidelines for the treatment of disease, government directives, etc.). the machine learning literature, significant efforts have been put into developing machinery to predict the effectiveness of policies efficiently. The challenge is that, in practice, the effectiveness of candidate policy is not always identifiable, i.e., not uniquely estimable from the combination of the available data and assumptions about the do- main at hand (e.g., encoded in a causal graph). In this paper, develop graphical characterizations and estimation tools to bound the effect of policies given a causal graph and observational data collected in non-identifiable settings. Specifically, our contributions are two-fold: (1) we derive analytical bounds for general proba- bilistic and conditional policies that tighter than existing results, (2) develop an estimation framework to estimate bounds from finite samples, applicable digher-dimennional spaces and continuously-valued data. We further show that the resulting estimators have favourable statistical properties such as fast convergence and robustness to model misspecification.
Introduction
Understanding how to act upon the world around us requires humans and artificial systems to thoughtfully evaluate the effect of different plans, policies, and interventions one might consider. Data and methods that facilitate this process are increasingly relevant for high-stakes decision-makinn notably in medical care with the rise of precision medicine [41], but also education (25), law
enforcement (13), public policy [44], and economics (2). The interventions that practitioner might consider, could consist of complex policies where a variable is set to follow a conditional or stochastic relationship depending
other variables in the system. For example, policy-makers might contemplate higher taxes on processed foods and stronger campaigns on its health risks targeted to overweight individuals. A sensible question in this context could be "whaat is the effect of policy that reduces the consumption of processed foods by 50% in people with a body mass index above 30?". Contrary atomic (also called hard) interventions that force particular value, this policy suggests a softer intervention.
The identifiability and estimation of policies from data is widely studied problem in the reinforce- ment learning [37] 39 and causal inference [2] [2 [5] 31 32 40) literatures. Despite the generality entailed by many common approaches in these fields, they often rely impractical condition: the assumption that the effectiveness of the policy is uniquely computable from the observed data and
*Correspondence to Alexis Bellot: abel fot @google com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

assumptions about the data generating process. This can be violated in real-world settings subject to unobserved confounding, and lead the non-identifiability of the effectiveness of given policy. This holds irrespective of the number of samples collected so that in reality multiple answers for the effect of a policy may be equally plausible. In their seminal work in the early 1990's, s, Manski and Robins [27 33 showed that, nevertheless, the effect of an atomic intervention may always be bounded in a non-trivial interval (i.e., probabilities strictly contained in [o, 1]. irrespective of unobserved confounders or the causal structure underlying the variables involved. Causal effects are therefore in general said to be partially identifiale, i.e., one can derive bounds that shrink the range of a priori possible values for a given effect and potentially serve useful support for decision- making.
Starting from this insight, the problem of partial identification has been gaining growing attention in the literature. To improve upon these bounds, one common restriction on the data generating mechanism is to assume knowledge of causal graph describing the phenomenon of interest. Several results have derived progressively tighter bounds by exploiting the independencies in observational For instance, Balke and Pearl [ (and subsequent refinements, e.g., [14 48) defined polynomial optimization programs to derive bounds that are provably optimal. Tighter analytical bounds have also been derived in selected settings, such as for the "instrumental variable" graph 47 and discrete systems with general graphs or equivalence classes [5) 46) These results, however, are almost exclusively concerned with atomic interventions and small systems of discretely-valued variables. Despite their generality, there is still a gap towards making the vartial-identification and estimation of the effectiveness of policies practical. In particular, making scalable inferences from finite samples with continuous outcomes and higher-dimensional covariates is not possible with existing tools.
and interventional distributions implied by the causal graph 4
46
48
This paper aims to provide novel estimation framework to support decision-making in non- identifiable settings, overcoming some of these challenges. We introduce several new results for the partial identification and estimation of the effect of stochastic and conditional policies from combination of observational data and assumptions on the domain, encoded in a causal graph. Our contributions may be summarized as follows. We introduce several graphical criteria to derive new analytical bounds on the effect of policies with continuous outcomes and covariates, that improve upon the non- con-paramttric bounds of (27 33 and [47). Given these analytical bounds, we then con- struct estimators leveraging the double machine learning [10] toolkit for scalable inference, and demonstrate that estimators exhibit favourable statistical properties such as robustness to noise and fast convergence. These results are applicable to arbitrary stochastic or conditional policies that an investigator might design, and high-dimensional covariate spaces. Recent work has made progress is developing powerful estimators for identifiable causal effects 66 8 9 20 36 49]. However, these estimators are not applicable in non-identifiable settings. To our knowledge, the proposed estimation machinery is the first result for bounding the effect of policies given a causal diagram.
Preliminaries. We use capital letters to denote variables (X), small letters for their values (x:). bold letters for sets of variables (X) and their values (æ), and use supp to denote their domains of definition (x supp (X) is the indicator function that equals 1 if the statement {X z) is true, and equal to otherwise.
The framework we use to underpin the estimation of the effect of policies rests on structural causal models (SCMs) [3I Def. 7.1.1]. An SCM M is tuple endogenous (observed) variables, U is set of exogenous latent variables, and F = (Jv]vev is a set of functions such that fv determines values of V taking as argument variables Pav = V and UV = U. i.e. V - fvvPaa Uv). Values of U are drawn from an exogenous distribution P(u). We assume the model to be recursive, i.c. that there are no cyclic dependencies among the variables. Graphically, each SCM M is associated with a causal diagram & over V, where V - W V appears as an argument of fw in M, and V + +----> W if Uv n Uw # Ø. L.e. V and W share an unobserved confounder.W. We will use graph-theoretic family abbreviations represent graphical relations, e.g. an, de, etc. Moreover, for a causal diagram 9 over V, the F--ower-maniiplattion of & deletes all those edges that out of variables in X, and otherwise keeps & as it is. The of G deletes all those edges that are into variables in X, and otherwise keeps G as it is. The resulting graph is denoted as 9x. We use l to denote d-separation in causal diagrams [31 Def. 1.2.3].
P(U)), where V is a set of
resulting graph is denoted as Gx. The --upper-manipullen
2

---

For a sample set D = {2(*) /ibl..... P. we use Eplf = (i/n) We use I (Brl(s(V)]] f op(rn) denotes that a function f is consistent estimator of at a rate Tno f f = Op(rx) denotes that it is bounded in probability at rate Tn.
2 Partial Identification of the Effect of Policies
policy π over a subset X V is a sequence of decision rules or plans π := [xx]XXx for determining the assignment of variables X. In its most general form, every xx : suppx x suppcx [o, 1] is a probability mapping from domain of Cx = C c V to the domain X.
Qualitatively different types of interventions may be modelled with лx. Specifically, a deterministic intervention setting X := g(cx) based the values of Cx be encoded as xx(x cx) := lolex) (z) while a probabilistic intervention may be written x (x Cx) := Pinduced xx (X x c.). The intervened model MMr represents different regime in which the assignment ( x is replaced with the assign- ment induced by For an outcome Y E V, the interventional distribution PM2(1 (Y), equivalently Pa (Y),, is defined as the distribution over in MM. will be useful to adopt the notation π Xxxxx Tx (z cx) ) to denote the product of policy assignments individual variables subject to intervention in π.
)x)
(a) G
(b) 9
Figure 1: Graphs for Example
Example (lllustration of policies). In the context of a public health program, let X be a measure of an individual's weekly physical exercise routine, C an individual's family history of metabolic diseases, and Y hospital admissions related heart disease. Consider the causal diagram Fig. [ap currently, an individuals exercise voluntarily. which depends on unobserved factors that could be associated with risk of heart disease. The government is considering an incentive plan aimed at increasing the frequency of exercise depending individual's s family history of metabolic diseases (with some probability). The proposed intervention can be encoded as π = {πx} and sets the new assignment J'x such that the variable follows the pre-specified distribution. Graphically, this policy is represented by Fig. [Tb (with the new edge corresponding to the implementation of the policy highlighted in blue) and may be evaluated with the following quantity,
EP,[Y]- = Ex, c| P)
The inferential challenge is that this quantity is not uniquely computable from P(x, y, and G. requiring more complex and nuanced notions of policy evaluation.
Formally, we interested in the evaluation of the effectiveness of plan or policy acting (potentially multivariate) discrete action X based on values of observed covariates C' e IRd, on an outcome of interest Y [0, 1
Definition (Average treatment effect). The effectiveness of a policy π on Y is Ep, [Y].
From the investigator's perspective, only the causal diagram G of the environment M is available. No assumptions about the form or shape of P(U) and F are made, but for the structural knowledge encoded in G. In general, there might exist multiple SCMs M that induce the same causal diagram G and entail P(V) but result in different values of [Y_ The identification of a set of possible solutions that contain the true value of Ep, [Y] leads the notion of partial identification.
Definition 2 (Partial Identification). The effectiveness of policy π is said to be partially identifiable from G and P(v) if
(P) & EPM & (0 for any M such that GM P. = P(v),
(2)
where (2b2, (²)) are functionals of P that are bounded away from 0 and 1, respectively.
The earliest bound on the effect of policies, the so-called natural bounds, were developed considering discrete atomic interventions, i.e. of the form π = (X), X [1x
2 An upper bound of 1 is assumed for simplicity; the following results could be generalized to more general bounded random variables.
3

---

X2
W
X2
X1
Y > X3
X2
(a)
(b)
(c)
(d)
Figure 2: Graphs used Sec.
and
Proposition (Natural Bounds). For any Ep(Y11(X)) & Ep, [Y] & EpI(Y 1X)x + +1.
Remarkably, these inequalities hold irrespective of the causal graph of the system G; result that dates back to [27 33]. Similarly, we could adapt the underlying proof strategy to derive functionals of P that bound the effect of more general probabilistic or conditional policies и,, given follows.
Proposition (Natural Policy Bounds (NPB)). For any π. Ep[Y & EP,[Y]] < Ep[(Y - 1
These inequalities also hold irrespective of the causal graph of the system. The natural bounds from 27 33 are a special case of the natural policy bounds (NPBs) by setting π := {πx Xex where 1g (X). We could show that the NPBs are tight in some cases. For example, in Example 1, provably, no better bounds for Ep, [Y] than those defined by Prop. 2 could be derived (a result given in Prop. T0 Appendix BB. For other systems that involve variables that are "separated" in G, however, better bounds may be derived by exploiting the implications the causal graph and definition of the policy the induced observational and interventional distributions. Consider the following example as a first illustration of this idea.
xx (X), and therefore
Example 2 (Tighter bounds with causal diagram). We are interested in evaluating the effect of a policy π := {xx, (æ1 c), (X2)} from P(r1, 22 y) and G in Fig. 2a In particular, this problem involves stochastic conditional intervention on X1 given C and deterministic intervention on X2; both variables having differing dependencies onto the rest of the system. We could show that,
Ep, [Y EPEx, [Y] > Ep|Yxx, (P).
(3)
The inequality gives an expression for the bound terms of P that exploits the fact that in- tervening on X2 does not influence Y, and is tighter that the natural policy lower bound as (P) > Ex[xxxx x [ P(X2 x2) Ep(Y xx, 1z, (X2)) (= NPB). For the upper bound we could similarly establish that,
Ey Ep I(Y 1)xx,++1== +1
(P).
(4)
which is smaller than the natural policy bound as (P) & EP((Y 1) (X2)] NPBP
This example, although relatively straightforward, serves to illustrate the potential of causal diagrams (and the constraints they imply the effect of policies) for defining tighter bounds.
3 Graphical Criteria for Partial Identification
This section aims to consider more general separation statements between variables encoded in a causal diagram and the decomposition they imply to provide a systematic algorithm to bound the effectiveness of policies. We start by introducing the notion of partial adjustment sets (Def, 33 that is applicable with multiple intervention variables.
(Tx,,Tx,) be a policy on (X1,X2) X2) with a conditioning set C. A set W C (XX, UXXUUXXUUYY is said to be partial adjustment set for nx, in G iY X2 W, C, C,X1)sxxxx>
Definition (Partial adjustment set), Let π :=
Proposition 3. Let π := ( xx,} be a policy mapping set of covariates C a set of treatment variables X2) Let W be partial adjustment set for nx, in G. Then,
Ep Y mT] & [P,[.] Ep[(Y - 1)x7] +1,
where 7 1X
1/P(X, C, W)
In words, partial adjustment sets are designed to exploit the unconfounded status of some intervention variables with respect to the outcome, and could be leveraged to derive tighter bounds when available.

---

Example 3 (Tighter bounds with partial adjustment sets), Consider the problem of evaluating the effect of policy
x (z) xx, })} from P(z), 22 u, y) and g in Fig. 2b
Following Def. 3 we could establish that W Ø is a valid partial adjustment set for xx, since we
can verify that g C, (2) X2'
Prop- 3 then gives us a valid expression for bounding the
effect of the policy,
[p] > Ep [Y C)]
( Ep[Y the NPB).
(6)
For the upper bound we similarly find that,
Ep, [Y] & Ep [(Y / C)] +1
(< Epl(Y 1) + 1, the NPB).
Next, we define a second useful notion, so-called partial conditional instrumental variables sets, that be exploited to evaluate bounds in z-specific distributions (Z) P())) instead of in P Pas described in Prop. E
Definition 4 (Partial conditional instrumental set). A set Z c V is said be a partial instrumental set conditional on R a policy π in 9 iY R)g..
For illustration, we give simple criterion below to show how this subgroup structure could be exploited derive tighter bounds.
Proposition 4. Let Z be an unconditional partial instrumental set with respect to a policy æ, i.e. (Y (/lLaZ)ss G. Then, max Ep (l=(Z)/P(Z) < EP, < min Ep((Y (1)12( +1
(8)
The following example shows that partial adjust- ment sets and partial conditional instrumental sets may be usefully combined to derive tighter bounds than would be available had each propo- been applied in isolation. Example 4 (Tighter bounds with partial ad- justment and instrumental sets). For this ex- ample, consider the evaluation of an atomic intervention π := 21 (X2), (,,, 1 (X2)) in the causal diagram G given in Fig. 2c Note that
Algorithm 1 Bounds for the effect of policies
Input: Graph G, policy π, outcome Y. Output: Bounds for Fr, [Y].
sition (Props. 3 and 4
1: Let C UXxx Cx K (XXCUYY)
/ 1. Omit redundant intervention variables. */
2: Let R X (Y) in 9x.
3: Let क =
/0 2. Find partial adjustment sets W. */
म == az (X). Following Def. 3 1 is a valid partial adjustment set for xx, since we can ver- ify that (Y
4: Initialize W - = R.
5: for R R do
X1 Ki) Further, we could verify that [Z} } is a partial instrumental set conditional on {X2) with respect to xx, since (Y L X2)g.x, These two separation state- ments in (manipulated versions of) & could be leveraged to derive a tighter bound than previ- ously considered:
6:
Let S = SR
7:
Let WR = An(CCU))KK in 9s.
8:
if (YLL | W. WR) in g3 then
9:
W = WUWR.
end if
11: end for
/ 3. Find partial instrumental sets Z. */ 12: T {R: WR E W), U = R.T. Initialize Z = Ø.
max Ep Y #1 (2) Z) < Ep, [Y]
< min E - -10 (Z) 2) + 1.
14: for Z K do
if (Y 1 W Z,T, T, C) in 9y then
To derive bounds a more systematic fash- ion, combining the notions developed so far, we present Alg. that recursively seeks to find partial adjustment and partial instrumental sets in an efficient and automatic manner.
16:
Z = ZUZ.
17:
end if
18: end for
/0 4. Return bounds. #/
19: Let y := 1x (2) PTT Z W
Intuitively, Alg. D seeks to recursively simplify
20: Let wh = YY
the query. First by omitting the intervened vari- ables that have no effect on the outcome; second
21: Let th EEp(( 1)>1 + 1.
by finding the set of intervened variables for which a partial adjustment set could be used to tighten the bound, and third by finding the set of variables that act as valid partial instrumental sets to evaluate bounds the most favorable conditional distributions rather than on the joint distribution.
22: Return bounds: (max, v/. ming 1/2)
5

---

Proposition 5. Alg. 7 is sound.
In words, Prop. says that for given causal diagram, policy क,, and joint distribution P(v), the average treatment effect [p,[Y] is contained in the bounds produced by Alg.
Proposition 6. Let be the number of variables and m be the number of edges in G. The run time of Alg. iis Ok((2² + m)).
Example 5 (Steps of Alg. 1.. For this example we consider evaluating a policy π == { (
C)J from observational data P C, 22, Z3. O. W, y) compatible with g in Fig. 2d by explicitly following Alg. [ On line 1, we define C = {C], K = (A, W). We start by omitting potentially irrelevant intervention variables by evaluating R (X1, X2) and noticing that Ep_ D EP, [Y] To find partial adjustment sets in line 4, we evaluate the for loop iterating
C) (- C)2 (- (-
{A, W but the if condition fails as Y is not d-separated from X1 given (X2, C, A, W.. We turn onto R X2. where S X1, Wx, W. The if condition is triggered as (Y 22 | X1, C, A, W) in 9 N2 and therefore we (A,W) (giving partial adjustment set). We continue with line 12, set T (x2), U (x1), and iterate over K {A, W in the for loop. For = A, we have that LY W, X2) in gax, the if condition is triggered and therefore we update Z {A). For Z = W, we find that (Y Hh W| A, X2) in 9 and therefore terminate the for loop. Finally putting together the pieces, we evaluate the bounds to be,
over R. For R = X1, we find that S = X2, Wx,
update W Wx,
max [Y & Ep, M min Er[(Y 1)7] + 1, 7 TRla(A)/P(XX. A WW,C)
(9)
Remark (Multiple bound expressions). Alg. iis designed make use of partial adjustment and conditional instrumental variable sets but it will return a single bounding expression. In general multiple different expressions could be derived for given causal diagram and data distribution. For example, given G in Fig. 3 and a policy æ on X2), Ep| PP(22 Wi)] and EplTY PPLTY/((XX (Xx2 W2)] give valid, but numerically different lower bounds for [Y]. The superiority of one bound over another might depend on the associations in the underlying distribution P(v). In this case, we could construct two different SCMs compatible with G that entail different orders in the numerical values for the two bounds, such that neither is always optimal. In general, we conjecture that finding an optimal analytical bound (using the proposed procedure) is undecidable from the graph alone.
X2 W1 W2
X
Figure 3: g
The observation in this remark motivates to develop a procedure to automatically enumerate all partial adjustment sets to facilitate the search for tighter bounds. The following proposition adapts algorithm by 43 to enumerate all partial adjustment sets.
the
Proposition (Enumerating partial adjustment sets). Let π :- (x,) be a policy on (X1, X2) with a conditioning set C. All partial adjustment sets may be enumerated in time Olk(k + m)) where & are the number of variables and m be the number of edges in g.
Remark (Related methods for policy evaluation). In parallel to the graphical approach for encoding structural assumptions about the domain at hand, a number of works have adopted sensitivity assumptions that quantify the degree of unobserved confounding through various data statistics, such as odds ratios, propensity scores, etc. A rich literature on sensitivity assumptions exists, including Tan's sensitivity model (38] and Rosenbaum's sensitivity model [34]. Under these models,
17 21 28 45), among others, present methods that achieve validity and rate properties for the resulting estimators. These approaches start with estimators that optimize the average or conditional treatment effect bounds subject to constraints implied by the sensitivity model. Several of these works leverage Neyman orthogonality techniques to obtain rate guarantees and doubly-robustmess properties. For instance, [17 28) study the estimation of bounds under Tan's sensitivity model which quantifies the degree of unobserved confounding through odds ratios, and propose estimators with various validity, sharpness, and favourable convergence rate guarantees. In contrast, [45] study the estimation of bounds under Rosenbaum's 's sensitivity model also deriving estimators with sharpness and fast convergence guarantees. Some works under these assumptions have also considered policy evaluation (as opposed to the evaluation of atomic interventions) under various sensitivity models in the context of Reinforcement Learning, e.g., 7 22 23 We interpret this line of work as complementary (applicable under different assumptions, i.e., sensitivity models rather than causal diagrams) to the techniques proposed in this paper.

---

Estimation of the Effect of Policies
This section aims to develop an estimation framework for the effect policy Ep, [Y] given finite samples from P that partially identifies its value. The key observation of this section is that multiple characterizations for estimation could be derived for given bound returned by lines 20-21 in Alg.
In first instance, parameterized by the probability ratio given in Alg. 1 defined by Y = (7) 72),
pW, == Brln2Y] (= v
7/2 := Tun, == i Z W C). (10)
In a second instance, parameterized by a collection of regression parameters = (No: Mi, (2) where
12 = |2 C, W 2 Ep[Y R,C, W,, 2], ji (R, W, Z) 7t(U C)p2 R C, W
(11)
(12)
= MI W, 2) = Eplii (R, W, W Z) T, C,C,W.W. Z].
(13)
No == Po(C, W, Z) Σ, Ci W. Z) C).
(14)
G.EG.. = Ep[po(C, W, =]] could be shown equal vig.
Both formulations define equivalent but different estimation targets for the lower bound and may be combined leveraging the double machine learning (DML) toolkit for more efficient and robust inferences
[0]
The following procedure is the main contribution of this section. It defines DML estimator for bounding the effectiveness [x,,[y [ of a policy π.
Definition 5 (DML Estimator). Given and G, let (R, W. C, Z, be defined as in Alg. 7 Consider finite sample of data D ~ P, randomly split into K folds. The k ''th partition of the sample denoted D(k) and D(-R) = D ()D((), For each k. learn approximate nuisances (frke px) with
D(-k), Then, define
max PIMLLEE min TDML,
(15)
to be an estimate for the bounds on Ep, [Y] where,
1
K
D-LLEE
E 1i2 f22, x EDO BialRii - AIN EDc [Ro.]
K
K
fDML, = 1
1
>
Epw (oo - p22,x Epw Pi. f11 1+1 EDO [oo
K
To analyse the error of the DML estimator, consider the case that nuisances be estimated consistently. Thus requirement is relatively mild in practice as accurate probability estimation employing off-the-shelff classification and regression methods is feasible in general. Its error with respect to the true bounds is given by the following proposition.
Proposition 8 (Error rates). Suppose the nuisance estimates (fa, f) are Legconnistent and bounded. Then, the error of the DML estimator fDM e (fDMIL, {f 3 in Def. 5 is given as follows
K Σ
fiDML 7-DML
RR + +0 (1 Tell - Hal) + Or (1 Alll - in
K
where Rg is a random variable that converges to zero at a rate Op(1//n).
In words, the DML estimator exhibits a robustness property since the eπrоr of DML. is bounded in probability at n -1/2 rate whenever the nuisances converge at rate n -1/4 Note that the term
FFor the upperbound. similar construction could be derived by defining Tw. Epl12(Y 1)] 1. R := [P(po W,=]] + with Y replaced by Y in the definition of μzz Both estimators could be shown to be unbiased, i.e. equal to eif, defined in Alg.
7

---

Setting No misspecfication
Setting 2:
Setting Misspciification of
Setting Slow muisance convergence
Misspocificaton of
DHL
0.1
190
2803
166 X0
Absolute
3
6.09
1.30
Sumple sizee (n)
Figure Experimental results on bounding the effectiveness of policies with the proposed estimators. Different rows highlight evaluations on different data generating mechanisms: the first row tests estimation with a partial instrumental set, the second row tests estimation with partial adjustment set, the third row tests estimation with a high-dimensional partial adjustment set (W e I gg100), and the fourth row estimation with combination of partial adjustment and instrumental sets.
|| - fi.x| quantifies the error approximating the conditional expectation Eq. (13 for a given üj estimated at that stage, and that the estimation error of 222 and î2 are equivalent since they are deterministic transformations of each other. The following proposition corolloray that shows that the DML estimator is unbiased under misspecification.
Proposition 9 (Bias under misspecification), Suppose either 71 71 μo2 μ2 and that either i1 1 or îu Ω1. Then, fIMM e (TDDMI.t fiDMM..u) 3 an unbiased estimator of the corresponding bound defined in Alg.
Experiments
This section evaluates the quality of policy effect estimation from finite samples. Our goal is to illustrate the computation of bounds and provide empirical evidence of the fast convergence and robustness to misspecification of estimators.
Finite sample bounds are estimated with gradient boosting classification and regression models (for conditional expectations) by taking sample averages (for unconditional expectations). We truncated estimates of probability mass functions in the interval [0.01, 0.99] to ensure positivity.
We assess the quality of an estimator I by computing the absolute average error (AAE) with respect to (a proxy for) the true bounds (estimated practice with larger sample sizes), i.e., AAE T ming 1b21 ! max, v'gl. Throughout, we report various statistics: 25th, 50th, 75th percentile, etc., across evaluation runs with ten different random seeds. Further details of the simulations are provided in Appendix
5.1 Synthetic Simulations
The synthetic simulations consider 4 data generating mechanisms constructed according to the graphs Fig. 4 They highlight the use of partial instrumental sets, partial adjustment sets, high-dimensional partial adjustment sets, and combinations of partial instrumental and adjustment sets of varying dimensionality. The task is estimate bounds on the effectiveness of policy ( defined as follows.
xx, = xx. (X, = 1) = 0.5, Tx. = (X2 c) ) = 1/(1 expl-ct)
(16)

---

That is the policy assigns X1 E {0, randomly with probability 0.5, and assigns X2 E {0, 1] as function of C' with the probability of X2 increasing with the value of C.
To estimate bounds on Ep r we consider the proposed estimators, labelled: TPW estimated with the nuisances in Eq. 10
TRG estimated with the nuisances in Eq. (11), and 7DML estimated with the procedure in Def. 5 Our evaluations test performance across 4 different settings designed to
highlight various properties.
Setting I: All nuisances estimated correctly. This setting aims to show that all estimators converge to the bound interest.
Setting 2: Nuisances Sy are sampled from a uniform distribution to induce misspecification in the estimation of
Setting 3: Nuisances ja are sampled from uniform distribution to induce misspecification in the estimation of J.. Settings 2 and 3 aim to demonstrate the doubly-robustness property of the DML estimator.
Setting 4: Noise is introduced in the estimation of all nuisances (7y p1) to emphasize error due to finite sample variation. Specifically, noise e ~ (ormal ", n .n"", ), 1/4, that induces slower rate of convergence as a function of sample size. inspired by [24 20]. This setting aims show that the fast convergence behavior of the DML estimator compared to competing estimators.
The results are given in Fig. We observe that across all data generating mechanisms, estimators improve with the size of the dataset and converge under no misspecification (Setting I) to the underlying bounds. is interesting to note also the differing accuracy of estimators in the small sample regime. TPW based the estimation of ratio of probabilities, can be unstable with low sample sizes if the ratio denominator is estimated to be close to zero while TREG based on a sequence of regression tasks, tends to be better behaved. TDML in contrast is constructed of elements of TPW and 7REG In particular, note in Def. 5 the use of nuisances la and 7. As a result, 7DML has quite a different performance profile. Settings and 3 in Fig. 4 show that the DML estimator 7.ML is robust to misspecification in either the nuisances or that highlights the robustness property. Further, when decaying noise is introduced in the estimation of nuisances (Setting 4), the DML estimator outperforms in general with a faster convergence rate. We also observe that performance remains close to optimal with high-ddmensiionl variables W, demonstrating that the DML estimator provides practical toolkit for bounding in practice.
combination
5.1.1 Width of Bounds According to Different Graphical Criteria
This section evaluates the width of the bounds returned by exploiting the different graphical criteria provided in Sec. 3
yop. irmomeerral only thep 41 Atseetent onty Ihts
The simulations are based on the data generating mechanism described by causal diagram illustrated in the fourth row of
Fig. We consider evaluating the policy in Eq. (6) and
Upperboand
compute the bounds obtained by applying Prop. 2 (most con-
Eg[V]
servative), Prop. 3 (using the partial adjustment set W only).
Lower bozdd catrato
Prop. 4 (using the partial instrumental set only), and finally Alg. (that combines all propositions and is the proposed ap- proach). Fig. gives the results over 10 seeds of the data and across multiple data sizes, highlighting the gain achieved by exploiting the causal structure using the proposed approaches.
Sample
Figure 5: Width of bounds.
Remark (Actual width of bounds in practice). The majority of our empirical evaluations are spent on evaluating the accuracy of different methods at estimating bounds, without addressing whether the returned bounds are actually informative. In practice, the graph structure can play an important role in tightening bounds but the actual width of the bounds particular problem are primarily driven by the distribution of data. Here is an example to make this more concrete.
For given policy n, Prop. 2 defines tight bounds (under some circumstances) on Ep, [Y]. The width of this bound is given by 1 Ep Φ] that is ultimately driven by P(x). This term may therefore evaluate to anything between and depending on P(x)) and π.
For the first row, the policy evaluated is: xx xx 1 1/(1 expl-ct).

---

Gument BM lovels
044
M
Expected BMI levels Ep(Y==
(a) Grieesity
(b)
Figure 6: Health campaign evaluations.
For more complex causal structures, the bounds proposed in Props. 3 and 4 (and Alg. 1 reduce the width of the interval above. Loosely written, from 1 - Ep [ to Ep || X for some that is function of the joint distribution P and the structure the graph. But again, the actual width of the interval is ultimately determined by the values of P and that may be large or small depending on the value probabilities involved.
5.2 Evaluating Health Campaigns
This section illustrates the evaluation of lifestyle recommendations for the mitigation of obesity in individuals from Colombia, Peru and Mexico 30). The data was collected from anonymous users using web platform, and includes reported obesity levels measured according to BMI (Y), age (A), smoking status (S), frequency of consumption of high caloric food (H), whether individuals monitored their calorie intake (M), family history being overweight (F). exercise frequency (E), and time using technology devices (T). Obesity
multi-factored medical condition for which several causes have been acknowledged in the literature; causal diagrams relating the variables above have been curated several related studies [ 9 We considered these findings to construct the causal diagram in Fig. that that we assumed for this example.
We aim to study the effect of a health campaign designed to lower the intake of high caloric food (H) and increase the frequency of exercise (E) on obesity levels (Y). For instance, we could hypothesize
that the campaign leads an increase in the observed proportion of individuals rarely consuming of
high caloric food (H) from 0.12 to 0.5 and that of individuals doing exercise (E) regularly from 0.05
to 0.5. These statements could be formulated as a stochastic policy лº := (T TB) acting on H and E,, with new assignments,
H == क [H = zarely) = : = x reqularly) = a.
(17)
We consider the evaluation of expected BMI levels E,,- [Y] that range from 12 to 50 in the population, with mean of 29.3. First note that this or other policies acting on (H, E) are not identifiable due to
the bi-directed edge {H 4----- Y). but may nevertheless be bounded using Alg. We find that
max E & Ep,, MY & min Ep[(Y - 1)>l+ +1 y: म.. ((TT)/((E.T S,
(18) To illustrate the inference of. bounds with the DML estimator, we consider evaluating policies with 0.2, 0.4, 0.6, 0.8. Fig, 6b gives the results. The end-points of the intervals denote estimated lower and upper bounds. We see that policies that promote a healthier lifestyle (larger values of a) expected to reduce obesity levels on average but substantial uncertainty is still expected.
6 Conclusions
The evaluation of policies is arguably one of the critical ingredients enabling more personalized decision-making systems. When the effect of policies is not identifiable, bounds can provide an effective support for making informed decisions. In this paper we developed partial identification and estimation tools for bounding the effect of (stochastic or conditional) policy given data and assumptions encoded in a causal graph. We introduced several graphical characterizations that induce tighter bounds, and developed an estimation framework that exhibit robustness to noise and fast convergence. The results of this paper were illustrated through synthetic simulations and a real- world health campaign example for the reduction of obesity levels.
10

---

Acknowledgements
We thank the anonymous reviewers for helpful comments.
References
[1] Steven Allender, Brynle Owen, Jill Kuhlberg, Janette Lowe, Phoebe Nagorcka-Smith. Jill Whelan, and Colin Bell. A community based systems diagram of obesity causes. PloS one, 10(7):e0129883, 2015.
[2] Susan Athey and Guido W Imbens. The state of applied econometrics: Causality and policy evaluation. Journal of Economic perspectives, 31(2):3-32, 2017.
[3] Vahid Balazadeh Meresht, Vasilis Syrgkanis, and Rahul G Krishnan. Partial identification of treatment effects with implicit generative models. Advances in Neural Information Processing Systems, 15:22216-222885 2022.
[4] Alexander Balke and Judea Pearl. Bounds on treatment effects from studies with imperfect compliance. Journal the
the American Statistical Association, 92(439::1171-1176 1997.
[5] Alexis Bellot. Towards bounding causal effects under Markov equivalence. In The 40th Conference on Uncertainty in Artificial Intelligence. PMLR, 2024.
[6] Alexis Bellot, Anish Dhir, and Giulia Prando. Generalization bounds and algorithms for estimating conditional average treatment effect of dosage. arXiv preprint arXiv:2205. 14692, 2022.
[7] Andrew Bennett and Nathan Kallus. Policy evaluation with latent confounders via optimal balance. Advances in neural information processing systems, 32, 2019.
[8] Rohit Bhattacharya, Razieh Nabi, and Iya Shpitser. Semiparametric inference for causal effects in graphical models with hidden variables. The Journal of Machine Learning Research, 23(1):13325-13400 2022.
[9] Bryony Butland, Susan Jebb, Peter Kopelman, Klim McPherson, Sandy Thomas, Jane Mardell, Vivienne Parry, al. Tackling obesities: future choices-project report, volume 10. Citeseer, 2007.
[10] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters, 2018.
[11] David Maxwell Chickering and Judea Pearl. A clinician's tool for analyzing non-compliance. In Proceedings of the National Conference on Artificial Intelligence, pages 1269-1276. 1996.
[12] Juan Correa and Elias Bareinboim. A calculus for stochastic interventions: Causal effect identification and surrogate experiments. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 10093-10100, 2020.
[13] Elizabeth K Drake, Steve Aos, and Marna G Miller. Evidence-based public policy options to reduce crime and criminal justice costs: Implications in washington state. Victims and offenders, 2009.
[14] Noam Finkelstein and Ilya Shpitser. Deriving bounds and inequality constraints using logical relations among counterfactuals. In Conference on Uncertainty in Artificial Intelligence, pages 1348 1357. PMLR, 2020.
[15] Limor Gultchin. Virginia Aglietti, Alexis Bellot, and Silvia Chiappa. Functional causal Bayesian optimization. In Uncertainty in Artificial Intelligence, pages 756-765. PMLR, 2023.
[16] Yaowei Hu, Yongkai Wu, Lu Zhang, and Xintao Wu. A generative adversarial framework for bounding confounded causal effects. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2214--1212. 2021.

---

[17] Andrew Jesson, Sören Mindermann. Yarin Gal, and Uri Shalit. Quantifying ignorance in individual-level cafsel- estimates under hidden confounding. In International Conference on Machine Learning, pages
PMLR, 2021.
[18] Shalmali Joshi, Junzhe Zhang, and Elias Bareinboim. Towards safe policy learning under partial identifiability: A causal approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13012, 2024.
[19] Yonghan Jung, Jin Tian, and Elias Bareinboim. Estimating identifiable causal effects on Markov equivalence class through double machine learning. In International Conference Machine Learning, pages 5168-5179. PMLR, 2021.
[20] Yonghan Jung, Jin Tian, and Elias Bareinboim. Estimating joint treatment effects by combining multiple experiments. In International Conference on Machine Learning, pages 15451-15527. PMLR, 2023.
[21] Nathan Kallus, Xiaojie Mao, and Angela Zhou. Interval estimation of individual-level causal effects under unobserved confounding. In The 22nd international conference on artificial intelligence and statistics, pages 2281-2290. PMLR, 2019.
[22] Nathan Kallus and Angela Zhou. Confounding-robust policy improvement. Advances in neural information processing systems, 31, 2018.
[23] Nathan Kallus and Angela Zhou. Confounding-rowe robust policy evaluation in infinite-horizon reinforcement learning, Advances in neural information processing systems, 33:22293- 2020.
[24] Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. arXiv preprint urXiv:2004. 14497, 2020.
[25] Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic, Offline policy evaluation across representations with applications educational games. In AAMAS, volume 1077, 2014.
[26] Henry B Mann and Abraham Wald. On stochastic limit and order relationships. The Annals of Mathematical Statistics, 14(3):217-226, 1943.
[27] Charles F Manski. Nonparametric bounds treatment effects. The American Economic Review, 80(2):319-323, 1990.
[28] Miruna Oprescu, Jacob Dorn, Marah Ghoummaid, Andrew Jesson, Nathan Kallus, and Uri Shalit, B-learner: Quasi-oracle bounds on heterogeneous causal effects under hidden confounding. arXiv preprint arXiv:2304. 10577,
[29] Kirtan Padh, Jakob Zeitler, David Watson, Matt Kusner, Ricardo Silva, and Niki Kilbertus. Stochastic causal programming for bounding treatment effects, arXiv preprint 2XXx:2202..1806 2022.
[30] Fabio Mendoza Palechor and Alexis de la Hoz Manotas. Dataset for estimation of obesity levels based eating habits and physical condition in individuals from colombia, peru and mexico. Data in brief. 25:104444, 2019.
[31] Judea Pearl. Causality. Cambridge university press, 2009.
[32] Judea Pearl and James M Robins. Probabilistic evaluation of sequential plans from causal models with hidden variables. In UAI, volume 95, pages 444-453. Citeseer, 1995.
[33] James M Robins. The analysis of randomized and non-randomized aids treatment trials using new approach to causal inference in longitudinal studies. Health service research methodology: focus on AIDS, pages 1 113-159, 1989.
[34] Paul R Rosenbaum, P Briskman Rosenbaum, and Briskman. Design of observational studies, volume 10. Springer, 2010.
12

---

[35] Andrea Rotnitzky, James Robins, and Lucia Babino. On the multiply robust estimation of the mean of the g-functional. arXiv preprint arXiv:1705.08582 2017.
[36] Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In International conference machine learning, pages 4076-3085 PMLR, 2017.
[37] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[38] Zhiqiang Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, (214476):161-16333 2006.
[39] Philip Thomas and Emma Brunskill. Data-efficient off- off-policy policy evaluation for reinforce- ment learning. In International Conference on Machine Learning, pages 2139-2148. PMIR, 2016.
[40] Jin Tian. Identifying dynamic sequential plans. arXiv preprint arXiv:
[41] Anastasios A Tsiatis, Marie Davidian, Shannon T Holloway, and Eric B Laber. Dynamic treatment regimes: Statistical methods for precision medicine. CRC press, 2019.
[42] Mark J van der Laan and Susan Gruber. Targeted minimum loss based estimation of causal effects of multiple time point interventions. The international journal of biostatistics, 8(1), 2012.
[43] Benito van der Zander. Algorithmics of Identifying Causal Effects in Graphical Models. PhD) thesis, Universität zu Lübeek, 2020.
[44] Evert Vedung. Public policy and progrum evaluation. Routledge, 2017.
[45] Steve Yadlowsky, Hongseok Namkoong, Sanjay Basu, John Duchi, and Lu Tian. Bounds on the conditional and average treatment effect with unobserved confounding factors. arXiv preprint 1808: 09521, 2018.
[46] Junzhe Zhang. Designing optimal dynamic treatment regimes: A causal reinforcement learning approach. In International Conference on Machine Learning, pages 11012-11022, PMLR, 2020.
[47] Junzhe Zhang and Elias Bareinboim. Bounding causal effects on continuous outcome. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12207-12215, 2021.
[48] Junzhe Zhang, Jin Tian, and Elias Bareinboim. Partial counterfactual identification from observational and experimental data. arXiv preprint arXiv:2110.05690, 2021.
[49] Yao Zhang, Alexis Bellot, and Mihaela Schaar. Learning overlapping representations for the estimation of individualized treatment effects. In International Conference Artificial Intelligence and Statistics, pages 1005-1014. PMLR, 2020.
13

---

Appendix
Table of Contents
A Preliminaries, Related Work, and Impact Statement
15
A Preliminaries.
15
A.2 Related Work
15
Broader Impact Statement
B
Proofs
18
Proofs of statements in Sec.
18
B.7 2
Proofs of statements in See
25
Details on experiments
30
Simulations
30
Health Campaign Evaluation
31
NeurlP'S Paper Checklist
33
14

---

A Preliminaries, Related Work, and Impact Statement
A.1 Preliminaries
For the derivation of results, we use the notion of counterfactuals defined as follows.
For SCM M, arbitrary subsets of endogenous variables X, Y. the potential outcome of Y to intervention do(æ), denoted by Y (u), is the solution for Y with = u the sub-model MM be read as the counterfactual sentence "the value that Y would have obtained in situation U u, had X been x.". Statistically, averaging u over the distribution P(u) leads to the counterfactual variables Y, The distribution of the variable Y2 is denoted P(Y With this formalism, we can express all the quantities in the main body of this paper in the language of counterfactuals. For instance, the distribution of Y in sub-model Ma denotes the conditional expectation of f(Y)
alternatively be written P(Y=) Pz (Y).
Moreover, Ep()(Y,) z]. also written [=)
over ( 2). Similarly, the distribution over Y in the sub-model M, can be written (Y,)), or P(Yx) (Y) for short. See [3 Chapter 7] for further context counterfactuals.
Definition (The Axioms of Counterfactuals, Chapter 7.3.1 [31). For any three sets of endogenous variables X, Y, W causal model and æ, w in the domains of X and W, the following holds:
Composition: Wa x = implies that Y.. w,w Y..
Effectiveness: X x = Z.
Reversibility: Y2,w and Wz.y w imply that Yv
Theorem (Soundness and Completeness the Axioms Theorems 7.3.3, 7.3.6 (3I). The Axioms of counterfactuals sound and complete for all causal models.
The following rules to manipulate experimental distributions produced by policies extend the do- calculus and will be used for the proof of several theoretical statements [12]
Theorem (Inference Rules o-calculus 12].. Let G be a causal diagram compatible with SCM M, with endogenous variables V. For any disjoint subsets X. Y, Z = V, two disjoint subsets T, W = (iUYY) (i.e., possibly including X), the following rules are valid for any intervention strategies nx, Tz, and 7 such that 9 9xx='s have no cycles:
Rule (Insertion/Deletion of observations):
Pxx (3) w. = Pax w)
iT W) in 9nx
Rule (Change of regimes under observation):
Pax, ( z, w) = a Lyyl 2, w) if 2 W) in 9nx 2 and Imxis'z
Rule (Change of regimes without observation):
P w) Pax (y w) (f 1 W) in g
Z(W] and g nx n'x Z(W)
where Z(W) is the set of elements in Z that ancestors W in G
A.2 Related Work
Active experimentation by physically manipulating reality is generally not feasible for many conse- quential applications. This motivates the study of causal effect identification and estimation. The former studies the problem inferring whether a unique expression or bound of the desired causal effect could be derived from the available data and assumptions. The latter studies the problem of providing efficient estimators from finite samples to compute causal effects or bounds on causal effects in practice. In the following, review these two lines of work to better contextualize our contribution.
Partial identification. causal effects could be drawn without making identifying assumptions beyond the observed data 27]. Relatedly, analysis of bounds of causal effects was also provided for studies with imperfect
The natural bounds were derived to demonstrate that useful inference about
15

---

compliance under set of instrumental variable assumptions that are not sufficient however to identify the causal effect of interest [33]. These bounds may be written closed form and have recently been
extended as a more general strategy for bounding causal effects given assumptions encoded in causal diagrams discrete systems [5] 46]. Recent work has also considered bounds in closed form with
access to both observational and interventional distributions [18). These techniques were developed alongside a second line of research that employs a polynomial optimization program to compute causal bounds given causal diagram [4] They proposed a family of canonical models parameterized according to the causal diagram, reducing the bounding problem to a series of equivalent linear programs. [1]] further used Bayesian techniques to investigate the sharpness of these bounds with regard to the observational sample size. Recently, [4] 48) describe a polynomial programming approach to solve the partial identification for general causal graphs. More recent proposals consider parameterizations in the space of linear combinations of a set of fixed basis functions (9 and neural networks
3 [6
In parallel, a number of works have adopted sensitivity assumptions (as an alternative or in combi- nation with causal diagram) that quantify the degree of unobserved confounding through various data statistics, such as odds ratios, propensity scores, etc. A rich literature on sensitivity assumptions exists, including Tan's ' sensitivity model [38 and Rosenbaum's 's sensitivity model [4]. Under these
models, [17 [2 28 (45), among others, present methods that achieve validity and rate properties for the resulting estimators. These approaches start with estimators that optimize the average or conditional treatment effect bounds subject to constraints implied by the sensitivity model. Several of these works leverage Neyman orthogonality techniques obtain rate guarantees and doubly- robustness properties. For instance, [17 21, 28 study the estimation of bounds under Tan's 's sensitivity model which quantifies the degree of unobserved confounding through odds ratios, and propose estimators with various validity, sharpness, and favourable convergence rate guarantees. In contrast, (45 study the estimation of bounds under Rosenbaum's sensitivity model also deriving estimators with sharpness and fast convergence guarantees.
While some works have considered policy evaluation under various sensitivity models in the context of Reinforcement Learning, e.g., 7 22 23 most works target estimation of bounds on the effect of atomic interventions. We interpret this line of work as complementary (applicable under different assumptions, i..., sensitivity models rather than causal diagrams) to the techniques proposed in this paper.
Causal effect estimation. With a causal diagram as input, causal effect estimation has traditionally focused on subset of identifiable scenarios, relying on assumptions such as backdoor criterion or the availability of adjustment sets. Identification expressions in those cases are given by (sequential) covariate adjustments that have lead to statistically appealing estimators from observational data
32.. Notable examples are doubly robust estimators (10) 35 42). Recently these techniques have
been extended to settings identifiable from multiple experimental distributions [8] 19 20]. To our
knowledge, no estimation framework specific to the estimation of analytical expressions derived for bounds on the effect of policies or atomic interventions has been developed.
A.3 Broader Impact Statement
Our work investigates the conditions under which policies may be bounded from observational data. Issues of policy evaluation are central to fields of application involved in decision-maken as well as Al and ML. We believe that tool to bound the effect of stochastic and conditional policies in systems with unobserved confounding is an important addition to the scientifie toolbox. Reasoning instead without acknowledging for the potential of unobserved confounding may lead researchers to operate on a more hcuristical basis. For instance, the hypothetical effect of public policies might be misrepresented if informed by the evaluation of idcalized atomic intervention that is difficult to implement in practice. And, once implemented, might have unintended consequences. With this background, we believe that research on the partial identification and estimation of policies based on stochastic or conditional interventions can help scientists and individuals make more informed decisions.
In this work, we start from the assumption that a causal diagram that consistent with the underlying data generating system of interest is available. In general, this requires domain knowledge. While some of the bounds provided do not require full knowledge of the causal graph, whenever a d- separation is assumed its truth value should be justified by prior knowledge or experiment. It is
16

---

important also to make the distinction between the task of partial identification, that is inferring an expression for bounds on causal effects, and that of estimation. that is providing efficient estimators from finite samples to compute bounds in practice. This set of results concerns both of these tasks. The first objective of our procedure is to provide an expression for lower and upper bounds, irresecctive of the accuracy with which one is able to approximate P(V) from finite samples. The second objective is introduce efficient estimators for bounds on the effect of policies using finite samples from P(V). In higher-dimensional systems, the computational complexity of estimating the conditional distributions that define lower and upper bounds on causal effects is substantial challenge. Consequently, practitioners must exercise caution when deploying the proposed method in small sample scenarios where estimators may be inaccurate. Moreover, we have stated our convergence guarantees in the infinite sample limit, without quantifying the finite-sample estimation uncertainty. Finite-sample properties could be explored similarly to [24] given a particular choice emphasize that
of function class extend our results with pigh-probbbility bounds. Finally, simulations on real and synthetic data are provided for illustration purposes only. These results do not recommend advocate for the implementation of particular policy, and should be considered in practice in combination with other aspects of the decision-making process.
17

---

B Proofs
This section presents proofs for theoretical statements in the main body of this paper.
B.1 Proofs of statements in Sec.
Prop. restated. For any æ, Ep ( & Er, [Y] & RE[(( xX)] + 1.
Proof. Consider the derivation of the lower bound.
Ep[Ya]
Σ EP <'1P(x))
Er P()(
@ Eply x()(2)
Σ yPl P(x)
Σ yl ( P(y.
y.2
(1) follows by the composition axiom of counterfactuals.
Consider the derivation of the upper bound. It holds that,
Ep[Ya] = Σ Eplx x')((x')
x²'
Eplye <))(()) Σ EplYs x')((x')
Ep|Y x P()(()) + Σ P(x') x xxv
Er[Y P()) + P(x)
EP[Y1 [)] + Rp X)]
- (
(I) follows by the boundedness of Y, here assumed bounded by I for simplifying the derivation. Proposition 10. The natural bounds are tight in general.
Proof. We show this statement by providing a pair of SCMs that agree on the input observational distribution but evaluate the lower and upperbounds, respectively, specified by the natural bounds. Let M[) denote the space of SCMs that induce the causal diagram G in Fig. Ia| We introduce a pair of SCMs compatible with the causal diagram G that evaluate to lower and upper bounds respectively. Let X be binary, Y E [0, 1]. and C' E IR, and consider Mr, M(G) defined as follows,
fx(U1)
c fc(u2)
M1
fr fz c, U1, U2)
if fx(1).
y
otherwise.
and,
fx(u1)
fo(uz)
M2
(r w1, W2) (x==x((m)
y
otherwise.
18

---

For a PM, (u) (u), both SCMs agree observational distributions PM, (z. c) = PM2 (r, c). However the following derivations show that the interventional expectation E PMM [Y] do(x = 1)] differs across models: for M₁ equal the analytical lower bound, and for M2 equal to the analytical upper bound demonstrating that (in this case) the bound is tight. In particular,
EPM, [ do(x 1)]
EPM, [Y] x :. fx fx(u))
BPM, [Y] x 1, Ix
EP [Y 1 1]
EPpY x= 1] = =)
(x)]
E ( do(x = 1)]
PP |Y x = = fx(u))
+Bmgg [Y] x=
(u
EPMM [Y] x = 1] = 1) Er (
1,:1, fw:
-EM, [Y 1 - -] p(x=1)
Prop. restated. For any n, Ep[Y#] & Ep, [Y] & Er((Y 1) +1
Proof. Consider the derivation of the lower bound. Let π denote a policy from an arbitrary set of covariates C to X. By marginalizing over X, C we find that
Ep YJ Ep(Yx]
= Er[y {
w cl π
xx(x P(c) dedæ.
Xex
Cx denotes the subset of C' that is used to inform the intervention on X. Moreover,
Ey(Yn xc, c] (1)
Er
Σ EplYn x' c((') c)
EplYa x c[ c)
E a. c P( c).
(I) holds by Rule 2 in Thm. 2 by swapping the policy π by the do intervention do(X = æ): given that the policy acts on X taking as inputs C it is always true that Y C' G and g Oxx in any graph G. The following equality follows by marginalizing over the domain of X, which assumed discrete. (2) follows by the composition axiom of counterfactuals. By combining these two expressions we get
Ep
>
क( (x) a, P(x XeX
I
e) dedæ
TyP(y æ. P(x cc) dydedæ
Ep|Ya].
19

---

Consider the derivation of the upper bound. It holds that,
Ep(Y, æ, cl Ep(Y cl
Σ Brlyn a', cc) c))
Ep[Ya n. c. c) Σ
EppV2 x', P( c)
Er[Y x. P(x
c) + Σ
P(a' (c)
#'
Ep(Y a,
c.c] P( c) +
P(æ: c).
(I) follows by the boundedness of Y, here assumed bounded by for simplifying derivation. As above by combining this inequality with the decomposition of the policy effect we get
Ep, M <
<< P(c) (EL x, c(c c) +1 P(æ c)} dedæ
XeX
Er(Yn] π| 1 Ep(T]
= Ep[(Y - 1)
Prop. 3 restated. Let π := { & be a policy mapping a set of covariates C to a set of treatment variables
(2) Let W be partial adjustment set for xx, in 9. Then,
Ep Y & Rr, [Y] & Ep[(Y - 1) 1,
where 7 (X, C, W) 1/P(X2 C)
Proof. Consider the derivation of the lower bound. By marginalizing over X, C we find that x. cl
Ep, [Y] Ep[Yx]
EP[Y>
II
x cx) dedæ.
XeX
Cx denotes the subset of C' that is used to inform the intervention on X. Denote म := Ixxx x cx). Let X,, X2 be partition X such that W (XUUUUY is partial adjustment set for
xx2- Then,
Ep(Yx x. cl = Eply ExplYx xx
c
Ep [Y c, w(w] c) dw
Ep Ep|Yen æz, w. c(w c) dw
x2, w. c)} P(w c) dw Σ Epl|Ya ,/. c(
z
Ep(Ye, 11.22. w. c| Ø2 w, c(w c) dw
EplY æ1 22, w. c(, æ2, w. P(w c) dw.
(1) follows by the definition of X2 as a partial adjustment set: it holds that IY,,,X, W C,) 9x,x, which induces the equality [y([Y,,a c, wi] = Er 22, w. c]. The last equality fol- lows by the composition axiom of counterfactuals. Combining this expression with the decomposition of the policy effect we get,
(Y >
< P() æι, Ø2, w. P(x: Ø2, w. P(w c) dwodedæ I XEX
RyP(yy E1, w, P(x, x2 w. c(w c)P( dydedæ
= EEp(Y W. W)C).
20

---

For the upper bound, consider the following derivation,
Ep az c]
ErlYn 22, w, cl
c) dw
x2, w. c)) c) dw (EPlYYe 22, w, P(a
ErlYa 02, w. c( æ2 w, c(w c)
Σ EplYa 0/ w. 1, w. P(w c) dw
sitm
| [r æ1 22 w. c(x
æ2 w. c) P(w c) +
Σ
22 w, P(w c)} dw P(
2 ボ
S (Er|Y æ1 02, w. P(E
T22 w. P(w c) (1 P(æ1 T2 w. P(w c) dw
Ep|Y 0i. 22 w. P(1 22 w, P(w c) dw + P(z1 22 w. c)) P(w c) dw.
The inequality follows from the boundedness of Y and the rest of the arguments analogous to the lower bound derivation. Combining this expression with the decomposition of the policy effect implies that,
Ep, [Y] & Ep((Y /)x/ W. +1.
Prop. restated. Let be an unconditional partial instrumental set with respect to a policy π, i.e. (Y 1)g. Then,
Ep, [Y] max mgxx:)\YT1.(2)//12
Ep. Y & min (1inB((((--)))1((2))22 +1.
Proof. Consider the derivation of the lower bound. Let denote policy from arbitrary set of covariates to X. Given that (Y a Z).. by marginalizing over X, C we find that
Ep, [Y] Ep[Y=] EplY= x] = [p[|> x, 2,
1l xx(x c() 2) dedx. XeX
Cx denotes the subset of C that is used inform the intervention X. Moreover,
(1) EplYa c
Ep|Yz x, c, 2]
Ep a e, P(a' c,
> Ep x I. 2 c,z)
Ey
c, P| c,z).
(1) holds by Rule 2 in Thm. 2 by swapping the policy π by the do intervention do(X = æ): given that the policy acts on X taking as inputs C it always true that Y C in G and g xxx in any graph G. The following equality follows by marginalizing over the domain of X, which assumed discrete. (2) follows by the composition axiom of counterfactuals. By combining these two
21

---

expressions we get
Er(Yx]]
I Xex
<< cx) 2)Br(Y| a. c, 2) 2) deda
RyP(y c, c, 2, z) dydeda:
Ep[YT π z] = (
And since the l.s.s. does not depend on Z we can further tighten the bound by writing,
Ep[Ya] > max
Consider the derivation of the upper bound. holds that,
Ep[Yx æ, c, x| = Ep[Y= c
Σ Eplya x' =1, c,2)
Ep|Y 2,c,2|((x c2 +
Σ EplYa ,',c, 2] c,,z)
x н
(1) Ep[Y 8. c, P(x c2 Σ P(a' c,,))
Am
Ep[Y x c, 2(x c, +1 P(æ c, 2).
(1) follows by the boundedness of Y, here assumed bounded by for simplifying the derivation. As above by combining this inequality with the decomposition of the policy effect get
Ep,[M] < f II ক cxe 2) [Ep[y æ, c, P|p( c, + 1 P(æ | c, 2)} dedæ
Xex
[E # =] + Ep[# z]
= EEI((Y 1) x1 11
(E Z / +
And since the l.h.s. does not depend on Z we can further tighten the bound by writing,
Ep(Y min - (2)1))) +1.
Prop- restated. Alg. 7 sound.
Proof. For the soundness of A we will consider each operation in turn and show that recovered adjustment sets and conditional instrumental sets lead valid bound.
1. Omit redundant intervention variables. For a policy æ == { xex denote R == {xx (cx) R X. Let R X AnA)((Y) in Gn. By Rule 3 of the o-calculus have that,
Ep. Y Ep, [Y]
since Y in R. and g x as there are no directed paths from X to by definition of R. The first two lines therefore reduce the number intervention variables. We proceed to bound the equivalent Er.,, [Y].
Find partial adjustment sets W. Line 4-11 consider finding partial adjustment sets by traversing the set of treatment variables R R. Recall that
Ep. YY Er RR)
Ep
.Y
r
πl XeR
xx(x Cx) dedr.
22

---

Denote TR := I Tx (r cx). In words lines 4-11 iteratively considers the existence of separators sets between R E R and Y reduce the scope of the policy on Y, i.. Yn By [43 Lemma 3. 3.1.. if there exists separator W between two sets of variables X and Y such that S S WWs = K in graph G, i.e. X W in 9, then W An UYYUUUUS US) K is also valid separator. The definition of WR in line aims to leverage this result to check for the existence of separators our case.
Consider EE R. let S = K - In particular, if there exist a set (C, R C W c K such that (YJL W) I then WR An An(RUUUC)NK in ATS satisfies (Y L W a) in Ins R. Lee. WR is a separator if one exists. Note that WR does include any member {C, R. J.. Assume this separation holds and that the if statement line
An(YUC) NK
8 is triggered. Then,
EplYTR R
r, cj = Ep[y, cl
Ep(Ya, orr |c]
E (a.rr c, w.n c) dwn
Ep (Y r, w.c.wnn c) dwn-
The last equality holds by assumption (Y & WR in 95
Now considering the term Ep|Y, r, c, wal for the second pass through the for loop, we can see that if the d-separation statement in the if condition is fulfilled we can further reduce the scope of the intervention on particular, for R' R,R, assuming that the if statement is triggered for Win n we have that,
E r, c, wR]
EplY, r, c, wR w WR, c) dw
ErlYar Ir wn, wr P( wn, c) dww.
Upon reaching the end of the for loop we have for W defined in line 9, T = (R: WR U) R.T that,
Er cr Erly. t, w[ c) dw.
Find partial instrumental sets Z. Starting line 12, we look for partial conditional instrumental sets. Consider first the case that potential instruments Z E W is evaluated in the for loop. On line 15, assume that the if statement is triggered, i.c. (Y Z) in Tr It then holds that,
Ep Yn r, cl == EplYu w, w, t, P(w c) dw
EplYu w 2, P(w c) dw
Ep(Yn w/2 1, P(w)) c) dwiiz.
In turn, for the case that some Z' W is evaluated the for loop. if aY W. T) in T7.
Ey c. = Er[Ya w. t,
t c) dw
EplYn w. t', c) dw
For the Z obtained in line 15, we derive that,
EplYn cr =
= EplY w/ w,,,,,,,, dwi,z.
23

---

4. Return bounds. We now proceed to use the expression above to lower and upper bound the effect of interest. particular, for the lower bound we have that,
EplYn x. cj
BplYn |
c) dw)z
Σ Eplyu w, ,, c)P('' t,) e) dw'r=
Er[Yu u, 22,,,,,,, ,z e) P(w)= c) dw)) =
Erly e, t, e) dw'zz.
By combining the inequality above with the decomposition of the policy effect we get
Er. [
I
<x< P() u, (,,, w 2,, e) dedrdwr'
XeR
TRYP(y (,,,,,,2,c))PP w u, c) P(w)= P(c) dedraw) ædy
P(t, = P(t, c)
w w/z.c)
2, TRyp(y|r,, I,,) 2 c)
P(w)z c))(()12(() dedrdwo, zdydz
1x(2)
F.l, c) P(t, z w),,c)
fadrdww.zdycc
where 7 := (2) W C, Note that we have used the definition R TUU. In turn for the upper bound we have that,
Ep|Y, x, cl
Ey|Yu w2,, P(w) c) dwiz
Σ EplYn
t/x.,,,,c))((u)z c) dw
EplYn u, w,,,,,,,2, c|Puu wlz,2,,,)) dw/z
Σ EplYn
c) dwi'= 1
Er|Yu 2, c) c) dwiz
Σ P(ul wiz. dw),
u'su
Eply | u, t, ,, c) dw),
+ S( P(u ((ulwis ((a c) dwl,
1 + f Ep 11 1 1/,,, dwi'
24

---

By combining the inequality above with the decomposition of the policy effect we get
Er.[M]
< +++]
11
π( cx) x,,,,,,,,c,P.... w/2, c)) dedrdw>>
XER
1
( 1) r, 2,2, 2, P()) (edrdw))zdl
(z)2)
=1+ + TR 1) P( w 2,2,2 P(t,z w 2,
= -1 + Ep((Y - 1)7]
where = #Rl= P Z W W2 Note that we have used the definition R = TUU.
Prop. restated. Let 1e be the number of variables and m be the number of edges in G. The run time of Alg. is O(k(A2 + m)).
Proof. Let & be the number of variables and be the number of edges in G. Operations in Alg. [
such as computing ancestors (e.g. lines 2,7), could be done in (kk²) time, e.g. with a Breadth-First
Search algorithm. Checking for d-separation is commonly done with the Bayes-Ball algorithm that can be implemented with a reachability search method, e.g. [4 Sec. 3.2.1], and could be done in
Olk + m) time by [43 Prop. 3.17J. The sets R and Z have size at most so the for loops in lines
and 14 are executed at most & times each. Combining these we get that Alg. [ requires O(k(k² 2 + m)) time return the bounds.
Prop. 7 restated. Let ग := ( Xx2) be a policy on {X,, X2) with a conditioning set C. All partial adjustment sets may be enumerated in time Okk(k + m)) where k are the number of variables and m be the number of edges in G.
Proof. This proposition is adapts Proposition 3.20 in 4 to find partial adjustment sets using the ListSep algorithm. List performs backtracking to enumerate all separator sets Z between X and Y such that I C Z s R, aborting branches that will not find valid separator. particular, it calls the TestSep and F FindSep algorithms recursively.
The Test Sep algorithm takes input graph, two sets of nodes be separated, and candidate separator set. By setting the graph to be 9 X,, the two sets to be separated to be Y and X2, and the separator set Z, TestSep will provably return whether W = Z (i(C, X1) is a partial adjustment set.
The Findsep algorithm uses the observation that if there exists a separator Z between X and Y such that I c Z S. R then Z := An(X Y I) R is separator. For each possible partition (X1, X2) of X, we can therefore find a separator by testing (using TestSep) whether An(X X O Y u C) n (V(UU)) in G X2 is separator. If it is then have found partial adjustment set.
To find all partial adjustment sets, we then proceed as follows. For each possible partition (X,, (x) of X, apply the ListSep algorithm with graph gxx, X.. sets to be separated Y and X2, and possible separator sets Z constrained as C. X, Z VVX C YY. Then, for every separator set Z found, output partial adjustment sets W Z (C,
This procedure finds all possible partial adjustment sets in time Okk(k + m]) where k are the number of variables and m be the number of edges in by Proposition 3.20 [k
B.2 Prool's of statements in Sec. 4
We make use of two auxiliary lemmas from the literature.
Lemma 1 (Continuous Mapping Theorem, (26). Let (Xx) X be random elements defined on
a metric spuce S. Consider a continuous function : - S' (where S' another metric space).
Then,
X, +p X = 9(Xn) 7p 9(X).
25

---

(Lemma 2, (24). Let := f(V;1) denote a finite and continuous functional and denote its nuisances. For independent samples of P. D == (0(l)) ii-...... P. let = Eplfal and := Ep[fn] some n. Let D- := E - Ep[ff] Then, the following decomposition holds:
Lemma 2
Eblía] - Er[sa] DD- ED-r(Sa - fol + +Ersfa -n]-
(19)
Suppose further that samples used for estimating y are independent and separate; and the nuisances are consistent. Then, A + B is random variable that converges to zero
Op(1//n)
Prop. restated. Suppose the nuisance estimates (js, 4) are Le ln-consistent and bounded. Then, the error of the DML estimator TDMe (TDMLL (DDML, in Def. is given as follows
1iDML T.M.
1 K
R2 + Op (1f2, 72 P22, Pell Op (1f1, alllin - i..l
where Rg is random variable that converges to zero rate Or(1/vn).
Proof. In this proof we consider the estimation the lower bund without loss of generality. Recall the definition of nuisance functions and estimators:
1/2 == t 71 == 1r ( Z) P(T, Z WZZ,C), with TPW = EphiM = v2. And μ = (Ju. H1, P22, (22) defined by
122 (2(RR, C, W, Z) Er(Y C,C, W, Z],
i = i2(R,, W, Z) 7U(U p) R C, W W)
#1 (T C, W, Z) Er[ea R, C. W. Z) T,C. C, W. 21. No = (C, W, Z) Σ, Ct, W, Z) Z) C).
We first verify that the population level value of the regression estimator coincides with the lower bound vig. This can be seen with the following derivation,
TREE.. = Er(uo( W Z, z)]
Σ polw, c, P(w',
wiz.c
= Σ
Σ (t, p(wiz,c)
w
Σ
wult t, c))
P(t. 2, w w,
P(L, = wiz
t
Σ w,ue
t, t,u, (,,,,,
aR P(t, 1 2,
utt,u, w.
Σ
Ply, t, ,',
P(t, x' w c)
Z W WZZC)
The DML estimator is similarly unbiased as
DML = Epliz(Y #2]] +
Eploo]
Ep]r2(Ep[y R, W W, - Eplm pill Epleo]
Ep(12(12 [2]] + ErIn T.C.W. 2] - -mil + Er[so]
Ep(Ti(u- [1]] Ep[uo]
Ep[so]
26

---

Consider the estimated value of TIDMI.. following the procedure in Def. 5
K
f.DML.. =
T
IDMI.E
I.DM..
T
= ED( B - -22t] + Di In luo - A1ll E [ãol
We could then write,
ADVI. b
TDMI, A+B+C
where,
A = E 12(Y - 122 + 71 - m1) + no
B Epm (2ef8 -22ll + mi/11 - H1l + No) - (fu(Y - d2) + 9/1/111 - -A13 + (iv)
C = Ep (718 - p2}] [123] + r(īn - m) + no) - (fa(Y - (12) + 21 - -n) + i0))
By the construction in Def. 5 the samples used for estimating (71 72: Po: M1, Mu, μ22 and evaluating the outer expectation are independent and separate. Under the assumption that nuisance parameters (fi, 12, no; ii, ju, (22) are consistent, R A + B converges to zero at rate Op(1/v/((k)) by Lem.
Before manipulating C' and deriving its large sample behaviour, consider the following intermediate results:
Ep[ea] = Σ
Hole, w, P), w)
6,30
2)
Σ
(tt,
Σ c,
P(t, x' c. c, c, (tt.
Er(n/d]
and similarly Ep[ão] Eplrdd] Note further that,
Ep(T2/2]
Σ
w. Plc.
P(t, ø c,w)
Σ
w,, 2 Pc,
c,m.t.u,s
Σ
w.
c,
Er[näd]
and similarly Er[12ń2] Ep[nâu]
Now consider C = Ci + C2 + C3 where,
C1 Erliz(Y frz) (2 u22l Epl12(22 fae
C₂ = Epli lú - (m] 7 - m1]]
C3 = Eplão - -oo = -EplT -Mp1
27

---

where the last equality follows from the result above. Therefore
C' Ep
fzlu2 -A23 + f1 P1} - 7im1 - m1 n( - pi)
Ep
(iz - 72 j + 72 - (2) + fn m1} n nii - 111 + (p m1)
Ep
(f2 - 72 [ + Ep 22/12 (2)) + 31 9::00 - -11 - - - -m13
@ Ep (32 72) 122)
Ep 11 m - :n1 + in fūr m²) --tii mm
Ep (f2 Ta (2)
Ep 1 pn) + fulii pn)
Op f1 72lll (2l)
Op (Iss 71 ?alllā Aall)
(1) holds by the equalities [22[12 Er(mūa] [r(22 Tr(nūd]
Finally, this implies that
fUML. TUML, = R + Op (1/- 1122-22llzz - pal) + OP 00(1/ (11m-1mm1)) millia Pull)
(20)
where is a random variable that converges to zero at a rate Op(L/Vn).
For the upper bound, consider the following defimition of nuisance functions and estimators:
?/2 nuh, n1 (Z) Z W Z,
with P = Epl12(Y -
v/. And μ (No, MM, #2 U2) defined by + 1]]
H2 M2(R, C, W Z) Er(Y - R. C, W Z].
A1 == P2(R,, C, W Z = āu(U C)p2( C, W, Z),
Z) T,C. W 2). μo = (C, W, Z) = Σ. E,mi (L, C, 2) it C).
#1 C, W, Z)
We first verify that the population level value of the regression estimator coincides with the lower bound vi.. This be seen with the following derivation,
T-REG, == Er|uo W (,2)] +1
Σ
nolw, e, 2) P(wkz,,,))
wizcc
1 Σ
uilt, w. wc,) P(w]z,c)
=1 Σ
Σ u. le,,ultz,,,,0 2,2, 23
WT
P(t, 2, w w22 c)
P(t,
t,w.
=1+
Σ
R Σ( 1)p(y t, u. u. (2,c) P(t, z w.z,c
P(t, u. u, w.
w.u.
;
1
Σ
(y- 1)
P(t, 11-1)P(t,z
w P(y, t, u, ,ly,,,u,"',,)''.
y. w'
1 Ep[(Y
Z W)
2/
The DML estimator is similarly unbiased as,
DMML := Ep(z2((Y - 1) - -22]] P 1p -4]] Ep[uo] +1
Ep[T2[Er[Y- - R. W W C, C, [2]] + Ep/n -ml]++ Er[uo] + 1
Eplx (EPlūn T,C, w. Z] - -Mil++ Ep[uo] +
Er[n(u- 1oalll1--ih m1]] + Er[uo]] +1
Ep[ua] +1
28

---

The arguments of the proof are now analogous to that of the lower bound. We can conclude therefore that,
TDML, TDML, R. Or Or (1 22ll1 Rall) Op Or(1 -illli -Aill..
(21)
restated. Suppose either 311 " or fúg = μ22 and that either ` = or jij = tA. Then, TDME (TDDMI.E " is an unbiased estimator of the corresponding bound defined in Alg. |
Prop.
Proof. Consider estimated value TDML, following the procedure Def.
5
K Σ
fDDML,
A
K
ADML
T
BD2 - -i23] + Da I (pt - A1]] RD [ão]
Under the assumption that,
Ep Ep[ -Epli fi2l Epliild- - -13l Ep[Ao]
The bias of the estimator is given by,
Ey Er.(TDML. TDML..
Ep [7(8 -22ll 7/01 - m1) + mo) - (f-(8 i22) + fn -n13 + R0)
= -Ep (f2 222 p22] + Ep (s1 nHirs Pr3
The last equality follows from the derivation of (C) in the proof of Prop. g
If either f1 Aá 142 and that either f1 "i or ji: ju then this expression equals 0 and we have that,
Ep(1 - D..... 0.
An analogous result holds for the upper bound.
29

---

C Details on experiments
This section provides details of the data generating mechanisms used for synthetic experiments, and implementations.
As described in the experimental section, for estimating nuisances (Y, 14) we used Gradi- ent Boosting models for classification and regression where appropriate. the models using Python using the commands eradientBoose wnngilasssiiiii. () and Gradient ingRgressor () using default hyperparameters. We systematically bound probability values in the interval [0.01, 0.99] to avoid propagating potentially large upproximating errors.
implemented
C.1 Simulations
We make use of the following 4 data generating systems, summarized by the graphs in Fig. In this section, we the notation [(X] that equals if the statement {X) is true, and equal otherwise.
Example 1.
The data generating mechanism for the first scenario given by:
Ux, NN,,,),
Z = fz(U2),
C fc(Uc),
== fx(Z, C,Uxy), Y fv(X,C
where,
[z(Uz) -1 1lU2 < 0.55 + 1[-0.5 Uz < 0.5) +1 1(U7 > 0.5),
fo(Uc): Uc.
[x(C,UxY) :- 11UxY > 1 exp(0.75C 0.5Z + 0.5)).
fx(, -2 -1) (2Uxy
The outcome Y is then re-scaled to lie in the [0, interval. The target for estimation can be derived with Alg. and equals, for the lower and upper bound respectively,
max ze(- 1e(-1.0,1)
(2) (PZ C)],
min Ep Ep( 1)/P(Z +)]++
ae 2e(-1,0,1)
Example 2.
The data generating mechanism for the second scenario is given by:
Uxy, Ux.w, Ux. Ux, Uc, Uw ~ 1),,
C Sc(Uc),
W = fw(Ux,w),
X1 = fx, (,,C,,,,,,,,,,)
X₂ fx,(Ux,w: Ux,),
Y = (r,,, X2, C, W. Wx,)),,
where,
Sc(Uc) = Uc,
sw(Uw) Ux,w,
fx, (C, Ux,Y) = 1(Ux,Y - (1 exp[0.077 + 0.5)) > 0.55)
Ux,),
[x, Ux,) 1[Ux,W Ux 0),
C,,,,,,,C Vx,y) =
(C 2sin (2X, - 1)) -2(2Ux, 1)(1 + 0.5C) X₂ W.
The outcome Y is then re-scaled to lie in the [0, 1] interval. The target for estimation can be derived with Alg. and equals, for the lower and upper bound respectively.
EplmY/P(Xe W. W.C)..
Ep( 1)/P(X, W,C) +
30

---

Example 3.
The data generating mechanism for the third scenario is given by:
U Uey.UxxW,Uann Ux, Uc NO0,1), 1),
W = {W1, W1o0) s.t. W; ~ (V(, 1), == 1,.... 100,
C : fc(Uc),
X1 = (C,C,,,,,
X2 (x,(W, ;Ux),,
Y = fy (X X2, C, W,Uxy)).
where,
fc(Uc) Uc.
(x, Ux,) y (1 exp(0.75C 0.5)) Uxi
(W, (W,Ux,) 11W + W/2 + Ux, > 0}.
Xy C, W, Ux,)) -22X, 1551-11-(( (C (2C (2X, - 1))
-2(2Ux,y 1)(1 + 0.5C) X2 +
Σ
W1/100.
i=1...,000 100
The outcome Y is then re-scaled to lie in the [0, 1] interval. The target for estimation can be derived with Alg. [ and equals, for the lower and upper bound respectively,
Ep|TY/P(X2 W. C),, EpiT(Y 11/(Y-1)/P(X) W.C)|+
Example 4.
The data generating mechanism for the fourth scenario is given by:
Uz, UX,Y Ux, Uc, Uz (N,,1))
Z = fz(U2)
W (Wi...., Wo} - AVO, 1), I,...,,,
C = fo(Uc),
X1 (x, 2,UX,, Ux,,
X2 = Nx(W,Ux, fr X2, C.W.
where,
fo(Uc) = Uc. (z(U2) 1(U2 > 0),
(x Ux, Z,Uc) : 1(Ux,Y (1 +
> Ux
(x,(W, (W,Ux.) 1(W, + W2 Ux, > 0). [r(X,, X2, C, W. W.UNU Ux,r) (2X,, 1) (C+ + 22n (2X1 - 1)) -2(2Ux,y 1)(1 0.5C) X₂ Wi
The outcome Y is then re-scaled to lie the [0. 1] interval. The target for estimation can be derived with Alg. and equals, for the lower and upper bound respectively.
max Ey Er( =E(0,1)
W, C)],
min (Z ==(0,1)
W,C)
C.2 Health Campaign Evaluation
The data was curated from anonymous from Colombia, Peru and Mexico, using web platform 30 The exact questions considered in the survey can be found in (30 The authors made the data available under a Creative Commons license anndd and is currently hosted by Kaggle as c.s.vv file, accessible through the following link: kaggle com/ code hwtp//pwlokkkkkooo ke
life steee/.te.....
creat ivecemmons http://t.oo...ece//e/ooo////////
31

---

F E
T
H
M
(a) g
(b) Fm
(c) 9 E
Figure 7: Obesity diagrams.
We considered a number of pre-processing steps. In particular, we created a BMI variable from weight and height data that were a posteriori removed from the dataset, imputed (with median mode among observed instances) missing values, scaled continuously-valuee covariates by subtracting the mean and dividing by the standard deviation, and scaled the outcome BMI to lie in the interval [0, 1]. Several covariates were ignored, including data on the consumption of water, consumption of vegetables, consumption of food between meals, consumption of alcohol, and number of main meals. These may be interpreted as unobserved variables, potentially confounding other relationships in the data as specified by the assumed causal diagram. The data from total of 2111 individuals was recorded in this study.
The bounds the effect of policy TO := (TT m acting on H and E, with new assignments defined as (ao H = rarely) = a), Paow regularly) = at) are approximated from the expression Prop- 4 Following this proposition we find that Age (A), Smoking (S), calories consumption Monitoring MM, and Family history with overweight (F), that is W = {.A, S, M, F), form partial adjustment set for T given that LL W W. H.)r We can verify also that Z = (T) is partial instrumental set for π since (YLL)))s. For illustration, the corresponding mutilated diagrams are shown in Fig- Finally, we partition the data equally to obtain two sets of samples for training first and second stage classifiers and regressors, respectively, obtaining a first estimate of bounds, before then switching the role of the two data samples and averaging over the resulting estimates.
32

---

D
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made match the theoretical and experimental results presented in the paper. The impact statement in the Appendix reflects how much the results can be expected to generalize to other settings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We include several remarks in the paper to qualify our contribution, e.g. the remark in Sec. 3, and include an Impact Statement in the Appendix to more thoroughly describe the limitations of our analysis, assumptions, and applicability real-world settings.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "LLimiations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
33

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical statement quoted in full in the paper. We have attempted to provide an example to illustrate the significance of each theoretical statement and highlight implications. The formal proof of all statements given in the Appendix.
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the data generating mechanisms, details of the target of estimation and information as to what python functions and libraries can be used to fit the proposed estimators. We do not, however, disclose an open source implementation the proposed methods at this moment.
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
34

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes] and [No]
Justification: The data is publicly available and we have provided full details as to where to access the data and how to run the synthetic data generation pipeline. The code will not be
open sourced at this moment we believe to have provided sufficient details to reproduce our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
publ heplic//uu des Codeeuulmm ss
cy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/eoiddeeeeeeeiiieeeeeeeee SodeeuumeaaSaooo Looooonocioooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provided details where applicable. In our case, data splits, hyper- parameters, optimizers, etc., are not significant for the implementation the method.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
We report results as box plots that include several statistics, e.g. median, 25th, 75th, percentiles, etc., to summarize our results for different seeds of the data and algorithm.
Justification:
Guidelines:
35

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: Experiments were single CPU in under few minutes of wall time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have read the guidelines and we do not think that our work presents any notable concerns.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
36

---

Justification:
We include broader impact statement in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
No risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
The paper does not use existing asset
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
37

---

The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, (papersw sepersswihcode com/ dat asats s has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
38

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "cQYxIVJKNe", "reviewer": "Reviewer_LyZM", "summary": "The paper proposes a new framework that uses structural causal graphs for estimating the effect of policies under unobserved confounding. Their approach first derives tighter analytical bounds on the estimates of the effects of the policies under the structural causal graph setting and the paper uses the results to develop estimators of these bounds using the double machine learning toolkit. The provide numerical experiments to show their approach produces good estimates of the effects of the policies under different settings such as misspecification.", "questions": "1. Can more detailed be given on differences between $T^{TW}$ and $T^{REG}$ that can explain the differences in the numerics? Is $T^{DML}$ derived from one of the two estimators combined with the DML toolkit and if so which one is used? \n2. Since the approaches provide an upper and lower bound on the effect of the policies, how do you select an estimate if the difference between the bounds is large?", "limitations": "Limitations have been addressed by authors.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper tackles an interesting problem of estimating the effect of policies under unobserved confounding and develops an effective structural causal models approach. The paper provides good discussion on the analytical bounds and is able to study multiple settings in the numerics.", "weaknesses": "The paper is somewhat hard to follow as the authors of the paper assume readers are familiar with existing notation for structural causal graphs. For example, they assume readers know the notation \"an, de\" for such graphs. Much of the notation is defined ahead of time, but without context for how and where they are used throughout the paper. Additionally, some new notation appear without explanation and are left to be interpreted by the reader. For example, in the partial identification section, $\\bf{X}_1, \\bf{X}_2$ are introduced, but the paper provides minimal intuition for what each set of variables means or should mean. As a result, the paper in general is hard to follow as it is overloaded with notation but limited intuition. Potentially, more concrete examples could be provided. The partial identification section in particular could benefit from figures of graphs that satisfy definition 3. \n\nIn the numerics, it may be helpful to develop some sort of baseline to demonstrate how the new estimation framework out performs existing approaches. \n\nThe paper also seems to have various typos and broken references. There are missing citations at line 67 and 311 and a missing proposition reference on line 316.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. The typos and missing / broken references have been corrected, thank you. In the following, we aim to address your concerns point by point.\n\n1. ***“The paper is somewhat hard to follow as the authors of the paper assume readers are familiar with existing notation for structural causal graphs.*** \n\nWe appreciate the feedback on the readability of the paper. We will aim to improve following your suggestions but we do feel that (some of) our choices are well justified. Please consider the points below to help address specific concerns mentioned in the review.\n\n> \"Much of the notation is defined ahead of time, but without context for how and where they are used throughout the paper.\"\n\nIt is not uncommon to describe the notation used throughout the paper in a short, self-contained section at the beginning of a paper. In our case, we found that defining different mathematical symbols in the main body, where required, creates a disconnect with the overall story line that makes the paper less readable.\n \n> \"Additionally, some new notation appear without explanation and are left to be interpreted by the reader. For example, in the partial identification section, $\\boldsymbol X_1, \\boldsymbol X_2$ are introduced, but the paper provides minimal intuition for what each set of variables means or should mean.\"\n\nDef. 3 mentions explicitly that $\\boldsymbol X_1, \\boldsymbol X_2$ are two sets of variables that are intervened on by the policy $\\boldsymbol\\pi$. We have also made an effort to introduce the intuition behind the partial adjustment strategy in Def. 3 gradually, first highlighting the idea in Example 2 and then again describing in words the proposition that uses Def. 3 in lines 166-167. Note also that Examples 3 and 4 both illustrate these ideas once more.\n\n> \"Potentially, more concrete examples could be provided. The partial identification section in particular could benefit from figures of graphs that satisfy definition 3.\"\n\nWe made a conscious effort to provide a concrete example after every proposition to illustrate the ideas involved. For example, Def. 3 is used and explicitly mentioned in Examples 3 and 4 using the graphs in Figs. 2b and 2c.\n\n2. ***”In the numerics, it may be helpful to develop some sort of baseline to demonstrate how the new estimation framework out performs existing approaches.”***\n\nThis suggestion is a very good one, thank you. \n\nWe are considering an additional experiment to give a sense of the benefit of the progressively tighter bounds given in Sec. 3. In particular, we can compute each one of these bounds separately for a concrete example and inspect their values. \n\nFor this experiment, we sample from the data generating mechanism associated with the causal graph given in the fourth row of Fig. 4, and evaluate the policy in Eq. (16). We compute bounds obtained by the natural bounds (most conservative, Prop. 2),  using the partial adjustment set $W$ only (Prop. 3), using the partial instrumental set $Z$ only (Prop. 4), and finally using Alg. 1 that combines all propositions and is the proposed approach. We display figures with the resulting bounds and causal graph in the attached pdf, demonstrating the gain that can be achieved in this particular example with our proposed bounding strategy.\n\nFinally, we would like to note that, to our knowledge, no existing method has been developed to estimate bounds on the effect of policies given an arbitrary causal diagram of the environment.\n\n3. ***“Can more detailed be given on differences between PW and REG that can explain the differences in the numerics?\"***\n\nOf course, thank you for the question. We could start by noting that $T^{PW}$ involves the estimation of a ratio of probabilities $\\boldsymbol\\\\gamma$, while $T^{REG}$ is based on a sequence of regression tasks. $T^{PW}$ can therefore be unstable with low sample sizes if the ratio denominator is estimated to be close to zero. This might explain the relatively larger errors of $T^{PW}$ in Fig. 4 for low sample sizes. $T^{REG}$, in contrast, is better behaved in small sample sizes, outperforming $T^{PW}$. \nFor equally good estimators of nuisance functions ($\\boldsymbol\\\\gamma$ for $T^{PW}$ and $\\boldsymbol\\mu$ for $T^{REG}$), both estimators have rates of convergence of the same order. With larger samples, we expect them to converge to the true values at a similar rate.\n\n4. ***\"Is DML derived from one of the two estimators combined with the DML toolkit and if so which one is used?”***\n\n$T^{DML}$ is constructed as a combination of elements of $T^{PW}$ and $T^{REG}$. In particular, note in the definition of $T^{DML}$ (Def. 5) the use of nuisances $\\boldsymbol\\gamma$ of $T^{PW}$ and $\\boldsymbol\\mu$ of $T^{REG}$. As a result, $T^{DML}$ has quite a different performance profile than $T^{PW}$ or $T^{REG}$, being more robust to noise in nuisance estimation and converging at a faster rate to the underlying population-level bounds as a function of sample size.\n\n5. ***“Since the approaches provide an upper and lower bound on the effect of the policies, how do you select an estimate if the difference between the bounds is large?”***\n\nThis is an important point, thank you for raising it.\n\nBy the definition of bounds, the true value of the effect of the candidate policy can only be constrained to be between the lower and upper bound. For all problems studied in the paper, given the data and causal diagram, it is not theoretically possible to pinpoint a single value for the effect of the policy of interest.\n\nIf the bound is wide, there is substantial uncertainty about the true value of the effect of the policy. In this case, the practitioner has to weigh this result along with other considerations (potential harm of the policy, cost of implementation, etc.) to arrive at a decision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gZiqFxPEie", "reviewer": "Reviewer_BoZN", "summary": "This paper proposes a new method to partially identify policy effects using a known causal graph and observational data. In theory, the paper derives general bounds for probabilistic and conditional policies, generalizing the existing results on discrete atomic policies. To estimate the bounds, the authors extend the well-known double machine learning framework to their setup. Finally, simulations are provided to demonstrate the convergence rate of the estimators. Then, the estimators are applied to estimate bounds for a range of policies in an obesity study.", "questions": "N.A.", "limitations": "N.A.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is overall well-written and organized. The contributions of the paper are clear in contrast with previous works. The authors not only derive new theoretical bounds but also provide an efficient double machine learning method to estimate the bounds. Finally, experiments are provided to demonstrate the robustness against the issue of model misspecification.", "weaknesses": "1. I think the paper can discuss a bit more about the cases in which the policy effect is identifiable, given a known graph and observational data, before moving to partial identification. Generally, partial identification is valuable when the assumption required for point identification may not hold in practice. The authors can potentially discuss the assumption and highlight the value of their method.\n2. I didn't see any assumption required to derive the natural bound and its extension proposed by the authors. If the bound is assumption-free, would it be too conservative to use in practice? The authors should clarify this. \n3. The estimation strategies in the paper are quite standard. The authors should explain why this part is novel compared to the existing literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and positive assessment of our work, we appreciate the feedback. Please find our response to specific concerns and questions in the following.\n\n1. ***”I think the paper can discuss a bit more about the cases in which the policy effect is identifiable, given a known graph and observational data, before moving to partial identification. Generally, partial identification is valuable when the assumption required for point identification may not hold in practice. The authors can potentially discuss the assumption and highlight the value of their method.”***\n\nThis is a good suggestion, thank you. Identification, and the type of graphs that enable identification of the effect of policies, will be mentioned more explicitly in the introductory sections.\n\n2. ***”I didn't see any assumption required to derive the natural bound and its extension proposed by the authors. If the bound is assumption-free, would it be too conservative to use in practice? The authors should clarify this.”***\n\nThe only requirement for the natural bounds (Props. 1 and 2) is for the treatment variables $\\boldsymbol X$ to be discretely-valued and for $Y$ to be bounded.\n\nAs the reviewer suggests, if more information is available, such as a causal diagram of the environment, the bounds can typically be improved by exploiting this structure, and the natural bounds will be too conservative. In our paper, we currently convey this idea in Example 2 that shows how the knowledge of the causal diagram leads to a bound tighter than the natural bounds. We will make sure that this is more clearly stated.\n\n3. ***”The estimation strategies in the paper are quite standard. The authors should explain why this part is novel compared to the existing literature.”***\n\nThe general principles behind the definition of double machine learning estimators have appeared in previous work. To a large extend we do build on previous intuition. In our view, however, adapting these techniques for the new expressions of the bounds in Alg. 1 is non-trivial. The nuisance functions, the definition of nested regressions, and target estimators, are all specifically tailored to our setting, and are novel contributions to the literature. Similarly for the corresponding robustness guarantee and rate of convergence result. We do believe that developing DML estimators for the bounds we derive and, as a consequence, providing a complete solution for estimating bounds on the effect of policies is novel and impactful.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gd8kL02oCi", "reviewer": "Reviewer_uSXJ", "summary": "This paper proposes a partial identification method for the value of policies that intervene on a potentially multivariate set of variables in a known causal graph. It searches for adjustment sets or IVs that apply to one set of variables intervened on conditioned on others.", "questions": "See (1) above.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Expanding the reach of partial identification methods to account for new kinds of action/covariate spaces is a valuable contribution. The paper is well-executed, with clean algorithmic strategies and statistical guarantees for the double ML estimator. I also appreciate the application to evaluating health campaigns -- although the bounds obtained in this case are not incredibly tight, they show that the method could provide suggestive evidence in a setting of practical interest.", "weaknesses": "(1) It would be valuable to have more intuition about what the method is doing. My heuristic understanding right now is that it is combining bounds for policies that intervene on two sets of variables, one of which can be made to satisfy a no-unobserved-confounders type assumption given the graph, and one of which cannot (and hence we get fairly trivial bounds for that portion of the treatment variables). What would the authors say is the right way to understand the strategy? Similarly, it would be useful to contrast explicitly with, e.g., previous strategies for partial identification with discrete variables. \n\n(2) The majority of the experiments are spent on evaluating accuracy of the DML method at estimating the bounds. This is valuable, but for partial identification methods the more important question is likely to be whether the bounds are informative. It would be more useful to have experiments that assess what kinds of graph structures lead to more or less informative bounds, to guide potential users about when the method is likely to be useful, even if it meant pushing some of the current experiments content to an appendix", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and positive feedback. In the following we hope to address the mentioned weaknesses. Please let us know if we can expand on any of it.\n\n1. ***”It would be valuable to have more intuition about what the method is doing. My heuristic understanding right now is that it is combining bounds for policies that intervene on two sets of variables, one of which can be made to satisfy a no-unobserved-confounders type assumption given the graph, and one of which cannot (and hence we get fairly trivial bounds for that portion of the treatment variables). What would the authors say is the right way to understand the strategy? Similarly, it would be useful to contrast explicitly with, e.g., previous strategies for partial identification with discrete variables.”***\n\nThank you for this question. Intuitively, we would describe the method as seeking to recursively simplify the query. First by omitting the intervened variables that have no effect on the outcome; second by finding the set of intervened variables for which a partial adjustment set could be used to tighten the bound, and third by finding the set of variables that act as valid partial instrumental sets to evaluate bounds on the most favorable conditional distributions rather than on the joint distribution. \n\nThe previous strategies for partial identification are described in Sec. A.2. We will refine that section to make a more explicit comparison. In our view, the biggest contrast with existing partial identification methods is that we provide analytical bounds (written in terms of $P(\\boldsymbol v)$) rather than as solutions to a constrained optimization problem. The biggest advantage of our approach is that analytical bounds allow for efficient estimation in high-dimensional systems of continuously-valued variables with the proposed estimation framework (Sec. 4).\n\n2. ***”The majority of the experiments are spent on evaluating accuracy of the DML method at estimating the bounds. This is valuable, but for partial identification methods the more important question is likely to be whether the bounds are informative. It would be more useful to have experiments that assess what kinds of graph structures lead to more or less informative bounds, to guide potential users about when the method is likely to be useful, even if it meant pushing some of the current experiments content to an appendix.”***\n\nThis is an interesting point. In practice, although the graph structure can play an important role in tightening bounds, the actual width of the bounds in a particular problem are also driven by the distribution of data. Here is an example to make this more concrete.\n\nFor binary variables $X$ and $Y$, consider bounding $P(Y=1 | do(X=1))$ from $P(X,Y)$ and the bow graph $\\\\{X \\rightarrow Y, X\\leftrightarrow Y\\\\}$. The tightest bounds in this case are given by:\n\n\\\\[P(Y=1,X=1) \\leq P(Y=1 | do(X=1)) \\leq P(Y=1,X=1) + 1 - P(X=1)\\\\]\n\nThe width of the bounds is given by $1 - P(X=1)$. This term may evaluate to anything between $(0,1)$ depending on the value of $P(X=1)$.\n\nFor more complex causal structures, the bounds proposed in Prop. 3 and Prop. 4 (and Alg. 1) reduce the width of the interval above. Intuitively, from $1 - P(X=1)$ to $1 - P(X=1)\\alpha$ for some $\\alpha\\geq 1$ that is a function of the joint distribution. But again, the actual width of the interval is ultimately determined by the values of $\\alpha$ and $P(X=1)$ that may be large or small depending on the value probabilities involved. A similar reasoning may be given for the effect of policies instead of the effect of hard interventions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V44KIir8Ju", "reviewer": "Reviewer_nLc5", "summary": "The authors develop graphical characterizations and estimation tools to bound the effect of policies given a causal graph and observational data collected in non-identifiable settings.", "questions": "Can the authors highlight the main challenge/novelty of the proposed bound for policy compared to bounds for treatment effect?", "limitations": "Please refer to Questions.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Strengths and contributions: (1) derive analytical bounds for general probabilistic and conditional policies that are tighter than existing results, (2) develop an estimation framework to estimate bounds from finite samples, applicable in higher-dimensional spaces and continuously-valued data. They further show that the resulting estimators have favourable statistical properties such as fast convergence and robustness to model misspecification.", "weaknesses": "Please refer to Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. Please consider the following descriptions and explanations to address your question.\n\n1. ***”Can the authors highlight the main challenge/novelty of the proposed bound for policy compared to bounds for treatment effect?”***\n\nWe would be happy to summarize what we consider to be the main contributions of our work. Below, we start by describing the context in which our methods are developed, followed by a description of the main novelty in our work. \n\n> Context (see also Sec. A.2 in the Appendix)\n\nBounding the effect of interventions and policies, whenever they cannot be uniquely computed, is a research agenda that dates back to the work of Manski in the 1990's. Since, to produce better bounds, two lines of research have emerged making different assumptions on the underlying structural model. One the one hand making assumptions on the strength of unobserved confounding, and testing the sensitivity of conclusions as the strength of unobserved confounding increases. And, on the other hand, making assumptions on the structural dependencies between variables, encoded in a causal diagram. \n\nOur approach considers the second setting. In other words, we attempt to provide better bounds and estimation methods for the effect of policies given data and an (arbitrary) causal diagram of the system. Very few practical solutions exist for that problem. Most existing work has focused on solving constrained optimization problems in parameterized models as a way to bound the effect of interventions, e.g. [42, 12, 25, 13],\n\\\\[\\min / \\max_{\\mathcal M_\\theta} \\mathbb E_{P_{\\mathcal M_\\theta}}[Y | do(x)], \\quad \\text{such that}\\quad P_{\\mathcal M_\\theta}(\\boldsymbol v)=P(\\boldsymbol v).\\\\]\nIn words, the problem is to search over the space of models $\\mathcal M_\\theta$ parameterized by $\\theta$ for the minimum and maximum values of $\\mathbb E_{P}[Y | do(x)]$ such that $\\mathcal M_\\theta$ entails the observed distribution, i.e. $P_{\\mathcal M_\\theta}(\\boldsymbol v)=P(\\boldsymbol v)$. The challenge with this approach is that the number of parameters $\\theta$ grows exponentially with the number of variables and their cardinality, and is difficult to scale to high-dimensional systems or continuous variables without assumptions on the function class and distributional families.\n\n> Novelty\n\nIn light of this summary, our first contribution is to derive analytical bounds (that is, mathematical expressions in terms of $P(\\boldsymbol V)$) on the effect of policies that exploit the structure of the causal diagram. The proof strategy is based on Manski’s bounds (Prop. 1) but extends them to leverage the independencies between variables given the causal diagram, producing new graphical criteria and tighter bounds (Props. 2, 3, 4, and Alg. 1). In particular, the bounds hold in systems with continuously-valued outcomes and covariates – which is a distinction with respect to previous work. Our second contribution is to develop an estimation strategy that would allow for the practical computation of bounds in high-dimensional systems, and with favorable statistical guarantees such as robustness to misspecification and fast convergence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
we thank the reviewers for their time reviewing our work.

In this global rebuttal, we include an additional experiment that compares the bounds obtained according to the different propositions in Sec. 3 of the paper. This addresses specifically a comment from Reviewer LyZM. The details of the experiment are described in the attached PDF and are summarized below.

> Evaluation of bounds given by the different propositions introduced in the paper

Our simulations for this experiment are based on one of the data generating mechanisms used in the experimental section and described in more details in Example 4 Appendix C.1. The causal diagram is illustrated in the attached PDF and can also be found in the fourth row of Fig. 4 in the paper. We consider evaluating the policy in Eq. (16) of the paper and compute the bounds obtained by applying Prop. 2 (most conservative), Prop. 3 (using the partial adjustment set $W$ only), Prop. 4 (using the partial instrumental set $Z$ only), and finally Alg. 1 (that combines all propositions and is the proposed approach). The figure in the attached PDF gives the results over 10 seeds of the data and across multiple data sizes, highlighting the gain achieved by exploiting the causal structure using the proposed approaches. In this particular example, we are able to tighten the natural bounds -- approximately equal to [0.1, 0.9] -- to approximately [0.32, 0.77] with Alg. 1.

Meta-review: I concur with the reviewers' positive view of the submission and think the paper would makes a nice addition to the nascent literature on this topic. I am happy to recommend its acceptance. However, I find that deferring all discussion of related work to the appendix to be inappropriate. Many papers on the precise subject of the paper (evaluation of policies/ATEs/HTEs under confounding) are cited but not discussed in the main text ([5,17,18,19,24,40]). I understand there are important differences to these works in the technical approach, and I appreciate that the works that the present papers' technical approach relies on are discussed somewhat in the main text, but additionally all works directly related to the problem itself, even with alternative technical approaches to it, should be discussed in the main text and compared to. So I ask the authors to prioritize discussing these in the main text, even over technical details or other extras that are more appropriate for deferring to the appendix. This should also ameliorate the concerns of the more negative reviewers regarding context in the literature. I fully trust that the authors can undertake this appropriately within the preparation of a camera ready version as this mostly involves moving content around from/to the appendix and/or adding a couple sentences here and there.
Decision: Accept (poster)