Towards Estimating Bounds on the Effect of Policies under Unobserved Confounding

Authors: alexis  bellot
chiappa  google
deepmind  london
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 15:06:41

Abstract: As many practical fields transition to provide personalized decisions, data is increasingly relevant to support the evaluation of candidate plans and policies (e.g., guidelines for the treatment of disease, government directives, etc.). In the machine learning literature, significant efforts have been put into developing machinery to predict the effectiveness of policies efficiently. The challenge is that, in practice, the effectiveness of a candidate policy is not always identifiable, i.e., not uniquely estimable from the combination of the available data and assumptions about the domain at hand (e.g., encoded in a causal graph). In this paper, we develop graphical characterizations and estimation tools to bound the effect of policies given a causal graph and observational data collected in non-identifiable settings. Specifically, our contributions are two-fold: (1) we derive analytical bounds for general probabilistic and conditional policies that are tighter than existing results, (2) we develop an estimation framework to estimate bounds from finite samples, applicable in higher-dimensional spaces and continuously-valued data. We further show that the resulting estimators have favourable statistical properties such as fast convergence and robustness to model misspecification.

Main paper: 
 introduction: 
 Understanding how to act upon the world around us requires humans and artificial systems to thoughtfully evaluate the effect of different plans, policies, and interventions one might consider. Data and methods that facilitate this process are increasingly relevant for high-stakes decision-making, notably in medical care with the rise of precision medicine [41] , but also in education [25] , law enforcement [13] , public policy [44] , and economics [2] . The interventions that a practitioner might consider, could consist of complex policies where a variable is set to follow a conditional or stochastic relationship depending on other variables in the system. For example, policy-makers might contemplate higher taxes on processed foods and stronger campaigns on its health risks targeted to overweight individuals. A sensible question in this context could be "what is the effect of a policy that reduces the consumption of processed foods by 50% in people with a body mass index above 30?". Contrary to atomic (also called hard) interventions that force a particular value, this policy suggests a softer intervention.The identifiability and estimation of policies from data is a widely studied problem in the reinforcement learning [37, 39] and causal inference [2, 12, 15, 31, 32, 40] literatures. Despite the generality entailed by many common approaches in these fields, they often rely on an impractical condition: the assumption that the effectiveness of the policy is uniquely computable from the observed data and ˚Correspondence to Alexis Bellot: abellot@google.com assumptions about the data generating process. This can be violated in real-world settings subject to unobserved confounding, and lead to the non-identifiability of the effectiveness of a given policy. This holds irrespective of the number of samples collected so that in reality multiple answers for the effect of a policy may be equally plausible. In their seminal work in the early 1990's, Manski and Robins [27, 33] showed that, nevertheless, the effect of an atomic intervention may always be bounded in a non-trivial interval (i.e., probabilities strictly contained in r0, 1s), irrespective of unobserved confounders or the causal structure underlying the variables involved. Causal effects are therefore in general said to be partially identifiable, i.e., one can derive bounds that shrink the range of a priori possible values for a given effect and potentially serve as a useful support for decision-making.Starting from this insight, the problem of partial identification has been gaining growing attention in the literature. To improve upon these bounds, one common restriction on the data generating mechanism is to assume knowledge of a causal graph describing the phenomenon of interest. Several results have derived progressively tighter bounds by exploiting the independencies in observational and interventional distributions implied by the causal graph [4, 5, 14, 46, 47, 48] . For instance, Balke and Pearl [4] (and subsequent refinements, e.g., [14, 48] ) defined polynomial optimization programs to derive bounds that are provably optimal. Tighter analytical bounds have also been derived in selected settings, such as for the "instrumental variable" graph [47] and discrete systems with general graphs or equivalence classes [5, 46] . These results, however, are almost exclusively concerned with atomic interventions and small systems of discretely-valued variables. Despite their generality, there is still a gap towards making the partial-identification and estimation of the effectiveness of policies practical. In particular, making scalable inferences from finite samples with continuous outcomes and higher-dimensional covariates is not possible with existing tools. This paper aims to provide a novel estimation framework to support decision-making in nonidentifiable settings, overcoming some of these challenges. We introduce several new results for the partial identification and estimation of the effect of stochastic and conditional policies from a combination of observational data and assumptions on the domain, encoded in a causal graph. Our contributions may be summarized as follows. We introduce several graphical criteria to derive new analytical bounds on the effect of policies with continuous outcomes and covariates, that improve upon the non-parametric bounds of [27, 33] and [47] . Given these analytical bounds, we then construct estimators leveraging the double machine learning [10] toolkit for scalable inference, and demonstrate that estimators exhibit favourable statistical properties such as robustness to noise and fast convergence. These results are applicable to arbitrary stochastic or conditional policies that an investigator might design, and high-dimensional covariate spaces. Recent work has made progress is developing powerful estimators for identifiable causal effects [6, 8, 19, 20, 36, 49] . However, these estimators are not applicable in non-identifiable settings. To our knowledge, the proposed estimation machinery is the first result for bounding the effect of policies given a causal diagram.Preliminaries. We use capital letters to denote variables (X), small letters for their values (x), bold letters for sets of variables (X) and their values (x), and use supp to denote their domains of definition (x P supp X ). 1 x pXq is the indicator function that equals 1 if the statement tX " xu is true, and equal to 0 otherwise. The framework we use to underpin the estimation of the effect of policies rests on structural causal models (SCMs) [31, Def. 7.1 .1] . An SCM M is a tuple xV , U , F, P pU qy, where V is a set of endogenous (observed) variables, U is a set of exogenous latent variables, and F " tf V u V PV is a set of functions such that f V determines values of V taking as argument variables P a V Ď V and U V Ď U , i.e. V Ð f V pP a V , U V q. Values of U are drawn from an exogenous distribution P puq. We assume the model to be recursive, i.e. that there are no cyclic dependencies among the variables. Graphically, each SCM M is associated with a causal diagram G over V , where V Ñ W if V appears as an argument of f W in M, and V W if U V X U W ‰ H, i.e. V and W share an unobserved confounder.We will use graph-theoretic family abbreviations to represent graphical relations, e.g. an, de, etc. Moreover, for a causal diagram G over V , the X-lower-manipulation of G deletes all those edges that are out of variables in X, and otherwise keeps G as it is. The resulting graph is denoted as G X . The X-upper-manipulation of G deletes all those edges that are into variables in X, and otherwise keeps G as it is. The resulting graph is denoted as G X . We use For a sample set D :" tv piq u i"1,...,n " P , we use E D rf pV qs :" p1{nq ř n i"1 f pv piq q. We use }f } P :" a E P rtf pV qu2 s. p f ´f " o P pr n q denotes that a function p f is a consistent estimator of f at a rate r n , p f ´f " O P pr n q denotes that it is bounded in probability at rate r n .π :" tπ X pc X qu XPX denote π R :" tπ X pc X qu XPR , R Ď X. Let R " X Ş AnpY q in G π . By Rule 3 of the σ-calculus we have that, E Pπ rY s " E Pπ R rY s, since Y |ù d XzR in G π,XzR and G π R ,XzR as there are no directed paths from XzR to Y by definition of R. The first two lines therefore reduce the number of intervention variables. We proceed to bound the equivalent E Pπ R rY s.2. Find partial adjustment sets W . Line 4-11 consider finding partial adjustment sets by traversing the set of treatment variables R P R. Recall thatE Pπ R rY s " E P rY π R s " ż E P rY π R | r, cs ź XPR π X px | c X qP pcq dcdr.Denote πR :"ś XPR π X px | c X q.In words lines 4-11 iteratively considers the existence of separators sets between R P R and Y to reduce the scope of the policy on Y , i.e. Y π R . By [43, Lemma 3.18.] , if there exists a separator W between two sets of variables X and Y such thatS Ď W Ď K in a graph G, i.e. X |ù d Y | W in G, then W " AnpX Ť Y Ť Sq ŞK is also a valid separator. The definition of W R in line 7 aims to leverage this result to check for the existence of separators in our case.Consider a R P R, let S " RzR, K " V zpX Ť C Ť Y q. In particular, if there exist a set tC, Ru Ă W Ă K such that pY |ù d R | W q in G π S ,R then W R " AnpR Ť Y Ť Cq Ş K " AnpY Ť Cq Ş K in G π S satisfies pY |ù d R | W R q in G π S ,R , i.e.W R is a separator if one exists. Note that W R does not include any member of tC, R, Y u. Assume this separation holds and that the if statement in line 8 is triggered. Then,E P rY π R | r, cs " E P rY r | cs " E P rY s,r | cs " ż E P rY s,r | c, w R sP pw R | cq dw R " ż E P rY s | r, c, w R sP pw R | cq dw R .The last equality holds by assumption pY|ù d R | W R q in G S,R .Now considering the term E P rY s | r, c, w R s, for the second pass through the for loop, we can see that if the d-separation statement in the if condition is fulfilled we can further reduce the scope of the intervention on S. In particular, for R 1 P RzR, assuming that the if statement is triggered for W R 1 we have that,E P rY s | r, c, w R s " ż E P rY s | r, c, w R , w R 1 sP pw R 1 | w R , cq dw R 1 " ż E P rY szr 1 | r 1 , r, c, w R , w R 1 sP pw R 1 | w R , cq dw R 1 .Upon reaching the end of the for loop we have for W defined in line 9, T " tR : W R P W u, U " RzT that,E P rY π | r, cs " ż E P rY u | t, c, wsP pw | cq dw.3. Find partial instrumental sets Z. Starting line 12, we look for partial conditional instrumental sets. Consider first the case that a potential instruments Z P W is evaluated in the for loop. On line 15, assume that the if statement is triggered, i.e. pY |ù d Z | W zZ, T q in G U It then holds that,E P rY π | r, cs :" ż E P rY u | w, t, csP pw | cq dw " ż E P rY u | wzz, z, t, cs ż z P pw | cq dw " ż E P rY u | wzz, z, t, csP pwzz | cq dwzz.In turn, for the case that someZ 1 R W is evaluated in the for loop, if pY |ù d Z 1 | W , T q in G U , E P rY π | r, cs :" ż E P rY u | w, t, csP pw | cq dw " ż E P rY u | w, t, z 1 , csP pw | cq dwFor the Z obtained in line 15, we derive that,E P rY π | r, cs " ż E P rY u | wzz, z, t, csP pwzz | cq dwzz.4. Return bounds. We now proceed to use the expression above to lower and upper bound the effect of interest. In particular, for the lower bound we have that, In turn, for the upper bound we have that, where γ :" πR 1 z pZq{P pT , Z | W zZ, Cq. Note that we have used the definition R " T Ť U .E P rY π | x, cs " ż E P rY u | wzz, z, t, csP pwzz | cq dwzz " ż ÿ u 1 E P rY u | u 1 ,E P rY π | x, cs " ż E P rY u | wzz, z, t, csP pwzz | cq dwzz " ż ÿ u 1 E P rY u | u 1 , wzz, z, t, csP pu 1 | wzz, z, t, cqP pwzz | cq dwzz " ż E P rY u | u, wzz, z, t, csP pu | wzz, z, t, cqP pwzz | cq dwzz `ż ÿ u 1 ‰u E P rY u | u 1 , wzz, z, t, csP pu 1 | wzz, z, t, cqP pwzz | cq dwzz ď ż E P rY u | u,Prop. 6 restated. Let k be the number of variables and m be the number of edges in G. The run time of Alg. 1 is Opkpk 2 `mqq.Proof. Let k be the number of variables and m be the number of edges in G. Operations in Alg. 1, such as computing ancestors (e.g. lines 2,7), could be done in Opk 2 q time, e.g. with a Breadth-First Search algorithm. Checking for d-separation is commonly done with the Bayes-Ball algorithm that can be implemented with a reachability search method, e.g. [43, Sec. 3.2.1], and could be done in Opk `mq time by [43, Prop. 3.17] . The sets R and Z have size at most k, so the for loops in lines 5 and 14 are executed at most k times each. Combining these we get that Alg. 1 requires Opkpk 2 `mqq time to return the bounds.Prop. 7 restated. Let π :" tπ X1 , π X2 u be a policy on tX 1 , X 2 u with a conditioning set C. All partial adjustment sets may be enumerated in time Opkpk `mqq where k are the number of variables and m be the number of edges in G.Proof. This proposition is adapts Proposition 3.20 in [43] to find partial adjustment sets using the ListSep algorithm. ListSep performs backtracking to enumerate all separator sets Z between X and Y such that I Ď Z Ď R, aborting branches that will not find a valid separator. In particular, it calls the TestSep and FindSep algorithms recursively.The TestSep algorithm takes as input a graph, two sets of nodes to be separated, and a candidate separator set. By setting the graph to be G π X 1 X2 , the two sets to be separated to be Y and X 2 , and the separator set Z, TestSep will provably return whether W " ZzpC, X 1 q is a partial adjustment set.The FindSep algorithm uses the observation that if there exists a separator Z between X and Y such that I Ď Z Ď R then Z :" AnpX Y Y Y Iq X R is a separator. For each possible partition pX 1 , X 2 q of X, we can therefore find a separator by testing (usingTestSep) whether AnpX Y Y Y Cq X pV zpX Y Y qq in G π X 1 X2 is a separator.If it is then we have found a partial adjustment set.To find all partial adjustment sets, we then proceed as follows. For each possible partition pX 1 , X 2 q of X, apply the ListSep algorithm with graph G π X 1 X2 , sets to be separated Y and X 2 , and possible separator sets Z constrained as C, X 1 Ď Z Ď V zpX Y Y q. Then, for every separator set Z found, output partial adjustment sets W " ZzpC, X 1 q.This procedure finds all possible partial adjustment sets in time Opkpk `mqq where k are the number of variables and m be the number of edges in G by Proposition 3.20 [43] . 
partial identification of the effect of policies: 
 A policy π over a subset X Ă V is a sequence of decision rules or plans π :" tπ X u XPX for determining the assignment of variables X. In its most general form, every π X : supp X ˆsupp C X Ñ r0, 1s is a probability mapping from domain of Qualitatively different types of interventions may be modelled with π X . Specifically, a deterministic intervention setting X :" gpc X q based on the values of C X can be encoded as π X px | c X q :" 1 gpc X q pxq while a probabilistic intervention may be written π X px | c X q :" P induced by π X pX " x | c x q.C X Ă C Ă V to the domain of X. X Y C (a) G X Y C (b) GπThe intervened model M π represents a different regime in which the assignment tf X u XPX is replaced with the assignment induced by π. For an outcome Y P V , the interventional distribution P Mπ pY q, equivalently P π pY q, is defined as the distribution over Y in M π . It will be useful to adopt the notation π :" ś XPX π X px | c X q to denote the product of policy assignments on individual variables subject to intervention in π. Example 1 (Illustration of policies). In the context of a public health program, let X be a measure of an individual's weekly physical exercise routine, C an individual's family history of metabolic diseases, and Y hospital admissions related to heart disease. Consider the causal diagram Fig. 1a ; currently, an individuals exercise voluntarily, which depends on unobserved factors that could be associated with risk of heart disease. The government is considering an incentive plan aimed at increasing the frequency of exercise depending individual's family history of metabolic diseases (with some probability). The proposed intervention can be encoded as π " tπ X u and sets the new assignment fX such that the variable X follows the pre-specified distribution. Graphically, this policy is represented by Fig. 1b (with the new edge corresponding to the implementation of the policy highlighted in blue) and may be evaluated with the following quantity,EQUATIONThe inferential challenge is that this quantity is not uniquely computable from P px, y, cq and G, requiring more complex and nuanced notions of policy evaluation.Formally, we are interested in the evaluation of the effectiveness of a plan or policy π, acting on a (potentially multivariate) discrete action X based on values of observed covariates C P R d , on an outcome of interest Y P r0, 1s 2 . Definition 1 (Average treatment effect). The effectiveness of a policy π on Y is E Pπ rY s.From the investigator's perspective, only the causal diagram G of the environment M is available.No assumptions about the form or shape of P pU q and F are made, but for the structural knowledge encoded in G. In general, there might exist multiple SCMs M that induce the same causal diagram G and entail P pV q but result in different values of E Pπ rY s. The identification of a set of possible solutions that contain the true value of E Pπ rY s leads to the notion of partial identification. Definition 2 (Partial Identification). The effectiveness of a policy π is said to be partially identifiable from G and P pvq if ψ ℓ pP q ď E P Mπ rY s ď ψ u pP q, for any M such that G M " G, P M pvq " P pvq,where pψ ℓ , ψ u q are functionals of P that are bounded away from 0 and 1, respectively.The earliest bound on the effect of policies, the so-called natural bounds, were developed considering discrete atomic interventions, i.e. of the form π X " 1 x pXq, X P t1, . . . , d X u.X 1 Y X 2 C (a) X 1 Y X 2 C (b) X 2 Y X 1 S Z (c) A X 1 Y X 3 X 2 C W (d)Figure 2 : Graphs used in Sec. 2 and 3.Proposition 1 (Natural Bounds). For any x, E P rY 1 x pXqs ď E Px rY s ď E P rpY ´1q1 x pXqs `1.Remarkably, these inequalities hold irrespective of the causal graph of the system G; a result that dates back to [27, 33] . Similarly, we could adapt the underlying proof strategy to derive functionals of P that bound the effect of more general probabilistic or conditional policies π, given as follows.Proposition 2 (Natural Policy Bounds (NPB)). For any π, E P rY πs ď E Pπ rY s ď E P rpY ´1q πs`1.These inequalities also hold irrespective of the causal graph of the system. The natural bounds from [27, 33] are a special case of the natural policy bounds (NPBs) by setting π :" tπ X u XPX where π X :" 1 x pXq, and therefore π " 1 x pXq. We could show that the NPBs are tight in some cases.For example, in Example 1, provably, no better bounds for E Pπ rY s than those defined by Prop. 2 could be derived (a result given in Prop. 10 in Appendix B). For other systems that involve variables that are "separated" in G, however, better bounds may be derived by exploiting the implications of the causal graph and definition of the policy on the induced observational and interventional distributions. Consider the following example as a first illustration of this idea.Example 2 (Tighter bounds with causal diagram). We are interested in evaluating the effect of a policy π :" tπ X1 px 1 | cq, 1 x2 pX 2 qu from P px 1 , c, x 2 , yq and G in Fig. 2a . In particular, this problem involves a stochastic conditional intervention on X 1 given C and a deterministic intervention on X 2 ; both variables having differing dependencies onto the rest of the system. We could show that,EQUATIONThe inequality gives an expression for the bound in terms of P that exploits the fact that intervening on X 2 does not influence Y , and is tighter that the natural policy lower bound as ψ ℓ pP q ě E P rY π X1 sP pX 2 " x 2 q " E P rY π X1 1 x2 pX 2 qs p" NPBq. For the upper bound we could similarly establish that,EQUATION)which is smaller than the natural policy bound as ψ u pP q ď E P rpY ´1qπ X1 1 x2 pX 2 qs `1 p" NPBq.This example, although relatively straightforward, serves to illustrate the potential of causal diagrams (and the constraints they imply on the effect of policies) for defining tighter bounds.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We include several remarks in the paper to qualify our contribution, e.g. the remark in Sec. 3, and include an Impact Statement in the Appendix to more thoroughly describe the limitations of our analysis, assumptions, and applicability in real-world settings.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
graphical criteria for partial identification: 
 This section aims to consider more general separation statements between variables encoded in a causal diagram and the decomposition they imply to provide a systematic algorithm to bound the effectiveness of policies. We start by introducing the notion of partial adjustment sets (Def. 3) that is applicable with multiple intervention variables. Definition 3 (Partial adjustment set). Let π :" tπ X1 , π X2 u be a policy on tX 1 , X 2 u with a conditioning setC. A set W Ď V zpX 1 Ť X 2 Ť C Ť Y q is said to be a partial adjustment set for π X2 in G if pY |ù d X 2 | W , C, X 1 q G π X 1 X 2 . Proposition 3.Let π :" tπ X1 , π X2 u be a policy mapping a set of covariates C to a set of treatment variables tX 1 , X 2 u. Let W be a partial adjustment set for π X2 in G. Then,EQUATIONwhere γ :" γpX, C, W q " 1{P pX 2 | C, W q.In words, partial adjustment sets are designed to exploit the unconfounded status of some intervention variables with respect to the outcome, and could be leveraged to derive tighter bounds when available. 1: Let C " Ť XPX C X , K " V zpX Ť C Ť Y q. /* 1. Omit redundant intervention variables. */ 2: Let R " X Ş AnpY q in G π . 3: Let π R :" tπ X pc X qu XPR ./* 2. Find partial adjustment sets W . */ 4: Initialize W " H, S " R. Let S " SzR.EQUATIONW " W Ť W R . To derive bounds in a more systematic fashion, combining the notions developed so far, we present Alg. 1 that recursively seeks to find partial adjustment and partial instrumental sets in an efficient and automatic manner.Intuitively, Alg. 1 seeks to recursively simplify the query. First by omitting the intervened variables that have no effect on the outcome; second by finding the set of intervened variables for which a partial adjustment set could be used to tighten the bound, and third by finding the set of variables that act as valid partial instrumental sets to evaluate bounds on the most favorable conditional distributions rather than on the joint distribution. In words, Prop. 5 says that for a given causal diagram, policy π, and joint distribution P pvq, the average treatment effect E Pπ rY s is contained in the bounds produced by Alg. 1. Proposition 6. Let k be the number of variables and m be the number of edges in G. The run time of Alg. 1 is Opkpk 2 `mqq. Example 5 (Steps of Alg. 1). For this example we consider evaluating a policy π :" tπ X1 p¨| Cq, π X2 p¨| Cq, π X3 p¨| Cqu from observational data P px 1 , c, x 2 , x 3 , a, w, yq compatible with G in Fig. 2d by explicitly following Alg. 1. On line 1, we define C " tCu, K " tA, W u. We start by omitting potentially irrelevant intervention variables by evaluating R " tX 1 , X 2 u and noticing that E Pπ rY s " E Pπ R rY s. To find partial adjustment sets in line 4, we evaluate the for loop iterating over R. For R " X 1 , we find that S " X 2 , W X1 " tA, W u but the if condition fails as Y is not d-separated from X 1 given tX 2 , C, A, W u. We turn onto R " X 2 , where S "X 1 , W X2 " tA, W u. The if condition is triggered as pY |ù d X 2 | X 1 , C, A, W q in G π X 1,X2 and therefore we update W " W X2 " tA, W u (giving a partial adjustment set). We continue with line 12, set T " tX 2 u, U " tX 1 u, and iterate over K " tA, W u in the for loop. For Z " A, we have thatpY |ù d A | W, X 2 q in G π X 1, the if condition is triggered and therefore we update Z " tAu. For Z " W , we find that pY K K d W | A, X 2 q in G π X 1 and therefore terminate the for loop. Finally putting together the pieces, we evaluate the bounds to be,max z E P rY γs ď E Pπ rY s ď min z E P rpY ´1qγs `1, γ :" πR 1 a pAq{P pX 2 , A | W, Cq (9) X 1 Y X 2 W 1 W 2 Figure 3: G Remark (Multiple bound expressions).Alg. 1 is designed to make use of partial adjustment and conditional instrumental variable sets but it will return a single bounding expression. In general multiple different expressions could be derived for a given causal diagram and data distribution. For example, given G in Fig. 3 and a policy π on tX 1 , X 2 u, E P r πY {P pX 2 | W 1 qs and E P r πY {P pX 2 | W 2 qs give valid, but numerically different lower bounds for E Pπ rY s. The superiority of one bound over another might depend on the associations in the underlying distribution P pvq. In this case, we could construct two different SCMs compatible with G that entail different orders in the numerical values for the two bounds, such that neither is always optimal. In general, we conjecture that finding an optimal analytical bound (using the proposed procedure) is undecidable from the graph alone.The observation in this remark motivates to develop a procedure to automatically enumerate all partial adjustment sets to facilitate the search for tighter bounds. The following proposition adapts the ListSep algorithm by [43] to enumerate all partial adjustment sets. Proposition 7 (Enumerating partial adjustment sets). Let π :" tπ X1 , π X2 u be a policy on tX 1 , X 2 u with a conditioning set C. All partial adjustment sets may be enumerated in time Opkpk `mqq where k are the number of variables and m be the number of edges in G.Remark (Related methods for policy evaluation). In parallel to the graphical approach for encoding structural assumptions about the domain at hand, a number of works have adopted sensitivity assumptions that quantify the degree of unobserved confounding through various data statistics, such as odds ratios, propensity scores, etc. A rich literature on sensitivity assumptions exists, including Tan's sensitivity model [38] and Rosenbaum's sensitivity model [34] . Under these models, [17, 21, 28, 45] , among others, present methods that achieve validity and rate properties for the resulting estimators. These approaches start with estimators that optimize the average or conditional treatment effect bounds subject to constraints implied by the sensitivity model. Several of these works leverage Neyman orthogonality techniques to obtain rate guarantees and doubly-robustness properties. For instance, [17, 21, 28] study the estimation of bounds under Tan's sensitivity model which quantifies the degree of unobserved confounding through odds ratios, and propose estimators with various validity, sharpness, and favourable convergence rate guarantees. In contrast, [45] study the estimation of bounds under Rosenbaum's sensitivity model also deriving estimators with sharpness and fast convergence guarantees. Some works under these assumptions have also considered policy evaluation (as opposed to the evaluation of atomic interventions) under various sensitivity models in the context of Reinforcement Learning, e.g., [7, 22, 23] . We interpret this line of work as complementary (applicable under different assumptions, i.e., sensitivity models rather than causal diagrams) to the techniques proposed in this paper.some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
estimation of the effect of policies: 
 This section aims to develop an estimation framework for the effect of a policy E Pπ rY s given finite samples from P that partially identifies its value. The key observation of this section is that multiple characterizations for estimation could be derived for a given bound returned by lines 20-21 in Alg. 1.In a first instance, parameterized by the probability ratio given in Alg. 1 defined by γ " pγ 1 , γ 2 q, T PW,ℓ :"EQUATIONIn a second instance, parameterized by a collection of regression parameters µ " pµ 0 , µ 1 , μ1 , µ 2 q whereEQUATIONEQUATIONEQUATIONEQUATIONT REG,ℓ :" E P rµ 0 pC, W , zqs could be shown to equal ψ ℓ z . Both formulations define equivalent but different estimation targets for the lower bound 3 and may be combined leveraging the double machine learning (DML) toolkit for more efficient and robust inferences [10] .The following procedure is the main contribution of this section. It defines a DML estimator for bounding the effectiveness E Pπ rY s of a policy π. Definition 5 (DML Estimator). Given π and G, let tR, T , W , C, Z, Y u be defined as in Alg. 1. Consider a finite sample of data D " P , randomly split into K folds. The k'th partition of the sample is denoted D pkq and D p´kq :" DzD pkq . For each k, learn approximate nuisances pγ k , μk q with D p´kq . Then, define´max z T DML,ℓ , min z T DML,u ¯(15)to be an estimate for the bounds on E Pπ rY s where,T DML,ℓ :" 1 K K ÿ k"1 E D pkq rγ 2,k tY ´μ 2,k us `ED pkq rγ 1,k t μ1,k ´μ 1,k us `ED pkq rμ 0,k s T DML,u :" 1 `1 K K ÿ k"1 E D pkq rγ 2,k tpY ´1q ´μ 2,k us `ED pkq rγ 1,k t μ1,k ´μ 1,k us `ED pkq rμ 0,k s.To analyse the error of the DML estimator, we consider the case that nuisances can be estimated consistently. Thus requirement is relatively mild in practice as accurate probability estimation employing off-the-shelf classification and regression methods is feasible in general. Its error with respect to the true bounds is given by the following proposition. Proposition 8 (Error rates). Suppose the nuisance estimates p μ, γq are L 2 -consistent and bounded.Then, the error of the DML estimator T DML P t T DML,ℓ , T DML,u u in Def. 5 is given as followsT DML ´T DML " 1 K K ÿ k"1 R k `OP ´}γ 2,k ´γ2 }}μ 2,k ´µ2 } ¯`O P ´}γ 1,k ´γ1 }}μ 1,k ´μ 1,k }where R k is a random variable that converges to zero at a rate O P p1{ ? nq.In words, the DML estimator exhibits a robustness property since the error of T DML is bounded in probability at n ´1{2 rate whenever the nuisances converge at a rate n ´1{4 . Note that the term } μ1,k ´μ 1,k } quantifies the error in approximating the conditional expectation in Eq. ( 13) for a given μ1 estimated at that stage, and that the estimation error of γ2 and γ2 are equivalent since they are deterministic transformations of each other. The following proposition is a corolloray that shows that the DML estimator is unbiased under misspecification. Proposition 9 (Bias under misspecification). Suppose either γ1 " γ 1 or μ2 " µ 2 and that either γ1 " γ 1 or μ1 " μ1 . Then, T DML P t T DML,ℓ , T DML,u u is an unbiased estimator of the corresponding bound defined in Alg. 1. 
experiments: 
 This section evaluates the quality of policy effect estimation from finite samples. Our goal is to illustrate the computation of bounds and provide empirical evidence of the fast convergence and robustness to misspecification of estimators.Finite sample bounds are estimated with gradient boosting classification and regression models (for conditional expectations) or by taking sample averages (for unconditional expectations). We truncated estimates of probability mass functions in the interval r0.01, 0.99s to ensure positivity. We assess the quality of an estimator T by computing the absolute average error (AAE) with respect to (a proxy for) the true bounds (estimated in practice with larger sample sizes), i.e., AAE = | T u ´min z ψ u z | `| T ℓ ´max z ψ ℓ z |.Throughout, we report various statistics: 25th, 50th, 75th percentile, etc., across evaluation runs with ten different random seeds. Further details of the simulations are provided in Appendix C.The synthetic simulations consider 4 data generating mechanisms constructed according to the graphs in Fig. 4 . They highlight the use of partial instrumental sets, partial adjustment sets, high-dimensional partial adjustment sets, and combinations of partial instrumental and adjustment sets of varying dimensionality. The task is to estimate bounds on the effectiveness of a policy π :" tπ X1 , π X2 u defined as follows.EQUATION)That is the policy assigns X 1 P t0, 1u randomly with probability 0.5, and assigns X 2 P t0, 1u as a function of C with the probability of X 2 " 1 increasing with the value of C.To estimate bounds on E Pπ rY s, we consider the proposed estimators, labelled: T PW , estimated with the nuisances in Eq. ( 10), T REG , estimated with the nuisances in Eq. ( 11), and T DML estimated with the procedure in Def. 5. Our evaluations test performance across 4 different settings designed to highlight various properties.• Setting 1: All nuisances estimated correctly. This setting aims to show that all estimators converge to the bound of interest.• Setting 2: Nuisances γ are sampled from a uniform distribution to induce misspecification in the estimation of γ.• Setting 3: Nuisances μ are sampled from a uniform distribution to induce misspecification in the estimation of µ. Settings 2 and 3 aim to demonstrate the doubly-robustness property of the DML estimator.• Setting 4: Noise ϵ is introduced in the estimation of all nuisances pγ, µq to emphasize error due to finite sample variation. Specifically, noise ϵ " Normalpn ´α, n ´αq, α " 1{4, that induces a slower rate of convergence as a function of sample size, inspired by [24, 20] . This setting aims to show that the fast convergence behavior of the DML estimator compared to competing estimators.The results are given in Fig. 4 4 . We observe that across all data generating mechanisms, estimators improve with the size of the dataset and converge under no misspecification (Setting 1) to the underlying bounds. It is interesting to note also the differing accuracy of estimators in the small sample regime. T PW , based the estimation of a ratio of probabilities, can be unstable with low sample sizes if the ratio denominator is estimated to be close to zero while T REG , based on a sequence of regression tasks, tends to be better behaved. T DML in contrast is constructed as a combination of elements of T PW and T REG . In particular, note in Def. 5 the use of nuisances µ and γ. As a result, T DML has quite a different performance profile. Settings 2 and 3 in Fig. 4 show that the DML estimator T DML is robust to misspecification in either the nuisances µ or γ that highlights the robustness property. Further, when decaying noise is introduced in the estimation of nuisances (Setting 4), the DML estimator outperforms in general with a faster convergence rate. We also observe that performance remains close to optimal with high-dimensional variables W , demonstrating that the DML estimator provides a practical toolkit for bounding in practice. This section evaluates the width of the bounds returned by exploiting the different graphical criteria provided in Sec. 3. The simulations are based on the data generating mechanism described by causal diagram illustrated in the fourth row of Fig. 4 . We consider evaluating the policy in Eq. ( 16) and compute the bounds obtained by applying Prop. 2 (most conservative), Prop. 3 (using the partial adjustment set W only), Prop. 4 (using the partial instrumental set Z only), and finally Alg. 1 (that combines all propositions and is the proposed approach). Fig. 5 gives the results over 10 seeds of the data and across multiple data sizes, highlighting the gain achieved by exploiting the causal structure using the proposed approaches.Remark (Actual width of bounds in practice). The majority of our empirical evaluations are spent on evaluating the accuracy of different methods at estimating bounds, without addressing whether the returned bounds are actually informative. In practice, the graph structure can play an important role in tightening bounds but the actual width of the bounds in a particular problem are primarily driven by the distribution of data. Here is an example to make this more concrete.For a given policy π, Prop. 2 defines tight bounds (under some circumstances) on E Pπ rY s. The width of this bound is given by 1 ´EP r πs that is ultimately driven by P pxq. This term may therefore evaluate to anything between 0 and 1 depending on P pxq and π. For more complex causal structures, the bounds proposed in Props. 3 and 4 (and Alg. 1) reduce the width of the interval above. Loosely written, from 1 ´EP r πs to 1 ´EP r πs ˆα for some α that is a function of the joint distribution P and the structure of the graph. But again, the actual width of the interval is ultimately determined by the values of P and π that may be large or small depending on the value probabilities involved.This section illustrates the evaluation of lifestyle recommendations for the mitigation of obesity in individuals from Colombia, Peru and Mexico [30] . The data was collected from anonymous users using a web platform, and includes reported obesity levels measured according to BMI pY q, age pAq, smoking status pSq, frequency of consumption of high caloric food pHq, whether individuals monitored their calorie intake pM q, family history being overweight pF q, exercise frequency pEq, and time using technology devices pT q. Obesity is a multi-factored medical condition for which several causes have been acknowledged in the literature; causal diagrams relating the variables above have been curated in several related studies [1, 9] . We considered these findings to construct the causal diagram in Fig. 6a that we assumed for this example.We aim to study the effect of a health campaign designed to lower the intake of high caloric food pHq and increase the frequency of exercise pEq on obesity levels pY q. For instance, we could hypothesize that the campaign leads to an increase in the observed proportion of individuals rarely consuming of high caloric food (H) from 0.12 to 0.5 and that of individuals doing exercise (E) regularly from 0.05 to 0.5. These statements could be formulated as a stochastic policy π α :" tπ α H , π α E u acting on H and E, with new assignments,EQUATIONWe consider the evaluation of expected BMI levels E P π α rY s that range from 12 to 50 in the population, with a mean of 29.3. First note that this or other policies acting on pH, Eq are not identifiable due to the bi-directed edge tH Y u, but may nevertheless be bounded using Alg. 1. We find that max t E P rY γs ď E P π α rY s ď min t E P rpY ´1qγs `1, γ :" ππα 1 t pT q{P pE, T | A, S, F q.(18) To illustrate the inference of. bounds with the DML estimator, we consider evaluating policies with α " 0.2, 0.4, 0.6, 0.8. Fig. 6b gives the results. The end-points of the intervals denote estimated lower and upper bounds. We see that policies that promote a healthier lifestyle (larger values of α) are expected to reduce obesity levels on average but substantial uncertainty is still expected. 
conclusions: 
 The evaluation of policies is arguably one of the critical ingredients enabling more personalized decision-making systems. When the effect of policies is not identifiable, bounds can provide an effective support for making informed decisions. In this paper we developed partial identification and estimation tools for bounding the effect of a (stochastic or conditional) policy given data and assumptions encoded in a causal graph. We introduced several graphical characterizations that induce tighter bounds, and developed an estimation framework that exhibit robustness to noise and fast convergence. The results of this paper were illustrated through synthetic simulations and a real-world health campaign example for the reduction of obesity levels.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We provided details where applicable. In our case, data splits, hyperparameters, optimizers, etc., are not significant for the implementation the method. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We report our results as box plots that include several statistics, e.g. median, 25th, 75th, percentiles, etc., to summarize our results for different seeds of the data and algorithm. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [NA] Justification: Experiments were run on a single CPU in under a few minutes of wall time.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: For the derivation of results, we use the notion of counterfactuals defined as follows.For an SCM M, arbitrary subsets of endogenous variables X, Y , the potential outcome of Y to intervention dopxq, denoted by Y x puq, is the solution for Y with U " u in the sub-model M x . It can be read as the counterfactual sentence "the value that Y would have obtained in situation U " u, had X been x.". Statistically, averaging u over the distribution P puq leads to the counterfactual variables Y x . The distribution of the variable Y x is denoted P pY x q. With this formalism, we can express all the quantities in the main body of this paper in the language of counterfactuals. For instance, the distribution of Y in sub-model M x can alternatively be written P pY x q " P x pY q. Moreover, E P rf pY x q | zs, also written E P p¨|zq rf pY x qs, denotes the conditional expectation of f pY q over P py x | zq. Similarly, the distribution over Y in the sub-model M π can be written P M pY π q, or P pY π q " P π pY q for short. See [31, Chapter 7] for further context on counterfactuals. Definition 6 (The Axioms of Counterfactuals, Chapter 7.3.1 [31]). For any three sets of endogenous variables X, Y , W in a causal model and x, w in the domains of X and W , the following holds:• Composition: W x " w implies that Y x,w " Y x .• Effectiveness: X w,x " x.• Reversibility: Y x,w " y and W x,y " w imply that Y x " y. Theorem 1 (Soundness and Completeness of the Axioms Theorems 7. 3.3, 7.3.6 [31] ). The Axioms of counterfactuals are sound and complete for all causal models.The following rules to manipulate experimental distributions produced by policies extend the docalculus and will be used for the proof of several theoretical statements [12] . Theorem 2 (Inference Rules σ-calculus [12] ). Let G be a causal diagram compatible with an SCM M, with endogenous variables V . For any disjoint subsets X, Y , Z Ď V , two disjoint subsets T , W Ď V zpZ Ť Y q (i.e., possibly including X), the following rules are valid for any intervention strategies π X , π Z , andπ 1 Z such that G π X π Z , G π X π 1Z have no cycles: • Rule 1 (Insertion/Deletion of observations):P π X py | w, tq " P π X py | wq if pT |ù d Y | W q in G π X .• Rule 2 (Change of regimes under observation):P π X ,π Z py | z, wq " P π X ,π 1 Z py | z, wq if pY |ù d Z | W q in G π X ,π Z ,Z and G π X ,π 1 Z ,Z• Rule 3 (Change of regimes without observation):P π X ,π Z py | wq " P π X ,π 1 Z py | wq if pY |ù d Z | W q in G π X ,π Z ,ZpW q and G π X ,π 1 Z ,ZpW qwhere ZpW q is the set of elements in Z that are not ancestors of W in G π X .Active experimentation by physically manipulating reality is generally not feasible for many consequential applications. This motivates the study of causal effect identification and estimation. The former studies the problem of inferring whether a unique expression or bound of the desired causal effect could be derived from the available data and assumptions. The latter studies the problem of providing efficient estimators from finite samples to compute causal effects or bounds on causal effects in practice. In the following, we review these two lines of work to better contextualize our contribution.Partial identification. The natural bounds were derived to demonstrate that useful inference about causal effects could be drawn without making identifying assumptions beyond the observed data [27] . Relatedly, an analysis of bounds of causal effects was also provided for studies with imperfect compliance under a set of instrumental variable assumptions that are not sufficient however to identify the causal effect of interest [33] . These bounds may be written in closed form and have recently been extended as a more general strategy for bounding causal effects given assumptions encoded in causal diagrams in discrete systems [5, 46] . Recent work has also considered bounds in closed form with access to both observational and interventional distributions [18] . These techniques were developed alongside a second line of research that employs a polynomial optimization program to compute causal bounds given a causal diagram [4] . They proposed a family of canonical models parameterized according to the causal diagram, reducing the bounding problem to a series of equivalent linear programs. [11] further used Bayesian techniques to investigate the sharpness of these bounds with regard to the observational sample size. Recently, [14, 48] describe a polynomial programming approach to solve the partial identification for general causal graphs. More recent proposals consider parameterizations in the space of linear combinations of a set of fixed basis functions [29] and neural networks [3, 16] .In parallel, a number of works have adopted sensitivity assumptions (as an alternative or in combination with a causal diagram) that quantify the degree of unobserved confounding through various data statistics, such as odds ratios, propensity scores, etc. A rich literature on sensitivity assumptions exists, including Tan's sensitivity model [38] and Rosenbaum's sensitivity model [34] . Under these models, [17, 21, 28, 45] , among others, present methods that achieve validity and rate properties for the resulting estimators. These approaches start with estimators that optimize the average or conditional treatment effect bounds subject to constraints implied by the sensitivity model. Several of these works leverage Neyman orthogonality techniques to obtain rate guarantees and doublyrobustness properties. For instance, [17, 21, 28] study the estimation of bounds under Tan's sensitivity model which quantifies the degree of unobserved confounding through odds ratios, and propose estimators with various validity, sharpness, and favourable convergence rate guarantees. In contrast, [45] study the estimation of bounds under Rosenbaum's sensitivity model also deriving estimators with sharpness and fast convergence guarantees.While some works have considered policy evaluation under various sensitivity models in the context of Reinforcement Learning, e.g., [7, 22, 23] , most works target estimation of bounds on the effect of atomic interventions. We interpret this line of work as complementary (applicable under different assumptions, i.e., sensitivity models rather than causal diagrams) to the techniques proposed in this paper.Causal effect estimation. With a causal diagram as input, causal effect estimation has traditionally focused on a subset of identifiable scenarios, relying on assumptions such as backdoor criterion or the availability of adjustment sets. Identification expressions in those cases are given by (sequential) covariate adjustments that have lead to statistically appealing estimators from observational data [32] . Notable examples are doubly robust estimators [10, 35, 42] . Recently these techniques have been extended to settings identifiable from multiple experimental distributions [8, 19, 20] . To our knowledge, no estimation framework specific to the estimation of analytical expressions derived for bounds on the effect of policies or atomic interventions has been developed.Our work investigates the conditions under which policies may be bounded from observational data. Issues of policy evaluation are central to fields of application involved in decision-making as well as AI and ML. We believe that a tool to bound the effect of stochastic and conditional policies in systems with unobserved confounding is an important addition to the scientific toolbox. Reasoning instead without acknowledging for the potential of unobserved confounding may lead researchers to operate on a more heuristical basis. For instance, the hypothetical effect of public policies might be misrepresented if informed by the evaluation of an idealized atomic intervention that is difficult to implement in practice. And, once implemented, might have unintended consequences. With this background, we believe that research on the partial identification and estimation of policies based on stochastic or conditional interventions can help scientists and individuals make more informed decisions.In this work, we start from the assumption that a causal diagram that is consistent with the underlying data generating system of interest is available. In general, this requires domain knowledge. While some of the bounds provided do not require full knowledge of the causal graph, whenever a dseparation is assumed its truth value should be justified by prior knowledge or experiment. It is important also to make the distinction between the task of partial identification, that is inferring an expression for bounds on causal effects, and that of estimation, that is providing efficient estimators from finite samples to compute bounds in practice. This set of results concerns both of these tasks. The first objective of our procedure is to provide an expression for lower and upper bounds, irrespective of the accuracy with which one is able to approximate P pV q from finite samples. The second objective is introduce efficient estimators for bounds on the effect of policies using finite samples from P pV q. In higher-dimensional systems, the computational complexity of estimating the conditional distributions that define lower and upper bounds on causal effects is a substantial challenge. Consequently, practitioners must exercise caution when deploying the proposed method in small sample scenarios where estimators may be inaccurate. Moreover, we have stated our convergence guarantees in the infinite sample limit, without quantifying the finite-sample estimation uncertainty. Finite-sample properties could be explored similarly to [24] given a particular choice of function class to extend our results with high-probability bounds. Finally, we emphasize that simulations on real and synthetic data are provided for illustration purposes only. These results do not recommend or advocate for the implementation of a particular policy, and should be considered in practice in combination with other aspects of the decision-making process.This section presents proofs for theoretical statements in the main body of this paper.Prop. 1 restated. For any x, E P rY 1 x pXqs ď E Px rY s ď E P rpY ´1q1 x pXqs `1.Proof. Consider the derivation of the lower bound. Consider the derivation of the upper bound. It holds that,E P rY x s " ÿ x 1 E P rY x | x 1 sP px 1 q ě E P rY x | xsP pxqE P rY x s " ÿ x 1 E P rY x | x 1 sP px 1 q " E P rY x | xsP pxq `ÿ x 1 ‰x E P rY x | x 1 sP px 1 q p1q ď E P rY | xsP pxq `ÿ x 1 ‰x P px 1 q" E P rY | xsP pxq `1 ´P pxq " E P rY 1 x pXqs `1 ´EP r1 x pXqs " E P rpY ´1q1 x pXqs `1.(1) follows by the boundedness of Y , here assumed bounded by 1 for simplifying the derivation.Proposition 10. The natural bounds are tight in general.Proof. We show this statement by providing a pair of SCMs that agree on the input observational distribution but evaluate to the lower and upperbounds, respectively, specified by the natural bounds. Let MpGq denote the space of SCMs that induce the causal diagram G in Fig. 1a . We introduce a pair of SCMs compatible with the causal diagram G that evaluate to lower and upper bounds respectively. Let X be binary, Y P r0, 1s, and C P R, and consider M 1 , M 2 P MpGq defined as follows,M 1 :" $ ' ' & ' ' % x :" f X pu 1 q c :" f C pu 2 q y :" " f Y px, c, u 1 , u 2 q if x " f X pu 1 q, 0 otherwise.and,M 2 :" $ ' ' & ' ' % x :" f X pu 1 q c :" f C pu 2 q y :" " f Y px, c, u 1 , u 2 q if x " f X pu 1 q, 1 otherwise.For a P M1 puq " P M2 puq, both SCMs agree on observational distributions P M1 px, y, cq " P M2 px, y, cq. However the following derivations show that the interventional expectation E P M 1 rY | dopx " 1qs differs across models: for M 1 equal to the analytical lower bound, and for M 2 equal to the analytical upper bound demonstrating that (in this case) the bound is tight. In particular,E P M 1 rY | dopx " 1qs " E P M 1 rY | x " 1, u : x " f X puqsP pu : x " f X puqq `EP M 1 rY | x " 1, u : x ‰ f X puqsP pu : x ‰ f X puqq " E P M 1 rY | x " 1sP px " 1q " E P rY | x " 1sP px " 1q " E P rY 1 1 pXqs E P M 2 rY | dopx " 1qs " E P M 2 rY | x " 1, u : x " f X puqsP pu : x " f X puqq `EP M 2 rY | x " 1, u : x ‰ f X puqsP pu : x ‰ f X puqq " E P M 2 rY | x " 1sP px " 1q `EP M 2 rY | x " 1, u : x " f X puqsP pu : x " f X puqq " E P M 2 rY | x " 1sP px " 1q `1 ´P px " 1q " E P rpY ´1q1 1 pXqs `1.Prop. 2 restated. For any π, E P rY πs ď E Pπ rY s ď E P rpY ´1q πs `1.Proof. Consider the derivation of the lower bound. Let π denote a policy from an arbitrary set of covariates C to X. By marginalizing over X, C we find that E Pπ rY s " E P rY π s "ż E P rY π | x, cs ź XPX π X px | c X qP pcq dcdx.C X denotes the subset of C that is used to inform the intervention on X. Moreover,E P rY π | x, cs p1q " E P rY x | cs " ÿ x 1 E P rY x | x 1 , csP px 1 | cq ě E P rY x | x, csP px | cq p2q " E P rY | x, csP px | cq.(1) holds by Rule 2 in Thm. 2 by swapping the policy π by the do intervention dopX " xq: given that the policy acts on X taking as inputs C it is always true that Y |ù d X | C in G πX and G XX in any graph G. The following equality follows by marginalizing over the domain of X, which is assumed discrete. (2) follows by the composition axiom of counterfactuals. By combining these two expressions we getE P rY π s ě ż ź XPX πpx | c X qP pcqE P rY | x, csP px | cq dcdx " ż πyP py | x, cqP px | cqP pcq dydcdx " E P rY πs.Consider the derivation of the upper bound. It holds that,E P rY π | x, cs " E P rY x | cs " ÿ x 1 E P rY x | x 1 , csP px 1 | cq " E P rY x | x, csP px | cq `ÿ x 1 ‰x E P rY x | x 1 , csP px 1 | cq p1q ď E P rY | x, csP px | cq `ÿ x 1 ‰x P px 1 | cq " E P rY | x, csP px | cq `1 ´P px | cq.(1) follows by the boundedness of Y , here assumed bounded by 1 for simplifying the derivation. As above by combining this inequality with the decomposition of the policy effect we getE Pπ rY s ď ż ź XPX πpx | c X qP pcq tE P rY | x, csP px | cq `1 ´P px | cqu dcdx " E P rY πs `1 ´EP r πs " E P rpY ´1q πs `1.Prop. 3 restated. Let π :" tπ X1 , π X2 u be a policy mapping a set of covariates C to a set of treatment variables tX 1 , X 2 u. Let W be a partial adjustment set for π X2 in G. Then,E P rY πγs ď E Pπ rY s ď E P rpY ´1q πγs `1, where γ :" γpX, C, W q " 1{P pX 2 | C, W q.Proof. Consider the derivation of the lower bound. By marginalizing over X, C we find thatE Pπ rY s " E P rY π s " ż E P rY π | x, cs ź XPX π X px | c X qP pcq dcdx.C X denotes the subset of C that is used to inform the intervention on X. Denote π :"ś XPX π X px | c X q. Let X 1 , X 2 be a partition of X such that W Ď V zpX Ť C Ť Y q is partial adjustment set for π X2 . Then, E P rY π | x, cs " E P rY π X 1 ,π X 2 | x, cs " ż E P rY x1,x2 | c, wsP pw | cq dw p1q " ż E P rY x1 | x 2 , w, csP pw | cq dw " ż ÿ x 1 1 ␣ E P rY x1 | x 1 1 , x 2 , w, csP px 1 1 | x 2 , w, cq ( P pw | cq dw ě ż E P rY x1 | x 1 , x 2 , w, csP px 1 | x 2 , w, cqP pw | cq dw " ż E P rY | x 1 , x 2 , w, csP px 1 | x 2 , w, cqP pw | cq dw.(1) follows by the definition of X 2 as a partial adjustment set: it holds that pY |ù d X 2 | W , C, X 1 q in G X1X2 which induces the equality E P rY x1,x2 | c, ws " E P rY x1 | x 2 , w, cs. The last equality follows by the composition axiom of counterfactuals. Combining this expression with the decomposition of the policy effect we get,E P rY π s ě ż ź XPX πpx | c X qP pcqE P rY | x 1 , x 2 , w, csP px 1 | x 2 , w, cqP pw | cq dwdcdx " ż πyP py | x 1 , x 2 , w, cqP px 1 | x 2 , w, cqP pw | cqP pcq dydcdx " E P rY π{P pX 2 | W , Cqs.For the upper bound, consider the following derivation,E P rY π | x, cs " ż E P rY x1 | x 2 , w, csP pw | cq dw " ż ÿ x 1 1 tE P rY x1 | x 1 1 , x 2 , w, csP px 1 1 | x 2 , w, cquP pw | cq dw " ż ! E P rY x1 | x 1 , x 2 , w, csP px 1 | x 2 , w, cqP pw | cq `ÿ x 1 1 ‰x1 E P rY x1 | x 1 1 , x 2 , w, csP px 1 1 | x 2 , w, cqP pw | cq ) dw ď ż ! E P rY | x 1 , x 2 , w, csP px 1 | x 2 , w, cqP pw | cq `ÿ x 1 1 ‰x1 P px 1 1 | x 2 , w, cqP pw | cq ) dw " ż ! E P rY | x 1 , x 2 , w, csP px 1 | x 2 , w, cqP pw | cq `p1 ´P px 1 | x 2 , w, cqqP pw | cq ) dw " ż E P rY | x 1 , x 2 , w, csP px 1 | x 2 , w, cqP pw | cq dw `1 ´ż P px 1 | x 2 , w, cqqP pw | cq dw.The inequality follows from the boundedness of Y and the rest of the arguments are analogous to the lower bound derivation. Combining this expression with the decomposition of the policy effect implies that,E Pπ rY s ď E P rpY ´1q π{P pX 2 | W , Cqs `1.Prop. 4 restated. Let Z be an unconditional partial instrumental set with respect to a policy π, i.e. Proof. Consider the derivation of the lower bound. Let π denote a policy from an arbitrary set of covariates C to X. Given that pY |ù d Zq Gπ , by marginalizing over X, C we find thatE Pπ rY s " E P rY π s " E P rY π | zs " ż E P rY π | x, c, zs ź XPX π X px | c X qP pc | zq dcdx.C X denotes the subset of C that is used to inform the intervention on X. Moreover, And since the l.h.s. does not depend on Z we can further tighten the bound by writing,E P rY π | x, c, zs p1q " E P rY x | c, zs " ÿ x 1 E P rY x | x 1 , c, zsP px 1 | c, zq ě E P rY x | x,E P rY π s ě max z E P rY π1 z pZq{P pZqs.Consider the derivation of the upper bound. It holds that,E P rY π | x, c, zs " E P rY x | c, zs " ÿ x 1 E P rY x | x 1 , c, zsP px 1 | c, zq " E P rY x | x, c, zsP px | c, zq `ÿ x 1 ‰x E P rY x | x 1 , c, zsP px 1 | c, zq p1q ď E P rY | x, c, zsP px | c, zq `ÿ x 1 ‰x P px 1 | c, zq " E P rY | x, c, zsP px | c, zq `1 ´P px | c, zq.(1) follows by the boundedness of Y , here assumed bounded by 1 for simplifying the derivation. As above by combining this inequality with the decomposition of the policy effect we getE Pπ rY s ď ż ź XPX πpx | c X qP pc | zq tE P rY | x, c, zsP px | c, zq `1 ´P px | c, zqu dcdx " E P rY π | zs `1 ´EP r π | zs " E P rpY ´1q π | zs `1 " E P rpY ´1q π1 z pZq{P pZqs `1.And since the l.h.s. does not depend on Z we can further tighten the bound by writing, E P rY π s ě min z E P rpY ´1q π1 z pZq{P pZqs `1.Prop. 5 restated. Alg. 1 is sound.Proof. For the soundness of Alg. 1, we will consider each operation in turn and show that recovered adjustment sets and conditional instrumental sets lead to a valid bound.We make use of two auxiliary lemmas from the literature. Lemma 1 (Continuous Mapping Theorem, [26] ). Let tX n u nPN , X be random elements defined on a metric space S. Consider a continuous function g : S Ñ S 1 (where S 1 is another metric space). Then,X n Ñ p X ñ gpX n q Ñ p gpXq.where the last equality follows from the result above. ThereforeEQUATIONholds by the equalities E P rγ 2 µ 2 s " E P rγ 1 μ1 s, E P rγ 2 μ2 s " E P rγ 1 μ1 s.Finally, this implies thatEQUATION)where R is a random variable that converges to zero at a rate O P p1{ ? nq.For the upper bound, consider the following definition of nuisance functions and estimators:γ 2 :" πU γ 1 , γ 1 :" πT 1 z pZq{P pT , Z | W zZ, Cq,with T PW,ℓ :" E P rγ 2 pY ´1qs `1 " ψ ℓ z . And µ " pµ 0 , µ 1 , μ2 , µ 2 q defined by µ2 :" µ 2 pR, C, W , Zq " E P rY ´1 | R, C, W , Zs, μ1 :" μ2 pR, C, W , Zq " πU pU | Cqµ 2 pR, C, W , Zq, µ 1 :" µ 1 pT , C, W , Zq " E P rμ 2 pR, C, W , Zq | T , C, W , Zs, µ 0 :" µ 0 pC, W , Zq " ÿ t µ 1 pt, C, W , Zqπ T pt | Cq.We first verify that the population level value of the regression estimator coincides with the lower bound ψ ℓ z . This can be seen with the following derivation, T REG,u :" E P rµ 0 pW zZ, C, zqs The arguments of the proof are now analogous to that of the lower bound. We can conclude therefore that,EQUATIONProp. 9 restated. Suppose either γ1 " γ 1 or μ2 " µ 2 and that either γ1 " γ 1 or μ1 " μ1 . Then, T DML P t T DML,ℓ , T DML,u u is an unbiased estimator of the corresponding bound defined in Alg. 1.Proof. Consider the estimated value of T DML,ℓ following the procedure in Def. 5, The last equality follows from the derivation of (C) in the proof of Prop. 8.T DML,ℓ :" 1 K K ÿ k"1If either γ1 " γ 1 or μ2 " µ 2 and that either γ1 " γ 1 or μ1 " μ1 then this expression equals 0 and we have that, E P r T DML,ℓ s ´T DML,ℓ " 0.An analogous result holds for the upper bound.This section provides details of the data generating mechanisms used for synthetic experiments, and implementations.As described in the experimental section, for estimating nuisances pγ, µq we used Gradient Boosting models for classification and regression where appropriate. We implemented the models using Python using the commands GradientBoostingClassifier() and GradientBoostingRegressor() using default hyperparameters. We systematically bound probability values in the interval r0.01, 0.99s to avoid propagating potentially large approximating errors.We make use of the following 4 data generating systems, summarized by the graphs in Fig. 4 . In this section, we use the notation 1tXu that equals 1 if the statement tXu is true, and equal to 0 otherwise.Example 1. The data generating mechanism for the first scenario is given by:U Z , U XY , U C " N p0, 1q, Z :" f Z pU Z q, C :" f C pU C q, X :" f X pZ, C, U XY q, Y :" f Y pX, C, U XY q,where, f Z pU Z q :" ´1 ¨1tU Z ď ´0.5u `0 ¨1t´0.5 ă U Z ă 0.5u `1 ¨1tU Z ě 0.5u, f C pU C q :" U C , f X pC, U XY q :" 1tU XY ą 1 `expt0.75C `0.5Z `0.5uu, f Y pX, C, U XY q :" p2X ´1q ´2 ¨p2U XY ´1q The outcome Y is then re-scaled to lie in the r0, 1s interval. The target for estimation can be derived with Alg. 1 and equals, for the lower and upper bound respectively, max zPt´1,0,1u E P rπ1 z pZqY {P pZ | Cqs, min zPt´1,0,1uE P rπ1 z pZqpY ´1q{P pZ | Cqs `1.Example 2. The data generating mechanism for the second scenario is given by:U X1Y , U X2W , U X2 , U X1 , U C , U W " N p0, 1q, C :" f C pU C q, W :" f W pU X2W q, X 1 :" f X1 pC, U X1Y , U X1 q, X 2 :" f X2 pU X2W , U X2 q, Y :" f Y pX 1 , X 2 , C, W, U X1Y q,where, f C pU C q :" U C , f W pU W q :" U X2W , f X1 pC, U X1 , U X1Y q :" 1tU X1Y ¨p1 `expt0.75C `0.5uq ą U X1 u, f X2 pU X2W , U X2 q :" 1tU X2W `UX2 ą 0u, f Y pX 1 , X 2 , C, W, U X1Y q :" p2X 1 ´1q ¨pC `1q `2 sinp2C ¨p2X 1 ´1qq ´2p2U X1Y ´1qp1 `0.5Cq `X2 `W. The outcome Y is then re-scaled to lie in the r0, 1s interval. The target for estimation can be derived with Alg. 1 and equals, for the lower and upper bound respectively,E P r πY {P pX 2 | W, Cqs, E P r πpY ´1q{P pX 2 | W, Cqs `1.Example 3. The data generating mechanism for the third scenario is given by: U X1Y , U X2W , U X2 , U X1 , U C " N p0, 1q, W :" tW 1 , . . . , W 100 u s.t. W i " N p0, 1q, i " 1, . . . , 100,C :" f C pU C q, X 1 :" f X1 pC, U X1Y , U X1 q, X 2 :" f X2 pW , U X2 q, Y :" f Y pX 1 , X 2 , C, W , U X1Y q,where,f C pU C q :" U C , f X1 pC, U X1Y , U X1 q :" 1tU X1Y ¨p1 `expt0.75C `0.5uq ą U X1 u, f X2 pW , U X2 q :" 1tW 1 `W2 `UX2 ą 0u, f Y pX 1 , X 2 , C, W , U X1Y q :" p2X 1 ´1q ¨pC `1q `2 sinp2C ¨p2X 1 ´1qq´2p2U X1Y ´1qp1 `0.5Cq `X2 `ÿ i"1,...,100W i {100.The outcome Y is then re-scaled to lie in the r0, 1s interval. The target for estimation can be derived with Alg. 1 and equals, for the lower and upper bound respectively,E P r πY {P pX 2 | W , Cqs, E P r πpY ´1q{P pX 2 | W , Cqs `1.Example 4. The data generating mechanism for the fourth scenario is given by:U Z , U X1Y , U X2 , U X1 , U C , U Z " N p0, 1q, Z :" f Z pU Z q W :" tW 1 , . . . , W 5 u s.t. W i " N p0, 1q, i " 1, . . . , 5, C :" f C pU C q, X 1 :" f X1 pC, Z, U X1Y , U X1 , Zq, X 2 :" f X2 pW , U X2 q, Y :" f Y pX 1 , X 2 , C, W, U X1Y q,where,f C pU C q :" U C , f Z pU Z q :" 1tU Z ą 0u, f X1 pU X1Y , U X1 , Z, U C q :" 1tU X1Y ¨p1 `expt0.75U C `0.5Z `0.5uq ą U X1 u, f X2 pW , U X2 q :" 1tW 1 `W2 `UX2 ą 0u, f Y pX 1 , X 2 , C, W , U X1Y q :" p2X 1 ´1q ¨pC `1q `2 sinp2C ¨p2X 1 ´1qq ´2p2U X1Y ´1qp1 `0.5Cq `X2 `W1The outcome Y is then re-scaled to lie in the r0, 1s interval. The target for estimation can be derived with Alg. 1 and equals, for the lower and upper bound respectively, max zPt0,1uE P rπ1 z pZqY {P pZ, X 2 | W , Cqs, min zPt0,1u E P rπ1 z pZqpY ´1q{P pZ, X 2 | W , Cqs `1.The data was curated from anonymous from Colombia, Peru and Mexico, using a web platform [30] .The exact questions considered in the survey can be found in [30] . The authors made the data available under a Creative Commons license 5 and is currently hosted by Kaggle as a c.s.v file, accessible through the following link: kaggle.com/code/mpwolke/obesity-levels-life-style/. We considered a number of pre-processing steps. In particular, we created a BMI variable from weight and height data that were a posteriori removed from the dataset, imputed (with median or mode among observed instances) missing values, scaled continuously-valued covariates by subtracting the mean and dividing by the standard deviation, and scaled the outcome BMI to lie in the interval r0, 1s. Several covariates were ignored, including data on the consumption of water, consumption of vegetables, consumption of food between meals, consumption of alcohol, and number of main meals. These may be interpreted as unobserved variables, potentially confounding other relationships in the data as specified by the assumed causal diagram. The data from a total of 2111 individuals was recorded in this study.The bounds on the effect of a policy π α :" tπ α H , π α E u acting on H and E, with new assignments defined as tP new pH " rarelyq " αu, P new pE " regularlyq " αu are approximated from the expression in Prop. 4. Following this proposition we find that Age pAq, Smoking pSq, calories consumption Monitoring pM q, and Family history with overweight pF q, that is W " tA, S, M, F u, form a partial adjustment set for π α E given that pY |ù d E | W , Hq G π H E . We can verify also that Z " tT u is a partial instrumental set for π since pY |ù d T q Gπ . For illustration, the corresponding mutilated diagrams are shown in Fig. 7 . Finally, we partition the data equally to obtain two sets of samples for training first and second stage classifiers and regressors, respectively, obtaining a first estimate of bounds, before then switching the role of the two data samples and averaging over the resulting estimates. Justification: The claims made match the theoretical and experimental results presented in the paper. The impact statement in the Appendix reflects how much the results can be expected to generalize to other settings.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We have read the guidelines and we do not think that our work presents any notable concerns.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).An upper bound of 1 is assumed for simplicity; the following results could be generalized to more general bounded random variables.For the upperbound, a similar construction could be derived by defining T PW,u :" EP rγ2pY ´1qs `1, and T REG,u :" EP rµ0pC, W , zqs `1 with Y replaced by Y ´1 in the definition of µ2. Both estimators could be shown to be unbiased, i.e. equal to ψ ℓ u defined in Alg. 1.For the first row, the policy evaluated is: πX :" πX pX " 1 | cq " 1{p1 ´expt´cuq.creativecommons.org/licenses/by/4.0/ 


Decision: I concur with the reviewers' positive view of the submission and think the paper would makes a nice addition to the nascent literature on this topic. I am happy to recommend its acceptance. However, I find that deferring all discussion of related work to the appendix to be inappropriate. Many papers on the precise subject of the paper (evaluation of policies/ATEs/HTEs under confounding) are cited but not discussed in the main text ([5,17,18,19,24,40]). I understand there are important differences to these works in the technical approach, and I appreciate that the works that the present papers' technical approach relies on are discussed somewhat in the main text, but additionally all works directly related to the problem itself, even with alternative technical approaches to it, should be discussed in the main text and compared to. So I ask the authors to prioritize discussing these in the main text, even over technical details or other extras that are more appropriate for deferring to the appendix. This should also ameliorate the concerns of the more negative reviewers regarding context in the literature. I fully trust that the authors can undertake this appropriately within the preparation of a camera ready version as this mostly involves moving content around from/to the appendix and/or adding a couple sentences here and there.

Reviews: review: paper_summary: paper_summary: The paper proposes a new framework that uses structural causal graphs for estimating the effect of policies under unobserved confounding. Their approach first derives tighter analytical bounds on the estimates of the effects of the policies under the structural causal graph setting and the paper uses the results to develop estimators of these bounds using the double machine learning toolkit. The provide numerical experiments to show their approach produces good estimates of the effects of the policies under different settings such as misspecification.
strength_weakness: strength_weakness: The paper is somewhat hard to follow as the authors of the paper assume readers are familiar with existing notation for structural causal graphs. For example, they assume readers know the notation "an, de" for such graphs. Much of the notation is defined ahead of time, but without context for how and where they are used throughout the paper. Additionally, some new notation appear without explanation and are left to be interpreted by the reader. For example, in the partial identification section, $\bf{X}_1, \bf{X}_2$ are introduced, but the paper provides minimal intuition for what each set of variables means or should mean. As a result, the paper in general is hard to follow as it is overloaded with notation but limited intuition. Potentially, more concrete examples could be provided. The partial identification section in particular could benefit from figures of graphs that satisfy definition 3. 

In the numerics, it may be helpful to develop some sort of baseline to demonstrate how the new estimation framework out performs existing approaches. 

The paper also seems to have various typos and broken references. There are missing citations at line 67 and 311 and a missing proposition reference on line 316.
questions: questions: 1. Can more detailed be given on differences between $T^{TW}$ and $T^{REG}$ that can explain the differences in the numerics? Is $T^{DML}$ derived from one of the two estimators combined with the DML toolkit and if so which one is used? 
2. Since the approaches provide an upper and lower bound on the effect of the policies, how do you select an estimate if the difference between the bounds is large?
limitations: limitations: Limitations have been addressed by authors.
paper_summary: paper_summary: This paper proposes a new method to partially identify policy effects using a known causal graph and observational data. In theory, the paper derives general bounds for probabilistic and conditional policies, generalizing the existing results on discrete atomic policies. To estimate the bounds, the authors extend the well-known double machine learning framework to their setup. Finally, simulations are provided to demonstrate the convergence rate of the estimators. Then, the estimators are applied to estimate bounds for a range of policies in an obesity study.
strength_weakness: strength_weakness: 1. I think the paper can discuss a bit more about the cases in which the policy effect is identifiable, given a known graph and observational data, before moving to partial identification. Generally, partial identification is valuable when the assumption required for point identification may not hold in practice. The authors can potentially discuss the assumption and highlight the value of their method.
2. I didn't see any assumption required to derive the natural bound and its extension proposed by the authors. If the bound is assumption-free, would it be too conservative to use in practice? The authors should clarify this. 
3. The estimation strategies in the paper are quite standard. The authors should explain why this part is novel compared to the existing literature.
questions: questions: N.A.
limitations: limitations: N.A.
paper_summary: paper_summary: This paper proposes a partial identification method for the value of policies that intervene on a potentially multivariate set of variables in a known causal graph. It searches for adjustment sets or IVs that apply to one set of variables intervened on conditioned on others.
strength_weakness: strength_weakness: (1) It would be valuable to have more intuition about what the method is doing. My heuristic understanding right now is that it is combining bounds for policies that intervene on two sets of variables, one of which can be made to satisfy a no-unobserved-confounders type assumption given the graph, and one of which cannot (and hence we get fairly trivial bounds for that portion of the treatment variables). What would the authors say is the right way to understand the strategy? Similarly, it would be useful to contrast explicitly with, e.g., previous strategies for partial identification with discrete variables. 

(2) The majority of the experiments are spent on evaluating accuracy of the DML method at estimating the bounds. This is valuable, but for partial identification methods the more important question is likely to be whether the bounds are informative. It would be more useful to have experiments that assess what kinds of graph structures lead to more or less informative bounds, to guide potential users about when the method is likely to be useful, even if it meant pushing some of the current experiments content to an appendix
questions: questions: See (1) above.
limitations: limitations: Yes
paper_summary: paper_summary: The authors develop graphical characterizations and estimation tools to bound the effect of policies given a causal graph and observational data collected in non-identifiable settings.
strength_weakness: strength_weakness: Please refer to Questions.
questions: questions: Can the authors highlight the main challenge/novelty of the proposed bound for policy compared to bounds for treatment effect?
limitations: limitations: Please refer to Questions.

Comments: comment: I appreciate the responses to my questions and the numerics. I will increase my score.
With the discussion period coming to its end, we were wondering whether you had a chance to check our rebuttal. Please don't hesitate to get in touch if there is any concern we could still help to clarify.

Thank you.
We hope to have answered all concerns to your satisfaction in the rebuttal above. Please don't hesitate to get in touch if there is any concern we could still help to clarify.

Thank you again for the positive feedback.
The discussion period is reaching its end. We hope you have had the chance to check our rebuttal and wonder whether it has answered your questions. If not, we would be happy to expand on any remaining concerns.

We appreciate your time and attention. Thank you!
The discussion period is ending soon. We were wondering whether you had a chance to check our rebuttal. We hope to have answered all concerns to your satisfaction. If not, we would be happy to provide further comments and clarifications, let us know.

Thank you again for your time and attention.