Scalable Sensitivity and Uncertainty Analyses for Causal-Effect Estimates of Continuous-Valued Interventions
Authors: Andrew Jesson
Alyson Rose Douglas
Peter Manshausen
Maëlys Solal
Nicolai Meinshausen
Philip Stier
Yarin Gal
Uri Shalit
Venue: NeurIPS 2022 Accept (2022)
Keywords: causal effect inference
hidden confounding
continuous intervention
climate
uncertainty
dose response
deep learning

Abstract: Estimating the effects of continuous-valued interventions from observational data is a critically important task for climate science, healthcare, and economics. Recent work focuses on designing neural network architectures and regularization functions to allow for scalable estimation of average and individual-level dose-response curves from high-dimensional, large-sample data. Such methodologies assume ignorability (observation of all confounding variables) and positivity (observation of all treatment levels for every covariate value describing a set of units), assumptions problematic in the continuous treatment regime. Scalable sensitivity and uncertainty analyses to understand the ignorance induced in causal estimates when these assumptions are relaxed are less studied. Here, we develop a continuous treatment-effect marginal sensitivity model (CMSM) and derive bounds that agree with the observed data and a researcher-defined level of hidden confounding. We introduce a scalable algorithm and uncertainty-aware deep models to derive and estimate these bounds for high-dimensional, large-sample observational data. We work in concert with climate scientists interested in the climatological impacts of human emissions on cloud properties using satellite observations from the past 15 years. This problem is known to be complicated by many unobserved confounders.

Scalable Sensitivity and Uncertainty Analyses for Causal-Effect Estimates of Continuous- Valued
Interventions
Andrew Jesson' OATML
Alyson Douglas AOPP
Peter Manshausen
AOPP
Department of Computer Science University of Oxford
Department of Physics University Oxford
Department of Physics University of Oxford
Maëlys Solal
Nicolai Meinshausen
Philip Stier
Department of Computer Science University of Oxford
Seminar for Statistics
AOPP
Department of Mathematics
Department Physies University of Oxford
ETH Zurich
Yarin Gal OATML
Uri Shalit
Machine Learning and Causal Inference in Healthcare Lab Technion Israel Institute of Technology
Department of Computer Science University Oxford
Abstract
Estimating the effects of continuous valued interventions from observational data is critically important task for climate science, healthcare, and economics. Recent work focuses on designing neural network architectures and regularization func- tions to allow for scalable estimation of average and individual-level dose-response curves from bigh-dimensional, large-sample data. Such methodologies assume ignorability (observation of all confounding variables) and positivity (observation of all treatment levels for every covariate value describing set of units), assump- tions problematic the continuous treatment regime. Scalable sensitivity and uncertainty analyses to understand the ignorance induced in causal estimates when these assumptions are relaxed are less studied. Here, we develop a continuous treatment-effect marginal sensitivity model (CMSM) and derive bounds that agree with the observed data and a researcher-define level hidden confounding. We introduce a scalable algorithm and uncertainty-ave -ware deep models to derive and estimate these bounds for high-dimensional. large-sample observational data. We work in concert with climate scientists interested in the climatological impacts of human emissions on cloud properties using satellite observations from the past 15 years. This problem is known to be complicated by many unobserved confounders.
Introduction
Understanding the causal effect of a continuous variable (termed "treatment") individual units and subgroups is crucial across many fields. In economics, we might like to know the effect of price on demand from different customer demographics. In healthcare, we might like to know the effect of medication dosage on health outcomes for patients of various ages and comorbidities. And in
"Correspondence to andrew. jew.ewssssssssss.ooo ax.ac. uk
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

climate science, we might like to know the effects of anthropogenic emissions on cloud formation and lifetimes under variable atmospheric conditions. In many cases, these effects must be estimated from observational data as experiments are often costly, unethical, or otherwise impossible to conduct.
Estimating causal effects from observational data can only be done under certain conditions, some of which are not testable from data. The most prominent are the common assumptions that all confounders between treatment and outcome are measured ('no hidden confounders"), and any level of treatment could occur for any observable covariate vector ("positivity"). These assumptions and their possible violations introduce uncertainty when estimating treatment effects. Estimating this uncertainty is crucial for paaisiionmaking and scientific understanding, For example, understanding how unmeasured confounding can change estimates about the impact of emissions on cloud properties can help to modify global warming projection models to account for the uncertainty it induces.
We present novel marginal sensitivity model for continuous treatment effects. This model is used to develop a method that gives the user a corresponding interval representing the "ignorance region" of the possible treatment outcomes per covariate and treatment level [D' A19] for specified level of violation of the no-hidden confounding assumption. We adapt prior work [Tan06, KMZI9, JMGS21] to the technical challenge presented by continuous treatments. Specifically, we modify the existing model to work with propensity score densities instead of propensity score probabilities (see Section 3 below) and propose a method relate ignorability violations to the unexplained range of outcomes. Further, we derive bootstrapped uncertainty intervals for the estimated ignorance regions and show how to efficiently compute the intervals, thus providing a method for quantifying the uncertainty presented by finite data and possible violations of the positivity assumption. We validate our methods on synthetic data and provide an application on real-world satellite observations of the effects of anthropogenic emissions on cloud properties. For this application, we develop a new neural network architecture for estimating continuous treatment effects that can take into account spatiotemporal covariates. We find that the model accurately captures known patterns of cloud deepening in response to anthropogenic emission loading with realistic intervals of uncertainty due to unmodeled confounders in the satellite data.
Problem Setting
Let the random variable X X' model observable covariates. For clarity, will assume that X' is d-dimensional continuous space: X' S IR but this does not preclude more diverse spaces. Instances of X are denoted by x. The observable continuous treatment variable is modeled as the random variable T E T S IR. Instances of are denoted by t. Let the random variable Y E ) C IR model the observable continuous outcome variable. Instances of Y are denoted by y. Using the Neyman- Rubin potential outcomes framework [Ney23, Rub74, Sek08], we model the potential outcome of a treatment level by the random variable Y. E V. Instances of Y, are denoted by yt. We assume that the observational data, Da. consists of n realizations of the random variables, Dn = -(xi, L4s vs));- We let the observed outcome be the potential outcome of the assigned treatment level, yv Yt, thus assuming on-inneerferenee and consistency [Rub80]. Moreover, we assume that the tuple (x)) are i.i.d. samples from the joint distribution P(X, T, YT), where YT= =Y,, : t 7}.
We are interested in the conditional average potential outcome (CAPO) function, dose-response function
u(x, t), and the average potential outcome (APO) (tt), for continuous valued treatments. These functions are defined by the expectations:
u(x, t) = E [Y, X x]
(1)
p(t) = E [x(,
Under the assumptions of ignorability. Yr 1 T X, and positivity, p(t
X
x) 0 : Vt E T, Vx E X jointly known strong ignorability [RR83]
the CAPO and APO are identifiable from the observational distribution P, as:
f((xt) = E|Y |T
(3)
mtt) E (X)
In practice however, these assumptions rarely hold. For example, there will almost always be unobserved confounding variables, thus violating the ignorability (also known as unconfoundedness or exogeneity) assumption, YT L T X. Moreover, due to both the finite sample of observed data, D, and also the continuity of treatment T, there will most certainly be values, T = i, that are unobserved for a given covariate measurement, X = x, leading to violations or near violations of the positivity assumption (also known as overlap).
2

---

3 Methods
We propose the continuous marginal sensitivity model (CMSM) as new marginal sensitivity model (MSM [TTn06]) for continuous treatment variables. The set of conditional distributions of the potential outcomes given the observed treatment assigned, {P(Y, T = t, X = x) :t T}. are identifiable from data, D. But, the set of marginal distributions of the potential outcomes, {P(Y, X x): : 7. each given as a continuous mixture,
P -f- f x)P T - x)dt',
are not. This is due to the general unidentifiability of the component distributions, P(Y, T t', X = =x), where Y, cannot be observed for units under treatment level T = t' for * the well- known "fundamental problem of causal inference" [Hol86]. Yet, under the ignorability assumption, the factual P(Y, T t, X x) and counterfactual P(Y, T t', X x) are equal for all E'E T. Thus, P(Y, x= and P(Y, T = I, X x) are identical, and any divergence between them is indicative of hidden confounding. But, such divergence is not observable in practice.
The CMSM supposes degree of divergence between the unidentifiable P(Y, X = x) and the identifiable P(Y. T =, x. = x) by assuming that the rate of change of P(Y, X x) with respect to P(Y, t, x) bounded by some value greater than or equal to 1. The Radon-
Nikodym derivative formulates the divergence, A(ys; x, 1) dP JPLY, X=xx] under the assumption that P(Y, X =xx x) is absolutely continuous with respect to P(Y, T=t,X x), Vt ET
dPIY, X-x)
Proposition 1. Under the additional assumption that P(Y, =Ttxx-x x) and the Lebesgue measure are mutually absolutely continuous, the Radon-Nikodym derivative above is equal to the ratio between the unidentifiable "complete" propensity density for treatent p(t yt, Xt,)) and the identifiable "nominal" propensity density for treatment p(t x).
p(t x)
Xxrixxt) = p(1 x)
(5)
Proof (Appendix A.3) and an analysis of this proposition are given in Appendix A.
The value A(y;: x, t) cannot be identified from the observational data alone; the merit of the CMSM that enables a domain expert to express their belief in what is a plausible degree hidden confounding through the parameter Z 1. Where, A-1 < p(t x)/p(t Yt. x) S A, reflects a hypothesis that the "complete". unidentifiable propensity density for subjects with covariates X = x can be different from the identifiable "nominal" propensity density by at most factor of A. These inequalities allow for the specification of user hypothesized complete propensity density functions, p(t y, x), and we define the CMSM as the set of such functions that agree with the inequalities.
Definition 1. Continuous Marginal Sensitivity Model (CMSM)
P(A) = (aA)=//
{
p(t {p(
1 p(t x) p(L X6,
x2
SA.
(6)
A
Remark. Note that the CMSM is defined in terms of density ratio, p(t x)) x)/p(t yt, x), whereas P(tx) ((tlv..))
the MSM for banury treatments defined in terms of an odds ratio, [I-((t [T-P(
Importantly, naively substituting densities into the MSM for binary-treatments would violate the condition that À > 0 as the densities p(t x) or p(t y). can each be greater than one, which would result in a negative less so. We offer transformation of the sensitivity analysis parameter in terms of the unexplained range of the outcome later.
p(t -). The odds ratio is familiar to practitioners. The density ratio is
3.1 Continuous Treatment Effect Bounds Without Ignorability
The CAPO and APO (dose-response) functions cannot be point identified from observational data without ignorability. Under the CMSM with given A, we can only identify set of CAPO and APO functions jointly consistent with the observational data D and the continuous marginal sensitivity model. All of the functions in this set are possible from the point of view of the observational data
3

---

alone. So to cover the range of all possible functional values, we seek an interval function that maps covariate values, X x, the upper and lower bounds of this set for every treatment value, t.
For E T and x E X, let p(y: t, t, denote the density of the distribution P(Y: = x, x). As reminder, this distribution is identifiable from observational data, but without further assumptions the CAPO, (x(xt) t) = E [Y, X = x). is not. We can express the CAPO in terms of its identifiable and unidentifiable components as
p(y,]txx] dy: f(x,t) = J Yt p(uy)..)
M(xx,t)+ 1)+ + Jy 1(, x),xx(xxx 7x, t))p(y [t, xxxddy (A2 - 1) 1)-1 + fu w(y, x)p(y (x,)ddy
J3 11,0 dy:
p(t)yr.))
= ju(uly, x): (,,, A)
where, by a one-to-one change of variables, p(tly.,x) Ap(t|x] + w(y, x)( p(t]x}
Ap(tix) ) with
y X' [0, 1]. Both [KMZ19] and later [JMGS21] provide analogous expressions for the CAPO in the discrete treatment regime under the MSM, and we provide our derivation in Lemma 1.
The uncertainty set that includes all possible values of w(y, x) that agree with the CMSM, i.e., the set of functions that violate ignorability by no more than A, can now be expressed as W {w w(y,x) [0. 1] Vy V/ Yx E X}.
With this set of functions, we can now define the CAPO and APO bounds under the CMSM. The CAPO lower, μu(x, t; A), and upper, 7i t; A), bounds under the CMSM with parameter A are:
10(x 1;
inf p(w(y, x); x, wEW
Ta(x, A) = sup p(w(y, x); t,
weW
(8)
(9)
inf ((w(y):xxt,,) wEW
= sup m(w(y); x, L, wEW
Where
{w w(y) = H(yh y)yyee>, and W {w: : wvy) = H(y - (o))ya))aee ya)lywe and (-) is the Heaviside step function. Lemma 2 in ap-
the sets W =
pendix D proves the equivalence in eq. (9) for bounded Y. The equivalence in eq. (8) can be proved analogously.
The APO lower, p(t; A). and upper, At; bounds under the CMSM with parameter are: (10)
n(t; A) = E [M(X, A]]
[ A) = E [ [2
(11)
Remark. It is worth pausing here and breaking down Equation (7) to get intuitive sense of how the specification of A in the CMSM affects the bounds on the causal estimands. When + 1, then the (A² - 1)-1 term (and thus the denominator) in Equation (7) tends to infinity. As result, the CAPO under A converges to the empirical estimate of the CAPO - p(w(y); x,, A 1) - 7((x, t) as expected. Thus, the supremum and infimum in Equations (8) and (9) become independent of w, and the ignorance intervals concentrate on point estimates. Next, consider complete relaxation of the ignorability assumption, A oc. Then, the (A² 1)-1 term tends to zero, and we are left with,
u(w; A 20) - ji(x, 1) fo w(y) (xx, t))p(y t, x) Jy w(y)p(( x,x),y,
M)(x + E Y - M(x, ()], p(w()))x.i)
(y))((ylxxx) Ty w(y"))(( 7 a distribution over Y given = x and T = t. Thus, when we relax the ignorability assumption entirely, the CAPO can be anywhere in the range of Y.
where, p(w(y) x, t)
The parameter A relates to the proportion of unexplained range in assumed to come from unob- served confounders after observing x and When user sets A to 1, they assume that the entire unexplained range of Y comes from unknown mechanisms independent of T. As the user increases A, they attribute some of the unexplained range of Y to mechanisms causally connected to T. For bounded Y,, this proportion can be calculated as:
x(x, A) - p(x, t; A)
TT(x, t; A) - ((x, t; A)
p(x,t: t;
7 t; + ,(x, A; x0)
Ymax
Ymin
X x1 T
The user can sweep over a set of A values and report the bounds corresponding to a P value they deem tolerable (c.g., p 0.5 yields bounds for the assumption that half the unexplained range in Y is due

---

to unobserved confounders). For unbounded outcomes, the limits can be estimated empirically by increasing to large value. Refer to Figure 10 the appendix for a comparison between p and A.
For another way to interpret A, in Appendix A.3.1 we A can be presented as a bound on the Kullback-Leibler divergence between the nominal and complete propensity scores through the relationship: log (A)l 2 DKL (P(Y, =
X =xx).
3.2 Parimearr Interval Estimator
Following [JMGS21], we develop marammrraettiic estimator of the bounds in eqs. (8) to (11). Un- der assumption A, the bounds on the expected potential outcome over ruw((y;; I, A) are com- pletely defined terms of identifiable quantities: namely, the conditional density of the outcome given the assigned treatment and measured covari- ates, p(y t, x); and the conditional expected out- come [x(, t). Thus, we define a density estimator, p(y x, x),, and estimator, /(x,t:: t; 0), parame- terized by instances A of the random variable @. The choice of density estimator is ultimately up to the user and will depend on the scale of the prob- lem examined and the distribution of the outcome variable Y. In Section 3.5, we will outline how define appropriate density estimators for high- 8:
Algorithm Grid Search Interval Optimizer
Require: x is an instance of X, is a treat- ment level evaluate, A is belief in the amount of hidden confounding, 0 are op- timized model parameters, y is a set of unique values {y ~ ply 1, x, 0)).
1:
function GRIDSEARCIIX, t, A, 0, 57)
2:
P 00, +
3:
P + xC, 4
4:
for YH E 5 do
5:
4- u(H(y -
6:
K 6- u(H((H - X),,,,
7:
if > then
ІT f- R. + YH
dimensional, large-sample, continuous-valued treat- 9:
if' < then
ment problems. Next, we need an estimator of the in- tegrals in u(w(y);x, t, A, 0), eq. (7). We use Monte- Carlo (MC) integration to estimate the expectation of arbitrary functions h(v) with respect to the para-
1 f & y + YH
11: return y,
metric density estimate p(y x, 0): I(h(y)) = m Σn h(y:), Y: ~ ply x, 0). We outline how the Gauss-Hermite quadrature rule is an alternate estimator of these expectations in Appendix C. The integral estimators allow for the semi-pane cami-parametric estimators for the CAPO and APO bounds under the CMSM to be defined.
The cemi-parametric CAPO bound estimators under the CMSM with sensitivity parameter A are:
(xx, 0) inf p(w(y):xx (12) (xx,;A,,) = sup t(,,,));x,,AA (13) wEW#
wEW!
where,
I (w(y)(y - (A² 11-111-1- 1 (w)(y)
u(w(y);x,t,A [xx t; 0) +
The semi parametric APO bound estimators under the CMSM with sensitivity parameter A are: 0)] (15)
(t:: = (x(,
(14)
(tt; 0) =
Theorem 1. In the limit of data (n + oc) and MC samples (m - 3c), for observed (X T t) E Di, we assume that p(y t. t,x,C 0) converges in measure to p(y t.,x,, fa(x, t; 0) is consistent estimator of p((x,t), t), and p(t X1,x) bounded away from O uniformly for all ye E V. Then, 4(x, t; A, 0) 2 p(x, A) and Ti(x A, 2n pi(x A). Proof in Appendix E.
3.3 Solving for w
We are interested in a scalable algorithm to compute the intervals on the CAPO function, eqs. (12) and (13), and the APO (dose-response) function, eqs. (14) and (15). The need for scalability stems not only from dataset size. The intervals also need to be evaluated for arbitrarily many values the continuous treatment variable, i, and the sensitivity parameter A. The bounds on the CAPO function can be calculated independently for each instance x, and the limits on the APO are an expectation over the CAPO function bounds.
5

---

The upper and lower bounds of the CAPO function under treatment, t, and sensitivity parameter, A, can be estimated for any observed covariate value, x, as
7x t; A, = (H y); 0)
[xx t; A, == F);
where y and y are found using Algorithm 1. See Algorithm 2 and Appendix for optional methods.
The upper and lower bounds for the APO (dose-response) function under treatment T I and sensitivity parameter A can estimated over any set observed covariates Dx = {x: as
((t; A.
Σ n
Mxx, t; A,
( A. 0)
(xx, A,
E Dx
3.4 Uncertainty about the Continuous Treatment Effect Interval
Following [ZSB19]. [DG21]. and [CCN+21], we construct (1 -x) statistical confidence intervals for the upper and lower bounds under the CMSM using the percentile bootstrap estimator. [JMSG20] and [JMGS21] have shown that statistical uncertainty is appropriately high for regions with poor overlap. Let Pp be the empirical distribution of the observed data sample, = {x, bt. N. = {S.)
k== Let PD = (Dx Ji be the bootstrap distribution over n,, datasets, D. = {S sampled with re- placement from the empirical distribution, Pp. Let Qa be the a-quantile of ((w(y);; x; A, 0) "²)
in the bootstrap resampling distribution: Qa inf,-- {
(PD(x(u());x,,
Finally, let 0% be the parameters of the model of the k-th bootstrap sample of the data.
Then, the bootstrap confidence interval of the upper and lower bounds of the CAPO function under the CMSM is given by: CI. (,,
a),
where,
(x, = Qa/2 (10(x,t; (i) (x, = Q1-a/2 1 ( 0n)
Furthermore, the bootstrap the APO CL (u(t:
confidence (dose-response)
interval of the function under H ())
upper and the CMSM
lower bounds of are given by:
where,
T, Q1-0/2 (fll:A..0))
2. (1; Qaj2 ((0(1),0=))
3.5 Scalable Continuous Treatment Effect Estimation
Following [SJS17], [S),+20), and [NYLN21], we propose using neural-network architectures with two basic components: feature extractor, Φ(x; 0) (Ø. for short) and conditional outcome prediction block fΦ6, t; 0). The feature extractor design will be problem and data specific. In Section 5, we look using both simple feed-forward neural network, and also a transformer [VSP+ 17]. For the conditional outcome block, we depart from more complex structures ([SLB* + 20, NYLN21)) and simply focus on residual [HZRS16], feed-forward, S-learner [KSBY19] structure. For the final piece of the puzzle, we follow [JMGS21] and propose ny component Gaussian mixture density:
nl
ply t, x2 = (6 (8, (), (,
(0)
and fi(x, t; = EØ (0 t: 7T (Ø, t: [Bis94]. Models are optimized by maximizing the log-likelihood of p(y x1 0)
4 Related Works
Scalable Continuous Treatment Effect Estimation. Using neural networks to provide scalable solutions for estimating the effects of continuous-valued interventions has received significant

---

attention in recent years. [BJvdS20] provide Generative Adversarial Network (GAN) approach. The dose-response network (DRNet) [SLB 20] provides a more direct adaptation of the TarNet [SJS17] architecture for continuous treatments. The varying coefficient network VCNet [NYLN22]] generalizes the DRNet approach and provides formal result for incorporating the target regularization technique presented by [SBV19]. The Riesz.et [CCN-2) 21] provides an alternative approach for targeted regularization. Adaptation of each method is straightforward for use in our sensitivity analysis framework by replacing the outcome prediction head of the model with suitable density estimator.
Sensitivity and Uncertainty Analyses for Continuous Treatment Effects. The prior litera- ture for continuous-valuce treatments has focused largely on parametric methods assuming linear treatmentouuccome, hidden-con caatoooooooooondrrrrtttttee
and mudden-confounder/wwwe [CHH16, DHCH16, MSDH16, 1st19, CH20a, CH20b]. addition to linearity, these parametric methods need to assume the structure and distribution of the unobserved confounding variable(s). [CKC+ 19] assumption. The MSM relaxes both the distributional and linearity assumptions, as does our CMSM extension. A two-parameter sensitivity model based on Riesz-Frechet representations of the target functionals, here the APO and CAPO, is proposed by [CCN+2] 21] way to incorporate confidence intervals and sensitivity bounds. In contrast, we use the theoretical background the marginal sensitivity model to derive one-paraeeter sensitivity model. [DBSC21] purport quantify the bias induced by unobserved confounding in the effects of continuous valued interventions, but they do not present formal sensitivity analysis. Simultancously and independently of this work, [MVSG] are deriving a sensitivity model that bounds the partial derivative of the log density ratio between complete and nominal propensity densities. Bounding the effects of continuous valued interventions has also been explored using instrumental variable models [KKS20, HWZW21, PZW+
relationships
allows for sensitivity analysis for arbitrary structural causal models under the linearity
Experiments
Here we empirically validate our method. First, we consider a synthetic structural causal model (SCM) demonstrate the validity of our method. Next, we show the scalability of our methods by applying them to a real-world climate-science-inspiii datasets (appendix G), and code are provided https: //gi httg//ttttuu/..////g com coom/aatml//mmmmmmmmmmmm/// /veeraast.
problem. Implementation details (appendix H),
(a) Observed Outcome (b) Observed Treatment
(c) CAPO functions
(d) APO function
1847740
1394:
(0) A=1.0
A=111
(g) A=1.2
(h) A=1.6
Figure Figures la Id: Synthetic data and ground truth functions. Figures 1e to 1h Causal uncertainty under hypothesized A values. Solid lines are ground truth; thick solid lines where the true X is within the range of hypothesized A, thin solid lines otherwise. The dotted lines are the estimated CAPO. Shaded regions are estimated CMSM intervals. 5.1 Synthetic
Figure presents the synthetic dataset (additional details about the SCM are given in Appendix G.1). Figure la plots the observed outcomes, y, against the observed confounding covariate, Each
7

---

datapoint is colored by the magnitude of the observed treatment, t. The binary unobserved confounder, u. induces bi-modal distribution in the outcome variable, y. at each measured value. x. Figure 1b plots the assigned treatment, against the observed confounding covariate, x. We can see that the coverage of observed treatments, i, varies for each value of x. For example, there is uniform coverage at X = 1, but low coverage for high treatment values at X = 0.1, and low coverage for low treatment values at X = 2.0. Figure Ic plots the true CAPO function over the domain of observed confounding variable, X, for several values of treatment (T 0.0, T = 0.5, and T = 1.0). For lower magnitude treatments, t, the CAPO function becomes more linear, and for higher values, we see more effect heterogeneity and attenuation of the effect size as seen from the slope of the CAPO curve for T = 0.5 and T 1.0. Figure 1d plots the the APO function over the domain of the treatment variable T.
Causal Uncertainty We want to show that in the limit of large samples (we set n. to 1004).. the bounds on the CAPO and APO functions under the CMSM include the ground truth when the CMSM is correctly specified. That is, when 1/ 1/A S X (t, x, u) S A, for user specified parameter A, the estimated intervals should cover the true CAPO or APO. This is somewhat challenging to demonstrate as the true density ratio X (t, X, u) (eq. (50)), varies with t, x, and u. Figures le to Ih work towards communicating this. In Figure le, we see that each predicted CAPO function (dashed lines) is biased away from the true CAPO functions (solid lines). We use thick solid lines to indicate cases where 1/A S X (t,, x, u) & A, and thin solid lines otherwise. Therefore thick solid lines indicate areas where expect the causal intervals cover the true functions. Under the erroneous assumption of ignorability (A = 1). the CMSM bounds have no width. In Figure 1f. we see that as we relax ignorability assumption (A 1.1) the intervals (shaded regions) start to grow. Note the thicker orange line: this indicates that for observed data described by X > 0.5 and T = 0.5, the actual density ratio is in the bounds of the CMSM with parameter A 0.5. We see that our predicted bounds cover the actual CAPO function for these values. We see our bounds grow again in Figure Ig when we increase A to 1.2. We see that more data points have X values that lie in the CMSM range and that our bounds cover the actual CAPO function for these values. In Figure Ih we again increase the parameter of the CMSM. We see that the bounds grow again, and cover the true CAPO functions for all of the data that satisfy 1/A S X (t, u) < A.
Statistical Uncertainty Now re- lax the infinite data assumption and set n 1000. This decrease in data will increase the estimator error for the CAPO and APO functions. So the estimated functions will not only be biased due hidden confound- ing, but they may also be erroneous due to finite sample variance. We show this in Figure 2b where the blue dashed line deviates from the ac- tual blue solid line as x increases be- yond 1.0. However, Figure 2b shows that under the correct CMSM, the un- certainty aware confidence intervals, section 3.4, cover the actual CAPO functions for the range of treatments considered. Figure 2a demonstrates that this holds for the APO function as well.
An A=16,
20ad
RKT EAI
41CT-1.0
Jlc. R.cT=100
(g403 T=03))
opulo.T T.500
(a) APO Function
(b) CAPO Functions
Figure 2: Statistical and causal uncertainty, is statistical significance level for the bootstrap. see Figure I for other details.
5.2 Estimating Aerosol-Cloud -almmaeC Effects from Satellite Data
Background The development of the model above, and the inclusion of treatment as a continuous variable with multiple, unknown confounders. is motivated by real-life use case for a prime topic in climate science. Aerosol-cloud interactions (ACI) occur when anthropogenic emissions in the form of aerosol enter cloud and act as cloud condensation nuclei (CCN). An increase in the number of CCN results in a shift in the cloud droplets to smaller sizes which increases the brightness of a cloud and delays precipitation. increasing the cloud's 's lifetime, extent, and possibly thickness [Two77, Alb89, TCGB 17.. However, the magnitude and sign of these effects are heavily dependent on the environmental conditions surrounding the cloud [DI.20]. Clouds remain the largest source of uncertainty our future climate projections [MDZP+21]: it pivotal to understand how human emissions may be altering their ability to cool. Our current climate models fail to accurately emulate
8

---

SST
Sea Surface Temperature Us
Surface winds
S
Supersaturation
Estimated inversion Strength Vertical motion at 500 mb
Relative sicity mm
Acrosal
w530
T
Temperalure profle
Acrosol proxy
Pressure profite
Cloud opticall depth
Unobserved Confeunding
Aerosol
Proxy
Meteorologica Proxies
Aerosol
Optical
Depth
Surrounding Meteorology
Metsorological Proxies
Cloud Optical Thickness
(b)
Figure 3: Causal diagrams. Figure 3a, a simplified causal diagram representing what we are reporting within; aerosol optical depth (AOD, regarded as the treatment T) modulates cloud optical depth (T, Y), which itself is affected by hidden confounders (U) and the meteorological proxies (X). Figure 3b, an expanded causal diagram of ACI. The aerosol (a) and aerosol proxy (AOD). the true confounders (light blue), their proxies (dark blue), and the cloud optical depth (red).
ACI, leading to uncertainty bounds that could offset global warming completely or double the effects of rising CO2 [BRA+13) 13].
Defining the Causal Relationships Clouds are integral to multiple components of the climate system, they produce precipitation, reflect incoming sunlight, and can trap outgoing heat [SF09]. Unfortunately, their interconnectedness often Ieads hidden sources of confounding when trying to address how anthropogenic emissions alter cloud properties.
Ideally, would like understand the effect of acrosols (a) on the cloud optical thickness, denoted However, this is currently impossible. Aerosols come in varying concentrations, chemical compositions, and sizes [SGW+ + 16] and we cannot measure these variables directly. Therefore, we use aerosol optical depth (AOD) as a continuous, 1-dimensional proxy for aerosols. Figure 3b accounts the known fact that AOD an imperfect proxy impacted by its surrounding meteorological environment [CNP+ 17]. The meteorological environment is also a confounder that impacts cloud thickness and aerosol concentration a. Additionally, we depend on simulations of the current environment in the form of reanalysis to serve as its proxy.
continuous treatment and the environmental variables as covariates. However, acrosol is the actual treatment, and AOD is only a confounded, imperfect proxy (Figure 3a). This model cannot accurately capture all causal effects and uncertainty due known and unknown confounding variables. We use this simplified model as a rest-bed for the methods developed within this paper and as a demonstration that they can scale the underlying problem. Future work will tackle the more challenging and realistic causal model shown in Figure 3b, noting that the treatment of interest a multi-dimensional and cannot measured directly.
Here we report AOD
Model We use daily observed 1 x means of clouds, aerosol, and the environment from sources shown in Table of Appendix G. To model the spatial correlations between the covariates on feature extractor.
given day, we use multi-headed attention [VSP+]] 17] to define a transformer-base Modeling the spatial dependencies between meteorological variables motivated by confounding that may be latent in the relationships between neighboring variables. These dependencies are unobserved from the perspective of single location. This architectural change respects both the assumed causal graph (fig. 3a) and some of the underlying physical causal structure. We see in Figure (Left) that modeling context with the transformer architecture significantly increases the predictive accuracy of the model when compared to simple feed-foree forward neural network (no context). Discussion & Results The results for the APO of cloud optical depth (r) as the "treatment". AOD, increases are shown Figure 4, As the assumed strength of confounding increases (A 1), the range of uncertainty

---

No contect
18
Rf 2 0.022
Contest
A=10
0330
A=11
16
A=12
1=
predicted
14
12
10
836
016
025
8.3
035
20
00
200 120
observed
AOD
Figure 4: Left: The values of the observed, true against the modeled Right: The curve for continuous treatment outcome of our aerosol proxy (AOD) on cloud optical depth (T). The darkest shaded region (A 1) represents the uncertainty in the treatment outcome from the ensemble due to finite data. As the strength of confounders increases (A 1.0), the range of uncertainty in the treatment outcome increases.
in the treatment outcome increases. Within this range of confounding, the modeled outcomes agree with two conflicting hypotheses. First, that aerosol acts invigorate the cloud, inducing a large response that would follow maximum curve within this uncertainty range [CS11, DL21J. And second, that aerosol has little impact on cloud depth, and the actual response is minimal, flat line [GGS+ 19]. We further find the reported dose-response curves in agreement with multiple estimates of acrosol-cloud interactions using satellite observations [BTG02, MSJ 07, TCQB 19]. The upper bound for log A = .2 agrees with measurements of the in-cloud environment and acrosol-cloud interactions from mounted sensors [PZ13], this may indicate the need for additional control variables when using satellite data.
The resolution of the satellite observations (1 daily means) could be averaging various cloud types and obscuring the signal. Future work will investigate how higher resolution (20km x 20km) data with constraints on cloud type may resolve some confounding influences. However, even our more detailed causal model (Figure 3b) cannot account for all confounders; we expected, and have seen, imperfections in our model this complex effect. The model's results require further expert validation to interpret the outcomes and uncertainty.
Societal Impact Geoengineering of clouds by acrosol seeding could offset some amount of warming due to climate change, but also have disastrous global impacts on weather patterns [DGL+ 22). Given the uncertainties involved understanding aerosol-cloud interactions, paramount that policy makers are presented with projected outcomes if proposals assumptions are wrong or relaxed.
Acknowledgments and Disclosure of Funding
We would like to thank Angela Zhou for introducing to the works of [ZSB19) 19] and [DG21]. These works use percentile bootstrap for finite sample uncertainty estimation within their sensitivity analysis methods. We would also like to thank Lewis Smith for helping understand the Marginal Sensitivity Model of [TTn06] in detail. Finally, we would like to thank Clare Lyle and all anonymous reviewers for their valuable feedback.
This research was supported by the European Research Council (ERC) project constRaining the EffeCts of Aerosols on Precipitation (RECAP) under the European Union's Horizon 2020 research and innovation program with grant agreement no. 724602 and from the European Union's 's Horizon 2020 research and innovation program project Constrained aerosol forcing for improved climate projections (FORCeS) under grant agreement No $21205. and Marie Skłodowska-Cunn agreement No 860100 (iMIR This work used JASMIN, the UK's collaborative data analysis environment (http:/jasmin.ac.uk))) U.S. was partially supported by the Israel Science Foundation (grant No. 1950/19).
grant
10

---

References
[A]b89] Bruce A Albrecht, Acrosols, cloud microphysics, and fractional cloudiness. Science, 245(4923).1227--2300 1989.
[BAC+15] Michael G Bosilovich, Santha Akella, Lawrence Coy, Richard Cullather, Clara Draper, Ronald Gelaro, Robin Kovach, Qing Liu, Andrea Molod, Peter Norris, et al. Merra-2: Initial evaluation of the climate. 2015.
[Bis94] Christopher M. Bishop. Mixture density networks. 1994.
[BJvdS20] Ioana Bica, James Jordon, and Mihaela van der Schaar. Estimating the effects of continuous- valued interventions using generative adversarial networks. arXiv preprint 2.Xv:22002.123226. 2020.
[BP06] Bryan Baum and Steven Platnick. Introduction to modis cloud products. In Earth science satellite remote sensing, pages 74-91. Springer, 2006.
13] Olivier Boucher, David Randall, Paulo Artaxo, Christopher Bretherton, Gragam Fein- gold, Piers Forster, V-M Kerminen, Yutaka Kondo, Hong Liao, Ulrike Lohmann, et al. Clouds and aerosols. In Climate change 2013: the physical science basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergoverumental Panel on Climate Change, pages 571-657. Cambridge University Press, 2013.
[BTG02]
Francois-Marie Bréon, Didier Tanré, and Sylvia Generoso. Aerosol effect on cloud droplet size monitored from satellite. Science, 295(5556):834-835) 2002.
[CCN*21] Victor Cheroozhukov. Carlos Cinelli, Whitney Newey, Amit Sharma, and Vasilis Syrgkanis. Omitted variable bias in machine learned causal models. arXiv preprint arXiv:21 12. 13398, 2021.
[CH20a] Carlos Cinelli and Chad Hazlett. Making sense of sensitivity: Extending omitted vari- able bias. Journal of the Royal Statistical Society: Series (Statistical Methodology), 82(1):39-66 67, 2020.
[CH20b] Carlos Cinelli and Chad Hazlett. An omitted variable bias framework for sensitivity analysis of instrumental variables. Work. Pap, 2020.
[CHH16] Nicole Bohme Carnegie, Masataka Harada, and Jennifer L Hill. Assessing sensitivity to unmeasured confounding using a simulated potential confounder. Journal of Research on Educational Effectiveness, 93):3955-420, 2016.
[CKC+ 19] Carlos Cinelli, Daniel Kumor, Bryant Chen, Judea Pearl, and Elias Bareinboim. Sen- sitivity analysis of linear structural causal models. In International conference on machine learning, pages 222--1261. PMLR, 2019.
[CNP+17] Matthew W Christensen, David Neubauer, Caroline A Poulsen, Gareth E Thomas, Gregory McGarragh, Adam Povey, Simon R Proud, and Roy G Grainger. Un- veiling aerosol-cloud interactions-part 1: Cloud contamination in satellite products enhances the acrosol indirect forcing estimate. Atmospheric Chemistry and Physics, 17(21):3151-13164 2017.
[CS11] Matthew W Christensen and Graeme I. Stephens. Microphysical and macrophysical responses marine stratocumulus polluted by underlying ships: Evidence of cloud deepening. Journal of Geophysical Research: Atmospheres, 116(D3), 2011.
[D'A19] Alexander D' Amour. On multi-cause approaches causal inference with unobserved counfounding: Two cautionary failure cases and a promising alternative. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3478-3488 PMLR, 2019.
[DBSC21] Gianluca Detommaso, Michael Brückner, Philip Schulz, and Victor Chemoozuuovv Causal bias quantification for continuous treatment, arXiv preprint 106.0962.. 2021.

---

[DG21] Jacob Dor and Kevin Guo. Sharp sensitivity analysis for inverse propensity weighting via quantile balancing. arXiv preprint arXiv:2102.04543, 2021.
[DGL*22] Michael S Diamond, Andrew Gettelman, Matthew D Lebsock, Allison McComiskey, Lynn M Russell, Robert Wood, and Graham Feingold. Opinion: To assess marine cloud brightening's technical feasibility, we need to know what to study-am and when to stop. Proceedings of the National Academy of Sciences, 11)(4), 2022.
[DHCH16] Vincent Dorie, Masataka Harada, Nicole Bohme Carnegie, and Jennifer Hill. A flexible, interpretable framework for assessing sensitivity to unmeasured confounding, Statistics in medicine, 35(20):3453-3470 2016.
[DI.20] Alyson Douglas and Tristan E'Euuy.r. Quantifying cloud adjustments and the radiative forcing due to acrosol-cloud interactions in satellite observations of warm marine clouds. Ammssheric Chemistry and Physics, (0(10):6225-624) 2020.
[DL21] Alyson Douglas and Tristan I. L'Ecuyer. Global evidence of aerosol-induced invigoration in marine cumulus clouds. Amospheric Chemistry and Physics, 11(19):5510--5111, 2021.
[FKH17] Stefan Falkner, Aaron Klein, and Frank Hutter. Combining hyperband and bayesian optimization. In NIPS 2017 Bayesian Optimization Workshop (Dec 2017), 2017.
[GGS+19] Edward Gryspeerdt, Tom Goren, Odran Sourdeval, Johannes Quaas, Johannes Mülmen- städt, Sudhakar Dipu, Claudia Unglaub, Andrew Gettelman, and Matthew Christensen. Constraining the aerosol influence cloud liquid water path. Atmospheric Chemistry and Physics, 19(8):5331-5344 2019.
[GMS+17] Ronald Gelaro, Will McCarty, Max J Suárez, Ricardo Todling, Andrea Molod, Lawrence Takacs, Cynthia A Randles, Anton Darmenov, Michael G Bosilovich, Rolf Reichle, al. The modern-era retrospective analysis for research and applications, version (merra-2). Journal of climate, 30(14):5419 5454, 2017.
[H104] Keisuke Hirano and Guido W Imbens. The propensity score with continuous treatments. Applied Bayesian modeling and causal inference from incomplete-de perspectives, 226164:73-88 84, 2004.
[Ho186] Paul W Holland. Statistics and causal inference. Journal of the American statistical Association, 81(36))995- 960, 1986.
[WZW22]] Yaowei Hu, Yongkai Wu, Lu Zhang, and Xintao Wu. A generative adversarial frame- work for bounding confounded causal effects. In Proceedings of the AAAI Conference on Artificial Intelligence volume 35, pages 12104-121 12, 2021.
[HZRS16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016
[JMGS21] Andrew Jesson, Sören Mindermann, Yarin Gal, and Uri Shalit. Quantifying ignorance in individual-level causal-effect estimates under hidden confounding. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4829- 4838. PMLR, 18-24 Jul 2021.
[JMSG20] Andrew Jesson, Sören Mindermann, Uri Shalit, and Yarin Gal. Identifying causal- effect inference failure with uncertainty-awarr models. Advances in Neural Information Processing Systems, 33: 1111-1111699 1149,, 2020.
[KKS20] Niki Kilbertus, Matt J Kusner, and Ricardo Silva. A class of algorithms for general instrumental variable models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20108-2011 Curran Associates, Inc., 2020.
12

---

[KMZ19] Nathan Kallus, Xiaojie Mao, and Angela Zhou. Interval estimation of individual-level causal effects under unobserved confounding. In The 22nd international conference on artificial intelligence and statistics, pages 2281-2290. PMLR, 2019.
[KSBY19] Sören R Künzel, Jasjeet S Sckhon, Peter Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, (16(10)4416-4166) 2019.
[LLN+18 18] Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv: 000700 2018.
[MDZP*21] V. Masson-Delmotte P. Zhai, A. Pirani, S. L. Connors, C. Péan, S. Berger, N. Caud, Chen, L. Goldfarb, M. I. Gomis, M. Huang, K. Leitzell, E. Lonnoy, J. B. R. Matthews, T. K. Maycock, T. Waterfield, O. Yelekçi, R. Yu, and B. Zhou. Ipce, 2021: Climate change 2021: The physical science basis. contribution working group to the sixth assessment report of the intergovernmental panel on climate change. 2021.
[MNW* 18] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. Ray: A distributed framework for emerging ai applications, 2018.
[MSDH16] Joel Middleton, Marc A Scott, Ronli Diakow, and Jennifer L Hill. Bias amplification and bias unmasking. Political Analysis, 24(3):307-333, 2016.
[MSJ+ 07] Gunnar Myhre, Frode Stordal, M Johnsrud, YJ Kaufman, D Rosenfeld, Trude Storelymo, Jon Egill Kristjansson, Terje Koren Berntsen, A Myhre, and Ivar SA Isaksen. Acrosol-cloud interaction inferred from modis satellite data and global aerosol models. Atmospheric Chemistry and Physics, 7(12):3081-3111 2007.
[MVSG] Myrl G Marmarelis, Greg Ver Steeg, and Aram Galstyan. Bounding the effects of continuous treatments for hidden confounders.
[Ney23] Jerzy Neyman. edited and translated by dorota m. dabrowska and terrence p. speed (1990). on the application of probability theory to agricultural experiments. essay on principles. section 9. Statistical Science, 5(4):465-47,, 1923.
[NYLN21] Lizhen Nie, Mao Ye, Qiang Liu, and Dan Nicolae. Venet and functional targeted regularization for learning causal effeets of continuous treatments. arXiv preprint axXiv:2103.07861, 2021.
[Ost19] Emily Oster. Unobservable selection and coefficient stability: Theory and evidence. Journal of Business & Economic Statistics, 3772)1187-204, 2019.
19GM+19] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmai- son, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[PVG+11] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-288 28330, 2011.
[PZ13] D Painemal and P Zuidema. The first aerosol indirect effect quantified through airborne remote sensing during vocals-rex. Atmospheric Chemistry and Physics, 13(2):917-99) 2013.
13

---

[PZW 22] Kirtan Padh, Jakob Zeitler, David Watson, Matt Kusner, Ricardo Silva, and Niki Kilbertus. Stochastic causal programming for bounding treatment effects. arXiv preprint 200.:22222.0800, 2022.
[RHW86] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representa- tions by back-propagating errors. nature, 323(6088):533-5366 1986.
[RR83] Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55, 1983.
[Rub74] Donald B Rubin. Estimating causal effects of treatments in randomized and nonran- domized studies. Journal of educational Psychology, 66(5):688, 1974.
[Rub80] Donald B Rubin. Randomization analysis of experimental data: The fisher randomiza- tion test comment. Journal of the American Statistical Association, 75(371):591-555) 1980.
[Rud16] Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv: 047477 2016.
[SBV19] Claudia Shi, David M Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 2507-2517, 2019.
[Sek08] Jasjeet S Sekhon. The neyman-rubin model of causal inference and estimation via matching methods. The Oxford handbook of political methodology, 2:1-32, 2008.
[SF09] Bjorn Stevens and Graham Feingold. Untangling aerosol effects on clouds and precipi- tation in a buffered system. Nature, 461(7264):607-613, 2009.
[SGW+16] Nick AJ Schutgens, Edward Gryspeerdt, Natalie Weigum, Svetlana Tsyro, Daisuke Goto, Michael Schulz, and Philip Stier. Will perfect model agree with perfect observations? the impact of spatial sampling. Atmospheric Chemistry and Physics, 16(10):6335-6353 2016.
[SJS17] Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. International Conference on Machine Learning, pages 3076-3085. PMLR, 2017.
[SLB+20] Patrick Schwab, Lorenz I Linhardt, Stefan Bauer, Joachim M Buhmann, and Walter Karlen. Learning counterfactual representations for estimating individual dose-response curves. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5612-5619, 2020.
[Tan06] Zhiqiang Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, (6(44))):619-1633 2006.
[TCGB17] Velle Toll, Matthew Christensen, Santiago Gassó, and Nicolas Bellouin. Volcano and ship tracks indicate excessive aerosol-induced cloud water increases in a climate model. Geophysical research letters, 44(24):12-492, 2017.
[TCQB19] Velle Toll, Matthew Christensen, Johannes Quaas, and Nicolas Bellouin. Weak average sqquid-loud-watte response to anthropogenic aerosols. Nature, 572(7767):51-55, 2019.
[Two77] Sean Twomey. The influence of pollution on the shortwave albedo of clouds. Journal of the ulmospheric sciences, 34(7):1 149-1152, 1977.
17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[ZSB19] Qingyuan Zhao, Dylan Small, and Bhaswar B Bhattacharya. Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 81(4):735-761, 2019.
14

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
i. we claim to introduce a novel marginal sensitivity model for continuous valued treatment effect (the CMSM). See Section
ii. we claim derive bounds for the CAPO and APO functions that agree with the CMSMS and observed data. See Section 3.1.
iil. we claim to provide tractable estimators of the CAPO and APO bounds. See Sections 3.2 and 3.3.
iv. we claim to provide bounds that account for finite-sample (statistical) uncertainty. See Section 3.4.
we claim to provide a novel architecture for scalable estimation of the effects of continuous valued interventions. See Section 3.5.
vi. we claim that the bounds cover the true ignorance interval for well specified A. See Figures to Ih and Theorem 1.
vii. we claim that this model scales to real-world, large-sample, high-dimensional data.
See Section 5.2
(b) Did you describe the limitations of your work? [Yes]
We have discussed the major limitation of sensitivity analysis methods, that unob- served confounding is not identifiable from data alone. We have tried to be honest and transparent that our method provides users with a way to communicate uncertainty induced when relaxing the ignorability assumption. We do not claim that lambda is in any way identifiable without further assumptions.
ii Section 5.2, we have clearly discussed the limitations of analyses of aerosol-cloud interactions using satellite data where only see underlying causal mechanisms through proxy variables. We hope this paper serves as a stepping stone for work that specifically addresses those issues.
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We have five theoretical results. Proposition 1, Equation (7), Equation (8), Equation (9), and Theorem 1. All assumptions are stated for each.
(b) Did you include complete proofs of all theoretical results? [Yes] The proof of Propo- sition is given in Appendix A.3. The proof of Equation (7) is given in Lemma 1. The proofs for Equation (8) and Equation (9) are given Lemma 2. The proof for Theorem is given in Appendix E. 3. If you experiments...
15

---

(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or URL)? [Yes] Code, data, and instructions are provided in the suppleemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We specify these details in Appendix and Appendix as well in the provided code.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] Both random seeds and random bootstrapped sampling of the training data.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] this is outlined in Appendix H
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use existing
satellite data and open source code libraries that we have cited.
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] provide new synthetic dataset and code base
(d) Did you discuss whether and how consent was obtained from people whose data you're you're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "ufB7-qd57G", "reviewer": "Reviewer_zMDP", "summary": "The paper derives bounds for causal treatment effect estimates when the intervention is continuous and there exists a user-specified amount of unobserved confounding. The main innovation is to extend previous bounds which were derived for discrete treatments to the case of continuous treatments. Non-parametric models are assumed for the treatment conditional on the covariates, and for the outcome conditional on the treatment and covariates.", "questions": "Given the multiple somewhat overlapping works currently under review, I would suggest expanding the \"Sensitivity and Uncertainty Analyses for Continuous Treatment Effects\" section in \"Related Works\" to include more details about the differences of the methods.\n\nAlso, I'm wondering if there aren't additional papers published in different fields (other than machine learning), where similar results could have been discussed earlier. On a quick googling, I found at least the following, which, based on title and abstract, might have a similar goal: \"Bias formulas for sensitivity analysis of unmeasured confounding for general outcomes, treatments, and confounders\" by VanderWeele and Arah, Epidemiology, 2011. Could you clarify the difference of the present work to this, and also check if you can find other similar papers published in, e.g., economics, epidemiology, or (bio)statistics?\n\nProposition 1 has an assumption that P(Y_t|T=t,X=x) is equivalent to a Lebesgue measure. I read this such that the distribution is uniform, which seems very restrictive. Could you clarify this, please?\n\nI don't follow eq (7). It's probably similar to KMZ19, but it would be good to make the present article self-standing. For example, what is the w(.) function and what is its role here intuitively? It's defined after eq (9) for one argument, but it seems to be used in the text interchangable with one or two arguments.\n\nSimilarly to the previous comment, I'm not sure what the y with the bar below/above in Section 3.3 is.\n\n\nVery minor:\nl. 210: D has subscript j while indexing uses k.\n\nl. 230: By mazimizing the log-likelihood, not \"minimizing\".\n", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nThe article has an important and timely topic. The derivations for the bounds seem rigorous, and are validated empirically in the experiments. The analytical results are derived for the asymptotic case, for finite case the article proposes bootstrapping to get a bounds for the bounds. Related works are covered well (though see questions below). The presentation is clear. The real-world application in climate change is important and well-described.\n\nWeaknesses: \n\nThis appears a crowded topic, as there seem to be at least two other recently arxiv'ed papers on the same topic (Chernozhukov et al., 2021; Marmarelis et al., 2022), which may differ in some details (I haven't read those papers closely). The idea for the topic seems to have been discussed in previous year's \"Causal Inference & Machine Learning: Why now?\" Neurips Workshop, which potentially has spawned the interest (my speculation). Anyway, this seems to slightly decrease the novelty of the present article. I am not sure if any of these related papers is more deserving to be published first, but at least the present article I found a rather complete and good package and tentatively can support its acceptance, unless other reviewers identify major issues. I don't find any other major problems.\n\nThere were some small problems with presentation, especially not all notation used in the formulas is defined/explained clearly (see below). The figures in the Experiments section are not sharp when printed out. Otherwise the presentation is clear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nV9D_dUnfY", "reviewer": "Reviewer_eRS3", "summary": "The paper tackles a very interesting an important question, that of bounding treatment effects when ignorability is violated and the treatment and covariates are continuous. They do this by proposing an extension of the marginal sensitivity model (MSM) to the so called continuous marginal sensitivity model (CMSM). In the CMSM, a density ratio is used to quantify the belief of how much ignorability is violated, in contrast to the odds ratio used in MSM.\n\nFollowing this the authors five a semi-parametric estimator for the bounds and an algorithm to compute them. The claims are then validated though synthetic and real world experiments.", "questions": "* Are there any baselines that the authors could have compare their methods to? For instance even something line shoehorning a continuous MSM into the binary setting (for a single dimensional continuous X of course)?\n* While the amount of compute used is reasonably small, I couldn't find any information on how long it took to get the bounds? Do you some estimate of this?\n", "limitations": "The  potential negative societal impact is discussed but the limitations could be described better.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**. \n* The paper is well written and easy to follow in my opinion. The flow is natural and it is clear how each part fits into the bigger picture while reading the paper.\n* The paper evaluates their method on a real world datasets in collaboration with the domain experts. The ideal application of causal inference is always said to be in alliance with domain experts, and using that as one of the ways to evaluate a proposed method speaks in favour of the method and the paper.\n* The experiments seem to justify the claims made in the paper.\n* The appendix provides a thorough description of the background on MSM and how the proposed method relates to it.\n\n**Weaknesses**. \n* The limitations of the method should be discussed to give a clearer picture of the method. One limitation is could be the need for a grid search.\n\n**Other related work**. Considering that the paper deals with bounding causal effects, the following papers could be mentioned in the background work. While they are not directly doing sensitivity analysis per se, I think they are relevant to getting a context of the related work since they also tackle the problem of bounding treatment effects in the presence of confounding.\n\n*Continuous setting*: \nKilbertus, N., Kusner, M.J. and Silva, R., 2020. A class of algorithms for general instrumental variable models. Advances in Neural Information Processing Systems, 33, pp.20108-20119.\n\nHu, Y., Wu, Y., Zhang, L. and Wu, X., 2021, May. A generative adversarial framework for bounding confounded causal effects. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 13, pp. 12104-12112).\n\nPadh, K., Zeitler, J., Watson, D., Kusner, M., Silva, R. and Kilbertus, N., 2022. Stochastic Causal Programming for Bounding Treatment Effects. arXiv preprint arXiv:2202.10806.\n\n*Discrete setting*: \nDuarte, Guilherme, Noam Finkelstein, Dean Knox, Jonathan Mummolo, and Ilya Shpitser. 2021a. “An Automated Approach to Causal Inference in Discrete Settings.” arXiv [stat.ME]. arXiv. http://arxiv.org/abs/2109.13471.\n\nDuarte, G., Finkelstein, N., Knox, D., Mummolo, J. and Shpitser, I., 2021. An automated approach to causal inference in discrete settings. arXiv preprint arXiv:2109.13471.\n\nZhang, Junzhe, Jin Tian, and Elias Bareinboim. 2021. “Partial Counterfactual Identification from Observational and Experimental Data.” arXiv [cs.AI]. arXiv. https://www.causalai.net/r78.pdf.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SM6ijFqeT0r", "reviewer": "Reviewer_UYQB", "summary": "In this paper, the authors develop a sensitivity analysis method for estimating causal effects with continuous-valued interventions. This is important because when performing causal inference from observational data, common assumptions like no unobserved confounding rarely hold. Building on a sensitivity analysis formulation for binary treatments, the authors develop a continuous marginal sensitivity model (CMSM) that accounts for possible unobserved confounding by bounding the density ratio between the observational and \"complete\" treatment propensity distribution. This sensitivity model allows the authors to compute upper and lower bounds on the conditional average potential outcome (which when marginalized over covariates yields the average potential outcome). Thus, for a given value of the sensitivity parameter, and for any value of the treatment and covariates, the method produces an interval for the conditional average potential outcome that can be subsequently used to estimate the range of causal effects compatible with the data. In synthetic experiments the authors validate various aspects of their approach. Finally, they apply their method to a simplified causal question in climate science, showing that a range of hypotheses are compatible with the data under varying degrees of possible unobserved confounding.", "questions": "* How does the proposed method address possible positivity violations? The authors allude to this in the introduction but (it seems to me) it is never explicitly spelled out. It seems that the uncertainty quantification via bootstrapping is meant to capture uncertainty in, e.g., regions with poor overlap? I think some explicit statement about this (say, in Section 3.4) would help tie up any loose ends regarding the authors' claims.\n\n* In the climate experiment, what guidance is there for how to pick relevant $\\Lambda$ values to consider? Could the authors add some kind of qualitative discussion/judgement here?", "limitations": "Yes, the authors adequately address limitations.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I really enjoyed this paper. I think the problem is important, the paper is well-written and well-executed, and the developed methodology seems both elegant and practical. As the authors note, I think sensitivity analysis is very important for bringing the value of causal inference from observational data to policy making in practical scenarios, and I think the current paper represents a useful method for doing so.\n\nI mainly have a few clarifying questions & minor suggestions:\n* The main possible difficulty with the proposed method that came to mind is how to select values of the sensitivity parameter. As opposed to the odds ratio (which, as the authors note, is generally interpretable to practitioners), the density ratio is somewhat difficult to make judgements about directly. The authors provide an alternative characterization in terms of the \"proportion of unexplained range in $Y$\", but even this is (to my knowledge) not a commonly considered statistic. Can this be related in any way to, e.g., the $R^2$ (i.,e., the fraction of *variance* unexplained)? Also, can one compute the proportion of unexplained range for a reference covariate(s) (e.g., patient age) and say something along the lines of \"unobserved confounding would need to account for at least as much of the range in $Y$ as patient age to reach a sensitivity value of $\\Lambda$\"? Would such comparisons be valuable for judging the plausibility of different $\\Lambda$ values?\n* Along these lines, I don't think the authors discussed what reasonable values of $\\Lambda$ might be in their cloud experiment. I think the exercise of reasoning about $\\Lambda$ in the context of this example would be very helpful for readers and potential users of the method.\n\n\n**Suggestions**\n* Within the main paper, the authors assume a basic familiarity with marginal sensitivity models as known method/object (Tan, 2006) (e.g., \"We propose CMSM as a new MSM\" Ln 77). While they provide more information in the supplementary material, perhaps there is some minimal definition of a \"marginal sensitivity model\" that they can put in the main paper to provide a baseline context that they can build upon.\n* To paint a fuller picture of the range of possible approaches to sensitivity analysis in the related work, the authors might consider adding a mention of works such as (Imbens (2003) and Veitch & Zaveri (2020)) which explicitly model the correlation due to unobserved confounding (for a binary treatment). Veitch & Zaveri gets around some criticism of existing methods that the authors discuss in Ln 246 by allowing for flexible, non-parametric models of the unobserved confounding relationship.\n\nImbens, G. W. (2003). Sensitivity to exogeneity assumptions in program evaluation. American Economic Review, 93(2), 126-132.\n\nVeitch, V., & Zaveri, A. (2020). Sense and sensitivity analysis: Simple post-hoc analysis of bias due to unobserved confounding. Advances in Neural Information Processing Systems, 33, 10999-11009.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "15QWNOFV9ea", "reviewer": "Reviewer_r7RZ", "summary": "Ignorability assumption is essential to ATE, however, this is often not satisfied in the real world. This means that there will be ignorance in the causal estimates when this assumption is relaxed. The paper addresses this issue by proposing a neural network based model that are scalability for ananlysing sensitivity and uncertainty when the ignorability assumption does not hold true for continuous data.", "questions": "- It's not clear to me why the exact formulation of the SCM of the synthetic data was chosen. Would be nice if the authors could elaborate on this.\n\n- Section 3 is a bit difficult to follow, would be nice to have more intutive explanations of the derivations.", "limitations": "The authors were transpararent about the limitations of their work, such as lambda may not be easily identified.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros,\n\n- The paper addresses an important issue of sensitivity analysis and uncertainty to measure the ignorance in causal estimate when the ignorability assumption is voileted. The paper also proposes a scalable neural network based approach, which can be applicable to a wilder set of data and problems.\n\n- The beignning of the paper is very well written and easy to follow. The settings, assumptions and derivations are clear and concise.\n\nCons,\n\n- The generation of the synthetic data seems to be in a particular form, it's not clear to me why this exact formulation was chosen. Could the authors explain a little more on this?\n\n- The latter sections (section 3) are slightly difficult to follow. It would be nice to have more intuitive explanation of the derivations throughout the section.\n\n- For the experiment, it's not exactly clear to me how the assumption of ignorability fits into the neural network architecture. \n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept