High Precision Causal Model Evaluation with Conditional Randomization
Authors: Chao Ma
Cheng Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: causality
causal inference
causal model evaluation
TL;DR: A simple yet novel method for estimating causal errors for causal model evaluation, under conditionally randomized experiments.

Abstract: The gold standard for causal model evaluation involves comparing model predictions with true effects estimated from randomized controlled trials (RCT). However, RCTs are not always feasible or ethical to perform. In contrast, conditionally randomized experiments based on inverse probability weighting (IPW) offer a more realistic approach but may suffer from high estimation variance. To tackle this challenge and enhance causal model evaluation in real-world conditional randomization settings, we introduce a novel low-variance estimator for causal error, dubbed as the pairs estimator. By applying the same IPW estimator to both the model and true experimental effects, our estimator effectively cancels out the variance due to IPW and achieves a smaller asymptotic variance. Empirical studies demonstrate the improved of our estimator, highlighting its potential on achieving near-RCT performance. Our method offers a simple yet powerful solution to evaluate causal inference models in conditional randomization settings without complicated modification of the IPW estimator itself, paving the way for more robust and reliable model assessments.

High Precision Causal Model Evaluation with Conditional Randomization
Chao Ma
Cheng Zhang Microsoft Research Cambridge, UK cheng.
Microsoft Research Cambridge, UK chao mammmaa@mie.. cww.of..oooo..oommmm com
Abstract
The gold standard for causal model evaluation involves comparing model predic- tions with effects estimated from randomized controlled trials (RCT). However, RCT's are not always feasible or ethical to perform. In contrast, conditionally ran- domized experiments based on inverse probability weighting (IPW) offer more realistic approach but may suffer from high estimation variance. To tackle this chal- lenge and enhance causal model evaluation in real-world conditional randomization settings, introduce novel low-variance estimator for causal error, dubbed the pairs estimator. By applying the same IPW estimator to both the model and true experimental effects, estimator effectively cancels out the variance due IPW and achieves a smaller asymptotic variance. Empirical studies demonstrate the improved of our estimator, highlighting its potential achieving near- RCT performance. Our method offers a simple yet powerful solution to evaluate causal inference models in conditional randomization settings without complicated modi- fication of the IPW estimator itself, paving the way for more robust and reliable model assessments.
Introduction
Experimental approaches for causal model evaluation Causal inference models aim to estimate the causal effects of treatments interventions on outcomes of interest, given observational (some- times experimental) data. A crucial step in developing and validating such models is to evaluate their prediction quality, that is, how well they can approximate the true treatment effects that would have been observed under different scenarios. A common approach for causal model evaluation is to launching new randomized controlled trial (RCT) separately, which provides reliable estimate of the true treatment effects by randomly assigning units to treatments. By comparing the RCT estimate with the prediction the causal model, one can assess the causal error (Equation 1), a key metric to measure how well the model reflects the true effects.
Nevertheless, RCTs are not always accessible preferable, random manipulation of treatments might be impractical, ethically questionable, or excessively expensive. For instance, it may be impossible or inappropriate randomly assign individuals different lifestyles, environmental exposures, or social policies, as these may involve personal choices, preferences, or constraints that affect the willingness or ability participate in the experiment. In such cases, causal machine learning researchers may resort to or augment their analyses with conditionally randomized designs, which the allocation of participants into treatment and control groups is not purely random, but instead based on predetermined condition or factor.
Conditionally randomized experiments vary in the level and unit of treatment assignment, such as individual, group, or setting. They also depend on different assumptions and methods to account for the potential confounding, selection, measurement biases that arise from the lack of randomization.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

A common scenario is the non non-random quantitative assignment paradigm [West et al., 2008], where a new set of treatment groups are selected given some quantitative covariates X, via an oracle model Pezp(( X), where 7 is the treatment variable. For example, in sales strategy assignment, is typical that resources are allocated towards products with higher revenue contributions, which introduces explicit confounding. A widely used method to estimate the true treatment effect in this setting is the inverse probability weighting (IPW) estimator [Rosenbaum and Rubin, 1983, Rosenbaum, 1987], which uses the propensity scores, the probability of receiving the treatment given the covariates, to weight the observed outcomes and adjust for the imbalance between the treatment groups. This IPW estimate can then be compared with the causal model prediction to evaluate the causal error.
Limitations of current approaches Despite the popularity of the IPW estimator for conditionally randomized experiments, it suffers from several limitations that affect its reliability for causal model evaluation. [Khan and Tamer, 2010] shows that IPW may have unbounded variance when the propensity scores are imbalanced. Moreover, the it may have poor finite sample performance and high sensitivity to the specification of the propensity score model [Busso et al., 2014]. Last but not the least, a surprising finding is that even the oracle IPW may be harmful for the estimation efficiency [Hahn, 1998, Hirano et al., 2003]. As a consequence, the causal error estimation based on the IPW estimator may not be accurate or robust, and may lead misleading conclusions about the causal model quality. Existing work on addressing the variance issue of the IPW estimator often involves modifying the IPW estimator itself [Crump et al., 2009, Chaudhuri and Hill, 2014, Sasaki and Ura, 2022, Busso et al., 2014, Robins et al., 2007, Lunceford and Davidian, 2004, Imbens, 2004, Liao and Rohde, 2022]. However, these methods are mainly designed for treatment effect estimation rather than causal error estimation, and may introduce additional bias or complexity that may not be optimal for the purpose of causal model evaluation.
Contributions overview We focus on causal model evaluation with conditionally randomized trials, propose a novel method for low- low-variance estimation of causal error (Equation 1), and demonstrate its effectiveness over current approaches by achieving near-RCT performance. Our key insight is: to estimate the causal error, we can design a simple and effective low-variance estimation procedure without improving the IPW estimator for the true treatment effect. As shown in Figure 1, denote the ground truth effect as ō and the treatment effect of a causal model M as â. Our goal to estimate the causal error À == dmm- &irw with low variance, Using conditionally randomized experiments, we have an estimator of the ground truth via IPW: JPPW Instead of improving the IPW estimator, we replace the commonly used sample mean estimator 8.M with its IPW counterpart, 8: having the same realizations of treatment assignments as in 8' This allows the variance of 8 to be hedged by 8TPW M, reducing the variance of the causal error estimator Å. Contrary to conventional estimation strategies, the pairs estimator, as we call it, effectively reduces the variance of the causal error estimation and provides more reliable evaluations of causal model quality, both theoretically and empirically.
The rest of this paper is structured follows. In Section 3, we formally describe our problem setting, necessary background and notations. In Section 4, we will formally define the pairs estimator for causal model evaluation, and study its theoretical properties (Proposition 1) on variance reduction. Finally, in Section 5, we will conduct simulation studies and demonstrate the effectiveness of the proposed estimation approach, highlighting its potential on achieving near-RCT performance using only conditionally randomized experimental data.
2 Related works
Alternative schemes randomized controlled trials There exists a number of different condi- tionally randomized trial schemes in the literature, for instance randomized encouragement designs [Angrist al., 1996], non-random quantitative treatment assignment [Imbens and Rubin, 2015], and fully observational studies [Rubin, 2005]. In this work, we mainly consider the non-random quantitative treatment assignment setting, where the treatment groups are determined by some known function of covariates. Nevertheless, we have an emphasized focus on using conditionally randomized trials for evaluating the quality of any given causal inference models (trained on observational data), which is fundamental setting for many practical causal model deployment procedure.

---

D = (X;, Tú Yi) ~ Pobs.
(B)= YT,,YT=00 13)
Y 1PW
causal learning
experiment
8IPW
B Peop. (T|x)
model prediction
IPW
8IPW
Poins
M
Figure 1: Overview of the proposed method. Å: the naive estimator for causal error evaluation. Å Pairs; the proposed estimator. Given a causal model M learned from observational data, our goal estimate its causal error (Equation 1) via conditionally randomized experiments (yellow). Naive method (red) would achieve this by comparing @rPW, the IPW estimation of ground truth effect from the experiment, with 8M. the predicted treatment effect from M, usually obtained via population mean (Equation 2). In our method (brown), we replace åm with its IPW counterpart, 8TPW, using the same treatment assignments used in estimating 8'PW, so that the variance from IPW are hedged.
Causal estimation methods with conditionally randomized experiments Apart from IPW men- tioned in Section 1, number of other methods were also developed for conditionally randomized trials settings, include matching [Ho al., 2007, Stuart, 2010, Rubin, 2006], regression adjustment (Neyman, 1923, Rubin, 1974, Lin, 2013, Wager et al., 2016), double robust methods [Robins et al., 1995, Bang and Robins, 2005, Kang and Schafer, 2007, Athey et al., 2018], and machine learning methods [Athey and Imbens, 2016, Wager and Athey, 2018, Shi et al., 2019, Chernozhukov et al., 2018]. Readers can refer to [Imbens and Rubin, 2015, Morgan and Winship, 2015] for a comprehen- sive review. This paper, on the contrary, aims to propose a general method for causal model evaluation applicable to any causal effect estimation method, particularly the widely used IPW estimator.
Variance reduction and robustness There has been a long standing discussion on robustness improvement [Robins al., 1995] and variance reduction of IPW estimators. Several techniques have been proposed to address the problem of extreme or uninformative weights, including weights trimming [Crump al., 2009, Chaudhuri and Hill, 2014, Sasaki and Ura. 2022], weights normalization [Busso et al., 2014, Robins et al., 2007, Lunceford and Davidian, 2004, Imbens, 2004], and more recently, imearly-modified IPW [Zhou and Jia, 2021]. etc. These techniques aim to reduce variance but may introduce bias, complexity, or tuning parameters. Our work differs from the existing literature in that, we focus on the causal error estimation, rather than the individual treatment effect estimation. We do not improve the IPW estimator for the true treatment effect, but rather propose to apply the same IPW estimator both the model and the true effects to hedge their variances.
3 Problem formulation
Consider the data generating distribution for the population on which the experiment is being carried over is given by p(X, T, Y), where X are some (multivariate) covariates, the outcome variable and T is the treatment variable. In this paper, we only consider continuous effect outcomes. Let yT= denote the potential outcome of the effect variable under the intervention T = f. Without loss of generality, assume T' E [0,1]. Then, the interventional means given by p² T= and n E T"D), respectively. The ground truth treatment effect is then given by
p - p. T-1 T-y)
Now, assume that given observational data sampled from p(X, T, Y), we have trained a causal model, denoted by M, whose treatment effect is given by
5M = PM HMM = -
Our goal is then to estimate the causal error of the model, which quantifies how well the model reflects the true effects (the closer to zero, the better):
((M) == 8M -
(I)
3

---

In practice, dM will be the model output, and can be estimated easily, For example, we can sample a pool of i.i.d. subjects D (X,, Yi). .... (XN, Yx) ~ p(X, Y). and the corresponding treatment effect estimation will be given
1 N
8m = 8m=
(2)
which forms the basics of many casual inference methodologies, both for potential-outcome ap- proaches and structural causal model approaches [Rubin, 1974, Rosenbaum and Rubin, 1983, Rubin, 2005, Pearl et al.. 2000]. On the contrary, obtaining the ground truth effect ô is usually not possible without real-world experiments/intereeniion [Imbens and Rubin, 2015]. By definition, can be (hypothetically) approximated by the population mean of potential outcomes:
due to the fundamental problem of causal inference
1 -1-
8 8
IED
However, given a subject only one version of the potential outcomes can be observed. Therefore. we often resort to the experimental approach, the randomized controlled trial (RCT). The RCT approach is always considered as the golden standard for treatment effect estimation, which we would randomly assign treatments to our pool of subjects D (X1, Y1). (XN, YN) p(X, by flipping an unbiased coin. Then the estimated treatment effect is given by:
3RCT =
Σ
(T=
B
N
B
iEB
JED(B
where B denotes the subset of patients that are assigned with the treatment. Together, we have the RCT estimator of the causal error:
RCT (M) = 8M - JRCT
However, when randomized trial is not available, we can only deploy a conditionally randomized test assignment plan, represented by T, which a vector of n Bernoulii random variables T [b..... by]. each determines that Y will be revealed with probability Pn- In practice, T can be given by an treatment assignment model Pexp 1|x). This is an oracle distribution that is known and designed by the experiment designer. A subset of patients B E 'D is selected given these assigment probabilities. Then, the inverse probability weighted (IPW) estimation of the treatment effect is given by
PI (T)
1 T N
TT= (B), (B)
< (D) B):
w(D \ B) w B) -
N
where w = [1/p1, .... 1/pN]. w(B) is created by sub-slicing w with subject indices in B. <, > is the inner product, and is a vector of ones. Finally, the model causal error can be estimated as (dubbed naive estimator in this paper):
((M.T) = 8m - (T)
(5)
In practice, when the size N of the subject pool is relatively small, the IPW estimated treatment effect 8PPW (T) will have high variance especially when Pezp T 1|X) is skewed. As a result, one will expect very high
unbounded variance in the estimation [Khan and Tamer, 2010, Busso al., 2014] (MM, T). The goal of this paper is then improve model quality estimation strategy ((M, such that has lower variance and error rates under conditionally randomized trials.
4 Pairs estimator for causal model quality evaluation
4.1 The pairs estimator
To resolve the problems with the naive estimator for causal error in Section 3, in this section, we propose novel yet simple estimator that will significantly improve the quality of the causal error estimation in model-agonist way. Intuitively, when estimating Á(M, T), we can simply apply the

---

same IPW estimator (with the same treatment assignment) for both the model treatment effect d. and the ground truth treatment effect 6. In this way, we anticipate that the estimators for &M and will become correlated: their estimation error will be canceled out and hence the overall variance is lowered. More formally, we have the following definition:
Definition 1 (Pairs estimator for causal model quality). Assume we have pool of i.i.d. subjects to be tested, namely D (I).,))...., (XN YN) p(X.Y) as well as a conditionally randomized treatment assignment plan, represented by T, which is vector of n. Bernoulli random variables
T = [b, ..., by), each determines that YT'=1 will be revealed with probability Pu- Assume that, for particular trial, a subset of patients B E DD are selected using these probabilities. Then, the IPW estimator of the model's 's treatment effect and the ground truth treatment effect is given by
3IPW MM (T) = N
< <YM (B), w(B)
1
YT=0 (D
\ B3,
w(D ())
N
w(D \ B)
and
81PW (T) = N
<X (B), w(B)
1
<YT=0(D) (B), w(D) \ ())
w(D B)
N
respectively. Then, the pairs estimator of causal model quality is defined as
À Pairs (M, () = = (7) - j (T).
(6)
We will formally show that this new estimator can effectively reduce estimation error. We
will provide the full assumptions and proposition below. Note that we will assume by default that the common assumptions for conditionally randomized experiments will hold, such as non-
inceffrenne/ommiseec///vvrrpp etc, even though we will not explicitly mention them for the rest of
the paper.
4.2 Core assumptions for achieving variance reduction
Our main assumption is regarding the estimation error of causal models, stated as below:
Assumption A. Causal model estimation error for potential outcomes. We assume that for each subject i, the trained causal model's 's potential outcome estimation can be expressed as
YT='() = (T) + V,(YT=(()) (i)) vi, i == 1,2, N, E {0,
(7)
where vs are i.i.d. distributed random error variables with unknown variance oz, that is independent from y-T= and by; and V:(-), = 1, 2, .... N is set of deterministic function that is indexed by i. This assumption is very general and models the modulation effect between the independent noise v and the ground truth counterfactual. One special example would be (i) (T) + YT=t(i) * Mi, where the estimation error will increase (on average) as TTT= increases. In practice, dependencies between error magnitude and ground truth value could arise when the model is trained on observational data that suffers from selection bias, measurement error, omitted variable bias, etc. For the rest of paper, we write Equation 7 in its vectorized form:
= + V(Y *
where all operations are pisnt-wise. Finally, we further require that the causal model's counterfactual prediction should be somewhat reasonable, in the sense that (())(=()()))))) < (E(YT=(()))) ( 1, O; i 1,2,..., N. This implies that the variance of the estimation error should at least be smaller than the ground truth counterfactual.
Assumption B. Asymptotic normality of mean estimators of potential outcome variables. Let be the corresponding mean estimator of the potential outcome variables yT= yT= =1=1 y' = We assume that these estimators are jointly asymptotic normal, i.e..
YT=I, YT=O yT=I
y
VN By YT BYT= y - EY Y - EY;
converge in distribution to zero mean multivariate Gaussian. This is reasonable due to the random- ization and the large samples used in experiments [Casella and Berger, 2021, Deng et al., 2018].
Note that this particular assumption is mainly for achieving asymptotic normality of the estimators, which is orthogonal to achieving variance reduction effect of our pairs estimator, that relies on Assumption A.
5

---

4.3 Theoretical results
Following our assumptions, we can derive the following theoretical result, which shows that given our assumptions described in the previous section, the pairs estimator å Pairs (MirssM,, will effectively reduce estimation variance compared to the naive estimator, (MM,T) Proposition (Variance reduction effect of the pairs estimator). With the assumptions stated in Section 4.2, we can show that our IPW estimators, 8IPW (T) and 81P (T) can be decomposed
81PW (T) = ((B),
M (T) 8. + f(B) (V,
where I and are random variables that depend B (or also v), and g(v, B) is orthogonal d, M and f(B), Furthermore, if we define the estimation error of model quality estimators as follows
e Paire (M,T)) = Å Paire (M, A(M)
e(A(M,T) = ((M, -
then both VNe(A Poirs (M, T)) and (Ne(A(M,T)) T)) are asymptotically normal with zero means, and their variances satisfies
Varle( Pairs (airs(M Varlc(A(M,TT))
Proof: See Appendix
This result provides theoretical justifications that our simple estimator will be effective for variance reduction. In the next section, will conduct simulation studies to empirically evaluate and validate the effectiveness proposed approach.
Simulation Studies
In this section, will evaluate the performance of the proposed pairs estimator, and validate our theoretical insights via simulation studics. We will also examine the robustness and sensitivity of the pairs estimator concerning diffferent scenarios of conditionally randomized trials, including treatment assignment mechanisms, degree of imbalance, choice causal machine learning models, etc.
5.1 Synthetic esuite dataset with hypothetical causal model
Setting. Following Geffner et al. [2022], we construct set of synthetic datasets, designed specifi- cally for evaluating causal inference performance, namely the esuite te datasets. The tata-generating process is based on structural causal models (SCMs), different levels of confounding, heterogeneity, and noise types are incorporated, by varying the strength, direction, and parameterize of the causal effects. We evaluated on three different datasets, namely esuite_ _1, csui te 2, and csui te_3. each with a different SCM. See Appendix B for more details. The corresponding causal model estimation is simulated using special form of Assumption A, that is:
(1) = x²= + YT= 14 = 1,..., N,t {0,
where are i.i.d. distributed zero-mean random variables with variance of that affects the ground truth causal error. To simulate the conditionally randomized trials, we use two different schemes to generate the treatment assignment plans T. The first scheme is based on logistic regression model of the treatment assignment probability given the covariates, that is,
Pezp (T = 1 1 -3x
where / is random vector sampled from multivariate Gaussian distributions with mean zero and variance o3. We vary the degree of imbalance the treatment assignment by changing the value of 0². A larger oz implies a more imbalanced treatment assignment, as the variance of the treatment assignment probability increases. The second scheme is based on a random subsampling of the units, where the treatment assignment probability is fixed for each unit, but different units are sampled with replacement to form different treatment assignment plans. We vary the number of treatment assignment plans by changing the sample size of each subsample.

---

Evaluation method. We compare the performance of the pairs estimator with the naive estimator (Equation 5), as well as the RCT estimator (Equation 4), which is the ideal benchmark for causal model evaluation. We also compare with two other baselines, by replacing the IPW component 8IPW (T) in the naive estimator (Equation 5) by its variance reduction variants. This includes the self-normalized estimator, as well as the linearly modified (LM) IPW estimator Zhou and Jia [2021], a state-of-the- art method for IPW variance reduction when the propensity score is known. We measure the performance of the estimators by the following metrics: the variance, the bias, and the MSE of the causal error estimation. See Appendix B.1 for detailed definitions. We compute these metries by averaging over 100 different realizations of the treatment assignment plans for each dataset.
The results are shown in Figure 2 and Figure Figure 2 shows the results for the logistic regression scheme, with different values of oZ and oz. Figure 3 shows the results for the random subsampling scheme, with different o² In both tables, we report the average and the standard deviation of the performance metrics of the estimators for each value of the true causal error, which is computed by the difference between the true treatment effect and the model treatment effect.
Results.
ve oj=0.1
Unea inea-modifed o2-0.1
Normiiieed IPW of-03
+ ACT
+ Pails o)-0.11
No 02-0.3
unear-modilied 00-033
Norroiized IPW of-0.3
-8- HCT 220.300.3
-8- Pails 00-0.3
s0-0.
caulte Verience
csu Verience
csubt Varierce
0.03
0.01
0.01
csu te_te_ Blos
csulte
cRuite3:: Blas
0.01
0.02
0.00
-0.01
0.03
0.4-
0.02
0.2
0.01
0.00
0.02
Figure 2: Comparison of various causal error estimators for the logistic assignment scheme across (csuite_1. _1, csuite_0 2, and csuite_3) __). In this 3 x 3 plot, each row corresponds to a specific performance metric (Variance/Bias/MSSDD) while each column represents different datasets. In each plot, the x-axis represents the degree of treatment assignment imbalance (a²), while the y-axis displays the performance metrics (lower the better). The different colors indicate the performance of different estimators, and different linestyles and markers indicate different of settings. The results demonstrate that the proposed estimator (purple) consistently outperforms the other estimators in terms of all performance metrics, with more robust behavior as the imbalance in treatment assignment increases.
From the tables, we can see that the variance of the naive estimator quickly increases when the treat- ment assignment is highly imbalanced. Nevertheless, our estimator (purple) consistently outperforms the naive estimator and its variance reduction variants in all metrics (variance/bias/MSSE) regardless of the value of (aī), and the degree of imbalance (o²). The pairs estimator also achieves comparable performance to the RCT estimator, the golden standard by design. The linearly-modified estimators and self- normalizedd estimators have lower variance than the naive estimator, but it also introduces some bias, and their performance is sensitive to the degree of imbalance. These demonstrate the effectiveness and robustness of the proposed pairs estimator when all assumptions are met.
7

---

Verlance
coutte Var
coute
Vorance
0.005
0.002
0.000
0:000
cuite 1: Bisn
cluiee 2: Bas
cuibe 3: Biis
0.02
0.03
0.03
0.000
0.02
0.00
-0.005
0.030
0.1
0.3
0.4
0.5
0.1
0.3
0.5
03
0.2
0.3
0.5
Casilee
Ihible
0.4
0.030
0.000
0.005
0.000
0.1
0.5
0.1
Figure 3: Comparative analysis of multiple causal emror estimators in the context of the random subsampling scheme for (csuite_ cwwsssssee.. __, and csuite__] __3.. In each plot, the x-axis represents the variance of the noise variable (o.), and the y-axis illustrates the performance metrics (Variance/Bi- as/MSE) for each estimator. Different colors denote the performance of different estimators. The findings reveal that, under the random subsampling scheme, the proposed estimator (purple) also consistently surpasses other estimators in every performance metric.
5.2 Synthetic counterfactual dataset with popular causal inference models
Settings. Based on Section 5.1. we now consider more realistic experimental settings, in which we apply a wide range of machine sarming-based causal inference methods by training them from synthetic observational data. Thus, we expect the assumptions in Section 4.2 might not strictly hold anymore, which can be used to test the robustness of our method. We include a wide range of methods, such as linear double machine DML (DML Kernel) [Nie and Wager, 2021], causal random forest (Causal Forest) [Wager and Athey, 2018, Athey et al., 2019], linear doubly robust learning(DR Linear), forest doubly robust learning (DR Forest), orthogonal forest learning (Ortho Forest) [Oprescu et al., 2019], and doubly robust orthogonal forest (DR Ortho Forest). All methods are implemented via EconMI. package [Battocchi et al., 2019]. See Appendix B for more details.
Coaming(Chernozhuuoo et al., 2018] (dubbed as DML Linear), kernel
We mainly focus on two aspects: 1), whether the proposed estimator can still be effective on variance reduction with non-hypothetical models as in Section 5.1; 2), whether the learned causal models' counterfactual predictions approximately follow the postulated Assumption A. Here We present the results for the first aspect, and results for the second can be found in Section 5.3.
Simulation procedure. We repeat the same simulation procedure as in Section 5.1, but using the learned causal inference models instead of the simulated causal model. We compare the performance of the pairs estimator with the same baselines as in Section 5.1, using the same metrics and the same treatment assignment schemes. For DML Linear, DML Kernel, Causal Forest, and Or tho Forest (that does not require propensity scores), we train on 2000 observational data points generated via the following data generating process of single continuous treatment [Battocchi al., 2019]:
W ~Normal (0, I ), X Uniform(O, 1)"* T =T, +7, Y T p(X) + (W. +
where T is the treatment, W is the confounder, X is the control variable, Y is the effect variable, and y and e are some uniform distributed noise. We choose the dimensionality X and W to be n, = 30 and n.p = 30, respectively. For other doubly robust-based methods, we use a discrete treatment that sampled from binary distribution P(T = 1) = sigmoid( (W, B3 + 7), while keeping the others unchanged. Once models are trained the generated observational datasets, we use the

---

Nape
Nomsttee PW
Hetdi: Vedincee 6-
Netric Varance, 6--5
Netric: Varianee. o
1.-
0.5
0.3
0.2-
Meeni: Blac oj-1
Uierte RaM, 55
Iteeren Sans 03-10
0.0
0.23
0.2
-02
0.0
-0.25
-12
-2.4
Hetii: NSC. ois)
MGT, oja5
Mebric: oj
1.5
1.2
0.3
DMt Usner
Clusal Fores
(OR famer
fores
Grtt hover
ceno Foostt
Fenel
Lawer
Ueto Harett
DR Oster Fovest
Unver
Cennal Foret
mi facott
oater fores
DR onea forms
DR
DB
Modda
Madeb
Modes
Figure Performance of different causal error estimators across different causal inference methods. Each row of the grid corresponds to a specific performance metric (Variance/Bias/MSEE) while each column represents different levels of treatment assignment imbalance (o%). each plot, the x-axis represents different causal models. The y-axis displays performance metrics. Different colors correspond to different estimators. The results highlight the differences in performance among the estimators, with the pairs estimator (purple) consistently outperforming others in most scenarios, emphasizing its robustness and effectiveness in assessing various causal models.
trained causal inference model to estimate the potential outcomes and the treatment effects for each unit. Then, we use the logistic regression-bastt based treatment assignment scheme as in Section 5.1 to simulate a hypothetical conditionally randonized experiment (for both continuous treatment and binary treatment, we will assign T 1 for the treatment group and T = 0 for the control group). Both our pairs estimator as well as other baselines presented Section 5.1 will be used estimate the causal error. This is repeated 100 times across 3 different settings for treatment assignment imbalance (o3 1, 5, 10).
Results. Figure 4 2 shows that using our estimator with conditionally randomized data, we achieved near-RCT performance for causal model evaluation quality across all metries. It is also more robust to different settings of oz. where no significant variance change is observed. Also, we note that each causal error estimator has a different sweet-spot: for instance, the naive estimator usually consistently works very well with Causal Forest, whereas estimator has relatively high variance and bias for DML Kernel. Nevertheless, the proposed estimator is still the most robust estimator (apart from RCT) that consistently achieves the best results across all causal models. This shows the feasibility of our method for reliable model evaluation with conditionally randomized experiments.
5.3 Validation of Assumption in Section 4.2
Settings Finally, provide experiments validate Assumption in Section 4.2. Specifically, we use numerical example to check if Assumption A holds in practice for various commonly used models, including the ones considered in Section 5.2, including DML Linear, DML kernel, Causal Forest, and variations of doubly robust algorithms (DR L Linear, DR forest, DR Or- tho Forest). We first fit different methods using the observational data generated in Section 5.2. and obtain the counterfactual predictions from each method; Then, we parameterize the function V(-) Equation a polynomial function and fit for the counterfactual predictions using Equation 7.
'The linearl-modified modified baseline is not displayed for clarity reasons, Appendix B, Figure 6 for full results.

---

Finally, we solve for the residuals i in Equation 7, and test whether they are independent from Y and b, as postulated Assumption
Correlation Coefficients and p-values for the residuals of different methods
Corralation Coefficient
Y'())
0.02
1 by
0.00
-0.02
p-valuc 0.05
D-value
0.4
dml, linear
causel forest
dml kerne
dr llnear
dr forest
artha forest
ortho forest
Figure 5: Validation of whether the learned causal models' counterfactual predictions approximately follow the postulated Assumption A.
Results As shown in Figure 5, we calculate the Pearson correlation coefficient between & and Y and b, respectively, as shown in the upper plot; we also compute the p-values for the null hypothesis that the samples are independent. We observe that the samples have close zero correlations as well as high p-values that are unable to reject the null assumption. Hence, both results shows suggests that the independence multiplicative noise assumption is likely to be true under this setting. We have also tried other tests such as Hoeffding's independence tests that captures non-linear dependencies, which yields similar results. This indicates that Assumption A is not overly restrictive and can be satisfied by a wide range of causal models that are popular in practical applications.
Conclusions
In this paper, we proposed the pairs estimator, a novel methodology for low-variance estimation of causal error in conditionally randomized trials. This approach applies the same IPW method both the model and ground truth effects, canceling out the variance due to IPW. Remarkably, the pairs estimator can achieve near-RCT performance using conditionally randomized experiments, signifying a novel contribution for enabling more reliable and accessible model evaluation, without depending on expensive infeasible randomized experiments. Future work may extend our method more complex scenarios, explore alternative ways to reduce causal eтror estimation variance, and apply our method to other causality applications such as policy evaluation, causal discovery, and counterfactual analysis.
Acknowledgements
We thank Colleen Tyler, Maria Defante, and Lisa Parks for conversations real-world use that inspired this work. We also thank Nick Pawloski, Wenbo Gong, Joe Jennings, Agrin Hilmkil and Meyer Scetbon for useful discussions and support that enables this work.
References
Joshua Angrist, Guido W Imbens, and Donald Rubin. Identification of causal effects using instrumental variables. Journal of the American statistical Association, 91(434):4444455 1996.
10

---

Susan Athey and Guido Imbens. Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences, 113(27):7333-736, 2016.
Susan Athey, Guido W Imbens, and Stefan Wager. Approximate residual balancing: debiased inference of average treatment effects in high dimensions. Journal of the Royal Statistical Society: Series (Statistical Methodology), 80(4))397-00 623, 2018.
Susan Athey, Julie Tibshirani, and Stefan Wager. Generalized random forests. 2019.
Heejung Bang and James M Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, (4):9622-973 2005.
Keith Battocchi, Eleanor Dillon, Maggie Hei, Greg Lewis, Paul Oka, Miruna Oprescu, and Vasilis Syrgkamis. EconML: A Python Package for ML-Based Heterogeneous Treatment Effects Estima- 2019.
tion.
Colin R Blyth. On simpson's 's paradox and the sure-thing principle. Journal the the American Statistical
Association, 67(338):364-366 1972.
Matias Busso, John DiNardo, and Justin McCrary. New evidence on the finite sample properties of propensity score reweighting and matching estimators. Review of Economics and Statistics, 96(5): 885-897, 2014.
George Casella and Roger L Berger. Statistical inference. Cengage Learning. 2021.
Saraswata Chaudhuri and Jonathan B Hill. Heavy tail robust estimation and inference for average treatment effects. Technical report, Working paper, 2014.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whit- ney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters, The Econometrics Journal, 21(1):C1-C68, 2018.
Richard K Crump, V Joseph Hotz, Guido W Imbens, and Oscar Mitnik. Dealing with limited overlap in estimation of average treatment effects. Biometrika, 96(1):187-199 2009.
Alex Deng, Ulf Knoblich, and Jiamnan Lu. Applying the delta method in metric analytics: A practical guide with novel ideas. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 233-242,14 2018.
Tomas Geffner, Javier Antoran, Adam Foster, Wenbo Gong, Chao Ma, Emre Kiciman, Amit Sharma, Angus Lamb, Martin Kukla, Nick Pawlowski, et al. Deep end-to-end causal inference. arXiv preprint XxXi:2222..02195 2022.
Jinyong Hahn. On the role of the propensity score in efficient semiparametric estimation average treatment effects. Econometrica, pages 315-331, 1998.
Keisuke Hirano, Guido W Imbens, and Geert Ridder. Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 1114)116111119, 2003.
Daniel E Ho, Kosuke Imai, Gary King, and Elizabeth A Stuart. Matching nonparametric prepro- cessing for reducing model dependence in parametric causal inference. Political analysis, 15(3): 199-236, 2007.
Guido W Imbens. Nonparametric estimation of average treatment effects under exogeneity: A review. Review of Economics and statistics, 86(1):4-29, 2004.
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
Joseph DY Kang and Joseph I. Schafer. Demystifying double robustness: comparison of alternative strategies for estimating a population mean from incomplete data. 2007.
Shakeeb Khan and Elie Tamer. Irregular identification, support conditions, and inverse weight estimation. Econometrica, 7866):20 2010.

---

Jiangang Liao and Charles Rohde. Variance reduction in the inverse probability weighted estimators for the average treatment effect using the propensity score. Biometrics, 78(2):660-66 66466666677.6 2022.
Winston Lin. Agnostic notes on regression adjustments to experimental data: Reexamining freedman's critique. 2013.
Jared K Lunceford and Marie Davidian. Stratification and weighting via the propensity score in estimation of causal treatment effects: comparative study. Statistics in medicine. 23(19): 2937-2960, 2004.
Stephen L Morgan and Christopher Winship. Counterfactuals and causal inference. Cambridge University Press, 2015.
Jersey Neyman. Sur les applications de la théorie des probabilités experiences agricoles: Essai des principes. Roczniki Nauk Rolniczych, 10:1-51, 1923.
Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299-33 119, 2021.
Miruna Oprescu, Vasilis Syrgkanis, and Zhiwei Steven Wu. Orthogonal random forest for causal inference. In International Conference on Machine Learning, pages 4932-4941. PMLR, 2019.
Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CambridgeUnivvesitypPrss,, 19 (2), 2000.
James Robins, Mariela Sued, Quanhong Lei-Gomez, and Andrea Rotnitzky. Comment: Performance of double-robust estimators when" inverse probability" weights are highly variable. Statistical Science, (4(4):544-55, 2007.
James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Analysis of semiparametric regression models for repeated outcomes in the presence of missing data. Journal of the umerican statistical association, 90(429): 1995.
Paul R Rosenbaum. Model-based direct adjustment. Journal of the American statistical Association, 82(398):387-394, 1987.
Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55 1983.
Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688, 1974.
Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 1022461) 2005.
Donald B Rubin. Matched sampling for causal effects. Cambridge University Press, 2006.
Yuya Sasaki and Takuya Ura. Estimation and inference for moments of ratios with robustness against large trimming bias. Econometric Theory, 38(1):66- 112, 2022.
Claudia Shi, David Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effeects Advances neural information processing systems, 32, 2019.
Elizabeth A Stuart. Matching methods for causal inference: A review and look forward. Statistical science: a review journal of the Institute of Mathematical Statistics, 25(1):1, 2010.
Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228-1244 2018.
Stefan Wager, Wenfei Du, Jonathan Taylor, and Robert J Tibshirani. High-dimensional regression adjustments in randomized experiments. Proceedings of the National Academy of Sciences, 113 (45): 2016.
Stephen G West, Naihua Duan, Willo Pequegnat, Paul Gaist, Don C Des Jarlais, David Holtgrave, José Szapoeznik, Martin Fishbein, Bruce Rapkin, Michael Clatts, et al. Alternatives to the randomized controlled trial. American journal of public health, 1359 1366. 2008
12

---

Kangjie Zhou and Jinzhu Jia. Variance reduction for causal inference. arXiv preprint
arXiv:2109.05750, 2021.
13


Reviews:
[{"review_id": "LGuMBL7tEn", "reviewer": "Reviewer_PdWH", "summary": "The paper considers the estimation of causal error using the IPW estimator in conditional randomized experiments. Given that the allocation probabilities are readily available in these types of experiments, IPW estimators are often used. The authors propose to use the same IP weights for both the causal prediction, and the the estimator of the ground truth, and show that this so-called pair estimation approach reduces the variance in the distance between the causal prediction and ground truth. They provide theoretical justifications for their approach in terms of variance reduction, and illustrate this on synthetic data sets.", "questions": "The authors refer to the phenomenon that the oracle IPW may not be as efficient as the one using a correctly specified model as one of the limitation of the IPW estimator. Why is this a limitation?", "limitations": "The author’s use of non-RCT data is very misleading; this is actually often called the conditional randomized studies (e.g. Imbens and Rubins’ book), and is one common type of randomized studies. A trial does not need to have the same coin for everyone!\n\nThe authors mention that their approach is different from the modern approaches that try to stablize the IP weights. It is unclear to me if one already use these stablization methods, then whether the authors’ method is still useful.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of using the same IP weights for both the causal prediction and ground truth estimation is simple yet effective. ", "weaknesses": "There is no evaluation of real data sets.\n\nOften the IPW is bad when the ps is unknown and the model is misspecified. When the ps is known by design, this seems to be less of a concern, and can/should probably be addressed by a better design. \n\nOften the IP weights are used in conjunction with an OR model to construct a DR estimator. In fact, there is really no good reasons to use the naive IPW estimator considered by the authors. I doubt the proposed method may also be useful for improving the DR estimator, but not as dramatic. The authors should probably have done this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback and suggestions. Below, we respond to each of your comments.\n\n\n> Often the IP weights are used in conjunction with an OR model to construct a DR estimator. In fact, there is really no good reasons to use the naive IPW estimator considered by the authors.\n\nWe acknowledge the effectiveness of DR estimators in causal inference settings. However, our paper focuses on **model evaluation in real-world scenarios** where IP weights are known by design, and model misspecification is less of a concern. As such, the introduction of the DR estimator might not be necessary or optimal. Instead, the high variance issue of IPW is more of a concern, which cannot be mitigated by D. We compared our method with existing IPW variance reduction techniques, such as self-normalized/renormalized [1, 2] and linearly modified (LM) IPW estimators[3], demonstrating the effectiveness of our approach.\n\n> Often the IPW is bad when the ps is unknown and the model is misspecified. When the ps is known by design, this seems to be less of a concern, and can/should probably be addressed by a better design.\n\nWhile it is true that known propensity scores alleviate some concerns, IPW might still suffer from high variance, leading to poor estimates. Addressing this issue through a different design is often infeasible due to practical limitations, such as financial concerns, or ethical constraints. Our method provides a solution that works within these constraints, helping to improve causal model evaluation even when propensity scores are known by design.\n\n\n> There is no evaluation of real data sets.\n\nWe recognize the absence of real dataset evaluations as a weakness of our paper. However, it is very difficult to find such real-world datasets that satisfies: 1) the non RCT treatment assignment has known oracle propensity scores; 2) the interventional outcomes are not generated in a semi-synthetic manner. Otherwise, these “real” datasets would be no different from our experiments. Our method was inspired by a real-world problem we encountered, and we achieved good results in our actual application, motivating us to share the method that we discovered. Due to confidentiality reasons, we cannot share these proprietary results and data publicly. That said, we believe our simulation studies sufficiently demonstrate the potential of our method in various scenarios.\n\n\n> The author’s use of non-RCT data is very misleading; this is actually often called the conditional randomized studies (e.g. Imbens and Rubins’ book), and is one common type of randomized studies. A trial does not need to have the same coin for everyone!\n\nThank you for the suggestion. We recognize that there is different convention of naming such as “non-RCT” used in [4] but CRS in other literatures.  In revision, we will use the suggested terms and mention different conventions in the literature.\n\n> It is unclear to me if one already use these stablization methods, then whether the authors’ method is still useful.\n\nIn our experiments, we have indeed compared the performance of our method versus some stabilization methods such as renormalization and linear corrections; and our results showed that our method will still be helpful even if these methods are already available.\n\n**Reference**\n\n[1] Imbens, Guido W. \"Nonparametric estimation of average treatment effects under exogeneity: A review.\" Review of Economics and statistics 86.1 (2004): 4-29.\n\n[2] Lunceford, Jared K., and Marie Davidian. \"Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study.\" Statistics in medicine 23.19 (2004): 2937-2960.\n\n[3] Zhou, Kangjie, and Jinzhu Jia. \"Variance Reduction for Causal Inference.\" arXiv preprint arXiv:2109.05150 (2021).\n\n[4] Rubin, Donald B. \"Estimating causal effects of treatments in randomized and nonrandomized studies.\" Journal of educational Psychology 66.5 (1974): 688.\n"}, {"review_id": "CDp3wTfc3h", "reviewer": "Reviewer_r45Q", "summary": "This work aims to evaluate the fidelity of causal models in estimating true treatment effects across different treatments. The golden approach involves comparing treatment effects derived from the target causal model and those obtained from Randomized Controlled Trials (RCT). Practical, time, cost, and ethical constraints often necessitate replacing the RCT estimate with non-RCT methods such as Inverse Probability Weighting (IPW). However, IPW may lead to unbounded variance due to imbalanced propensity scores. To address this, the authors introduce a procedure that applies the IPW estimator to both the model and the actual effects. This aligns the estimated treatment effects, thus offsetting their estimation errors, and results in a lower variance causal error estimate. Under the two stated assumptions, the authors show that the variance of the causal error estimated from their approach, namely pairs estimator, is upper bounded by variance of the the causal error estimated from the naive estimator. In their experiments, the authors compared their approach with the naive estimator, RCT estimator, and existing state-of-the-art variance reduction estimators such as the self-normalized estimator and the LW IPW estimator. They carried out these comparisons on three synthetic datasets, under various non-RCT scenarios, which included different treatment assignment units across sub-populations and varying degrees of propensity score imbalance. The results demonstrated that their approach consistently produced low estimation errors, often on par with those from the RCT estimator. Moreover, they also evaluated the performance of their approach when the existing machine learning-based causal models are used in treatment effect estimation. Still, the pairs estimator yield low estimation errors and yielded results comparable to those from the RCT estimator.", "questions": "1. Regarding Figure 2, can you clarify why the variance of the Linear-modified estimator and Normalized IPW initially increase and subsequently decrease as the imbalance degree increases? Wouldn't these variance reduction methods yield improved results when the degree of imbalance is less pronounced?\n\n2. In assumption A, what is $b_i$?\n\n3. Given that your theoretical findings strongly depend on Assumption A, the compliance of the learned causal model's counterfactual predictions with this assumption becomes a key aspect of your experimental inquiry. Could you please provide the appendix and discuss these results in more detail?\n\n####################################################################################\n\n[08/19/ 2023] Reviewer r45Q: The experiment validation on Assumption A and the proof for Proposition 1 are provided, and hence I adjust my review accordingly.\n\n####################################################################################", "limitations": "To my knowledge, this work does not have potential negative societal impacts. However, the authors did not provide a section with these discussions.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The author proposes a simple yet powerful procedure for estimating causal error without modifying IPW, which might lead to other complexity, such as parameter tuning, and bias introduction. Their experiments demonstrate their approach has the capability in estimating true causal error faithfully under many existing non-RCT scenarios that are often encountered in practice.\n\n2. The problem statement, formulation, and illustration are clearly stated and well structured, allowing readers to follow easily.", "weaknesses": "Despite their approach being supported by the theoretical results and extensive experiments presented, the authors have not provided an appendix. The thorough justification of their theoretical results and experimental details could only be further substantiated with access to this supplementary material.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your constructive and positive feedback to our paper. We would address your comments below. \n\n> Despite their approach being supported by the theoretical results and extensive...the authors have not provided an appendix.\n\nWe apologize for the missing appendix, this is due to our unpurposive mistakes. We will add it in the revision. Regarding the theoretical results, they are appended at the end of this response since the proof is not complicated.\n\n> The compliance of the learned causal model's counterfactual predictions with this assumption becomes a key aspect of your experimental inquiry. Could you please provide the appendix?\n\nYes, please see the pdf file our general reply.\n\n> In assumption A, what is $b_i$?\n\n$b_i$ is given in the paragraph below Eq (4), which is the Bernoulli random variable of treatment assignment.\n\n**Proof of Prop. 1**:\n\nFirst, it is straightforward to verify that under **Assumption A**, $\\hat{\\Delta}^{Pairs}(\\mathcal{M}, \\mathcal{T})$ and $\\hat{\\Delta}(\\mathcal{M}, \\mathcal{T})$ can be decomposed as\n\n$ \\hat{\\Delta}^{Pairs}(\\mathcal{M}, \\mathcal{T}) =  \\hat{\\delta}_\\mathcal{M} - \\hat{\\delta}  + g(\\mathbf{\\nu}, \\mathcal{B}) $\n\n$ \\hat{\\Delta}(\\mathcal{M}, \\mathcal{T}) = \\hat{\\delta}_\\mathcal{M} - \\hat{\\delta}   - f(\\mathcal{B})$\n\nWhere:\n\n $f(\\mathcal{B}) = \\frac{1}{N} <\\mathbf{Y}^{T=1} (\\mathcal{B}), \\mathbf{w}(\\mathcal{B}) - 1 >   + \\frac{1}{N} <\\mathbf{Y}^{T=0}(\\mathcal{B}), \\mathbf{1}>  $  \\\n$- \\frac{1}{N}<\\mathbf{Y}^{T=0}(\\mathcal{D} \\setminus \\mathcal{B}), \\frac{1}{\\mathbf{w}(\\mathcal{D} \\setminus \\mathcal{B}) - 1} > - \\frac{1}{N}<\\mathbf{Y}^{T=1}(\\mathcal{D} \\setminus \\mathcal{B}), \\mathbf{1}> $,\n\nAnd: \n\n$g(\\mathbf{\\nu}, \\mathcal{B})  = \\frac{1}{N}<\\mathbf{\\nu}(\\mathcal{B}) * \\mathbf{v}(\\mathbf{Y}^{T=1} (\\mathcal{B})), \\mathbf{w}(\\mathcal{B}) - 1 >    \\frac{1}{N}<\\mathbf{\\nu}(\\mathcal{B}) *\\mathbf{v}(\\mathbf{Y}^{T=0}(\\mathcal{B})), \\mathbf{1}> $ \\\n$ - \\frac{1}{N}<\\mathbf{\\nu}(\\mathcal{D} \\setminus \\mathcal{B}) *\\mathbf{v}(\\mathbf{Y}^{T=0}(\\mathcal{D} \\setminus \\mathcal{B})), \\frac{1}{\\mathbf{w}(\\mathcal{D} \\setminus \\mathcal{B}) - 1} > - \\frac{1}{N}<\\mathbf{\\nu}(\\mathcal{D} \\setminus \\mathcal{B})*\\mathbf{v}(\\mathbf{Y}^{T=1}(\\mathcal{D} \\setminus \\mathcal{B})), \\mathbf{1}>  $ .\n\nTheir estimation error can then be expressed as\n\n$ e(\\hat{\\Delta}^{Pairs}(\\mathcal{M}, \\mathcal{T}))  =  \\hat{\\delta}_\\mathcal{M} - \\hat{\\delta}  + g(\\mathbf{\\nu}, \\mathcal{B}) - \\Delta(\\mathcal{M}) $ \n\n$ e(\\hat{\\Delta}(\\mathcal{M}, \\mathcal{T})) =  \\hat{\\delta}_\\mathcal{M} - \\hat{\\delta}   - f(\\mathcal{B}) - \\Delta(\\mathcal{M}) $ \n\n\nAccording to delta method, both $\\sqrt{N} e(\\hat{\\Delta}^{Pairs})$ and $\\sqrt{N} e(\\hat{\\Delta})$ are asymptotically normal with zero mean under Assumption B. \n\nNote that $g(\\mathbf{\\nu}, \\mathcal{B})$ can be rewritten as\n\n   $  g(\\mathbf{\\nu}, \\mathcal{B})  = \\frac{1}{N}<\\pmb{\\mathbf{\\nu}} *\\mathbf{b}*\\mathbf{v}(\\mathbf{Y}^{T=1}), \\mathbf{w} - 1 >   + \\frac{1}{N}<\\mathbf{\\nu} *\\mathbf{b} *\\mathbf{v}(\\mathbf{Y}^{T=0}), \\mathbf{1}> $ \\\n   $ - \\frac{1}{N}<\\mathbf{\\nu} *(1-\\mathbf{b}) *\\mathbf{v}(\\mathbf{Y}^{T=0}), \\frac{1}{\\mathbf{w} - 1} > - \\frac{1}{N}<\\mathbf{\\nu}*(1-\\mathbf{b}) *\\mathbf{v}(\\mathbf{Y}^{T=1}), \\mathbf{1}> $\n\nwhere $b_i$ is the Bernoulli variable with $P(b_i=1) = p_i$, and $b_i = 1$ if $i \\in \\mathcal{B}$.\n\nNote also that $\\nu$ is independent from $(Y^{T}(i), b_i)$ and has zero mean and variance $\\sigma$, we have \n\n$ Cov(Y^{T=t_a}(i), \\nu_i b_i V_i(Y^{T=t_b}(i)) ) = E(\\nu_i b_i) Cov(Y^{T=t_a}(i), V_i(Y^{T=t_b}(i)) ) = 0$ \n\nholds for all $i$ and all treatments $t_a$ and $t_b$. Similarly, we have $ Cov(Y_\\mathcal{M}^{T=t_a}(i), \\nu_i b_i V_i(Y^{T=t_b}(i)) ) =0$. \n\nTherefore, it is not hard to show that $ Cov( g(\\mathbf{\\nu}, \\mathcal{B}), \\hat{\\delta}_\\mathcal{M}) = 0 $, and $Cov( g(\\mathbf{\\nu}, \\mathcal{B}), \\hat{\\delta} ) = 0$,  \n\nwhich implies $Cov( g(\\mathbf{\\nu}, \\mathcal{B}), \\hat{\\delta}_\\mathcal{M} - \\hat{\\delta} ) = 0 $. Thus, we have:\n\n$\n    Var[ \\sqrt{N} e(\\hat{\\Delta}^{Pairs}(\\mathcal{M}, \\mathcal{T})) ]  =  Var[ \\sqrt{N} (\\hat{\\delta}_\\mathcal{M} - \\hat{\\delta})]  + Var[\\sqrt{N}g(\\mathbf{\\nu}, \\mathcal{B})] $ \n\n$ = Var[Y_\\mathcal{M}^{T=1} - Y_\\mathcal{M}^{T=0} + Y^{T=1} - Y^{T=0}] +  Var[\\sqrt{N}g(\\mathbf{\\nu}, \\mathcal{B})]\\\\\n     Var[\\sqrt{N}e(\\hat{\\Delta}(\\mathcal{M}, \\mathcal{T}))]  $\n\n$= Var[\\sqrt{N}( \\hat{\\delta}_\\mathcal{M} - \\hat{\\delta})]  + Var[ \\sqrt{N(}f(\\mathcal{B}))] $\n\n  $ = Var[Y_\\mathcal{M}^{T=1} - Y_\\mathcal{M}^{T=0} + Y^{T=1} - Y^{T=0}] + Var[ \\sqrt{N(}f(\\mathcal{B}) )] $.\n\n\nSince $\\nu$ has zero mean and variance $\\sigma^2_\\nu$ and is independent of $(Y_i^{T}, b_i)$ as in Assumption A, this expression be further simplified as, according to the rules of variance of the product of independent variables:\n\n$    Var[g(\\mathbf{\\nu}, \\mathcal{B})]  = \\frac{1}{N^2}<\\sigma^2_\\nu *\\mathbf{p}*E(\\mathbf{v}(\\mathbf{Y}^{T=1})^2), (\\mathbf{w} - 1)^2 >   + \\frac{1}{N^2}<\\sigma^2_\\nu *\\mathbf{p} * E(\\mathbf{v}(\\mathbf{Y}^{T=0})^2), \\mathbf{1}>  $\n\n$     + \\frac{1}{N^2}<\\sigma^2_\\nu *(1-\\mathbf{p}) *E(\\mathbf{v}(\\mathbf{Y}^{T=0})^2), \\frac{1}{\\mathbf{w} - 1} > + \\frac{1}{N^2}<\\sigma^2_\\nu*(1-\\mathbf{p}) * E(\\mathbf{v}(\\mathbf{Y}^{T=1})^2), \\mathbf{1}> $\n\n$     = \\frac{\\sigma^2_{\\nu}}{N^2}[ <\\mathbf{p}(\\mathbf{w}-1)^2 + (\\mathbf{1}-\\mathbf{p}), E((\\mathbf{v}(\\mathbf{Y}^{T=1})^2)> +  <\\frac{1-\\mathbf{p}}{\\mathbf{w} - 1} + (\\mathbf{p}), E\\mathbf{v}(\\mathbf{Y}^{T=0})^2)>] $\n\n$     < \\frac{1}{N^2}[ <\\mathbf{p}(\\mathbf{w}-1)^2 + (\\mathbf{1}-\\mathbf{p}), E((\\mathbf{Y}^{T=1})^2)> +  <\\frac{1-\\mathbf{p}}{\\mathbf{w} - 1} + (\\mathbf{p}), E((\\mathbf{Y}^{T=0})^2)>]     = Var[f(\\mathcal{B})] $.\n\nWhere the third equality is due to the fact that $ \\sigma^2_\\nu E [(V_i Y^{T=t}(i))^2] < E [Y^{T=t}(i)^2]$. Therefore, we finally conclude that the variances of the error estimators will satisfy\n$ Var[ \\sqrt{N} e(\\hat{\\Delta}^{Pairs}(\\mathcal{M}, \\mathcal{T})) ] < Var[\\sqrt{N}e(\\hat{\\Delta}(\\mathcal{M}, \\mathcal{T}))]$.\n\n**QED**\n\n"}, {"review_id": "7x2Y8eoFES", "reviewer": "Reviewer_zDTG", "summary": "This paper constructs a new estimator for IPW evaluation by comparing the IPW estimator applied on the model-predicted treatments versus the observed treatments. The paper presents a theoretical result that this estimator has lower variance than the naive one and aims to demonstrate this via empirical experiments.", "questions": "1. Could the authors briefly describe the derivation of the IPW estimator (the unlabeled equation between eq. 4 and 5). I have not seen it in this form before. \n\n2. How do the authors explain the large spike in variance at \\sigma_\\beta = 10 in Figure 2 for the baseline?", "limitations": "The main limitation is the lack of the Appendix which contains significant details regarding the contributions of this paper. The authors should also discuss potential limitations of their theoretical assumptions.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The structure of the paper and most of the writing are very good. \n\n2. The theoretical result is important. \n\n3. The empirical results seem to show that the estimator is better than the naive one.", "weaknesses": "1. The paper could benefit from more clarity in the writing. For example, in line 124, it would be great if there can be some intuition or example on when P(T=1|X) is skewed and what that means (is it overfitting or mis-specified)? Furthermore, it would be great if there was an example of a commonly used model and how it satisfies Assumption A, and why this assumption is not a strong one. \n\n2. While the theoretical result seems important, the supplementary file with the proof is missing, which makes it impossible to review. \n\n3. Similarly, the empirical experiments, while extensive, rely on datasets and details that are supposed to be in the supplementary file, but that file is missing. Therefore, it is unfortunately impossible to review the setting in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging review and valuable suggestions to improve.\n\n> The paper could benefit from more clarity in the writing. For example, in line 124, it would be great if there can be some intuition or example on when P(T=1|X) is skewed and what that means (is it overfitting or misspecified)?\n\nWe appreciate your suggestion and will clarify this point in our revision. In our paper, P(T=1|X) is a given oracle distribution that is known and designed by the experiment designer to evaluate the causal predictions of another causal model; and the goal of such experiment is to evaluate the predictions of a existing causal model. \n\nThe skewness of P(T=1|X) is not due to overfitting or misspecification; rather, it often results from practical constraints such as ethical, legal, or financial considerations. For example, in business scenarios, any experimental interventions could have financial impact in the real world, leading to a trade-off between utility and test power. A more skewed policy might be preferred to minimize the negative impact of random assignment, at the cost of higher effect estimation variance. Our proposed method aims to address this challenge by providing a more reliable evaluation in such scenarios.\n\n\n\n> Furthermore, it would be great if there was an example of a commonly used model and how it satisfies Assumption A, and why this assumption is not a strong one.\n\nThanks for your suggestion, relevant results are provided in the attached pdf in our general reply. We demonstrate with different models including linear DML, kernel DML, causal random forest, and variations of doubly robust algorithms (linear, forest, orthogonal), this assumption holds in practice naturally.\n \n> While the theoretical result seems important, the supplementary file with the proof is missing, which makes it impossible to review.\n\nWe apologize for the omission of the appendix in our submission. This was an unintentional mistake, and we appreciate your understanding. Please refer to the proof provided in our response to Reviewer r45Q to review our theoretical results.\n\nWe hope that our revised responses address your concerns more effectively. Please let us know if there are any further questions or suggestions.\n\n> Could the authors briefly describe the derivation of the IPW estimator (the unlabeled equation between eq. 4 and 5). I have not seen it in this form before.\n\nThis is just a vectorized form of the usual IPW definition. Note that\n\n$\n\\hat{\\delta}^{IPW}(\\mathcal{T})  = \\frac{1}{N} \\sum_{i\\in \\mathcal{B}} \\frac{\\mathbf{Y}^{T=1}(i)}{p_i} - \\frac{1}{N} \\sum_{j\\in \\mathcal{D} \\setminus \\mathcal{B}} \\frac{\\mathbf{Y}^{T=1}(j)}{(1 - p_j)} $\n\n$ = \\frac{1}{N} \\sum_{i\\in \\mathcal{B}} \\frac{\\mathbf{Y}^{T=1}(i)}{p_i} - \\frac{1}{N} \\sum_{j\\in \\mathcal{D} \\setminus \\mathcal{B}} \\frac{\\mathbf{Y}^{T=1}(j)/p_j}{(1/p_j - 1)} $\n\n$ = \\frac{1}{N}<\\mathbf{Y}^{T=1}(\\mathcal{B}), \\mathbf{w}(\\mathcal{B})> - \\frac{1}{N}<\\mathbf{Y}^{T=0}(\\mathcal{D} \\setminus \\mathcal{B}), \\frac{\\mathbf{w}(\\mathcal{D} \\setminus \\mathcal{B})}{\\mathbf{w}(\\mathcal{D} \\setminus \\mathcal{B}) - 1}> .\n$\n\n\n> How do the authors explain the large spike in variance at \\sigma_\\beta = 10 in Figure 2 for the baseline?\n\nThe spike of variance is expected since when $\\sigma_\\beta$ increases, the imbalance of treatment assignment would also increase.  Sometimes the variance might go down again at $ \\sigma^2_\\beta = 20$, but this almost only happens for the combination of certain baselines (e.g., naive IPW), certain $\\sigma_\\nu$ values (e.g., very small) and certain datasets (e.g., csuite_3 dataset). This might be due to that when $\\sigma_\\nu$ is very small, the model's bias will also be small, resulting in a causal error that has a much smaller scale, hence the smaller variance. \n\n"}, {"review_id": "CuAdoSlRdC", "reviewer": "Reviewer_t8vT", "summary": "This paper proposes a new estimator for the causal error, that achieves lower variance than previous approaches.\nThe estimator consists in the difference between a IPTW-like estimator using the causal model and a direct IPTW causal effect estimator.\nThe paper shows that under clear assumptions, this estimator results in lower variance than a naive estimator, which only uses a typical IPTW. The authors further test this estimator empirically on a wide range of setups that both satisfy and potentially violate their assumptions.", "questions": "Same as above:\n\nThe main theoretical comparison of this paper seems to be the naive IPTW estimator. As the authors state in the related works section, other estimators have been proposed in the literature to reduce the variance of the causal effect estimator. How does this estimator compare to those theoretically ? And can you leverage some the improvements of IPTW in your method too ?\n\nI think the whole goal of the paper would deserve more clarity. For instance, the overall goal is usually to estimate the true causal effect rather the causal error. Having a good estimator of the causal effect would directly result in low causal error. I believe this would deserve some more motivation / details in the text. \n\nFurthemore, I would encourage the author to make the problem setup more clear. In my opinion, it is not fully transparent from the paper if the model is trained on a different set of samples than the ones used for the IPTW estimator. I believe this is the case but this should be state more clearly. ", "limitations": "The limitations and assumptions of this work were carefully adressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper propose a simple yet effective way to reduce the variance of the causal error estimation.\n\nThe assumptions under which the main result hold are very clearly stated and discussed.\n\nThe authors have extensively tested their approach. In particular, they have tested both on simulated data that satisfy their assumptions as well as on data that potentially violate them, to stress test the method.", "weaknesses": "The main theoretical comparison of this paper seems to be the naive IPTW estimator. As the authors state in the related works section, other estimators have been proposed in the literature to reduce the variance of the causal effect estimator. How does this estimator compare to those theoretically ? And can you leverage some the improvements of IPTW in your method too ?\n\nI think the whole goal of the paper would deserve more clarity. For instance, the overall goal is usually to estimate the true causal effect rather the causal error. Having a good estimator of the causal effect would directly result in low causal error. I believe this would deserve some more motivation / details in the text.\n\nFurthemore, I would encourage the author to make the problem setup more clear. In my opinion, it is not fully transparent from the paper if the model is trained on a different set of samples than the ones used for the IPTW estimator. I believe this is the case but this should be state more clearly. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive opinions for our work. We are pleased to see that you appreciate our efforts in technical soundness, presentation, as well as evaluations.\n\n> The main theoretical comparison of this paper seems to be the naive IPTW estimator. As the authors state in the related works section, other estimator s have been proposed in the literature to reduce the variance of the causal effect estimator. How does this estimator compare to those theoretically ? And can you leverage some the improvements of IPTW in your method too ?\n\nThanks for your question. First, due to the variety of existing estimators, we cannot assert the theoretical advantages for each of them. However, we believe our results can be extended to the case of any extensions of self-normalized IPW, such as the Adaptive normalization for IPW estimation [1]. We will add discussions regarding this in the revision. Moreover, we would like to enhance that our paper focus on improving evaluation of causal inference models, rather than improving causal effect estimations itself.\n\n> Having a good estimator of the causal effect would directly result in low causal error. I believe this would deserve some more motivation / details in the text.\n\nWe appreciate your suggestion and will provide better motivation in our revision. We would like to enhance that we did evaluate our method against other IPW variance reduction methods that are designed to obtain better causal effects; and our results indeed show that our method can still be very valuable when other effect-oriented estimators are not working well.  \n\nOur method aims to offer a new, easy-to-use tool for causal practitioners, complementing other existing tools. By focusing on causal error estimation, we can address specific challenges in evaluating causal models.\n\n> Furthemore, I would encourage the author to make the problem setup more clear. In my opinion, it is not fully transparent from the paper if the model is trained on a different set of samples than the ones used for the IPTW estimator. I believe this is the case but this should be state more clearly.\n\nThank you for pointing this out. We will make this clearer in our revision. As illustrated in Figure 1, there are two data sources: an observational data source for training the model and an interventional data source obtained through a designed oracle treatment assignment mechanism P(T=1|X) for validating the model. The IPW estimator is used to estimate the ground truth effect based on the interventional data, leveraging the known propensity scores P(T=1|X).\n\n\n**Reference**\n\n[1] Khan, Samir, and Johan Ugander. \"Adaptive normalization for IPW estimation.\" Journal of Causal Inference 11.1 (2023): 20220019.\n"}, {"review_id": "W3RcjQ1Gty", "reviewer": "Reviewer_yvJa", "summary": "The authors formulate and evaluate an approach to solving a non-standard problem: evaluating a causal model M when additional data (not used to construct M) is available from a non-randomized experiment. In particular, the authors focus on comparing IPW estimates from the non-RCT data and from the inferences of the model.", "questions": "1. What are examples of practical scenarios in which researchers have a model of treatment effect and then they collect data non-RCT data to evaluate that model?\n\n2. Under what circumstances would the pairs estimator fail to provide estimates with low variance and low bias?", "limitations": "The experiments do not identify cases in which the pairs estimator will fail. In the synthetic experiments whose results are shown in Figure 2 and 3 (which could be used to identify such failure modes), the results for the pairs estimator are freakishly good.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea is a simple and apparently powerful one: Remove the variability due to IPW by performing IPW on both the actual data and the estimates from the model. This largely removes an apparently extraneous source of variability (IPW itself) and allows direct comparison of the estimates.", "weaknesses": "The basic idea of the pairs estimator assumes some basic properties of the IPW estimator. Specifically, if IPW was a terrible estimator whose estimates were unrelated to the data (for example, it always output a single value for ATE: 0.5), then the pairs estimator would always show that the model was essentially perfect, regardless of the model’s estimates or the non-RCT data. I don’t think this scenario likely, but it is possible. This implies that, at least, some diagnostic tests are in order to increase confidence in the output of the pairs estimator. For example, you could introduce noise into the model’s estimates and see if the estimated error increases. \n\nThe results in Figures 2 and 3 are very good. Indeed, they are *freakishly* good. They are so good that it makes me wonder whether the experiments reported in these figures are really evaluating anything important about how the pairs estimator works in practice. This bears some discussion in the description of the results.\n\nThe introduction to the paper may be confusing to many readers. When first encountering the term “non-random experiment”, many readers will balk, thinking that randomization is the *sine qua non* of experimentation and seeing “non-random experiment” as a contradiction in terms. The authors could save readers this confusion by introducing the example of explicit non-random assignment (line 38) earlier or by moving the first paragraph of Section 2 (Related Works) to early in the introduction.\n\nAnother issue may confuse readers: The contrast between an IPW estimate and the “model’s” estimate of treatment effect. For many readers, the goal of analyzing observational data is to get a single estimate of treatment effect, perhaps through IPW. In this scenario, there is no “model” (or, the model is a model of treatment propensity). The authors could substantially improve the paper by explaining one or more practical scenarios in which a researcher has both a model of causal effect and a non-RCT data set that has not been used to create that model.\n\nThe paper has occasional small grammar errors that detract from the authors’ message. An example is the first sentence of the contributions (with corrections noted in brackets): “We focus on causal model evaluation with non-RCT[s], propose a novel method for low-variance estimation of causal error (Equation 1), and demonstrate its effective[ness] over current approaches by achieving near-RCT performance.” A little more care in editing would improve the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. Below we address your question respectively.\n\n> Some diagnostic tests are in order to increase confidence in the output of the pairs estimator. For example, you could introduce noise into the model’s estimates and see if the estimated error increases.\n\nThank you for your suggestion, we will append failure scenarios of our method and try to introduce certain practical diagnoses. \n\n\n> The results in Figures 2 and 3 are very good. Indeed, they are freakishly good. They are so good that it makes me wonder whether the experiments reported in these figures are really evaluating anything important about how the pairs estimator works in practice. This bears some discussion in the description of the results.\n\nWhile the results in Figures 2 and 3 are indeed very good, they are based on the fact that the assumptions of our method are met or expected to be met. These results mainly validate the correctness of our theoretical results and demonstrate the practicality of our assumptions. In the revision, we will provide further discussion including potential failure cases and scenarios where the assumptions may not hold.\n\n> The introduction to the paper may be confusing to many readers. When first encountering the term “non-random experiment”, many readers will balk, thinking that randomization is the sine qua non of experimentation and seeing “non-random experiment” as a contradiction in terms. The authors could save readers this confusion by introducing the example of explicit non-random assignment (line 38) earlier or by moving the first paragraph of Section 2 (Related Works) to early in the introduction.\n\nThank you so much for these suggestions. We recognize the naming conventions and may leads to confusion, which we originally followed [1].  In revision, we will apply your suggestions and mention different naming conventions in the literature. \n \n> What are examples of practical  scenarios in which researchers have a model of treatment effect and then they collect data non-RCT data to evaluate that model? The authors could substantially improve the paper by explaining one or more practical scenarios in which a researcher has both a model of causal effect and a non-RCT data set that has not been used to create that model.\n\nSuch scenarios can be found in various applications. For example, in personalized medicine domain, researchers build complex models such as structural causal models (SCMs) to predict various types of causal quantities beyond ATE. In practice, performing trials on every single types of causal queries is expensive and ethically challenging, making the collection of non-RCT data a valuable downstream task to gain insights into the model's performance.\n\n> The paper has occasional small grammar errors that detract from the authors’ message. An example is the first sentence of the contributions (with corrections noted in brackets): “We focus on causal model evaluation with non-RCT[s], propose a novel method for low-variance estimation of causal error (Equation 1), and demonstrate its effective[ness] over current approaches by achieving near-RCT performance.” A little more care in editing would improve the paper.\n\nThank you for your input. In revision, we will further polish and improve the paper.\n\n\n**References**\n\n[1] Rubin, Donald B. \"Estimating causal effects of treatments in randomized and nonrandomized studies.\" Journal of educational Psychology 66.5 (1974): 688.\n"}]

Author rebuttal:
We thank the reviewers for your encouraging review and valuable suggestions to improve. We acknowledge that the reviewers highlighted the **effectiveness and simplicity of our approach (PdWH, r45Q,  t8vT, yvJa), novelty or soundness of our results (PdWH, zDTG,  t8vT, yvJa), extensiveness of experiments (r45Q, r45Q,  t8vT), and clarity of presentation (PdWH, r45Q, zDTG,  t8vT, yvJa)**. 

- An essential aspect of our work that we would like to emphasize is its focus on **model evaluation in real-world testing scenarios**. Our novel method for low-variance estimation of causal error facilitates reliable evaluation of causal inference models through **oracle conditional/non-randomized trial**. This is particularly relevant in situations where the oracle design cannot be improved due to real-world constraints or interests. Evaluating causal models in these settings is crucial to ensure their validity and applicability across various domains. Our approach is simple yet powerful, complementing existing tools in the causal inference toolbox, effectively addressing specific challenges faced by practitioners in a wide range of real-world applications. 

- Moreover, we have **addressed the main concerns raised by the reviewers**, including: 

    - as requested, providing **experiments to validate Assumption A**. In the attached pdf file, using an numerical example we have demonstrated that Assumption A holds in practice for various commonly used models, including linear DML, kernel DML, causal random forest, and variations of doubly robust algorithms (linear, forest, orthogonal). **This indicates that Assumption A is not overly restrictive and can be satisfied by a wide range of causal models that are popular in practical applications**. 

    - as requested, providing **theoretical proofs** to the main result,  which can be found in our response to **r45Q**.  

We believe that our responses effectively address the concerns raised by the reviewers and hope that they find our work more compelling and valuable. We are grateful for the reviewers' insightful feedback and look forward to incorporating their suggestions to further improve the quality of our paper.




Meta-review: The paper presents a novel estimator for evaluating causal models in the context of non-randomized experiments. This approach aims to reduce the variance in causal error estimation. The authors provide theoretical justifications for the method and conduct empirical experiments to demonstrate its effectiveness. There were some questions and concerns raised by reviewers including the lack of discussion about failure modes or the practical scenarios where the required model of treatment effect and non-randomized data for evaluation are both available. The paper would benefit from addressing these comments.
Decision: Accept (poster)