SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chao Ma
chao.ma@microsoft.com
Cheng Zhang
cheng.zhang@microsoft.com
The gold standard for causal model evaluation involves comparing model predictions with true effects estimated from randomized controlled trials (RCT). However, RCTs are not always feasible or ethical to perform. In contrast, conditionally randomized experiments based on inverse probability weighting (IPW) offer a more realistic approach but may suffer from high estimation variance. To tackle this challenge and enhance causal model evaluation in real-world conditional randomization settings, we introduce a novel low-variance estimator for causal error, dubbed as the pairs estimator. By applying the same IPW estimator to both the model and true experimental effects, our estimator effectively cancels out the variance due to IPW and achieves a smaller asymptotic variance. Empirical studies demonstrate the improved of our estimator, highlighting its potential on achieving near-RCT performance. Our method offers a simple yet powerful solution to evaluate causal inference models in conditional randomization settings without complicated modification of the IPW estimator itself, paving the way for more robust and reliable model assessments.
Experimental approaches for causal model evaluation Causal inference models aim to estimate the causal effects of treatments or interventions on outcomes of interest, given observational (sometimes experimental) data. A crucial step in developing and validating such models is to evaluate their prediction quality, that is, how well they can approximate the true treatment effects that would have been observed under different scenarios. A common approach for causal model evaluation is to launching a new randomized controlled trial (RCT) separately, which provides a reliable estimate of the true treatment effects by randomly assigning units to treatments. By comparing the RCT estimate with the prediction of the causal model, one can assess the causal error (Equation 1), a key metric to measure how well the model reflects the true effects.
Nevertheless, RCTs are not always accessible or preferable, as random manipulation of treatments might be impractical, ethically questionable, or excessively expensive. For instance, it may be impossible or inappropriate to randomly assign individuals to different lifestyles, environmental exposures, or social policies, as these may involve personal choices, preferences, or constraints that affect the willingness or ability to participate in the experiment. In such cases, causal machine learning researchers may resort to or augment their analyses with conditionally randomized designs, in which the allocation of participants into treatment and control groups is not purely random, but is instead based on a predetermined condition or factor.
Conditionally randomized experiments can vary in the level and unit of treatment assignment, such as individual, group, or setting. They also depend on different assumptions and methods to account for the potential confounding, selection, or measurement biases that arise from the lack of randomization.
A common scenario is the non-random quantitative assignment paradigm [West et al., 2008], where a new set of treatment groups are selected given some quantitative covariates  $X$ , via an oracle model  $P_{exp}(T|X)$ , where  $T$  is the treatment variable. For example, in sales strategy assignment, it is typical that resources are allocated towards products with higher revenue contributions, which introduces explicit confounding. A widely used method to estimate the true treatment effect in this setting is the inverse probability weighting (IPW) estimator [Rosenbaum and Rubin, 1983, Rosenbaum, 1987], which uses the propensity scores, the probability of receiving the treatment given the covariates, to weight the observed outcomes and adjust for the imbalance between the treatment groups. This IPW estimate can then be compared with the causal model prediction to evaluate the causal error.
Limitations of current approaches Despite the popularity of the IPW estimator for conditionally randomized experiments, it suffers from several limitations that affect its reliability for causal model evaluation. [Khan and Tamer, 2010] shows that IPW may have unbounded variance when the propensity scores are imbalanced. Moreover, the it may have poor finite sample performance and high sensitivity to the specification of the propensity score model [Busso et al., 2014]. Last but not the least, a surprising finding is that even the oracle IPW may be harmful for the estimation efficiency [Hahn, 1998, Hirano et al., 2003]. As a consequence, the causal error estimation based on the IPW estimator may not be accurate or robust, and may lead to misleading conclusions about the causal model quality. Existing work on addressing the variance issue of the IPW estimator often involves modifying the IPW estimator itself [Crump et al., 2009, Chaudhuri and Hill, 2014, Sasaki and Ura, 2022, Busso et al., 2014, Robins et al., 2007, Lunceford and Davidian, 2004, Imbens, 2004, Liao and Rohde, 2022]. However, these methods are mainly designed for treatment effect estimation rather than causal error estimation, and may introduce additional bias or complexity that may not be optimal for the purpose of causal model evaluation.
Contributions overview We focus on causal model evaluation with conditionally randomized trials, propose a novel method for low-variance estimation of causal error (Equation 1), and demonstrate its effectiveness over current approaches by achieving near-RCT performance. Our key insight is: to estimate the causal error, we can design a simple and effective low-variance estimation procedure without improving the IPW estimator for the true treatment effect. As shown in Figure 1, denote the ground truth effect as  $\delta$  and the treatment effect of a causal model  $\mathcal{M}$  as  $\delta_{\mathcal{M}}$ . Our goal to estimate the causal error  $\hat{\Delta} \coloneqq \hat{\delta}_{\mathcal{M}} - \hat{\delta}^{IPW}$  with low variance. Using conditionally randomized experiments, we have an estimator of the ground truth via IPW:  $\hat{\delta}^{IPW}$ . Instead of improving the IPW estimator, we replace the commonly used sample mean estimator  $\hat{\delta}\mathcal{M}$  with its IPW counterpart,  $\hat{\delta}_{\mathcal{M}}^{IPW}$ , having the same realizations of treatment assignments as in  $\hat{\delta}^{IPW}$ . This allows the variance of  $\hat{\delta}^{IPW}$  to be hedged by  $\hat{\delta}^{IPW}\mathcal{M}$ , reducing the variance of the causal error estimator  $\hat{\Delta}$ . Contrary to conventional estimation strategies, the pairs estimator, as we call it, effectively reduces the variance of the causal error estimation and provides more reliable evaluations of causal model quality, both theoretically and empirically.
The rest of this paper is structured as follows. In Section 3, we formally describe our problem setting, necessary background and notations. In Section 4, we will formally define the pairs estimator for causal model evaluation, and study its theoretical properties (Proposition 1) on variance reduction. Finally, in Section 5, we will conduct simulation studies and demonstrate the effectiveness of the proposed estimation approach, highlighting its potential on achieving near-RCT performance using only conditionally randomized experimental data.
Alternative schemes to randomized controlled trials There exists a number of different conditionally randomized trial schemes in the literature, for instance randomized encouragement designs [Angrist et al., 1996], non-random quantitative treatment assignment [Imbens and Rubin, 2015], and fully observational studies [Rubin, 2005]. In this work, we mainly consider the non-random quantitative treatment assignment setting, where the treatment groups are determined by some known function of covariates. Nevertheless, we have an emphasized focus on using conditionally randomized trials for evaluating the quality of any given causal inference models (trained on observational data), which is a fundamental setting for many practical causal model deployment procedure.
Figure 1: Overview of the proposed method.  $\hat{\Delta}$ : the naive estimator for causal error evaluation.  $\hat{\Delta}^{Pairs}$ : the proposed estimator. Given a causal model  $\mathcal{M}$  learned from observational data, our goal is to estimate its causal error (Equation 1) via conditionally randomized experiments (yellow). Naive method (red) would achieve this by comparing  $\hat{\delta}^{IPW}$ , the IPW estimation of ground truth effect from the experiment, with  $\hat{\delta}_{\mathcal{M}}$ , the predicted treatment effect from  $\mathcal{M}$ , usually obtained via population mean (Equation 2). In our method (brown), we replace  $\hat{\delta}_{\mathcal{M}}$  with its IPW counterpart,  $\hat{\delta}_{\mathcal{M}}^{IPW}$ , using the same treatment assignments used in estimating  $\hat{\delta}^{IPW}$ , so that the variance from IPW are hedged.
Causal estimation methods with conditionally randomized experiments Apart from IPW mentioned in Section 1, a number of other methods were also developed for conditionally randomized trials settings, include matching [Ho et al., 2007, Stuart, 2010, Rubin, 2006], regression adjustment [Neyman, 1923, Rubin, 1974, Lin, 2013, Wager et al., 2016], double robust methods [Robins et al., 1995, Bang and Robins, 2005, Kang and Schafer, 2007, Athey et al., 2018], and machine learning methods [Athey and Imbens, 2016, Wager and Athey, 2018, Shi et al., 2019, Chernozhukov et al., 2018]. Readers can refer to [Imbens and Rubin, 2015, Morgan and Winship, 2015] for a comprehensive review. This paper, on the contrary, aims to propose a general method for causal model evaluation applicable to any causal effect estimation method, particularly the widely used IPW estimator.
Variance reduction and robustness There has been a long standing discussion on robustness improvement [Robins et al., 1995] and variance reduction of IPW estimators. Several techniques have been proposed to address the problem of extreme or uninformative weights, including weights trimming [Crump et al., 2009, Chaudhuri and Hill, 2014, Sasaki and Ura, 2022], weights normalization [Busso et al., 2014, Robins et al., 2007, Lunceford and Davidian, 2004, Imbens, 2004], and more recently, linearly-modified IPW [Zhou and Jia, 2021], etc. These techniques aim to reduce variance but may introduce bias, complexity, or tuning parameters. Our work differs from the existing literature in that, we focus on the causal error estimation, rather than the individual treatment effect estimation. We do not improve the IPW estimator for the true treatment effect, but rather propose to apply the same IPW estimator to both the model and the true effects to hedge their variances.
Consider the data generating distribution for the population on which the experiment is being carried over is given by  $p(X, T, Y)$ , where  $X$  are some (multivariate) covariates,  $Y$  is the outcome variable and  $T$  is the treatment variable. In this paper, we only consider continuous effect outcomes. Let  $Y^{T = t}$  denote the potential outcome of the effect variable under the intervention  $T = t$ . Without loss of generality, we assume  $T \in \{0, 1\}$ . Then, the interventional means are given by  $\mu^1 = \mathbb{E}[Y^{T=1}]$  and  $\mu^0 = \mathbb{E}[Y^{T=0}]$ , respectively. The ground truth treatment effect is then given by
$$
\delta = \mu^ {1} - \mu^ {0} = \mathbb {E} [ Y ^ {T = 1} ] - \mathbb {E} [ Y ^ {T = 0} ].
$$
Now, assume that given observational data sampled from  $p(X,T,Y)$ , we have trained a causal model, denoted by  $\mathcal{M}$ , whose treatment effect is given by
$$
\delta_ {\mathcal {M}} = \mu_ {\mathcal {M}} ^ {1} - \mu_ {\mathcal {M}} ^ {0} = \mathbb {E} [ Y _ {\mathcal {M}} ^ {T = 1} ] - \mathbb {E} [ Y _ {\mathcal {M}} ^ {T = 0} ].
$$
Our goal is then to estimate the causal error of the model, which quantifies how well the model reflects the true effects (the closer to zero, the better):
$$
\Delta (\mathcal {M}) := \delta_ {\mathcal {M}} - \delta \tag {1}
$$
In practice,  $\delta_{\mathcal{M}}$  will be the model output, and can be estimated easily. For example, we can sample a pool of i.i.d. subjects  $\mathcal{D} = (X_1,Y_1),\ldots ,(X_N,Y_N)\sim p(X,Y)$ , and the corresponding treatment effect estimation will be given as
$$
\delta_ {\mathcal {M}} \approx \hat {\delta} _ {\mathcal {M}} = \frac {1}{N} \sum_ {i \in \mathcal {D}} \left[ Y _ {\mathcal {M}} ^ {T = 1} (i) - Y _ {\mathcal {M}} ^ {T = 0} (i) \right], \tag {2}
$$
which forms the basics of many casual inference methodologies, both for potential-outcome approaches and structural causal model approaches [Rubin, 1974, Rosenbaum and Rubin, 1983, Rubin, 2005, Pearl et al., 2000]. On the contrary, obtaining the ground truth effect  $\delta$  is usually not possible without real-world experiments/interventions, due to the fundamental problem of causal inference [Imbens and Rubin, 2015]. By definition,  $\delta$  can be (hypothetically) approximated by the population mean of potential outcomes:
$$
\delta \approx \hat {\delta} := \frac {1}{N} \sum_ {i \in \mathcal {D}} \left[ Y ^ {T = 1} (i) - Y ^ {T = 0} (i) \right] \tag {3}
$$
However, given a subject  $i$ , only one version of the potential outcomes can be observed. Therefore, we often resort to the experimental approach, the randomized controlled trial (RCT). The RCT approach is always considered as the golden standard for treatment effect estimation, in which we would randomly assign treatments to our pool of subjects  $\mathcal{D} = (X_1, Y_1), \ldots, (X_N, Y_N) \sim p(X, Y)$  by flipping an unbiased coin. Then the estimated treatment effect is given by:
$$
\hat {\delta} ^ {R C T} = \frac {1}{| \mathcal {B} |} \sum_ {i \in \mathcal {B}} Y ^ {T = 1} (i) - \frac {1}{N - | \mathcal {B} |} \sum_ {j \in \mathcal {D} \backslash \mathcal {B}} Y ^ {T = 0} (j)
$$
where  $\mathcal{B}$  denotes the subset of patients that are assigned with the treatment. Together, we have the RCT estimator of the causal error:
$$
\hat {\Delta} ^ {R C T} (\mathcal {M}) := \hat {\delta} _ {\mathcal {M}} - \hat {\delta} ^ {R C T} \tag {4}
$$
However, when a randomized trial is not available, we can only deploy a conditionally randomized test assignment plan, represented by  $\mathcal{T}$ , which is a vector of  $n$  Bernoulli random variables  $\mathcal{T} = [b_1,\dots,b_N]$  each determines that  $Y_{n}^{T = 1}$  will be revealed with probability  $p_n$ . In practice,  $\mathcal{T}$  can be given by an treatment assignment model  $p_{exp}(T = 1|X)$ . This is an oracle distribution that is known and designed by the experiment designer. A subset of patients  $\mathcal{B}\in \mathcal{D}$  is selected given these assignment probabilities. Then, the inverse probability weighted (IPW) estimation of the treatment effect is given by
$$
\hat {\delta} ^ {I P W} (\mathcal {T}) := \frac {1}{N} <   \mathbf {Y} ^ {T = 1} (\mathcal {B}), \mathbf {w} (\mathcal {B}) > - \frac {1}{N} <   \mathbf {Y} ^ {T = 0} (\mathcal {D} \setminus \mathcal {B}), \frac {\mathbf {w} (\mathcal {D} \setminus \mathcal {B})}{\mathbf {w} (\mathcal {D} \setminus \mathcal {B}) - 1} >,
$$
where  $\mathbf{w} = [1 / p_1,\dots,1 / p_N]$ ,  $\mathbf{w}(\mathcal{B})$  is created by sub-slicing  $\mathbf{w}$  with subject indices in  $\mathcal{B}$ .  $<,>$  is the inner product, and  $\mathbf{1}$  is a vector of ones. Finally, the model causal error can be estimated as (dubbed naive estimator in this paper):
$$
\hat {\Delta} (\mathcal {M}, \mathcal {T}) := \hat {\delta} _ {\mathcal {M}} - \hat {\delta} ^ {I P W} (\mathcal {T}) \tag {5}
$$
In practice, when the size  $N$  of the subject pool is relatively small, the IPW estimated treatment effect  $\hat{\delta}^{IPW}(\mathcal{T})$  will have high variance especially when  $p_{exp}(T = 1|X)$  is skewed. As a result, one will expect a very high or even unbounded variance in the estimation [Khan and Tamer, 2010, Busso et al., 2014]  $\hat{\Delta} (\mathcal{M},\mathcal{T})$ . The goal of this paper is then to improve model quality estimation strategy  $\hat{\Delta} (\mathcal{M},\mathcal{T})$ , such that it has lower variance and error rates under conditionally randomized trials.
4 Pairs estimator for causal model quality evaluation
4.1 The pairs estimator
To resolve the problems with the naive estimator for causal error in Section 3, in this section, we propose a novel yet simple estimator that will significantly improve the quality of the causal error estimation in a model-agonist way. Intuitively, when estimating  $\hat{\Delta} (\mathcal{M},\mathcal{T})$ , we can simply apply the
same IPW estimator (with the same treatment assignment) for both the model treatment effect  $\delta_{\mathcal{M}}$  and the ground truth treatment effect  $\delta$ . In this way, we anticipate that the estimators for  $\delta_{\mathcal{M}}$  and  $\delta$  will become correlated; their estimation error will be canceled out and hence the overall variance is lowered. More formally, we have the following definition:
Definition 1 (Pairs estimator for causal model quality). Assume we have a pool of i.i.d. subjects to be tested, namely  $\mathcal{D} = (X_1,Y_1),\ldots,(X_N,Y_N)\sim p(X,Y)$ , as well as a conditionally randomized treatment assignment plan, represented by  $\mathcal{T}$ , which is a vector of  $n$  Bernoulli random variables  $\mathcal{T} = [b_1,\dots,b_N]$ , each determines that  $Y_{n}^{T = 1}$  will be revealed with probability  $p_n$ . Assume that, for a particular trial, a subset of patients  $\mathcal{B}\in \mathcal{D}$  are selected using these probabilities. Then, the IPW estimator of the model's treatment effect and the ground truth treatment effect is given by
$$
\hat {\delta} _ {\mathcal {M}} ^ {I P W} (\mathcal {T}) := \frac {1}{N} <   \mathbf {Y} _ {\mathcal {M}} ^ {T = 1} (\mathcal {B}), \mathbf {w} (\mathcal {B}) > - \frac {1}{N} <   \mathbf {Y} _ {\mathcal {M}} ^ {T = 0} (\mathcal {D} \setminus \mathcal {B}), \frac {\mathbf {w} (\mathcal {D} \setminus \mathcal {B})}{\mathbf {w} (\mathcal {D} \setminus \mathcal {B}) - 1} >
$$
$$
\hat {\delta} ^ {I P W} (\mathcal {T}) := \frac {1}{N} <   \mathbf {Y} ^ {T = 1} (\mathcal {B}), \mathbf {w} (\mathcal {B}) > - \frac {1}{N} <   \mathbf {Y} ^ {T = 0} (\mathcal {D} \setminus \mathcal {B}), \frac {\mathbf {w} (\mathcal {D} \setminus \mathcal {B})}{\mathbf {w} (\mathcal {D} \setminus \mathcal {B}) - 1} >
$$
, respectively. Then, the pairs estimator of causal model quality is defined as
$$
\hat {\Delta} ^ {P a i r s} (\mathcal {M}, \mathcal {T}) := \hat {\delta} _ {\mathcal {M}} ^ {I P W} (\mathcal {T}) - \hat {\delta} ^ {I P W} (\mathcal {T}). \tag {6}
$$
We will formally show that this new estimator can effectively reduce estimation error. We will provide the full assumptions and proposition below. Note that we will assume by default that the common assumptions for conditionally randomized experiments will hold, such as non-interference/consistency/overlap, etc, even though we will not explicitly mention them for the rest of the paper.
4.2 Core assumptions for achieving variance reduction
Our main assumption is regarding the estimation error of causal models, stated as below:
Assumption A. Causal model estimation error for potential outcomes. We assume that for each subject  $i$ , the trained causal model's potential outcome estimation can be expressed as
$$
Y _ {\mathcal {M}} ^ {T = t} (i) = Y ^ {T = t} (i) + V _ {i} \left(Y ^ {T = t} (i)\right) * \nu_ {i}, i = 1, 2, \dots , N, t \in \{0, 1 \} \tag {7}
$$
where  $\nu_{i}$  are i.i.d. distributed random error variables with unknown variance  $\sigma_{\nu}^{2}$ , that is independent from  $Y_{i}^{T = t}$  and  $b_{i}$ ; and  $V_{i}(\cdot), i = 1,2,\dots,N$  is a set of deterministic function that is indexed by  $i$ . This assumption is very general and models the modulation effect between the independent noise  $\nu$  and the ground truth counterfactual. One special example would be  $Y_{\mathcal{M}}^{T = t}(i) = Y^{T = t}(i) + Y^{T = t}(i)*\nu_{i}$ , where the estimation error will increase (on average) as  $Y^{T = t}$  increases. In practice, dependencies between error magnitude and ground truth value could arise when the model is trained on observational data that suffers from selection bias, measurement error, omitted variable bias, etc. For the rest of paper, we write Equation 7 in its vectorized form:
$$
\mathbf {Y} _ {\mathcal {M}} ^ {T = t} = \mathbf {Y} ^ {T = t} + \mathbf {V} (\mathbf {Y} ^ {T = t}) * \boldsymbol {\nu},
$$
where all operations are point-wise. Finally, we further require that the causal model's counterfactual prediction should be somewhat reasonable, in the sense that  $\sigma_{\nu}^{2}\mathbb{E}[(V_{i}Y^{T = t}(i))^{2}] < \mathbb{E}[Y^{T = t}(i)^{2}], t = 1,0; i = 1,2,\dots,N$ . This implies that the variance of the estimation error should at least be smaller than the ground truth counterfactual.
Assumption B. Asymptotic normality of mean estimators of potential outcome variables. Let  $\overline{Y^{T = 1}}, \overline{Y^{T = 0}}, \overline{Y_{\mathcal{M}}^{T = 1}}, \overline{Y_{\mathcal{M}}^{T = 0}}$  be the corresponding mean estimator of the potential outcome variables  $Y^{T = 1}, Y^{T = 0}, Y_{\mathcal{M}}^{T = 1}, Y_{\mathcal{M}}^{T = 0}$ . We assume that these estimators are jointly asymptotic normal, i.e.,
$$
\sqrt {N} \big (\overline {{Y ^ {T = 1}}} - \mathbb {E} Y ^ {T = 1}, \overline {{Y ^ {T = 0}}} - \mathbb {E} Y ^ {T = 0}, \overline {{Y _ {\mathcal {M}} ^ {T = 1}}} - \mathbb {E} Y _ {\mathcal {M}} ^ {T = 1}, \overline {{Y _ {\mathcal {M}} ^ {T = 0}}} - \mathbb {E} Y _ {\mathcal {M}} ^ {T = 0} \big)
$$
converge in distribution to a zero mean multivariate Gaussian. This is reasonable due to the randomization and the large samples used in experiments [Casella and Berger, 2021, Deng et al., 2018].
<sup>1</sup>Note that this particular assumption is mainly for achieving asymptotic normality of the estimators, which is orthogonal to achieving variance reduction effect of our pairs estimator, that relies on Assumption A.
4.3 Theoretical results
Following our assumptions, we can derive the following theoretical result, which shows that given our assumptions described in the previous section, the pairs estimator  $\hat{\Delta}^{Pairs}(\mathcal{M},\mathcal{T})$  will effectively reduce estimation variance compared to the naive estimator,  $\hat{\Delta} (\mathcal{M},\mathcal{T})$ .
Proposition 1 (Variance reduction effect of the pairs estimator). With the assumptions stated in Section 4.2, we can show that our IPW estimators,  $\hat{\delta}^{IPW}(\mathcal{T})$  and  $\hat{\delta}_{\mathcal{M}}^{IPW}(\mathcal{T})$  can be decomposed as
$$
\hat {\delta} ^ {I P W} (\mathcal {T}) = \hat {\delta} + f (\mathcal {B}), \quad \hat {\delta} _ {\mathcal {M}} ^ {I P W} (\mathcal {T}) = \hat {\delta} _ {\mathcal {M}} + f (\mathcal {B}) + g (\boldsymbol {\nu}, \mathcal {B}),
$$
where  $f$  and  $g$  are random variables that depend on  $\mathcal{B}$  (or also  $\pmb{\nu}$ ), and  $g(\pmb{\nu},\mathcal{B})$  is orthogonal to  $\hat{\delta}$ ,  $\hat{\delta}_{\mathcal{M}}$  and  $f(\mathcal{B})$ . Furthermore, if we define the estimation error of model quality estimators as follows
$$
e \left(\hat {\Delta} ^ {P a i r s} (\mathcal {M}, \mathcal {T})\right) := \hat {\Delta} ^ {P a i r s} (\mathcal {M}, \mathcal {T}) - \Delta (\mathcal {M})
$$
$$
e (\hat {\Delta} (\mathcal {M}, \mathcal {T})) := \hat {\Delta} (\mathcal {M}, \mathcal {T}) - \Delta (\mathcal {M}),
$$
then both  $\sqrt{N} e(\hat{\Delta}^{Pairs}(\mathcal{M},\mathcal{T}))$  and  $\sqrt{N} e(\hat{\Delta} (\mathcal{M},\mathcal{T}))$  are asymptotically normal with zero means, and their variances satisfies
$$
\mathbb {V} a r \left[ e \left(\hat {\Delta} ^ {P _ {a i r s}} (\mathcal {M}, \mathcal {T})\right) \right] <   \mathbb {V} a r \left[ e \left(\hat {\Delta} (\mathcal {M}, \mathcal {T})\right) \right].
$$
Proof: See Appendix A
This result provides theoretical justifications that our simple estimator will be effective for variance reduction. In the next section, we will conduct simulation studies to empirically evaluate and validate the effectiveness proposed approach.
5 Simulation Studies