SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we will evaluate the performance of the proposed pairs estimator, and validate our theoretical insights via simulation studies. We will also examine the robustness and sensitivity of the pairs estimator concerning different scenarios of conditionally randomized trials, including treatment assignment mechanisms, degree of imbalance, choice of causal machine learning models, etc.
5.1 Synthetic csuite dataset with hypothetical causal model
Setting. Following Geffner et al. [2022], we construct a set of synthetic datasets, designed specifically for evaluating causal inference performance, namely the csuite datasets. The data-generating process is based on structural causal models (SCMs), different levels of confounding, heterogeneity, and noise types are incorporated, by varying the strength, direction, and parameterize of the causal effects. We evaluated on three different datasets, namely csuite_1, csuite_2, and csuite_3, each with a different SCM. See Appendix B for more details. The corresponding causal model estimation is simulated using a special form of Assumption A, that is:
$$
Y _ {\mathcal {M}} ^ {T = t} (i) = Y ^ {T = t} (i) + Y ^ {T = t} (i) * \nu_ {i}, i = 1, \dots , N, t \in \{0, 1 \}
$$
where  $\nu_{i}$  are i.i.d. distributed zero-mean random variables with variance  $\sigma_{\nu}^{2}$  that affects the ground truth causal error. To simulate the conditionally randomized trials, we use two different schemes to generate the treatment assignment plans  $\mathcal{T}$ . The first scheme is based on a logistic regression model of the treatment assignment probability given the covariates, that is,
$$
p _ {e x p} (T = 1 | X) = \frac {1}{1 + \exp (- \beta^ {T} X)}
$$
, where  $\beta$  is a random vector sampled from multivariate Gaussian distributions with mean zero and variance  $\sigma_{\beta}^{2}$ . We vary the degree of imbalance in the treatment assignment by changing the value of  $\sigma_{\beta}^{2}$ . A larger  $\sigma_{\beta}^{2}$  implies a more imbalanced treatment assignment, as the variance of the treatment assignment probability increases. The second scheme is based on a random subsampling of the units, where the treatment assignment probability is fixed for each unit, but different units are sampled with replacement to form different treatment assignment plans. We vary the number of treatment assignment plans by changing the sample size of each subsample.
Evaluation method. We compare the performance of the pairs estimator with the naive estimator (Equation 5), as well as the RCT estimator (Equation 4), which is the ideal benchmark for causal model evaluation. We also compare with two other baselines, by replacing the IPW component  $\hat{\delta}^{IPW}(\mathcal{T})$  in the naive estimator (Equation 5) by its variance reduction variants. This includes the self-normalized estimator, as well as the linearly modified (LM) IPW estimator Zhou and Jia [2021], a state-of-the-art method for IPW variance reduction when the propensity score is known. We measure the performance of the estimators by the following metrics: the variance, the bias, and the MSE of the causal error estimation. See Appendix B.1 for detailed definitions. We compute these metrics by averaging over 100 different realizations of the treatment assignment plans for each dataset.
Results. The results are shown in Figure 2 and Figure 3. Figure 2 shows the results for the logistic regression scheme, with different values of  $\sigma_{\nu}^{2}$  and  $\sigma_{\beta}^{2}$ . Figure 3 shows the results for the random subsampling scheme, with different  $\sigma_{\nu}^{2}$ . In both tables, we report the average and the standard deviation of the performance metrics of the estimators for each value of the true causal error, which is computed by the difference between the true treatment effect and the model treatment effect.
Figure 2: Comparison of various causal error estimators for the logistic assignment scheme across (csuite_1, csuite_2, and csuite_3). In this  $3 \times 3$  plot, each row corresponds to a specific performance metric (Variance/Bias/MSE), while each column represents different datasets. In each plot, the x-axis represents the degree of treatment assignment imbalance  $(\sigma_{\beta}^{2})$ , while the y-axis displays the performance metrics (lower the better). The different colors indicate the performance of different estimators, and different linestyles and markers indicate different  $\sigma_{\nu}^{2}$  settings. The results demonstrate that the proposed estimator (purple) consistently outperforms the other estimators in terms of all performance metrics, with a more robust behavior as the imbalance in treatment assignment increases.
From the tables, we can see that the variance of the naive estimator quickly increases when the treatment assignment is highly imbalanced. Nevertheless, our estimator (purple) consistently outperforms the naive estimator and its variance reduction variants in all metrics (variance/bias/MSE) regardless of the value of  $(\sigma_{\nu}^{2})$ , and the degree of imbalance  $(\sigma_{\beta}^{2})$ . The pairs estimator also achieves comparable performance to the RCT estimator, the golden standard by design. The linearly-modified estimators and self-normalized estimators have a lower variance than the naive estimator, but it also introduces some bias, and their performance is sensitive to the degree of imbalance. These demonstrate the effectiveness and robustness of the proposed pairs estimator when all assumptions are met.
Figure 3: Comparative analysis of multiple causal error estimators in the context of the random subsampling scheme for (csuite_1, csuite_2, and csuite_3). In each plot, the x-axis represents the variance of the noise variable  $(\sigma_{\nu}^{2})$ , and the y-axis illustrates the performance metrics (Variance/Bi-as/MSE) for each estimator. Different colors denote the performance of different estimators. The findings reveal that, under the random subsampling scheme, the proposed estimator (purple) also consistently surpasses other estimators in every performance metric.
5.2 Synthetic counterfactual dataset with popular causal inference models
Settings. Based on Section 5.1, we now consider more realistic experimental settings, in which we apply a wide range of machine learning-based causal inference methods by training them from synthetic observational data. Thus, we expect the assumptions in Section 4.2 might not strictly hold anymore, which can be used to test the robustness of our method. We include a wide range of methods, such as linear double machine learning[Chernozhukov et al., 2018] (dubbed as DML Linear), kernel DML (DML Kernel) [Nie and Wager, 2021], causal random forest (Causal Forest) [Wager and Athey, 2018, Athey et al., 2019], linear doubly robust learning(DR Linear), forest doubly robust learning (DR Forest), orthogonal forest learning (Ortho Forest) [Oprescu et al., 2019], and doubly robust orthogonal forest (DR Ortho Forest). All methods are implemented via EconML package [Battocchi et al., 2019]. See Appendix B for more details.
We mainly focus on two aspects: 1), whether the proposed estimator can still be effective on variance reduction with non-hypothetical models as in Section 5.1; 2), whether the learned causal models' counterfactual predictions approximately follow the postulated Assumption A. Here We present the results for the first aspect, and results for the second can be found in Section 5.3.
Simulation procedure. We repeat the same simulation procedure as in Section 5.1, but using the learned causal inference models instead of the simulated causal model. We compare the performance of the pairs estimator with the same baselines as in Section 5.1, using the same metrics and the same treatment assignment schemes. For DML Linear, DML Kernel, Causal Forest, and Ortho Forest (that does not require propensity scores), we train on 2000 observational data points generated via the following data generating process of single continuous treatment [Battocchi et al., 2019]:
$$
W \sim \operatorname {N o r m a l} (0, I ^ {n _ {w}}), X \sim \operatorname {U n i f o r m} (0, 1) ^ {n _ {x}}
$$
$$
T = \langle W, \beta \rangle + \eta , \quad Y = T \cdot \theta (X) + \langle W, \gamma \rangle + \epsilon ,
$$
where  $T$  is the treatment,  $W$  is the confounder,  $X$  is the control variable,  $Y$  is the effect variable, and  $\eta$  and  $\epsilon$  are some uniform distributed noise. We choose the dimensionality of  $X$  and  $W$  to be  $n_x = 30$  and  $n_w = 30$ , respectively. For other doubly robust-based methods, we use a discrete treatment that is sampled from a binary distribution  $P(T = 1) = \mathrm{sigmoid}(\langle W,\beta \rangle +\eta)$ , while keeping the others unchanged. Once models are trained on the generated observational datasets, we use the
Figure 4: Performance of different causal error estimators across different causal inference methods. Each row of the grid corresponds to a specific performance metric (Variance/Bias/MSE), while each column represents different levels of treatment assignment imbalance  $(\sigma_{\beta}^{2})$ . In each plot, the x-axis represents different causal models. The y-axis displays performance metrics. Different colors correspond to different estimators. The results highlight the differences in performance among the estimators, with the pairs estimator (purple) consistently outperforming others in most scenarios, emphasizing its robustness and effectiveness in assessing various causal models.
trained causal inference model to estimate the potential outcomes and the treatment effects for each unit. Then, we use the logistic regression-based treatment assignment scheme as in Section 5.1 to simulate a hypothetical conditionally randomized experiment (for both continuous treatment and binary treatment, we will assign  $T = 1$  for the treatment group and  $T = 0$  for the control group). Both our pairs estimator as well as other baselines presented in Section 5.1 will be used to estimate the causal error. This is repeated 100 times across 3 different settings for treatment assignment imbalance  $(\sigma_{\beta}^{2} = 1,5,10)$ .
Results. Figure  $4^2$  shows that using our estimator with conditionally randomized data, we achieved near-RCT performance for causal model evaluation quality across all metrics. It is also more robust to different settings of  $\sigma_{\beta}^{2}$ , where no significant variance change is observed. Also, we note that each causal error estimator has a different sweet-spot: for instance, the naive estimator usually consistently works very well with Causal Forest, whereas our estimator has relatively high variance and bias for DML Kernel1. Nevertheless, the proposed estimator is still the most robust estimator (apart from RCT) that consistently achieves the best results across all causal models. This shows the feasibility of our method for reliable model evaluation with conditionally randomized experiments.
5.3 Validation of Assumption A in Section 4.2
Settings Finally, we provide experiments to validate Assumption A in Section 4.2. Specifically, we use a numerical example to check if Assumption A holds in practice for various commonly used models, including the ones considered in Section 5.2, including DML Linear, DML kernel, Causal Forest, and variations of doubly robust algorithms (DR Linear, DR forest, DR Ortho Forest). We first fit different methods using the observational data generated in Section 5.2, and obtain the counterfactual predictions from each method; Then, we parameterize the function  $V(\cdot)$  in Equation 7 as a polynomial function and fit for the counterfactual predictions using Equation 7.
2The linearly-modified baseline is not displayed for clarity reasons, see Appendix B, Figure 6 for full results.
Finally, we solve for the residuals  $\hat{\nu}$  in Equation 7, and test whether they are independent from  $Y^T$  and  $b$ , as postulated in Assumption A.
Correlation Coefficients and p-values for the residuals  $\hat{\nu}$  of different methods
Figure 5: Validation of whether the learned causal models' counterfactual predictions approximately follow the postulated Assumption A.
Results As shown in Figure 5, we calculate the Pearson correlation coefficient between  $\hat{\nu}$  and  $Y^{T}$  and  $b$ , respectively, as shown in the upper plot; we also compute the p-values for the null hypothesis that the samples are independent. We observe that the samples have close zero correlations as well as high p-values that are unable to reject the null assumption. Hence, both results show suggests that the independence multiplicative noise assumption is likely to be true under this setting. We have also tried other tests such as Hoeffding's independence tests that captures non-linear dependencies, which yields similar results. This indicates that Assumption A is not overly restrictive and can be satisfied by a wide range of causal models that are popular in practical applications.
In this paper, we proposed the pairs estimator, a novel methodology for low-variance estimation of causal error in conditionally randomized trials. This approach applies the same IPW method to both the model and ground truth effects, canceling out the variance due to IPW. Remarkably, the pairs estimator can achieve near-RCT performance using conditionally randomized experiments, signifying a novel contribution for enabling more reliable and accessible model evaluation, without depending on expensive or infeasible randomized experiments. Future work may extend our method to more complex scenarios, explore alternative ways to reduce causal error estimation variance, and apply our method to other causality applications such as policy evaluation, causal discovery, and counterfactual analysis.
We thank Colleen Tyler, Maria Defante, and Lisa Parks for conversations on real-world use cases that inspired this work. We also thank Nick Pawloski, Wenbo Gong, Joe Jennings, Agrin Hilmkil and Meyer Scutbon for useful discussions and support that enables this work.
Joshua D Angrist, Guido W Imbens, and Donald B Rubin. Identification of causal effects using instrumental variables. Journal of the American statistical Association, 91(434):444-455, 1996.
Kangjie Zhou and Jinzhu Jia. Variance reduction for causal inference. arXiv preprint arXiv:2109.05150, 2021.