SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Different incarnations of the Bayesian framework correspond to different choices of the prior  $(\Pi_0, \Pi_1)$ . Bayesian Additive Regression Trees (BART) [34] is one popular model for Bayesian nonparametric regression that was later adapted for causal effect estimation and inference [8, 35]. While [8] applies BART for estimation of CATE and inference of ITE out-of-the-box, [35] introduces new regularization techniques to incorporate different forms of inductive biases on the CATE target parameter. Another
popular choice of the prior  $(\Pi_0,\Pi_1)$  is based on Gaussian processes where inductive biases are incorporated through different choices of the kernel function of a reproducible kernel Hilbert space [9].
Unlike the conformal framework, Bayesian methods do not provide coverage guarantees on their credible intervals. Frequentist coverage can be achieved in an asymptotic sense by Bayesian method under certain technical conditions [36]. Existing methods typically do not provide any finite-sample guarantees and the achieved empirical coverage depends on the choice of the prior. Models such as BART are typically applied with a default prior for all datasets, undermining their achieved coverage in diverse experimental setup. In [1], it was shown that Bayesian methods have a tendency to undercover ITEs in some data generation processes (e.g., in high-dimensional covariate spaces).
Bayesian methods also suffer from two key advantages compared to the conformal framework. First, Bayesian inference procedures and the computation of posterior distributions are typically tailored to specific model architectures. Generalizing exact Bayesian inference to arbitrary model architectures, including modern deep learning architectures, is not straightforward. Moreover, Bayesian inference is conducted either using computationally exhaustive Monte Carlo method (e.g., BART) or through expensive evaluation of an analytical posterior distribution that does not scale well with the number of training points (e.g., Gaussian process posteriors are  $\mathcal{O}(n^3)$ ).
(2) Frequentist methods. Another model-specific approach to inference on causal parameters is the Causal Forest model in [7]. This procedure uses constructs pointwise confidence intervals on the CATE function  $\tau(x)$  using conditional variance estimates based on the infinitesimal jackknife [37]. These intervals provide asymptotically valid coverage of CATE under mild regularity assumptions, but they provide no (asymptotic or finite-sample) guarantees on coverage for ITEs.
Another non-Bayesian approach for predictive inference of ITEs uses a quantile regression approach to train models that construct upper and lower bounds on the observed potential outcomes using a pinball loss [38]. While quantile regression can be repurposed for different model architectures, this approach does not provide finite-sample coverage guarantees and is empirically found to undercover outcomes, hence it is typically supplemented with a conformal prediction procedure [25].
(3) Conformal methods. The application of conformal prediction to ITE inference traces its origins back to the covariate shift problem. In [26], a variant of conformal prediction was proposed to address problem setups where training and testing data are drawn from different distributions, breaking the exchangeability assumptions required for valid inference.
In the treatment effect estimation setup, covariate shift arises because the distributions of the treatment groups differ from that of the target population. Recall from (8) that the conformity scores for the nuisance estimates  $\widehat{\mu}_0$  and  $\widehat{\mu}_1$  can be written as follows:
$$
V _ {k} ^ {(0)} (\widehat {\mu} _ {0}) \triangleq V (X _ {k}, Y _ {k} (0); \widehat {\mu} _ {0}), \forall k \in \mathcal {D} _ {c, 0}, \quad V _ {k} ^ {(1)} (\widehat {\mu} _ {1}) \triangleq V (X _ {k}, Y _ {k} (1); \widehat {\mu} _ {1}), \forall k \in \mathcal {D} _ {c, 1},
$$
Both sets of conformity scores are sampled from the populations  $P_{X|W = 0}$  and  $P_{X|W = 1}$ , respectively, and hence they are not exchangeable with the conformity scores sampled from target covariate distribution  $P_{X}$ . To retain the validity of intervals constructed using the sample quantiles of conformity scores, [26] introduces the notion of "weighted exchangeability" and proposes a weighted sample quantile with weights based on the likelihood ratios between training and testing distributions. [1] uses this method to construct valid intervals for potential outcomes and then proposes different approaches for combining these intervals to construct intervals for ITEs. Other work utilized the conformal prediction framework to conduct sensitivity analysis by devising hypothesis tests for the presence of hidden confounders [39, 40]. In [41, 42], conformal prediction has been utilized to address the closely related problem of off-policy evaluation. More recently, there has been further work developing methods for conformal prediction under covariate shift [43, 44, 45].
The key difference between our work and the aforementioned approaches is that we do not apply conformal prediction to the nuisance parameters, and instead conduct inference directly on the target parameter. We do so by applying conformalization with respect to transformed targets  $\widetilde{Y}_{\varphi}$  for each data point in  $\mathcal{D}$ , and this way the covariate shift problem does not arise since  $(X,\widetilde{Y}_{\varphi})$  is sampled from the same distribution in calibration and testing data. One way to think about our approach as opposed to weighted conformal prediction (WCP) is that ours applies re-weighting to the outcomes whereas WCP re-weights the conformity scores. By moving the re-weighting step to the outcomes, we decouple the conformal procedure from the model, allowing for more flexible choice of inductive priors and accommodating models that do not estimate potential outcomes directly.
Appendix C: Experimental Setup and Results
C.1. Overview of the experimental setup
In all experiments, we used a Gradient Boosting model with 100 trees as the base model for nuisance estimation and quantile regression on pseudo-outcomes. We used the same base model in conformal meta-learners as well as the weighted CP (WCP) baselines to control for potential performance differences resulting from different choices of base models. Unless otherwise stated, all experiments followed a  $90\% / 10\%$  train/test split of each dataset, and each training set with further split into a  $75\% / 25\%$  proper training/calibration sets. All performance metrics (empirical coverage, expected interval width and RMSE) are evaluated on the testing data and averaged over 100 runs. The target coverage in all experiments was set  $1 - \alpha = 0.9$ .
We used the official implementations of the (naïve, exact and inexact) WCP baselines in the R package available at https://github.com/lihualei71/cfcausal. We executed these baselines through rpy2 (https://rpy2.github.io/) wrappers within our Python codebase.
C.2. Details of the IHDP and NLSM datasets
In this Section, we provide details of the semi-synthetic datasets used in the experiments in Section 5.
IHDP dataset. The Infant Health and Development Program (IHDP) was a multi-site randomized controlled trial that studied the effect of early educational intervention on the cognitive development of low birth weight premature infants [46]. The trial targeted low-birth-weight, premature infants, and administered an intensive high-quality child care and home visits from a trained provider to the treatment group. In [8], a semi-synthetic benchmark was developed based on the covariate data in IHDP with simulated outcomes to assess the performance of different CATE estimation and ITE inference models when ground-truth counterfactual are known. The dataset comprised 25 covariates (6 continuous and 19 binary variables) capturing aspects related to children and their mothers. The benchmark dataset excluded a non-random subset of treated individuals [8]. The final dataset consists of 747 samples (139 treated and 608 control). The outcomes of all individuals were simulated whereas the treatment assignments followed the true assignments in the study.
The simulated outcomes model in IHDP followed simulation Setup B in [8]. In this Setup, the potential outcomes were simulated as follows:  $Y(0) \sim \mathcal{N}(\exp((X + W)\beta), 1)$  and  $Y(1) \sim \mathcal{N}(\exp(X\beta) - \omega, 1)$ , where  $W$  is an offset matrix,  $\omega$  is set so that the ATE on the treated is always equal to 4, and the coefficients  $\beta$  are sampled independently at random from  $(0, 0.1, 0.2, 0.3, 0.4)$  with probabilities  $(0.6, 0.1, 0.1, 0.1, 0.1)$ . In our experiments, we used the 100 realization of the training and testing data released by [6] in https://www.fredjo.com/files/ihdp_npci_1-100.train.npz and https://www.fredjo.com/files/ihdp_npci_1-100.test.npz.
NLSM dataset. The National Study of Learning Mindsets (NLSM) is a large-scale randomized trial that studied the effect of a behavioral intervention, designed to instill students with a growth mindset, on the academic performance of students in secondary education in the US [3]. In [33], a semisynthetic benchmark was developed based on the covariates of the NLSM study. Unlike IHDP, this benchmark does not use the real covariates of NLSM but rather buids a synthetic process to emulate NSLM in terms of the covariate distributions, data structures, and effect sizes. The final dataset comprised 10,000 data points. The data was generated based on 5 principles (See Section 2 and Appendix A in [33]). The principles are: (1) ATEs should be well-estimated from synthetic data by any reasonable procedure, (2) variability in CATE should be relatively modest, (3) treatment effect heterogeneity can be approximately recovered given complete knowledge of the correct functional form, (4) no additional unmeasured treatment effect moderation at the individual level, and (5) unexplained treatment effect heterogeneity at the group level should be present at reasonable levels.
C.3. Further experiments using the ACIC dataset
In addition to the IHDP, NLSM and fully-synthetic experiments, we also conducted further experiments on 77 datasets from the 2016 Atlantic Causal Inference Competition (ACIC2016) [47]. We followed the processing steps in [16], creating a dataset of 4,802 data points and 55 covariates. Out of this base dataset, we created 770 simulation setups using the 77 settings proposed in [47]. The 77 settings represent different levels of complexity of response surfaces, varying degrees of confounding,
Figure C.1: Performance of all baseline across the 77 ACIC experiments.
overlap and effect heterogeneity. All settings use the same covariates but simulate different treatment assignments and response surfaces. We created 10 realizations of each of the 77 settings and average the performance of all baselines across the 10 runs.
The empirical coverage, average interval width and RMSE results across all 77 experiments are provided in Figure C.1. The comprehensive experiments in Figure C.1. align with the overall conclusions of the experiments in the main paper. For a target coverage of  $1 - \alpha = 0.9$ , all models achieved the target coverage with the exception of the X-learner. All models achieving the target coverage displayed conservative predictive intervals, achieving coverage levels that exceed the 0.9 target. In terms of efficiency, the DR-learner and WCP-Inexact achieved comparable interval widths, significantly outperforming all other valid procedures (IPW-learner, WCP-exact and WCP-Naive). Finally, the DR-learner outperformed all valid procedure in terms of RMSE in almost all experiments, providing point estimation accuracy that is on par with the X-learner.
These results align with the overall conclusions of the experiments in the main paper. The DR-learner generally inherits the accurate point estimation of its underlying CATE meta-learner, provides a coverage guarantee for practically relevant values of target coverage and achieves this coverage with competitive efficiency making it a favorable approach for both point estimation of CATE and predictive inference of ITE.