Conformal Meta-learners for Predictive Inference of Individual Treatment Effects
Authors: Ahmed Alaa
Zaid Ahmad
Mark van der Laan
Venue: NeurIPS 2023 oral (2023)
Keywords: Heterogeneous treatment effects
conformal prediction

Abstract: We investigate the problem of machine learning-based (ML) predictive inference on individual treatment effects (ITEs). Previous work has focused primarily on developing ML-based “meta-learners” that can provide point estimates of the conditional average treatment effect (CATE)—these are model-agnostic approaches for combining intermediate nuisance estimates to produce estimates of CATE. In this paper, we develop conformal meta-learners, a general framework for issuing predictive intervals for ITEs by applying the standard conformal prediction (CP) procedure on top of CATE meta-learners. We focus on a broad class of meta-learners based on two-stage pseudo-outcome regression and develop a stochastic ordering framework to study their validity. We show that inference with conformal meta-learners is marginally valid if their (pseudo-outcome) conformity scores stochastically dominate “oracle” conformity scores evaluated on the unobserved ITEs. Additionally, we prove that commonly used CATE meta-learners, such as the doubly-robust learner, satisfy a model- and distribution-free stochastic (or convex) dominance condition, making their conformal inferences valid for practically-relevant levels of target coverage. Whereas existing procedures conduct inference on nuisance parameters (i.e., potential outcomes) via weighted CP, conformal meta-learners enable direct inference on the target parameter (ITE). Numerical experiments show that conformal meta-learners provide valid intervals with competitive efficiency while retaining the favorable point estimation properties of CATE meta-learners.

Conformal Meta-learners for Predictive Inference of Individual Treatment Effects
Ahmed M. Alaa UC Berkeley and UCSF amalaa@berkeley.. edu
Zaid Ahmad
Mark van der Laan UC Berkeley laan@stat. berkele..eeeeee............. edu
UC Berkeley
zaidahmad@@erkelleyycoo
Abstract
We investigate the problem of machine learning-based (ML) predictive inference individual treatment effects (ITEs). Previous work has focused primarily on devel- oping ML-based Imeanerearners" that can provide point estimates of the conditional average treatment effect (CATE) these are model- aggnosicc approaches for com- bining intermediate nuisance estimates to produce estimates of CATE. In this paper, we develop conformal meta-learners, a general framework for issuing predictive intervals for ITEs by applying the standard conformal prediction (CP) procedure top of CATE meta-learners. We focus on a broad class of meta-learners based on two-stage pseudo-outcome regression and develop stochastic ordering framework to study their validity. We show that inference with conformal meta-learners is marginally valid if their (pseudo-outcome) conformity scores stochastically domi- nate "oracle" conformity scores evaluated on the unobserved ITES. Additionally, we prove that commonly used CATE meta-learners, such as the doubly-robus learner, satisfy a model- and distribution -feeeefeeeeee stochastic (or convex) dominance condition, making their conformal inferences valid for practically-reled releevann levels of target cov- erage. Whereas existing procedures conduct inference nuisance parameters (i.e., potential outcomes) via weighted CP [1], conformal meta-learners enable direct ference on the target parameter (ITE). Numerical experiments show that conformal meta-learners provide valid intervals with competitive efficiency while retaining the favorable point estimation properties of CATE meta-learners.
Code: https//github..omm/ laal ab/conformal cattrnoormmmmmmmalaareeee
Introduction
Identifying heterogeneity in the effects of interventions across individuals is central problem in various fields, including medical, political, and social sciences [2, 3, 4]. In recent years, there has been growing interest in developing machine learning (ML) models to estimate heterogeneous treatment effects using observational or experimental data [5, 6, 7, 8, 9]. However, most of these models only provide point estimates of the conditional average treatment effect (CATE), which is deterministic function that describes the expected treatment effect based on given individual's covariates. In this paper, we focus on quantifying uncertainty in these estimates, which arises from both errors in the model and the variation of individual treatment effects (ITEs) for individuals with the same covariates. We adopt a predictive inference approach to this problem, with the goal of devising valid procedures to issue predictive intervals that cover ITEs on unseen data with a predetermined probability.
Traditionally, predictive inference on ITEs has been conducted through Bayesian methods such as BART [8] and Gaussian processes [9]. These methods can provide interval-valued predictions of ITEs through their induced posterior distributions (e.g., posterior credible intervals). However, Bayesian methods tend to be model-specifii and cannot be straightforwardly generalized to modern ML models, e.g., transformer-based architectures used to model visual and textual covariate spaces [10]. More importantly, Bayesian methods generally do not provide guarantees on the frequentist coverage of
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

their credible intervals-achievell (finite- (finite-sample sample) coverage depends on the prior [11]. This paper is motivated by the advent of conformal prediction (CP), a frequentist altemaatve that can be used to conduct model-agnostic, distribution-fiee valid predictive inference on top of any MI. model [12, 13, 14]. Throughout this paper, we will study the validity of CP-based procedures for inference of ITEs.
What makes CP-based inference of ITEs different from its application to the standard regression (supervised) setup? The "fundamental problem of causal inference" is that we never observe counter- factual outcomes [15]. That is, our "label" is the ITE which is a difference between two potential outcomes (treated and untreated) for an individual subject- this label is never observed for any given subject because we only ever observe factual outcomes. This poses two challenges [16]:
(1) How to handle covariate shift? When treatments assigned to individuals with probabilities that depend on their covariates, then the distributions of covariates in treated and untreated groups differ. Consequently, the distribution of training data differs from that of the target population.
(2) How to incorporate inductive biases? Unlike supervised learning wherein we fit a single function using examples of covariates and observed targets, models of treatment effects cannot be directly fit to the unobserved effects. Thus, estimates of treatment effects comprise intermediate estimates of nui- sance parameters. Different approaches for combining nuisance estimates entail different inductive priors on the potential outcomes that affect the accuracy of the resulting ITE estimates.
The literature on ML-based CATE estimation focuses on addressing the two questions above. Covari- ate shift affects the generalization performance of MI. models models-existingg CATE estimation models address this problem using importance weighting [17] or balanced representation learning methods for unsupervised domain adaptation [6. 18, 19]. [5], the notion of "meta-learners" was coined to de- scribe various model-agnostic approaches to incorporating inductive biases and combining nuisance estimates. In [5. 20J, it was shown that the choice of the meta-learner influences the CATE estimation rates. While the impact of (1) and (2) on the generalization performance of CATE estimators has been extensively investigated, their impact on the validity and efficiency of predictive inference methods This forms the central focus of our paper.
for ITE is less
Contributions. We propose CP procedure for predictive inference of ITEs that jointly addresses (1) and (2) in an end-to-end fashion. Our proposed inference strategy applies the standard CP procedure on top of broad class of CATE meta-learners based on two-stage pseudo-outcome regression. These meta-learners operate by first estimating pseudo-outcomes, i.e., transformed targets that depend on observable variables only, and then regressing the pseudo-outcomes on covariates to obtain point estimates of CATE. We then construct intervals for ITEs by computing the empirical quantile of conformity scores evaluated on pseudo-outcomes in a held-out calibration set. Conformal meta- learners address (I) because the distribution of covariates associated with pseudo-outcomes is the same for training and testing data, and they address (2) since the calibration step is decoupled from model architecture, enabling flexible choice of inductive priors and the possibility of re-purposing existing meta-lcarners and architectures that have been shown to provide accurate estimates of CATE.
Conformal meta- meta-learners inherit the guarantees of CP, i.e., their resulting intervals cover pseudo- outcomes on test data with high probability. However, the original CP guarantees do not immediately translate to guarantees on coverage of ITES. To this end, we develop a unified stochastic ordering framework to study the validity of conformal meta-learners for inference on ITEs. We show that in- ference with conformal meta-learners is valid if their conformity scores satisfy certain stochastic ordering conditions with respect to "oracle" conformity scores evaluated on unobserved ITEs. We prove that some of the commonly used meta-learners, such as the doubly-robust learner [20], satisfy a weaker stochastic (or convex) dominance condition which makes them valid for relevant levels of target coverage. Our numerical experiments show that, with careful choice of the pseudo-outcome transformation, conformal meta- letanlesrners inherit both the coverage properties of CP as well as the efficiency and point estimation accuracy of their underlying CATE meta-learners.
2 Predictive Inference of Individual Treatment Effects (TTEs)
2.1 Problem setup
We consider the standard potential outcomes (PO) framework with a binary treatment ([21, 22]). Let W E (0,1) be the treatment indicator, X E X' be the covariates, and Y be the outcome of inter- est. For each subject i, let (Y)(0), (x)) be the pair of potential outcomes under W 0 and W 1,
2

---

respectively. The fundamental problem of causal inference is that we can only observe the factual outcome, i.e. the outcome Y, (1Y+(11++ (1 (0) determined by W,, but we cannot observe the counterfactual (1 W.). For n subjects, we assume that the data generation process
Rid ~ P(X, W, (0), Y(1),, i i= 1111..... n, X, satisfies the following assumptions: (1) Unconfoundedness: (Y (Y))),(())) 1 | (2) Consistency:
x W (O)), Y(O))
Y = Y (W), and (3) Positivity: 0 < P(W = x = x) < 1, Vz E X. These assumptions are necessary for identifying the causal effects of the treatment from dataset {2, - (X,, W, Y=))==1- The causal effect of the treatment on individual i, known as the individual treatment effect (ITE), is defined as the difference between the two potential outcomes. i.e., (1 Y()
Previous modeling efforts (e.g.. [5, 6, 7, 7J) have focused primarily on the (deterministic) conditional average treatment effect (CATE), i.e., r(x) _ (1) Y(0) X = xr. In this paper, we focus on the (random) ITE as the inferential target of interest. That is, goal is to infer the ITE for given subject j given their covariate X, and the observed sample {Z, = (X,, W,, Y=)
The distribution of the observed variable Z = (X, W, Y) is indexed by the covariate distribution Px as well as the nuisance functions (Ir),, (40 (a) and fii (1) defined as follows:
X ===x)) x), Mn (x) = B|y = x, W = w), w 10,
(x) P(W
(2)
The function (x)) is known as the propensity score and it captures the treatment mechanism underly- ing the data generation process. Throughout this paper, we assume that π (z) is known, i.c., data is drawn from an experimental study or the treatment assignment mechanism is known.
Predictive Inference of ITEs. Given the sample {Z, (X,, Wr, Y.) our goal is to infer the ITE for new individual n + 1 with covariate Xn+1 In particular, we would like to construct predictive band (()) that covers the true ITE for new test points with high probability, i.c.,
P(Y,+1(1) (1) (0) E C(Xx+1)) 2 1 a,
for predetermined target coverage of - cx, with a E (0, 1), where the probability in (3) accounts for the randomness of the training data { and the test point (Xn+1, (1) - (0)) Predictive intervals that satisfy the coverage condition in (3) are said to be marginally valid.
2.2 Conformal prediction
Conformal prediction (CP) is model- and distribution- free framework for predictive inference that provides (finite-sampl) sante-)smmpl) marginal coverage guarantees. In what follows, we describe a variant of CP. known as split (or inductive) CP [12. 13, 14], for the standard regression setup. Given a training dataset D [(x,, s, the CP procedure starts by splitting DD into two disjoint subsets: proper training set [(X,, j. = j. E }. and a calibration set {(Xx, Y&) h E De). Then, an MI. model (ixx) is fit
using the samples in Dr and a conformity score V() is evaluated for all samples in D. as follows:
(T) A V(X,, M); Vk E D..
The conformity score measures how unusual the prediction looks relative to previous examples. A common choice of V.) is the absolute residual, i.e., V y; īi) & [7xx) y J. For target coverage level of a, we then compute a quantile of the empirical distribution conformity scores, i.e..
(v - a.) _ (1 - (1 + 1//P.l) quantile of V()).
(5)
where V(ū) (V)( A E D.). Finally, the predictive interval at new point X = is
Ō(x) [mxx) Qv - a), ((x) a)]
(6)
1-a. The interval in (6) is guaranteed to satisfy marginal coverage, i.e., P(Y,+1 E ((X++1)) 21 The only assumption needed for this condition to hold is the exchangeabiliry between calibration and test data [12, 23, 24J. Note that the interval in (6) has a fixed length of 2Qv(1-q) a) that is independent of x. To enable adaptive intervals, [25] proposed variant of the CP procedure where the base model is quantile regression with interaa-valuue predictions (Po/2 (x), Pu /)-a//(x)), and the conformity score is defined as the signed distance Vi (f) Maxx(aa/2(22)) (Xx) Yk, Yv (1-a/2(Xx))
3

---

2.3 Oracle conformal prediction of ITES
How can we adapt the CP framework for predictive inference of ITEs? In a hypothetical world where we have access to counterfactual outcomes, we can apply the standard CP in Section 2.2 to a dataset of covariates and ITE tuples, D = {(X,, (1) (())) and compute conformity scores as:
V (7) - R (1) (0) :7), Vk E D
where is an ML model fit to estimate the CATE function T T(x) using D;, and D = DÇUD: We will refer to this procedure "oracle" conformal prediction and to (F) as the oracle conformity scores. Since the oracle problem is a standard regression, the oracle procedure is marginally valid-ie..
satisfies the guarantee in (3), (1Y - (0) E (x)) 2 1 1-14 a. However, oracle CP is infeasible since we can only observe of the potential outcomes (colored in red and blue in (7)), hence need an alternative procedure that operates only on the observed variable Z = (X, W, W)
2.4 The two challenges of predictive inference on ITES
A naive approach to inference of ITEs is to split the observed sample {2, (X,, W,, Y)), by the treat-
ment group and create two datasets: Da {(X,, Y)) : W, = h. D2 [(X,, Yi): W; = h. then generate two sets of conformity scores for the nuisance estimates Po and アi as follows:
V (1) (Po) A V(XA, (=) Ak E De,
(7:) A V(X M+(1): Ak E Da,1,
(8)
where De.0 and De,1 are calibration subsets of Do and DJ In order to construct valid predictive inter- vals for ITE using the conformity scores in (8), we need reconsider how the two distinct character- istics of CATE estimation, previously discussed in Section 1, interact with the CP procedure:
(I) Covariate shift
(2) Inductive biases
The distributions of covariates for treated and un- treated subjects differ from that of the target pop- ulation: Pu * Px|w=1 * Px, i.e. the follow-
Different choices for the joint model of Nn and μi encode different inductive biases that impose different forms of regularization
the implied
ing holds for the conformity scores in (8):
CATE function, i.e., f(x) Ai (x) (x( [5]. These biases influence the induced distributions Pycc and Pyc of the conformity scores in (8).
Px Px,v() # Px.v(0) B.vo)/== # Px.v
Covariate shift breaks the exchangeability assumption necessary for the validity of CP. Current meth- ods have primarily focused on (1) with Y (0) and (1) as inference targets, and developed approaches for handling covariate shift by reweighting conformity scores [1, 26]. The resulting intervals for POs are then combined to produce intervals for ITEs. However, these method tie the CP procedure to model architecture, requiring inference nuisance parameters, and hence lose the desirable post-hoc nature of CP. Furthermore, inference on POs is likely to provide conservative ITE intervals, and limits the inductive priors that can be assumed since not all CATE models provide explicit PO estimates.
3 Conformal Meta-learners
In [5], a taxonomy of "meta-learners" was introduced to categorize different inductive priors that can be incorporated into CATE estimators by structuring the regression models for μlI and μ1, For example, the T-learner estimates Ãu and f independently using D, and D1, while the S-learner models the treatment variable W
additional covariate in joint regression model p(X. W) and estimates CATE as f(z) = p(z, 1) f(x 0). In this Section, we propose an end-to-end solution to (1) and (2) by applying CP top of CATE meta-learners in post-hoe fashion, thereby decoupling the CP procedure from the CATE model and allowing direct inference on ITEs. In the next Section, we develop a unified framework for analyzing the validity of this broad class of procedures.
3.1 Pseudo-outcome regression for CATE estimation
We focus on broad subclass of CATE meta-learners based on two-stage pseudo-outcome regression. These models replace the (unobserved) oracle ITES with "proximal" targets that are estimated from ob- served variables only, and then train MI. model to predict the estimated targets from covariates. The two stages of this general pseudo-outcome regression framework can be described follows:

---

(8) Pseudo-outtcomee regression for CATE
(b) Conformal pseudo-intevvels for ITE
Algorithm 1: Conformal Meta-learner
Po
Conformity scores
Input ((X,, W Yill Xn. Output: (X...
X
Nuisence estimator
Pv-
Split D into D.. D. and De;
Qv (1 oo)
2 Pseudo-outcome regression:
Y
Po,
8 (1) Estimate ş = (a. uai ūi) using Pei
Oracle
Pv-
4 (2) Regress Y, f(2, @) on X using D;;
CATE estimator
Conformal pseudo-intervals:
X
Ety.
Evaluate conformity scores v. using Dzi
Qy- (1 o)
Return C₂2 [f(X.+1) Q1 aj]
Figure I: Pictorial depiction of conformal meta-learners,
We obtain plug-in estimate ç of the nuisance parameters &p (TT, (o, f1). Note that since we assume that the propensity score is known, we only need to estimate μi and μu using Do and Dj.
Stage 1.
We use the nuisance estimates obtained in Stage I to create pseudo-outcomes Y, that depend only on @ and the observable variables Z = (X, W. Y), i.e., Y, = f(Z, for some function f. The CATE estimate then obtained by regressing the pseudo-outcome Y, on the covariate X. This is typ- ically conducted using a different dataset than the one used to obtain the nuisance estimate ç.
Stage 2.
The general framework de- scribed above captures vari- ous models previous liter- ature. We study 3 examples of such meta-learners: X-
Pseudo-outcome
IPW-learner [27] Y, W W-+(X)) Y =(X)(1=((X))
X-learner [5] Vę W(Y - -xo(X)) - (1- (1))) -M VY
learner, Inverse propensity weighted (IPW) learner and
DR-leamer [20] Y, =(X](1-x(X)) (Y -x)) (X) (X)
W-n(X)
doubly-robust (DR) learner. Table 1: Existing meta- meta-learners instantiations of pscudo-outcoms. regression.
Table lists the pseudo-outcomes Yú for the three meta- learners: IPW-leamner reweights factual out- comes using propensity scores match CATE, i.c., Bi,, X = x] r(x); X-Iearner uses regression adjustment to impute counterfactuals; DR-learner combines both approaches. DR- and X-learners' coupled with specific architectures for joint modeling of Çao and fú, have shown competitive perfor- mance for CATE estimation in previous studies [5, 16, 20]. The conformal meta-learner framework decouples the CP procedure (Section 3.2) from the inductive priors encoded by these meta-learners, hence it inherits their favorable CATE estimation properties and enables potentially more efficient direct inference ITES as opposed to inference on POs. This addresses challenge (2) in Section 2.4.
3.2 Conformal pseudo-intervals for ITES
Pseudo-outcome regression is based on the notion that accurate proxies for treatment effects can pro- duce reliable CATE point estimates. This concept can be extended to predictive inference: using CP to calibrate meta-learners via held-out pseudo-outcomes can yield accurate "pseudo-intervals" for ITEs.
Given dataset = [Z. (X,, W., Y.) we create three mutuullyy-xxclustt subsets: Da, De and De. D, is used to estimate the nuisance parameters yp Next, the estimates & = (T, Fo. Ai are used to trans- form {Z, = (X, W,, Yi) E D,) into covariate/seaudooouutoomm pairs ((X,, Y.,i) :): i E D.} which are used to train CATE model F. Finally, we compute conformity scores for f on pseudo-outcomes, i.e.,
(7) V(X,, ?), Vk E De.
For a target coverage of 1 we construct predictive interval at a new point Xn+1 as follows:
(,(2) = [f(x) (1- F(x) + Qv (1-)),
(10)
where V.-- = (V). : E D.). We call Cx((x) a pseudo-interval. The conformal meta-leamer ap- proach is depicted in Figure 1 and summary of the procedure is given in Algorithm 1.
Here, we consider special case of the learner in [5] which involves weighted sum of two regression adjusted models fo and fi trained separately the treated and control datasets Do and Di.
5

---

(a) First-order stochastic dominance F Eas G
(b) Second-order stochastic dominance F Zez G
G
F
9
Qa(a: Qr(a)
Qr(a") = Qa(a")
Figure 2: Graphical illustration of stochastic dominance among two exemplary distributions F and G.
Note that conditional on p. the pseudo-outcomes (X, Y₂) in calibration data are drawn from the target distribution, which maintains the exchangeability of conformity scores and addresses covariate shift (challenge (1) in Section 2.4). However, the conformity scores Vip are evaluated on transformed outcomes, which means that VP and V are not exchangeable, even though they are drawn from the same covariate distribution. Consequently, the usual CP guarantees, i.e., F(Y,, E Cx(X)) 2 a, do not immediately translate to coverage guarantees for the true ITE Y(1) Y()). In the next section, we show that for certain choices of the provide valid inferences for ITE without requiring the exchangeability of VvP and V*.
the corresponding pseudo-inted intervals can
Validity of Conformal Meta-learners: A Stochastic Ordering Framework
Under what conditions are pscudo-intervals valid for inference of ITEs? Recall that these intervals are constructed by evaluating the empirical quantile of naeudo- conformity scores. Intuitively, the iseudo-intervals will cover the true ITE if the conformity scores are "stochastically larger" than the oracle scores in Section 2.3, ie., Qv (a) 2 Qy-(a) in some stochastic sense (Figure 1(b)). Hence, study the validity of conformal meta-learners, identify conditions under which pseudo-intervale cover oracle intervals.
analyze the stochastic orders of V₂P and V*, and
Stochastic orders are partial orders of random variables used to compare their location, magnitude, or variability [28, 29]. In our analysis, we utilize two key notions of stochastic order among cumulative distribution functions (CDFs) F and G, which we formally define below.
Definition 1 (Stochastic dominance) F has first-order stochastic dominance (FOSD) on G, F F. G, iff F(z)= S G(z), Vx, with strict inequality for some æ. F has second-orde stochastic dominance (SOSD) over G. Zey G, iff J" [C(t) F(]] di 2 0, Vz, with striet inequality for some
Definition 2 (Convex dominance) F has monotone conver dominance (MCX) over G, F Emcx G, iff Ex-ep(u<Xe [x])] 2 Ex=c[u(X)] for all non-decreasing convex functions u R - I.
Stochastic ordering is useful tool in decision theory and quantitative finance used to analyze the deci- sions of utility maximizers with varying risk attitudes [30]. A distribution has FOSD over G if is favored by any decision-man maker with non-decreasing utility function, i.e., F is more likely to give higher outcomes than G because its CDF strictly lower (Figure 2(a)). i.e., f has smaller spread than g and is favored by all
has SOSD over G, then
it is favored by risk-averse decision-makee decision-maie with a non-decreasing concave utility function [31]. J. this case, the CDFs can cross but G is always lower after the last crossing point (Figure 2(b)). F has MCX over G if it is favored by decision-maken with a non-decreasing convex utility-iin this case, the CDFs can cross but always lower after the last crossing point (See Appendix A for detailed analysis).
In the following Theorem, we provide sufficient conditions for the validity of conformal meta-leamers in terms of the stochastic orders of their conformity scores.
Theorem 1. (X,, W,, Y,),), Y(()), i 1,...,, n + 1 are exchangeable, then 3a* (0, 1) such that the sveudo-interval C.p (Xx+1) constructed using the dataset 'D {(X,, W;, Y:)) satisfies
P(Yn+1 (1) (0) (xp(xn+))) 2 a, Va E (0,a");
the if at least one of the following stochastic ordering conditions hold: (i) V2 Ei V", (ii) V 312 V*, and (iii) Vi Emex V* Under condition (i), we have a* =
and
All proofs are provided in Appendix A. Theorem states that the conformity score V₂ of the meta- learner is stochastically larger (FOSD) or has a larger spread (SOSD and MCX) than the oracle con- formity score, then the conformal meta-learner is valid for high-probabilityy coverage (Figure 3). (This is the range of target coverage that is of practical relevance, i.., a is typically set to 0.05 or 0.1.)

---

Validity region
Meta-learner
Conformity score
conformal meta learners
Torget coverope
Absolute residual
Signed distance
CDF
X-learner
No stochastic order Vp Zmez V
No stochastic order
PV
Weleearner
Vp Zay V*
DR-learner
V Emer V
Ve Iz V
Pv.
Table 2: Stochastic orders of conformity scores for the three meta-learners considered in Table 1.
Conformity score
Figure 3: Validity condition in Theorem 1.
Because stochastic (or convex) dominance pertain to more variable conformity scores, the predictive intervals of conformal meta-leamers will naturally be more conservative than the oracle intervals.
Whether meta-learner meets conditions (i)-(ii)) of Theorem depends on how the gpeudo-outoome Y, ((: is constructed. The following Theorem provides an answer the question of which of the meta-learners listed in Table satisfy the stochastic ordering conditions in Theorem 1.
Let Vi((7) f(x) -Ypl and assume that the propensity score function : - [0. known. Then, the following holds: For the X X-learner, V₂ and V do not admit model- and distribution-fee stochastic order. (ii) For any distribution P(X, W, (0), (1)). CATE estimate F, and nuisance estimate G, the IPW- and the DR-learners satisfy V4 Emer V*.
Theorem 2.
Theorem states that the stochastic ordering of V, and V depends on the specific choice the con- formity score function V(X, Yg; well as the choice of the meta-learner, i.e., the pseudo-outcome generation function Y, f(Z: G). The IPW- and De-learners ensure that, by construction, the pseudo- outcome is equal to CATE in expectation: Y X x| 7xx) This construction enables the IPW- and DR-learners to provide unbiased estimates of average treatment effects (ATE) independent the data distribution and the ML model used for the nuisance estimates to and Çus. By the same logic, IPW- and DR-learners also guarantee stochastic (convex) dominance of their conformity scores irrespective of the data distribution and the ML model choice, hence preserving the model- and distribution-fie nature of the CP coverage guarantees. Contrarily, the X-learner does not use the knowledge of n to construct its pseudo-outcomes, hence it does not guarantee (distribution-fir free)) stochastic order and the achieved coverage depends on the nuisance estimates Ño and د In Table 2, we list the stochastic orders achieved for different choices of meta- leta-learners and conformity scores. (The analysis of stochas- tic orders for the signed distance score used in [25] and [32] is provided in Appendix A.)
Key limitations of conformal meta-learners. While conformalized meta-learners can enable valid end-to-end predictive inference of ITEs, they have two key limitations. First, the propensity score π must be known to guarantee model- and distribution craebbttoo-feeeeeeeeee stochastic ordering of conformity scores. However, we note that this limitation in not unique to our method and also encountered methods based on weighted CP [1, 26]. The second limitation is peculiar to method: exact characterization of a is difficult and depends on the data distribution. Devising procedures for inferring a based observable variables or deriving theoretical upper bounds future work. Here, we focus on empirical evaluation of a" in semi-synthetic experiments. A detailed comparison between our method and previous work is provided in Appendix B.
a" are interesting directions for
Experiments
5.1
Experimental setup
Since the true ITEs are never observed in real-world datasets, follow the common practice of conducting numerical experiments using synthetic and semi-synthetic datasets [1, 8, 19]. We present a number of representative experiments in this Section and defer further results to Appendix C.
Synthetic datasets. We consider a variant of the data- geta-gennration process Section 3.6 in [1] which was originally proposed in [7]. We create synthetic datasets by sampling covariates X ~ U 1]") and treatments W|X æ Bern (x:) with (()) = (1 + [2, 4))/4, where (2, 4) is the regularized incomplete beta function (i.e., CDF of Beta distribution with shape parameters 2 and 4). Outcomes
are modeled as (1(x) = {(x1) {(x2) and pa pa(x) = 7 ((z1) ) ((x2), where 7 E [0. 1]
parameter
that controls the treatment effect, and is function given by ((x) 1/(1 exp( (- 12(( 0.5))).
7

---

[a] Empirical assessment stochastic orders
[b) Coverage, efficiency and RMSE for Setup A (top) and Setup B (bottom)
DRlleamer
IPW WWW-lcarner
Xieerner
(Sctup A)
1.4
Inesact
WCP CM
0.5
V
Exact
0.5
Naive
9.0
25
(Seupp
1.4
+
+
0.3
IPW
0.0
DA
H0
Conformity score
Performance different levels Larget coverage
5.0
15
H
Empriical coverage
Aug, interol length
BMS
4
B
Setup
Naire
+ +--
X
18+
Setup
IPW
HHH
4
63 35
Target coMmrage
Empirical coverage (m:0.9)
Interval Length
RMIS
Figure 4: Performance of baseline in the synthetic setup described in Section 5.1. In (b), red vertical lines correspond to target coverage (1 - a = 0.9), and blue vertical lines correspond to optimal interval width. In (c), baseline methods are color-coded as follows: CM-DR, CM-IPW, CM-X, WCP-Naive, WCP-Exac and WCP-Inexact. Here, WCP stands for weighted CP and CM stands for conformal meta-learners
We assume that POs are sampled from Y (()) heteroscedastic noise model o² (x) log(z1). We define two setups within this model: Setup A where the treatment has not effect (C 1), and Setup where the effects are heterogeneous (C 0).
{o. 1} and consider
Semi-synthetic datasets. We also consider two well well-known semi-synthetic datasets that involve real covariates and simulated outcomes. The first the National Study of Learning Mindsets (NLSM) [3], and the second is the IHDP benchmark originally developed in [8]. Details on the data generation process for NLSM can be founded in Section 2 in [33]. Details the IHDP benchmark can be found in [6, 8, 16, 19J. Appendix C provides detailed description of both datasets for completeness.
Baselines. We consider baseline models that provide valid predictive intervals for ITEs. Specifically, we consider state-of-the-aae methods based on weighted conformal prediction (WCP) proposed [1]. These methods apply weighted CP to construct intervals for the two POs or plug-in estimates of ITEs. We consider the three variants of WCP [1]: (1) Naïve WCP which combines the PO intervals using Bonferroni correction, (2) Exact Nested WCP which applies WCP to plug-in estimates of ITEs in treatment and control groups followed by a secondary CP procedure, and (3) Inexact Nested WCP which follows the same steps of the exact version but replaces the secondary CP with conditional quantile regression. (Note that Inexact Nested WCP does not provide coverage guarantees.) For all baselines, we the model (Gradient Boosting) for nuisance and pseudo-outcome modeling, and we use the conformal quantile regression method in [25] to construct predictive intervals.
5.2 Results and discussion
Our experimental findings yield the following key takeaways: Firstly, the IPW- and DR-learners demonstrate a robust FOSD (i.e., a = 1) in the majority of experiments, surpassing the MCX condi- tions outlined in Theorem 2. Secondly, the DR-learner exhibits superior point estimation accuracy and interval efficiency in most experiment compared to all other baselines that ensure valid inference. Thirdly, the effectiveness of conformal meta-learners depends the discrepancy between the CDFs of conformity scores and oracle the resulting conformtiy scores cause conformal meta-learners to umder-perfone
transformations that induce thicker tails in
Empirical assessment stochastic orders. Figure 4(a) depicts the empirical CDF of the conformity scores Vg, and oracle scores V* for the three meta-learners under study (DR-, IPW- and X-learners). These CDFs are averaged over 100 runs of Setups and the synthetic generation process outlined in Section 5.1. (The shaded regions represent the lowest and highest bounds the empirical

---

IHDP
NLSM
DR-lcamer
IPW-leamer
Xileamer
CDF 10FINN CD [HDR)
2.0
Coverage
Avg. len.
RMSE
Coverage
Avg. len.
RMSE
05
Natve
0.89 (0.02)
(8.9
(1.00)
0.99 (0.00)
4,82 (8.020.02)
0.15 (0.00))
Fxact
0.99 (0.00)
(9.60)
(0.97)
0.99 (0.03)
4.92 (0.07)
0.19 (0.01)
0.0
0.61
(13())
4.61
(0.00)
(0.00)
0.65 (0:04)
(1,0(3.04)
(0.56
0.27 (0.03)
(0,000
0.14 (0.00)
0.99 (0.00)
112 (23.0)
19.9 (3/44)
(0.00)
(0.07)
0.61 (0.02)
DR
0.96 (0.01)
16.7 (3.30)
3.32 (0.53)
0.99 (0.00)
6.24 (0.07)
0.37 (0.01)
-2.9
Table 3: Performance of all baselines in semi-synthetic datasets. Bold numbers correspond to hest performance.
Conformity score
Figure 5: Stochastic orders in IHDP//LSM
CDFs evaluated across all runs.) In both setups, the conformity scores for the DR- and IPW-learners demonstrate FOSD over the oracle scores with respect to the average CDFs and in almost all realiza- tions. This aligns with the result of Theorem 2, and shows that the stochastic dominance condition achieved in practice even stronger than our theoretical guarantee since FOSD (V, žα V*) implies the weaker conditions of V,, Za V and Vi Emcr V On the contrary, the conformity scores of the X-learner are dominated by oracle scores in the FOSD sense. This is not surprising in light of Theo- rem 2, which indicates that X-learners do not guarantee a distribution-fii firee stochastic order. These observations are also replicated in the semi-synthetic datasets as shown in Figure 5.
Based on Theorem 1, the empirical stochastic orders observed in Figures 4(a) and 5 predict that the IPW- and DR-learners will cover ITEs, whereas the X-learner will not achieve coverage. This is con- firmed by the results in Figures 4(b), 4(c) and Table 3. The fact that the IPW- and DR-learners satisfy a stronger FOSD condition is promising because it indicates that the range of validity for these models spans all levels of coverage (a" = 1 in Figure 3). It also means that a stronger version of Theorem 2 outlining the conditions under which IPW- and DR-learners achieve FOSD could be possible.
Coverage, efficiency and point estimation accuracy. The performance of a predictive inference procedure can be characterized in terms of three metries: achieved coverage for true ITEs, expected length of predictive intervals, and root-mean-squee periments, we find that the DR-learner strikes balance between these metrics (See Appendix C for further experiments). In Figure 4(b), we can see that the DR- DR-learner outperforms the valid (naïve and exact) WCP procedures in terms of RMSE and interval length, while achiveing the target coverage of 90%. The X-learner outperforms all baselines in terms of RMSE, but as expected, it under-covers ITFss in all experiments. The inexact WCP baseline offers competitive efficiency and calibration, however, in addition to not offering coverage guarantees it also lacks consistency in RMSE performance under different inductive biases (i.e., no treatment effects in Setup A and heterogeneous effects in Setup B). These performance trends hold true across all levels of target coverage as shown in Figure 4(c).
error (RMSE) in CATE estimates. In most ex-
The semi-synthetic experiments on IHDP and NLSM datasets shed light on when meta-learners may perform poorly. The DR-leamer outperforms all baselines on the IHDP dataset in terms of RMSE, interval efficiency, while achieving the desired coverage of 90%. However, we observe that empirical performance depends on how closely the CDF of conformity scores matches the oracle CDF. The DR- learner performance deteriorates when conformity scores have "very strong" dominance over oracle scores, as observed in the NL.M dataset (Figure 5, bottom). Conversely, when the CDF of conformity scores is closer lower bound on the oracle CDF, the DR-learner performance is competitive (Figure and Figure 5, top). This is intuitive because if the pseudo-outcome transformation induces significant variability in regression targets, it will result in lower CDF, poorer accuracy of pseudo-outcome regression, and longer predictive intervals. This is why the DR-learner consistently outperforms the IPW-learner, as provides a closer approximation of the oracle CDF. Future work could focus on analyzing the gap between pseudo-outcome and oracle score CDFs and designing pseudo-outcome transformations that optimize efficiency while preserving stochastic orders.
6 Conclusions
Estimation and inference of treatment effects is challenging because causal effects are not directly observable. In this paper, we developed a general framework for inference of treatment effects, dubbed conformal meta-learners, that is compatible with any machine learning model. Our framework inherits the model- and distribution-fire validity of conformal prediction as well as the estimation accuracy of model-agnostic meta- meta-learners of treatment effects. Additionally, we introduce a new theoretical framework based on stochastic ordering to assess the validity of our method, which can guide the development of new models optimized for both accurate estimation and valid inference.

---

References
[1] Lihua Lei and Emmanuel J Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(5):911-938, 2021.
[2] Kosuke Imai and Aaron Strauss. Estimation of heterogeneous treatment effects from randomized experiments, with application the optimal planning of the get-out-the-vots campaign. Political Analysis, 19(1):1-19, 2011.
[3] David S Yeager, Paul Hanselman, Gregory M Walton, Jared S Murray, Robert Crosnoe, Chandra Muller, Elizabeth Tipton, Barbara Schneider, Chris S Hulleman, Cintia Hinojosa, et al. A national experiment reveals where a growth mindset improves achievement. Nature, 573(7774)364-369) 2019.
[4] Sheldon Greenfield, Richard Kravitz, Naihua Duan, and Sherrie H Kaplan. Heterogeneity of treatment effects: implications for guidelines, payment, and quality assessment. The American journal of medicine, 120(4):53-S9, 2007.
[5] Sören R Künzel, Jasject S Sckhon, Peter Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, 1 116110)4156-4465 2019.
[6] Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In International Conference on Machine Learning, pages 3076-3085. PMLR, 2017.
[7] Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228-1222 2018.
[8] Jennifer L Hill. Bayesian nonparametric modeling for causal inference. Journal of Computa- tional and Graphical Statistics, 20 (0(1):2172240 2011.
[9] Ahmed M Alaa and Mihaela Van Der Schaar. Bayesian inference of individualized treatment effects using multi-task gaussian processes. Advances neural information processing systems, 30, 2017.
[10] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-We Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.088 1908.03557, 2019.
[11) Botond Szabó, Aad W Van Der Vaart, and JH Van Zanten. Frequentist coverage of adaptive nonparametric bayesian credible sets. The Annals of Statistics, 2015.
[12] Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine-leamning applications of algorithmic randomness. International Conference on Machine Learning, 1999.
[13] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer Science & Business Media, 2005.
[14] Harris Papadopoulos. Inductive conformal prediction: Theory and application to neural networks. INTECH Open Access Publisher Rijeka, 2008
[15] Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical
sciences. Cambridge University Press, 2015.
[16] Alicia Curth and Mihaela van der Schaar. On inductive biases for heterogeneous treatment effect estimation. Advances in Neurul Information Processing Systems, 34: 15883-1584, 2021.
[17] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
[18) Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.
[19] Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual inference. In International conference on machine learning, pages 3020-3029. PMLR, 2016.
[20] Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. arXiv preprint arXiv:2004.14497, 2020.
10

---

[21] Jerzy S Neyman. On the application of probability theory to agricultural experiments. essay on principles. section 9. (tlanslatedd and edited by dm dabrowska and tp speed, statistical science (1990), 5, 465-480). Annals of Agricultural Sciences, 10:1-51, 1923.
[22] Donald B Rubin. [on the application of probability theory to agricultural experiments. essay on principles. section 9.] comment: Neyman (1923) and causal inference in experiments and observational studies. Statistical Science, 5(4))472-440, 1990.
[23] Jing Lei, James Robins, and Larry Wasserman. Distribution-free prediction sets, Journal of the American Statistical Association, 108(501):278-287, 2013.
[24] Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Associ- ation, (136523):1094-111
2018.
[25] Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression.
Advances in Neural Information Processing Systems, 32:3543-3855 -553,, 2019.
[26] Ryan Tibshirani, Rina Foygel Barber, Emmanuel Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. arXiv preprint arXiv:/904.066 16009, 2019.
[27] Daniel G Horvitz and Donovan J Thompson. A generalization of sampling without replacement from finite universe. Journal the American statistical Association, 47(260):663-685, 1952.
[28] Moshe Shaked and J George Shanthikumar. Stochastic orders. Springer, 2007.
[29] Michael Rothschild and Joseph E Stiglitz. Increasing risk: definition. In Uncertainty in Economics, pages 99-121. Elsevier, 1978.
[30] Søren Asmussen and Mogens Steffensen. Risk and insurance. Springer, 2020.
[31] Josef Hadar and William R Russell. Rules for ordering uncertain prospects. The American economic review, 59(1):25-34, 1969.
[32] Henrik Linusson, Ulf Johansson, and Tuve Löfström. Signed-error conformal regression. In Advances in Knowledge Discovery and Data Mining: 18th Pacific-Asia Conference, PAKDD 2014, Tainan, Taiwan, May 13 13-16, 2014. Proceedings. Part 18, pages 224-236. Springer, 2014.
[33] Carlos Carvalho, Avi Feller, Jared Murray, Spencer Woody, and David Yeager. Assessing treatment effect variation in observational studies: Results from a data challenge. Observational Studies, 5(2):21-35, 2019.
[34] Hugh A Chipman, Edward George, and Robert E McCulloch. Bart: Bayesian additive regression trees. The Annals of Applied Statistics, 4(1):266-299 2010.
[35] P Richard Hahn, Jared S Murray, and Carlos M Carvalho. Bayesian regression tree models for causal inference: Regularization, confounding, and heterogeneous effects (with discussion). Bayesian Analysis, 15(3):965-1055 2020.
[36] Suzanne Sniekers and Aad van der Vaart. Adaptive bayesian credible sets in regression with a gaussian process prior. Electronic Journal of Statistics, 9(2):2475-22 2527, 2015.
[37] Stefan Wager, Trevor Hastie, and Bradley Efron. Confidence intervals for random forests: The jackknife and the infinitesimal jackknife. The Journal of Machine Learning Research,
15())11625-165, 2014.
[38] Maggie Makar, Fredrik Johansson, John Guttag, and David Sontag. Estimation of bounds on potential outcomes for decision making. In International Conference on Machine Learning, pages 6661-6671. PMLR, 2020.
[39] Ying Jin, Zhimei Ren, and Emmanuel J Candès. Sensitivity analysis of individual treatment effects: A robust conformal inference approach. Proceedings of the National Academy of Sciences, 120(6))e22148))) 14889 120. 2023.
[40] Mingzhang Yin, Claudia Shi, Yixin Wang, and David M Blei. Conformal sensitivity analysis for individual treatment effects. Journal of the American Statistical Association, pages 1-14, 2022.
[41] Yingying Zhang, Chengchun Shi, and Shikai Luo. Conformal off-policy prediction. In In- ternational Conference on Artificial Intelligence and Statistics, pages 2751-2768. PMLR, 2023.
11

---

[42] Muhammad Faaiz Taufiq, Sean-Francor Ton, Rob Cornish, Yee Whye Teh, and Arnaud Doucet. Conformal off-policy prediction contextual bandits. arXiv preprint arXiv:2206. 04405, 2022.
[43] Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34: 14:16661177, 2021.
[44] Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. arXiv preprint arXfv:2202.13415, 2022.
[45] Hongxiang Qiu, Edgar Dobriban, and Eric Tchetgen Tchetgen. Distribution-fres prediction sets adaptive to unknown covariate shift. arXiv preprint wrXiv:2203.06/222 2022.
[46] Jeanne Brooks-Gume Fong-ruey Liaw, and Pamela Kato Klebanov. Effects of early intervention on cognitive function of low birth weight preterm infants. The Journal of pediatrics, 120(3):350-
359, 1992.
[47] Vincent Dorie, Jennifer Hill, Uri Shalit, Mare Scott, and Dan Cervone. Automated versus dourt-yourself methods for causal inference: Lessons learned from a data analysis competition. Statistical Science, 2019.
12

---

Appendix A: Technical Proofs
A.1. Equivalent Definitions for Stochastic Orders
The following section introduces alternative definitions for stochastic dominance between two cumulative distribution functions (CDFs), denoted as F and G. These definitions will be used in the proof for Theorem 2 (Section A.4.). Note that these definitions are equivalent to Definition 1 provided in the main text but are expressed in decision-theoretic terms.
Definition A1 (First-order stochastic dominance) F has first-order stochastic dominance (FOSD) over G. F Za: G, if and only if Ex~ [u(X)] 2 (xx=(((X)) for all non-decre (utility) func- tions 2 IR + R (Theorem 1.A.3 in [28] proves the equivalence of Definitions 1 and A1).
Definition A2 (Second-order stochastic dominance) F has second-order stochastic dominance (SOSD) over G, Ley G, if and only if (xxxr_(x()) 2 xxx [u( X)] for all non-decreasing con- cave functions I- + I (Refer to Theorem 4.A.1 and Eq. (4.A.7) in [28]).
In addition the alternative definitions of FOSD and SOSD above, we also state the definitions for two additional notions of stochastic ordering that we will use in the proofs.
Definition A3 (Statewise stochastic dominance) A random variable X has statewise stochastic dom- inance (SWD) over a random variable Y, X sww Y, if and only if X 2 for every draw of the bivariate random variable (X,Y) Px, Note that SWD implies FOSD.
Definition A4 (Convex dominance) has convex dominance (CX) over G, F Zex G, if and only Ex-r ((X) 2 Ex [x((X) for all convex functions u R R.
A.2. Useful Lemmas
Lemma A1. (Pointwise and marginal stochastic orders) For random variables X, Y, and Z, the following conditions hold for any marginal distribution Px:
(i) Y X x .. Z X x, Vo E X
Y
Ea) Z,
(ii) Y X = x 2
Y Eej Z,
Emor Z. (ili) Y X = x E Emer x - Va E
Proof. Recall from the definition of FOSD (Definition A1) that if Y X = En) Z| X x, then:
Ev~pyix=2 (V) 2 Eww.Pix=,, [W)], Vr E X,
(A.1)
for non-decreasing functions u - RR. Since expectation is positive linear operator, marginal- izing both sides with respect to the density Px preserves the inequality in (A.1):
Ex-Px [Ev~Prix (tv))rix(u(V)]]] 2 Ex-px (Ew~P [u [w))))),
(A.2)
which recovers the definition of FOSD between random variables Y and Z, i.e.,
Ey-py [u(Y)] 2 [xm-)[u(]])
(A.3)
and concludes the proof for (i). Note that (A.3) holds for all non-decreasing functions : RR + R. which also includes all con-decreasing concave and convex functions that define SOSD and MCX. Hence, combining (A.3)) with Definitions 2 and A2 concludes the proof for (ii) and (iii).
Lemma A2. (Convex dominance of mixture distributions over convex combinations random variables) Let X E R and Y R. be two real-valuet random variables with finite means, and let т e [0, 1] be a constant. If Z and are two random variables constructed as:
ZZ is the absolute value of the convex combination of X and Y. i.e., Z | + (1
V is a mixture of X and YY), i.e., V = W + (1 - Y) W ~ Bernoulli(T),
then we have V maz Z for any म E [0, 1] and any distribution P(X,))
Proof. Recall from Definition 2 that V Zmcz if and only if
Evlu(V)] > Ez [[([((Z) Z)),
(A.4)
13

---

for all non non-decreasing convex functions u. : R + R. Given the definition of V above, we have
[v([([)] = P(W = By (v(V) W 1| + P(W = 0) lEv [u(V) W = 0]
= π
+ - T) B[u((
IV Iv@ B[u(T|X| + (1 -) [[u[[TX] + (1-x)
= E[u[|TX = E[] + V Emex Z,
(A.5)
where the last two inequalities follow from (a) the convexity of and (b) the monotonicity of along
with the application of the triangle inequality | |x| + [(1 x) YY 2 1 + - (1-r)\/.
Lemma A3. (Stochastic dominance and sign changes in CDFs) Let X and Y be two non-negative random variables with distribution functions F and G and with finite means such that B[x] s E Let S (f) be the number of sign changes of a function f defined as
8 (f) = sup S [f(z1); ((,2)....,
(A.6)
where S 5a1,, (0..0....... am is the number of sign changes of the sequence (a1, 02: Aml, with the zero terms being discarded, and the supremum in (A.6) is taken over all sets 21 < 2i2 <... < Fm such that m oc. Then, F Ee) mmcu G if and only if there exist random variables Zz, Zz..., with
distribution functions G1, G2..., such that Zi st X. [[[2]] S E[Y), j = 1.2,................ Zj - at Y and E|Z,]] + E as j + 30 and S (Gj+1- -G))
1 and the sign sequence is - [[t,,)] [[]+ Here,
sst denotes equality in law and s denotes convergence in distribution.
Proof. Refer to Theorems 4.A.22 and 4.A.23 in [28] for full proof.
Corollary A4. (CDF crossings under SOSD and MCX) Let X and Y be non-negative random variables with CDFs F and G with finite means such that B[X] S [ Then, if F Z2) [Zmez] GG we
have S (F' - G) 2 1 and the corresponding sign sequence is { ..... +Hl +, -H. Equivalently,
there exists a E (0, 1) such that F F-"(a") = GG-((aa))
(a") = v² and F(u) 2 [siG(v), Vu 2 v².
Proof. From Lemma A3, we know that if F m mmcz G then there exists a sequence of random variables Z1, Zz.... with distributions G1, G2...., such that Z, st X, Zj + Y. and the CDFs satisfy= S S-(G++1 G2) = with sign sequence (-,+ [[++. -}]-
Now observe that for any three integers & < m < if 5 (G.. Ga) = 1 with sign sequence of [[++ -)), and S (G, G,.)
with sign sequence {-, + [[[ }]. then by mono- tonicity of CDFs follows that in the last point of crossing, G2, crosses G₁. from below [above], ie.. H. Thus, for any j. the three with sign Hf +, -},, which implies that in the last point of crossing, F crosses G
Gn G2 has at least one sign change with +...,-.+)II F) I and S (G- G2)
random variables Z₁ =d X, Z, and Y satisfy 5 sequence f...,.,th7 -, from below [above] and concludes the statement of the corollary.
A.3. Proof of Theorem 1
Theorem 1. (X,, W., ((()), i = are exchangeable, then Eo* E (0, 1) such that the pseudo-interval Cip (Xx+1) constructed using the datasel D = {(Xi,, ((, Y:)) satisfies
P(Yn-1 (1) (0) E C,
21 a, Va E [0,a"),
at least one the following stochastic ordering conditions hold: (i) Vi Ein V*, (i)) V In V*, "nd (ii)) V& Emex V*h Under condition (i), we have a* 1.
Proof. Without loss of generality, assume that the conformity scores are sorted, V < < V Vpnno
where n. [Del- Recall from (10) that the pseudo-interval is constructed as:
(x) = (f(x) - Qv (1 - a), F(x) + Qv (1 a))]
(A.7)
where Qv (1 - a)) is the empirical quantile defined as
a 2 nc+
Qv (1 a)) = x,,
(A.8)
o.w.
14

---

Combining (A.7) and (A.8), we notice that the following two events are equivalent
( (++1(0)) E Cx
(v++1 S Qv, (1-a);
(A.9)
equivalently,
(1)+ - (0) E (Xp(X=1))) = (v S -=
(A.10)
Hence, the achieved coverage probability for the pseudo-interval given by
P(V++1 (V..(oaat))(-a)))
(A.11)
By exchangeability of the variables (X1, Y1() (1(()), ... (Xn+1, 11) Yn+1 (0)), we have
k
PVV S VV,kk)
(A.12)
nh 1
any integer k. 12) holds because ranks are uniformly distributed under exchangeability, i.e., Vg,nn411 is equally likely to fall in anywhere between the calibration points Vpll,.... Vome From (A.12), follows that for [mne + 1)(1 - a)] we have:
[(no + 1)) ail
P(Vg.0+1 S (V..tt..+)(aaaaa)
a.
(A.13)
Th +
Now, we examine the three conditions: (i) Vi Eas V", (ii) Vip Ia V and (iii) Vi Emer V*.
(i) FOSD V Zo V*:
If V. Co V then from Definition 1: Fv, (v) Fv-(U), Vu. Equivalently, FOSD can be written as:
Ve Eo V P(V S u) s P(V* S Su), Vo.
(A.14)
Since (A.14) applies for any v, then the following holds:
Vpxt+1 Eos V = P(V S +o. s P(V I (o(())+)((-)))) (A.15) Combining (A.13) and (A.15) we have
Ve,tt Zo V + P(V S Ve..on+tt((aa1)) 2 P(Vpn41 S (Vo.(ne+3)aaaa)) a.,,
which holds for any a, hence a 1. This concludes statement (i).
(ii) SOSD Vp Za V*:
If V. 3zz V*, then from Corollary A4 we know that 3a* (0, 1) where F7.(a")) Fyt o² and Fv. (v) 2 Fv, (v), Ve 2 n* Equivalently, SOSD can be written as:
Ve Z0 V Eor, s.1. PV, v) S P(V vu
(A.16)
Hence, the following holds: Vpnt1 Se V P(V S 5) S P(V
n+1 5 v), V4 2 v²
= PVV & Ve.[(m+t))1-a)1 $ P(V S ())(0)))))))))) (A.17)
for all a S a". Combining (A.13) and (A.17) we have
Vpm+1 Se V = P(V S (= 2 P(Vent1 S Vo. a.
for all 0 5 a". This concludes statement (ii).
(iii) MCX Vi Emcz V*: Since Corollary A4 holds for SOSD and MCX, the proof identical to the proof for (ii).
15

---

A.4. Proof of Theorem 2
Theorem Let ((( -Ypl
and assume that the propensity score function : [0.1] is known. Then, the following holds: (i) For the X-learner, V₂ and V do not admit to model- and distribution-free stochastic order, (ii) For any distribution P(X, W. (O), ( CATE estimate and nuisance estimate G the IPW- and the DR-learners satisfy V4 Emer
Proof. For CATE estimate 7, the oracle scores are
(- - -(()))))))( (Y(1))
(A.18)
In what follows, we use the notation (z) to denote the conditional random variable V₂ = x, ie., the conformity score evaluated at a given feature point. Similarly, use V*(x) to denote the conditional random variable V*|X = z (oracle score evaluated at given feature point).
Proof of statement (i):
Recall from Table that the pseudo-outcome for the X-learner is given by:
Y = w(Y - (x) + (1 - (x) Y).
(A.19)
With (F) [F(x) YpI. the conformity score for the X-learner can be written as: V (F) [(()) W(Y - (x)(X) - (1 (X) - Y)1-
(A.20)
To prove statement (i), it suffices to find counter-examples for estimators 7, 75 and 7i, or data distributions that cause violations of FOSD, SOSD and MCX between Vi and V* Let IR - R be non-decreasing function. The expected value of u (V)()))) can be evaluated as:
E E[u(VV ([xx) W(Y - (1 (x)
= ==x)) (x)(
- (V(1) - (o(2))))) +
(1-x(x)) -(u(((()) -
(A.21)
We start by showing that there exists nuisance and CATE estimates and a data distribution for which Vi does not have FOSD on V*. Assume that u is non-decreasing concave. By Jensen inequality, we have:
(x))) < ((u(T(x)) [F(x) (Y(1) 7 (x))] (1 - x (xx) [F(x) (īi (2)) Y(O))
Now consider a nuisance estimates of i%o = and ū = 0, a data distribution for which Y()) > 0
and Y(1) < 0 almost surely, and CATE estimator
F(z) 0, Vr E X, then have
(l)(V((x)) < [[u(T(z)- [f(z) - Y()] (1 x(x)) [f(=) (O))
[]](T(x) (f(x) Y()) + x(x) +
B[u(F(r)-(Y Y(0)))]
S (lu([[(2)- (1
=[[[)
(A.22)
and hence (*) Zas (x) for all and by Lemma A1, we have V Zo, Vg.. The last inequality in (A.22) holds because u is non-decreasing and (1 - (I)) - T is always non-posilee non-positive when (0) > 0 and (11) < 0 (almost surely). Since SOSD is necessary condition for FOSD, then it follows that V₂ does not have FOSD over V in this counter-example.
Now we show that there exists nuisance and CATE estimates and a data distribution for which Vip does not have MCX over V*, This follows directly from the previous counter-example. Since V Zg ViP implies that Vv Emea V (Theorem 4.A.1. in [28]), then VpP does not have MCX over V*
Finally, we show that there exists nuisance and CATE estimates and data distribution for which V does not have SOSD over Vg. Note that [E[V*] > E[[₂] is a necessary condition for V* Za Vi (See Eq. (4.A.6) in [28]). To compare E[V* and [V]. we rewrite [V & (x)) as:
(x)] = E[]F(x) - W(Y -mo(=)) (1 () (2) - Y)J],
= Txx) [=] (Y(1)) + (11-+((11))) (x)) Bl]((z) (f)(z) - (O))I,.
= EE(T(x) [F(x) - + (F()) (
2 E[[f(()) - (0((1)))o(x))) - (1-)))
- -E])(=))(((())) --
Y(O)).
=
16

---

Σ [z) ( 1
(1 - ( (.) (z))
the same random variables, For nuisance estimates that satisfy (1 (1) and (z( > (0) almost surely, it follows that
+ ( (
- (0)) is always positive and hence we have:
2 Elf((2) (Y(1))
B[Va(=)) -
(1())) + + (1)) (())I),
-
(A.23)
which implies that V' does not have SOSD over Vpp.
Proof of statement (ii):
From Table 1, the pseudo-outcomess for the IPW- and DR-learners can be written as:
Vx(+) W W - (1 W) [+(z) Y (i)=(=)),
(A.24)
where Y and Yi-s are defined as follows:
(1),,
for IPW-learner, (x)) + ((a())))7o( (0))))) for DR-learner, and (Y(1) ((1)-(((1))
(x())=
x
for IPW-learner, m(a) (Y(0) - -m())) + (71(x) - -)(x)), for DR-learner.
Y(0),
(1-=(x))=
(A.25)
Note that, following the definition in (A.25), the following holds for both the IPW- and DR-learners:
π (x) (z) (1 (z)) (x) (1) (0).
(A.26)
Hence, applying (A.26) to the definition of the oracle score, we can write V (x) as follows:
(x)x) [f()) (Y (1) Y() [f(x) (() (1 (1- (x))) Y
[T(x) - + + - (x)) (f(x) - Y1-1(1)) (z))).
(A.27)
From the expressions in (A.24) and (A.27), we can see that Vz(xx) is a mixture of the absolute values of the two random variables
and (f(z) -(==)> with mixing proportions :Tx:) and 1 (z), and Vxx) is the absolute value of the convex combination
(f(z) (x)) and (f(x) (x)), with weights xxx) and 1 T()). Applying Lemma A2, it
follows that (i) Emer (x), Ve E X, and from Lemma A1 we have that Vi Emer V
Li
Stochastic orders for the signed distance conformity score:
We have shown that for the absolute residual conformity score. = Yol. the DR- and
IPW-learners guarantee Vi Zmer V* irrespective of the data distribution and underlying models. We now study the stochastic orders achieved by the signed distance conformity score proposed [32],
defined as: (7) = f(X)- Yp. Let u be non-decreaiing concave function, then we have:
Blu(VV(F)) Elu(F(X)) KoDI
= =TE((F((X) - + (1- Fi--
s E[x( - π TÝ. + (1- - (1 10-)))in 8-=))
= E[u(f(X) -( (Y(1)Y(()))) = =[l(v"
(A.28)
where the inequality follows from the concavity of u(.) and the last equality follows from (A.26). From Definition 1, (A.28) implies that V žoz V₂ (Table 2). Note, however, that the construction of pre- dictive intervals with the signed еттоr distance does not follow (5) since the signed distance score does not sort absolute errors from largest smallest, but sorts conformity scores from maximum positive error to maximum negative error. Hence, Theorem 1 does not apply to the signed distance error and the SOSD condition * Ze Vp does not guarantee coverage.
Another more commonly used variant of the signed distance score was proposed in [25]. Given a quan- tile regression model [R, 7x], [25] uses variant of the signed error score defined as:
(F( max (xx Yz, Y. - (X)).
(A.29)
17

---

The construction of the predictive intervals in [25] follows (5). Note that m vt can be written as:
(ax y) = I + y
x
yl
(A.30)
2
2
Applying (A.30) to (A.29), we can rewrite the conformity scores as:
Th
πr + F
2Y
VF((F)
2
π 4h +
7 +
Th
Y
(A.31)
2
Similarly, the oracle scores can be written as:
Th
+ Th
-(Y(1)
(A.32)
2
2
Let δr = R 88, F and let u be non-decreasing convex function, then we have:
E(u(Vv(F))] E (0(or= (ōT + 1 |]
ग Elu(OT_ + 1ōT+ - Y₂1) + (1 T) Blu((r- + 1ōT+ - -Y1-a))
2 Blu(T or_ + π 1ō++ Yř, + (1 (1-x) ÖT_ + T) JōT+ V1-1
2 Blu(ôT- + 1o++ x Y, (1 - (11-))Bi-- --
Blu(8r- + 1б++ - -1 -()
(A.33)
where the first inequality follows from the convexity of u, the second inequality is application of the triangle inequality, and the last equality follows from (A.26). Following Definition 2, (A.33) implies that V₂ Emer V when the base models apply quantile regression and the conformity scores follow the signed distance score [25]. Hence, the coverage guarantee in Theorem applies to conformal meta-learners based conditional quantile regression.
Appendix B: Literature Review
In this Section, we provide a comprehensive overview of the relevant literature. We categorize the previous literature into three distinct strands: (1) Bayesian methods for modeling individualized causal effects, (2) frequentist methods for inference on parameters pertaining to individualized causal effects, and (3) conformal methods for predictive inference of ITES.
(1) Bayesian methods. Predictive inference on ITTES has been traditionally conducted using Bayesian methods. These methods place prior distribution over the nuisance parameters lo and μl, and then estimate the CATE function through the posterior distribution computed conditional on the observa- tional dataset D {(X;, W,, Y:) More formally, Bayesian procedures operate as follows:
Y(w) Mw Mo Io, #1 I1I
(A.34)
where Hy and IT, are priors over function spaces, and e (O o²). Given the dataset D, the posterior distributions over the two nuisance functions are used to estimate CATE as follows:
f(x) = - yp [u (x)]
(A.35)
Furthermore, the posterior distributions To ID and IT, DD can be used to construct predictive intervals on ITTES through the posterior credible intervals of Pro (x)(x) and Pn, (z))) i.e.
P Poo.ar- (Y - (0) Ō(X) X x) =1
(A.36)
Different incarnations of the Bayesian framework correspond to different choices of the prior (Il, II,). Bayesian Additive Regression Trees (BART) [34] is one popular model for Bayesian nonparametric regression that was later adapted for causal effect estimation and inference [8, 35J. While [8] applies BART for estimation of CATE and inference of ITE out-of-the-box [35] introduces new regularization techniques to incorporate different forms of inductive biases the CATE target parameter. Another
18

---

popular choice of the prior (Il,, I11) is based on Gaussian processes where inductive biases are incor- porated through different choices of the kernel function of a reproducible kernel Hilbert space [9].
Unlike the conformal framework, Bayesian methods do not provide coverage guarantees on their credible intervals. Frequentist coverage can be achieved in asymptotic sense by Bayesian method under certain technical conditions [36]. Existing methods typically do not provide any finite-sample guarantees and the achieved empirical coverage depends on the choice of the prior. Models such as BART are typically applied with a default prior for all datasets, undermining their achieved coverage in diverse experimental setup. In [1], it was shown that Bayesian methods have a tendency to undercover ITEs in some data generation processes (e.g,, in
covariate spaces).
Bayesian methods also suffer from two key advantages compared to the conformal framework. First, Bayesian inference procedures and the computation of posterior distributions are typically tailored to specifie model architectures. Generalizing exact Bayesian inference to arbitrary model architectures, including modern deep learning architectures, is not straightforward. Moreover, Bayesian inference conducted either using computationally exhaustive Monte Carlo method (e.g.. BART) or through expensive evaluation of an analytical posterior distribution that does not scale well with the number of training points (e.g.. Gaussian process posteriors are O(n)))
(2) Frequentist methods. Another model-specific approach to inference on causal parameters is the Causal Forest model in [7]. This procedure uses constructs pointwise confidence intervals on the CATE function 7 (z)) using conditional variance estimates based on the infinitesimal jackknife [37]. These intervals provide asymptotically valid coverage of CATE under mild regularity assumptions, but they provide no (asymptotic or finite-sample) guarantees on coverage for ITEEs.
Another non-Bayesian approach for predictive inference of ITEs uses a quantile regression approach to train models that construct upper and lower bounds on the observed potential outcomes using a pinball loss [38]. While quantile regression can be repurposed for different model architectures, this approach does not provide
sample coverage guarantees and empirically found to undercover outcomes, hence it is typically supplemented with conformal prediction procedure [25].
(3) Conformal methods. The application of conformal prediction to ITE inference traces its origins back to the covariate shift problem. In [26], variant of conformal prediction was proposed to address problem setups where training and testing data are drawn from different distributions, breaking the exchangeability assumptions required for valid inference.
In the treatment effect estimation setup, covariate shift arises because the distributions of the treatment groups differ from that of the target population. Recall from (8) that the conformity scores for the nuisance estimates fio and ğ can be written follows:
v (To) A V(XK, Yx(0); Po), Vk E De..:
(M) ) (X V.)1): Vk: E Da,1:
Both sets of conformity scores are sampled from the populations Px = and PxIW= respectively, and hence they are not exchangeable with the conformity scores sampled from target covariate distri- bution Px. To retain the validity of intervals constructed using the sample quantiles of conformity scores, [26] introduces the notion of "weighted exchangeability" and proposes a weighted sample quantile with weights based on the likelihood ratios between training and testing distributions. [1] uses this method to construct valid intervals for potential outcomes and then proposes different approaches for combining these intervals to construct intervals for ITEs. Other work utilized the conformal prediction framework to conduct sensitivity analysis by devising hypothesis tests for the presence of hidden confounders [39, 40]. [41, 42], conformal prediction has been utilized to address the closely related problem of off-policy evaluation. More recently, there has been further work developing methods for conformal prediction under covariate shift [43, 44, 45].
The key difference between our work and the aforementioned approaches is that we do not apply conformal prediction to the nuisance parameters, and instead conduct inference directly on the target parameter. We do so by applying conformalization with respect to transformed targets Y, for each data point in D, and this way the covariate shift problem does not arise since (X, Y.) is sampled from the same distribution in calibration and testing data. One way to think about our approach as opposed to weighted conformal prediction (WCP) is that ours applies re- re-weighting to the outcomes whereas WCP re-weights the conformity scores. By moving the re- weighting step to the outcomes, we decouple the conformal procedure from the model, allowing for more flexible choice of inductive priors and accommodating models that do not estimate potential outcomes directly.
19

---

Appendix C: Experimental Setup and Results
C.1. Overview of the experimental setup
In all experiments, we used Gradient Boosting model with 100 trees as the base model for nuisance estimation and quantile regression on pseudo-outcomes. We used the same base model in conformal meta-learners as well as the weighted CP (WCP) baselines to control for potential performance dif- ferences resulting from different choices of base models. Unless otherwise stated, all experiments followed 90%/10% train/test split of each dataset, and each training set with further split into 75%/25% proper training/calibratinn sets. All performance metrics (empirical coverage, expected interval width and RMSE) are evaluated on the testing data and averaged over 100 runs. The target coverage in all experiments was set 1 = 0.9.
We used the official implementations of the (naīve, exact and inexact) WCP baselines in the R package available at https://github.com/ii cot//ihualliiii///////// 1/efcausal. We executed these baselines through rpy2 (htttss://rpy2.. github.io/) htttt::://ppy2.iituuubo/// wrappers within our Python codebase.
C.2. Details of the IHDP and NLSM datasets
In this Section, we provide details of the semi-synthetic datasets used in the experiments in Section 5.
IHDP dataset. The Infant Health and Development Program (IHDP) was a multi-site randomized controlled trial that studied the effect of early educational intervention on the cognitive development of low birth weight premature infants [46]. The trial targeted low-birth-weige -irthh, premature infants, and administered an intensive high- child care and home visits from a trained provider to the treatment group. [8]. a semi-synthetic benchmark was developed based on the covariate data in IHDP with simulated outcomes to assess the performance of different CATE estimation and ITE inference models when ground-truth counterfactual are known. The dataset comprised 25 covariates (6 continuous and 19 binary variables) capturing aspects related to children and their mothers. The benchmark dataset excluded a non-random subset of treated individuals [8]. The final dataset consists of 747 samples (139 treated and 608 control). The outcomes of all individuals were simulated whereas the treatment assignments followed the true assignments in the study.
The simulated outcomes model in IHDP followed simulation Setup B in [8]. In this Setup, the po- tential outcomes were simulated as follows: Y()) ((exp((X ((exp(X w, 1), where W is an offset matrix, w is set so that the ATE on the treated is always equal to 4, and the coefficients 3 are sampled independently at random from 0.1, 0.2, 0.3, 0.4) with probabilities (0.6, 0. 0.1, 0.1, 0.1). ). In our experiments, we used the 100 realization of the train- ing and testing data released by [6] in https://www.freet wwt.://wwwo.ooo..ooo wwt...oooo/eeesdpp...... https://www.frrt www.fffddo..ooooco wwtp.//e////////ppppppp pap___pc_______ 1el0o.tes...p..
8) 1) and (1)
1-1000100000 to0..traii.pp.... and
NLSM dataset. The National Study of Learning Mindsets (NLSM) is large-scale randomized trial that studied the effect of a behavioral intervention, designed to instill students with a growth mindset, on the academic performance of students secondary education in the US [3]. In [33], a semi- synthetic benchmark was developed based on the covariates of the NLSM study. Unlike IHDP, this benchmark does not use the real covariates of NLSM but rather buids a synthetic process to emulate NSLM in terms of the covariate distributions, data structures, and effect sizes. The final dataset com- prised 10,000 data points. The data was generated based on 5 principles (See Section 2 and Appendix A in [33]). The principles are: (1) ATES should be well-estimated from synthetic data by any reason- able procedure, (2) variability in CATE should be relatively modest, (3) treatment effect heterogeneity can be approximately recovered given complete knowledge of the correct functional form, (4) no addi- tional unmeasured treatment effect moderation at the individual level, and (5) unexplained treatment effect heterogeneity at the group level should be present reasonable levels.
C.3. Further experiments using the ACIC dataset
In addition to the IHDP, NLSM and fully-synthetic experiments, we also conducted further experi- ments on 77 datasets from the 2016 Atlantic Causal Inference Competition (ACIC2016) [47]. We followed the processing steps in [16], creating dataset of 4,802 data points and 55 covariates. Out of this base dataset, we created 770 simulation setups using the 77 settings proposed in [47]. The 77 settings represent different levels of complexity of response surfaces, varying degrees of confounding.
20

---

1.00
Averoge Coverage Lavels
mmmmm wwwm
CMXX
0.90
WCP-Naive
WCP-Exoc!
WCP-Inexac
0.80
10
20
30
50
70
30
25
CMX
WCP-Naive
Average
WCN-nexact
10
10
20
40
50
60
10
Aucrage RASE
CM-X
WCP Nal-
WCP inckantxact
10
Experiment number
Figure C.I: Performance of all baseline across ACIC experiments.
21

---

overlap and effect heterogeneity. All settings use the same covariates but simlate different treatment assignments and response surfaces. We created 10 realizations of each of the 77 settings and average the performance of all baselines across the 10 runs.
The empirical coverage, average interval width and RMSE results across all 77 experiments are pro- vided in Figure C.I. The comprehensive experiments in Figure C.I. align with the overall conclusions of the experiments in the main paper. For target coverage larget coverage with the exception of the X-learner. All models achieving the target coverage dis- played conservative predictive intervals, acheiving coverage levels that exceed the 0.9 target. In terms of efficiency, the DR-learner and WCP-Inexact achieved comparable interval widths, significantly outperforming all other valid procedures (IPW-leamne,, WCP-exace and WCP-Naive) Finally, the the DR-learner outperformed all valid procedure in terms of RMSE in almost all experiments, providing point estimation accuracy that is on par with the X-learner.
a 0.9, all models achieved the
These results align with the overall conclusions of the experiments in the main paper. The DR-learner generally inherits the accurate point estimation of its underlying CATE meta-learner, provides cov- erage guarantee for practically relevant values of target coverage and achieves this coverage with competitive efficiency making favorable approach for both point estimation of CATE and predic- tive inference of ITE.
22


Reviews:
[{"review_id": "o5xrDYkxJh", "reviewer": "Reviewer_855y", "summary": "The paper deals with confidence intervals for the ITE prediction task, where the authors propose to use the framework of conformal prediction for meta learners. The learning of nuisance models for generating pseudo outcomes is done one separate data split, and the final regression models trained to predict pseudo outcomes are evaluated on a validation set to compute the empirical calibration distribution. The proposed approach for calibration removes the issues of covariate shift and inductive biases on nuisance models with the prior approaches of weighted conformal prediction. The authors also provide guarantees on when the confidence intervals for ITE with pseudo outcomes cover the confidence interval for ITE with true outcomes; and further validate their findings with synthetic and semi-synthetic datasets.", "questions": "* I do not understand the argument of authors that conformal prediction with meta learners is not affected by the choice of inductive biases of nuisance models. The pseudo outcomes are a function of the nuisance models, hence the predicted effect (ITE), which is learnt using pesudo outcomes, indeed depends indirectly on the nuisance models. Hence, the choice of nuisance models would affect the empirical calibration distribution, consequently the confidence intervals for ITE. \n\n* I do not follow the argument of authors that conformal meta learners are not affected by the covariate shifts. The covariate shift between the control and treatment population should still affect the computation of pseudo outcomes with finite samples; hence the confidence intervals.\n\n* Please clarify the assumption of knowing the true propensity model. What challenges would the authors in their analysis if they do not make this assumption.", "limitations": "Yes, the authors have addressed any potential negative societal impact of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* ITE prediction is an extremely relevant problem in causal inference as population level effects (ATE, CATE) would not necessarily transfer to the individual level effects. Further, having an estimate of uncertainity along with ITE prediction is crucial for high-stakes application scenarios. Hence, the approach of conformal meta learners is highly relevant for the current challenges in causal inference. \n\n* The proposed combination of conformal prediction with meta learners is novel to the best of my knowledge. Further, the ITE coverage results with pseudo outcomes for standard meta learners (DR Learner) are quite significant as it is non-trivial to understand when the predicted confidence intervals with observed information can represent the confidence intervals with counterfactual information. \n\n* The proposed conformal meta learner approach is technically sound with theoretical proofs on their validity and empirical justification on a decently diverse set of benchmarks.", "weaknesses": "I do not think the paper has any serious weaknesses, but I have listed some of them in the questions section ahead. One major suggestion is that the paper could be written with better clarification for the section on conformal meta learner being robust to covariate shift and inductive biases. The arguments are stated informally and it is a bit hard to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the thoughtful and detailed feedback. Please see below our point-by-point responses to your comments. \n\n- **Conformal meta-learners and inductive biases**\n\nIn the points below, we clarify our argument on how the proposed framework enables a more flexible set of choices of inductive priors for the CATE function.\n\n1. First, we would like to stress that we *did not claim* that conformal meta-learners are *not affected by the choice of inductive priors* or that they are *robust to inductive biases*. In fact, inductive priors are not a problem that we need to overcome, but rather a design dimension that we need to consider when building our estimators. As we mention in Lines 146-147, covariate shift and inductive priors are *”two distinct characteristics of CATE estimation that interact with the conformal prediction procedure”*. As the reviewer rightly pointed out, the performance of the model (in terms of RMSE and efficiency) will depend on the choices for the models of $\\mu_0$ and $\\mu_1$. This is also evident in all our empirical results that compare the different meta-learners (i.e., different inductive biases) in Figures 4 and 5, as well as Tables 2 and 3.\n\n2. The key finding of our paper is that a stochastic ordering framework can be used to evaluate the **validity (and not performance) of any meta-learner architecture** that encodes a specific inductive prior. This means that through the conformal meta-learners approach, we can apply CP to a broader class of models that correspond to a wider range of inductive priors. This enables us to conduct CP on top of models that typically perform well in terms of point estimation (RMSE), such as the DR-learner, instead of restricting to models that estimate and conformalize potential outcomes separately as in [1].\n\n3. To sum up, our work does not propose a method that is *robust to inductive biases*, but proposes a framework for conducting valid inferences with a broader class of inductive priors than what was previously possible.\n\n4. We believe that this confusion might have been caused by the wording in Line 47 which presents covariate shift and inductive biases as *two challenges*. We will improve the phrasing of this paragraph in the final version of the manuscript.\n\n- **Conformal meta-learners and covariate shift**\n\nCovariate shift arises in the treatment effect estimation problem because the covariates in the datasets $(X_i, Y_i(0))$ and $(X_j, Y_j(1))$ are drawn from distributions $P_{X|W=1}$ and $P_{X|W=0}$, respectively, which differ from the target distribution $P_X$.\n\nOn the contrary, when we apply the pseudo-outcome transformation, we get a single dataset $(X_i, \\widetilde{Y}_{\\varphi, i})$, where $\\widetilde{Y}_{\\varphi}$ depends on the observables $X$, $Y$, and $W$. This transformation merges both the control and treated group into one dataset. Our framework applies the regression model and CP procedure to $(X_i, \\widetilde{Y}_{\\varphi, i})$. Since the covariates in this dataset are drawn from $P_X$, there is no covariate shift. Intuitively, for the IPW and DR-learners, the pseudo-outcome transformation adjusts for covariate shift in the output space rather than the covariate space.\n\n- **Knowledge of the propensity scores**\n\nTo the best of our knowledge, all existing valid inference procedures for ITE require knowledge of the propensity scores. This assumption holds true when the data comes from a randomized trial with or when treatment assignments follow a strictly known policy. When the data comes from an observational study, we need to estimate the propensity score $\\pi$ and use this estimate in the pseudo-outcome transformation. The challenge that would arise in this case is that the results on Theorem 2 will no longer hold and one will need to establish the conditions under which approximate propensity scores would give rise to valid inferences.\n\nSince the results of Theorem 1 still applies under approximate propensities, we note that our stochastic ordering framework can be useful in analyzing the validity of conformal meta-learners with approximate propensity scores. To approach this problem, one can use a PAC-style generalization bound on approximate propensities along with Theorem 1 to derive conditions for stochastic dominance (i.e. validity). \n\n- **Clarity of writing in Section 3**\n\nThank you for this suggestion. In the final manuscript, we will improve the clarity of writing in Section 3 by clarifying how conformal meta-learners handle inductive biases and covariate shifts as explained above.\n"}, {"review_id": "40dyiBw7d0", "reviewer": "Reviewer_vDHA", "summary": "This paper adresses individual treatment effect (ITE) inference using conditional average treatment effect (CATE) meta learners. \n\nThe authors propose to wrap the meta-learners with a conformal prediction step so as to obtain valid confidence intervals for the estimated treatment effect. Since the CATE estimator induces a distribution shift wrt conformity scores, the coverage guarantee does not immediately translate into the desired coverage of actual ITE. In a who can do more can do less spirit, the authors point out that ITE coverage is ensured if conformity scores obtained from pseudo-outcomes stochastically dominate the scores obtained from actual « oracle » outcomes. The authors succeed at providing stochastic dominance results for several stochastic order / meta-learner pairs. \n\n", "questions": "In my humble opinion, this is a very good paper. The problem is well formulated, the proposed solution is clearly exposed and easy to reproduce. The solution is theoretically proved to work (under some clearly stated conditions) which is confirmed by numerical studies.\n\nFurthermore, the authors have correctly self-identified the limitations of their approach (knowledge of propensity score and hard to assess level range). \n\nMy only remark is that I would have like to see a synthetic experiment how far is the proposed solution to the optimum/oralce especially in terms of interval length.", "limitations": "- knowledge of the propensity score is required\n- validity range of level $\\alpha$ not always easy to determine\n- over-conservatism of the prediction intervals\n\nThe two first points are already discussed by the authors in the submission. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- the paper is very clear and pleasant to read\n- the tackled problem is challenging and impactful\n- the proposed solution is backup by both theoretical and experimental results", "weaknesses": "- the method is by-design over-conservative \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and the feedback on our paper. Please see below some points that we would like to clarify in response to your comments. \n\n- **Knowledge of the propensity scores**\n \nPlease note that this is not a unique limitation to our method. To date, all valid inference procedures for ITE in the literature require knowledge of the propensity scores. There are practical setups where this assumption will hold true, i.e., when the data comes from a randomized trial with or when treatment assignments follow a strictly known policy.  However, when the data comes from a real-world observational study, we need to estimate the propensity score $\\pi$ and use this estimate in the pseudo-outcome transformation. In this case, the results in Theorem 2 will no longer hold and we will need to establish the conditions under which approximate propensity scores would give rise to valid inferences.\n\nOn the positive side, the theoretical framework we proposed in this paper can help address this problem. Since the results of Theorem 1 still applies under approximate propensities, our stochastic ordering framework can still be used to analyze the validity of conformal meta-learners with approximate propensity scores. To approach this problem, one can use a PAC-style generalization bound on approximate propensities along with Theorem 1 to derive conditions for stochastic dominance (i.e. validity).\n\n- **Conservatism of conformal meta-learners**\n\nWhile we have not conducted a theoretical analysis of the efficiency of the DR-learner, our empirical results do not show that its intervals are more conservative than other valid procedures. In most of our experiments (Fig. 4(b) and 4(c), Table 2 (IHDP)), the DR-learner intervals are comparable to other valid procedures (WCP exact and naive) and are only outperformed by invalid procedures (inexact WCP). We chose to highlight the NLSM experiment in the main text in order to show the cases where conformal meta-learners under-perform, i.e., when the CDF of pseudo-scores and oracle scores deviate significantly (Figure 5). In our extensive experiments on 77 datasets from the ACIC challenge provided in the Appendix, we found that conformal DR-learners significantly outperforms all valid procedures based on WCP in terms of efficiency (Figure C.1 in the Appendix).\n\nIn theory, there is no reason to believe that conformal meta-learners are destined to issue conservative intervals compared to the naive and exact WCP methods. These methods use Bonferroni correction or other multiple testing approaches to combine intervals on POs, which is also going to lead to conservative intervals. The conservatism of ITE intervals is a natural consequence of not observing counterfactuals—analyzing the optimal length of these intervals is an interesting direction for future work.\n\n- **Comparison with the oracle interval length in synthetic experiments**\n\nThank you for this suggestion. We believe that we have already done this comparison in Figure 4(b) (blue vertical lines correspond to optimal interval widths). Please let us know if this addresses your question or if we misunderstood the oracle length you are referring to.\n"}, {"review_id": "BFfTT6hu1n", "reviewer": "Reviewer_V6dY", "summary": "The paper introduces a framework for inferring treatment effects, merging concepts of conformal prediction and meta-learner. This framework is characterized by its distribution-free validity and coverage guarantees. The authors utilize stochastic ordering techniques to substantiate the framework's validity.", "questions": "(1). Assuming the propensity score is unknown, could we estimate it first and then follow the same procedure outlined in the paper?\n\n(2). For the experimental results, it seems your method can be overly conservative.\n\n(3). Figure 4(c) could be adjusted to resolve the issue of elements overlapping.", "limitations": "See weakness", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and sounded. The innovative approach of employing stochastic ordering to validate the use of pseudo outcomes in conformal prediction is particularly noteworthy.", "weaknesses": "The concept of employing stochastic ordering is innovative; however, its practical application is impeded by the fact that the assumptions are inherently uncheckable\n\nIn comparison to the method proposed in [1], the current method exhibits better performance only in terms of the RMSE of the CATE. However, when considering coverage and average length, it does not surpass the performance of the inexact method. \n\n\n[1] Lihua Lei and Emmanuel J Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(5):911–938, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the thoughtful feedback and comments. In the final manuscript, we will fix the overlapping elements in Fig. 4(c).\n\nPlease see below our point-by-point responses to your comments. \n\n- **Uncheckable assumptions involved in the stochastic ordering framework**\n\nWe are unsure what assumptions the reviewer is referring to. The assumptions required for Theorems 1 and 2 in our paper to hold are:\n\n-- Exchangeability of training and testing points (Line 230). Note that this is a typical assumption that is guaranteed if the data points are drawn from i.i.d distributions.\n\n-- The propensity score is known. This is not an uncheckable assumption, and it holds in application domains where the treatment mechanism is known. \n\nPlease note that **we do not assume stochastic ordering** between the pseudo-scores and oracle scores. Instead, we prove that if such ordering holds, then the resulting predictive intervals are valid (Theorem 1). In Theorem 2, we show that some of the well known pseudo-outcome transformations guarantee that stochastic ordering holds. Both results rely only on the two assumptions adobe and both assumptions are checkable.\n\n- **Performance comparison with other methods**\n\nThe reviewer mentions that the proposed method only improves the RMSE of the CATE and not the interval width. Please note that the main goal of the paper is to develop an inference procedure that reuses some of the meta-learner architectures that perform well in terms of the RMSE while retaining validity of inference. That being said, the conformal DR-learner also outperforms all valid methods in terms of interval length (i.e., efficiency) in many experiments. In 77 experiments within the ACIC benchmark (provided in the Appendix), we found that DR-learners consistently outperformed all valid baselines (WCP exact and naive) in terms of efficiency and not just RMSE (Fig. c.1 in the Appendix).\n\n- **Plug-in estimates of the propensity scores**\n\nYes, when the propensity scores are unknown, a plug-in estimate of $\\pi$ can be used instead of the true propensities in the pseudo-outcome transformation. However, Theorem 2 will not hold in this case and a new analysis is needed. Fortunately, our proposed stochastic order framework can still be used to analyze the setting when propensity scores are approximate (please also refer to our responses to **Reviewers 855y** and **vDHA**).\n\n- **Conservatism of conformal meta-learners**\n\nWe found the DR-learner intervals to be overly conservative only in the NLSM experiment. This is why we chose to highlight the NLSM experiment in the main text in order to demonstrate the cases when conformal meta-learners under-perform. \n\nOur empirical results do not show that its intervals are generally more conservative than other valid procedures. In most of our experiments (Fig. 4(b) and 4(c), Table 2 (IHDP)), the DR-learner intervals were comparable to the valid inference baselines (WCP exact and naive). In our extensive experiments on 77 datasets from the ACIC challenge (provided in the Appendix), we found that conformal DR-learners significantly outperforms all valid procedures based on WCP in terms of efficiency (Figure C.1 in the Appendix). \n\nIn theory, there is no reason to believe that conformal meta-learners are destined to issue conservative intervals compared to the naive and exact WCP methods. These methods use Bonferroni correction or other multiple testing approaches to combine intervals on POs, which is also going to lead to conservative intervals. The conservatism of ITE intervals is a natural consequence of not observing counterfactuals.\n"}, {"review_id": "CPgfiebgqE", "reviewer": "Reviewer_SDYU", "summary": "This paper develops a meta-learner-based approach for conformal inference of individual treatment effects (ITEs). Instead of predicting the missing outcome (the counterfactual), the proposed approach uses plug-in pseudo outcomes as the inference proxy. The validity of ITE inference is based on careful analysis of the statistical dominance of the ITE and the inference proxy. In this way, one no longer needs to adjust for the covariate shifts in applying conformal inference, and the validity holds in a model-free fashion (although the coverage guarantee is more limited). Overall, this paper finds a novel approach to predictive inference of ITEs, and the method works well in synthetic and real-world datasets. ", "questions": "1. What if the propensity score is unknown? \n\nA natural question is whether the proposed method is robust to fitted propensities in the pseudo outcomes. I guess the dominance condition should still be (approximately) reasonable if the estimation is accurate (at least for DR?) since the expectation is the same, and the linear relation still holds? Currently the discussion on this point is a bit passive. But given the existing methods' good performance in this aspect, I was wondering whether more can be said about this problem. \n\n\n2. Do we have a sense of how large $\\alpha^*$ is for conditions (ii) and (iii) in Theorem 1, or practically used conformity scores? \n\nIt is also discussed that $\\alpha^*$ is difficult to know, which I think is a hurdle to the current method. While it's said that $\\alpha^*$ is evaluated in the experiments, I didn't understand how this can be done. Can we have some examples of the $\\alpha^*$ under some parametric models + gaussian noise? Can we know how it changes with characteristics of the distributions? \n\n\n3. A clarification question: Does it only apply to the setting without covariate shift? \n\nFrom my understanding the method requires that the calibration data are exchangeable with the future point. This means for both-PO-missing parts we should also find such individuals in the calibration data, and similarly for single-PO-missing ones. If this is the case, it will be helpful to explicitly mention this because it would be helpful to practitioners. \n\n\n4. Clarification in the presentation of the experiment part. \n\nWhen reading the experiments part I am not sure which situation we are in. Are we inferring the ITE for both POs missing case, and what is the calibration data here? Did you evaluate the performance for the case with only one PO missing? \n\n\n5. What affects the relative performance of meta-learner-based approach? \n\nI see that the performance of the proposed approach is not always the best, and it can yield very long intervals for the NLSM dataset. It is important for practitioners to understand in which cases this method will perform well or poorly. Is it possible to give some guidance on how the performance changes with the data generating process, such as the coupling of POs, the strength of signal/noise? \n\n\n\n---\nMinor points & typos:\n\n1. Should condition (ii) in Theorem 1 be $V_\\varphi \\succeq_{(2)} V^*$? Seems the current order is opposite to other types of conditions. ", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel contribution to an important problem \n\nThis paper makes novel contributions, with a new methodology, to an important problem: uncertainty quantification and predictive inference of individual treatment effects. The proposed methodology significantly differs from the PO-prediction-based approaches in the literature, and the analysis contains interesting findings/perspectives. I believe these are important contributions to the literature of conformal inference, and might inspire more developments in the future. \n\n2. Good writing quality and clarity \n\nThis paper is well-written and enjoyable to read. The challenges are clearly stated and the contributions are easy to capture. I still have some questions/suggestions for improving the presentation; please see my questions. ", "weaknesses": "1. The analysis is somewhat limited to the known propensity case. \n\nSince previous ITE methods already achieve validity in the known propensity case, the contribution of this paper is somewhat marginal in the sense that it does not push the limit of how well/ how much we can do for this problem (although i still appreciate the novel perspective provided here). However, the authors only mentioned the challenge of unknown propensity score at the end of the paper, which is a bit unsatisfactory. \n\n2. The setting in synthetic datasets should be stated more clearly. \n\nI think the most challenging part of ITE inference is when both POs are missing, and this is where the proposed approach becomes most interesting. However, when reading the experiments part I am not sure which situation we are in. Are we inferring the ITE for both POs missing case, and what is the calibration data here? I think further clarification will be helpful. \n\n3. The benefits of this method in experiments is a bit unclear. \n\nThe experiment part contains rich information. However, I am not sure I fully understand in which cases the proposed method performs the best - is it particularly beneficial for both-PO-missing case, or one-PO-missing part? I thought in the latter case the previous approach should suffice, or it still can be improved due to no covariate shift adjustment? I also do not understand why IPW and DR lead to so long prediction intervals for the NLSM dataset. It is mentioned that \"conformity scores have “very strong” dominance over oracle scores\", but it is difficult to understand in which cases this might happen (strong signal? large noise? high nonlinearity?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the thoughtful feedback and comments. Regarding condition (ii) in Theorem 1, the ordering is written correctly for the SOSD condition. Because these notations come from the decision theory literature, they are written to express risks associated with different distributions. SOSD is conventionally written in an opposite fashion to other types of conditions because it means that $V^*$ is \"less risky\" (i.e., has less spread) than $V_\\varphi$. \n\nPlease see below our point-by-point responses to your comments and questions. \n\n- **What if the propensity score is unknown?**\n\nWhen the data comes from an observational study with an unknown treatment mechanism, we need to estimate the propensity score $\\pi$ and use this estimate in the pseudo-outcome transformation. In this case, the general conditions in Theorem 1 still hold, but Theorem 2 does not hold anymore. While an accurate estimate of $\\pi$ will lead to a \"approximate\" validity, no formal result along these lines follow directly from Theorems 1 and 2. \n\nThe good news is: to make progress in establishing validity of inference with estimated propensity scores, we can still use our stochastic ordering result in Theorem 1 combined with generalization bounds on estimated propensities to identify the conditions under which meta-learners with approximate propensities are valid. This is an interesting and natural extension of our work that can build on the novel stochastic dominance framework proposed in this paper to solve an even more challenging problem.\n\nWhile our analysis focuses on the case of known propensities, we believe that it still pushes the boundary of what we can do for the ITE inference problem, since all previous work applies CP to POs rather than ITE. To the best of our knowledge, this is the first paper that applies CP to infer ITE in an end-to-end fashion and enables the utilization of a broader range of meta-learners as the underlying CATE estimators. The novel stochastic ordering framework can also be used to solve more challenging versions of our setup, e.g., when hidden confounding exists or when propensity scores are unknown.\n\n- **How large is the validity region for conditions (ii) and (iii) in Theorem 1 or practically used conformity scores?**\n\nIn almost all our experiments, we found that the pseudo-scores have FOSD over the oracle scores, which means that $\\alpha^*=1$ and conformal meta-learners are valid for all target coverage. This is the case in all experiments in the main text (see Figures 4(c) and 5) as well as in the 77 experiments in the ACIC benchmark (provided in the Appendix). We believe that the IPW and DR may satisfy stronger stochastic orders (i.e., FOSD) than what we proved in Theorem 2. This is because our proof technique is based on an analysis of pointwise stochastic orders conditional on any feature point $X=x$. This is a sufficient but not necessary condition for the marginal scores to satisfy the MCX conditions in Theorem 2. It is likely that a stronger version of Theorem 2, showing that the meta-learner scores have FOSD over the oracle scores is viable.\n\nCharacterizing $\\alpha^*$ is a challenging problem because it requires inference on the crossing points of the CDFs of unknown distributions (i.e., the blue and red CDFs in Figures 4(c) and 5). This is a very interesting but technically-involved analysis that is of broad interest beyond the ITE inference problem, and would warrant a separate paper on its own. \n\n- **Both POs missing case vs. Only one PO missing**\n\nPlease note that we only focus on the more challenging problem of inferring the ITE on a new point $X$ when both POs are missing. All our experiments and analyses are focused on this case. In fact, it is impossible to infer one PO with the IPW- and DR-learners because they do not provide explicit estimates of the POs in the first place. We will clarify this in the final manuscript.\n\n- **Does it only apply to the setting without covariate shift?**\n\nAs mentioned in the previous point, we only focus on the case when both POs are missing. Please note that the key idea of our method is to apply a pseudo-outcome transformation to obtain a single dataset $(X_i, \\widetilde{Y}_{\\varphi, i})$, where $\\widetilde{Y}_{\\varphi}$ depends on the observables $X$, $Y$, and $W$. This transformation merges both the control and treated group into one dataset. Our framework applies the regression model and CP procedure to $(X_i, \\widetilde{Y}_{\\varphi, i})$. Since the covariates in this dataset are drawn from $P_X$, the tuple $(X_i, \\widetilde{Y}_{\\varphi, i})$ is exchangeable between calibration and test data, and there is no covariate shift between calibration and test pseudo-outcome labeled data points. Intuitively, for the IPW and DR-learners, the pseudo-outcome transformation adjusts for covariate shift in the output space rather than the covariate space.\n\n- **What affects the relative performance of meta-learner-based approaches?**\n\nWe agree that a pre-fitting test for whether meta-learners would be a good approach for the problem at hand is important for practitioners to best use our method. It is hard to attribute performance to a single aspect of the data generation process (i.e., noise or non-linearity). Our hypothesis is that the “very strong dominance of pseudo-scores over oracle scores” (Line 339) that leads to the under-performance of meta-learners in terms of interval length is also associated with poor RMSE. We are working on a theory to establish the formal relationship between the meta-learner’s interval length and RMSE. If both quantities correlate, we can devise a hypothesis test for whether the RMSE of the conformal meta-learner exceeds that of the WCP methods, we can use this test to determine whether we should use conformal meta-learners for inference. Luckily, there are already plenty of formal model selection procedures for CATE estimation models that can be used for this purpose. \n"}]

Author rebuttal:

Meta-review: This paper addresses an important and timely questions, namely to provide uncertainty quantification of individual treatment effect in causal inference. The reviewers agree that this is a well written paper and it is pleasant to read, that the tackled problem is challenging and impactful, and that the proposed solution is backup by both theoretical and experimental results. While some reviewers questioned the conservatism of the procedure, it is not showing in the simulation. The known propensity setting is also very common in the literature. It is determined that this paper should be accepted.
Decision: Accept (oral)