Title: Conformal Meta-learners for Predictive Inference of Individual Treatment Effects
Venue: NeurIPS 2023
Accepted: yes
Keywords: Heterogeneous treatment effects

conformal prediction
Abstract: We investigate the problem of machine learning-based (ML) predictive inference on individual treatment effects (ITEs). Previous work has focused primarily on developing ML-based “meta-learners” that can provide point estimates of the conditional average treatment effect (CATE)—these are model-agnostic approaches for combining intermediate nuisance estimates to produce estimates of CATE. In this paper, we develop conformal meta-learners, a general framework for issuing predictive intervals for ITEs by applying the standard conformal prediction (CP) procedure on top of CATE meta-learners. We focus on a broad class of meta-learners based on two-stage pseudo-outcome regression and develop a stochastic ordering framework to study their validity. We show that inference with conformal meta-learners is marginally valid if their (pseudo-outcome) conformity scores stochastically dominate “oracle” conformity scores evaluated on the unobserved ITEs. Additionally, we prove that commonly used CATE meta-learners, such as the doubly-robust learner, satisfy a model- and distribution-free stochastic (or convex) dominance condition, making their conformal inferences valid for practically-relevant levels of target coverage. Whereas existing procedures conduct inference on nuisance parameters (i.e., potential outcomes) via weighted CP, conformal meta-learners enable direct inference on the target parameter (ITE). Numerical experiments show that conformal meta-learners provide valid intervals with competitive efficiency while retaining the favorable point estimation properties of CATE meta-learners.

Conformal Meta-learners for Predictive Inference of Individual Treatment Effects

Abstract

We investigate the problem of machine learning-based (ML) predictive inference on individual treatment effects (ITEs). Previous work has focused primarily on developing ML-based 'meta-learners' that can provide point estimates of the conditional average treatment effect (CATE)-these are model-agnostic approaches for combining intermediate nuisance estimates to produce estimates of CATE. In this paper, we develop conformal meta-learners , a general framework for issuing predictive intervals for ITEs by applying the standard conformal prediction (CP) procedure on top of CATE meta-learners. We focus on a broad class of meta-learners based on two-stage pseudo-outcome regression and develop a stochastic ordering framework to study their validity. We show that inference with conformal meta-learners is marginally valid if their (pseudo-outcome) conformity scores stochastically dominate 'oracle' conformity scores evaluated on the unobserved ITEs. Additionally, we prove that commonly used CATE meta-learners, such as the doubly-robust learner, satisfy a model- and distribution-free stochastic (or convex) dominance condition, making their conformal inferences valid for practically-relevant levels of target coverage. Whereas existing procedures conduct inference on nuisance parameters (i.e., potential outcomes) via weighted CP [1], conformal meta-learners enable direct inference on the target parameter (ITE). Numerical experiments show that conformal meta-learners provide valid intervals with competitive efficiency while retaining the favorable point estimation properties of CATE meta-learners.

Code:

https://github.com/AlaaLab/conformal-metalearners

1 Introduction

Identifying heterogeneity in the effects of interventions across individuals is a central problem in various fields, including medical, political, and social sciences [2, 3, 4]. In recent years, there has been growing interest in developing machine learning (ML) models to estimate heterogeneous treatment effects using observational or experimental data [5, 6, 7, 8, 9]. However, most of these models only provide point estimates of the conditional average treatment effect (CATE), which is a deterministic function that describes the expected treatment effect based on a given individual's covariates. In this paper, we focus on quantifying uncertainty in these estimates, which arises from both errors in the model and the variation of individual treatment effects (ITEs) for individuals with the same covariates. We adopt a predictive inference approach to this problem, with the goal of devising valid procedures to issue predictive intervals that cover ITEs on unseen data with a predetermined probability.

Traditionally, predictive inference on ITEs has been conducted through Bayesian methods such as BART [8] and Gaussian processes [9]. These methods can provide interval-valued predictions of ITEs through their induced posterior distributions (e.g., posterior credible intervals). However, Bayesian methods tend to be model-specific and cannot be straightforwardly generalized to modern ML models, e.g., transformer-based architectures used to model visual and textual covariate spaces [10]. More importantly, Bayesian methods generally do not provide guarantees on the frequentist coverage of

their credible intervals-achieved (finite-sample) coverage depends on the prior [11]. This paper is motivated by the advent of conformal prediction (CP), a frequentist alternative that can be used to conduct model-agnostic, distribution-free valid predictive inference on top of any ML model [12, 13, 14]. Throughout this paper, we will study the validity of CP-based procedures for inference of ITEs.

What makes CP-based inference of ITEs different from its application to the standard regression (supervised) setup? The 'fundamental problem of causal inference' is that we never observe counterfactual outcomes [15]. That is, our 'label' is the ITE which is a difference between two potential outcomes (treated and untreated) for an individual subject-this label is never observed for any given subject because we only ever observe factual outcomes. This poses two challenges [16]:

(1) How to handle covariate shift ? When treatments are assigned to individuals with probabilities that depend on their covariates, then the distributions of covariates in treated and untreated groups differ. Consequently, the distribution of training data differs from that of the target population.

(2) How to incorporate inductive biases ? Unlike supervised learning wherein we fit a single function using examples of covariates and observed targets, models of treatment effects cannot be directly fit to the unobserved effects. Thus, estimates of treatment effects comprise intermediate estimates of nuisance parameters. Different approaches for combining nuisance estimates entail different inductive priors on the potential outcomes that affect the accuracy of the resulting ITE estimates.

The literature on ML-based CATE estimation focuses on addressing the two questions above. Covariate shift affects the generalization performance of ML models-existing CATE estimation models address this problem using importance weighting [17] or balanced representation learning methods for unsupervised domain adaptation [6, 18, 19]. In [5], the notion of 'meta-learners' was coined to describe various model-agnostic approaches to incorporating inductive biases and combining nuisance estimates. In [5, 20], it was shown that the choice of the meta-learner influences the CATE estimation rates. While the impact of (1) and (2) on the generalization performance of CATE estimators has been extensively investigated, their impact on the validity and efficiency of predictive inference methods for ITE is less well-understood. This forms the central focus of our paper.

Contributions. We propose a CP procedure for predictive inference of ITEs that jointly addresses (1) and (2) in an end-to-end fashion. Our proposed inference strategy applies the standard CP procedure on top of a broad class of CATE meta-learners based on two-stage pseudo-outcome regression. These meta-learners operate by first estimating pseudo-outcomes, i.e., transformed targets that depend on observable variables only, and then regressing the pseudo-outcomes on covariates to obtain point estimates of CATE. We then construct intervals for ITEs by computing the empirical quantile of conformity scores evaluated on pseudo-outcomes in a held-out calibration set. Conformal metalearners address (1) because the distribution of covariates associated with pseudo-outcomes is the same for training and testing data, and they address (2) since the calibration step is decoupled from model architecture, enabling flexible choice of inductive priors and the possibility of re-purposing existing meta-learners and architectures that have been shown to provide accurate estimates of CATE.

Conformal meta-learners inherit the guarantees of CP, i.e., their resulting intervals cover pseudooutcomes on test data with high probability. However, the original CP guarantees do not immediately translate to guarantees on coverage of ITEs. To this end, we develop a unified stochastic ordering framework to study the validity of conformal meta-learners for inference on ITEs. We show that inference with conformal meta-learners is valid if their conformity scores satisfy certain stochastic ordering conditions with respect to 'oracle' conformity scores evaluated on unobserved ITEs. We prove that some of the commonly used meta-learners, such as the doubly-robust learner [20], satisfy a weaker stochastic (or convex) dominance condition which makes them valid for relevant levels of target coverage. Our numerical experiments show that, with careful choice of the pseudo-outcome transformation, conformal meta-learners inherit both the coverage properties of CP as well as the efficiency and point estimation accuracy of their underlying CATE meta-learners.

2 Predictive Inference of Individual Treatment Effects (ITEs)

2.1 Problem setup

We consider the standard potential outcomes (PO) framework with a binary treatment ([21, 22]). Let W ∈ { 0 , 1 } be the treatment indicator, X ∈ X be the covariates, and Y ∈ R be the outcome of interest. For each subject i , let ( Y i (0) , Y i (1)) be the pair of potential outcomes under W = 0 and W = 1 ,

respectively. The fundamental problem of causal inference is that we can only observe the factual outcome, i.e., the outcome Y i = W i Y i (1) + (1 -W i ) Y i (0) determined by W i , but we cannot observe the counterfactual Y i (1 -W i ) . For n subjects, we assume that the data generation process

satisfies the following assumptions: (1) Unconfoundedness: ( Y (0) , Y (1)) ⊥ W | X , (2) Consistency: Y = Y ( W ) , and (3) Positivity: 0 < P ( W = 1 | X = x ) < 1 , ∀ x ∈ X . These assumptions are necessary for identifying the causal effects of the treatment from a dataset { Z i = ( X i , W i , Y i ) } n i =1 . The causal effect of the treatment on individual i , known as the individual treatment effect (ITE), is defined as the difference between the two potential outcomes, i.e., Y i (1) -Y i (0) .

Previous modeling efforts (e.g., [5, 6, 7, 8]) have focused primarily on the (deterministic) conditional average treatment effect (CATE), i.e., τ ( x ) ≜ E [ Y (1) -Y (0) | X = x ] . In this paper, we focus on the (random) ITE as the inferential target of interest. That is, our goal is to infer the ITE for a given subject j given their covariate X j and the observed sample { Z i = ( X i , W i , Y i ) } i =1 .

The distribution of the observed variable Z = ( X,W,Y ) is indexed by the covariate distribution P X , as well as the nuisance functions π ( x ) , µ 0 ( x ) and µ 1 ( x ) defined as follows:

The function π ( x ) is known as the propensity score and it captures the treatment mechanism underlying the data generation process. Throughout this paper, we assume that π ( x ) is known, i.e., data is drawn from an experimental study or the treatment assignment mechanism is known.

Predictive Inference of ITEs. Given the sample { Z i = ( X i , W i , Y i ) } n i =1 , our goal is to infer the ITE for a new individual n +1 with covariate X n +1 . In particular, we would like to construct a predictive band ̂ C ( x ) that covers the true ITE for new test points with high probability, i.e.,

P ( Y n +1 (1) -Y n +1 (0) ∈ ̂ C ( X n +1 )) ≥ 1 -α, (3) for a predetermined target coverage of 1 -α , with α ∈ (0 , 1) , where the probability in (3) accounts for the randomness of the training data { Z i } i and the test point ( X n +1 , Y n +1 (1) -Y n +1 (0)) . Predictive intervals that satisfy the coverage condition in (3) are said to be marginally valid .

2.2 Conformal prediction

Conformal prediction (CP) is a model- and distribution-free framework for predictive inference that provides (finite-sample) marginal coverage guarantees. In what follows, we describe a variant of CP, known as split (or inductive ) CP [12, 13, 14], for the standard regression setup. Given a training dataset D = { ( X i , Y i ) } i , the CP procedure starts by splitting D into two disjoint subsets: a proper training set { ( X j , Y j ) : j ∈ D t } , and a calibration set { ( X k , Y k ) : k ∈ D c } . Then, an ML model ̂ µ ( x ) is fit using the samples in D t and a conformity score V ( . ) is evaluated for all samples in D c as follows:

Q V (1 -α ) ≜ (1 -α )(1 + 1 / |D c | ) -th quantile of V ( ̂ µ ) , (5) where V ( ̂ µ ) = { V k ( ̂ µ ) : k ∈ D c } . Finally, the predictive interval at a new point X n +1 = x is ̂ C ( x ) = [ ̂ µ ( x ) -Q V (1 -α ) , ̂ µ ( x ) + Q V (1 -α ) ] . (6) The interval in (6) is guaranteed to satisfy marginal coverage, i.e., P ( Y n +1 ∈ ̂ C ( X n +1 )) ≥ 1 -α . The only assumption needed for this condition to hold is the exchangeability between calibration and test data [12, 23, 24]. Note that the interval in (6) has a fixed length of 2 Q V (1 -α ) that is independent of x . To enable adaptive intervals, [25] proposed a variant of the CP procedure where the base model is a quantile regression with interval-valued predictions [ ̂ µ α/ 2 ( x ) , ̂ µ 1 -α/ 2 ( x )] , and the conformity score is defined as the signed distance V k ( ̂ µ ) ≜ max { ̂ µ α/ 2 ( X k ) -Y k , Y k -̂ µ 1 -α/ 2 ( X k ) } .

V k ( ̂ µ ) ≜ V ( X k , Y k ; ̂ µ ) , ∀ k ∈ D c . (4) The conformity score measures how unusual the prediction looks relative to previous examples. A common choice of V ( . ) is the absolute residual, i.e., V ( x, y ; ̂ µ ) ≜ | ̂ µ ( x ) -y | . For a target coverage level of 1 -α , we then compute a quantile of the empirical distribution of conformity scores, i.e.,

2.3 Oracle conformal prediction of ITEs

How can we adapt the CP framework for predictive inference of ITEs? In a hypothetical world where we have access to counterfactual outcomes, we can apply the standard CP in Section 2.2 to a dataset of covariates and ITE tuples, D ∗ = { ( X i , Y i (1) -Y i (0)) } i , and compute conformity scores as:

V ∗ k ( ̂ τ ) ≜ V ( X k , Y k (1) -Y k (0) ; ̂ τ ) , ∀ k ∈ D ∗ c , (7) where ̂ τ is an ML model fit to estimate the CATE function τ ( x ) using D ∗ t , and D ∗ = D ∗ t ∪D ∗ c . Wewill refer to this procedure as 'oracle' conformal prediction and to V ∗ k ( ̂ τ ) as the oracle conformity scores. Since the oracle problem is a standard regression, the oracle procedure is marginally valid-i.e., it satisfies the guarantee in (3), P ( Y (1) -Y (0) ∈ ̂ C ∗ ( X )) ≥ 1 -α . However, oracle CP is infeasible since we can only observe one of the potential outcomes (colored in red and blue in (7)), hence we need an alternative procedure that operates only on the observed variable Z = ( X,W,Y ) .

2.4 The two challenges of predictive inference on ITEs

Anaïve approach to inference of ITEs is to split the observed sample { Z i = ( X i , W i , Y i ) } i by the treatment group and create two datasets: D 0 = { ( X i , Y i ) : W i = 0 } i , D 1 = { ( X i , Y i ) : W i = 1 } i , then generate two sets of conformity scores for the nuisance estimates µ 0 and µ 1 as follows:

V (0) k ( ̂ µ 0 ) ≜ V ( X k , Y k (0); ̂ µ 0 ) , ∀ k ∈ D c, 0 , V (1) k ( ̂ µ 1 ) ≜ V ( X k , Y k (1); ̂ µ 1 ) , ∀ k ∈ D c, 1 , (8) where D c, 0 and D c, 1 are calibration subsets of D 0 and D 1 . In order to construct valid predictive intervals for ITE using the conformity scores in (8), we need to reconsider how the two distinct characteristics of CATE estimation, previously discussed in Section 1, interact with the CP procedure:

(1) Covariate shift

̸

The distributions of covariates for treated and untreated subjects differ from that of the target population: P X | W =0 = P X | W =1 = P X , i.e., the following holds for the conformity scores in (8):

̸

̸

̸

(2) Inductive biases

Different choices for the joint model of µ 0 and µ 1 encode different inductive biases that impose different forms of regularization on the implied CATE function, i.e., ̂ τ ( x ) = ̂ µ 1 ( x ) -̂ µ 0 ( x ) [5]. These biases influence the induced distributions P V (0) and P V (1) of the conformity scores in (8).

Covariate shift breaks the exchangeability assumption necessary for the validity of CP. Current methods have primarily focused on (1) with Y (0) and Y (1) as inference targets, and developed approaches for handling covariate shift by reweighting conformity scores [1, 26]. The resulting intervals for POs are then combined to produce intervals for ITEs. However, these method tie the CP procedure to model architecture, requiring inference on nuisance parameters, and hence lose the desirable post-hoc nature of CP. Furthermore, inference on POs is likely to provide conservative ITE intervals, and limits the inductive priors that can be assumed since not all CATE models provide explicit PO estimates.

3 Conformal Meta-learners

In [5], a taxonomy of 'meta-learners' was introduced to categorize different inductive priors that can be incorporated into CATE estimators by structuring the regression models for µ 0 and µ 1 . For example, the T-learner estimates ̂ µ 0 and ̂ µ 1 independently using D 0 and D 1 , while the S-learner models the treatment variable W as an additional covariate in a joint regression model ̂ µ ( X,W ) and estimates CATE as ̂ τ ( x ) = ̂ µ ( x, 1) -̂ µ ( x, 0) . In this Section, we propose an end-to-end solution to (1) and (2) by applying CP on top of CATE meta-learners in a post-hoc fashion, thereby decoupling the CP procedure from the CATE model and allowing direct inference on ITEs. In the next Section, we develop a unified framework for analyzing the validity of this broad class of procedures.

3.1 Pseudo-outcome regression for CATE estimation

We focus on a broad subclass of CATE meta-learners based on two-stage pseudo-outcome regression. These models replace the (unobserved) oracle ITEs with 'proximal' targets that are estimated from observed variables only, and then train an ML model to predict the estimated targets from covariates. The two stages of this general pseudo-outcome regression framework can be described as follows:

<latexi sh

1_b

64="J2

0B3

u9V

dO

GvoI

Z

w

>A

n

c

S

N

E

LXzF+R

P

K

UQ

g

M

y

H

kr

Y

C

/

8

qD

fj

5

W

m

7

p

T

<latexi sh

1_b

64="J2

0B3

u9V

dO

GvoI

Z

w

>A

n

c

S

N

E

LXzF+R

P

K

UQ

g

M

y

H

kr

Y

C

/

8

qD

fj

5

W

m

7

p

T

<latexi sh

1_b

64="Zo uvX/w+

FUVy5GmR

n

r

>A

CL

c

D

S

N

J3

P

d

K

I

Q

OB

fpq

M

zg

H

j

T

7

0

2

8kY

W

E

9

<latexi sh

1_b

64="Uz

2TJdI+OE

j/GMWwrm

Aqno

>

C

3

c

VDLSgNB

X

FP

pZ

H

K

R

ku

Y

v

5

f

8

Q

0

9

y

7

<latexi sh

1_b

64="

AqrLHQMo

v

C3nNEg

f

JzZm

>

c

V

S8

U

/

j

W

wFTy

p

G

9

0

YXd

K

R

7

5

2O

P

k

D

I

u

<latexi sh

1_b

64="UET

CgdrY

7m

Dj qoz

LO

p

>A

B+n

c

V

NS8

3

/

W

wF

y

G

Q

9

0

X

K

HR

5

2

PM

J

k

I

Z

u

f

v

<latexi sh

1_b

64="AJU

gc yrT

uCnM

7If qSKd

8

>

B+X

V

N

E

3

/o

9L

p5

Q

0

2Wz

Z

G

D

YO

w

v

FH

m

k

R

j

P

<latexi sh

1_b

64="F

wSu

Ok

B0TYdMUy5o

8

7GA

>

H

c

V

9

N

EJ2LXz

+R

W

C

j

f

p

n

3

D

K

r

P

m

v

Z

/

g

q

Q

I

<latexi sh

1_b

64="

LmK

Bn

HRocS

zk

2d

v

w

>A

+

V

N

8

EJ3Ur

/G

X

Z

P

W

D

Y

9g

T

u

j

p

7

O

C

F

0

5

Q

I

f

M

q

y

<latexi sh

1_b

64="w

+j gQJ

WN

zKT0S7Pf

X

C9

8

>A

Bn

c

V

E

3Ur q/

5F

LZ

o

D

Y

v2

p

u

H

k

G

M

y

d

m

R

I

O

<latexi sh

1_b

64="Jr cnj+0X3DEV

vP2gW

LKST

>A

C

H

N

F

U

qk

w

o

R

Y

9QB

z

p

8m

u

f

M

/

d

7

yI

G

5

Z

O

Nuisance

estimator

CATE estimator

(a) Pseudo-outcome regression for CATE

(b) Conformal pseudo-intervals for ITE

X

X

̂

µ

0

̂

µ

1

˜

Y

ϕ

=

f

(

Z

;

π,

̂

µ

0

,

̂

µ

1

)

P

V

ϕ

P

V

∗

Q

V

ϕ

(1

-

α

)

Conformity scores

Oracle scores

Algorithm 1: Conformal Meta-learner

Input :

D = { ( X i , W i , Y i ) } n i =1 , X n +1

Output:

̂ C φ ( X n +1 )

1 Split D into D φ , D t and D c ;

2 Pseudo-outcome regression:

3 (1) Estimate ̂ φ = ( π, ̂ µ 0 , ̂ µ 1 ) using D φ ;

4 (2) Regress ˜ Y φ = f ( Z, ̂ φ ) on X using D t ;

5 Conformal pseudo-intervals:

6 Evaluate conformity scores V φ using D c ;

7 Return ̂ C φ = [ ̂ τ ( X n +1 ) ± Q V φ (1 - α )]

Stage 2. We use the nuisance estimates obtained in Stage 1 to create pseudo-outcomes ˜ Y φ that depend only on ̂ φ and the observable variables Z = ( X,W,Y ) , i.e., ˜ Y φ = f ( Z, ̂ φ ) for some function f . The CATE estimate is then obtained by regressing the pseudo-outcome ˜ Y φ on the covariate X . This is typically conducted using a different dataset than the one used to obtain the nuisance estimate ̂ φ .

The general framework described above captures various models in previous literature. We study 3 examples of such meta-learners: Xlearner, Inverse propensity weighted (IPW) learner and doubly-robust (DR) learner.

Pseudo-outcome

Table 1 lists the pseudo-outcomes ˜ Y φ for the three meta-learners: IPW-learner reweights factual outcomes using propensity scores to match CATE, i.e., E [ ˜ Y φ | X = x ] = τ ( x ) ; X-learner uses regression adjustment to impute counterfactuals; DR-learner combines both approaches. DR- and X-learners 1 , coupled with specific architectures for joint modeling of ̂ µ 0 and ̂ µ 1 , have shown competitive performance for CATE estimation in previous studies [5, 16, 20]. The conformal meta-learner framework decouples the CP procedure (Section 3.2) from the inductive priors encoded by these meta-learners, hence it inherits their favorable CATE estimation properties and enables a potentially more efficient direct inference on ITEs as opposed to inference on POs. This addresses challenge (2) in Section 2.4.

3.2 Conformal pseudo-intervals for ITEs

Pseudo-outcome regression is based on the notion that accurate proxies for treatment effects can produce reliable CATE point estimates. This concept can be extended to predictive inference: using CP to calibrate meta-learners via held-out pseudo-outcomes can yield accurate 'pseudo-intervals' for ITEs.

Given a dataset D = { Z i = ( X i , W i , Y i ) } i , we create three mutually-exclusive subsets: D φ , D t and D c . D φ is used to estimate the nuisance parameters φ . Next, the estimates ̂ φ = ( π, ̂ µ 0 , ̂ µ 1 ) are used to transform { Z i = ( X i , W i , Y i ) : i ∈ D t } into covariate/pseudo-outcome pairs { ( X i , ˜ Y φ,i ) : i ∈ D t } which are used to train a CATE model ̂ τ . Finally, we compute conformity scores for ̂ τ on pseudo-outcomes, i.e.,

For a target coverage of 1 -α , we construct a predictive interval at a new point X n +1 = x as follows:

̂ C φ ( x ) = [ ̂ τ ( x ) -Q V φ (1 -α ) , ̂ τ ( x ) + Q V φ (1 -α ) ] , (10) where V φ = { V φ,k ( ̂ τ ) : k ∈ D c } . We call ̂ C φ ( x ) a pseudo-interval . The conformal meta-learner approach is depicted in Figure 1 and a summary of the procedure is given in Algorithm 1.

Input :

D = { ( X i , W i , Y i ) } n i =1 , X n +1

Output:

̂ C φ ( X n +1 )

(a) First-order stochastic dominance

(b) Second-order stochastic dominance

Note that conditional on ̂ φ , the pseudo-outcomes ( X, ˜ Y φ ) in calibration data are drawn from the target distribution, which maintains the exchangeability of conformity scores and addresses covariate shift ( challenge (1) in Section 2.4). However, the conformity scores V φ are evaluated on transformed outcomes, which means that V φ and V ∗ are not exchangeable, even though they are drawn from the same covariate distribution. Consequently, the usual CP guarantees, i.e., P ( ˜ Y φ ∈ ̂ C φ ( X )) ≥ 1 -α , do not immediately translate to coverage guarantees for the true ITE Y (1) -Y (0) . In the next section, we show that for certain choices of the pseudo-outcomes, the corresponding pseudo-intervals can provide valid inferences for ITE without requiring the exchangeability of V φ and V ∗ .

4 Validity of Conformal Meta-learners: A Stochastic Ordering Framework

Under what conditions are pseudo-intervals valid for inference of ITEs? Recall that these intervals are constructed by evaluating the empirical quantile of pseudo-outcome conformity scores. Intuitively, the pseudo-intervals will cover the true ITE if the conformity scores are 'stochastically larger' than the oracle scores in Section 2.3, i.e., Q V φ ( α ) ≥ Q V ∗ ( α ) in some stochastic sense (Figure 1(b)). Hence, to study the validity of conformal meta-learners, we analyze the stochastic orders of V φ and V ∗ , and identify conditions under which pseudo-intervals cover oracle intervals.

Stochastic orders are partial orders of random variables used to compare their location, magnitude, or variability [28, 29]. In our analysis, we utilize two key notions of stochastic order among cumulative distribution functions (CDFs) F and G , which we formally define below.

Definition 1 (Stochastic dominance) F has first-order stochastic dominance (FOSD) on G , F ⪰ (1) G , iff F ( x ) ≤ G ( x ) , ∀ x , with strict inequality for some x . F has second-order stochastic dominance (SOSD) over G , F ⪰ (2) G , iff ∫ x -∞ [ G ( t ) -F ( t )] dt ≥ 0 , ∀ x , with strict inequality for some x .

Definition 2 (Convex dominance) F has monotone convex dominance (MCX) over G , F ⪰ mcx G , iff E X ∼ F [ u ( X )] ≥ E X ∼ G [ u ( X )] for all non-decreasing convex functions u : R → R .

Stochastic ordering is useful tool in decision theory and quantitative finance used to analyze the decisions of utility maximizers with varying risk attitudes [30]. A distribution F has FOSD over G if it is favored by any decision-maker with a non-decreasing utility function, i.e., F is more likely to give higher outcomes than G because its CDF is strictly lower (Figure 2(a)). If F has SOSD over G , then it is favored by risk-averse decision-makers, i.e., f has smaller spread than g and is favored by all decision-makers with a non-decreasing concave utility function [31]. In this case, the CDFs can cross but G is always lower after the last crossing point (Figure 2(b)). F has MCX over G if it is favored by decision-makers with a non-decreasing convex utility-in this case, the CDFs can cross but F is always lower after the last crossing point (See Appendix A for a detailed analysis).

In the following Theorem, we provide sufficient conditions for the validity of conformal meta-learners in terms of the stochastic orders of their conformity scores.

Theorem 1. If ( X i , W i , Y i (0) , Y i (1)) , i = 1 , . . . , n +1 are exchangeable, then ∃ α ∗ ∈ (0 , 1) such that the pseudo-interval ̂ C φ ( X n +1 ) constructed using the dataset D = { ( X i , W i , Y i ) } n i =1 satisfies P ( Y n +1 (1) -Y n +1 (0) ∈ ̂ C φ ( X n +1 )) ≥ 1 -α, ∀ α ∈ (0 , α ∗ ) , if at least one of the following stochastic ordering conditions hold: (i) V φ ⪰ (1) V ∗ , (ii) V φ ⪯ (2) V ∗ , and (iii) V φ ⪰ mcx V ∗ . Under condition (i), we have α ∗ = 1 .

All proofs are provided in Appendix A. Theorem 1 states that if the conformity score V φ of the metalearner is stochastically larger (FOSD) or has a larger spread (SOSD and MCX) than the oracle conformity score, then the conformal meta-learner is valid for high-probability coverage (Figure 3). (This is the range of target coverage that is of practical relevance, i.e., α is typically set to 0.05 or 0.1.)

Conformity score

CDF

Target coverage

Validity region for

conformal meta-learners

Because stochastic (or convex) dominance pertain to more variable conformity scores, the predictive intervals of conformal meta-learners will naturally be more conservative than the oracle intervals.

Whether a meta-learner meets conditions (i)-(iii) of Theorem 1 depends on how the pseudo-outcome, ˜ Y φ = f ( Z ; ̂ φ ) , is constructed. The following Theorem provides an answer to the question of which of the meta-learners listed in Table 1 satisfy the stochastic ordering conditions in Theorem 1.

Theorem 2. Let V φ ( ̂ τ ) = | ̂ τ ( X ) -˜ Y φ | and assume that the propensity score function π : X → [0 , 1] is known. Then, the following holds: (i) For the X -learner, V φ and V ∗ do not admit to a model- and distribution-free stochastic order, (ii) For any distribution P ( X,W,Y (0) , Y (1)) , CATE estimate ̂ τ , and nuisance estimate φ , the IPW- and the DR-learners satisfy V φ ⪰ mcx V ∗ .

̂ Theorem 2 states that the stochastic ordering of V φ and V ∗ depends on the specific choice of the conformity score function V ( X, ˜ Y φ ; ̂ τ ) as well as the choice of the meta-learner, i.e., the pseudo-outcome generation function ˜ Y φ = f ( Z ; ̂ φ ) . The IPW- and DR-learners ensure that, by construction, the pseudooutcome is equal to CATE in expectation: E [ ˜ Y φ | X = x ] = τ ( x ) . This construction enables the IPWand DR-learners to provide unbiased estimates of average treatment effects (ATE) independent of the data distribution and the ML model used for the nuisance estimates ̂ µ 0 and ̂ µ 1 . By the same logic, IPWand DR-learners also guarantee stochastic (convex) dominance of their conformity scores irrespective of the data distribution and the ML model choice, hence preserving the model- and distribution-free nature of the CP coverage guarantees. Contrarily, the X-learner does not use the knowledge of π to construct its pseudo-outcomes, hence it does not guarantee a (distribution-free) stochastic order and the achieved coverage depends on the nuisance estimates ̂ µ 0 and ̂ µ 1 . In Table 2, we list the stochastic orders achieved for different choices of meta-learners and conformity scores. (The analysis of stochastic orders for the signed distance score used in [25] and [32] is provided in Appendix A.)

Key limitations of conformal meta-learners. While conformalized meta-learners can enable valid end-to-end predictive inference of ITEs, they have two key limitations. First, the propensity score π must be known to guarantee model- and distribution-free stochastic ordering of conformity scores. However, we note that this limitation in not unique to our method and is also encountered in methods based on weighted CP [1, 26]. The second limitation is peculiar to our method: exact characterization of α ∗ is difficult and depends on the data distribution. Devising procedures for inferring α ∗ based on observable variables or deriving theoretical upper bounds on α ∗ are interesting directions for future work. Here, we focus on empirical evaluation of α ∗ in semi-synthetic experiments. A detailed comparison between our method and previous work is provided in Appendix B.

5 Experiments

5.1 Experimental setup

Since the true ITEs are never observed in real-world datasets, we follow the common practice of conducting numerical experiments using synthetic and semi-synthetic datasets [1, 8, 19]. We present a number of representative experiments in this Section and defer further results to Appendix C.

Synthetic datasets. We consider a variant of the data-generation process in Section 3.6 in [1] which was originally proposed in [7]. We create synthetic datasets by sampling covariates X ∼ U ([0 , 1] d ) and treatments W | X = x ∼ Bern ( π ( x )) with π ( x ) = (1 + I x (2 , 4)) / 4 , where I x (2 , 4) is the regularized incomplete beta function (i.e., CDF of a Beta distribution with shape parameters 2 and 4). Outcomes are modeled as µ 1 ( x ) = ζ ( x 1 ) · ζ ( x 2 ) and µ 0 ( x ) = γ ζ ( x 1 ) · ζ ( x 2 ) , where γ ∈ [0 , 1] is a parameter that controls the treatment effect, and ζ is a function given by ζ ( x ) = 1 / (1 + exp( -12( x -0 . 5))) .

Avg. interval length

(a) Empirical assessment of stochastic orders

Conformity score

CDF

(Setup A)

DR-learner

IPW-learner

X-learner

CDF

(Setup B)

(c) Performance at different levels of target coverage

Setup A

Setup B

Target coverage

Empirical coverage

RMSE

Empirical coverage (

⍺

=0.9)

Interval Length

RMSE

Inexact

Exact

Naive

X

IPW

DR

Inexact

Exact

Naive

X

IPW

DR

(b) Coverage, efficiency and RMSE for

Setup A

(top) and

Setup B

(bottom)

WCP

CM

1.0

亞·

IB工：··

0.5

-Vy

101101

HI·

■V*

0.0

2.5

0.0

2.5

-5

0

0

2

口

I·

IBI

1.0

·i

+

0.5

1o1

HI

IT:

0.0

-5

0

-5

0

-2

0

2

臣

臣·

0.0

0.5

1.0

0

5

10

15

0.0

0.5

1.0

I

。

1.0 -

15

0.6

J01

P

H·

10

0.4

$1

t·

P

5 -

0.0

.

0.2

口

工

+

T

E工·

1.0

1.8

IOT

工

5

1.6

I·

IBI

00

1.4

0.0

0.5

1.00.0

0.5

1.00.0

0.5

1.0

0.0

0.5

1.0

0

5

10

15

1.4

1.6

We assume that POs are sampled from Y ( w ) | X = x ∼ N ( µ w ( x ) , σ 2 ( x )) , w ∈ { 0 , 1 } and consider a heteroscedastic noise model σ 2 ( x ) = -log( x 1 ) . We define two setups within this model: Setup A where the treatment has not effect ( ζ = 1 ), and Setup B where the effects are heterogeneous ( ζ = 0 ).

Semi-synthetic datasets. We also consider two well-known semi-synthetic datasets that involve real covariates and simulated outcomes. The first is the National Study of Learning Mindsets (NLSM) [3], and the second is the IHDP benchmark originally developed in [8]. Details on the data generation process for NLSM can be founded in Section 2 in [33]. Details on the IHDP benchmark can be found in [6, 8, 16, 19]. Appendix C provides detailed description of both datasets for completeness.

Baselines. We consider baseline models that provide valid predictive intervals for ITEs. Specifically, we consider state-of-the-art methods based on weighted conformal prediction (WCP) proposed in [1]. These methods apply weighted CP to construct intervals for the two POs or plug-in estimates of ITEs. We consider the three variants of WCP in [1]: (1) Naïve WCP which combines the PO intervals using Bonferroni correction, (2) Exact Nested WCP which applies WCP to plug-in estimates of ITEs in treatment and control groups followed by a secondary CP procedure, and (3) Inexact Nested WCP which follows the same steps of the exact version but replaces the secondary CP with conditional quantile regression. (Note that Inexact Nested WCP does not provide coverage guarantees.) For all baselines, we use the same model (Gradient Boosting) for nuisance and pseudo-outcome modeling, and we use the conformal quantile regression method in [25] to construct predictive intervals.

5.2 Results and discussion

Our experimental findings yield the following key takeaways: Firstly, the IPW- and DR-learners demonstrate a robust FOSD (i.e., α ∗ = 1 ) in the majority of experiments, surpassing the MCX conditions outlined in Theorem 2 . Secondly, the DR-learner exhibits superior point estimation accuracy and interval efficiency in most experiment compared to all other baselines that ensure valid inference. Thirdly, the effectiveness of conformal meta-learners depends on the discrepancy between the CDFs of conformity scores and oracle scores -pseudo-outcome transformations that induce thicker tails in the resulting conformtiy scores can cause conformal meta-learners to under-perform.

Empirical assessment of stochastic orders. Figure 4(a) depicts the empirical CDF of the conformity scores V φ and oracle scores V ∗ for the three meta-learners under study (DR-, IPW- and X-learners). These CDFs are averaged over 100 runs of Setups A and B of the synthetic generation process outlined in Section 5.1. (The shaded regions represent the lowest and highest bounds on the empirical

IHDP

NLSM

Coverage

Avg. len.

RMSE

Coverage

Avg. len.

RMSE

Naïve

0.89 (0.02)

18.9 (4.04)

4.73 (1.00)

0.99 (0.00)

4.82 (0.02)

0.15 (0.00)

Exact

0.99 (0.00)

29.8 (7.60)

4.50 (0.97)

0.99 (0.00)

4.92 (0.07)

0.19 (0.01)

Inexact

0.61 (0.04)

8.49 (1.36)

4.61 (0.99)

0.96 (0.00)

2.38

(0.01)

0.18 (0.00)

X

0.65 (0.04)

11.0 (3.04)

3.34

(0.56)

0.27 (0.00)

0.38 (0.00)

0.14

(0.00)

IPW

0.99 (0.00)

112 (23.0)

19.9 (3.44)

0.99 (0.00)

6.48 (0.07)

0.61 (0.02)

DR

0.96 (0.01)

16.7

(3.30)

3.32

(0.53)

0.99 (0.00)

6.24 (0.07)

0.37 (0.01)

Conformity score

CDF

(IHDP)

DR-learner

IPW-learner

X-learner

CDF

(NLSM)

CDFs evaluated across all runs.) In both setups, the conformity scores for the DR- and IPW-learners demonstrate FOSD over the oracle scores with respect to the average CDFs and in almost all realizations. This aligns with the result of Theorem 2, and shows that the stochastic dominance condition achieved in practice is even stronger than our theoretical guarantee since FOSD ( V φ ⪰ (1) V ∗ ) implies the weaker conditions of V φ ⪯ (2) V ∗ and V φ ⪰ mcx V ∗ . On the contrary, the conformity scores of the X-learner are dominated by oracle scores in the FOSD sense. This is not surprising in light of Theorem 2, which indicates that X-learners do not guarantee a distribution-free stochastic order. These observations are also replicated in the semi-synthetic datasets as shown in Figure 5.

Based on Theorem 1, the empirical stochastic orders observed in Figures 4(a) and 5 predict that the IPW- and DR-learners will cover ITEs, whereas the X-learner will not achieve coverage. This is confirmed by the results in Figures 4(b), 4(c) and Table 3. The fact that the IPW- and DR-learners satisfy a stronger FOSD condition is promising because it indicates that the range of validity for these models spans all levels of coverage ( α ∗ = 1 in Figure 3). It also means that a stronger version of Theorem 2 outlining the conditions under which IPW- and DR-learners achieve FOSD could be possible.

Coverage, efficiency and point estimation accuracy. The performance of a predictive inference procedure can be characterized in terms of three metrics: achieved coverage for true ITEs, expected length of predictive intervals, and root-mean-square error (RMSE) in CATE estimates. In most experiments, we find that the DR-learner strikes a balance between these metrics (See Appendix C for further experiments). In Figure 4(b), we can see that the DR-learner outperforms the valid (naïve and exact) WCP procedures in terms of RMSE and interval length, while achiveing the target coverage of 90%. The X-learner outperforms all baselines in terms of RMSE, but as expected, it under-covers ITEs in all experiments. The inexact WCP baseline offers competitive efficiency and calibration, however, in addition to not offering coverage guarantees it also lacks consistency in RMSE performance under different inductive biases (i.e., no treatment effects in Setup A and heterogeneous effects in Setup B ). These performance trends hold true across all levels of target coverage as shown in Figure 4(c).

The semi-synthetic experiments on IHDP and NLSM datasets shed light on when meta-learners may perform poorly. The DR-learner outperforms all baselines on the IHDP dataset in terms of RMSE, interval efficiency, while achieving the desired coverage of 90%. However, we observe that empirical performance depends on how closely the CDF of conformity scores matches the oracle CDF. The DRlearner performance deteriorates when conformity scores have 'very strong' dominance over oracle scores, as observed in the NLSM dataset (Figure 5, bottom). Conversely, when the CDF of conformity scores is a closer lower bound on the oracle CDF, the DR-learner performance is competitive (Figure 4 and Figure 5, top). This is intuitive because if the pseudo-outcome transformation induces significant variability in regression targets, it will result in a lower CDF, poorer accuracy of pseudo-outcome regression, and longer predictive intervals. This is why the DR-learner consistently outperforms the IPW-learner, as it provides a closer approximation of the oracle CDF. Future work could focus on analyzing the gap between pseudo-outcome and oracle score CDFs and designing pseudo-outcome transformations that optimize efficiency while preserving stochastic orders.

6 Conclusions

Estimation and inference of treatment effects is challenging because causal effects are not directly observable. In this paper, we developed a general framework for inference of treatment effects, dubbed conformal meta-learners, that is compatible with any machine learning model. Our framework inherits the model- and distribution-free validity of conformal prediction as well as the estimation accuracy of model-agnostic meta-learners of treatment effects. Additionally, we introduce a new theoretical framework based on stochastic ordering to assess the validity of our method, which can guide the development of new models optimized for both accurate estimation and valid inference.

References

Lihua Lei and Emmanuel J Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology , 83(5):911-938, 2021.

Kosuke Imai and Aaron Strauss. Estimation of heterogeneous treatment effects from randomized experiments, with application to the optimal planning of the get-out-the-vote campaign. Political Analysis , 19(1):1-19, 2011.

David S Yeager, Paul Hanselman, Gregory M Walton, Jared S Murray, Robert Crosnoe, Chandra Muller, Elizabeth Tipton, Barbara Schneider, Chris S Hulleman, Cintia P Hinojosa, et al. A national experiment reveals where a growth mindset improves achievement. Nature , 573(7774):364-369, 2019.

Sheldon Greenfield, Richard Kravitz, Naihua Duan, and Sherrie H Kaplan. Heterogeneity of treatment effects: implications for guidelines, payment, and quality assessment. The American journal of medicine , 120(4):S3-S9, 2007.

Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences , 116(10):4156-4165, 2019.

Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In International Conference on Machine Learning , pages 3076-3085. PMLR, 2017.

Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association , 113(523):1228-1242, 2018.

Jennifer L Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics , 20(1):217-240, 2011.

Ahmed M Alaa and Mihaela Van Der Schaar. Bayesian inference of individualized treatment effects using multi-task gaussian processes. Advances in neural information processing systems , 30, 2017.

Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 , 2019.

Botond Szabó, Aad W Van Der Vaart, and JH Van Zanten. Frequentist coverage of adaptive nonparametric bayesian credible sets. The Annals of Statistics , 2015.

Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of algorithmic randomness. International Conference on Machine Learning , 1999.

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world . Springer Science & Business Media, 2005.

Harris Papadopoulos. Inductive conformal prediction: Theory and application to neural networks . INTECH Open Access Publisher Rijeka, 2008.

Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences . Cambridge University Press, 2015.

Alicia Curth and Mihaela van der Schaar. On inductive biases for heterogeneous treatment effect estimation. Advances in Neural Information Processing Systems , 34:15883-15894, 2021.

Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference , 90(2):227-244, 2000.

Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning , pages 1180-1189. PMLR, 2015.

Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual inference. In International conference on machine learning , pages 3020-3029. PMLR, 2016.

Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. arXiv preprint arXiv:2004.14497 , 2020.

Jerzy S Neyman. On the application of probability theory to agricultural experiments. essay on principles. section 9.(tlanslated and edited by dm dabrowska and tp speed, statistical science (1990), 5, 465-480). Annals of Agricultural Sciences , 10:1-51, 1923.

Donald B Rubin. [on the application of probability theory to agricultural experiments. essay on principles. section 9.] comment: Neyman (1923) and causal inference in experiments and observational studies. Statistical Science , 5(4):472-480, 1990.

Jing Lei, James Robins, and Larry Wasserman. Distribution-free prediction sets. Journal of the American Statistical Association , 108(501):278-287, 2013.

Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association , 113(523):1094-1111, 2018.

Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in Neural Information Processing Systems , 32:3543-3553, 2019.

Ryan J Tibshirani, Rina Foygel Barber, Emmanuel J Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. arXiv preprint arXiv:1904.06019 , 2019.

Daniel G Horvitz and Donovan J Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association , 47(260):663-685, 1952.

Moshe Shaked and J George Shanthikumar. Stochastic orders . Springer, 2007.

Michael Rothschild and Joseph E Stiglitz. Increasing risk: I. a definition. In Uncertainty in Economics , pages 99-121. Elsevier, 1978.

Søren Asmussen and Mogens Steffensen. Risk and insurance . Springer, 2020.

Josef Hadar and William R Russell. Rules for ordering uncertain prospects. The American economic review , 59(1):25-34, 1969.

Henrik Linusson, Ulf Johansson, and Tuve Löfström. Signed-error conformal regression. In Advances in Knowledge Discovery and Data Mining: 18th Pacific-Asia Conference, PAKDD 2014, Tainan, Taiwan, May 13-16, 2014. Proceedings, Part I 18 , pages 224-236. Springer, 2014.

Carlos Carvalho, Avi Feller, Jared Murray, Spencer Woody, and David Yeager. Assessing treatment effect variation in observational studies: Results from a data challenge. Observational Studies , 5(2):21-35, 2019.

Hugh A Chipman, Edward I George, and Robert E McCulloch. Bart: Bayesian additive regression trees. The Annals of Applied Statistics , 4(1):266-298, 2010.

P Richard Hahn, Jared S Murray, and Carlos M Carvalho. Bayesian regression tree models for causal inference: Regularization, confounding, and heterogeneous effects (with discussion). Bayesian Analysis , 15(3):965-1056, 2020.

Suzanne Sniekers and Aad van der Vaart. Adaptive bayesian credible sets in regression with a gaussian process prior. Electronic Journal of Statistics , 9(2):2475-2527, 2015.

Stefan Wager, Trevor Hastie, and Bradley Efron. Confidence intervals for random forests: The jackknife and the infinitesimal jackknife. The Journal of Machine Learning Research , 15(1):1625-1651, 2014.

Maggie Makar, Fredrik Johansson, John Guttag, and David Sontag. Estimation of bounds on potential outcomes for decision making. In International Conference on Machine Learning , pages 6661-6671. PMLR, 2020.

Ying Jin, Zhimei Ren, and Emmanuel J Candès. Sensitivity analysis of individual treatment effects: A robust conformal inference approach. Proceedings of the National Academy of Sciences , 120(6):e2214889120, 2023.

Mingzhang Yin, Claudia Shi, Yixin Wang, and David M Blei. Conformal sensitivity analysis for individual treatment effects. Journal of the American Statistical Association , pages 1-14, 2022.

Yingying Zhang, Chengchun Shi, and Shikai Luo. Conformal off-policy prediction. In International Conference on Artificial Intelligence and Statistics , pages 2751-2768. PMLR, 2023.

Muhammad Faaiz Taufiq, Jean-Francois Ton, Rob Cornish, Yee Whye Teh, and Arnaud Doucet. Conformal off-policy prediction in contextual bandits. arXiv preprint arXiv:2206.04405 , 2022.

Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems , 34:1660-1672, 2021.

Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. arXiv preprint arXiv:2202.13415 , 2022.

Hongxiang Qiu, Edgar Dobriban, and Eric Tchetgen Tchetgen. Distribution-free prediction sets adaptive to unknown covariate shift. arXiv preprint arXiv:2203.06126 , 2022.

Jeanne Brooks-Gunn, Fong-ruey Liaw, and Pamela Kato Klebanov. Effects of early intervention on cognitive function of low birth weight preterm infants. The Journal of pediatrics , 120(3):350359, 1992.

Vincent Dorie, Jennifer Hill, Uri Shalit, Marc Scott, and Dan Cervone. Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition. Statistical Science , 2019.

Appendix A: Technical Proofs

A.1. Equivalent Definitions for Stochastic Orders

The following section introduces alternative definitions for stochastic dominance between two cumulative distribution functions (CDFs), denoted as F and G . These definitions will be used in the proof for Theorem 2 (Section A.4.). Note that these definitions are equivalent to Definition 1 provided in the main text but are expressed in decision-theoretic terms.

Definition A1 (First-order stochastic dominance) F has first-order stochastic dominance (FOSD) over G , F ⪰ (1) G , if and only if E X ∼ F [ u ( X )] ≥ E X ∼ G [ u ( X )] for all non-decreasing (utility) functions u : R → R (Theorem 1.A.3 in [28] proves the equivalence of Definitions 1 and A1).

Definition A2 (Second-order stochastic dominance) F has second-order stochastic dominance (SOSD) over G , F ⪰ (2) G , if and only if E X ∼ F [ u ( X )] ≥ E X ∼ G [ u ( X )] for all non-decreasing concave functions u : R → R (Refer to Theorem 4.A.1 and Eq. (4.A.7) in [28]).

In addition to the alternative definitions of FOSD and SOSD above, we also state the definitions for two additional notions of stochastic ordering that we will use in the proofs.

Definition A3 (Statewise stochastic dominance) Arandom variable X has statewise stochastic dominance (SWD) over a random variable Y , X ⪰ SWD Y , if and only if X ≥ Y for every draw of the bivariate random variable ( X,Y ) ∼ P X,Y . Note that SWD implies FOSD.

Definition A4 (Convex dominance) F has convex dominance (CX) over G , F ⪰ cx G , if and only if E X ∼ F [ u ( X )] ≥ E X ∼ G [ u ( X )] for all convex functions u : R → R .

A.2. Useful Lemmas

Lemma A1. (Pointwise and marginal stochastic orders) For random variables X , Y , and Z , the following conditions hold for any marginal distribution P X :

(i) Y | X = x ⪰ (1) Z | X = x, ∀ x ∈ X ⇒ Y ⪰ (1) Z,

(iii) Y | X = x ⪰ mcx Z | X = x, ∀ x ∈ X ⇒ Y ⪰ mcx Z.

(ii) Y | X = x ⪰ (2) Z | X = x, ∀ x ∈ X ⇒ Y ⪰ (2) Z,

Proof. Recall from the definition of FOSD (Definition A1) that if Y | X = x ⪰ (1) Z | X = x , then:

for all non-decreasing functions u : R → R . Since expectation is a positive linear operator, marginalizing both sides with respect to the density P X preserves the inequality in (A.1):

which recovers the definition of FOSD between random variables Y and Z , i.e.,

and concludes the proof for (i). Note that (A.3) holds for all non-decreasing functions u : R → R , which also includes all non-decreasing concave and convex functions that define SOSD and MCX. Hence, combining (A.1)-(A.3) with Definitions 2 and A2 concludes the proof for (ii) and (iii).

Lemma A2. (Convex dominance of mixture distributions over convex combinations of random variables) Let X ∈ R and Y ∈ R be two real-valued random variables with finite means, and let π ∈ [0 , 1] be a constant. If Z and V are two random variables constructed as:

Z is the absolute value of the convex combination of X and Y , i.e., Z = | πX +(1 -π ) Y | ,

V is a mixture of | X | and | Y | , i.e., V = W | X | +(1 -W ) | Y | , W ∼ Bernoulli ( π ) ,

then we have V ⪰ mcx Z for any π ∈ [0 , 1] and any distribution P ( X,Y ) .

Proof. Recall from Definition 2 that V ⪰ mcx Z if and only if

for all non-decreasing convex functions u : R → R . Given the definition of V above, we have

where the last two inequalities follow from (a) the convexity of u , and (b) the monotonicity of u along with the application of the triangle inequality | πX | + | (1 -π ) Y | ≥ | πX +(1 -π ) Y | .

Lemma A3. (Stochastic dominance and sign changes in CDFs) Let X and Y be two non-negative random variables with distribution functions F and G and with finite means such that E [ X ] ≤ E [ Y ] . Let S -( f ) be the number of sign changes of a function f defined as

where S -[ a 1 , a 2 , . . . , a m ] is the number of sign changes of the sequence [ a 1 , a 2 , . . . , a m ] , with the zero terms being discarded, and the supremum in (A.6) is taken over all sets x 1 < x 2 < . . . < x m such that m< ∞ . Then, F ⪰ (2) [ ⪰ mcx ] G if and only if there exist random variables Z 1 , Z 2 ,. . . , with distribution functions G 1 , G 2 ,. . . , such that Z 1 = st X , E [ Z j ] ≤ E [ Y ] , j = 1 , 2 , . . . , Z j → st Y and E [ Z j ] → E [ Y ] as j →∞ and S -( G j +1 -G j ) = 1 and the sign sequence is { + , -} [ {-, + } ] . Here, = st denotes equality in law and → st denotes convergence in distribution.

Proof. Refer to Theorems 4.A.22 and 4.A.23 in [28] for a full proof.

Corollary A4. (CDF crossings under SOSD and MCX) Let X and Y be non-negative random variables with CDFs F and G with finite means such that E [ X ] ≤ E [ Y ] . Then, if F ⪰ (2) [ ⪰ mcx ] G , we have S -( F -G ) ≥ 1 and the corresponding sign sequence is { . . . , -, + } [ { . . . , + , -} ] . Equivalently, there exists α ∗ ∈ (0 , 1) such that F -1 ( α ∗ ) = G -1 ( α ∗ ) = v ∗ and F ( v ) ≥ [ ≤ ] G ( v ) , ∀ v ≥ v ∗ .

Now observe that for any three integers k < m < n , if S -( G m -G k ) = 1 with a sign sequence of {-, + } [ { + , -} ] , and S -( G n -G m ) = 1 with a sign sequence {-, + } [ { + , -} ] , then by monotonicity of CDFs it follows that in the last point of crossing, G n crosses G k from below [above], i.e., G n -G k has at least one sign change with { . . . , -, + } [ { . . . , + , -} ] . Thus, for any j , the three random variables Z 1 = st X , Z j and Y satisfy S -( G j -F ) = 1 and S -( G -G j ) = 1 with a sign sequence { . . . , -, + } [ { . . . , + , -} ] , which implies that in the last point of crossing, F crosses G from below [above] and concludes the statement of the corollary.

Proof. From Lemma A3, we know that if F ⪰ (2) [ ⪰ mcx ] G then there exists a sequence of random variables Z 1 , Z 2 ,. . . , with distributions G 1 , G 2 ,. . . , such that Z 1 = st X , Z j → st Y , and the CDFs satisfy S -( G j +1 -G j ) = 1 with a sign sequence {-, + } [ { + , -} ] .

A.3. Proof of Theorem 1

Theorem 1. If ( X i , W i , Y i (0) , Y i (1)) , i = 1 , . . . , n +1 are exchangeable, then ∃ α ∗ ∈ (0 , 1) such that the pseudo-interval C φ ( X n +1 ) constructed using the dataset D = { ( X i , W i , Y i ) } n i =1 satisfies

̂ if at least one of the following stochastic ordering conditions hold: (i) V φ ⪰ (1) V ∗ , (ii) V φ ⪯ (2) V ∗ , and (iii) V φ ⪰ mcx V ∗ . Under condition (i), we have α ∗ = 1 .

Proof. Without loss of generality, assume that the conformity scores are sorted, V φ, 1 < . . . < V φ,n c , where n c = |D c | . Recall from (10) that the pseudo-interval is constructed as:

where Q V φ (1 -α ) is the empirical quantile defined as

Combining (A.7) and (A.8), we notice that the following two events are equivalent or equivalently,

{ Y n +1 (1) -Y n +1 (0) ∈ ̂ C φ ( X n +1 ) } ⇐⇒ { V ∗ n +1 ≤ V φ, ⌈ ( n c +1)(1 -α ) ⌉ } . (A.10) Hence, the achieved coverage probability for the pseudo-interval is given by

By exchangeability of the variables ( X 1 , Y 1 (1) -Y 1 (0)) , ..., ( X n +1 , Y n +1 (1) -Y n +1 (0)) , we have

for any integer k . (A.12) holds because ranks are uniformly distributed under exchangeability, i.e., V φ,n +1 is equally likely to fall in anywhere between the calibration points V φ, 1 , . . . , V φ,n c . From (A.12), it follows that for ⌈ ( n c +1)(1 -α ) ⌉ we have:

Now, we examine the three conditions: (i) V φ ⪰ (1) V ∗ , (ii) V φ ⪯ (2) V ∗ , and (iii) V φ ⪰ mcx V ∗ .

(i) FOSD V φ ⪰ (1) V ∗ :

If V φ ⪰ (1) V ∗ , then from Definition 1: F V φ ( v ) ≤ F V ∗ ( v ) , ∀ v . Equivalently, FOSD can be written as:

Since (A.14) applies for any v , then the following holds:

Combining (A.13) and (A.15) we have

V φ,n +1 ⪰ (1) V ∗ n +1 ⇒ P ( V ∗ n +1 ≤ V φ, ⌈ ( n c +1)(1 -α ) ⌉ ) ≥ P ( V φ,n +1 ≤ V φ, ⌈ ( n c +1)(1 -α ) ⌉ ) ≥ 1 -α., which holds for any α , hence α ∗ = 1 . This concludes statement (i).

(ii) SOSD V φ ⪯ (2) V ∗ :

If V φ ⪯ (2) V ∗ , then from Corollary A4 we know that ∃ α ∗ ∈ (0 , 1) where F -1 V φ ( α ∗ ) = F -1 V ∗ ( α ∗ ) = v ∗ and F V ∗ ( v ) ≥ F V φ ( v ) , ∀ v ≥ v ∗ . Equivalently, SOSD can be written as:

Hence, the following holds:

V φ,n +1 ⪯ (2) V ∗ n +1 ⇒ P ( V ∗ n +1 ≤ V φ, ⌈ ( n c +1)(1 -α ) ⌉ ) ≥ P ( V φ,n +1 ≤ V φ, ⌈ ( n c +1)(1 -α ) ⌉ ) ≥ 1 -α., for all 0 ≤ α ≤ α ∗ . This concludes statement (ii).

for all α ≤ α ∗ . Combining (A.13) and (A.17) we have

(iii) MCX V φ ⪰ mcx V ∗ :

Since Corollary A4 holds for SOSD and MCX, the proof is identical to the proof for (ii).

A.4. Proof of Theorem 2

Theorem 2. Let V φ ( ̂ τ ) = | ̂ τ ( X ) -˜ Y φ | and assume that the propensity score function π : X → [0 , 1] is known. Then, the following holds: (i) For the X -learner, V φ and V ∗ do not admit to a model- and distribution-free stochastic order, (ii) For any distribution P ( X,W,Y (0) , Y (1)) , CATE estimate ̂ τ , and nuisance estimate ̂ φ , the IPW- and the DR-learners satisfy V φ ⪰ mcx V ∗ .

Proof. For a CATE estimate ̂ τ , the oracle scores are

Proof of statement (i):

V ∗ ( ̂ τ ) = | ̂ τ ( X ) -( Y (1) -Y (0)) | . (A.18) In what follows, we use the notation V φ ( x ) to denote the conditional random variable V φ | X = x , i.e., the conformity score evaluated at a given feature point. Similarly, we use V ∗ ( x ) to denote the conditional random variable V ∗ | X = x (oracle score evaluated at a given feature point).

Recall from Table 1 that the pseudo-outcome for the X-learner is given by:

˜ Y φ = W ( Y -̂ µ 0 ( X )) + (1 -W )( ̂ µ 1 ( X ) -Y ) . (A.19) With V φ ( ̂ τ ) = | ̂ τ ( X ) -˜ Y φ | , the conformity score for the X-learner can be written as: V φ ( ̂ τ ) = | ̂ τ ( X ) -W ( Y -̂ µ 0 ( X )) -(1 -W )( ̂ µ 1 ( X ) -Y ) | . (A.20) To prove statement (i), it suffices to find counter-examples for estimators ̂ τ , ̂ µ 0 and ̂ µ 1 , or data distributions that cause violations of FOSD, SOSD and MCX between V φ and V ∗ . Let u : R → R be a non-decreasing function. The expected value of u ( V φ ( x )) can be evaluated as:

̂ ̂ Westart by showing that there exists nuisance and CATE estimates and a data distribution for which V φ does not have FOSD on V ∗ . Assume that u is non-decreasing concave. By Jensen inequality, we have:

E [ u ( V φ ( x ))] ≤ E [ u ( π ( x ) · | ̂ τ ( x ) -( Y (1) -̂ µ 0 ( x )) | +(1 -π ( x )) · | ̂ τ ( x ) -( ̂ µ 1 ( x ) -Y (0)) | )] . Now consider a nuisance estimates of ̂ µ 0 = 0 and ̂ µ 1 = 0 , a data distribution for which Y (0) > 0 and Y (1) < 0 almost surely, and a CATE estimator τ ( x ) > 0 , ∀ x ∈ X , then we have

̂ and hence V ∗ ( x ) ⪰ (2) V φ ( x ) for all x , and by Lemma A1, we have V ∗ ⪰ (2) V φ . The last inequality in (A.22) holds because u is non-decreasing and (1 -π ( x )) Y (1) -π ( x ) Y (0) is always non-positive when Y (0) > 0 and Y (1) < 0 (almost surely). Since SOSD is a necessary condition for FOSD, then it follows that V φ does not have FOSD over V ∗ in this counter-example.

Now we show that there exists nuisance and CATE estimates and a data distribution for which V φ does not have MCX over V ∗ . This follows directly from the previous counter-example. Since V ∗ ⪰ (2) V φ implies that -V φ ⪰ mcx -V ∗ (Theorem 4.A.1. in [28]), then V φ does not have MCX over V ∗ .

Finally, we show that there exists nuisance and CATE estimates and a data distribution for which V ∗ does not have SOSD over V φ . Note that E [ V ∗ ] ≥ E [ V ϕ ] is a necessary condition for V ∗ ⪰ (2) V φ (See Eq. (4.A.6) in [28]). To compare E [ V ∗ ] and E [ V ϕ ] , we rewrite E [ V ϕ ( x )] as:

For nuisance estimates that satisfy ̂ µ 1 ( x ) < Y (1) and ̂ µ 0 ( x ) > Y (0) almost surely, it follows that (1 -π ( x ))( Y (1) -µ 1 ( x )) + π ( x )( µ 0 ( x ) -Y (0)) is always positive and hence we have:

Proof of statement (ii):

From Table 1, the pseudo-outcomes for the IPW- and DR-learners can be written as:

where Y π and Y 1 -π are defined as follows:

Hence, applying (A.26) to the definition of the oracle score, we can write V ∗ ( x ) as follows:

From the expressions in (A.24) and (A.27), we can see that V φ ( x ) is a mixture of the absolute values of the two random variables ( ̂ τ ( x ) -Y π ( x )) and ( ̂ τ ( x ) -Y 1 -π ( x )) with mixing proportions π ( x ) and 1 -π ( x ) , and V ∗ ( x ) is the absolute value of the convex combination of the same random variables, ( ̂ τ ( x ) -Y π ( x )) and ( ̂ τ ( x ) -Y 1 -π ( x )) , with weights π ( x ) and 1 -π ( x ) . Applying Lemma A2, it follows that V φ ( x ) ⪰ mcx V ∗ ( x ) , ∀ x ∈ X , and from Lemma A1 we have that V φ ⪰ mcx V ∗ .

Stochastic orders for the signed distance conformity score:

We have shown that for the absolute residual conformity score, V φ ( ̂ τ ) = | ̂ τ ( X ) -˜ Y φ | , the DR- and IPW-learners guarantee V φ ⪰ mcx V ∗ irrespective of the data distribution and underlying models. We now study the stochastic orders achieved by the signed distance conformity score proposed in [32], defined as: V φ ( τ ) = τ ( X ) -Y φ . Let u be a non-decreasing concave function, then we have:

̂ ̂ where the inequality follows from the concavity of u ( . ) and the last equality follows from (A.26). From Definition 1, (A.28) implies that V ∗ ⪰ (2) V φ (Table 2). Note, however, that the construction of predictive intervals with the signed error distance does not follow (5) since the signed distance score does not sort absolute errors from largest to smallest, but it sorts conformity scores from maximum positive error to maximum negative error. Hence, Theorem 1 does not apply to the signed distance error and the SOSD condition V ∗ ⪰ (2) V φ does not guarantee coverage.

Another more commonly used variant of the signed distance score was proposed in [25]. Given a quantile regression model [ ̂ τ l , ̂ τ h ] , [25] uses a variant of the signed error score defined as:

The construction of the predictive intervals in [25] follows (5). Note that max { x, y } can be written as:

Applying (A.30) to (A.29), we can rewrite the conformity scores as:

Similarly, the oracle scores can be written as:

∣ ∣ Let δτ -= ̂ τ l -̂ τ h 2 , δτ + = ̂ τ l + ̂ τ h 2 , and let u be a non-decreasing convex function, then we have:

̂ where the first inequality follows from the convexity of u , the second inequality is an application of the triangle inequality, and the last equality follows from (A.26). Following Definition 2, (A.33) implies that V φ ⪰ mcx V ∗ when the base models apply quantile regression and the conformity scores follow the signed distance score in [25]. Hence, the coverage guarantee in Theorem 1 applies to conformal meta-learners based on conditional quantile regression.

Appendix B: Literature Review

In this Section, we provide a comprehensive overview of the relevant literature. We categorize the previous literature into three distinct strands: (1) Bayesian methods for modeling individualized causal effects, (2) frequentist methods for inference on parameters pertaining to individualized causal effects, and (3) conformal methods for predictive inference of ITEs.

(1) Bayesian methods. Predictive inference on ITEs has been traditionally conducted using Bayesian methods. These methods place a prior distribution over the nuisance parameters µ 0 and µ 1 , and then estimate the CATE function through the posterior distribution computed conditional on the observational dataset D = { ( X i , W i , Y i ) } i . More formally, Bayesian procedures operate as follows:

where Π 0 and Π 1 are priors over function spaces, and ϵ ∼ N (0 , σ 2 ) . Given the dataset D , the posterior distributions over the two nuisance functions are used to estimate CATE as follows:

̂ τ ( x ) = E µ 1 ∼ Π 1 |D [ µ 1 ( x )] -E µ 0 ∼ Π 0 |D [ µ 0 ( x )] . (A.35) Furthermore, the posterior distributions Π 0 |D and Π 1 |D can be used to construct predictive intervals on ITEs through the posterior credible intervals of P Π 0 ( µ 1 ( x ) |D ) and P Π 1 ( µ 1 ( x ) |D ) , i.e.,

P µ 0 ,µ 1 ∼ Π 0 , Π 1 |D ( Y (1) -Y (0) ∈ ̂ C ( X ) | X = x ) = 1 -α. (A.36) Different incarnations of the Bayesian framework correspond to different choices of the prior (Π 0 , Π 1 ) . Bayesian Additive Regression Trees (BART) [34] is one popular model for Bayesian nonparametric regression that was later adapted for causal effect estimation and inference [8, 35]. While [8] applies BARTfor estimation of CATE and inference of ITE out-of-the-box, [35] introduces new regularization techniques to incorporate different forms of inductive biases on the CATE target parameter. Another

popular choice of the prior (Π 0 , Π 1 ) is based on Gaussian processes where inductive biases are incorporated through different choices of the kernel function of a reproducible kernel Hilbert space [9].

Unlike the conformal framework, Bayesian methods do not provide coverage guarantees on their credible intervals. Frequentist coverage can be achieved in an asymptotic sense by Bayesian method under certain technical conditions [36]. Existing methods typically do not provide any finite-sample guarantees and the achieved empirical coverage depends on the choice of the prior. Models such as BARTare typically applied with a default prior for all datasets, undermining their achieved coverage in diverse experimental setup. In [1], it was shown that Bayesian methods have a tendency to undercover ITEs in some data generation processes (e.g., in high-dimensional covariate spaces).

Bayesian methods also suffer from two key advantages compared to the conformal framework. First, Bayesian inference procedures and the computation of posterior distributions are typically tailored to specific model architectures. Generalizing exact Bayesian inference to arbitrary model architectures, including modern deep learning architectures, is not straightforward. Moreover, Bayesian inference is conducted either using computationally exhaustive Monte Carlo method (e.g., BART) or through expensive evaluation of an analytical posterior distribution that does not scale well with the number of training points (e.g., Gaussian process posteriors are O ( n 3 ) ).

(2) Frequentist methods. Another model-specific approach to inference on causal parameters is the Causal Forest model in [7]. This procedure uses constructs pointwise confidence intervals on the CATE function τ ( x ) using conditional variance estimates based on the infinitesimal jackknife [37]. These intervals provide asymptotically valid coverage of CATE under mild regularity assumptions, but they provide no (asymptotic or finite-sample) guarantees on coverage for ITEs.

Another non-Bayesian approach for predictive inference of ITEs uses a quantile regression approach to train models that construct upper and lower bounds on the observed potential outcomes using a pinball loss [38]. While quantile regression can be repurposed for different model architectures, this approach does not provide finite-sample coverage guarantees and is empirically found to undercover outcomes, hence it is typically supplemented with a conformal prediction procedure [25].

(3) Conformal methods. The application of conformal prediction to ITE inference traces its origins back to the covariate shift problem. In [26], a variant of conformal prediction was proposed to address problem setups where training and testing data are drawn from different distributions, breaking the exchangeability assumptions required for valid inference.

In the treatment effect estimation setup, covariate shift arises because the distributions of the treatment groups differ from that of the target population. Recall from (8) that the conformity scores for the nuisance estimates µ 0 and µ 1 can be written as follows:

̂ ̂ ̂ ̂ Both sets of conformity scores are sampled from the populations P X | W =0 and P X | W =1 , respectively, and hence they are not exchangeable with the conformity scores sampled from target covariate distribution P X . To retain the validity of intervals constructed using the sample quantiles of conformity scores, [26] introduces the notion of 'weighted exchangeability' and proposes a weighted sample quantile with weights based on the likelihood ratios between training and testing distributions. [1] uses this method to construct valid intervals for potential outcomes and then proposes different approaches for combining these intervals to construct intervals for ITEs. Other work utilized the conformal prediction framework to conduct sensitivity analysis by devising hypothesis tests for the presence of hidden confounders [39, 40]. In [41, 42], conformal prediction has been utilized to address the closely related problem of off-policy evaluation. More recently, there has been further work developing methods for conformal prediction under covariate shift [43, 44, 45].

The key difference between our work and the aforementioned approaches is that we do not apply conformal prediction to the nuisance parameters, and instead conduct inference directly on the target parameter. We do so by applying conformalization with respect to transformed targets ˜ Y φ for each data point in D , and this way the covariate shift problem does not arise since ( X, ˜ Y φ ) is sampled from the same distribution in calibration and testing data. One way to think about our approach as opposed to weighted conformal prediction (WCP) is that ours applies re-weighting to the outcomes whereas WCP re-weights the conformity scores. By moving the re-weighting step to the outcomes, we decouple the conformal procedure from the model, allowing for more flexible choice of inductive priors and accommodating models that do not estimate potential outcomes directly.

Appendix C: Experimental Setup and Results

C.1. Overview of the experimental setup

In all experiments, we used a Gradient Boosting model with 100 trees as the base model for nuisance estimation and quantile regression on pseudo-outcomes. We used the same base model in conformal meta-learners as well as the weighted CP (WCP) baselines to control for potential performance differences resulting from different choices of base models. Unless otherwise stated, all experiments followed a 90%/10% train/test split of each dataset, and each training set with further split into a 75%/25% proper training/calibration sets. All performance metrics (empirical coverage, expected interval width and RMSE) are evaluated on the testing data and averaged over 100 runs. The target coverage in all experiments was set 1 -α = 0 . 9 .

We used the official implementations of the (naïve, exact and inexact) WCP baselines in the R package available at https://github.com/lihualei71/cfcausal. We executed these baselines through rpy2 (https://rpy2.github.io/) wrappers within our Python codebase.

C.2. Details of the IHDP and NLSM datasets

In this Section, we provide details of the semi-synthetic datasets used in the experiments in Section 5.

IHDP dataset. The Infant Health and Development Program (IHDP) was a multi-site randomized controlled trial that studied the effect of early educational intervention on the cognitive development of low birth weight premature infants [46]. The trial targeted low-birth-weight, premature infants, and administered an intensive high-quality child care and home visits from a trained provider to the treatment group. In [8], a semi-synthetic benchmark was developed based on the covariate data in IHDP with simulated outcomes to assess the performance of different CATE estimation and ITE inference models when ground-truth counterfactual are known. The dataset comprised 25 covariates (6 continuous and 19 binary variables) capturing aspects related to children and their mothers. The benchmark dataset excluded a non-random subset of treated individuals [8]. The final dataset consists of 747 samples (139 treated and 608 control). The outcomes of all individuals were simulated whereas the treatment assignments followed the true assignments in the study.

The simulated outcomes model in IHDP followed simulation Setup B in [8]. In this Setup, the potential outcomes were simulated as follows: Y (0) ∼ N (exp(( X + W ) β ) , 1) and Y (1) ∼ N (exp( Xβ ) -ω, 1) , where W is an offset matrix, ω is set so that the ATE on the treated is always equal to 4, and the coefficients β are sampled independently at random from (0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4) with probabilities (0 . 6 , 0 . 1 , 0 . 1 , 0 . 1 , 0 . 1) . In our experiments, we used the 100 realization of the training and testing data released by [6] in https://www.fredjo.com/files/ihdp_npci_1-100.train.npz and https://www.fredjo.com/files/ihdp_npci_1-100.test.npz.

NLSM dataset. The National Study of Learning Mindsets (NLSM) is a large-scale randomized trial that studied the effect of a behavioral intervention, designed to instill students with a growth mindset, on the academic performance of students in secondary education in the US [3]. In [33], a semisynthetic benchmark was developed based on the covariates of the NLSM study. Unlike IHDP, this benchmark does not use the real covariates of NLSM but rather buids a synthetic process to emulate NSLM in terms of the covariate distributions, data structures, and effect sizes. The final dataset comprised 10,000 data points. The data was generated based on 5 principles (See Section 2 and Appendix A in [33]). The principles are: (1) ATEs should be well-estimated from synthetic data by any reasonable procedure, (2) variability in CATE should be relatively modest, (3) treatment effect heterogeneity can be approximately recovered given complete knowledge of the correct functional form, (4) no additional unmeasured treatment effect moderation at the individual level, and (5) unexplained treatment effect heterogeneity at the group level should be present at reasonable levels.

C.3. Further experiments using the ACIC dataset

In addition to the IHDP, NLSM and fully-synthetic experiments, we also conducted further experiments on 77 datasets from the 2016 Atlantic Causal Inference Competition (ACIC2016) [47]. We followed the processing steps in [16], creating a dataset of 4,802 data points and 55 covariates. Out of this base dataset, we created 770 simulation setups using the 77 settings proposed in [47]. The 77 settings represent different levels of complexity of response surfaces, varying degrees of confounding,

1.00

0.95

Average Coverage Levels

CM-DR

CM-X

0.90

CM-IPW

WCP-Naive

WCP-Exact

WCP-Inexact

0.85

0.80

0

10

20

30

40

50

60

70

30

25

Average Interval Widths

CM-DR

CM-X

CM-IPW

WCP-Naive

WCP-Exact

ytwy/wA/W/AA

WCP-Inexact

10-

MM

0

10

20

30

40

50

60

70

10-

8

Average RMSE

CM-DR

6

CM-X

CM-IPW

WCP-Naive

WCP-Exact

4

WCP-Inexact

2

0

0

10

20

30

40

50

60

70

Experiment number

overlap and effect heterogeneity. All settings use the same covariates but simlate different treatment assignments and response surfaces. We created 10 realizations of each of the 77 settings and average the performance of all baselines across the 10 runs.

The empirical coverage, average interval width and RMSE results across all 77 experiments are provided in Figure C.1. The comprehensive experiments in Figure C.1. align with the overall conclusions of the experiments in the main paper. For a target coverage of 1 -α = 0 . 9 , all models achieved the target coverage with the exception of the X-learner. All models achieving the target coverage displayed conservative predictive intervals, acheiving coverage levels that exceed the 0.9 target. In terms of efficiency, the DR-learner and WCP-Inexact achieved comparable interval widths, significantly outperforming all other valid procedures (IPW-learner, WCP-exact and WCP-Naïve). Finally, the the DR-learner outperformed all valid procedure in terms of RMSE in almost all experiments, providing point estimation accuracy that is on par with the X-learner.

These results align with the overall conclusions of the experiments in the main paper. The DR-learner generally inherits the accurate point estimation of its underlying CATE meta-learner, provides a coverage guarantee for practically relevant values of target coverage and achieves this coverage with competitive efficiency making it a favorable approach for both point estimation of CATE and predictive inference of ITE.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a meta-learner-based approach for conformal inference of individual treatment effects (ITEs). Instead of predicting the missing outcome (the counterfactual), the proposed approach uses plug-in pseudo outcomes as the inference proxy. The validity of ITE inference is based on careful analysis of the statistical dominance of the ITE and the inference proxy. In this way, one no longer needs to adjust for the covariate shifts in applying conformal inference, and the validity holds in a model-free fashion (although the coverage guarantee is more limited). Overall, this paper finds a novel approach to predictive inference of ITEs, and the method works well in synthetic and real-world datasets.\n\nSTRENGTHS:\n1. Novel contribution to an important problem \n\nThis paper makes novel contributions, with a new methodology, to an important problem: uncertainty quantification and predictive inference of individual treatment effects. The proposed methodology significantly differs from the PO-prediction-based approaches in the literature, and the analysis contains interesting findings/perspectives. I believe these are important contributions to the literature of conformal inference, and might inspire more developments in the future. \n\n2. Good writing quality and clarity \n\nThis paper is well-written and enjoyable to read. The challenges are clearly stated and the contributions are easy to capture. I still have some questions/suggestions for improving the presentation; please see my questions.\n\nWEAKNESSES:\n1. The analysis is somewhat limited to the known propensity case. \n\nSince previous ITE methods already achieve validity in the known propensity case, the contribution of this paper is somewhat marginal in the sense that it does not push the limit of how well/ how much we can do for this problem (although i still appreciate the novel perspective provided here). However, the authors only mentioned the challenge of unknown propensity score at the end of the paper, which is a bit unsatisfactory. \n\n2. The setting in synthetic datasets should be stated more clearly. \n\nI think the most challenging part of ITE inference is when both POs are missing, and this is where the proposed approach becomes most interesting. However, when reading the experiments part I am not sure which situation we are in. Are we inferring the ITE for both POs missing case, and what is the calibration data here? I think further clarification will be helpful. \n\n3. The benefits of this method in experiments is a bit unclear. \n\nThe experiment part contains rich information. However, I am not sure I fully understand in which cases the proposed method performs the best - is it particularly beneficial for both-PO-missing case, or one-PO-missing part? I thought in the latter case the previous approach should suffice, or it still can be improved due to no covariate shift adjustment? I also do not understand why IPW and DR lead to so long prediction intervals for the NLSM dataset. It is mentioned that \"conformity scores have “very strong” dominance over oracle scores\", but it is difficult to understand in which cases this might happen (strong signal? large noise? high nonlinearity?).\n\nQUESTIONS:\n1. What if the propensity score is unknown? \n\nA natural question is whether the proposed method is robust to fitted propensities in the pseudo outcomes. I guess the dominance condition should still be (approximately) reasonable if the estimation is accurate (at least for DR?) since the expectation is the same, and the linear relation still holds? Currently the discussion on this point is a bit passive. But given the existing methods' good performance in this aspect, I was wondering whether more can be said about this problem. \n\n\n2. Do we have a sense of how large $\\alpha^*$ is for conditions (ii) and (iii) in Theorem 1, or practically used conformity scores? \n\nIt is also discussed that $\\alpha^*$ is difficult to know, which I think is a hurdle to the current method. While it's said that $\\alpha^*$ is evaluated in the experiments, I didn't understand how this can be done. Can we have some examples of the $\\alpha^*$ under some parametric models + gaussian noise? Can we know how it changes with characteristics of the distributions? \n\n\n3. A clarification question: Does it only apply to the setting without covariate shift? \n\nFrom my understanding the method requires that the calibration data are exchangeable with the future point. This means for both-PO-missing parts we should also find such individuals in the calibration data, and similarly for single-PO-missing ones. If this is the case, it will be helpful to explicitly mention this because it would be helpful to practitioners. \n\n\n4. Clarification in the presentation of the experiment part. \n\nWhen reading the experiments part I am not sure which situation we are in. Are we inferring the ITE for both POs missing case, and what is the calibration data here? Did you evaluate the performance for the case with only one PO missing? \n\n\n5. What affects the relative performance of meta-learner-based approach? \n\nI see that the performance of the proposed approach is not always the best, and it can yield very long intervals for the NLSM dataset. It is important for practitioners to understand in which cases this method will perform well or poorly. Is it possible to give some guidance on how the performance changes with the data generating process, such as the coupling of POs, the strength of signal/noise? \n\n\n\n---\nMinor points & typos:\n\n1. Should condition (ii) in Theorem 1 be $V_\\varphi \\succeq_{(2)} V^*$? Seems the current order is opposite to other types of conditions.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper introduces a framework for inferring treatment effects, merging concepts of conformal prediction and meta-learner. This framework is characterized by its distribution-free validity and coverage guarantees. The authors utilize stochastic ordering techniques to substantiate the framework's validity.

STRENGTHS:
The paper is well written and sounded. The innovative approach of employing stochastic ordering to validate the use of pseudo outcomes in conformal prediction is particularly noteworthy.

WEAKNESSES:
The concept of employing stochastic ordering is innovative; however, its practical application is impeded by the fact that the assumptions are inherently uncheckable

In comparison to the method proposed in [1], the current method exhibits better performance only in terms of the RMSE of the CATE. However, when considering coverage and average length, it does not surpass the performance of the inexact method. 


[1] Lihua Lei and Emmanuel J Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(5):911–938, 2021.

QUESTIONS:
(1). Assuming the propensity score is unknown, could we estimate it first and then follow the same procedure outlined in the paper?

(2). For the experimental results, it seems your method can be overly conservative.

(3). Figure 4(c) could be adjusted to resolve the issue of elements overlapping.

LIMITATIONS:
See weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper adresses individual treatment effect (ITE) inference using conditional average treatment effect (CATE) meta learners. \n\nThe authors propose to wrap the meta-learners with a conformal prediction step so as to obtain valid confidence intervals for the estimated treatment effect. Since the CATE estimator induces a distribution shift wrt conformity scores, the coverage guarantee does not immediately translate into the desired coverage of actual ITE. In a who can do more can do less spirit, the authors point out that ITE coverage is ensured if conformity scores obtained from pseudo-outcomes stochastically dominate the scores obtained from actual « oracle » outcomes. The authors succeed at providing stochastic dominance results for several stochastic order / meta-learner pairs.\n\nSTRENGTHS:\n- the paper is very clear and pleasant to read\n- the tackled problem is challenging and impactful\n- the proposed solution is backup by both theoretical and experimental results\n\nWEAKNESSES:\n- the method is by-design over-conservative\n\nQUESTIONS:\nIn my humble opinion, this is a very good paper. The problem is well formulated, the proposed solution is clearly exposed and easy to reproduce. The solution is theoretically proved to work (under some clearly stated conditions) which is confirmed by numerical studies.\n\nFurthermore, the authors have correctly self-identified the limitations of their approach (knowledge of propensity score and hard to assess level range). \n\nMy only remark is that I would have like to see a synthetic experiment how far is the proposed solution to the optimum/oralce especially in terms of interval length.\n\nLIMITATIONS:\n- knowledge of the propensity score is required\n- validity range of level $\\alpha$ not always easy to determine\n- over-conservatism of the prediction intervals\n\nThe two first points are already discussed by the authors in the submission.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with confidence intervals for the ITE prediction task, where the authors propose to use the framework of conformal prediction for meta learners. The learning of nuisance models for generating pseudo outcomes is done one separate data split, and the final regression models trained to predict pseudo outcomes are evaluated on a validation set to compute the empirical calibration distribution. The proposed approach for calibration removes the issues of covariate shift and inductive biases on nuisance models with the prior approaches of weighted conformal prediction. The authors also provide guarantees on when the confidence intervals for ITE with pseudo outcomes cover the confidence interval for ITE with true outcomes; and further validate their findings with synthetic and semi-synthetic datasets.\n\nSTRENGTHS:\n* ITE prediction is an extremely relevant problem in causal inference as population level effects (ATE, CATE) would not necessarily transfer to the individual level effects. Further, having an estimate of uncertainity along with ITE prediction is crucial for high-stakes application scenarios. Hence, the approach of conformal meta learners is highly relevant for the current challenges in causal inference. \n\n* The proposed combination of conformal prediction with meta learners is novel to the best of my knowledge. Further, the ITE coverage results with pseudo outcomes for standard meta learners (DR Learner) are quite significant as it is non-trivial to understand when the predicted confidence intervals with observed information can represent the confidence intervals with counterfactual information. \n\n* The proposed conformal meta learner approach is technically sound with theoretical proofs on their validity and empirical justification on a decently diverse set of benchmarks.\n\nWEAKNESSES:\nI do not think the paper has any serious weaknesses, but I have listed some of them in the questions section ahead. One major suggestion is that the paper could be written with better clarification for the section on conformal meta learner being robust to covariate shift and inductive biases. The arguments are stated informally and it is a bit hard to follow.\n\nQUESTIONS:\n* I do not understand the argument of authors that conformal prediction with meta learners is not affected by the choice of inductive biases of nuisance models. The pseudo outcomes are a function of the nuisance models, hence the predicted effect (ITE), which is learnt using pesudo outcomes, indeed depends indirectly on the nuisance models. Hence, the choice of nuisance models would affect the empirical calibration distribution, consequently the confidence intervals for ITE. \n\n* I do not follow the argument of authors that conformal meta learners are not affected by the covariate shifts. The covariate shift between the control and treatment population should still affect the computation of pseudo outcomes with finite samples; hence the confidence intervals.\n\n* Please clarify the assumption of knowing the true propensity model. What challenges would the authors in their analysis if they do not make this assumption.\n\nLIMITATIONS:\nYes, the authors have addressed any potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-31"}