Unveiling the Potential of Robustness in Selecting Conditional Average Treatment Effect Estimators
Authors: Yiyan HUANG
Cheuk Hang LEUNG
WANG Siyi
YIJUN LI
Qi WU
Venue: NeurIPS 2024 poster (2024)
Keywords: Causal inference
Treatment effect
CATE estimator selection
Model Validation
Robustness
Primary area: causal_inference

Abstract: The growing demand for personalized decision-making has led to a surge of interest in estimating the Conditional Average Treatment Effect (CATE). Various types of CATE estimators have been developed with advancements in machine learning and causal inference. However, selecting the desirable CATE estimator through a conventional model validation procedure remains impractical due to the absence of counterfactual outcomes in observational data. Existing approaches for CATE estimator selection, such as plug-in and pseudo-outcome metrics, face two challenges. First, they must determine the metric form and the underlying machine learning models for fitting nuisance parameters (e.g., outcome function, propensity function, and plug-in learner). Second, they lack a specific focus on selecting a robust CATE estimator. To address these challenges, this paper introduces a Distributionally Robust Metric (DRM) for CATE estimator selection. The proposed DRM is nuisance-free, eliminating the need to fit models for nuisance parameters, and it effectively prioritizes the selection of a distributionally robust CATE estimator. The experimental results validate the effectiveness of the DRM method in selecting CATE estimators that are robust to the distribution shift incurred by covariate shift and hidden confounders.

Unveiling the Potential of Robustness in Selecting Conditional Average Treatment Effect Estimators
Yiyan Huang
Cheuk Hang Leung City University of Hong Kong chleung87@ci edu.
The Hong Kong Polytechnic University yiyhuang3-comy c0my. cityu.edu
edu hk
Siyi Wang
Yijun Li
City University of Hong Kong siyi. wang@my cityu. edu. hk
City University of Hong Kong yijunli5- cityu.eeee edu, hk
Qi Wu f City University of Hong Kong qi. wu@cityu edu. hk
Abstract
The growing demand for personalized Decision-making has led to a surge of interest in estimating the Conditional Average Treatment Effect (CATE). Various types of CATE estimators have been developed with advancements machine learning and causal inference. However, selecting the desirable CATE estimator through conventional model validation procedure remains impractical due to the absence of counterfactual outcomes in observational data. Existing approaches for CATE estimator selection, such as plug-in and pscudo-outcome metrics, face challenges. First, they must determine the metric form and the underlying machine learning models for fitting nuisance parameters (e.g., outcome function, propensity function, and plug-in learner). Second, they lack specific focus selecting a robust CATE estimator. To address these challenges, this paper introduces a Distributionally Robust Metric (DRM) for CATE estimator selection. The proposed DRM is nuisance- freee eliminating the need to fit models for nuisance parameters, and effectively prioritizes the selection CATE estimator. The experimental results validate the effectiveness of the DRM method in selecting CATE estimators that are robust to the distribution shift incurred by covariate shift and hidden confounders.
distributionally robust
Introduction
The escalating demand for decision-making has sparked an increasing interest in Causal Inference across various research domains, such as economics 23
13,
43
1. statistics 770
49
26
41],
healthcare 78
27
61
g 42), and financial application [ 15 37
21
35 24
50
The primary
goal in personalized decision-making is to quantify the causal effect of a specific treatment (or policy/intervention) on the target outcome, and understanding such causal effects is closely connected with identifying the Conditional Average Treatment Effect (CATE). In observational studies, identify- ing the CATE faces a significant and fundamental challenge: the absence of counterfactual knowledge. According to Rubin Causal Model [64), the CATE is determined by comparing potential outcomes
"The co-first authors.
t "oorrsppndiigg author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

under different treatment assignments (i.e., treat and control) for specific covariate. Nonetheless, in real-world applications. we can only observe the potential outcome under the actual treatment (i.e.. factual outcome), while the potential outcome under the alternative treatment (i.e., counterfactual outcome) remains unobserved. The unavailability of the counterfactual outcome widely recognized as the fundamental problem in causal inference [33], making it difficult to accurately determine the true value of the CATE.
The advancement of machine learning (ML) has opened up promising opportunity to improve the CATE estimation from observational data. Several innovative CATE estimation approaches, such as meta- meta-learners and causal ML models, have been proposed to tackle the fundamental challenge in causal inference and enhance the predictive accuracy of CATE estimates (as discussed in Section (3). Nevertheless, the emergence of various CATE estimation methods has brought forth new question: Given multifarious options for CATE estimators, which should be chosen? In observational data, treatment is often non-random and propensity scores remain unknown. Conventional model validation procedures, unfortunately, are not suitable for CATE estimator selection in this case due to the absence of ground truth CATE labels. Therefore, exploring proper metries for CATE estimator selection remains an essential yet challenging research topic in causal inference.
Recent research has emphasized the significance of model selection for CATE estimators, as high- lighted in [66 20 53 These works have proposed and summarized two types of criteria for CATE estimator selection, the plug-in metric R. (f) and the pseudo-outcome metrie Rpseudo (f):
Ry
RS plug (f)
1 n
(F(x,) - F(X,)),
R
pasuddo (t)
12 (F(X,) 8,y3. (1)
n
One can establish a plug-in estimator F or construct a pseudo-outcome estimator Y using the validation data select CATE estimator +. The previous studies [66 20 53 have shown that these metrics offer some assistance in identifying well-performing CATE estimators. However, two additional challenges are still encountered in these two metrics.
How to determine the metric form and underlying ML models for nuisance pa- rameters? As previously discussed, plug-in and pseudo-outcome metrics have various forms, and both of them rely on estimating nuisance parameters îj using MI. algorithms such as linear models, tree-based models, etc. Plug-in metries even need fit an additional ML model for the plug-in learner 7. However, selecting the suitable metric form and ML algorithms can be very difficult without the knowledge of true data generating process. Consequently, we might go round in circles as this challenge leads us back to the original estimator selection problem (2)
Challenge 1:
These metries are not well-targeted for selecting a robust CATE estimator. In potential outcome framework [64), the factual distribution pr and the counterfactual distribution per for t {0, 1} can be defined as follows:
Challenge 2:
pF = P(X, T = T
per = : P =
- =1
The above 2 reveals that the covariate shift P(X]-= =) + P(XTT=11 t) leads to distribution shift between pF and pCF and such distribution shift can be further exacerbated once the uncon- foundedness assumption P(Y*|X,
t) P[ 1 1-t) is violated. It is widely recognized that ML models often struggle when the training and test data do not adhere the same distribution. Therefore, it becomes essential to select CATE estimator learned on p" that demonstrates robust performance to the counterfactual distribution PCF, This need for robustness holds even greater significance than the pursuit of an ideal "stellar" estimator because striving for the perfect estimator can be futile the absence of ground truth counterfactual labels.
Contributions. In this paper, we propose Distributionally Robust Metric (DRM) for CATE estimator selection. The main contributions are summarized as follows: (1) The proposed DRM method is nuisance-free, eliminating the need to fit models for nuisance parameters (outcome function, propensity function, and plug-in learner). (2) The DRM method is designed to prioritize selecting distributionally robust CATE estimator. (3) We provide a finite sample analysis of the proposed

---

distributionally robust value (+) for E {0, 1), showing it decays to (*( at rate of n 1/2 (4) Experimental results validate the effectiveness of the DRM method in selecting CATE estimator that is robust to the distribution shift incurred by covariate shift and hidden confounders.
2 Background of CATE Estimator Selection
Suppose the observational data contain n i..d. samples { ti 9:) with the associated random variables being ((x; T, Y=)).=. For each unit i, X; E X C Rd is d-dimensional covariates and T; {0, 1 is the binary treatment. Potential outcomes for treat (T 1) and control (I' 0) are denoted by Y y E y C R. The observed (factual) outcome is
TY1 + (1 T)Y The X z). The conditional mean potential
propensity score 63 is defined (2x := P(T
outcome surface defined as (xr) == E [ X = x] for E {0, 1]. The true CATE defined as
Ttrue (z) == E [y] yo X = x] = frr (a) - (o(x))
Following the standard and necessary assumptions in potential outcome framework 64 Assumption 2.1 that ensure treatment effects are identifiable.
impose
Assumption 2.1 (Consistency, Overlap, and Unconfoundedness) Consistency: the treatment is I, then the observed outcome Y equals Yt. Overlap: The propensity score is bounded away from to 1, i.., < क (z) < Vo E X. Unconfoundedness Y IL X, Vt E {0, 13.
The goal of CATE estimator selection is to select the best CATE estimator, denoted by Abests from set of candidate estimators [f1,...,f] fy):
Testt = arg min Roracle ff
R Roract (f)
= (F(X.) Tirue (X,))²
Ttrue (X)) known the Precision of Estimating Heterogeneous Effects (PEHE) w.r.t 7 32 68 Note that Roraci (f) cannot be employed to evaluate CATE estimators' performances in real applications as we do not have access to Ttras Previous studies have introduced plug-in in and pseudo-outcon metrics to aid in CATE estimator selection, as
Here, Rorade (f) is associated with [f (X)
shown in equation (T) Then, CATE estimator isefect is selected on validation data by
Psetect arg min
R5 Rolug (+) or Tselect arg min FE
Re
pseudo (f).
Notably, both the plug-in and pseudo-outcome metrics necessitate the fitting of nuisance parameters nj (e.g.. 7/ = (ū1, Po, R)) using off-the-shelf ML models. While some papers like [16] address the selection nuisance parameters for Averate Treatment Effect (ATE) estimators, e.g., the doubly robust estimator [23] 13 36). our paper focuses on the selection of CATE estimators rather than nuisance parameters. For the plug-in metric, 7 can be constructed using any CATE estimator discussed in Appendix A.1 yielding metrics such as plug-T, plug-DR, etc. For the pseudo-outcome metric, Y can be constructed using specific formula discussed in Appendix A.2 yielding metrics such as pseudo-DE DR, pseudo-R, etc. The metrics based on the influence function and the R-learner objective (55] are categorized into the pssudd-outcome metric. The categorization of plug-in and pseudo-outcome metrics maintains consistency with 20 53
Related Work
CATE estimation. Recent advancements in ML have emerged powerful tools for estimating
CATE from observational data, and researchers pay particular attention to meta--learnrss and causal ML models. Existing meta-learners mainly include traditional learners such S-Iearner, T-learer, PS-learner, and IPa-learner, as well as new learners such as X-learner [47]. U-learner 25 55],
DR-learner 41 26). R-leamer (55), and RA-learner (18). The specific details of these meta-learners are stated in Appendix A.1 Additionally, some studies also focus on developing innovative causal ML
'Note that in the setting C of our experiments, the unconfoundedness assumption is violated, leading to misspecified nuisance parameters in CATE estimators and baseline selectors.
3

---

models for CATE estimation, such as Causal BART [30, Causal Forest [70] 58 generative models
like CEVAE [T] and GANITE [77]. representation learning nets including SITE [76]. TARNet 68
Dragonnet 669 FlexTENet [9] and HTCE [0,, disentangled learning nets like DO 44 45
DeR-CHF [4]. and DR-CFR 31 and representation balancing nets such as BNN [39, CHRNet
68
DKLITE 77), IGNITE 29). BWCER [6], DRRB [55, and DIGNet (38. Recent surveys
28
75
56] have also conducted a systematic review various causal inference methods.
CATE estimator selection. Compared to the diverse range of CATE estimation methods, selecting CATE estimators has received limited attention in existing causal inference research. Current methods for selecting CATE estimators can be broadly classified into two main categories. The first category. which also considered in this paper, involves using plug-in and pseudo-outcome methods to evaluate CATE estimators. These methods share two common characteristics: 1) Both methods require fitting MI. models for nuisances (e.g.. outcome function, propensity function, CATE function) on validation set and then implementing the learned ML models in either the plug-in surrogate or the pseudo-outcone surrogate; 2) Both methods serve as surrogates for the expected error between the CATE estimator and the true CATE, i.e., Roracie (f) in equation 3 The difference between the two methods is that the plug-in method directly approximates the true CATE function, where only covariate variables are involved, while the pseudo-outcome
method typically constructs specific formula incorporating covariates, treatment, and outcome variables. For example, the pscudo-DR proposed in [65] is constructed by the outcome predictors learned with representation balancing has conducted thorough empirical investigations into exploring these methods for selecting CATE estimators. Their findings suggest that no single selection criterion can universally outperform others in all scenarios in the task of selecting CATE estimators. More details of the two selection methods are stated Appendix [A.2] The second category considers leveraging the data generating process (DGP) to generate synthetic data with the known true CATE function, allowing the validation of CATE estimators' performance on this synthetic data. For example, authors in [2] find that placebo and structured empirical Monte Carlo methods are helpful for estimator selection under some restrictive conditions. In addition, researchers in [67 [59] focus on training generative models to enforce the generated data to approximate the distribution of the observed data. However, the DGP-based method still faces some limitations in CATE estimator selection: 1) it only guarantees the resemblance of the generated data to the factual distribution, without considering the counterfactual distribution; and ii) there a potential risk of the method favoring estimators that closely resemble the generative models [17].
objective 66
Recent research 666 20 53
4 The Distributionally Robust Metric
In this section, we introduce the Distributionally Robust Metric (DRM) for CATE estimator selection. First, we capture the uncertainty in PEHE in distributionally robust manner (Section 4.1) We then establish the DRM based on the distributionally robust value of PEHE (Section 4.2
4.1 Capturing the Uncertainty in PEHE
Proposition 4.1. The PEHE w.r.. the CATE estimator can be decomposed as follows:
(x) ( Terue (X)) [[+(())] + 2X(+(X) + 2F(-F(X)Y") +
(5)
where Ç - (((()(() - (x))) The proof deferred to Appendix B.1
Proposition 4. indicates that the PEHE is equal to four terms, where [[)(X) (l and B depend on f. while is constant that is independent of f. The term (lf(X) for t E {0, 1} can be further decomposed as follows:
B[+(X)r*}=
E P(T = -1--00
(6)
(a) Empiriallly computable
(b) Empirically uncomputable
Equation (6a) can be computed empirically since the potential outcome Yt is observable in the group of T However, equation (b) empirically uncomputable due to the unavailability of yt the group of T ===1 I. The unknown term E[+(X) = therefore determines the uncertainty PEHE. To capture such an uncertainty, we therefore establish distributionally robust values for [[f(X)) = and E| El-+(x)) 0) based on Kullback-Leble (KL) ambiguity set.

---

Definition 4.2 (KL ambiguity set), Given two distributions Q and P and the ambiguity radius >> 0.
The KL ambiguity (uncertainty) set (P) is defined
where DxL(QIIP) 1,x g(x))) log dr.
((z) p(z)
(P = D sth,
Here, D P) denotes the KL divergence of some arbitrary distribution Q from the reference distribution P. Now we define the distribution of (x, y ,,,'' in the treated and controlled groups as
Pr P(X,YO, I' 1); Pc P(X, Yo Y T==0) 0).
By setting an adcquately large ambiguity radius in Definition 4.2 the following incqualities hold for Po
= = P [f(x)r*) and (E(- =
(-
sup = 1] = (F(X)))) (EB..(Pc) E
E [=((()))))==
(9)
= E| =)
E {"1-(())'))=
sup QEB,
To provide clearer understanding, let consider the example of (+)))() Since the term E[f(X)) is computable on its factual distribution Pe but uncomputable on its counterfactual distribution Pr, it is large enough to contain the distribution Pr. By doing so, we can capture the uncertainty of (*r(f(X) (F()) w.r.t. f. In other words, the value of the uncomputable quantity E (+ will be at most (f). Similarly, the value of the uncomputable quantity Ee (PFe(-+() will be at most (f)... Obviously, the uncertainty in PEHE will be larger if the distribution shift between factual and counterfactual distribution severer. Consequently, we can obtain the distributionally robust value of PEHE in Corollary 4.3 which measures the uncertainty in PEHE.
construct an ambiguity set centered around the distribution Po such that
Corollary 4.3. Let (+) and (²) be the quantities defined in equation (9)
C be the constant given in Proposition 4.1 u1 P(T =1), and un un P(T = 0). The distributionally robust value of PEHE w.rl. 7 is defined as VPEHE(f) such that
[[(((X) Ttrae (X)) I VPEHE(F)
(10)
= =E(f(X)²] (ooo)((((X))") -
+ 2 (to (F) +C.
4.2
Establishing Distributionally Robust Metric
provides the distributionally robust (worst-case) value of PEHE, it can naturally measure the robustness of the CATE estimator ê against distribution shift between counterfactual distribution and factual distribution. In this section, we will provide two steps involved in using construct the DRM method for CATE estimator selection.
As Corollary
4.3
Corollary 4.3
Step 1: Establishing computational tractability of V* The distributionally robust values (F) and V (f) in equation (10) are initially defined as supremum problems over infinite support, presenting a substantial computational challenge. Theorem reformuullte the infeasible supremum problems into tractable minimum problems.
Theorem 4.4. The distributionally robust values (+) and (f) in equation 2 are equivalent to
(+) min AgEo + do log E Po (exp((((( /No]]; 2g
(11)
('( min λιEι + A log Br (ex(=//xx((=xxxxx///// A120
The proof deferred to Appendix
B.
In the finite-sample scenario, (+) and (+) can be empirically approximated as follows:
(+( = min AoEo + do log A020
(1 --) exp(+(()))x))))
(12)
(²()) min λιEι + X log A1
Teep Texp(-=(X)\\/AA)
T1t
5

---

Algorithm Using DRM for CATE Estimator Selection
Input: The candidate CATE estimators ( f.). The validation dataset with n i.i.d. observa-
tional samples {(X,, Ti H=) The number of iterations K. The initialization Ao and 11 The ambiguity radius 6o and E1.
1: for j = to J do
2: for k = K - do
3:
Compute f((A) (k)
fxjtt) fort {0, 1} by equation (14a).
4: Compute ( (k)
ce: 0022, (k) for t {0, 1 by equation (4b).
5: X
(k+1) + max
(A SE (OE (R) Eri f;)/ for t (0,1).
6: Save (ft][[]]] FA (1+1) sg for t E 13.
7: Return (f,) arg mink Ko.....-11 K-1) D (+,)]K fort E (0.1).
8: Use (f,) and (ft) to compute R RDM (fz) by equation 15
Output:
ioelect arg min ts) R RRM (f).
Note that in equation 12 the potential outcomes y and y are replaced by the observed outcome Y due to the fact that (1 Y) - T)Y and 7y1 = TY, which aligns with the Consistency
assumption in Assumption 2.1 We then provide a finite-sample analysis of the gap between (f) and (f) in the following Theorem 4.5 which suggests the gap decays at rate of n 11/2
Theorem 4.5. Let un := P(T = t) for t E {0.1}. Assume 0 < À I Ao A1 I and f(X)Y
bounded within the range of M to M. Define Ceep 1 (MEMsO) exp (M/ā - M/2)
1 (M M220) exp (M/2 - M/2) + LOSssMY exp (M/2 M/A). For 2 2/1²² log(2/6) and
t {0, 13, with probability 1 - 8, we have
(V
(V22
SA2 log 3 nut
2X 222 log(
- -())
Ceap
+0
(13)
nuf
The proof is deferred to Appendix B.4
Step 2: Finalizing Distributionally Robust Metric for CATE estimator selection. two functions that are useful in obtaining (f) and V (f):
We first define
1
1 Σ
Fo f) Ageo + An log
(( = = + Ai log
Ar
(14a)
The
14.
i=1
i=1
ar. aNo
A
Zi
E
21 Z,e
of
-2
R+
Z,
AI
-2,
= c0 + log
= =1 + log
(14b)
nh
Ao Ri
2
θλ1
R4
A1
EE e A1
2.
Here, Z denotes F(X)Y for notational simplicity. We then use the Newton-Raphsee method to find
the empirical solution for D'(F), exploiting the convexity of (ic,, f); w.r.t. A₁- Based on the distributionally robust value of PEHE,, i.c., VPEHE(F) in equation (10). we finally obtain the selected arg min
estimator fselect
DIEM (f) such that
R DRM (7)
2
2 n
Σ
Th
i(X))
Σ +(X.)V+++ D. n. -.)
(t)) (15)
Algorithm provides complete procedure of using the DRM method for CATE estimator selection.
Discussion on the ambiguity radius c. The ambiguity radius plays critical role in real-wor worldworld applications (2 60). However, determining an appropriate value for can be challenging as it
requires striking a balance between ensuring the bound in equation D holds and maintaining its tightness. Specifically, if is set too small, it fails to guarantee that the counterfactual distribution is contained within the ambiguity set centered factual distribution (the bound Corollary 4.3 can hold). On the other hand, if is set too large, even though the ambiguity set can encompass more

---

distributions to ensure the counterfactual distribution is contained, the bound in Corollary 4.3 can be less tight. In general, selecting a proper ambiguity radius an open problem in distributioanlly robust optimization (DRO) literature 34 54
46
48 72
In this paper, we provide guidance for determining the ambiguity radius for our DRM method. Based on the above discussion, an ideal radius should be eî = ( PP)) and e = (Prllet) which ensures that the bound in Corollary 4.3 holds and is tight. However, as defined in equation (8) both Pc and PT involve counterfactual information, making it unattainable to directly compute D (Pe Pr)) and D (PrllPc)) To overcome this challenge, we demonstrate that Proposition 4.6 provides an intriguing alternative approach to acquire DKL (PclPP))Pr) and (ri(Pr(IPe) when unconfoundedness in Assumption i. satisfied.
Proposition 4.6. Let P PT 1) and PX := P(X = 0) denote the covariates distri- bution in the treat and control group, respectively. Assuming that random variables (X, T, Y Y satisfy the unconfoundedness in Assumption 2.1 we have
(Pe(PP))PT) = ( PX);
(xi(Pr||Po) = (
(16)
The proof is deferred to Appendix B.2
Proposition 4.6 provides an important insight that the uncomputable term DRI(PelIPy (or D (Pr Pc)) can be replaced by computable quantity DRz(Px (or OKI(Pxl
where P and PK are empirically observable. Consequently, the ideal ambiguity radius can be set as of = DDDi ( x and ej = DxL ( P)). While the KL divergence can be approximated
we recommend setting the ambiguity using empirical algorithm (e.g, Nearest- Neighbt 73 57
radius larger than the empirically approximated KL divergence (see specific explanations in Appendix C.I. This is necessary because it ensures that the ambiguity set large enough to contain the target distribution. It is also important to note that though the Algorithm D iinvolves approximating i DK (P& [IP}) and ej = Drr. ( P PC), the DRM itself remains free of nuisances, as this approach only determines the ambiguity radius but does not involve learning any nuisance function such as the outcome function, propensity function, and plug-in learner.
5 Experiments
5.1 Experimental Setup.
Estimators & Selectors. We consider total of 36 CATE estimators, comprising the combination of 4 base ML models and 9 meta-learners. Specifically, the base ML models are Linear Regression (LR), Support Vector Machine (SVM), Random Forests (RF), and Neural Net (Net). We consider these ML models for CATE estimators because they are representative of both rigid and flexible models, with each encoded distinct inductive biases, as highlighted by 199 20 Note that for the LR method, we employ Ridge regression for regression tasks and Logistic regression for classification tasks. As for the remaining methods, we utilize their corresponding regressors and classifiers for regression and classification tasks, respectively. Regarding the meta-learners, we select set of both traditional basic learners (S-, T-, PS-, and IPW-learners) and recently developed learners (X-, DR-, U-. R-, and RA-learners), as detailed in Appendix A. We consider 14 CATE selectors, consisting of plug-in methods that rely on the above learners, 3 pseudo-outcome methods (pseudo-DR, -R, and -IF), the random selection, the factual selection (from the 6-learner pool with S-, T-), the Nearest- Neighbor Matching (62), and our proposed DRM. The specific details of baseline selectors are stated in Appendix A.2 We employ the eXtreme Gradient Boosting (XGB) [2] as the underlying ML model for both plug-in and pseudo-outcome methods. We choose XGB because: i) it demonstrates superior performance in various scenarios, ensuring a good performance of baseline selectors; ii) the need to avoid potential congeniality bias that may arise from using the similar ML models employed in CATE estimators 20); iii) aligning with [5] where XGB is used for their proposed pseudo-IF metric. The details of hyperparameters for nuisance models are stated in Section C.2 of Appendix.
Dataset. Since the ground truth of CATE unavailable real-world data, previous studies commonly utilize semi-synthetic datasets to compare model performance. In line with [19 [1] 20 we collect the covariates with n = 4802 data points from ACIC2016 dataset [22]. Then, we generate treatment with X, Bern(1/(1 ( + 3]))), where Bern indicates the Bernoulli
7

---

Table 1: Comparison of Regret for different selectors across Settings A, B, and C (Note that B ( = 1) matches A (p- = 0.1)). Reported values (mean standard deviation) are computed over 100 experiments. Bold denotes the best three results among all selectors. Smaller value is better.
(p
(p 0.1)
A Go 0.3)
B ( 0)
B 2)
Cin 0.1)
C (on 0.5)
( 0.9)
Plug-U
47.87294.89
39.22.100788
32.13161.49
0.51=2.09
151,98_291.20
39.41
54.47:229.36
15.58126.41
Plug-S
3.3817.73
2.66
2.2515.61
0.22-1088
5.91:
2.7214.70
3.7426.34
4.65:1.33
Plug-PS
3.0847.27
2.65-565
2.2445564
0,22-1.08
5,91 10,61
2.7244470
3.55+6.11
4.65+6.31
Plug-1 T
59.12+21 87
56.38+25502
55,35+21.36
10.48=10.72
18.00
59,87=18.75
43.284:2.64
36.28+19.35
Plug-X
8.10
1.77
1.76110.81
13.95413.94
7.78&155022
14.76114.62
10.8020082
Plug-IPW
33.37::28 34
35.78122750
35.261127.21 21
4.43=7.10
58.64 28.64123388
38,43:231.16
29.777
20.54119.67
Plug-DR
43.1 1::26.54
43.76:226:2
44.20++76.4
4.22=7.97
64.60=18.88
46.57=332.93
28.06::23.06
18.59
Plug-R
1.92111.91
2.62-15.63
1.47133.00
0,43=22.9
9.78131143
1.91149944
2.5316.51
2.12.14.94
Plug-RA
06.60::24:06
17.69:19.98
54.60+
6.60=9.16
64.50=17.72
15.87=1966
20,48:22.55
33.34:119.04
Pseudo-DR DR
61.35+:22.41 41
61.09+20.08
59.06+19.46 46
14.75:22..95
70.02+17.55
62.08=19.68
48.83+-25.77 78
44.99+23.53
Pseulo-R
9.854227.04
14.12:445.74
5.94:21.05
4.73+:2040
14,86+30.6 64
10.58-24.63
15.93429.84
21.251332.11
Pscudo-IF
64.541115.18
62.491116661
62,69116613
26.73-123.60
65.74
60.06:2:1.6
54,961220.3
38.601.22.11
Random
7214::22745
651 6511=21151
4196217049
1135=5596
7549
3768: 16625
6214:19942
3445:14591
51.09+118.0
10.86++19440
51.01+21.08
14.33=16654
65,23+27755
48.92-17.199
47.40+222.51
40.37+22.14
Matching
60,85::2145
18:777777
19.91
13,33-22.86
68,98=17.2
61.52=19.04
52.83++23.85
40,01
DRM
0.96113.6
0.84
1.25.5..97
0.38
15.51&112.62
1.5618.73
140.8.67
1.2613.52
distribution. The potential outcomes are generated by linear function with interaction terms:
Y
Σ 8
ΣΣ j=1k=)
X X + ΣΣΣ
B R X X Nu + T, 7t Xis + es.
The coefficient values are set as follows: 22.3,,B3.k. B3r. Bern(0.2),, Vj Bern(p), and ~ &~N(0, 0.1). The parameter & in treatment assignment represents the level of selection bias, and the parameter p Y represents the complexity of the CATE function. We adopt the above data generating process to randomly generate 100 datasets, each with a training/valddtioonnesingg ratio of 49%/219933%%.
Settings. In this section, we mainly investigate whether the estimator selected by DRM can demonstrate robustness to the selection bias and unobserved confounders. In addition, as demonstrated in [19 20), the complexity of CATE function also affects relative performance of estimators and selectors. Given these considerations. we design the following three settings to compare the CATE selectors. Setting A: With the unconfoundedness assumption, let p vary in 0.1 0.3) with fixing < 1. Setting With the unconfoundedness assumption, let vary in {0, 1,2] with fixing / = 0.1. Setting C: Without unconfoundedness assumption, fix p = 0.1 and = 1. Then randomly remove Im- d covariates such that the dimension of observed covariates is d - [m dJ. where m denotes the ratio of missing covariates varying in {0.1, 0.5, 0.9}. All the experiments are run on Dell 3640 with Intel Xeon W-1290P 3.60GHZ CPU.
Comparison criteria. The CATE estimator + is believed better if it achieves smaller difference between Roruce (7) and Roracie (fhsst,, where Abest the actual best estimator equation (3). We therefore use the following Regret criteria to compare estimators chosen by different selectors:
Regret = oracle (fsetect) R Roadle (fbest).
To further assess the ranking ability of each selector, we calculate the Spearman rank correlation between the rank order determined by the oracle metric Roraele (f) and the rank order determined by each selector. All the reported values (Mean + Standard deviation) are computed over 100 runs.
5.2 Experimental Results
Regret comparison. The results presented in Table demonstrate consistently good performance from the DRM selector across various settings. In setting A, the DRM selector outperforms other selectors as the CATE complexity (p) varies. Additionally, Plug-R, Plug-S, and Plug-PS also perform well in terms of the Regret criterion, which aligns with prior findings in [66] that the R-objective excellent many cases. Note that the strong performance of Plug-S and Plug-PS may be due to less pronounced heterogeneity in the CATE function compared to the outcome function in the data generating process. We also compare the PEHE performance (i.e., Apprcie(fsatenn )) different selectors in Table 3 of Section
C.3
The results indicate that Plug-R, R, Plug-S, and Plug-PS tend to
8

---

Table 2: Comparison of rank correlation for different selectors across Settings A, B, and C (Note that (5 1) matches (p- 0.1)). Bold denotes the best three results among all selectors. Reported values (mean + standard deviation) are computed over 100 experiments. Larger is better.
A (p
(p 0.1)
A (p= 0.3)
(C 0)
BC 2)
C (m 0.1) Cm 0.5)
Cm 0.9)
Plug-U
0.69
0.7010.35
0.7510029
0.95100.04
0.53.1..0
0.68.0.33
0.7310.34
0.8310.24
Plug-S
0.95=20.06
0.95:0.06
0.95
0.95
0.95:10.05
0.95+0.03
0.95
0.91
Plug-PS
0.95+00.6
0.95+0.06
0.95=0005
0.95+00.04
0.95+00.5
0.95+0003
0,95::0.55
0.91=00077
Plug-T 0.54=0.18
0.54
0.54=0016
0.89+0.07
0.57=0.16
0.51
0,58+0021
0.59=00211
Plug-X
0,94:00.55
0.94
0.94
0.93=+0.05
0.93++0.05
0.93++0.04
0.92++0.06
0.85+00.1 13
Piug- Pug-IPW
0.72:10.19
0.71
0.71
0.92-10.06
0.68
1.691.0.19
0.76.10.18
0.77
Plug-DR
0.65=0.19
0.63::0.20
0.63=0018
0.93
0.59:0.16
0.61
0.71
0.73
Plug-R
0.96=+0.03
0.96=20.03
0.96=20.03
0.95
0.93++0.07
0.96+0003
0,96#00.05
0.96=#0.04
Plug-RA
0.55++0.19
0.54+-0.17
0.55++0.17
0.924+0.06
0.574:0.15
0.53++0.17
0.604-0.22
0.62:+0.21
Pseudo-DR
0.54:10.8
0.53:10.18
0.53
0.87
0.55:10.15
0.50
0.54
0.58110.23
Pseudo-R
0.86=00.11
0.87=0.09
0.88=0008
0.93=#0.06
0.83++0.15
0.85
0.85#0.12
0.80=0.16
Pseudo-IF
0.52++0.17
0.52:40.17
0.51=00115
0.66++0.18
0.64+0.16
0.524+0.16
0.53-+0.19
0.62++0.11
Random
0.26.10.13
0.26110.13
0.27
0.4710.11
0.23.10.13
0.28
0.2810.11
0.24
Fact
0.35=20.08
0,36::0.08
0.35
0.48:10.08
0.31
0.35
0.33-10.09
0.29=0.11
Matching
0.53+0017
0.51 18
0.52 16
0.89+0008
0,58+0.15
0.51 16
0.55-+0.21
0.60+0.21
DRM
0.81
0.801.0.88
0.80
0.85 1..05
0.77
0.791.0.99
0.81 1.0.0
0.00
Setting (p=
Setling A (p= 0.11
Setting A 1p 0.3)
Setting (E
Masling
PNN
Randble
Ban
Pando
Randon
10
Rug
MUS
Po
4
011
0.1
Ratio
Rafio
Setting (6 22
Setting Cm-= 0.1)
Setting C (m 0.5)
Setting cm 0.9)
Metching 141
Matchng led
Matcing
Matchnn let
Banair
Rando0
Fanedio
PseUce
Rugel
0.1
Ratio
Rabo
Ratio
Rabio
Rank
the Rank 4-11
the Rank 12-19
Rank 20-27
the Rank 20-36
Figure 1:
The stacked bar chart showing the distribution of the selected estimator's rank for each
evaluation metric across rank intervals: [1-3], [4-11], [12-19], [20-27], and [28-36]. The greener (or redder) color indicates that the selected estimator ranks higher (or lower). For example, the dark red (or green) indicates the percentage of cases (out of 100 experiments) where the selected estimator ranks among the worst 9 estimators, specifically as ranks 28, 29, .... or 36 (or among the best 3 estimators, specifically as ranks 1, 2, or 3).
exhibit better PEHE as the CATE complexity decreases, aligning with the findings in 200] In setting B, the DRM selector demonstrates robustness against selection bias (controlled by { compared to many baselines. However, for the case & 2, DRM selects a poor estimator 1 or 2 times out of 100 experiments, as shown in Figure Although this weakens its overall performance, DRM still outperforms many baselines in this scenario. In the scenario = 0 where no selection bias is present, the factual selection criterion performs better in this specific setting. In this case, DRM does not demonstrate a significant advantage, as there no distribution shift caused by selection bias. In setting C where the unconfoundedness assumption is violated, most selectors exhibit inferior performance. In contrast, DRM demonstrates consistent outperformance across all three cases, and its superiority becomes particularly significant m increases to 0.9, showcasing its robustness against the distribution shift arising from unobserved confounders.
Ranking ability. Table 2 the DRM method demonstrates favorable performance in ranking estimators, surpassing certain Plug- (c.g., U, T, IPW, DR, RA) and Pseudo- (c.g., DR, IF) selectors.

---

In comparison to other nuisance-free baselines (Random, Fact, and Matching), DRM achieves significantly superior ranking ability. However, compared to Plug-S, -PS, -X, and -R, does not exhibit remarkable performance in ranking CATE estimators,, possibly due to the fact that DRM selects estimators based on their distributionally robust (worst-case) performance. Indeed, the definition of ranking inherently involves the concept of expected (average) performance, which is not determined solely by either the best or worst performance. While distributionally robust performance serves as suitable criterion for selecting players to participate in the Olympics, it may not be reasonable standard for ranking players' average performance. Therefore, it would be intriguing to explore some ways in future research that can enhance the ranking ability of our DRM selector.
Variance analysis. Table Iiniicates that baseline selectors tend to exhibit higher variances in Regret performance. This is primarily due to the wide range of PEHE performances across the 36 CATE estimators. If a selector consistently selects either good or bad estimators, the variance would not be very large. To investigate this further, we sorted all 36 estimators in ascending order based on their Rfyad((f) values, resulting in the sorted list: [R determine the actual rank of the selected estimator within this list and visualize the distribution of these 100 ranks using a stacked bar chart. shows that many baseline methods tend to select CATE estimators from various percentile ranges, leading to high variance across the 100 selections. Notably, the DRM selector consistently chooses higher-ranked (i.e., better performing in PEHE) estimators, demonstrating its robustness in CATE estimator selection.
Ro (f))) []] We then
Potential improvements. There are several potential improvements based the current experi- mental settings. First, the existing results suggest that Plug-S performs better than Plug-T, indicating that the complexity of CATE function is relatively simple. It would help provide more compre- hensive analysis if investigating how DRM compares to baselines when the CATE function is more complex. Second, since the impact of selection bias can vary with sample size 3. it is important to compare different selectors when the sample size is sufficiently large. Third, considering baselines that are specifically designed for addressing hidden confounders could provide valuable insights for testing different selectors under such conditions. We encourage deeper investigation of causal model selection without assuming unconfoundedness. Finally, it would be good if future studies will apply DRM and other selectors in Healthcare, Economics, and Business applications with real-world data, as CATE estimator selection plays an important role in personalized decision makings.
Conclusion
This paper sheds lights on the potential of robustness in CATE estimator selection. We propose a distributionally robust metric (DRM). The proposed metric is nuisance-free, eliminating the need to fit models for nuisance parameters (outcome function, propensity function, and plug-in learner). Additionally, it is well-targeted for selecting a robust CATE estimator. We provide a finite sample analysis that demonstrates the gap between (F)(F) and (*) reduces at rate of n -1/2 for /. E {0, 1}. The experimental results showease that the CATE estimator selected by DRM demonstrate robustness to the distribution shift incurred by covariate shift and hidden confounders.
Limitations and future work. This paper explores the potential of robustness in CATE estimator selection. However, we must acknowledge that our DRM method is not a silver bullet, as consistent estimation on the CATE are never attainable 14]. Here, we outline some challenges and suggest future research directions. First, while Proposition 4.6 provides useful guidance for setting ambiguity radius in the DRM algorithm, we cannot guarantee that the empirically
radius is optimal due to potential bias in the algorithm's 's approximation of KL-divergence. Second, as discussed in Section [.2) enhancing the ranking capability of DRM is promising area for further research. Moreover, our findings are based on KL-divergence. However, using other divergences, such as the Wasserstein distance, to construct the ambiguity set could incorporate more diverse distributions, despite the challenges in solving the dual formulation of the Wasserstein distributionally robust value. Simultaneouslyy, exploring whether alternative divergences can yield a tighter bound for the PEHE error is also interesting
[4]. Finally, inspired by [16], understanding how nuisance parameters influence metrics like plug-DR and pseudo-DR might be helpful in CATE estimator selection. We hope our methods and findings will spur interest model selection for causal inference, as well as in related fields like domain adaptation and out-of-distribution generalization.
10

---

Acknowledgement
Qi WU acknowledges the support from The CityU-JD Digits Joint Laboratory in Financial Tech- nology and Engineering, The Hong Kong Research Grants Council [General Research Fund 1121940090433088,, and The CityU APRC Grant 9610643. The work described in this paper was partially supported by the InnoHK initiative, the Govermennt of the HKSAR, and the Laboratory for Al-Powered Financial Technologies. We finally thank all the anonymous reviewers for their constructive suggestions.
References
[1] Alberto Abadie, Susan Athey, Guido W Imbens, and Jeffrey M Wooldridge. When should you adjust standard errors for clustering? The Quarterly Journal of Economics, 138(1):1-35, 2023.
[2] Arun Advani, Toru Kitagawa, and Tymon Stoczyáski. Mostly harmless simulations? using monte carlo studies for estimator selection. Journal of Applied Econometrics, 34(6):893-910. 2019.
[3] Ahmed Alaa and Mihaela Schaar. Limits of estimating heterogeneous treatment effects: Guide- lines for practical algorithm design. In International Conference on Machine Learning, pages 129-138. PMLR, 2018
[4] Ahmed Alaa and Mihaela der Schaar. Limits of estimating heterogeneous treatment effects: Guidelines for practical algorithm design. In Jennifer Dy and Andreas Krause, editors, Proceed- ings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 129-138. PMLR, 10-15 Jul 2018.
[5] Ahmed Alaa and Mihaela Van Der Schaar. Validating causal inference models via influence functions. In International Conference on Machine Learning, pages 191-201. PMLR, 2019.
[6] Serge Assaad, Shuxi Zeng, Chenyang Tao, Shounak Datta, Nikhil Mehta, Ricardo Henao, Fan Li, and Lawrence Carin. Counterfactual representation learning with balancing weights. In International Conference on Artificial Intelligence and Statistics, pages 1972-1980. PMLR, 2021.
[7] Susan Athey, Guido W Imbens, Jonas Metzger, and Evan Munro. Using wasserstein generative adversarial networks for the design of monte carlo simulations. Journal of Econometrics, 2021.
[8] SUSAN ATHEY, JULIE TIBSHIRANI, and STEFAN WAGER. Generalized random forests. The Annals of Statistics, 47(2): 148-117477 2019.
[9] Toana Bica, Ahmed M Alaa, Craig Lambert, and Mihaela Van Der Schaar. From real-world patient data to individualized treatment effects using machine learning: current and future methods to address underlying challenges. Clinical Pharmacology & I'herapeutics, 10971):87- 100. 2021.
[10] Ioana Bica and Mihaela van der Schaar. Transfer learning on heterogeneous feature spaces for treatment effects estimation. Advances in Neural Information Processing Systems, 35:37184- 37198, 2022.
[11] Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14(11), 2013.
[12] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd aem sigkdd international conference knowledge discovery and data mining, pages 785-794, 2016.
[13] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters, 2018.

---

[14] Victor Chernozhukov, Mert Demirer, Esther Duflo, and Ivan Fernandez- Val. Generic machine learning inference on heterogeneous treatment effects in randomized experiments, with an application to immunization in india. Technical report, National Bureau of Economic Research, 2018.
[15] Zhixuan Chu, Stephen L Rathbun, and Sheng Li. Graph infomax adversarial learning for treatment effect estimation with networked observational data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 176 176,184, 2021.
[16] Yifan Cui and EJ Tchetgen Tchetgen. Selective machine learning of doubly robust functionals. Biometrika, 11 12)555117-335 2024.
[17] Alicia Curth, David Svensson, Jim Weatherall, and Mihaela van der Schaar. Really doing great at estimating cate? critical look at ml benchmarking practices in treatment effect estimation. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 2), 2021.
[18] Alicia Curth and Mihaela van der Schaar. Nonparametric estimation of heterogeneous treat- ment effects: From theory to learning algorithms. In International Conference on Artificial Intelligence and Statistics, pages 1810-1818. PMLR, 2021.
[19] Alicia Curth and Mihaela van der Schaar. On inductive biases for heterogencous treatment effect estimation. Advances in Neural Information Processing Systems, 34:15883-1558 15894, 2021.
[20] Alicia Curth and Mihaela Van Der Schaar. In search of insights, not magic bullets: Towards demystification of the model selection dilemma heterogeneous treatment effect estimation. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 6623-6642. PMLR, 23-29 Jul 2023.
[21] Robert Donnelly, David Blei, Susan Athey, et al. Correction to: Counterfactual inference for consumer choice across many product categories. Quantitative Marketing and Economics, 19(3-4):4094409 2021.
[22] Vincent Dorie, Jennifer Hill, Uri Shalit, Mare Scott, and Dan Cervone. Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition. Statistical Science, 34(1):43-68, 2019.
[23] Max Farrell. Robust inference on average treatment effects with possibly more covariates than observations. Journal of Econometrics, 189(1):1-23, 2015.
[24] Carlos Fernández-Loorf, Foster Provost, Jesse Anderton, Benjamin Carterette, and Praveen Chandar. A comparison of methods for treatment assignment with application playlist generation. Information Systems Research, 34(2):786-80) 2023.
[25] Aaron Fisher. Inverse-variance weighting for estimation of heterogeneous treatment effects. In Forty-first International Conference on Machine Learning.
[26] Dylan J Foster and Vasilis Syrgkanis. Orthogonal statistical learning. The Annals of Statistics,
51 (1)):87999088 2023.
[27] Jared C Foster, Jeremy MG Taylor, and Stephen J Ruberg. Subgroup identification from randomized clinical trial data. Statistics in medicine, 30(24):2867-2888 2011.
[28] Ruocheng Guo, Lu Cheng, Jundong Li, P Richard Hahn, and Huan Liu. A survey of learning causality with data: Problems and methods. ACM Computing Surveys (CSUR), 53(4):1-37, 2020.
[29] Ruocheng Guo, Jundong Li, Yichuan Li, K Selçuk Candan, Adrienne Raglin, and Huan Liu. Ignite: A minimax game toward learning individual treatment effeets from networked observa- tional data. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences Artificial Intelligence, pages 4534 4540, 2021.
[30] P Richard Hahn, Jared S Murray, and Carlos M Carvalho. Bayesian regression tree models for causal inference: Regularization, confounding, and heterogeneous effects (with discussion). Bayesian Analysis, 15(3):965-105) 2020.
12

---

[31] Negar Hassanpour and Russell Greiner. Learning disentangled representations for counterfactual regression. In International Conference on Learning Representations, 2019.
[32] Jemnifer L Hill. Bayesian nonparametric modeling for causal inference. Journal of Computa- lional and Graphical Statistics, 20(1):217-240, 2011.
[33] Paul W Holland. Statistics and causal inference. Journal of the American statistical Association,
8 (1(3))::45-96) 1986.
[34] Zhaolin Hu and Jeff Hong. Kullback-leibler divergence constrained distributionally robust optimization. Available at Optimization Online, 1(2):, 2013.
[35] Yiyan Huang, Cheuk Hang Leung, Shumin Ma. Zhiri Yuan, Qi Wu, Siyi Wang. Dongdong Wang. and Zhixiang Huang. Towards balanced representation learning for credit policy evaluation. In International Conference on Artificial Intelligence and Statistics, pages 3677-3692. PMLR, 2023.
[36] Yiyan Huang, Cheuk Hang Leung, Qi Wu, Xing Yan, Shumin Ma, Zhiri Yuan, Dongdong Wang, and Zhixiang Huang. Robust causal learning for the estimation of average treatment effects. In 2022 International Joint Conference Neural Networks (IJCNN), pages 1-9. IEEE, 2022.
[37] Yiyan Huang, Cheuk Hang Leung, Xing Yan, Qi Wu, Nanbo Peng, Dongdong Wang, and Zhixiang Huang. The causal learning of retail delinguency. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 204-212, 2021.
[38] Yiyan Huang, WANG Siyi, Cheuk Hang Leung, WU Qi. WANG Dongdong, and Zhixiang Huang. Dignet: Learning decomposed patterns in representation balancing for treatment effect estimation. Transactions Machine Learning Research, 2024.
[39] Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual inference. In International conference on machine learning, pages 3020-3029 PMLR, 2016.
[40] Fredrik D Johansson, Uri Shalit, Nathan Kallus, and David Sontag. Generalization bounds and representation learning for estimation of potential outcomes and causal effects. The Journal of Machine Learning Research, 23(1):749--538, 2022.
[41] Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics, 17(2):3008-3499, 2023.
[42] Newton Mwai Kinyanjui and Fredrik D Johansson. Adob: An alzheimer's disease simulator for benchmarking observational estimators of causal effects. In Conference on Health, Inference, and Learning, pages 103-118. PMLR, 2022.
[43] Toru Kitagawa and Aleksey Tetenov. Who should be treated? empirical welfare maximization methods for treatment choice. Econometrica, 86(2):591-616, 2018.
[44] Kun Kuang, Peng Cui, Bo Li, Meng Jiang, Shiqiang Yang, and Fei Wang. Treatment effect estimation with data-driven variable decomposition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[45] Kun Kuang, Peng Cui, Hao Zou, Bo Li, Jianrong Tao, Fei Wu, and Shiqiang Yang. Data-driven variable decomposition for treatment effect estimation. JEEE Transactions on Knowledge and Data Engineering. 34(5):2120-2133 2020.
[46] Daniel Kuhn, Peyman Mohajerin Esfahani, Viet Anh Nguyen, and Soroosh Shafieezadeh- Abadeh. Wasserstein distributionally robust optimization: Theory and applications in machine learning. In Operations research & management science in the age of analytics, pages 130-166. Informs, 2019.
[47] Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, 1 116(10):4166-416 2019.
13

---

[48] Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization. Advances Neural Information Processing Systems, 33:8847-8860, 2020.
[49] Shuangning Li and Stefan Wager. Random graph asymptotics for treatment effect estimation under network interference. The Annals of Statisticx, 50(4):2334-2358, 2022.
[50] Yijun Li, Cheuk Hang Leung, Xiangqian Sun, Chaoqun Wang, Yiyan Huang, Xing Yan, Qi Wu, Dongdong Wang. and Zhixiang Huang. The causal impact credit lines on spending distributions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 180-187, 2024.
[51] Christos Louizos, Uni Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep vaaiilee models. Advances in neural information processing systems, 30, 2017.
[52] Shumin Ma, Cheuk Hang Leung, Qi Wu, Wei Liu, and Nanbo Peng. Understanding distributional ambiguity via non-robust chance constraint. In Proceedings of the First ACM International Conference Al in Finance, pages 1-8, 2020.
[53] Divyat Mahajan, Ioannis Mitliagkas, Brady Neal, and Vasilis Syrgkanis. Empirical analysis of model selection for heterogenous causal effect estimation. International Conference Learning Representations, 2024.
[54] Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the wasserstein metric: performance guarantees and tractable reformulations. Mathemati- cal Programming. 171(1-2): 1766162):15--66, 2018.
[55] Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299-311 2021.
[56] Ana Rita Nogueira, Andrea Pugnana, Salvatore Ruggieri, Dino Pedreschi, and João Gama. Methods and tools for causal discovery and causal inference. Wiley interdisciplinary reviews: data mining and knowledge discovery, 12(2);e 1449, 2022.
[57] Yung-Kyun Noh, Masashi Sugiyama, Song Liu, Marthinus C Plessis, Frank Chongwoo Park, and Daniel D Lee. Bias reduction and metric learning for nearest-neighbor estimation of kullback-leibler divergence. In Artificial Intelligence and Statistics, pages 669-677. PMLR, 2014.
[58] Miruna Oprescu, Vasilis Syrgkanis, and Zhiwei Steven Wu. Orthogonal random forest for causal inference. In International Conference on Machine Learning, pages 2019.
PMLR,
[59] Harsh Parikh, Carlos Varjao, Louise Xu, and Eric Tchetgen Tchetgen. Validating causal inference methods. In International Conference on Machine Learning, pages 17346-17358. PMLR, 2022.
[60] Georg Ch Pflug, Multistage stochastic decision problems: Approximation by structusss and ambiguity modeling. European Journal of Operational Research, 306(3):027--109, 2023.
[61] Zhaozhi Qian, Yao Zhang, Ioana Bica, Angela Wood, and Mihaela van der Schaar, Synctwin: Treatment effect estimation with longitudinal outcomes. Advances in Neural Information Processing Systems, 14:3178-31991 2021.
[62] Craig A Rolling and Yuhong Yang. Model selection for estimating treatment effects. Journal of the Royal Statistical Society Series Statistical Methodology, 76(4):749-769, 2014.
[63] Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55 1983.
[64] Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions.
Journal of the American Statistical Association, 100(469):322331,, 2005.
14

---

[65] Yuta Saito and Shota Yasui. Counterfactual cross-validation: Stable model selection procedure for causal inference models. In International Conference on Machine Learning, pages 8398- 8407. PMLR, 2020.
[66] Alejandro Schuler, Michael Baiocchi, Robert Tibshirani, and Nigam Shah. A comparison of methods for model selection when estimating individual treatment effects. arXiv preprint arXiv:1804.05146. 2018.
[67] Alejandro Schuler, Ken Jung, Robert Tibshirani, Trevor Hastie, and Nigam Shah. Synth- validation: Selecting the best causal inference method for a given dataset. arXiv preprint arXiv:1711.00083. 2017.
[68] Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In International conference on machine learning, pages 3076-3085. PMLR, 2017.
[69] Claudia Shi, David Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. Advances in neural information processing systems, 32, 2019.
[70] Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effeets using random forests. Journal of the American Statistical Association, 1136623):1228-12444 2018.
[71] Chi Wang, Qingyun Wu, Markus Weimer, and Erkang Zhu. Flaml: A fast and lightweight automl library. Proceedings of Machine Learning and Systems, 3:434-447, 2021.
[72] Jie Wang, Rui Gao, and Yao Xie. Sinkhorn distributionally robust optimization. arXiv preprint arXiv:2109.11 11926, 2021.
[73] Qing Wang, Sanjeev Kulkarni, and Sergio Verdú. A nearest-neighbor approach to estimating divergence between continuous random vectors. In 2006 IEEE International Symposium Information Theory, pages 242-246. IEEE, 2006.
[74] Anpeng Wu, Junkun Yuan, Kun Kuang, Bo Li, Runze Wu, Qiang Zhu, Yueting Zhuang, and Fei Wu. Learning decomposed representations for treatment effect estimation. IEEE Transactions on Knowledge and Data Engineering, 35(5):4989 5001, 2022.
[75] Liuyi Yao, Zhixuan Chu, Sheng Li, Yaliang Li, Jing Gao, and Aidong Zhang. A survey on causal inference. ACM Transactions on Knowledge Discovery from Data (TKDD), 15(5):1-46, 2021.
[76] Liuyi Yao, Sheng Li, Yaliang Li, Mengdi Huai, Jing Gao, and Aidong Zhang. Representation learning for treatment effect estimation from observational data. Advances neural information processing systems, 31, 2018.
[77] Jinsung Yoon, James Jordon, and Mihaela Van Der Schaar. Ganite: Estimation of individualized treatment effects using generative adversarial nets. In International conference on learning representations, 2018
[78] Linying Zhang, Yixin Wang, Anna Ostropolets, Jami J Mulgrave, David M Blei, and George Hripcsak. The medical deconfounder: assessing treatment effects with electronic health records. In Machine Learning for Healthcare Conference, pages 490-512, PMLR, 2019.
[79] Yao Zhang, Alexis Bellot, and Mihaela Schaar. Learning overlapping representations for the estimation of individualized treatment effects. In International Conference on Artificial Intelligence and Statistics, pages 1005-1014. PMLR, 2020.
15

---

Appendix
A
CATE Estimation Strategies
A.1
CATE Learners
We now detail how to construct CATE learners using the observed samples {(x, T, Y:) Note that CATE learners are learned on the training set, so the sample size n here equals the training sample size. Denote n, by the sample size in the treat group, and n. by the sample size in the control group such that n. + Re.
S-learner: Let predictors=(X, T). response=Y. Train a model ((X, Then we obtain tx()):
is ts(X) = ((X, -
T-learner: Let predictors=X" (covariates in the treat), response=Y (outcome in the treat). Train model (x)). Let predictors=XC (covariates the control), response=y (out- come in the control). Train model (i0 (X) Then obtain (X)::
XX i (X) (x))
PS-Iearner: Fisrt-step: Train (X) using the above-mentioned step in S-learner. Second- step: Let predictors=XX, response=fg(X). Train model fps(X) from the following objective:
Σ (T(X.) -s()))
ApS arg min
7
IPW-learner: First-step: let predictors=X,, response=T. Train propensity score model # (X). Construct surrogate of CATE using pseudo-outeomes with inverse propensity weight-
TY and Yfrw = (1-TY ing (IPW) formula: Y/PW = Yirw - Yirw. where Y/rw #(X]
-i-((X)
Train model (Xpw(X) from the following objective:
frew = arg min 1 Σ (r(X.) - 11.
)
X-leamer [47]: First-step: Train (1 (X) and (x using the the mentiooee proce- dure in T-leaner. Train propensity score model #(X) using the the above-mentiones procedure in IPW-learner. Second-step. Let predictors=X response=ji (X*) Y and
predictors= response=jinX'9 yc, Obtain model fx(X) by learning two separate functions i (X) and # (X):
(x) (1
+
* arg min
(r(x)) - -(Y; - -ooo)))))))
X arg min > (r(X,) - (x.) - 22))
First-step: Let predictors=X, response=Y. Train a model ((X) to approximate the conditional mean outcome X] Train propensity score model â (X) using the the ahove-mentioned procedure IPW-leamer. Second-step: Compute the outcome residual ş Y fx(X) and treatment residual T #(X). Train model ty (X) from the following objective:
U-learne 25 55
ीF = arg min
Ui
16

---

[41 26]: First-step: Train (x( and Po(X) using the the above-mentionel procedure in T-learner. Train propensity score model ( using the the above- mentioned procedure in IPW-learner. Second-step: Construct surrogate of CATE us- ing pseudo-outcomes with doubly robust (DR) formula: y-1.0
DR YDR YD where I-T (Y (x))) Train DR #1 (X) + FY ji(()) and YD Ao(X) I- model (x)(X) from the following objective:
*DR = arg min
- (r(x,) V(DDR)
R-learner [55] First-step: Let predictors=X, response=Y. Train model jx(X) to approxi- mate the conditional mean outcome [X] Train propensity score model #(X) using the the above-mentioned procedure in IPW-learner. Second-step: Compute the outcome residual Y (xX) and treatment residual T #(X). Train model fx(X) from the following objective:
fR arg min Σ (5 vTT((x)) 1=1
RA-learner [8]] First-step: Train (X) and (X using the the above-mentioned pro- cedure in T-learner. Second-step: Construct surrogate of CATE using pseudo-outcomess with regression adjustment (RA) formula: YRA TRA=T' =T(Y - (o(X)) + (1 Y).
Train model fRA(X (X) from the following objective:
RRA = arg min
(T(())) n2n
72
A.2 CATE Selectors
We now detail how to construct CATE selectors using the observed samples {(x, Tú Y:) Note that CATE selectors are constructed on the validation set, so the sample size n here equals the validation sample size.
Plug-in selector: Obtain any CATE learners F using the observational validation data. Then plug-in F into the following metric Relug (+):
Relug (f)
E(((x) - F(X.))2
For each plug-in selector 7, the selected j' CATE estimator is fg- where j arg minnje(,......) R Reing (fy).
Pseudo-outcome selector:
1. Pseudo-DR: Utilize validation data to estimate nuisance parameters (Ø1, iu 7), fol- lowing the procedure described in Section A.1 YDR
V Y where I-$
YDR (X) + #(x) (Y in (X)) and DR Aa(X) + --x(x) (x)))
Then the pseudo-DR metric is
Raseudo (f) D
(+(x)) YTDDR)
For pseudo-DR selector, the selected -" CATE estimator is fj., where j* arg migmijee(1....))) Reseudo
2. Pseudo-R: Utilize validation data to estimate nuisance parameters (Ā, #). following the procedure described in Section A.I Then the pseudo-R metric is
1
poeeudo (f)
(LG ((X))) +(X) -xx)))
72
1=1
17

---

For pseudo-R R selector, the selected j'hth CATE estimator is 6g., where j
arg (ig.in R Re (fz).
3. Pseudo-IF [3]: Utilize validation data to estimate nuisance parameters (Ā1, Mo; 7t,
following the procedure described in Section A.1 Let F(X) (iI - (x())). Then the pseudo-IF metric is
Rpseudo (f) -
Σ =31
(11 - (.))()))) B..((())) - A.(F(X.) - -FAX)))--(XXj))) +
where A, = Ti -
- F(,)). B, =
C. (1=
For pseudo-IF selector, the selected j" CATE estimator is fg*, where jr =
arg arg. R/F Re (fz).
4. Other pseudo-outcome selector: By manipulating the formula of Y. it is possible to create additional pseudo-outcome selectors, such the pseudo-IPW selector. In our paper, we choose pseudo-DD MR as the baseline because it is representative in the causal inference literature and often demonstrates superior performance, owing to doubly robust property.
B Proofs
B.1 Proof of Proposition 4.1
Proof.
B[(f(X) - Terue (X))²
= =E[(f(X) - (A1(X) (x())))))))
[[( (x(X)+++++++)))+)))) (x
[[(+(X) -12
[FI1:(3971-11|/1333--22111111111 YY |=,p::/:00000011///1101/11-11111
Yoy
+
|3 (X)]
=177:351-00000000002277777711701 2E (X (X)))
B.2 Proof of Proposition 4.6
The following Proposition B.I is useful in proving Proposition 4.6
Proposition B.1. Assuming the random variable tuple (X, T, Y Y²) satisfies Assumption 2.1 we
have
v Y 0) = p Y X)p(X 0); , X 1) = x xXx(XX
(17)
18

---

Proof.
p(X,y yo VX 1-1T= =0)
=p(Y", Y' =)p((X[T'= =
=P(Y", r xx)) = =0).
(Unconfoundedness)
p(X,y' x-1T==1 =)
=p(r",
= "[XxII=1)(XX] =I)
= Y (1xx,, = 1).
(Unconfoundedness)
Now can prove Proposition 4.6
Proof.
(kI(Po|(PT)
=DKL(P(X, Y O Y Y' =1)
= -/x yo yi
p( 1' 00) log p(u, 1. yy T = 0) dyy dy da
p(z, 10, y 1)
= -Ix /
plu M 111)p((/|TTT- 0) log p( 3'1x)p(<x11 0)
p(0.. vy |z)p(x(TT =1) dy dy dr
(By Proposition B.1
= -Ix 1 u
p(x[ = 0)
(x)p(=]T 0) log
p(x 1) dy' dr
Ix -Ix (1. [, p(y", 1x) p(x log p(ar T = 0)
-
p( = dx
= =f, P(alTT= = 0o
p dx
p(x = 1)
= O)
1))
=DKL -D (
Similarly, it easy show DEL (P) DRI (PEIIP)
B.3 Proof of Theorem 4.4
Lemma B.2 (Theorem in Let fo(X) denote the loss function of X and is bounded almost surely. E θ represents the model parameters of the function fe(X). Let (P) be the uncertainty ball centered at distribution P with ambiguity radius Define the mass of the distribution P on its essential supremum (Proposition 2 in (349). Assume fo(X) is bounded and log k + < 0, then we have
sup [f] = min de + X log E (ox((((((((X)/1) (EB.(P)
=
ASO
Our Theorem follows by directly applying the above Lemma B.2
B.4 Proof of Theorem 4.5
For notational simplicity, we denote W = (X,T,Y) e W and 2 = f(X)Y. Assume is bounded within the range M and M. Define the following functions:
Go(Ao: W) Elgo(Ao: W)), Go(Mo W)
n Σ 90 (Ao: W.).
1
where (o0; W) = (1 - (T) exp (Z/2o);
Gi(A;W) = W)) Gi(A;;W) = Σ (A1;
im1
where ()1s W) = T exp (- (-Z/>1).
19

---

Then we have the following lemma that guarantees the convergence for Go(Do; W) and G1 (A(: W). Lemma B.3. Assume 0 Ao A1 S A, and f(X)Y bounded within the range of M M.
Then with probability 8, we have
If M S M 0:
2 log 3/d (exp (M/
Go(A: W)
n
2og 3 (exp (-M/)))
(G](A;; GI(A;; W)
n
If M S0
2 2og Ah (exp (M/)))²
Goo(>: W) Go 5I
(18)
2 log 3 (exp (M22))
(G((a: G) 0
n
FOS M S M
2 3/ (exp (M/
Go(Ao: W) Go(Ao: 50
n
2 2og & (exp (-M/2))
Proof. Denote ho(W, 2 W = E Ao W.). We notice that ho W W.)
satisfies the bounded difference inequality:
sup
loo (WW,...... W,, Ww) ho(W,, W/. W.)l
Wi.... W....,... WWW
(Ao: W.) 90 (Ao: Wi)
sup W. W.W
190 (oo: Ws) sup
2 2xxp (M/Ao)
W.ew
n
n
Note that Go W) Go(An; W = 1 (W,, W1. W2...., W., E[ho (w, We,-.. W.)]l Then using McDiarmid's inequality, for any 0, we have
Go(Ao: W) - Go(Ao: W) 2()
P (Who(Wi, W2, Wn) B[ho(W1, W2, ...... Wn]] 2 e)
2
-nE² 2(exp (M/20))
2xxp
2exp
2exp( (exp(Xi/Ma) M1/2a)
n
For some d > we have
-nE?
P ((Go(2o: w)
GolAo: w) 26) 2exp
88.
2 (exp (M1))
This solves such that
2 2 2h (exp (M/ (M/
20

---

The above inequality should hold for any A such that < A S Ao S X. Therefore, we have
2 2 2 exp (M/)
If 20:
c2
2 21og 2 exp (M/ M/
M 50:
62
Similarly, denote h1 (W1, W Wn 1 Σ
"h 91 (A1; Wt). We note that h1 (W1, W2,..., Wn)
satisfies the bounded difference inequality:
sup
(W1, W..... W.) (W,, W ... W.)
WI.... WJWW
(AI; W.)
91 (A1; (!1:W
sup W/W
n
191 (A1; W:)l 2 exxp (-M/A))
2 sup W.ew
n
Then using McDiarmid's inequality, for any > have
P ((Gi(Mi:W) Gi ()i;; 2e)
(1h W2, ..,, Wi.) E[h, (W1, W
W 2)
2c²
nc²
< 2xxp
exxp
n( 2expl
2 (exp (-M/X1))
For some d > 0, we have
-ne?
-G( 2() 1 2exp
2 (exp (-epp-MM/II (-M/>1))² 5.6
This solves such that
2 log 2 (exp (-M/A1))
<
The above inequality should hold for any d such that < 2 A1 S A. Therefore, we have
2 2log (exp (-M/X))
If 2 0:
EZ
n
2 2oo (exp (- 2/22) If 5o: 0
n
In the following content, we will bound terms log(Er (A1; W)) log (G1(A1; W)) Lemma is useful for bounding these two terms.
[og[GG (Ao: W)) log (Go (Ao: ()o:
and
Lemma B.4, Let be constant. For any 11, 32 such that 21,42 2 <>> O, we have
[oo((x)) - (2) < 20 x22
(19)
Proof. Without loss of generality, assume 0 < S σ1
< Ø2 We then have
T2
22
I2 21
log(22) log(z1) log z1
log(1 +
I1
σ1
Taking the absolute value both the left-hand side and the right-hand side, we have
1
[oo((11) log(z2) <
21 Ø2/
21

---

Next, we introduce Lemma B.5 that bounds terms
log((G(Aoo: W)) log (Go(Ao; W)) and
log(Gi (A1; W)) - log (GI(1;;))) W))
Lemma B.S. Let denote the probability of treat, ie., P(T =1). Assume that Xo A1 E A
(d. Aj and F(X)Y is bounded within M and M. Then for 2 max f7t log (3) (1-u77 ( log (z)
2
with probability
we have
If M < M 0::
log Go (oo; W)) log (Go(An; W))
<
2
W) - Go(Ao; w)|; Go(Ao: W)
exp( M/2)(1 - u)
log(G](A) W)) log (GI(A;;W))
(G(A::1)-CG(A1:W
exp(-M/ -M/A)u
If M S 0, M 2 20:
log(Go (Ao: W)) log (Go(An; W))]
2
Go(Ao: W) - Go(do; w//: (20)
5 (xp(M/A)((
log((i(A1: W)) log (GI(A); exp(-M/AJ)
f M S M
2 (xp(M/A)(11 - w) W) - Goldo:W)) -
log(Go(Ao: W)) log (Go(Ao:W))
log(Gi W)) log (Gi(Ar;; exp(-M/A)a
-
Proof. First, we bound the term [log(Go W)) - log (Go(Ao: w))(
Go(Ao; W) and Go(Ag; W') are greater than 0 and bounded because Z = +())Y is bounded within the range M and M. Therefore, applying Lemma B.4
we have
log(Gu(Ao: W)) log (Gu(Au: W\)
Go(Au: W) Go(du: w\
c
GolAo: W3}
where e min
inf ADEA,WEW
Go(Ao; W),
inf We
Moreover, for any Ao E A, have If M 20: Go(Ao: W)
E((1 T) exp(Z/Ao) =
P 0)
2 E(exp(( (/ 0|(1 - u) = cxp(M/A)((1-)) u): Go(Ao: W)
1 Σ
(1 Ti) (1-T:)exp(Z//M)
(21)
i=1
1
(1 T.) (xp(M exp( i)
71
If M S0: Go(Ao:W)
(1 T) exp (x/p(/)))))))) =Bcxp((/>)) PT 0)
2 Elexp( M/ / 0/(1 -w) = exp(M/A)(1 -u);
Go(Ag:Wy
Σ
(1 - (1-T,) cxp((/AA))
(22)
n
i=1
1 n
(1 T.) exp M/2) exp(MM/)(1 ú).
7
Given á -i=1 T. and E Σn
Til using Hoeffding's 's inequality, have
P
Σ
(1 - T:) - E
1
Σ
T:)l
B E (1
T:)
< 2exp
2( 2(122)2 n {
S 6.
n
71
2
1=1
22

---

We can solve n by
n211 - u)²
2
2 exp
<d=n> (1 u)² log
7
2
This indicates that (1- ú) 2 u2/2 with probability 1 1 when n 2 [T-u]² log (3). Combining
this with equations (1 and 22
with probability 6, when n 2 [I- log ( we have
If M20: 0:
inf EWW
Go(Ao; W) 2 exp(M -u);
inf
Go(Ao: W) 2 exp(M/ (11)(1 a) 2 exp(M/A)(1 - -2/2.
WEW
If MS:
inf WEW Go(Ao: W) 2 exp(M// u);
inf WeW
Co(Ao: W) 2 exp( M/2)(1 û) 2 exp(M/A)(1 - -u)/2.
Therefore, with probability 1- 8, when 2 uy- log (3). we have
If M22
[oo((G((oo: W)) - log (Go(An; W)) S
2
GolAo:W) (o((oo:))
exp(A (xp(A//A)( u)
If M :
Go W)) log (Go(An: w)) S exp(AM/A) (M/
2
Go(Ao: W) Go W
Next, we bound the term log( (A1: (A11 W)) - log (Gr (G1 W)) G1(4;; W) and Gi(A:: are greater than and bounded above. Therefore, applying Lemma B.4 we have
1 G1(Nn:W) Gi(:::
log(Gi(A1:)) (e1;G((11:W)) log (GI(A1;;)))
c
where c min
inf WEEAWEW
Gi();;W), inf X.A.WEW
G(Ai:W)
Moreover, for any A E A, have
If M 00: Gi(A1; W) E[T exp( ///1]]] = ((exp(-Z/A))TT=1PPP((
2 Elexp(- M/ -1/u- exp(-MMA)uu;
Gı(A;:W) =
T, exp(-Zi/A1)
(23)
iml
Ti exep(-MM/2 M/2) = eXp(-
If M <0: G((A:) = E exp( - 2/\1]] = E/ssp(-Z/AAI
2 T 1|u exp( cxp(-M/A)u;
G((A:W)= =
Ti (xxpp(22/A1)
(24)
i=1
2 n
T3 exp( M/A) = e/p(--M/Aja.
Given á = Σn T. and
E Σ
Til. using Hoeffding's inequality, have
1
E ΣE
T.]
(- 22)))² )2
Ti
E
M
T.
< 2exp
S
n
2
n 2
23

---

We can solve n by
nu²
2
2 exp
< + n 2
log
2
u
8
This indicates that hatu 2 2/2 with probability å when n2 2 2 log (g). Combining this with
equations
(23 and
24
with probability 6, when n 2 -/2 log (7),
we have
If FM26
inf WEW.WEW
G1(A1; W) 2 exp(-M/2)u;
inf MEA,WEW
G1(A1; W) > exp(-M/ MM A) exp( exp(-M/A)u/2
If S0:
inf WEW
Gi(A1: W) 2 exp( exp(-M/\)u;
inf WEW.WWW
GI(:;: W) 2 M/Aja 2 exp(M/A)u/2.
Therefore, with probability 1- 8, when n 3 log (3) we have
If 2 0:
2
log (GI (Gi():; s exp(-M/AJu
If 50:
log( (11; log (Gi (Gi;
2 exp(-M/AJu
This completes the proof of Lemma B.5
Additionally, the following Lemma B.6 provides the bound of log(ū) - log(u)]
Lemma B.6. Let u Σm= T. and u E Σi= Ti]. For 1 log (g). with probability
1 8, we have
(V2200
(200g(3) (3)
log(ū) - log(u)) 50
(25)
nur
Proof. Using Hoeffding's inequality, we have
P([ā u| 2 2) P ( T.
E
Til
2
2exp (-2ne?)
log
2 exp (-2ne)) <
solves
2n
Notably, using the results in the previous lemma, we know for 2 log (2), u/2. Therefore, we have
log (i) log(u)) 5 min ât u)
u/.
(By Lemma B.4
2 20 ul
log :
(v
2 ?) nu?
=0
u
u1
2n
24

---

with In the following, we will bound the term
- (V(F)-((F) using above lemmas. We first define functions
Fo(Ao), Fo(oo) Fi (A1), and (A1):
Fo(An) Aoco do log ( (exp(+(X)\/Ao)))
=X0E0 do log
u E[(1 - exxxxx(X(((((
Fo(An) = Aoco + do log(
(1 - exx((()))))))
nee 1=1
= = + do log
Σ (1 T:) (xpe)epp (xp(((X))Y//A))
n(1 - ù)
(((A) λleι A log(R
(exp(==(X)Y/AA))
= A log
FF[Texp(->(X)>/>
A())) = dLY
+ d log(
T.
1=1
D11 =
+ Ao
T, expl ( Σ
nu
1=1
The following Lemma B.7 bounds the term (F(A)=((A))
Lemma B.7. Let
P(T 1). Assuming that 0 < S X S X and f(X)Y is
bounded within the range of M to M. Define Cezp (MSMSO) exp (M/X M/2)
1 (M exp (M/2 M/2) 1 -1oosMs exp (M/2- M M/X). For 2 2/ /u2 log probability 8, we have
(2/5), log(2/8)
SA8 log
2x log ) V
(Bo()o) (6000)
(v
C
+0
n11
eup
n(1 u²²
(26)
8X7 log 2 nu²
2x log
(AA) -A(A 50
C2
+ 0
nua
Proof.
(Fo()o) -Fo(Do)
a
No log 11
E[(1 T) exp(+(XX)//Ao)) log
Σa T:) exp exp(+(X.)Vv/AA))
n(1 - ú)
1 n
An log (E[(1 T) exp(F(X))/AA))) log
+ -log(1 - -i) - -log(1 (1 - -u)
n
1=1
1 n
5 An log (E[(1 -
log
(1 (x--T)xxx(((()xAAA))))
+ log(l - -)) - -log(1 - -w)l
71 1=1
25

---

If M S M 50:
(oo(Ao) - Po(()
-Co(Ao) w/| + do |log(1 u) llog(1 u)1
- Go(do:
(By Lemma B.5
exp(1//2((1 u)
0 8AD log 3
(exp (MI/ M/21)) +0 (1
(22
223 log(?)
(By Lemma B.3 and Lemma B.6
721 u)²
n11 u)²
If M S M, 20:
(o() (oo)
2λo
Go(Ao: W) - Go(Ao: W3) + do [log(1 -i) -log(1 u)l
(By Lemma B.5
exp(1//2(1 u)
(exp (M/A M/21) +0 (1
8A8 log 3
228 log(
0
(By Lemma B.3 and Lemma B.6
n11 u)²
n(1 u)²
If 0 < M < M:
Fo(Ao) Fo(1o)l
2λo
Go(Api W) Go(Ao: w)| No |log(1 úi -log(1 u)
(By Lemma B.5
exp(1f/2)(1 u)
(V
(exp (M/2 - +0 (1 (1222 log(
8AA log 3
(By Lemma B.3 and Lemma B.6
12(1 u)²
nn1
u)²
(F((A) F((A))
d log (1 a
1 1 "T, (xxx((((X)))/A))
E(T [[Txxx(-+()))//] log
nu
A log (E(Texp(((X)V/A)) log
7, Texp(((X.)5/AA)) + +log(m) log(u)
72
< A log (ETTex((((X)Y/A))) log
Σ Texp(+((.))/AA))
> [log(1) - log(u)).
26

---

If M S M 50:
1 - Pi(Ai)
2Å1
Go(No: Go(Ao: W) + A1 [log(() log(u)]
(By Lemma B.5
exp( HOp(-M/Aju
+ (1
8AG log
2 log(
S O
(exp (M/² (M/)
(By Lemma B.3 and Lemma B.6
nuu
nu²
If M S O, 20:
(i(A)) FI(A1)
2λ1
G) Gi(;: + + [log(() log(u)l
(By Lemma B.5
exp( - M/2Ju
+ +0 (v
BAG log
2X log(
(exp (M/2- 2/21)²
(By Lemma B. and Lemma B.6
nu²
If 0 & M & M:
(1 (A1)
2A1
G) Gi(1;; + +A1 |log(() - log(u))
(By Lemma B.5
exp( exp(-M/A)u
+0 (1
SA7 log nuu
2A log(
(exp (M/2- -
(By Lemma B.3 and Lemma B.6
nu²
Now, we can prove the result in Theorem 4.5
Proof. Let Åg arg miny Fo(Au), Ao = arg minx, Fo(Aa), Ån = arg minA (11)) and X;
arg minx, (A1) Then we have
(+( - (F) = Fo(A5) Fo(Ao)
Fo(A)) Fo(Do) Fo(Ao) Fo(Ao)
Folda) Fo())) Fo(A5) Fo(do)
I (Fo(So) - (So)) +0
I sup Fo(2o) Po(M))).
(0( - (+) Po))
1o(25)
(20) - Po(A5) Fo(A5) Fo(Ad)
Fo(A5) - Fo(25) + Fo()v) Fo(A5)
I (E6(AA) Fo(A6) +0
S sup Fo(Ao) Fo(2o))-
No
((A)-D1G = (AT) fi(AA)
Fi (Xt) Ai(A + Fi()) Fi(2.)
F()) AAA) F(())) F(()
$1FI(A)) - AA()) +0
s sup Fi(A,) (.D)).
A
27

---

-' - A(A)) P((A))
A(A)) RI(A)) F((A)) F((2))
PI(AT) - +1())) + (A())))))))))))))))))))))))))))) -())))((A))
S (FF( - FIDI +0
S sup (F(())) Fi(A))
A
Therefore, we have If M S M 0:
(v
8X2 2
2X log( ( nn1 un2
- - (+)l I sup 5O
n11 u)² (exp (M/² 2/22)²
+
8X2 AX2log? 3 nu²
2Xa log nu?
(b) - v'(()) S sup ((A)) F0 -FAN)
-M2)))²
+0
If M S 0, M2 20:
(1
8A 872 log 2 n11 w)²
2X log( ?
(F) (F) s sup ((()) =O X
(exp (1/ 2/A2))²
+0
nn1 u)²
(1
8X2 log nuu2
2X log(?)) 2 nuu2
(F) v'(+)| s sup - F(WI
(exp (M/A M/2))²
+0
S M:
(1
8X2 log 2h n2(1 - u)²
2X log )
(D) (+) S sup supp -FA))
F(JI
(exp (M/2 -M/X))
+
(v
n2(1 u)²
(V
882 log 3 1 nu²
2X2 log( 2 nuu2
-(A) ( S sup
(exp (MI/2 (M//)-M/1))
M/1))²
+0
X
Finally, we have
vi+)) (V
8821088 nuf
882 log 2
2X log( 2
Coep
+0
nuf
Note that d1 P(T 1) and uo = P(T = 0). Ceep 1 (MveNTs)) exp (M/A MM2)
1 (M50,M20) exp (M/2 - M/2) + (OSMEM) exp (M/2 M/X).
C Additional Materials
C.1 Additional Explanations
QI. Why DRM can select CATE estimators that are robust to the uncertainty in PEHE caused by selection bias and unobserved confounders? In Section 3.1 and Section 4.1, we have presented theoretical explanations for the reason why DRM can measure a CATE estimator's 's robustness against selection bias and unobserved confounding. Below we will explain it more specifically.
In causal inference, all the CATE estimators are constructed on the observational factual data. But how reliable the CATE estimator that learned on factual data is? This question can be never known unless we have the knowledge of the oracle PEHE, As shown in equation (6). we know that the PEHE is equal to two f-dependent terms, ( = and (I) = 1 - t).
28

---

Unfortunately, T = = -1 is uncomputable empirically because we can only observe the factual distribution p P Ł) but not the counterfactual distribution per P(X. YKIY = f). The unobserved counterfactual distribution can be regarded as an uncertain distribution varying around the observed and certain factual distribution p If we could assume a "God's perspective" and observe PCF directly, the counterfactual distribution will be certain like a quantum world! Such an uncertainty in pCF results in the uncertainty in PEHE. Now we will analyze the source of such uncertainty by analyzing the relationship between the uncertain distribution per and the certain distribution p based on equation (2):
P(X,Y = 1 - P(X, I =1) P = - x) P(X = 1 P(X[== =
-
From above, we find the unobservable distribution P(X, Y = 11 t) is equal to the observ- able distribution P(X, Y 1) multiplied with PlY'T=I- P(X(T=1-1) P(Y")=LX) P(X]T=)) In other words,
T(y(T=l p(x(T=1-t) controls the discrepancy between pF and PCF, Note that if there is no P(y")T=t,a) p(a|==1)
1,2x) - ply' IT 1, and if there is no = selection bias (covariate shift), then we have =1 = p(x t). Now we understand
unmeasured confounders, then we have p(y" |T
the root cause of the discrepancy between p and pcr (or between E[ = E|T(X)Y* = f)) lies at the unobserved confounders and selection bias (covariate shift). In the DRM method, the uncertainty caused by potential unobserved confounders and selection bias in PEHE can be further measured as the distributionally robust values V and D Then the PEHE w.r.t. the CATE estimator ē will be at most R RDMM as shown in equation (15). An estimator f that attains smallest RDRM (f) by definition reflects the distributional robustness against potential unobserved confounders and selection bias.
and
Q2. How to set E" when there are unobserved confounders? When unobserved confounders are present, Proposition 3.6 can also provide guidance for setting e Taking ei = (Px(Pc))P) as an example, we have
D (PP(Pt
-I I / T = 0) log
p(y y zz, T = 0)p(2 T 0)
p'y,, y x T 1)p(x T= dy dy
y
-Ix (L ( p(u lo dyy) p(z|T = 0) log p(x|T 0)
dx
p(x|T'= =
* I p(y |x, T 0)p(a/7 = -00(aa==-)0000000000000000
log p1 y² x T 0)
x, T = dy dy dx
p(y5 1 y
I I plo | 0)p(2|T = T=ppa 0) log p yy x. T 0) (PL(F)
ply M,y,, y x, T = dy 2 dy
Therefore, when unobserved confounders present, we can set c* to a larger value than the one guided by Proposition 4.6 Simultaneously, the empirical approximation of cî = Dx ( (P&) PP) and 6 D P I can be biased, also suggest set c" to a large value than the empirically- computed ones to ensure the ambiguity set is large enough to contain the target distribution. Therefore, we generally set e D (P(()))) + 5.2 and o D (P& (IP)))) ) + 5.2 for all settings in our experiment. Theoretically. a larger e" should guarantee the DRM-selected estimator to be more robust, it allows for broader range of possible counterfactual distributions in the ambiguity set. However, setting c* too large can result in overly conservative estimator selection (similar to the well-known accuracy-robustrre
tradeoff). Therefore, how to determine proper ambiguity radius still remains an open challenge in both our work and distributionally robust optimization literature.
C.2 H Hyperparameters
For linear model, we use LogisticRegressionnCV and RidgeCV (both are with 3-fold cross- validation) from sklearn package to tune hyperparameters: Logistic regression: Cs E (0.01, 0.1, 1, 10); Ridge Regression: a E (0.01, 0.1, 1. 10, 100).
29

---

For Neural Net. we set the hidden layers as [200, 200, 200, 100, 100], each with the ReLU activation function. The model is trained using the Adam optimizer with a learning rate of 0.001, batch size of 64, and 300 epochs.
For RF, XGBoost, and SVM model, we use AutoMI. 71 53 from flaml package tune hyperparameters.
(with 3-fold cross-validation)
C.3 The Complementary Results
First, we would like to emphasize that the experimental results in this final version of the paper differ from those in the original version. We made several revisions based the feedback from anonymous reviewers: 1) We add Neural Net model to the base ML models and add the U-learner the meta-learners, increasing the number of CATE estimators from 24 to 36; 2) We adopted AutoMI. for hyperparameter tuning when training SVM, RF, and XGBoost. All code is available at https://gi http:///gttuuu..c thub. wtt.:/oooyyyyynn/////m hatpaaannn/AEEEetttaatttttrrrre wer.EEesttmtttr.eememmm wettimattorsseletttioo.
Below, we prsent the complementary PEHE results for 36 candidate CATE estimators, where the candidate pool contains 4 ML models (LR, SVM, RF, and Neural Net) x 9 learners (U-, S-, T-, PS-, IPW-, X-, DR-, R-, RA-).
Table 3: Comparison of PEHE for different selectors across Settings B, and C (Note that B (E matches A (p = 0.1)), with base model for CATE estimator being (LR, SVM, RF, Net]. Reported values (mean standard deviation) are computed over 100 experiments. Smaller better.
(p
(p- 0.1)
0.3)
0)
( 2)
0.1)
0.5)
Cm 0.9)
Plug-U
49.59195.07
41.93:61.07
36.16161.77
2.28-2.32
155.24 291.74
42.45:52265
59,51:210.51
24.37426.51
Plug-S
5.10:8.29
5.36+5884
9.18+11.57
5.7615.46
8.78:7.55
13.45:9.53
Plug PS
4.80+1.74
5.36+5885
6.28+1.75
1.99-141
9.17+11.58
5.76+5446
8.58+7.40
13.45+99.33
Plug-T
59.39421.24
Plug-X
9.821100.6
10.39112.20
9.81
6,521
14:82-14:25
10.82:15521
15.80115.03
20.39113.03
Plug-IPW
35.09:28844
28.50=27.78
39.29
6.19-7.39
61 24.9112
41,47=31.37
30.02::-2.57
29.33::20:45
Plug-DR
44.83+226.77
46.47+42.02
48.23++26.56
5.98-8.09
67.87-18.94
49.61=13.34
33.694-23.5
32.39++9.22
Plug-R
3.6445.001
5.33=
5.51
2.19=2.31
13.04-31
4.95455.38
7.5627.88
70.91
Plug-RA
58.32.24.02
60.40120.13
38.631.22.56 86
8.37-9928
67.77
58.91
45.52421.50
42.13:120.35
Pscudo-DR
63.07::22.54
63 23.80:220
63, 10:11414
16.51:223005
73.29
65.12::20.1
53.871266
53.79124.91
16.83-445.81
9.97+21.23
18:13-30061
20.96+300.2
30.05+332.02
Pseuda IIF
66.26::15:20
16.21
15.72+15.84
28 28.49#23.55
69.01 16.
63.09=20.86
60,00+ 19.18
47.40+200.6
75824-23408
3453+14590
Fact
52.81=:18.01
53.58=:19.42
55.05 10
16.09:16.50
68.50=27 69
$1.96=17:45
$2.44+222.51
49.16::22.47
Matching
62.57::21.57
17.80
63.94118.72
64.56119.55
DRM
2.68:1:1.73
3.65
5.28116.37
2,14=1.70
18.77:112.78
4.60-19.58
6,44
10.05:1..19
30

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuii wwwticccguie//eeieeeeeeii CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
33

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ftttps://neurippe http//uurrr.cccuuu/////e http:///.cccpuccccttttttttteeeea
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
34

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
35

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "364IvWEGWb", "reviewer": "Reviewer_T6Zv", "summary": "The paper proposes a new model selection method for choosing an estimator of the conditional average treatment effect (CATE), namely, a Distributional Robust Metric (DRM). The proposed method, DRM, is nuisance-free: It does not require an additional estimation of the nuisance functions, unlike the majority of existing approaches. The DRM splits a precision in the estimation of the heterogeneous effects (PEHE) into two parts: the first term can be estimated nuisance-free and the second term is upper-bounded using the KL-divergence ambiguity set. Thus, the proposed method can only utilize the observational data.  Furthermore, the authors provided an estimation procedure for the DRM and the finite-sample convergence rates. Finally, multiple synthetic experiments were provided to prove the effectiveness of the method in comparison to the other existing approaches.", "questions": "- What ‘uncertainty in PEHE’ is meant in lines 164-165?", "limitations": "The main limitation of the method, i.e., the lack of consistency, was not discussed by the authors.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper studies an important and heavily discussed problem in causal inference, CATE model selection. I find the idea of using the upper bound on the PEHE based on the distributional ambiguity set original and novel. The paper is well-structured and easy to follow.", "weaknesses": "Two major weaknesses of the paper are, in my opinion, the following:\n1. A lack of understanding of the asymptotic behaviour of the DRM. Specifically, when the data size grows, the upper bound on the term (b) of Eq. (5) will still stay the same, as the epsilon-ball around $P_C$ or $P_T$ stays constant. Hence, there will be always a gap between the ground-truth PEHE and the DRM from Eq. (14). This renders the DRM not consistent. I encourage the authors to provide a discussion about this issue and some experiments with the varying data sizes, where they compare the DRM with other (consistent) nuisance-free selectors, e.g., nearest-neighbours matching.   \n2. It was not directly clear to me, why the optimal $\\epsilon^*$ is a forward KL-divergence but not, e.g., a reverse KL-divergence between treated and untreated covariates distributions. Note, that KL-divergence is not symmetric. Does it make sense to use the symmetric distributional distances? \n \nThere are also several minor suggestions for improvement:\n1. The hidden confounding setting needs to be considered more carefully. This setting requires different approaches, e.g., partial identification or sensitivity models, and, thus, fitting/selecting point CATE models is inadequate in this case.\n2. Some baselines were not mentioned in the related work or the experiments, e.g., U-learner [1]. \n\nI am open towards raising my score if the authors address the above-mentioned issues during the rebuttal.\n\nReferences:\n- [1] Fisher, Aaron. \"The Connection Between R-Learning and Inverse-Variance Weighting for Estimation of Heterogeneous Treatment Effects.\" arXiv preprint arXiv:2307.09700 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer T6Zv,\n\nWe are grateful for your thorough summary and greatly appreciate your recognition of the motivation and novelty underlying our proposed method. Thank you for your time and effort in providing constructive and helpful feedback. We will respond to each of your comments below.\n\n**Q1.** DRM is not consistent with ground-truth PEHE. Better discuss this issue and do some experiments with the varying data sizes, comparing with other (consistent) nuisance-free selectors, e.g., nearest-neighbours matching.\n\n**R1:** Thank you for this suggestion. First, we would like to emphasize that we did not claim the proposed DRM metric converges to the ground-truth PEHE. Our theoretical results only state that $\\hat{\\mathcal{V}}^t$ should converge to the $\\mathcal{V}^t$ at a rate of $1/\\sqrt{n}$. Consequently, together with the law of large numbers for other terms in $\\mathcal{R}^{DRM}(\\hat{\\tau})$ in Eqn. (14) should converge to $\\mathcal{V}_{PEHE}(\\hat{\\tau})$ in Eqn. (9) at a rate of $1/\\sqrt{n}$. Additionally, we believe it is not a weakness that DRM is not consistent with PEHE for the following two reasons:\n\n* By definition, DRM measures the robustness of PEHE w.r.t. $\\hat{\\tau}$. We can never observe the ground-truth PEHE due to the uncertainty incurred by selection bias and hidden confounders (as discussed in GR_1 of the top-page rebuttal). Therefore, instead of pursuing the unavailable PEHE, we aim to quantify its uncertainty using a distributionally robust value of PEHE: If some $\\hat{\\tau}$ has a small $\\mathcal{R}^{DRM}(\\hat{\\tau})$, then $\\hat{\\tau}$ is robust to the PEHE uncertainty caused by the uncertainty in counterfactual distribution.\n* To the best of our knowledge, no existing nuisance-free estimator/selector (e.g., the mentioned matching) is assured to be a consistent estimator of the ground-truth CATE/PEHE, but some nuisance-free estimators are consistent estimators of ATE (not CATE) given additional assumptions. E.g., (Abadie & Imbens, 2006) show the matching estimator converges to ATE at a rate slower than $1/\\sqrt{n}$, given the assumption that the density of covariates is bounded.\n\nPer your suggestion, we have compared DRM and nearest-neighbor matching with varying sample sizes in Table 2 of the attached rebuttal pdf reports the details and results. The findings show that DRM and matching exhibit a decreasing trend in regret with increasing sample size. However, when the sample size exceeds 10,000, the average regret for matching fluctuates between 6.5 and 7, while the average regret for DRM becomes steady. These results suggest that neither DRM nor matching converges to the ground-truth PEHE because a consistent metric should have a regret that tends to zero. Nevertheless, DRM demonstrates more stable and effective performance than matching as the sample size varies. \n\nRef: (Abadie & Imbens, 2006) Large sample properties of matching estimators for average treatment effects, Econometrica, 2006\n\n**Q2.** Why the $\\epsilon^*$ is determined as $D_{KL}(P_C||P_T)$ instead of $D_{KL}(P_T||P_C)$? Does it make sense to use the symmetric distributional distances?\n\n**R2.** Indeed, we set $\\epsilon_1^*=D_{KL}(P^C_X||P^T_X)$ and $\\epsilon_0^*=D_{KL}(P^T_X||P^C_X)$ in the original paper. In lines 222-237, we intended to explain the choice of $\\epsilon^*$ with $\\epsilon_1^*=D_{KL}(P^C_X||P^T_X)$ as an example, but forgot to explain $\\epsilon_0^*$, which led to a misleading presentation. We are grateful for your pointing this out, and we will revise the incomplete expression to make it clearer.\n\nRegarding your second question, we believe using symmetric distances would be more meaningful. The non-symmetric nature of KL divergence makes it fail to satisfy the triangle inequality and can make the divergence values difficult to interpret and compare between distributions. This restricts its practical use in fields like domain adaptation and transfer learning. Exploring the use of symmetric divergences, such as the Wasserstein distance, would be very useful in DRM (as discussed in line 343). However, solving the distributionally robust optimization problem with a Wasserstein-based ambiguity set can be computationally challenging due to the intractable dual problem involving an infinite \"sup\" objective. Therefore, considering alternative symmetric divergences that may provide a more tractable optimization framework for DRM would be a promising direction.\n\nMinor suggestions:\n\n**S1.** More baseline CATE estimators aiming at the hidden confounding setting should be considered.\n\n**R1.** At the current stage, one key focus is developing a CATE selection method that can robustly handle the distributional shifts. Therefore, the paper is mainly around “selection” rather than “estimation”. However, we strongly agree that considering more CATE estimators aiming at hidden confounder issues would provide more insights for testing different selectors under the hidden confounder setting, which will be further investigated. We will mention this interesting suggestion and leave it for future research.\n\n**S2.** U-learner can be included as a baseline CATE estimator.\n\n**R2.** We have incorporated the U-learner [1] into our experimental evaluation, and the updated results are reported in Table 1 of the top-page rebuttal pdf. We will add a discussion of the U-learner and incorporate the updated experimental results in the final version of the paper.\n\n**Q.** What 'uncertainty in PEHE' is meant in lines 164-165?\n\n**R.** We hope GR_1 of the top-page rebuttal has addressed this question well, and we will add these details in the Appendix to provide clearer explanations for the proposed method.\n\nFinally, thank you again for providing a thorough review and constructive suggestions! Your comments are valuable in helping us improve our work, and we will carefully incorporate your suggestions when finalizing the paper. We are very welcome any further questions or comments you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qxRuYdvjSl", "reviewer": "Reviewer_e7g2", "summary": "The paper introduces a new metric for CATE model evaluation and selection. Specifically, they derive a distributionally robust metric (DRM) which is nuisance-free and robust against selection bias. They show and explain its robust performance in extensive benchmarking experiments against existing baselines.", "questions": "- For selecting the ambiguity radius in Proposition 3.6, unconfoundedness is assumed. So why should the metric be still robust against unobserved confounding? Was the ambiguity radius in setting C selected differently?\n- Why is the ranking performance clearly worse than the average regret? The high standard deviation in the regret could imply that single runs with bad performance skew the evaluation which could be due to insufficient tuning of the used baseline models for the metrics. Clarification here would be appreciated. \n- What exactly do the authors mean in the limitations when they state “considering the ambiguity set constructed with other divergence such as Wasserstein may contain more diverse distributions”? What does more diverse mean here and what are possible shortcomings of KL divergence in this setting?", "limitations": "Limitations are mentioned sufficiently.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Robust and reliable CATE model evaluation is an important and often widely overlooked challenge in causal inference\n- The paper is well-structured and easy to follow\n- The authors provide proper theoretical derivation and extensive experimental validation for their method.", "weaknesses": "- The stated robustness against unobserved confounding sounds like a silver bullet. Here, the authors, provide experiments, but should more clearly discuss in theory why and to which degree of unobserved confounding their metric is more robust compared to others. Especially, since for selecting the optimal ambiguity radius they assume unconfoundedness. Hence, it is not clear if or why the metric is still robust, even though the experiments seem to support that.\n- The experimental results seem to be untransparent. The experimental setup for the baselines should be described in more detail, e.g. in the Appendix. It is not clear, if the models used for the baseline metrics were properly tuned. In particular, there is high standard variation in the Regret in Table 1, and large discrepancy compared to the observed performance in ranking in Table 2. Here, the setup should be described in full detail and possible reasons for this discrepancy should be discussed more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer e7g2,\n\nWe greatly appreciate your thoughtful feedback and insightful comments. Thank you for your recognition of our theoretical and experimental analysis! We will address each of your comments below.\n\n**Q1.** Should more clearly discuss why and to which degree of unobserved confounding their metric is more robust compared to others.\n\n**R1.** Regarding \"why\": Please kindly refer to GR1 of the top-page rebuttal.\n\nRegarding \"to what degree\", the level of robustness depends on the choice of the ambiguity radius $\\epsilon$. Theoretically, a larger $\\epsilon$ should guarantee the DRM-selected estimator to be more robust to hidden confounding, as it allows for a broader range of possible counterfactual distributions in the ambiguity set. However, setting $\\epsilon$ too large can result in overly conservative estimator selection (similar to the well-known accuracy-robustness tradeoff). Therefore, as lines 216-220 explain, determining $\\epsilon$ involves a careful balance between ensuring the counterfactual distribution is contained in the ambiguity set (i.e., robustness) and maintaining a tight upper bound (i.e., accuracy). \n\n**2.** In Proposition 3.6, unconfoundedness is assumed. Why should the metric still be robust against unobserved confounding? Was the ambiguity radius in setting C selected differently?\n\n**R2.** As discussed above, the DRM-selected estimators should be robust against unobserved confounders provided that $\\epsilon$ is set appropriately. However, determining the proper value of $\\epsilon$ remains an open challenge in the distributionally robust optimization literature [29, 46, 39, 41, 63]. In our work, Proposition 3.6 can guide us to set $\\epsilon$ when unconfounded. When unobserved confounders are present, we can set $\\epsilon$ to a larger value than the one guided by Proposition 3.6. The reason is:\n\n$D_{KL}(P_C||P_T)$\n\n$=\\int_{\\mathcal{X}}\\int_{\\mathcal{Y}^0}\\int_{\\mathcal{Y}^1}p(y^0,y^1|x,T=0)p(x|T=0) \\log \\frac{p(y^0,y^1|x,T=0)p(x|T=0)}{p(y^0,y^1|x,T=1)p(x|T=1)}dy^1dy^0dx$\n\n$=D_{KL}(P^C_X || P^T_X)+\\int_{\\mathcal{X}}\\int_{\\mathcal{Y}^0}\\int_{\\mathcal{Y}^1}p(y^0,y^1|x,T=0)p(x|T=0) \\log \\frac{p(y^0,y^1|x,T=0)}{p(y^0,y^1|x,T=1)}dy^1dy^0dx$\n\n$>D_{KL}(P^C_X || P^T_X)$\n\nIn setting C, yes, the ambiguity radius is set differently. We set $\\epsilon_1=D_{KL}(P^C_X || P^T_X)+5$ and $\\epsilon_0=D_{KL}(P^T_X || P^C_X)+5$. This ensures that the ambiguity set is sufficiently large to contain the possible uncertain counterfactual distributions. Based on your insightful question, we will further emphasize the ambiguity radius for unmeasured confounding in the revised paper.\n\n**Q3.** The high variance for baselines in regret and worse ranking performance of DRM.\n\n**R3.** The high variance phenomenon is discussed in GR2 of the top-page rebuttal and the discussion will be added in the revised paper. Regarding the ranking performance of DRM, we have provided some explanation in lines 310-317 of the original paper. Now we provide some additional discussion.\n\n* First, high rank correlation does not necessarily imply low regret. E.g., in Table 3 of the rebuttal pdf, we have 24 CATE estimators with true performance rank in ascending order: [1, 2, 3, ..., 24]. A plug-X selector may give a surrogate PEHE value, 11.59, then it selects the 6th estimator and gives a rank order [6, 7, 8, 5, 4, 3, 9, 10, 11, 2, 1, 12, 13, 14, ..., 24]. The Spearman correlation between the true rank and Plug-X rank is 0.89. However, the regret of the selected 6th-ranked estimator is 11.63, much larger than 2.00 produced by the top-ranked estimator.\n\n* Second, the DRM approach is designed to select estimators based on their distributionally robust (worst-case) performance. Consider that a coach selects athletes to attend the Olympics based on their scores over 100 games. The first athlete consistently scores between 90-95, with an average of 93, while the second athlete's scores fluctuate between 85-96, with an average of 94. The coach may prefer to select the first athlete, as the worst-case performance is much higher, even though the second athlete has a higher average score (94) and best score (96).\n\n**Q4.** Should discuss baseline setups: whether models for baseline metrics were properly tuned.\n\n**R4.** The baseline setups and hyperparameter tuning method in our paper are in line with [16]. Details are provided in GR3 of the top-page rebuttal. We will add these in the revised paper.\n\n**Q5.** What does “more diverse mean” and what are possible shortcomings of KL divergence?\n\n**R5.** By \"more diverse\", we mean the ability to consider a broader range of distribution types and supports within the ambiguity set. The KL-divergence defined in Eqn (6) is a non-symmetric measure and requires P and Q to have the same distribution type and support. \n\nThe limitations of the KL divergence: \n* Non-symmetry: The lack of symmetry means the KL divergence does not satisfy the triangle inequality, which can limit its applicability in certain areas, such as domain adaptation and transfer learning. Additionally, it makes the distributional distance hard to interpret.\n* Requirement for the same distribution type and support: This restriction can reduce the flexibility of the KL divergence-based ambiguity set. E.g., it prevents comparisons between continuous and discrete distributions.\n\nTherefore, We recognize that using a Wasserstein-based ambiguity set would be a promising research direction. It can be used to compare distributions of different types and supports, capturing a wider range of possible counterfactual distributions. However, as mentioned in the paper, the dual problem of the Wasserstein version involves an intractable \"sup\" problem, which presents additional challenges that require further investigation.\n\nThank you again for your time and effort in providing these constructive insights that help improve the clarity and rigor of our work! We are very welcome to any additional feedback you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "suvRuF9Bha", "reviewer": "Reviewer_9qbS", "summary": "The paper addresses the challenging problem of Counterfactual Average Treatment Effect (CATE) model selection, where the goal is to develop a model selection metric using only observed data, as counterfactual labels are not available. Previous work has focused on model selection metrics that involve learning nuisance parameters on the validation set, making the task of choosing the appropriate CATE model selection criteria complex. The authors propose a solution with a model selection criterion, termed DRM, that avoids the need for training additional nuisance models and is designed to be robust to distribution shifts between treatment and control groups. They first derive the optimization objective for DRM and then propose a relaxation that is tractable with observed data samples. The paper includes a finite sample analysis of the proposed DRM estimator and benchmarks it against existing model selection criteria using synthetic datasets.", "questions": "* In Table 1, I don't understand why DRM is more robust to selection bias as stated by the authors. Are authors comparing the performance gains with DRM over the best baselines across scenarios ($\\mathcal{E}= \\{0, 1, 2\\}$)? It doesn't seem like the performance gap becomes better with more selection bias. Also, the performance of Plug-R or other baselines across different scenarios ($\\mathcal{E}= \\{0, 1, 2\\}$) is statistically indistinguishable given the difference in means lies within the confidence interval. So I am not sure whether any of the prior metrics are susceptible to selection bias in this empirical study. \n\n* Why is the Plug-T metric much worse than the Plug-S metric? T-Learner should provide more flexibility in choosing the regression models than S-Learner, so I don't understand why would the estimates of the ground truth be worse with T-Learner (hence worse model selection).", "limitations": "Yes, the authors have properly addressed the limitations of their work.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper makes a significant contribution to CATE model selection by introducing the DRM criterion, which is notable for not requiring the training of additional nuisance models. Prior work often involves specific inductive biases (e.g., S-Learner, T-Learner, DRLearner) and necessitates training extra models, which can lead to sub-optimal choices and affect model selection performance. The DRM criterion addresses these limitations by not being tied to any particular CATE estimation design. Instead, it derives an upper bound on the ideal PEHE metric that is tractable with observed data alone, avoiding the need for additional nuisance models.\n\n*  The theoretical formulation of DRM is compelling and enhances interpretability by linking the bound to the magnitude of the shift between control and treatment distributions. This directly addresses the challenge of distribution shift in effect estimation. Additionally, the authors derive an equivalent formulation in terms of the KL divergence between the control covariates ($P(X|T=0$) and treatment covariates ($P(X|T=1$), which can be computed using observed data. This approach provides a practical way to assess the impact of distribution shift on CATE model selection.\n\n* The derivation of the DRM criteria from the PEHE metric, along with the proposed estimation strategy, appears to be novel to the best of my knowledge.\n\n* The paper is well-written, with the methodology behind DRM presented in an accessible manner. The background section effectively motivates their approach, and the experimental results are clearly presented, accompanied by a thorough discussion of the performance compared to baselines.", "weaknesses": "While the authors have done a good job in the experiments by considering diverse synthetic datasets and interesting ablation studies, I think the empirical analysis can still be improved. I have summarized my concerns with the experiments below, which would help to provide a clearer understanding of the performance differences and strengthen the comparison of DRM with other model selection criteria.\n\n* The authors use only three base ML models to train the CATE estimators, which limits the diversity of models considered for model selection. To make the model selection problem more challenging and provide a more thorough evaluation, it would be beneficial to train a larger number of CATE estimators. This would offer a more comprehensive assessment of the DRM criterion's performance and robustness across a broader range of models.\n\n* Estimating the nuisance parameters for model selection metrics is crucial for their performance, as highlighted by Mahajan et al. [1], who showed that plug-in metrics based on T-Learner/X-Learner are effective for model selection. To ensure a fair analysis, the authors should consider using state-of-the-art techniques, such as AutoML, for estimating the nuisance parameters associated with the model selection metrics. Currently, the performance of many model selection metrics might be suboptimal due to potentially poor choices of nuisance parameters\n\n* The comparison of different model selection criteria in the results raises some concerns, particularly due to the high variance observed in the performance of several baselines. This variance might stem from poor results on a few datasets, while these baselines could be comparable to DRM on a majority of datasets. The authors should clarify this in the paper and offer more detailed insights, such as results broken down by dataset or other methods to highlight trends and performance variations.\n\nReferences:\n\n[1] Mahajan, Divyat, Ioannis Mitliagkas, Brady Neal, and Vasilis Syrgkanis. \"Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation.\" arXiv preprint arXiv:2211.01939 (2022)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 9qbs,\n\nThank you for your thorough review of our paper. We are delighted that you recognize the novelty and significance of our DRM method in advancing CATE model selection, particularly its nuisance-free and robustness to distribution shift. Below we will address your comments.\n\n* Suggestions:\n\n**S1.** Considering a larger set of base models for CATE estimators.\n\n**R1.** As you may note, prior research has conducted comprehensive empirical investigations conducted a thorough and comprehensive empirical investigation, providing valuable insights for baseline CATE selectors. In our work, our primary focus has been on proposing a new metric for CATE estimator selection, and we believe the 24 CATE estimators with 8 widely-used meta-learners and 3 base ML models were adequate for the current examination. However, we do agree that considering a larger set of base models could offer a more comprehensive analysis. As such, we have expanded our experiments to include an additional Neural Net model, and the results are reported in Table 4 of the rebuttal pdf, and the updated results will be added into the revised paper. Specifically, the Net has 5 hidden layers [200, 200, 200, 100, 100], each with the ReLU activation function. The model is trained using the Adam optimizer with a learning rate of 0.001, a batch size of 64, and 300 epochs. \n\n**S2.** Using AutoML for estimating nuisance parameters.\n\n**R2.** In the original paper, we have tuned hyperparameters for each model in line with [16]. The details are discussed in GR3 of the top-page rebuttal. Thank you for recommending AutoML. A recent ICLR paper [45] has shown the benefits of using AutoML to search a broader grid of hyperparameters, ensuring well-trained nuisance functions. We will use AutoML to update our results and also in future research for hyperparameter tuning.\n\n**S3.** The high variance phenomenon should be further discussed.\n\n**R3.** Thank you for this insightful suggestion. The high variance phenomenon is discussed in GR2 of the top-page rebuttal and the discussion will be added into the revised paper.\n\n* Questions\n\n**Q1.** Not sure whether any of the prior metrics are susceptible to selection bias. Why DRM is robust to selection bias compared to baselines?\n\n**R1.** It is widely acknowledged in the causal inference literature that increasing selection bias can lead to larger biases in estimators and selectors (e.g., [8, 30, 67, 68]). Accordingly, for most selectors, the regret is expected to increase with higher selection bias - a trend clearly evident in the results presented in Table 1. Therefore, we would not expect a regret-decreasing trend with higher selection bias for most selectors, including our DRM method.\n\nDRM metric is robust to selection bias when comparing its performance to other baseline selectors, such as Plug-R. Specifically, as the level of selection bias increases from $\\xi=1$ to $\\xi=2$, the average regret of Plug-R increases significantly, from 1.27 to 4.38 - a jump of 3.11. In contrast, the average regret of DRM only increases from 0.11 to 1.21, a much smaller change of 1.10. However, it is important to note that in the case of no selection bias ($\\xi=0$), DRM does not show a clear advantage over some baselines. This aligns with our expectations because DRM is designed to select estimators that are robust to distribution shifts.\n\n**Q2.** Why is Plug-T worse than Plug-S?\n\n**R2.** The key factor affecting the performance of Plug-T relative to Plug-S appears to be the level of heterogeneity. Heterogeneity refers to the complexity of the CATE function relative to the potential outcome functions. In our paper, the parameter $\\rho$ controls the degree of heterogeneity. According to Table 1, the average regret gap between Plug-T and Plug-S decreases as $\\rho$ increases from 0 to 0.3. Specifically, the regret gap decreases from 38.12 - 3.40 = 34.72 to 34.38 - 2.34 = 32.04 as heterogeneity increases. This suggests a trend that Plug-T would perform on par with (or better than) Plug-S as the level of heterogeneity increases.\n\nThis observation aligns with insights from prior literature. For example, the authors in [16] found that the T-learner outperforms the S-learner in terms of PEHE when the heterogeneity is much larger. Additionally, [40] noted that the S-learner performs better when the CATE complexity is lower than the potential outcome functions, while the T-learner is preferred when the CATE is more complex. Furthermore, the authors in [45] used the original ACIC data but discarded instances with low-variance CATE to ensure sufficient heterogeneity, where they found that T-learner outperforms S-learner.\n\nTherefore, based on our results and the insights from previous studies, we believe the level of heterogeneity is a key factor that affects the relative performance of Plug-T and Plug-S. It would be interesting to further investigate other possible factors that may influence their comparative performance, as they are widely used meta-learners in real-world applications. \n\nFinally, we would like to express our sincere gratitude for your time and effort in reviewing our paper and providing valuable suggestions and feedback! We are very open to addressing and discussing any further comments or questions you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xk9U66ceAo", "reviewer": "Reviewer_1tzr", "summary": "This paper presents a method for selecting a conditional average treatment effect estimator using the distributional robust optimization (DRO) technique. The proposed method does not require specifying models for nuisance functions.", "questions": "1. What is the relation with the paper [[1911.02029] Selective machine learning of doubly robust functionals (arxiv.org)](https://arxiv.org/abs/1911.02029)? \n2. Is the empirical result matched with previous literature [58, 16, 45]?", "limitations": "The paper assumes the ignorability, which is pretty strong assumption in general.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "1. The result is strong, since this paper provides a method of evaluating multiple CATE estimators without specifying other baseline CATE estimator. As a result, more reliable assessment of CATE estimators is possible.  \n2. The empirical comparison is extensive. This paper evaluates many existing CATE estimator and transparently exhibits the performance of the proposed method.   \n\nGiven that the paper contains several weaknesses discussed below, I think the paper’s result is interesting and powerful in practice.", "weaknesses": "**Misleading statements** \n\nThe following statement is generally not true. \n\n> According to Rubin Causal Model [56], the CATE is determined by comparing potential outcomes under different treatment assignments (i.e., treat and control) for a specific individual.\n> \n\nI think the authors are confusing the “unit-level treatment effect” with the “conditional treatment effect”. The unit-level treatment effect $Y_i(T=1) - Y_i(T=0)$ “*for a specific individual*” $i$ cannot be estimated. The CATE is a group-level treatment effect, where the group is specified by the conditioned covariates. Please explicitly distinguish between the term “CATE” and an individual treatment effect.\n\n**Noninformative Introduction**\n\nThe key research question in line 38, “Given multifarious options for CATE estimators, which should be chosen” are not well-defined in the Introduction section. As a reader, the minimum expectation for Introduction section is to see what were the limitations of previous methods with a simple example illustrating the limitations. However, by the design of the paper, until reaching to Section 3, Equation (2), it’s impossible to know (1) what are the plug-in and pseudo-outcome metrics and (2) what were their problems. The unclarity of the problem setting in Section 1 and 2, makes the first two sections non-informative. \n\n**Redundancy**\n\nI don’t see the point of categorizing previous methods as plug-in and pseudo-outcome. All previous methods rely on specifying nuisances, while the proposed method does not. I think such categorization is redundant and unnecessary for the literature review and explanation of the proposed method.\n\n**Subjective statements**\n\nI think this paper contains subjective evaluations of previous methods, which, as a reader, feels somewhat embarrassing. For example, in line 125, \"previous high-quality paper\" seems awkward and redundant — \"previous paper\" is enough. Similarly, \"standing on the shoulders of giants\" is also embarrassing. These kinds of subjective assessments of previous papers dilute the attractiveness of this paper and distract the authors from focusing on their own work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 1tzr,\n\nWe sincerely appreciate your positive recognition of the soundness of our theoretical results and the extensiveness and transparency of our empirical experiments. We are also grateful for your valuable suggestions in improving our presentation and bringing new insights to future work. Below we will respond to your comments one by one.\n\n**W1. Distinguishing Conditional Average Treatment Effect (CATE) and Individualized Treatment Effect (ITE)**\n\n**R1:** We agree with your argument about the difference between CATE and ITE. In the revised version, we will clarify this in the Introduction, explicitly defining ITE as $Y^1-Y^0$ (unit-level) and CATE as $\\mathbb{E}[Y^1-Y^0|X]$ (subgroup-level).\n\n**W2. The presentation of Introduction**\n\n**R2:** We will restructure it as follows:\n* Background on causal inference and CATE.\n* Motivation for CATE estimator selection and existing methods for CATE estimator selection.\n* Challenges faced by current methods (moving content from Section 3.1 here)\n* Introducing our DRM method.\n\nThis restructuring will allow readers to quickly grasp existing CATE model selection metrics and their limitations before introducing our method, thus providing clearer motivation for our work. Thank you for your suggestion!\n\n**W3. Redundancy of categorizing previous methods as plug-in and pseudo-outcome**\n\n**R3:** Thank you for this comment, while we still think it is proper to keep such categorization for the following reasons.\n\n* This maintains consistency with [16, 45]. \n\n* The plug-in and pseudo-outcome approaches differ fundamentally in their construction form, as detailed in Section A.2:\n\n1. Plug-in metrics approximate the ground-truth CATE function using $\\tilde{\\tau}(X)$, which relies on off-the-shelf nuisance functions and only covariates variable $X$. \n\n2. Pseudo-outcome metrics approximate the ground-truth CATE function using $\\tilde{Y}(X,T,Y)$, incorporating off-the-shelf nuisance functions and variables $(X,T,Y)$.\n\n* Both DR-based and R-based objectives can be used to construct plug-in and pseudo-outcome metrics (see Section A.2). This categorization helps prevent potential confusion between Plug-R/Plug-DR and Pseudo-R/Pseudo-DR.\n\nWe will emphasize their distinction in lines 115-123 of the revised paper.\n\n**W4. Subjective statements**\n\n**R4:** We appreciate this helpful suggestion for our presentation. We have removed the subjective statements and will ensure our tone remains objective and rigorous in future paper writing. \n\n**Q1. What is the relation with the paper [arxiv 1911.02029] Selective machine learning of doubly robust functionals?**\n\n**R1:** \n\n**(a) Difference**: While both papers address model selection in causal inference, they differ significantly in the interested quantity and selection objects:\n\nInterested quantity:\n\n* Our paper: Conditional Average Treatment Effect (CATE), i.e., $E[Y^1 - Y^0|X]$.\n\n* Their paper: Average Treatment Effect (ATE), i.e., $E[Y^1 - Y^0]$.\n\nSelection objects:\n\n* Our paper: Selects candidate CATE functions. The specific details are stated in Section 3, Background of CATE Estimator Selection.\n\n* Their paper: Selects candidate nuisance functions in the doubly robust (DR) estimator of ATE. The specific details are as follows:\n\nThe DR estimator of ATE is\n$\\theta_{DR}=\\frac{1}{n}\\sum_{i=1}^{n} [\\mu(X_i,1)-\\mu(X_i,0)+\\frac{T_i}{\\pi(X_i)}(Y-\\mu(X_i,1))-\\frac{1-T_i}{1-\\pi(X_i)}(Y-\\mu(X_i,0))]$.\nIn $\\theta_{DR}$, the nuisance function $\\mu(X,T)$ can be fitted with different machine learners, e.g., Ridge regression, Support Vector Machine, Random Forest, etc, and the nuisance function $\\pi(X)$ can be fitted with different machine learners, e.g., Logistic Regression, Support Vector Machine, Random Forest, etc. This learner-fitting necessity is the same as plug-in and pseudo-outcome methods in CATE estimator selection. But differently, the main goal of their paper is to select nuisance parameters $\\mu$ and $\\pi$ for the DR estimator of ATE, from $J_{\\mu}$ candidate outcome nuisance functions $\\mu_1(X,T), ..., \\mu_{J_{\\mu}}(X,T)$ and $J_{\\pi}$ candidate propensity score nuisance functions $\\pi_1(X), ..., \\pi_{J_{\\pi}}(X)$.\n\n**(b) Connection**:\nInterestingly, the nuisance function selection method in their paper offers insights for improving the baseline CATE selection methods Plug-DR and Pseudo-DR. Both Plug-DR and Pseudo-DR metrics involve the process of constructing $\\tilde{Y}_i=\\mu(X_i,1)-\\mu(X_i,0)+\\frac{T_i}{\\pi(X_i)}(Y-\\mu(X_i,1))-\\frac{1-T_i}{1-\\pi(X_i)}(Y-\\mu(X_i,0))$ (see Sections A.1 and A.2). Note that similar to the mentioned paper, one also needs to select a machine learner to fit the nuisance functions $\\mu(X,T)$ and $\\pi(X)$ for constructing $\\tilde{Y}$. Their approach could be helpful in choosing appropriate machine learning learners for fitting nuisance functions $\\mu(X,T)$ and $\\pi(X)$ in constructing $\\tilde{Y}$. This opens an interesting avenue for future research, i.e., exploring nuisance function selection methods for Plug-DR and Pseudo-DR metrics in CATE model selection. We will discuss this interesting research direction in Related Work.\n\n**Q2.** Is the empirical result matched with [58, 16, 45]?\n\n**R2:** Our results align with some key findings from previous literature. For example, in lines 288-289, we observe the excellence of the R-objective in many scenarios, corroborating the findings in [58]. In lines 290-292, the results confirm the outperformance of R-based selectors as CATE complexity decreases, consistent with the conclusions in [16]. Notably, our analysis extends beyond previous work by examining CATE estimator performance against the level of unobserved confounders and selection bias, providing new insights into CATE estimator selection. \n\nFinally, we would like to thank you again for your time and effort in reviewing our paper. Your feedback is helpful and valuable in improving our work! We remain very open to discussing any additional comments you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We are grateful for your comments and suggestions, which are very helpful in improving our paper. Here are some general responses (GR) that might be useful for individual questions. In the separate rebuttal, we may remind you to refer to this general response. Thank you for your time and effort during the review process!

**GR1. DRM is able to select CATE estimators that are robust to the uncertainty in PEHE caused by selection bias and unobserved confounders.**

In Sections 3.1 and 4.1, we have presented theoretical explanations for why the DRM metric can measure a CATE estimator's robustness against selection bias and unobserved confounding. To address any potential questions from reviewers on this topic, we will provide a more specific explanation below, and these details will be further added to the Appendix.

In causal inference, all the CATE estimators are constructed using observational factual data. However, we can never know how reliable the CATE estimator is due to the unavailable Oracle PEHE in Eqn. (1). As shown in Eqn. (5),  the PEHE is equal to two $\hat{\tau}$-dependent terms, $\mathbb{E}[\hat{\tau}(X)Y^t|T=t]$ and $\mathbb{E}[\hat{\tau}(X)Y^t|T=1-t]$. Unfortunately, $\mathbb{E}[\hat{\tau}(X)Y^t|T=1-t]$ is  in practice. This is because we can only observe the factual distribution $P^F = P(X, Y^t | T=t)$, but not the counterfactual distribution $P^{CF} = P(X, Y^t | T=1-t)$. The unobserved counterfactual distribution can be regarded as an uncertain distribution varying around the observed and certain factual distribution $P^{F}$. If we could assume a "God's perspective" and observe $P^{CF}$ directly, the counterfactual distribution would be certain - like a quantum world! Such an uncertainty in $P^{CF}$ results in the uncertainty in PEHE. In the following, we analyze the source of this uncertainty by examining the relationship between the uncertain distribution $P^{CF}$ and the certain distribution $P^F$:

$P(X,Y^t|T=1-t)=P(X,Y^t|T=t) \frac{P(Y^t|T=1-t, X)}{P(Y^t|T=t, X)} \frac{P(X|T=1-t)}{P(X|T=t)}$.

Therefore, the unobservable distribution $P(X, Y^t | T=1-t)$ can be expressed as the observable distribution $P(X, Y^t | T=t)$ multiplied by the term $\frac{P(Y^t | T=1-t, X)}{P(Y^t | T=t, X)} \frac{P(X | T=1-t)}{P(X | T=t)}$. In other words, the ratio $\frac{p(y^t | T=1-t, x)}{p(y^t | T=t, x)} \frac{p(x | T=1-t)}{p(x | T=t)}$ controls the discrepancy between the factual distribution $P^F$ and the counterfactual distribution $P^{CF}$. Note that if there are no unmeasured confounders, then we have $p(y^t | T=1-t, x) = p(y^t | T=t, x)$; and if there is no selection bias (covariate shift), then we have $p(x | T=1-t) = p(x | T=t)$. Now we understand that the root cause of the discrepancy between $P^F$ and $P^{CF}$ (or between $\mathbb{E}[\tau(X)Y^t | T=1-t]$ and $\mathbb{E}[\tau(X)Y^t | T=t]$) lies in the presence of unobserved confounders and selection bias. In Section 4.1, the uncertainty caused by potential unobserved confounders and selection bias in PEHE can be further measured as the distributionally robust values $\mathcal{V}^t$. Then the PEHE w.r.t. the CATE estimator $\hat{\tau}$ will be at most $\mathcal{V}^t$, which reflects the distributional robustness of $\hat{\tau}$.


**GR2. The high variance regret in some baseline selectors.** 

This phenomenon is primarily due to the wide range of PEHE performances produced by a total of 24 CATE estimators. For example, in setting A ($\rho=0.1$), the average PEHE ranged from 2.00 for the best estimator to 141.01 for the worst estimator (we will present the PEHE performances of all 24 CATE estimators in the Appendix). This significantly large gap between the good and bad estimators leads to high variance of some baseline selectors - if a selector is able to consistently select either good or bad estimators, the variance would not be as pronounced. Therefore, it is important to further analyze the performance of different selectors in each experiment.

To achieve this, we sorted all 24 estimators in ascending order of PEHE and determined the rank of the estimator selected by each selector in each of the 100 experiments. In lines 318-326 and Figure 1 of Appendix C.1, we find that many baseline methods tend to select CATE estimators ranked across different percentile ranges, resulting in a high variance in 100 selections. In contrast, the DRM selector is able to consistently select higher-ranked (i.e., better performing in PEHE) estimators, while mitigating the risk of selecting lower-ranked estimators in most cases. This not only confirms the robustness strength of the DRM but also helps explain its lower variance observed in Table 1.

**GR3. Hyperparameters tuning for nuisance training.**

We have tuned hyperparameters for each model whenever there is a model training process. Specifically, we used RidgeCV and LogisticRegressionCV for the linear regression and logistic regression models, respectively. We also employed GridSearchCV for the SVM and Random Forest models. The number of cross-validation folds was set to 3. The specific hyperparameter ranges we searched are as follows:

Ridge regression: $\alpha \in$ \{ 0.01, 0.1, 1.0, 10.0, 100.0 \}. 

Logistic regression: $C \in$ \{0.01, 0.1, 1, 10\}. 

SVM: Kernel $\in$ \{Sigmoid, RBF\}, $C \in$ \{1, 5\}. 

RF and XGBoost: Max depth $\in$ \{1, 3, 6\}, n\_estimator $\in$ \{20, 100\}.

Meta-review: This paper proposes a distributionally-robust method for selecting CATE estimators. Reviewers were generally very positive about the paper's contributions and emphasized many of its strength. Some limitations (e.g., the non-symmetric nature of D_KL) were discussed in length in a way that seems to have both grounded the author's choices and results, as well as improved the paper. Others should be implemented into the next revision (e.g., details and clarity regarding experimental setup and evaluation, inaccuracies and redundancies of some claims). One point that was brought up multiple times is the extended use of subjective statements - these should be reduced considerably (or removed altogether) in the final version. Nonetheless, overall this is a good paper that is likely to advance the current state of research on CATE model selection and promote the importance of robust models in this literature.
Decision: Accept (poster)