Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation
Authors: Divyat Mahajan
Ioannis Mitliagkas
Brady Neal
Vasilis Syrgkanis
Venue: ICLR 2024 spotlight (2024)
Keywords: Heterogeneous Treatment Effect Estimation
Conditional Average Treatment Effect
Causal Inference
Model Selection
Primary area: causal reasoning

Abstract: We study the problem of model selection in causal inference, specifically for conditional average treatment effect (CATE) estimation. Unlike machine learning, there is no perfect analogue of cross-validation for model selection as we do not observe the counterfactual potential outcomes. Towards this, a variety of surrogate metrics have been proposed for CATE model selection that use only observed data. However, we do not have a good understanding regarding their effectiveness due to limited comparisons in prior studies. We conduct an extensive empirical analysis to benchmark the surrogate model selection metrics introduced in the literature, as well as the novel ones introduced in this work. We ensure a fair comparison by tuning the hyperparameters associated with these metrics via AutoML, and provide more detailed trends by incorporating realistic datasets via generative modeling. Our analysis suggests novel model selection strategies based on careful hyperparameter selection of CATE estimators and causal ensembling.

Published as a conference paper at ICLR 2024
EMPIRICAL ANALYSIS OF MODEL SELECTION FOR HETEROGENEOUS CAUSAL EFFECT ESTIMATION
Divyat Mahajan'
Ioannis Mitliagkas'
Brady Neal
Vasilis Syrgkanis
1 Mila, Université de Montréal
2 Stanford University
ABSTRACT
We study the problem of model selection in causal inference, specifically for con- ditional average treatment effect (CATE) estimation. Unlike machine learning, there is no perfect analogue of cross-validation for model selection we do not observe the counterfactual potential outcomes. Towards this, a variety of surro- gate metrics have been proposed for CATE model selection that use only observed data. However, we do not have a good understanding regarding their effectiveness due to limited comparisons in prior studies. We conduct an extensive empirical analysis to benchmark the surrogate model selection metrics introduced in the literature, well as the novel ones introduced in this work. We ensure fair com- parison by tuning the hyperparameters associated with these metrics via AutoMI, and provide more detailed trends by incorporating realistic datasets via generative modeling. Our analysis suggests novel model selection strategies based on careful hyperparameter selection of CATE estimators and causal ensembling.
INTRODUCTION
Several Docision-making tasks require us to compute the personalized effect of interventions on an individual. If interventions are assigned based on the average effect, then it might result in sub- optimal outcomes (Segal et al.] 2012) as the heterogeneity of the data is not taken into account. Hence, identifying which individuals benefit the most from an intervention would result in better policies. The emphasis on individual treatments effects has been shown in multiple domains, from personalised healthcare (Foster al. 2011) to social sciences (Xie et al. 2012).
This has led to several techniques that estimate flexible and accurate models of heterogeneous treat- ment effects. These approaches range from adapting neural networks (Shi al.) 2019) to random forests [Wager & Athey] 22018) along with frameworks like double machine learning (Chernozhukov al. 2016, Foster & Syrgkanis 2019 Nie & Wager] 2021), instrumental variables (Hartford al.]
2017) meta learners (Künzel et al.] 2019), etc. But how do we select between the different estima- tors? While in some situations we can choose between the estimators based on domain knowledge and application requirements, it is desirable to have model-free approach for model selection. Further, the commonly used practice of cross-validation in supervised learning problems (Bengio et al. 2013) cannot be used for model selection in causal inference, as we never observe both of the potential outcomes for an individual (fundamental problem of causal inference (Holland 19861).
Towards this, surrogate metrics have been proposed that perform model selection using only obser- vational data. Farlier proposals were based on evaluating the nuisance models associated with the estimators, and the utility of decision policy (Zhao et al. 2017) based the heterogeneous treat- ment effects of the estimator. Recently, the focus has shifted towards designing surrogate metrics that approximate the true effect and compute its deviation from the estimator's treatment effect (Nie) & Wager 2021 Saito & Yasui 2020), and they have also been shown to be more effective than other metrics (Schuler et al. 2018 Alaa & Van Der Schaar 2019). However, most of these evalua- tion studies have been performed only on a few synthetic datasets, therefore, the trend in such studies could be questionable. Also, there often lack of fair comparison between the various metrics as
"Equal Advising, Correspondence di vyat. .aatttmaaajaae jangij quebec

---

Published as a conference paper at ICLR 2024
some of them excluded from the baselines when authors evaluate their proposed metrics. Hence, we have a poor understanding of which surrogate criteria should be used for model selection.
Contributions. In this work, we perform a comprehensive empirical study | over 78 datasets to understand the efficacy of 34 surrogate metrics for conditional average treatment effect (CATE) model selection, where the model selection task is made challenging by training a large number of estimators (415 CATE estimators) for each dataset. Our evaluation framework encourages unbiased evaluation of surrogate metrics by proper tuning of their nuisance models using AutoML (Wang ict al. 2021), which were chosen in a limited manner even in recent benchmarking studies (Curth & van der Schaar 2023). We also provide a novel two-level model selection strategy based on careful hyperparameter selection for each class
estimaasti,,oos, and causal ensembling which improves the performance of several surrogate metrics significantly.
To ensure we have reliable conclusions, unlike prior works, we also make use of recent advances in generative modeling for causal inference (Neal et al. (220)) include realistic benchmarks in analysis. Further, we introduce several new surrogate metrics inspired by other related strands of the literature such as TMLE, policy learning, calibration, and uplift modeling.
Our analysis shows that metrics that incorporate doubly robust aspects significantly dominate the rest across all datasets. Interestingly, also find that plug-in metrics based T-Learner are never dominated by other metrics across all datasets, which suggests the impact of tuning the nuisance models properly with AutoML for CATE model selection.
Capital letter denote random variables (X) and small case letters (x) denote their realiza- tions. The nuisance models of the CATE estimators have upward hat v = (át, f), while the nuisance models of surrogate metrics have downward hat ij = (f4, Ā). Potential outcomes are denoted as (Y (0), (1)) while the pseudo-outcomes are represented as (()) = (n) Yo(n).
Notations:
2 CATE MODEL SELECTION: SETUP& & BACKGROUND
We work with the potential outcomes framework (Rubin) 2005) and have samples of random vari- ables (Y, W, X), where X are the pre-treatment covariates, W is the treatment assignment, and Y is the outcome of interest. We consider binary treatments E {0, 1}, and have two potential outcomes (Y (0), (()) corresponding to the interventions (do(W = 0), do W = 1)). The observa- tional data {z, w, y are sampled from an unknown joint distribution Px, W, (0),
Typical causal inference queries require information about the propensity (treatment assignment) distribution (T (x) = P(W = x|X = z)) and the expected potential outcomes (zw ( z)), commonly referred the nuisance parameters 1) (Hoo f1, TT.
Our target of inference is the conditional average treatment effect (CATE), that represents the aver- age effect of intervention (Y (0)) the population with covariates X z.
CATE: (xx) (1) Y() = μi (x) fo(2).
Under the standard assumptions of ignorability Peters et al. tified using observational data [[y("))X x
2017) the expected outcomes are iden- w, X æz). which further implies
CATE is identified as follows (more details in Appendix A.
(x) E EY 1, X 2| E Y W X 21
Meta-Learners for CATE Estimation. We consider the meta-learner framework (Künzel et al. 2019) that relies on estimates of nuisance parameters (f) to predict CATE. E.g.. if we can reliably estimate the potential outcomes YE W = w, = x)) from observational data by learning regres- sion functions A. that predict the outcomes from the covariates æ for treatment groups E {0, 1). then we can estimate the CATE as follows, also known as the T-Learner
fr (x)
ف (z) (A)(2)
(1)
'The code repository he accessed here: wwt.hh/w.wcdddaaaaa/aaaeeettetee
2

---

Published as a conference paper at ICLR 2024
Similarly, could also learn single regression function (Az,w) to estimate the potential outcomes, also known as the S-Learner
*5(x) 1 (xx
-
(2)
Following Curth & Van der Schaar 2021p. such estimating strategies are called indirect meta- learners, their main learning objective to estimate potential outcomes and not CATE directly. In contrast, with direct meta-leamnrr we learn additional regression models (f) to estimate CATE from covariates X, which provides additional regularization. One popular direct meta-learner the Doubly Robust (DR) Learner (Kennedy) 2020), where we first estimate the DR pseudo-outcomes (f) and then learn the CATE predictor by regressing the pseudo-outcomes on the covariates.
DR (f) = fā(a, w) + y - Max,, w) y DR (f)) = 1/ pR (fi yo (r) where y
(3)
tw (x)
.DR SDR arg min
Σ
(u (n) - f(x))
JEF
(2.00.9)
Please refer to Appendix A for a detailed recap on meta-learners used in this study.
CATE Model Selection. Given a set of CATE estimates {fi..., FMr from estimators {Ei..., EM], CATE model selection refers to finding the best estimator, Em- s.t. m arg min, [(f;), where the I(f) denotes the precision of heterogeneous effects (PEHE) (Hill) 2011
L(f) (x((( ((x))²
(5)
If we had access to counterfactual data (observed both Y()), (1)), then we could compute the true effect r(X) and use the ideal metric PEHE for model selection. Hence. the main difficulty stems from not observing both potential outcomes for each sample, and we need to design surrogate metrics (M(+) that use only observational data for model selection.
Surrogate Metries for CATE Model Selection. common approach for designing surrogate metrics (f)) is to learn approximation of the ground truth CATE as F(z) and then compute the PEHE as follows (Schuler et al. 2018).
M(F) Σ (f(x))
Different choices for would give rise to different surrogate metrics. We briefly describe a few tech- niques commonly used for estimating F, with more detailed description of the various surrogate metrics considered in our work can be found in Appendix B
One class of surrogate metrics are the plug-in in surrogate metries which estimate by training another CATE estimator on the validation set, and we could employ similar estimation strategies as meta- learners. E.g., analogous to T-Learner, we can learn f(x) as the difference in the estimated potential outcomes Ú1 (I)) [z), known the T-Score (Alaa & Van Der Schaar 2019).
1 Σ (f(z.) - +x:))))
M (F())= =
(x(x) = (1(x) - Po(x)
(T Score)
N
Another class of surrogate metries are the pseudo-outcome surrogate metrics that estimate as pseudo-outcomes (Y (f)). E.g., we can construct the pscudo-outcome metric using DR pseudo- outcome (F.q. 3 known as the DR Score (Saito Yasui 2020).
M (f)
N Σ (f(x)) fibR(:)))*
PD = (R)
(DR Score)
Note that DR-Leamer would require training the CATE predictor (/DR) as well, however with the pseudo-outcome metries don't train such direct predictors of CATE. Infact, training a direct CATE predictor (fDR) for the metric well would make it plug-in surrogate metric.
3

---

Published as a conference paper at ICLR 2024
Which surrogate criteria to use? While there have been several surrogate metries proposed in the literature that enable CATE model selection using only observed data, we have a poor understanding
of their relative advantages/dissadvaaaeee
Towards this, there have been couple of benchmarking
studies, the first one by Schuler et al. (2018) where they found the R-Score (Nie & Wager] 2021) strategy to be the most effective. However, since their work, there have been new surrogate cri- teria proposed (Alaa & Van Der Schaar] [2019] Saito & Yasui 2020) and also they experimented with only single synthetic data generation process. The latest study by Curth & van der Schaar 2023) considers an exhaustive set of surrogate metrics and analyzes their performance on a care- fully designed synthetic data generation process. Their analysis shows the limitations of the factual prediction criteria that rely on evaluating the generalization of nuisance models for model selection. They also find that pseudo-outcome variants are less susceptible to congeniality bias as compared to their plug-in counterparts. In this work, we build further in the same spirit of conducting a more thorough analysis to obtain insightful trends regarding the performance of different surrogate met- rics. The next section provides details on the proposed evaluation framework and highlights the associated important design choices ignored in the prior works.
3 FRAMEWORK FOR COMPARING MODEL SELECTION STRATEGIES
Consider set of trained CATE estimators (E) and set of surrogate metries ( (f)(,))) our task is to determine the effectiveness of each metric. Let Eúr denote the set of estimators that are optimal w.r.t the metric M, i..., EM arg min.e E A (fe). Then, similar prior works. we judge the performance of any surrogate metric M by computing the ideal PEHE metric (E.q.5) for the best estimators selected by it, i.e., PE [r EeER, I(fe). Since PEHE(E As) determines the quality of the best estimators selected by a metrie M, hence it can be used to compare the different surrogate metries. We now state the novel aspects of our evaluation framework (Figure [
for comparing the surrogate metrics for CATE model selection.
Well-tuned surrogate metries via Auto Since surrogate metrics involve approximating the ground-truth CATE (F) (E.q. 6). need to infer the ussociated nuisance models (r)) on the validation set. The nuisance models (i)) play a critical role in the performance of these metrics as sub-optimal choices for them can lead to a biased approximation (F) of the true CATE. Despite its importance, tuning of metric's nuisance models is done by searching over a small manually specified grid of hyperparameters in prior works. Hence, we use AutoML, specifically FLAML (Wang et al. 2021 to select the best-performing nuisance model class as well as its hyperparameters. Since AutoML can select much better nuisance models than grid search or random search would for the same amount of compute. the surrogate metrics would have less tendency to be biased.
Two-level model selection strategy. The set of trained CATE estimators can be grouped based on the different learning criteria. E.g., consider the population of CATE estimators be comprised of two groups, where the first group E, (fr
fr (f)) contains all the estimators of type T-Learner and the second group EDR = [fRR (7) ).......... ÂDR (fr)) contain all the estimators of type DR-Learner. Given a surrogate metric M (f). prior works select over the entire estimator population, F = arg min M(te), which we term as single-level model selection strategy.
However, another approach would be to first select amongst the estimators within each meta-learner using a criterion better suited for that specific meta-learner, and then select over the remaining population of meta-learner using the surrogate metric. In the example above, we could use [T Score to select amongst the T-Learner group, i.e., E = arg min E, M (f.). Similarly, we could use DR Score to select amongst the DR-Learner group, i.e, F = arg min Eo MDD Then we could select between E' = E using the surrogate metric M, i.e., F arg min,g (fa)- We term this two-level model selection strategy, and since we were more careful in selecting over hyperparameters of each meta-learner, might help the surrogate metric in model selection.
Hence, denoting the CATE estimator population as E = {,} where E, represents all the estimators of type meta-leamer J, the two-level selection strategy can be summarized as follows.
1. Select using meta-learner based metric (M (f)). FG arg min, M YJ
2. Select using the surrogate metrie (7), Fl = arg min M(fa) where F' U, Fj
4

---

Published as a conference paper at ICLR 2024
Casual Ensembling. The prior works typically judge the performance of any metric per its best performing CATE estimator however, this approach is prone to outliers where the top-1 choice selected using the metric is bad but the top-k choices good. Analogous to super-learning (used successfully for predictive model selection (Ju et al. 2018)), instead of returning the best CATE estimator using a metric M, instead return weighter combination of CATE estimators, where the weight of each CATE estimator is proportional to exp (x. (f:) i.c. softmax weight with K as the temperature which can be tuned. This helps to avoid the sharp discontinuities of the best CATE estimator selected using any surrogate metric as we select an ensemble of CATE estimators.
Realistic benchmarks. While surrogate metrics themselves do not require counterfactual data, the same would be needed for the ideal PEHE score(E.q. 5) to judge the quality of the best CATE estimators returned by any metric. Hence, the prior works have experimented only with synthetic datasets where counterfactuals
known. We overcome this issue by using RealCause (Neal) let al., (020), which closely models the distribution of real datasets using state-of-the-arr genera- live modeling techniques such as normalizing flows (Huang et al.) (0118) and verifies its closeness to the original datasets using variety of visualizations and statistical tests. They model the se- lection mechanism (F(WIX)) and the output mechanism (I I,, X)) using generative models (I moded (W (X). P (Y X)), where the covariates X are sampled from the observed real- istic dataset. This gives us access the interventional distributions (IP modet (Y (Yo 0). X), Prodet (Y |d( =)), X)), hence we can sample both potential outcomes in realistic datasets.
4 NOVEL SURROGATE CRITERIA FOR CATE MODEL SELECTION
We also propose a variety of new metrics that are based on blending ideas from other strands of the literature and which have not been examined in prior works. The primary reason for including these new metrics was to have a more comprehensive evaluation, not necessarily to beat the prior metrics.
4.1 ADAPTIVE PROPENSITY CLIPPING METRICS
Consider the DR Score where the pseduo-outcomes depend upon the inverse of the propensity func- tion (=x)... Hence, if some samples have an extremely small propensity for the observed treatment, then their pseudo-outcon
estimates might be biased. Therefore, we introduce propensity clipping techniques from the policy learning and evaluation literature (Wang et al., 2017] [Thomas & Brun- skill 2016 Su al. 2019) for surrogate metrics that depend on the propensity function. We start with clipping the propensity estimate the range [e, 1 - cJ. (xx) = max {e, min {1 e, म (x:))) Then can create variant that uses the adaptive approach of switching approximate 7 fol-
lows:
ADR (/s/T
if S (x) if #e <e
DDR Switch
(DR Switch)
This metric is the same the DR Score for samples that do not have an extremely small propensity of the observed treatment, otherwise. it uses another surrogate metric like S/T Score that doesn't t depend on propensity function for reliable estimates of F(x).
Another idea in policy learning is blending (Thomas & Brunskill 2016), where we consider a convex combination of the DR pseudo-outcome and the potential-outconee based estimate, i.e. #Blend
a-!PW (1 a)7 where is some constant. A successor to blending is Continuous Adaptive Blending (CAB) (Su et al. (209)), which makes a adaptive to the propensity of the sample and combines it with switching ideas. We present here adaptation of CAB for CATE estimation:
*DR Ru (a)
if M (m) (x)
7DR-CAB
FDR +
(x)
1s/T
if # (x) <C
(DR CAB)
Recent prior work of Han & Wu loss and based regression.
2022) also considers variant of causal ensembling using particular
5

---

Published as a conference paper at ICLR 2024
4.2 TARGETED LEARNING
An alternative to alleviate extreme propensities is instead to learn how much inverse propensity correction we need to add. This roughly the idea in targeted learning, which has been explored for average treatment effect (ATE) estimation, but estimation. We describe the S-Learner variant of TMLE, but the same can be done with T-Learner. We learn conditional linear predictor of the residual outcome (Y (x((X, W)) in boosting manner W W
not aware of any prior application for CATE
from the inverse propensity a(X, W) = (1-)(1-=(X))
? arg argmin min fEF
(y: Mxx wi) c(x.)) a w;))
N
weighted regression problem with weights ä(z;, w,)² and labels (y: - çaa,, w;) w;))/ā(z; w;). In our implementation, we used a causal forest approach to solve this regression problem, viewing Y f(X, W) the outcome, a(X, W) as the treatment, and c(X) as the heterogeneous effect. Then we add the correction term to obtain the updated regression model (XMLL(X W) ixX, W) + E(X) ä(X, W) and define the corresponding metric:
The above corresponds
Σ (f(x.) - (x:)) 1 N M (F)
FIMLE -/IMLE(1, 1) WIMLLE 0) (TMLE Score)
4.3 CALIBRATION SCORES
Calibration scores do not plug-in a proxy for the true T(x), rather they check for consistency of the CATE predictions (f(z)) within quantiles on the validation set. We split the CATE predictions (f(r)) into K percentiles (bottom 25%, next 25% etc.), and within each group G&(f) calculate the out-of-sample group ATE using DR pseudo-outcomes (E.q.) 3 and also using the CATE predictions.
GATEE (f)
Σ
(R,))))
GATE& = [G)(F) (EGu(A)
Σ
f(x,).
(+)
(EGu(F)
Viewing G (+) as the unbiased estimate of group ATE, we measure its weighted absolute discrepancy from the estimate of group ATE computed via input CATE predictions (GATE&(F),
K
MCIDR
GR(()]
GATE GATE (+)
(Cal DR Score)
The calibration has been studied RCT' in Dwivedi et al. (0020) and its variants in (Cher- nozhukov et al. 2018 Athey & Wager 2019); we adapted it to be used for CATE model selection.
4.4
QINI SCORES
The Qini score is based on the uplift modeling literature (Surry & Radcliffe 2011 and measures the benefit with the policy of assigning treatment based on the top-k percentile of input CATE estimates as opposed to the policy of assigning treatments uniformly at random. Let G (f) denote the group with treatment effects in the top k-th percentile of the input CATE estimates. We measure the group ATE for it using DR pseudo-outcomes (E.q. 3). GATTERR()
1 [G2x(F) Eieos, (x:():)
The cumulative effect from this group should be much better than treating the same population uniformly random, which can be approximated ATEDR = Σ (x)();). This yields the following score (higher is better):
100
Qini-DR = Σ G2x(7) (GATEDR( - A
(Qini DR Score)

---

Published as a conference paper at ICLR 2024
5 EMPIRICAL ANALYSIS
We now present our findings from the extensive benchmarking study of 34 metrics for selecting amongst total of 415 CATE estimators across 78 datasets over 20 random seeds for each dataset.
5.1 EXPERIMENT SETUP
We work with the ACIC 2016 DDoiie et al. 2019) benchmark, where we discard datasets that have variance in true CATE lower than 0.01 to ensure heterogeneity; which leaves with 75 datasets from the ACIC 2016 competition. Further, we incorporate three realistic datasets, LaLonde PSID, LaLonde CPS (LaLonde 1986). and Twins (Louizos et al. 2017), using RealCause. For each dataset, the CATE estimator population comprises 7 different types of meta-learners where the nuisance models (i)) are learned using AutoMI. (Wang et al. 2021). For the CATE predictor (f) in direct meta-learners, we allow for multiple choices with variation across the regression model class and hyperparameters, resulting in diverse collection of estimators for each direct meta-learner. Even the most recent benchmarking study by Curth & van der Schaar (2023) did not consider a large range of hyperparameters for direct meta-learners, while make the task of model selection more challenging with a larger grid of hyperparameters. For the set of surrogate metrics, we in- corporate all the metrics used in the prior works and go beyond consider various modifications of them, along with the novel metrics described in Section ! As stated before in Section 3 we AutoML for selecting the nuisance models (ij) of surrogate metrics on the validation set. More details regarding the experiment setup can be found in Appendix C
5.2 RESULTS
Following the discussion in Section 3 we compute PEHE of the best estimators selected by a sur- rogate metric to judge its performance, PEHE(E Ežg * Eee L(fx). Since the scale of the true CATE can vary lot across datasets, compute a normalized version where we take the % difference of the PEHE of the best estimators chosen by each metric (E ) from the PEHE of the PEHE(E?;) PEHE(E*) ] PEHE(E*),
overall best estimator (E*), Hermalized-PHHE(M// For each dataset, we report the mean (standard error) Normalized-PEHHE over 20 random seeds. Since we have multiple datasets under the ACIC 2016 benchmark, we first compute the mean per- formance across them and then compute the mean and standard error across the random seeds. For each dataset group, we bold the dominating metrics using the following rule; A metric M is said to be a dominating metric if the confidence interval of the performance metric M either overlaps or lies strictly below the confidence interval of the performance of any other metric M o M.
[
5.2. SINGLE-LEVEL MODEL SELECTION STRATEGY
We first provide results with the single-level model selection strategy for selected list of metrics in Results with the complete list of surrogate metrics can be found in Table 7 in Appendix DD
Table T
Doubly Robust and TMLE variants globally dominating metrics. Across all the datasets, DR Score (and its variants) and TMLE T score are optimal as compared to the other metrics. They produce even better results than Calibration and Qini based scores. Further, the improvements due to adaptive propensity clipping techniques (Switch, CAB) over the basic DR score are not significant.
Plug-in surrogate metrics are globally optimal. It is interesting to observe that plug-in metrics like T/X Score are rarely dominated by other metrics! This highlights the importance of learning nuisance models with AutoMI, as enhances the model selection ability due to lower bias in the estimated nuisance parameters. Since the prior works did not search over a large grid for learning nuisance models, that could explain why the plug-in in metrics were sub-optimal in their results.
Superior performance of T-Learner based metrics. In Table we compare the metrics that have the choice of estimating the potential outcomes (fo. (A1) using either S-Learner or T-Learner. We find that metrics with the T-Learner strategy better than those with S-Learner strategy in all the cases, which further highlights that choice of nuisance models is critical the performance of surrogate metrics.
7

---

Published as a conference paper at ICLR 2024
Metric
ACIC 2016
LaLonde CPS
LaLonde PSID
TWINS
Value Score
1.05c+7 (4.31e+6)
6.63 (5.52)
0.48 (0.06)
0.57 (0.15)
Value DR Score
13.02 (11.73)
2.33 (1.41)
0.46 (0.05)
1.61 (1.02)
Match Score
3.60 (0.16)
0.23 (0.04)
0.50 (0.06)
0.38 (0.08)
S Score
0.95 (0.02)
0.90 (0.04)
0.74 (0.04)
0.29 (0.05)
Score
0.56 (0.02)
0.16 (0.03)
0.42 (0.03)
0.31 (0.05)
X Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.35 (0.06)
R Score
4.0 (0.11)
0.83 (0.04)
0.67 (0.03)
0.60 (0.11)
Influence Score
1455.75 (1439.46)
0.95 (0.04)
0.80 (0.02)
1.08 (0.1)
IPW Score
3.21 (0.12)
0.25 (0.05)
0.32 (0.02)
0.37 (0.06)
DR T Score
0.56 (0.02)
0.16 (0.02)
0.41 (0.03)
0.32 (0.07)
DR Switch T Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.28 (0.05)
DR CAB T Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.33 (0.06)
TMLE T Score
0.64 (0.03)
0.16 (0.03)
0.42 (0.03)
0.31 (0.05)
Cal DR T Score
3.45 (0.11)
0.17 (0.03)
0.42 (0.03)
0.21 (0.03)
Qini DR T Score
1.32 (0.07)
2.87 (1.53)
0.57 (0.05)
2.08e+7 (1.90e+7)
Table 1: Normalized PEHE of the best estimators chosen by each metric with the single-level model selection strategy: results report the mean (standard error) across 20 seeds and also across
datasets for the ACIC 2016 benchmark. Lower value is better.
Metric
ACIC 2016
LaLonde CPS
LaLonde PSID
TWINS
S Score
0.95 (0.02)
0.90 (0.04)
0.74 (0.04)
0.29 (0.05)
T Score
0.56 (0.02)
0.16 (0.03)
0.42 (0.03)
0.31 (0.05)
DR S Score
0.93 (0.02)
0.85 (0.05)
0.73 (0.04)
0.35 (0.06)
DR T Score
0.56 (0.02)
0.16 (0.02)
0.41 (0.03)
0.32 (0.07)
TMLE S Score
1.06 (0.04)
0.91 (0.04)
0.74 (0.04)
0.26 (0.05)
TMLE T Score
0.64 (0.03)
0.16 (0.03)
0.42 (0.03)
0.31 (0.05)
Cal DR Score
5.78 (0.19)
0.87 (0.05)
0.72 (0.04)
0.19 (0.03)
Cal DR T Score
3.45 (0.11)
0.17 (0.03)
0.42 (0.03)
0.21 (0.03)
Table 2: Comparing the S-Learner vs T-Learner based metrics. Each cell represents the Normalized PEHE of the best estimators with the single-level strategy; results report the mean (standard error) across 20 seeds and also across datasets for the ACIC 2016 benchmark. Lower value is better.
5.2.2 TWO-LEVEL MODEL SELECTION STRATEGY
We now provide results with the two-level model selection strategy for a selected list of metries in Results with the complete list of metrics can be found in Table 8 in Appendix D
Table 3
Better performance than single-level strategy. We find that the two-level selection strategy per- forms much better as compared to the single-level selection strategy, and we find better performance in approximately 28.7% cases over all datasets and metrics; with statistically indistinguishable per- formance for the other cases. Since in no scenario it happens that this strategy gets dominated by the single-level selection strategy, recommend this good practice for CATE model selection.
In fact, Qini DR score ended up as a dominating metric for almost all of the datasets with the two- level strategy, while it was among the worst metrics with the lengle-level strategy for the TWINS dataset. Also, the Value DR score ends up as globally dominating metric with this strategy, which is big improvement in contrast to its performance before. Further, the major conclusions from
8

---

Published as a conference paper at ICLR 2024
Metric
ACIC 2016
LaLonde CPS
LaLonde PSID
TWINS
Value Score
3.97 (1.98)
0.34 (0.09)
0.43 (0.03)
0.21 (0.03)
Value DR Score
0.64 (0.03)
0.25 (0.08)
0.47 (0.04)
0.21 (0.03)
Match Score
1.76 (0.09)
0.17 (0.03)
0.45 (0.03)
0.21 (0.03)
S Score
0.93 (0.02)
0.90 (0.04)
0.75 (0.04)
0.21 (0.03)
T Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.21 (0.03)
X Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.21 (0.03)
R Score
3.88 (0.11)
0.86 (0.03)
0.62 (0.03)
0.21 (0.03)
Influence Score
3.26 (0.1)
0.93 (0.04)
0.77 (0.03)
0.16 (0.02)
IPW Score
1.41 (0.06)
0.16 (0.04)
0.38 (0.02)
0.21 (0.03)
DR T Score
0.56 (0.02)
0.16 (0.02)
0.41 (0.03)
0.21 (0.03)
DR Switch Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.21 (0.03)
DR CAB Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.21 (0.03)
TMLE T Score
0.61 (0.03)
0.16 (0.03)
0.42 (0.03)
0.21 (0.03)
Cal DR T Score
0.62 (0.02)
0.19 (0.04)
0.42 (0.03)
0.22 (0.03)
Qini DR T Score
0.58 (0.02)
0.14 (0.03)
0.52 (0.03)
0.24 (0.04)
Table Normalized PEHE of the best estimators chosen by each metric with the two-level model selection strategy; results report the mean (standard error) across 20 seeds and also across datasets for the ACIC 2016 benchmark. Lower value is better.
before regarding the dominance of DR/TMLE and the plug-in T/X metrics are still valid with the two-level strategy, along with the superior performance of T-Learner over S-Learner based metrics.
Hence, a two-level selection strategy can lead to significant benefits, and designing better methods towards the same can be fruitful direction. Note the proposed choice of using meta-learner based metric (M' (f)) to select amongst all meta-estimators of type J is not guaranteed to be optimal, and chose it to mimic the inductive bias of meta-learner J. In Appendix D Tables to 13 provide results for selecting amongst only a particular class of meta-learners using any surrogate metric, and we can see that in some cases the optimal choice is not Af J (f). E.g., in Table 13 the Score not always optimal for selecting amongst estimators of type Projected S S-Learner.
Enhanced performance with causal ensembling. Since we are still selecting the best meta- leamner with the two-level strategy in Table 3 we now consider selecting an ensemble of meta- learners with the two-level strategy and provide its results in Table 9 (Appendix D). We find that ensembling is statistically better than con-ennembling x 5.8% of the experiments (across all datasets and metrics), and otherwise has statistically indistinguishable performance.
6 CONCLUSION
Our work shows the importance of consistent evaluation across a wide range of datasets for surrogate model selection metrics, which leads to more detailed trends opposed to prior works. With well- tuned nuisance models via AutoML., we show that even plug-in surrogate metrics (T Score) can be competitive for model selection. Further, we present novel strategies of two-level model selection and causal ensembling, which can be adopted to enhance the performance of any surrogate metric. Among all the metrics, the DR/TMLE based variants always seem to be among the dominating metrics, hence if one were to use a global rule, such metrics are to be preferred. However, we believe that a more contextual metric is the right avenue and has great potential for future research.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The authors would like to thank the reviewers for their detailed feedback and suggestions! We also thank Amit Sharma for helpful pointers regarding the presentation of the work. The experiments were enabled in part by computational resources provided by Calcul Québec (calcul Iquebe.. ca) and the Digital Research Alliance of Canada (alliancecall ca). Ioannis Mitliagkas ac- knowledges support by NSERC Discovery grant (RGPIN-2019-06512), Microsoft Research collaborative grant and Canada CIFAR Al chair.
REFERENCES
Ahmed Alaa and Mihaela Van Der Schaar. Validating causal inference models via influence func- tions. In International Conference on Machine Learning, pp. 191-201. PMLR, 2019.
Susan Athey and Stefan Wager. Policy learning with observational data. arXiv preprint
arXiv:1702.028%6 2017.
Susan Athey and Stefan Wager. Estimating treatment effects with causal forests: An application. Observational Studies, 5(2):37-51, 2019.
Keith Battocchi, Eleanor Dillon, Maggie Hei, Greg Lewis, Paul Oka, Miruna Oprescu, and Vasilis Syrgkanis. EconML: Python Package for ML-Based Heterogeneous Treatment Effects Estima- 2019. Version 0.x.
tion. https///ghhub.ccmmnnccssssffffft
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 7788)1178--1888, 2013.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and causal parame- ters. arXiv preprint 160i 00060, 2016.
Victor Chernozhukov, Mert Demirer, Esther Duflo, and Ivan Fernandez-Va Generic machine learn- ing inference on heterogeneous treatment effects in randomized experiments, with an application to immunization in india. Technical report, National Bureau of Economic Research, 2018.
Alicia Curth and Mihaela Van der Schaar. Nonparametric estimation of heterogeneous treatment effects: From theory to learning algorithms. In International Conference on Artificial Intelligence and Statistics, pp. 1810-1818 PMLR. 2021.
Alicia Curth and Mihaela van der Schaar. In search of insights, not magic bullets: Towards de- mystification of the model selection dilemma heterogeneous treatment effect estimation. arXiv preprint arXiv:2302.0292: 2023.
Vincent Dorie, Jennifer Hill, Uri Shalit, Mare Scott, and Dan Cervone. Automated versus do-it- yourself methods for causal inference: Lessons learned from data analysis competition. Statis- tical Science, 34(1):43 2019.
Raaz Dwivedì, Yan Shuo Tan, Briton Park, Mian Wei, Kevin Horgan, David Madigan, and Bin Yu. Stable discovery of interpretable subgroups via calibration in causal studies. International Statistical Review, 88:5135-S178. 2020.
Dylan J Foster and Vasilis Syrgkanis. 2019.
Orthogonal statistical learning.
arXiv preprint
Jared C Foster, Jeremy MG Taylor, and Stephen J Ruberg. Subgroup identification from randomized clinical trial data. Statistics in medicine, 30624)28877-2888 2011.
Kevin Wu Han and Han Wu. Ensemble method for estimating individualized treatment effects. arXiv preprint a00i:: /2445, 2022.
10

---

Published as a conference paper at ICLR 2024
Jason Hartford, Greg Lewis, Kevin Leyton-Brown and Matt Taddy. Deep iv: A flexible approach for counterfactual prediction. In International Conference on Machine Learning, pp. 1423- PMLR, 2017.
Jennifer Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217-240, 2011.
Paul Holland. Statistics and causal inference. Journal of the American statistical Association, 81 (396):945-90 1986.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference on Machine Learning, pp. 2078-2087. PMLR, 2018.
Cheng Ju, Aurélien Bibaut, and Mark der Laan. The relative performance of ensemble methods with deep convolutional neural networks for image classification. Journal of Applied Statistics, 45(15))2800-281 2018.
Edward H Kennedy. Optimal doubly robust estimation heterogeneous causal effects. arXiv preprint arXiv:2004.14433 2020.
Sören R Künzel, Jasjeet Sekhon, Peter J Bickel, and Bin Yu. Metalearners for estimating heteroge- neous treatment effects using machine learning. Proceedings of the national academy of sciences, 116(10):455-41655 2019.
Robert LaLonde. Evaluating the econometric evaluations of training programs with experimental data. The American economic review, pp. 604-620, 1986.
Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. Advances in neural information pro- cessing systems, 30, 2017.
Brady Neal, Chin-Wei Huang, and Sunand Raghupathi. Realcause: Realistic causal inference bench- marking. arXiv preprint arXiv:20/ 15007, 2020.
Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299-319, 2021.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Craig Rolling and Yuhong Yang. Model selection for estimating treatment effects. Journal of the Royal Statistical Society: Series B (StatisticaMeehaadlloog))) 76:749-769, 2013.
Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469):322-331, 2005.
Yuta Saito and Shota Yasui. Counterfactual cross-validation: Stable model selection procedure for causal inference models. In International Conference Machine Learning, pp. 8398-8407 PMLR, 2020.
Alejandro Schuler, Michael Baiocchi, Robert Tibshirani, and Nigam Shah. A comparison of methods for model selection when estimating individual treatment effects. arXiv preprint arXiv:1804.05146, 2018.
Jodi B Segal, Carlos Orval Weiss, and Ravi Varadhan. Understanding heterogeneity of treatment effects in pragmatic trials. 2012.
Claudia Shi, David Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. Advances in neural information processing systems, 32, 2019.
Yi Su, Lequn Wang. Michele Santacatterina, and Thorsten Joachims. Cab: Continuous adaptive
blending for policy evaluation and learning. International Conference on Machine Learning, pp. 6005-6014. PMLR, 2019.

---

Published as a conference paper at ICLR 2024
Patrick D Surry and Nicholas J Radeliffe. Quality measures for uplift models. submitted to KDD2011, 2011.
Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference Machine Learning, pp. 2139-2148. PMLR, 2016.
Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 1126623):128--222, 2018
Chi Wang, Qingyun Wu, Markus Weimer, and Eric Zhu. FLAML: A Fast and Lightweight AutoML Library, 2021.
Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudik. Optimal and adaptive off-policy evaluation in contextual bandits. In International Conference Machine Learning, pp. 3589-3597, PMLR, 2017.
Yu Xie, Jennie E Brand, and Ben Jann. Estimating heterogeneous treatment effects with observa- tional data. Sociological methodology, 42(1):314-347, 2012.
Yan Zhao, Xiao Fang, and David Simchi-Levi. Uplift modeling with multiple treatments and general response types. In Proceedings of the 2017 SIAM International Conference Data Mining, pp- 588-596. SIAM, 2017.
12

---

Published as a conference paper at ICLR 2024
Appendix
LIST OF CONTENTS
The content in the Appendix has been organized as follows.
Appendix
provides a review of the meta-learners CATE estimation.
Appendix provides review of the surrogate metrics for CATE models selection.
Appendix
provides the implementation details of our empirical study.
Figure Overview of the proposed framework for comparing the different surrogate
model selection strategies
Figure Illustrating the construction of indirect and direct meta-learners used in our empirical study.
Figure 3
Illustaating the use of AutoML selecting the nuisance parameters of sur- rogate metrics for CATE model selection.
Appendix provides the additional results from our empirical study.
Table 7
Results for single-level model selection strategy.
Table 8
Results for two-level model selection strategy.
Table
Results for ensemble selection with two-level selection strategy
Table 10
Results for model selection amongst only DR-Learners.
Table
Results for model selection amongst only DML-Leamers.
Table 12
Results for model selection amongst only X-Learners
TTbble
13
Results for model selection amongst only Projected S-Learners.
13

---

Published as a conference paper at ICLR 2024
A REVIEW OF CATE ESTIMATION
A.1 IDENTIFICATION OF CATE
The objective of CATE by definition relies on interventional data, which implies we cannot estimate this using only observational data.
CATE:
T(x) = F Y()) = pu (a:) (3))
However, we can make the ignorability assumptions (Peters et al. 2017 which enables us to identify
the expected potential outcomes from observational data (ElY X = E x, x)), hence identify CATE from observational data.
The ignorability assumption consists of the following three assumptions:
Consistency: It implies that the outcome observe reflects the true potential outcome under the observed treatment, Y = W (L) -
W) (0).
Exchangeability: It implies that we do not have any unobserved confounders, (0), (1) IL W | X. Unobserved confounders would lead to open backdoor paths between Y(O), Y(I) and W, hence we would not have conditional independence.
Overlap: It implies that the treatment assignment for each sample is probabilistic, 0 < x(x) < 1 Vz X. Therefore, each sample has non-trivial chance of being assigned to either group (W = or W = 1)).
A.2 META-LEARNERS FOR CATE ESTIMATION
We consider only meta-learners for CATE estimation as they reduce CATE estimation to a series of (weighted) regression and classification problems, which makes easy to apply out-of-the-bon machine learning techniques to solve each sub-problem. Such approaches have been extensively studied in the literature and are heavily used in industry practice. Following (Curth Van der Schaar 2021), we divide the meta-learners into two categories, indirect and direct meta-learners
Indirect Meta-L.crrners The main learning objective of these estimators is to accurately model the potential outcomes (fo, fin) using the observational data and then we can obtain CATE estimate indirectly as the difference in the learned potential outcomes.
The T-Learner approach approximates the potential outcome E regressing Y on X using samples from the un-treated population {z, y v>==o. and E|Y W = 1, as fi by regressing Y X using samples {2, y) w== from the treated population.
X x| as jo by
(z) (z) (o(x))
While the S-Leaner approach learns single regression model ja(r, w), regressing Y jointly on the features X and the treatment assignments W from observational data {x. w,y).
ts(x) = p(x, (xx.0)
(8)
the nuisances models (fo(z), A (x), f(x,, w)) are universal function approximators, then S- Learner and T-Leaner would offer unbiased estimates of CATE. However, there will be statistical errors in practice due to the nuisance model class not capturing the true potential outcome. This will be amplified more compared to standard regression tasks while computing CATE, i.e. use ji (x) to make predictions for data points with the same features observed data points but different treatment assignment (Kennedy 2020
also do model-extrapolation
Direct Meta-Learners. Direct meta-learners also learn the nuisance parameters (β,, f) from ob- servational data but then learn a regression function to directly predict CATE. For example, we can transform the indirect S L-Learner described above into direct meta- leaaner known the Projected S-Learner (Battocchi et al. 2019). which learns a final projection step by regressing the difference in the estimated potential outcomes (i fa(:, 0) the covariates.
14

---

Published as a conference paper at ICLR 2024
irs arg min
Σ (id(r, (x 0) f(x))²
SeF
This introduces extra regularization step which could avoid potential overfitting, especially if we have some prior belief on the functional form or the smoothness of the CATE function.
Another example is the the X-Learner (Künzel et al. 2019) approach, which first trains the nui- sance models (fo(2), Pi (x)) as in the T-Learner approach. But rather than using the difference in estimated potential outcomes to predict CATE, model the CATE directly for the control and the treatment group using regression models (fg, fō) as follows.
70 arg min
Σ (z)( y/ - fx;))
JEF 1,00;
(10)
71 arg min TEF
Σ 1,0,=1
(ye - Mo(a:)) - fxx)))
The first approach imputes treatment counterfactuals for each control sample and views (x) - y) as a proxy for the individual treatment effect. Then learn the CATE predictor by regressing that on covariates z. The second approach takes the analogous approach using the treated population. Finally, we use the learned propensity model (I) to combine the CATE predictions from both groups:
+ (1) (1 (())
(11)
The intuition behind the weighting is that the estimate 20 relies on the function #r, which performs well in regions of X where we have many treated units (i.e. when (() is large). Similarly, the estimate 7 relies on AS which performs well regions where we have many controls units.
We now discuss in detail the widely used direct meta-leamers; doubly robust and double machine learning techniques for CATE estimation.
Doubly Robust Learner (DR-Leerner) To avoid the heavy dependence on the potential out- come regression functions with indirect meta-learners, recent works have proposed generalizations of the doubly robust estimator (traditionally used for ATE inference), for CATE estimation (Foster) & Syrgkanis 2019: Kennedy 2020). The DR learner is mixture of the S-learner with inverse propensity (IPW) based approach. First, we state the vanilla inverse propensity based learner and then the DR leaner The inverse propensity weighted (IPW) estimator, uses the learned propensity model (ft (x)) to compute the IPW pseudo-outcome as follows:
IPW
v/
HPW (1)
y0 (1))
where
y
(12)
can be shown that under the true propensity model, we have the following (Peters et al., 2017):
B|y (x) [[()))
Hence, /IPW (v) gives an unbiased estimate of the potential outcome Y()) and the pseudo-outcome /IPW (f) Y(M Y!PW (f) can be used to approximate the CATE. Therefore, we can learn the CATE predictor (fiww) by regressing the IPW pseudo outcomes the covariates.
arg min FEF (2,2,0)
Σ
(1 (n) f(x))²
(13)
Unlike the S-Learner, the IPW-Learner does not depend any potential outcome regression model but heavily relies on the learned propensity model f... Further, there are numerical stability issues if the true probability of certain treatment is low, then dividing very small numbers. Hence, the IPW-Learner can have very high variance, which can overcome by doubly robust learning (Foster & Syrgkanis, 2019 Kennedy 2020).
be interpreted hybrid approach,
15

---

Published as a conference paper at ICLR 2024
where it first directly approximates the potential outcomes (Ø(zz w)) like S-Learner, and then it debiases the estimate using the IPW based approach on the residual (y p1 w)), as shown below:
DR (f) = fú(z, w) + Mr w)
DR (i) = -1 DR (f)
DR (f) where yo
(14)
Tw (x)
Finally, learn the CATE predictor (SDR) that maps the covariates the DR pseudo-outcomes:
DDR arg min Σ ((rr))
(15)
SEF (z.m,y)
Double Machine Learning (R-Learner) The Dobule ML approach to CATE estimation (Cher-
nozhukov et al. 2016; Nic & Wager 2021). also known in the literature the R-Learner (Nie &
Wager] 2021), solves the CATE estimation task by learning the following nuisance models ( Q) where स is estimate of the true propensity model, and q estimate of [E] i.e, predicting potential outcomes only from the covariates. Then it computes the residuals from both the potential
outcome prediction task :- y ((x)) and the treatment assignment task (w w +x))). In the
final step it learns the CATE predictor (fDMI.) by minimizing the following loss function:
7DM == arg min FEP
Σ - f(x) w)² arg min (1.3)00)
Σ w² ('(0
(16)
JEF (M.N.0)
This can viewed as weighted regression problem with weights पł and labels / / This learning objective can be justified by simply observing that for binary treatment, we have:
Y| X, W| T(X) W = E|Y - B| X X, W. = T(X) (W - E(W X))
Therefore, the R-Learner loss is the square loss that corresponds to the latter regression equation. The R-Learner has been shown to enjoy theoretical robustness properties similar to the DR learner (i.e. that the impact of errors in the nuisance models ît, (i) on the final estimate is alleviated, albeit for the R-Learner the accuracy of the propensity is more important and the method is not consistent the propensity model is not consistent (while the DR-Leamer would be if the regression model was consistent but the propensity model was inconsistent). Moreover, if the model space F does contain the true CATE model then the outcome of the DR Learner can be interpreted asymptoti- cally as projection of on the model space, with respect to the L2 norm, while the outcome of the R-Learner is a weighted projection, weighted by the variance of the treatment; hence approximating the true 7 more accurately in regions where the treatment is more random.
16

---

Published as a conference paper at ICLR 2024
B REVIEW OF CATE MODEL SELECTION SURROGATE METRICS
Surrogate PEHE metrics. Majority of the surrogate metrics for CATE model selection that we consider in our analysis learn an approximation of the ground truth CATE as F(x) and then compute the PEHE (E.q.5) 5. i.e mean squared loss of the input CATE estimator's prediction (f(x)) from F(x) on the validation set.
N N Σ (f(x,) F(x)))
M(F)
(17)
Since the approximate true CATE (f(1)) is, in general, a function nuisance parameters as well, to differentiate the nuisance parameters of the surrogate metries (M) compared those associated with the CATE estimators (E), we represent the nuisance models of the metrics with downward hats (j,, #), while nuisance models of the estimators have upward hats (Ø, R). Note that the nuisance models associated with the surrogate metrics are trained only on the validation set and we do not assume any access to training data in this stage. These metrics are broadly classified into two categories, plug-in in surrogate and psendo-outcome surrogate metrics, as discussed below.
Plug-in surrogate metries. The plug-in surrogate metries learn F(x) in an analogous way to meta- leamnrrs and use it to score the input CATE estimate (f(z).. For example, we can construct the plug-in metrics based on the indirect meta-leamer strategies as follows, which were also used as baselines by Alaa & Van Der Schaar (2019).
1
(+)
(f(x;) - =s(x,))²
is(x) p(x, (0 0)
(S Score)
N
1 MT = N - +x:)))
Σ
ir(x) = (1(1) #o(
Score)
Another plug-in surrogate metric consider is the matching score, which estimates F(x) using the matching method (Rolling & Yang 2013), for each point Ii, find its nearest neighbour from the opposite treatment group, i = arg ming ||ær
x: then define 73 y/ Vi-
N Match (+) = N Σ (f(2,) - (M,ac((2;))
MMach(())) = y y7
(Matching Score)
Note that in general, we could use direct meta-learnen as well for estimating F(x) in plug-in metrics, however, we do not consider them they more susceptible congeniality bias (Curth & van der Schaar 2023). For surrogate metrics based on direct meta-learner strategies, we only consider the pseudo-outcome variant described ahead.
Pseudo-outcome surrogate metrics. These surrogate metrics approximate F(x) using pseduo- outcomes (i)) such that under the assumption of true nuisance parameters, have [| T(r), hence justifying their choice for approximating the ground truth CATE.
For example, construct the useudo-outcome metries using IPW (E.q. 12) and DR pseudo- outcome (E.q. which was first proposed for model selection by Saito & Yasui 2020).
N
M IPW (f) =
(f(x,) (aw();)))
7PW == y (7)
(IPW Score)
N
N (f(z.) - =x)(2
DR (f) =
JDR = DR
(DR Score)
N
Since we use the S-Learner strategy to obtain the nuisance models for computing the DR pseudo outcomes, we will refer to it as DR S Score. If instead we used the T-Leamnr strategy (No. fa) for computing DR pscudo outcomes, DR (î) (x) + == (w) will refer to it as DR T Score.
17

---

Published as a conference paper at ICLR 2024
Further, define propensity clipped versions of metrics that depend on the propensity function. We update the propensity function with the clipped propensity estimates in the range [e, c]:
कt ( max (e, min (1 e, (c:))}
(18)
Whenever such propensity clipping is introduced in the score, we will annotate it with the extra keyword "Clip"" (e.g. IPW Clip Score, DR Clip 7 Score, etc.).
We also consider R-Score (Nie & Wager 2021 that uses Double Machine Learning (DML) to ap- proximate Fxx), which was found to the best performing surrogate metric in the evaluation study of MfSchhler et al. 2018). Using the approximation of F(X) via the DML method, we know that F(X) would be the solution to the regression problem (E.q. Hence, to compute the deviation of the input CATE estimator's prediction fzx) from Fzx), we substitute f(x) in the DML regression prob- lem (equation 16). as shown below, where # (24) is the probability of x; belonging to the treatment class w = 1, i.e., the propensity.
N
MR := N
Σ ((1s (xx)) - (1xa)) (wi - # (xx)))²
(R-Score)
The Influence Score proposed by Alaa & Van Der Schaar (019)) also falls in the category of pseudo-outcome surrogate metrics. To compute the influence score, first compute plug-in sur-
rogate metric using T-Learner (E.q. Score) and then debias it using influence functions. Following Theorem 2 in their work, the correction term is defined follows:
IF- (F-CCorretion( (, 7) =(1 ()((())) (F( - +(X)) (A+ +
where A = W - $ (X), B = 2 AC-1, and C. = equation below, where we add influence correction to the plug-in surrogate metric.
=X)) This is summarized in the
N
M T (f)
1 N
Σ [(f(x)) (x: (=()))))) + IF-CCorection(f.77 (Influence Score)
Finally, we also propose the X-Score that computes pseudo pseudo-outcomes based on the X-Leaner methodology (E.q. [T.. Following the learning objective of CATE predictors per group (E.q. 10). the potential outcomes associated with each group are given as follows:
7 (x) j4 (x)
(z) y ()()))
These combined using the propensity model (+(z)) (z)) to construct the final potential outcome based approximation the true CATE as F(z) = # ( (z) + +(=)) (x)
1 Σ (f(x:) -xx)))))())))) N
M (f)
# #(x)= fx(x)
+ +(x)) 7 (x)
(X Score)
Utility of treatment policy based metries. As opposed to approximating the true CATE, we also consider the surrogate metrics based on the estimates of the value of an optimal policy designed via the input estimator's CATE prediction (f(z)). We derive the optimal treatment policy corresponding to the input CATE estimator as d(x)) := 1(f(x) > 0) (assuming larger potential outcome is better).
Then, one such metrie, value score (Zhao et al. 2017), constructs an IPW (Equation based estimates of the policy value:
M valuc (f)
1 1(w; d(z,)))
(Value Score)
N
#u, (2:)
An issue with the Value Score is that it only utilizes a portion of the data where the decision policy and the observed treatment agree. This is handled in the value DR score (Athey & Wager 2017).
where we use the DR pseudo-outcomes (Equation 14), to estimate the policy value:
M = N Σ d(x,)) DR (n)
(Value DR Score)
18

---

Published as a conference paper at ICLR 2024
Proposed surrogate metries. In Section 4 we introduced novel surrogate metries based ideas from related areas like policy learning, uplift modeling, etc. The primary reason for including these new metrics was have more comprehensive evaluation and include more good candidates for model selection, not necessarily to beat the prior metrics. The ideas of switching, blending, etc. have been well-studied and extensively used in the policy learning literature, but their applicationn in the context of CATE estimation/ model selection has not been shown. Similarly, targeted learning has been used for learning ATE estimators but remains unexplored for CATE model selection.
The motivation for adaptive propensity clipping and targeted learning metrics is the extreme propen- sity regime, where the metrics that depend on inverse propensity scores (IPW. DR) can become bi- ased. Hence, these proposed metrics can handle the extreme propensity region (very small propen- sity for the observed treatment) by using estimates from the regression- baased learners (S/T Learner) as in DR Switch DR CAB or adding the inverse propensity correction as in [MLE Score Since we only described the adaptive propensity clipped variants of DR-Score in the main text, we now provide the same for the IPW-Score for convenience.
if E S tw (z)
Switch
Fs/T
(x) <
(IPW Switch)
fiw (w)
if c S #w(
CAP
TiPw + (1 (1- Xm(())
(IPW CAB)
Fs/T
if x (T) <E
For the Calibration score, the intuition is to match the group ATE across subgroups denoted by the different percentiles of the CATE estimates. We compute the group ATE using DR/TMLE which is viewed as an unbiased sample of the group ATE. Subsequently, we compute the weighted absolute error of the group ATE computed using the input CATE estimates against the unbiased group ATE. For the Qini score, rather than computing the group ATE with the imput CATE estimates as in Calibration, here compare the improvement with unbiased group ATE estimates (computed using DR/TMLE) versus the uniform sampling estimate. Intuitively, if there is heterogeneity in the data, then we should expect the policy of uniformly treating an individual to be worse compared to the policy that assigns treatment to the top-k percentile of the input CATE estimates. Hence, the Qini score is qualitatively different than most of the other metrics as does not directly compare the CATE estimates, rather it evaluates whether the groups most likely to be treated would actually benefit from the treatment. Since we only described the case of unbiased group ATE estimates computed using DR pseudo-outcomes (Cal DR Score
Qini DR Score), we now provide the same
for the case of TMLE for convenience.
K
CCal-TMME(()
Gx((F]
GATE&(F) - GATE TMLE (+)
(Cal TMLE Score)
100
M Qini- SoniTMLE// (A) =
Mini
Σ G2a(f) (GATE (F) TTEE
(Qini TMLE Score)
k=1
where GATE TMLE
(M)E [N(F] Eiec. iTMLE(Z;),
GATER = TO([F] EVEGU F(x:)
GATE TEEETMLF(F (f) := [G>((f] Lieage FIMLE(X,).
ATE N E. AIMLE(D1)
Zk
19

---

Published as a conference paper at ICLR 2024
C
EXPERIMENT SETUP DETAILS
Select using (t)
Meta-Learners E
Meta-Learners A
CATE Estimation
Select using (i)
Meta-Learners Ep
Meta-Learners E
List of CATE Estimators
List of optimal Meta-Learners
Dataset
Select using M(
Real Cause
Potential Outcomes (Y(O), Y(1))
PEHE to judge M(+)
Ensemble E
Counterfactual Data
Ensemble of optimal Meta-Learners
Figure I: The proposed framework for comparing the different surrogate model selection strategies (f). We first perform intra-meta-learner selection using meta-learner based metrics, and then con- struct an ensemble
the optimal meta-learners using the input surrogate metric M()). Further,
RealCause enables us to sample counterfactual data for realistic datases as well and benchmark the performance of each surrogate metric M (f) the PEHE of the ensemble returned by it.
C.I DATASET STATISTICS
Dataset Group
ACIC 2016
LaLonde CPS
LaLonde PSID
TWINS
Training Data Size
3841
8089
1338
5992
Evaluation Data Size
961
6470
1069
4794
Covariate Dimension
82
8
8
75
Table Statistics for the various datasets used in our analysis
Dataset
LaLonde CPS
LaLonde PSID
TWINS
CATE (Mean)
10274.49
57280.61
0.02
CATE (Variance)
3.73e + 07
5.92e + 08
0.01
Treatment Class & (Train)
0.01
0.06
0.74
Treatment Class % (Eval)
0.01
0.06
0.75
Table 5: Extra statistics for the RealCause datasets used in our analysis. Train and Eval correspond the different splits of the dataset used for training and evaluating CATE estimators.
We provide details regarding the training and evaluation sample size, along with the covariate di- mensions Table We also provide more details regarding the mean and variance of CATE as well as the true propensity (fraction of samples in the treatment class) in 55 the realistic datasets, and the same for ACIC 2016 datasets T Note that the ACIC 2016 synthetic benchmark con- tains several datasets, where we had discarded datasets that have variance in true CATE lower than 0.01 to ensure heterogeneity; which leaves us with 75 datasets. Further, LaLonde CPS, LaLonde PSID and TWINS are the realistic benchmarks that do not contain the counterfactual potential out- comes for each individual. Hence, we used RealCause (Neal et al.) (2200) to model the counterfactual potential outcomes for these datasets, essentially making them semi-synthetic datasets.
20

---

Published as a conference paper at ICLR 2024
C.2
CATE ESTIMATORS IMPLEMENTATION DETAILS
Training Dataset
AutoML
Nuisance Parameters (f)
Indirect Meta-Learners
Large Grid of Hyperparameters
Direct Meta-Learners
Predictor (f)
Figure 2: Illustrating the construction of indirect and direct meta-learners used in our empirical study. We use large grid over different regression model classes and hyperparameters for choosing the CATE predictor in direct meta-learners, resulting in 103 different CATE estimators per direct meta-learner. This design choice is an improvement from prior works which consider only a few choices for hyperparameters of direct meta- meta-learners.
For each dataset, we consider 7 different type of meta-leamers for CATE estimation, including both indirect and direct meta-learners. For indirect meta-learners, we consider both the S-Learner and T-Learner; while for direct meta-learners, we consider the Projected S-Learner, X-Learner, DR- Learner, R-Learner, and Causal Forest Learner. Details regarding each of these meta-learners can be found in Appendix[A. A Note that we do not consider deep learning based CATE estimators since the scale of our study is already quite large. Also, the basic ML techniques with appropriate hyper- parameter tuning should have good performance as well since we work with tabular datasets.
The nuisance models (i)) associated with all these various meta-learners are learned using Au- toML (Wang et al., 202T) with a budget of 30 minutes. Specifically, we have five different types of nuisance models; propensity model (f (X)), outcome model used in S-Learner, Projected S- Learner, DR-Learner (í X, W)), outcome models used in T-Learner, X-Learner (Xx(X), fu (X)), and outcome model for R-Learner, Causal Forest Learner (f(X)): each learned using AutoML.
Further, for the CATE predictor (f) direct meta-learners, use total of 103 different regression models with variation across regression model class and associated hyperparameters, specified be- low. We used sklearn for implementing all the regression models and we use the same notation from sklearn for representing the regression model class and the corresponding hyperparameter names. For representing the hyperparameter ranges, we use the numpy logspace function to sample from range (unless specified otherwise), with the syntax (start value, end value, total entries).
Linear Regression; No Hyperparameter
Linear Regression; Degree 2 polynomial features; No interaction terms; No Hypereparam-
Linear Regression; Degree 2 polynomial features; Interaction term; No Hyperparameter
Ridge Regression; Hyperparameters (a): np.logspacee-44,,5 5, 10)
Kernel Ridge Regression; Hyperparameters (a): 1wpplogsspacee-44,,. 10)
Lasso Regression; Hyperparameters (a): np.logspacee-4455.00 10)
Elastic Net Regression; Hyperparameters (a)): mp.logspace 5, 10)
SVR; Sigmoid Kerne:; Hyperparameters (C): mp.logspacce-44.] 10)
SVR; RBF Kernel; Hyperparameters (C): mp..ogspacee 5, 10)
Linear SVR; Hyperparameters (C): np.logspace(-4.5,7
Decision Tree: Hyperparameters (max depth): list(range 11)) + [None]
Random Forest: Hyperparameters (max depth): list(range 11)) + [None]
Gradient Boosting: Hyperparameters (max depth): list(range(2 11)) + [None]
21

---

Published as a conference paper at ICLR 2024
Hence, have 103 different CATE estimators for each type of direct meta-learner, and single CATE estimator for each type of indirect meta-learner, resulting in total of 415 CATE estimators per dataset. This difference between the construction of indirect direct meta-leamnrss is also visualized in Figure 2 Note that Causal Forest DMI--Leamnr technically but the default random forest provided in the EconMI. implementation (Battocehi et al.)
direct meta-learner,
2019) hence effectively we do not consider variations across its CATE predictor (f) function.
C.3
SURROGATE METRICS IMPLEMENTATION DETAILS
Validation Dataset
AutoML
Nuisance
Surrogate Metrics
Parameters 6i)
Figure 3: Illustrating the use of AutoML in selecting the nuisance parameters of surrogate metrics for CATE model selection. This is an important design choice in contrast to prior works which relied on small grid searches to infer the nuisance parameters associated with surrogate metrics, potentially resulting in biased estimates and affecting the model selection ability of surrogate metrics.
Please refer to Appendix for full details regarding the implementation for each surrogate criteria of CATE model selection. An important point to highlight is that for surrogate metrics based on direct meta-learner strategies, we only consider the pseudo-outcome variant. B.g., DR Score only uses gpeudo-outcomer to approximate F(x) and does not train any regression model for direct CATE estimation (like we do in DR-Learner). Further, all nuisance models (7)) are trained on the validation
set using AutoMI., specifically FL.AMI (Wang et al. (022), with budget of 30 minutes (Figure
Regarding the two-level model selection strategy, each metric (²) for selecting amongst meta- estimators of type J is provided below.
DR T Score for selecting amongst DR-Learners
R Score for selecting amongst DML-Learners
X Score for selecting amongst X-Learners
S Score for selecting amongst Projected S-Learners
Note that for the remaining meta-learners (S-Learner, T-Learner, Causal Forest Learner) we did not have any hyperparameters
select over as their nuisance models have been learned via AutoML. Hence, the first step in the two-level model selection strategy is trivial for them and they can be directly for the second step model selection via a general surrogate metric.
22

---

Published as a conference paper at ICLR 2024
Dataset
CATE (Mean)
CATE (Vartance)
'Treatueent Class (Trait)
Treatment Class (Eval)
ACHC,3016_0
144
10.99
0.29
0.29
ACIC.201
0.37
ACIC,2016, 6_4
548
30
0.28
0.30
ACK_2001_5
3.38
4.46
0.29
0.29
ACIC.2016.6
3.79
6.85
0.28
0.30
ACTC_2016_7
4.45
9.40
0.30
ACIC,2016.8
400
9.37
0.29
0.30
ACIC.2016.9
200
8.44
0.32
0.31
ACTC_2016_10
3.71
11:77
0.29
0.31
ACIC_2016.11
5332
0.32
1CIC.2016.12
295
17.74
0.50
0.19
1C
3.27
7.14
0.50
0.54
1CIC.2016.11
2331
45.37
0.47
0.46
ACTC_2016_15
ACTC_2016_16
380
23.87
0.57
0.56
1C16_2016.17
021
18.76
0.37
0.38
10
3.16
16.24
0.52
0.56
1CIC_2016.19
1.85
ACIC.2016.20
4.45
5.94
0.29
0.27
ACTC_2016.21
6.39
$6.62
0.32
0.32
ACIC.2016.22
575
3.96
0.33
0.35
ACTC_2016.23
344
11:47
0.27
ACTC_2016.24
4.73
5.97
0.35
0.35
2016.2016.25
3.92
6.93
0.31
0.32
2016.2
2.88
40.74
0.31
0.29
2C1C.2016.27
3119
AC1C.2016.28
3.25
5.42
0.33
0.36
2916.
3.75
9.41
0.35
0.36
ACIC.2016.80
335
10.82
0.36
ACIC_2016.31
384
20.17
ACTC.2016.32
283
14.85
0.33
0.34
ACIC.2016.3
5.22
19.58
0.63
0.63
ACIC 2016 34
5.10
4.55
0.52
0.53
2C16.2016.35
272
0.36
ACIC.2016.36
3.15
5.98
0.57
0.53
ACIC 2016 37
5.38
11:32
0.60
0.63
ACIC22016.38
3.66
3.565
0.63
ACIC.2016.39
5.12
0.63
ACIC 2016.40
0.95
5,30
0.62
0.57
ACIC.2016.41
337
3.06
0.59
0.57
ACIC 2016 42
303
15.11
0.64
ACIC 43
387
25.95
0.66
AC1C.2016.44
437
28.33
0.67
0.70
ACIC 2016 45
4.9
25.60
0.65
0.89
2016..016.4
2.88
0:65
AC1C.2016.47
535
1.21
0.32
ACIC 2016 48
3.79
4.84
0.27
0.27
2016..
585
1.05
0.25
0.21
2016
3662
0.32
4.15
12.54
ACIC.2016.52
490
79.71
0.31
0.35
ACIC 2016 33
2:58
33.89
0.29
0.29
2016
0.36
2016.2016.55
363
0.36
ACIC 2016 36
2:30
9.71
0.34
0.36
ACIC 2016 57
3.31
13.43
0.36
0.34
20165201,58
4.66
41.14
0.58
434
40.29
0.38
ACIC 2016 60
2.51
34.84
0.35
0.36
ACIC 2016,61
4.3
50.43
0.35
0.34
2016
400
1.29
0.56
ACIC 2016
3331
15.72
0.51
ACIC 64
3:22
9.29
0.54
0.57
ACIC 2016 65
3,00
28.50
0.56
0.55
6016,001666
2.16
ACIC
5.1
117.43
0.61
ACIC 2016 68
304
18:27
0.60
0.58
2016
282
4.75
0.65
0.63
ACTC.2016.15
2.48
0.65
ACIC
2.55
18,66
0.69
ACIC 22
1.58
4.61
0.63
0.62
ACIC 20166.3
4.31
5.44
0.63
0.62
4.49
34.10 16
0.66
2016,
320
0.64
ACIC 2016.76
4.96
15660
0.63
0.61
Table 6: Extra statistics for the various ACIC 2016 datasets used in our analysis. Train and Eval correspond to the different splits of the dataset used for training and evaluating CATE estimators.
23

---

Published as a conference paper at ICLR 2024
D ADDITIONAL RESULTS
Metric
ACIC 2016
LaLonde CPS
LaLonde PSID
TWINS
Value Score
1.05c+7 (1.31e+6)
6.63 (5.52)
0.48 (0.06)
0.57 (0.15)
Value DR Score
13.02 (11.73)
2.33 (1.41)
0.46 (0.05)
1.61 (1.02)
Value DR Clip Score|
13.02 (11.73)
0.43 (0.09)
3.96 (3.51)
1.68 (1.02)
Match Score
3.60 (0.16)
0.23 (0.04)
0.50 (0.06)
0.38 (0.08)
S Score
0.95 (0.02)
0.90 (0.04)
0.74 (0.04)
0.29 (0.05)
T Score
0.56 (0.02)
0.16 (0.03)
0.42 (0.03)
0.31 (0.05)
Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.35 (0.06)
Score
4.0 (0.11)
0.83 (0.04)
0.67 (0.03)
0.60 (0.11)
Influence Score
1455.75 (1439.46)
0.95 (0.04)
0.80 (0.02)
1.08 (0.1)
Influence Clip Score
1149.74 (1/39.722)
87.13 (60.29)
1.16 (0.16)
1.06 (0.09)
IPW Score
3.21 (0.12)
0.25 (0.05)
0.32 (0.02)
0.37 (0.06)
IPW Clip Score
3.21 (0.12)
0.60 (0.07)
0.32 (0.02)
0.37 (0.06)
IPW Switch S Score
3.21 (0.12)
0.29 (0.06)
0.31 (0.02)
0.37 (0.06)
IPW Switch T Score
3.21 (0.12)
0.29 (0.06)
0.31 (0.02)
0.37 (0.06)
IPW CAB S Score
3.20 (0.12)
0.29 (0.06)
0.31 (0.02)
0.37 (0.06)
IPW CAB T Score
3.20 (0.12)
0.29 (0.06)
0.31 (0.02)
0.37 (0.06)
DR S Score
0.93 (0.02)
0.85 (0.05)
0.73 (0.04)
0.35 (0.06)
DR S Clip Score
0.93 (0.02)
0.90 (0.04)
0.74 (0.04)
0.34 (0.06)
DR Score
0.56 (0.02)
0.16 (0.02)
0.41 (0.03)
0.32 (0.07)
DR Clip Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.33 (0.06)
DR Switch S Score
0.93 (0.02)
0.90 (0.04)
0.75 (0.04)
0.32 (0.06)
DR Switch T Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.28 (0.05)
DR CABS S Score
0.93 (0.02)
0.90 (0.04)
0.74 (0.04)
0.34 (0.06)
DR CAB T Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.33 (0.06)
TMLE S Score
1.06 (0.04)
0.91 (0.04)
0.74 (0.04)
0.26 (0.05)
TMLE T Score
0.64 (0.03)
0.16 (0.03)
0.42 (0.03)
0.31 (0.05)
Cal DR S Score
5.78 (0.19)
0.87 (0.05)
0.72 (0.04)
0.19 (0.03)
Cal DR T Score
3.45 (0.11)
0.17 (0.03)
0.42 (0.03)
0.21 (0.03)
Cal TMLES S Score
6.12 (0.21)
0.90 (0.04)
0.75 (0.04)
0.17 (0.02)
Cal TMLE T Score
4.63 (0.18)
0.17 (0.03)
0.42 (0.03)
0.22 (0.03)
Qini DR S Score
1.54 (0.06)
917.0 (896.57)
16.12 (15.38)
1.78e+6 (1.77e+6)
Qini DR T Score
1.32 (0.07)
2.87 (1.53)
0.57 (0.05)
2.08c+7 (1.90e+7)
Qini TMLES S Score
1.55 (0.06)
295.06 (173.91)
0.70 (0.04)
1.78e46 (1.78c+6)
Qini TMLE T Score
1.32 (0.07)
2.59 (1.49) 0.57 (0.04)
1.78e+6 (1.78e+6)
Table 7: Normalized PEHE of the best estimators chosen by each metric with the single-level model selection strategy; results report the mean (standard error) across 20 seeds and also across datasets for the ACIC 2016 benchmark. Lower value better.
24

---

Published as a conference paper at ICLR 2024
Metric
ACIC 2016
LaLonde CPS
LaLonde PSID
TWINS
Value Score
3.97 (1.98)
0.34 (0.09)
0.43 (0.03)
0.21 (0.03)
Value DR Score
0.64 (0.03)
0.25 (0.08)
0.47 (0.04)
0.21 (0.03)
Value DR Clip Score
0.64 (0.03)
0.22 (0.06)
0.47 (0.04)
0.21 (0.03)
Match Score
1.76 (0.09)
0.17 (0.03)
0.45 (0.03)
0.21 (0.03)
S Score
0.93 (0.02)
0.90 (0.04)
0.75 (0.0M)
0.21 (0.03)
T Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.21 (0.03)
X Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.21 (0.03)
Score
3.88 (0.11)
0.86 (0.03)
0.62 (0.03)
0.21 (0.03)
Influence Score
3.26 (0.1)
0.93 (0.04)
0.77 (0.03)
0.16 (0.02)
Influence Clip Score
3.19 (0.1)
68.67 (27.37)
1.26 (0.23)
0.16 (0.02)
IPW Score
1.41 (0.06)
0.16 (0.04)
0.38 (0.02)
0.21 (0.03)
IPW Clip Score
1.40 (0.06)
0.18 (0.04)
0.37 (0.02)
0.21 (0.03)
IPW Switch S Score
1.41 (0.06)
0.16 (0.04)
0.37 (0.02)
0.21 (0.03)
IPW Switch T Score
1.41 (0.06)
0.16 (0.04)
0.37 (0.02)
0.21 (0.03)
IPW CAB S Score
1.40 (0.06)
0.16 (0.04)
0.38 (0.02)
0.21 (0.03)
IPW CAB T Score
1.40 (0.06)
0.16 (0.04)
0.38 (0.02)
0.21 (0.03)
DR S Score
0.92 (0.02)
0.85 (0.04)
0.74 (0.04)
0.20 (0.03)
DR S Clip Score
0.92 (0.02)
0.90 (0.04)
0.74 (0.04)
0.20 (0.03)
DR T Score
0.56 (0.02)
0.16 (0.02)
0.41 (0.03)
0.21 (0.03)
DR T Clip Score
0.56 (0.02)
0.14 (0.03)
0.41 (0.03)
0.21 (0.03)
DR Switch S Score
0.92 (0.02)
0.90 (0.04)
0.74 (0.04)
0.21 (0.03)
DR Switch T Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.21 (0.03)
DR CAB S Score
0.92 (0.02)
0.90 (0.04)
0.74 (0.04)
0.20 (0.03)
DR CAB T Score
0.56 (0.02)
0.16 (0.03)
0.41 (0.03)
0.21 (0.03)
TMLE S Score
1.0 (0.03)
0.90 (0.04)
0.75 (0.04)
0.21 (0.03)
TMLE T Score
0.61 (0.03)
0.16 (0.03)
0.42 (0.03)
0.21 (0.03)
Cal DR S Score
1.18 (0.03)
0.86 (0.04)
0.74 (0.04)
0.21 (0.03)
Cal DR T Score
0.62 (0.02)
0.19 (0.04)
0.42 (0.03)
0.22 (0.03)
Cal TMLE S Score
1.39 (0.05)
0.90 (0.04)
0.74 (0.04)
0.18 (0.02)
Cal TMLE T Score
0.82 (0.02)
0.2 (0.04)
0.42 (0.03)
0.21 (0.02)
Qini DR S Score
0.62 (0.02)
21.18 (15.26)
0.68 (0.12)
0.21 (0.03)
Qini DR T Score
0.58 (0.02)
0.14 (0.03)
0.52 (0.03)
0.24 (0.04)
Qini TMLE Score
0.67 (0.03)
14.68 (9.97)
0.60 (0.07)
0.21 (0.03)
Qini TMLE T Score 0.63 (0.03)
0.15 (0.03)
0.53 (0.03)
0.21 (0.03)
Table 8: Normalized PEHE of the best estimators chosen by each metric with the two-level model selection strategy; results report the mean (standard error) across 20 seeds and also across datasets for the ACIC 2016 benchmark. Lower value is better.
25

---

Published as a conference paper at ICLR 2024
Metric
ACIC 2016
LaLonde CPS
LaLonde PSID
TWINS
Value Score
1.94 (0.31)
0.58 (0.08)
0.47 (0.03)
0.21 (0.03)
Value DR Score
0.71 (0.03)
0.37 (0.08)
0.51 (0.04)
0.21 (0.03)
Value DR Clip Score
0.71 (0.03)
0.27 (0.07)
0.51 (0.04)
0.21 (0.03)
Match Score
1.56 (0.07)
0.17 (0.03)
0.45 (0.03)
0.21 (0.03)
S Score
0.86 (0.02)
0.90 (0.04)
0.75 (0.0M)
0.21 (0.03)
T Score
0.45 (0.02)
0.16 (0.03)
0.41 (0.03)
0.21 (0.03)
X Score
0.45 (0.02)
0.16 (0.03)
0.41 (0.03)
0.21 (0.03)
Score
3.33 (0.09)
0.86 (0.03)
0.62 (0.03)
0.21 (0.03)
Influence Score
2.98 (0.11)
0.93 (0.04)
0.77 (0.03)
0.16 (0.02)
Influence Clip Score
2.89 (0.11)
68.67 (27.37)
1.26 (0.23)
0.16 (0.02)
IPW Score
1.33 (0.05)
0.16 (0.04)
0.38 (0.02)
0.20 (0.03)
IPW Clip Score
1.32 (0.05)
0.18 (0.04)
0.37 (0.02)
0.20 (0.03)
IPW Switch S Score
1.33 (0.05)
0.16 (0.04)
0.37 (0.02)
0.20 (0.03)
IPW Switch T Score
1.33 (0.05)
0.16 (0.04)
0.37 (0.02)
0.20 (0.03)
IPW CAB S Score
1.33 (0.06)
0.16 (0.04)
0.38 (0.02)
0.20 (0.03)
IPW CAB T Score
1.33 (0.06)
0.16 (0.04)
0.38 (0.02)
0.20 (0.03)
DR S Score
0.85 (0.02)
0.85 (0.04)
0.74 (0.04)
0.20 (0.03)
DR S Clip Score
0.85 (0.02)
0.90 (0.04)
0.74 (0.04)
0.21 (0.03)
DR T Score
0.45 (0.02)
0.16 (0.02)
0.41 (0.03)
0.21 (0.03)
DR T Clip Score
0.45 (0.02)
0.14 (0.03)
0.41 (0.03)
0.21 (0.03)
DR Switch S Score
0.85 (0.02)
0.90 (0.04)
0.74 (0.04)
0.21 (0.03)
DR Switch T Score
0.45 (0.02)
0.16 (0.03)
0.41 (0.03)
0.21 (0.03)
DR CAB S Score
0.85 (0.02)
0.90 (0.04)
0.74 (0.04)
0.21 (0.03)
DR CAB T Score
0.45 (0.02)
0.16 (0.03)
0.41 (0.03)
0.21 (0.03)
TMLE S Score
0.93 (0.03)
0.90 (0.04)
0.75 (0.04)
0.21 (0.03)
TMLE T Score
0.51 (0.03)
0.16 (0.03)
0.42 (0.03)
0.21 (0.03)
Cal DR S Score
1.42 (0.25)
0.89 (0.04)
0.74 (0.04)
0.20 (0.03)
Cal DR T Score
0.62 (0.02)
0.20 (0.04)
0.42 (0.03)
0.20 (0.03)
Cal TMLE S Score
1.43 (0.25)
0.89 (0.04)
0.74 (0.04)
0.21 (0.03)
Cal TMLE T Score
0.82 (0.02)
0.19 (0.04)
0.42 (0.03)
0.20 (0.03)
Qini DR S Score
0.62 (0.02)
24.86 (14.92)
0.63 (0.04)
0.21 (0.03)
Qini DR T Score
0.58 (0.02)
0.14 (0.03)
0.50 (0.03)
0.21 (0.03)
Qini TMLE Score
0.67 (0.03)
11.96 (4.04)
0.63 (0.04)
0.21 (0.03)
Qini TMLE T Score 0.63 (0.03)
0.15 (0.03)
0.51 (0.02)
0.21 (0.03)
Table 9: Normalized PEHE of the ensemble estimators chosen by each metric with the two-level model selection strategy: results report the mean (standard error) across 20 seeds and also across datasets for the ACIC 2016 benchmark. Lower value is better.
26

---

Published as a conference paper at ICLR 2024
Metric
ACIC 2016
LaLonde CPS
LaLonde PSID
TWINS
Value Score
1.21e+7 (6.21e+6)
1.15 (0.4)
0.64 (0.09)
0.57 (0.15)
Value DR Score
13.11 (11.72)
1.38 (0.41)
0.61 (0.04)
0.58 (0.14)
Value DR Clip Score
13.11 (11.72)
1.63 (0.46)
0.59 (0.05)
0.64 (0.16)
Match Score
3.49 (0.16)
0.43 (0.06)
0.51 (0.05)
0.38 (0.08)
S Score
0.98 (0.03)
0.83 (0.05)
0.75 (0.0M)
0.29 (0.05)
Score
0.61 (0.02)
0.35 (0.05)
0.48 (0.04)
0.31 (0.05)
X Score
0.61 (0.02)
0.35 (0.05)
0.48 (0.04)
0.35 (0.06)
Score
3.31 (0.08)
0.83 (0.05)
0.68 (0.03)
0.60 (0.11)
Influence Score
1.04e+6 (7.76e45)
0.91 (0.04)
0.79 (0.03)
0.19 (0.03)
Influence Clip Score
5.90c+5 (5.90c+5)
2.18 (0.38)
0.94 (0.06)
0.2 (0.03)
IPW Score
2.53 (0.09)
0.59 (0.1)
0.43 (0.03)
0.40 (0.07)
IPW Clip Score
2.53 (0.09)
0.80 (0.14)
0.38 (0.04)
0.40 (0.07)
IPW Switch Score
2.53 (0.09)
0.59 (0.1)
0.43 (0.03)
0.40 (0.07)
IPW Switch Score
2.53 (0.09)
0.59 (0.1)
0.43 (0.03)
0.40 (0.07)
IPW CAB S Score
2.53 (0.09)
0.59 (0.1)
0.43 (0.03)
0.40 (0.07)
IPW CAB T Score
2.53 (0.09)
0.59 (0.1)
0.43 (0.03)
0.40 (0.07)
DR S Score
0.95 (0.02)
0.81 (0.06)
0.74 (0.04)
0.37 (0.06)
DR S Clip Score
0.95 (0.02)
0.83 (0.05)
0.75 (0.04)
0.35 (0.06)
DR T Score
0.61 (0.02)
0.35 (0.05)
0.48 (0.04)
0.32 (0.07)
DR Clip Score
0.61 (0.02)
0.35 (0.05)
0.48 (0.04)
0.33 (0.06)
DR Switch S Score
0.95 (0.02)
0.83 (0.05)
0.75 (0.04)
0.32 (0.06)
DR Switch T Score
0.61 (0.02)
0.35 (0.05)
0.48 (0.04)
0.28 (0.05)
DR CABS S Score
0.95 (0.02)
0.83 (0.05)
0.75 (0.04)
0.35 (0.06)
DR CAB T Score
0.61 (0.02)
0.35 (0.05)
0.48 (0.04)
0.33 (0.06)
TMLE S Score
1.04 (0.03)
0.83 (0.05)
0.76 (0.04)
0.26 (0.05)
TMLE T Score
0.65 (0.02)
0.35 (0.05)
0.49 (0.03)
0.31 (0.05)
Cal DR S Score
5.77 (0.14)
0.87 (0.07)
0.74 (0.04)
0.23 (0.04)
Cal DR T Score
3.60 (0.1)
0.38 (0.05)
0.48 (0.03)
0.22 (0.04)
Cal TMLES S Score
6.09 (0.19)
0.85 (0.05)
0.75 (0.04)
0.23 (0.05)
Cal TMLE T Score
4.62 (0.15)
0.39 (0.04)
0.49 (0.03)
0.24 (0.03)
Qini DR S Score
1.36 (0.05)
8.89 (2.74)
0.68 (0.08)
1.78c+6 (1.77e+6)
Qini DR T Score
1.16 (0.05)
0.79 (0.08)
0.66 (0.05)
1.78c46 (1.77c46)
Qini TMLE S Score
1.41 (0.06)
21.78 (14.52)
1.39 (0.67)
1.78e+6 (1.77e+6)
Qini TMLE T Score
1.17 (0.04)
0.89 (0.12)
0.65 (0.05)
1.78+46 (1.77e+6)
Table 10: Normalized PEHE of the best estimators chosen by each metric amongst only the set of DR-Learners; results report the mean (standard error) across 20 seeds and also across datasets for the ACIC 2016 benchmark. Lower value is hetter.
27

---

Published as a conference paper at ICLR 2024
Metric
ACIC 2016
LaLonde CPS
LaL.nde PSID
TWINS
Value Score
2.78c+6 (1.58e+6)
800.39 (792.56)
0.46 (0.05)
1.21c+4 (1.216+4)
Value DR Score
7.63e45 (3.83c+5)
3.78 (1.82)
21.25 (15.63)
1.21c+4 (1.21e+4)
Value DR Clip Score
7.63e+5 (3.83e+5)
2.22 (0.77)
21.25 (15.63)
1.21e+4 (1.21e+4)
Match Score
5.48 (0.17)
0.27 (0.05)
0.48 (0.06)
0.17 (0.04)
S Score
4.39 (0.14)
0.9 (0.04)
0.73 (0.03)
0.10 (0.03)
T Score
4.26 (0.14)
0.2 (0.03)
0.43 (0.03)
0.12 (0.03)
X Score
4.26 (0.14)
0.2 (0.03)
0.43 (0.03)
0.13 (0.03)
Score
6.35 (0.15)
0.86 (0.03)
0.64 (0.04)
0.32 (0.1)
Influence Score
2.81e+4 (2.49e+4)
0.96 (0.03)
0.81 (0.03)
1.08 (0.1)
Influence Clip Score
1.23c+4 (1.23e+44
58.5 (56.51)
1.24 (0.17)
1.06 (0.09)
IPW Score
5.87 (0.18)
0.38 (0.06)
0.33 (0.02)
0.14 (0.03)
IPW Clip Score
5.87 (0.19)
0.57 (0.07)
0.33 (0.02)
0.12 (0.03)
IPW Switch S Score
5.86 (0.19)
0.41 (0.06)
0.32 (0.02)
0.12 (0.03)
IPW Switch T Score
5.86 (0.19)
0.41 (0.06)
0.32 (0.02)
0.12 (0.03)
IPW CAB S Score
5.86 (0.18)
0.41 (0.06)
0.32 (0.02)
0.12 (0.03)
IPW CAB T Score
5.86 (0.18)
0.41 (0.06)
0.32 (0.02)
0.12 (0.03)
DR S Score
4.38 (0.14)
0.84 (0.05)
0.72 (0.03)
0.13 (0.04)
DR S Clip Score
4.38 (0.14)
0.9 (0.04)
0.72 (0.03)
0.15 (0.04)
DR T Score
4.26 (0.14)
0.2 (0.03)
0.43 (0.03)
0.14 (0.04)
DR T Clip Score
4.26 (0.14)
0.2 (0.03)
0.43 (0.03)
0.14 (0.04)
DR Switch Score
4.38 (0.14)
0.9 (0.04)
0.73 (0.03)
0.14 (0.04)
DR Switch T Score
4.26 (0.14)
0.2 (0.03)
0.43 (0.03)
0.15 (0.04)
DR CAB S Score
4.38 (0.14)
0.9 (0.04)
0.72 (0.03)
0.15 (0.04)
DR CAB T Score
4.26 (0.14)
0.2 (0.03)
0.43 (0.03)
0.14 (0.04)
TMLE S Score
4.45 (0.14)
0.91 (0.04)
0.74 (0.03)
0.1 (0.03)
TMLE T Score
4.3 (0.14)
0.18 (0.03)
0.43 (0.03)
0.13 (0.03)
Cal DR Score
6.23 (0.19)
0.86 (0.05)
0.72 (0.03)
0.17 (0.03)
Cal DR T Score
5.94 (0.18)
0.19 (0.03)
0.43 (0.03)
0.19 (0.03)
Cal TMLES S Score
6.28 (0.2)
0.9 (0.04)
0.74 (0.03)
0.16 (0.02)
Cal TMLE T Score
6.02 (0.2)
0.19 (0.03)
0.43 (0.03)
0.19 (0.02)
Qini DR S Score
6.196 (3.75e+6)
2.82c+3 (1.51e+3)
19.4 (15.6)
1.13e+7 (1.12e+7)
Qini DR T Score
8.22c46 (1.09c+6)
3.47 (1.54)
3.92 (3.51)
1.13c+7 (1.13c+7)
Qini TMLE Score
7.38e+6 (4.08e+6)
916.03 (599.74)
6.34 (4.07)
1.13e+7 (1.13e+7)
Qini TMLE T Score
6.91e+6 (4.00e+6)
4.71 (1.86)
3.94 (3.51)
2.13c+7 (1.47++7)
Table 11: Normalized PEHE of the best estimators chosen by each metric amongst only the set of DML-Learners; results report the mean (standard error) across 20 seeds and also across datasets for the ACIC 2016 benchmark. Lower value is better.
28

---

Published as a conference paper at ICLR 2024
Metric
ACIC 2016
LaLonde CPS
LaLonde PSID
TWINS
Value Score
1.04c+7 (4.23e+6)
1.47 (0.53)
0.45 (0.04)
0.28 (0.05)
Value DR Score
1.04 (0.1)
0.35 (0.1)
0.42 (0.04)
0.29 (0.04)
Value DR Clip Score
1.04 (0.1)
0.39 (0.09)
0.38 (0.04)
0.29 (0.04)
Match Score
3.40 (0.16)
0.31 (0.05)
0.47 (0.05)
0.26 (0.04)
S Score
0.88 (0.02)
0.72 (0.03)
0.72 (0.0M)
0.22 (0.04)
Score
0.34 (0.01)
0.20 (0.03)
0.42 (0.02)
0.31 (0.01)
X Score
0.34 (0.01)
0.20 (0.03)
0.40 (0.02)
0.29 (0.04)
Score
3.14 (0.09)
0.72 (0.03)
0.61 (0.03)
0.28 (0.05)
Influence Score
1.45e+3 (1.44e+3)
0.75 (0.03)
0.76 (0.02)
0.41 (0.07)
Influence Clip Score
1.45e+3 (1.44c43)
31.67 (24.68)
1.32 (0.21)
0.46 (0.09)
IPW Score
2.40 (0.11)
0.40 (0.06)
0.33 (0.02)
0.29 (0.04)
IPW Clip Score
2.39 (0.11)
0.58 (0.08)
0.33 (0.02)
0.29 (0.04)
IPW Switch Score
2.40 (0.11)
0.40 (0.06)
0.32 (0.02)
0.29 (0.04)
IPW Switch Score
2.40 (0.11)
0.40 (0.06)
0.32 (0.02)
0.29 (0.04)
IPW CAB S Score
2.40 (0.11)
0.40 (0.06)
0.32 (0.02)
0.29 (0.04)
IPW CAB T Score
2.40 (0.11)
0.40 (0.06)
0.32 (0.02)
0.29 (0.04)
DR S Score
0.85 (0.02)
0.68 (0.04)
0.7 (0.05)
0.28 (0.04)
DR S Clip Score
0.85 (0.02)
0.71 (0.04)
0.71 (0.04)
0.28 (0.04)
DR T Score
0.34 (0.01)
0.20 (0.03)
0.40 (0.02)
0.28 (0.04)
DR Clip Score
0.34 (0.01)
0.20 (0.03)
0.40 (0.02)
0.28 (0.04)
DR Switch S Score
0.85 (0.02)
0.72 (0.03)
0.71 (0.04)
0.28 (0.04)
DR Switch T Score
0.34 (0.01)
0.20 (0.03)
0.40 (0.02)
0.32 (0.04)
DR CABS S Score
0.85 (0.02)
0.71 (0.04)
0.71 (0.04)
0.28 (0.04)
DR CAB T Score
0.34 (0.01)
0.20 (0.03)
0.40 (0.02)
0.28 (0.04)
TMLE S Score
0.96 (0.03)
0.73 (0.03)
0.73 (0.04)
0.25 (0.04)
TMLE T Score
0.39 (0.02)
0.20 (0.03)
0.42 (0.02) )
0.31 (0.04)
Cal DR S Score
3.15 (0.1)
0.71 (0.04)
0.71 (0.04)
0.22 (0.03)
Cal DR T Score
1.40 (0.06)
0.24 (0.04)
0.41 (0.03)
0.22 (0.03)
Cal TMLES S Score
3.80 (0.11)
0.79 (0.04)
0.73 (0.04)
0.17 (0.02)
Cal TMLE T Score
2.04 (0.07)
0.2 (0.03)
0.41 (0.03) )
0.20 (0.03)
Qini DR S Score
0.59 (0.03)
1.43 (0.75)
0.62 (0.15)
6.23c+5 (6.22e+5)
Qini DR T Score
0.47 (0.03)
0.89 (0.35)
0.33 (0.04)
1.96c+7 (1.90c+7)
Qini TMLE S Score
0.75 (0.04)
2.89 (1.14)
1.13 (0.6)
6.23e+5 (6.22e+5)
Qini TMLE T Score
0.51 (0.02)
1.47 (0.44)
0.31 (0.04)
6.23e+5 (6.22e+5)
Table 12: Normalized PEHE of the best estimators chosen by each metric amongst only the set of
X-Learners; results report the mean (standard error) across 20 seeds and also across datasets for the ACIC 2016 benchmark. Lower value is better.
29

---

Published as a conference paper at ICLR 2024
Metric
ACIC 2016
LaLonde CPS
LaLonde PSID
TWINS
Value Score
1.25e+7 (4.13e+6)
0.83 (0.06)
0.73 (0.03)
2.39e+6 (2.39c+6)
Value DR Score
1.24 (0.09)
0.83 (0.05)
0.73 (0.03)
0.22 (0.04)
Value DR Clip Score
1.23 (0.09)
0.83 (0.05)
8.53 (7.8)
0.22 (0.04)
Match Score
3.36 (0.16)
0.75 (0.06)
0.7 (0.02)
0.23 (0.04)
S Score
0.9 (0.02)
0.91 (0.05)
0.76 (0.04)
0.22 (0.04)
T Score
0.55 (0.01)
0.75 (0.06)
0.68 (0.02)
0.22 (0.04)
X Score
0.55 (0.01)
0.75 (0.06)
0.68 (0.02)
0.22 (0.04)
Score
3.19 (0.09)
0.88 (0.04)
0.72 (0.03)
0.22 (0.04)
Influence Score
1.32c+6 (9.54e+5)
0.91 (0.04)
0.78 (0.02)
0.17 (0.03)
Influence Clip Score
7.49c+5 (7.49c+5)
0.94 (0.07)
0.83 (0.08)
0.16 (0.03)
IPW Score
2.33 (0.11)
0.75 (0.06)
0.68 (0.02)
0.21 (0.04)
IPW Clip Score
2.33 (0.11)
0.75 (0.06)
0.68 (0.02)
0.22 (0.04)
IPW Switch S Score
2.33 (0.11)
0.75 (0.06)
0.68 (0.02)
0.21 (0.04)
IPW Switch Score
2.33 (0.11)
0.75 (0.06)
0.68 (0.02)
0.21 (0.04)
IPW CAB S Score
2.33 (0.11)
0.75 (0.06)
0.68 (0.02)
0.21 (0.04)
IPW CAB T Score
2.33 (0.11)
0.75 (0.06)
0.68 (0.02)
0.21 (0.04)
DR S Score
0.88 (0.02)
0.89 (0.04)
0.75 (0.04)
0.22 (0.04)
DR S Clip Score
0.88 (0.02)
0.91 (0.04)
0.75 (0.04)
0.21 (0.04)
DR T Score
0.55 (0.01)
0.75 (0.06)
0.68 (0.02)
0.22 (0.04)
DR Clip Score
0.55 (0.01)
0.75 (0.06)
0.68 (0.02)
0.22 (0.04)
DR Switch S Score
0.88 (0.02)
0.91 (0.04)
0.75 (0.04)
0.22 (0.04)
DR Switch T Score
0.55 (0.01)
0.75 (0.06)
0.68 (0.02)
0.22 (0.04)
DR CABS S Score
0.88 (0.02)
0.91 (0.04)
0.75 (0.04)
0.21 (0.04)
DR CAB T Score
0.55 (0.01)
0.75 (0.06)
0.68 (0.02)
0.22 (0.04)
TMLE $ Score
0.95 (0.03)
0.91 (0.04)
0.76 (0.04)
0.22 (0.04)
TMLE T Score
0.58 (0.01)
0.75 (0.06)
0.68 (0.02)
0.22 (0.04)
Cal DR S Score
5.14 (0.15)
0.9 (0.05)
0.75 (0.04)
0.22 (0.03)
Cal DR T Score
3.88 (0.1)
0.76 (0.06)
0.69 (0.02)
0.22 (0.03)
Cal TMLE S Score
5.66 (0.17)
0.9 (0.05)
0.76 (0.04)
0.2 (0.03)
Cal TMLE T Score
4.51 (0.15)
0.75 (0.06)
0.69 (0.02)
0.23 (0.03)
Qini DR S Score
1.1 (0.03)
0.85 (0.06)
0.8 (0.07)
0.22 (0.04)
Qini DR T Score
0.98 (0.03)
0.81 (0.06)
0.75 (0.03)
0.2 (0.04)
Qini TMLES S Score
1.16 (0.05)
185.85 (184.98)
0.81 (0.07)
0.22 (0.04)
Qini TMLE T Score
0.98 (0.03)
0.81 (0.06)
0.75 (0.03)
0.22 (0.04)
Table 13: Normalized PEHE of the best estimators chosen by each metric amongst only the set of Projected S-Learners; results report the mean (standard error) across 20 seeds and also across datasets for the ACIC 2016 benchmark. Lower value is better.
30


Reviews:
[{"review_id": "3n7fPHyPO9", "reviewer": "Reviewer_RQGf", "summary": "This paper focuses on evaluating CATE estimators.  The author discuss common CATE estimators and propose some additional estimation metrics.  They then describe an evaluation pipeline, using semi-synthetic data from a few sources, getting CATE estimates from 103 estimators, and evaluating the performance of 34 metrics.  They present results across four classes of datasets, evaluating a large number of metrics and comparing single-level model selection to a novel two-level model selection strategy.  The authors come to a few high-level conclusions, including that doubly-robust and TMLE methods tend to perform well and that two-level model selection performs better than single-level.", "questions": "In Section 3.1, the IPTW Switch equation shows using the IPW approach if the propensity score is >= epsilon, and the regression learner when it's < epsilon.  Presumably, epsilon is a small number.  However, the sentence after that equation says that you use IPW \"if the propensity of the observed treatment for that sample is large.\"  It looks like the regression learner is being used when the propensity score is small (< epsilon), which means that IPW is being used whenever the propensity score is just 'not small' (>= epsilon), which is very different from it being 'large'.  Am I misunderstanding something here?\n\nIn Section 4, under \"CATE estimators with well-tuned nuisance models\", the 3rd paragraph start with \"Further for the CATE estimators with final models (f)\" - what does it mean for an estimator to have a 'final model' vs not?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The problem the authors are addressing is important.  The authors correctly point out that much evaluation in the literature is lacking, often performed on synthetic data and comparing to only a small subset of other methods.  The authors overall approach is very principled and comprehensive.  The chosen CATE estimators seem representative, and the authors seem to have made reasonable steps towards evaluating on semi-realistic data.  The narrative flow of the paper is clear and it's overall well-written.  I also really liked the authors' analysis of the empirical results.  The conclusions they drew in Section 5 are interesting and well-described, suggesting real, useful recommendations for the application of CATE estimators.\n\nBased on authors' response, I have updated my score from a 6 to an 8.", "weaknesses": "The introduction suffers from a bit of hyperbole.  For example, \"This [...] problem is at the heart of EVERY decision making problem\", \"With the emergence of rich data sets\" (implying that 'rich data sets' are new), and \"The challenging task is to identify which individuals...\" ('The' implies it is the only challenging task - it should 'A challenging task' or even 'An important yet challenging task').\n\nSome terminology clarification could be helpful here.  There are a few terms which, while present in the literature, are not necessarily universally used.  For example, 'nuisance models' and the task being 'model selection' rather than causal modeling/discovery/inference.  Also, the first sentence of Section 2 says that X \"represents the controls.\"  However, 'control' is a pretty overloaded term in causality (generally referring to treatment = 0 in a binary treatment setting), and, especially since you're only considering binary treatment, calling the X's controls just feels unnecessarily confusing.  'Covariates' or 'confounders' would be clearer.  The term 'metric' also feels a bit overloaded and could at least benefit from some clarification of what definition you're using here.  In evaluation literature, I generally expect a 'metric' to be a measure of performance.  So, for example, In Table 1, I would call 'PEHE' the evaluation metric the things under the 'Metric' column 'Estimators'.  Just a sentence or two early on to clarify these definitions would go a long way for increasing clarity.\n\nIn Section 2, you define tau(x) as CATE and tau-hat as a CATE estimate.  However, I don't see where you define tau\\~, which appears for the first time, I believe, under 'CATE Model Selection Metrics' as a term in the equation.  The next sentence mentions learning tau~, but I don't see anywhere where you describe what it actually represents.  Actually, it looks like you may just be missing the ~ in the sentence before the equation (where it reads: \"includes approximating the ground-truth effect (tau(X)) on the validation set\").\n\nWhile I appreciate at some level the inclusion of novel metrics, given that you already are considering quite a lot of metrics, I would have liked more of a discussion as to the motivation for including each of these new ones.  Is there something that's lacking in the existing set of metrics that led you to the new ones?  Or are there aspects of the new metrics that you think could capture something that isn't currently captured?  Performance-wise, none of the IPTW modifications (IPTW Switch T Score, IPTW CAB T Score) appear to do better than just IPTW score, and Qini and Calibration scores rarely do better either.  That doesn't mean that these metrics weren't worth considering, or that the results aren't interesting.  As it is, while IPTW and Targeted Learning at least have a sentence each about why they are being included here (due to their handling of extreme propensities which other metrics don't do), the Calibration Score discussion just says that it \"has been studied on RCTs\", and Qini just says it comes from the uplift modeling literature.  Without a strong justification for including them, and without them performing particularly well in the results, they just come across as being included for the sake of it, and they contribute little to the paper as a whole.\n\nGiven that the two-level model selection strategy is described by the authors as 'novel', I wish a bit more detail were given.  As it is, it says \"we first select its hyperparameters using a metric that is designed with similar inductive bias\" - what metric?  Does it vary by estimator?  Also, is this a contribution? (it's described as novel but I don't see it listed under 'Contributions' in the introduction)\n\nThere are some typos/grammatical issues.  Just looking at the introduction, the first sentence of the 3rd paragraph should say \"This has led\", not \"This has lead\", in the 4th paragraph, \"which have also been shown to be effect than other metrics\" is missing the word 'more', and in the Contribution, the first sentence should not contain a semi-colon (since the phrase that comes after is a dependent clause - just replace with a regular comma).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rArjh6ZVBt", "reviewer": "Reviewer_i3fD", "summary": "The authors proposed several new conditional average treatment effect (CATE) model selection metrics and performed a extensive empirical analysis to judge the performance of the new metrics and the metrics in the existing literature.", "questions": "1. The third paragraph of section 1: \"However, the issue is that these estimators often contradict each other and we would need to perform model selection.\"\n     Is there any literature that demonstrates this issue?\n2. Section 3: the proposed model selection metrics seem ad hoc to me. Can you provide some intuition behind these metrics?\n3. As I mentioned in the Weakness, 10 random seeds are not enough.\n4. As I mentioned in the Weakness, the experiment results are not very informative on LaLonde CPS, LaLonde PSID and TWINS. The authors may want to included the results on other datasets to support their conclusion.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors provided a guidance on CATE model selection without access to counterfactual data. This can be useful in practice.\n2. The authors included 78 datasets and 34 metrics, and trained 415 CATE estimators for each dataset. The analysis should be comprehensive enough with such large scale.", "weaknesses": "The design and results of the experiment don't seem to be convincing to me. First, the authors only used 10 replicates (random seeds) to estimate the mean (standard error) Normalized-PEHE, which is probably not large enough. Second, in Tables 1 and 3, most of the metrics are \"dominating\" on LaLonde CPS, LaLonde PSID and TWINS while the metrics that are not \"dominating\" are in the minority. This means most metrics perform similarly on those datasets if I understand correctly, which suggests that the conclusions the authors drew are not well supported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4iDf2RhGGl", "reviewer": "Reviewer_5dxk", "summary": "This work concerns model selection in conditional average treatment effect (CATE) estimation. Compared with prior works, the authors conduct more comprehensive evaluations involving 78 datasets, 34 metrics and 415 CATE estimators. The authors also propose novel strategies and new metrics for the model selection task. Finally, the author refine the existing evaluation protocols by leveraging advanced generative models and realistic datasets.", "questions": "1. The author propose some new metrics (e.g., the \"Switch\" approach) which seem reasonable but show marginal performance gaps in the evaluation. I am wondering the motivation and if there is other advantage of using these new metrics?\n2. Although the typical cross validation method could be non-ideal in CATE estimation in general, I am wondering if reporting those numbers can make this work more complete and highlight the effectiveness of the proposed metrics?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors provide a very comprehensive and detailed studies in CATE evaluations. The discovered strategies of model selection can benefit both developers and practitioners. In particular, the observations that T and X scores are dominant and the effectiveness of the proposed 2-level model selection are very interesting. The authors also propose multiple effective metrics and improved evaluation protocols, facilitating development of CATE estimation models.\n2. The paper is well-written and easy to follow. Prior works regarding evaluations are thoroughly reviewed and presented.", "weaknesses": "1. The deep learning or representation-based CATE methods are excluded. The adopted estimators are numerous indeed, while the underlying machine learning models (Appendix A.2) are rather basic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "okvtzwwsxa", "reviewer": "Reviewer_3TcV", "summary": "This work uses extensive empirical analysis to explore the possibility as well as the best way to do model selection for CATE estimation. This is a really challenging problem because we don't have an ideal model selection metric for this task. If we had access to hypothetical counterfactual data, then we could have used the ideal metric PEHE. Hence, the main difficulty stems from not observing both potential\noutcomes for each sample, and we need to design model selection metrics which do not rely on counterfactual data.\n\nHowever, this is an important direction to make causal inference more democratization, especially for those who are not familiar with statistics or causal inference.", "questions": "(1) Due to this is an empirical study, there may be some concerns about the results of the experiments, such as whether all experimental conditions have been taken into account, whether the way the training and validation datasets are partitioned will affect the final experimental conclusions, and etc.\n(2) The general framework proposed in the paper seems to have high requirements for the amount of data.\n(3) Some relevant papers were missing, e.g. Nguyen et al., OpportunityFinder: A Framework for Automated Causal Inference.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This work uses extensive empirical analysis to explore the possibility as well as the best way to do model selection for CATE estimation. This is a really challenging problem because we don't have an ideal model selection metric for this task. If we had access to hypothetical counterfactual data, then we could have used the ideal metric PEHE. Hence, the main difficulty stems from not observing both potential\noutcomes for each sample, and we need to design model selection metrics which do not rely on counterfactual data.\n\nHowever, this is an important direction to make causal inference more democratization, especially for those who are not familiar with statistics or causal inference.", "weaknesses": "(1) Due to this is an empirical study, there may be some concerns about the results of the experiments, such as whether all experimental conditions have been taken into account, whether the way the training and validation datasets are partitioned will affect the final experimental conclusions, and etc.\n(2) The general framework proposed in the paper seems to have high requirements for the amount of data.\n(3) Some relevant papers were missing, e.g. Nguyen et al., OpportunityFinder: A Framework for Automated Causal Inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)