Do Contemporary Causal Inference Models Capture Real-World Heterogeneity? Findings from a Large-Scale Benchmark
Authors: Haining Yu
Yizhou Sun
Venue: ICLR 2025 Poster (2025)
Keywords: causal inference
Primary area: causal reasoning
TL;DR: We find contemporary CATE model fail to recover real-world heterogeneity through a large-scale real-world benchmark

Abstract: We present unexpected findings from a large-scale benchmark study evaluating Conditional Average Treatment Effect (CATE) estimation algorithms. By running 16 modern CATE models across 43,200 datasets, we find that: (a) 62\% of CATE estimates have a higher Mean Squared Error (MSE) than a trivial zero-effect predictor, rendering them ineffective; (b) in datasets with at least one useful CATE estimate, 
80\% still have higher MSE than a constant-effect model; and (c) Orthogonality-based models outperform other models only 30\% of the time, despite widespread optimism about their performance.  These findings expose significant limitations in current CATE models and suggest ample opportunities for further research.

Our findings stem from a novel application of \textit{observational sampling}, originally developed to evaluate Average Treatment Effect (ATE) estimates from observational methods with experiment data. To adapt observational sampling for CATE evaluation, we introduce a statistical parameter, $Q$, equal to MSE minus a constant and preserves the ranking of models by their MSE. We then derive a family of sample statistics, collectively called $\hat{Q}$, that can be computed from real-world data. We prove that $\hat{Q}$ is a consistent estimator of $Q$ under mild technical conditions. When used in observational sampling, $\hat{Q}$ is unbiased and asymptotically selects the model with the smallest MSE. To ensure the benchmark reflects real-world heterogeneity, we handpick datasets where outcomes come from field rather than simulation. By combining the new observational sampling method, new statistics, and real-world datasets, the benchmark provides a unique perspective on CATE estimator performance and uncover gaps in capturing real-world heterogeneity.

Published as a conference paper at ICLR 2025
Do CONTEMPORARY CAUSAL INFERENCE MODELS HETEROGENEITY?
CAPTURE FINDINGS FROM A LARGE-SCALE BENCHMARK
Haining Yu Amazon
Yizhou Sun
Department of Computer Science University of California, Los Angeles yzsun@cs ucla.
hainiy@amazon.com com
ABSTRACT
We present unexpected findings from a large-scale benchmark study evaluating Conditional Average Treatment Effect (CATE) estimation algorithms, i..., CATE models. By running 16 modern CATE models on 12 datasets and 43,200 sampled variants generated through diverse observational sampling strategies, we find that: (a) 62% of CATE estimates have a higher Mean Squared Error (MSE) than a trivial zero-effect predictor, rendering them ineffective; (b) in datasets with least useful CATE estimate, 80% still have higher MSE than a constant-effee model; and (c) Orthogonality-
models outperform other models only 30% of the time, despite widespread optimism about their performance. These findings highlight significant challenges in current CATE models and underscore the need for broader evaluation and methodological improvements.
Our findings stem from a novel application of observational sampling, originally developed to evaluate Average Treatment Effect (ATE) estimates from observa- tional methods with experiment data. To adapt observational sampling for CATE evaluation, we introduce a statistical parameter, Q, equal to MSE minus a constant and preserves the ranking of models by their MSE. We then derive a family of sample statisties, collectively called Q, that can be computed from real-world data. When used in observational sampling, Q is an unbiased estimator of Q and asymptotically selects the model with the smallest MSE. To ensure the benchmark reflects real-world heterogeneity, we handpick datasets where outcomes come from field rather than simulation. By integrating observational sampling, new statistics, and real-world datasets, the benchmark provides new insights into CATE model performance and reveals gaps in capturing real-world heterogeneity, emphasizing the need for more robust benchmarks.
INTRODUCTION
Conditional Average Treatment Effect (CATE) models are increasingly used to answer causal infer- ence questions in fields such as medicine, economics, and policy. But how well do these models capture real-world heterogeneity? We present unexpected findings from a large-scale benchmark study on contemporary CATE estimation algorithms. Based on 43,200 sampled variants derived from 12 datasets and evaluated across 16 CATE models, we find: (a) 62% of CATE estimates have a higher Mean Squared Error (MSE) than trivial estimator that consistently predicts zero effect, rendering them ineffective; (b) cases where at least one useful CATE estimate exists, 80% have higher MSE than constant-effect estimator, and (c) orthogonality- -based models outperform other models only 30% of the time. These findings raise important questions about the current models' ability to fully reflect the complexities of real-world heterogeneity.
Rather than introducing new models, our benchmark study focuses on evaluating existing CATE estimation models. The past decade has seen significant advancements in CATE estimation, with new methods emerging from statistics, econometrics. and machine learning (see (Chrnnozuuko
et al. (2017); Athey & Imbens] (2016); Kennedy 2023); (Salit et al. (2017) Alaa & van der Schaar

---

Published as a conference paper at ICLR 2025
(017): ; Chernozhukov et al. (2023): ; Künzel et al. (2019)) Widely available and casy-to-use tools
like EconML (Battocchi et al., 2219)) and DoubleML (Bach et al. (022) have made CATE models accessible to users with minimal expertise, leading to their broad application in high-stakes business
and scientific decisions, where accuracy is critical.
Understanding real-world model accuracy is challenging because CATE estimation models lack access to ground truth CATE. To compensate, these models rely on estimates of potential outcomes and/or propensity, known as nuisance functions. As a result, models face two key risks inaccurate potential outcome estimation and inaccurate propensity estimation- - forcing difficult trade-offs. When potential outcome risk approaches zero, the ground truth CATE estimate can be recovered, which is the core idea behind S-learner and T-learner models (Künzel et al. 2019). Conversely, when propensity is known, the Horvitz- Thrompsn estimator (Horvitz & Thompson 1952) provides unbiased CATE estimates. Most contemporary CATE models attempt hybrid approach, estimating both potential outcomes and propensities to generate the final CATE estimate. The error of nuisance estimates impacts the accuracy of CATE estimates. To minimize the effect of errors, modern CATE models employ loss functions with robustness guarantees, ensuring that errors in nuisance estimates do not have first-order effect on the CATE estimate. Such guarantees come from family of closely related theories, including but not limited to doubly robustness (Kennedy, 2023), Neyman Orthogonality
(Chernozhukov al., 2017: Nie & Wager 2020 Foster & Syrgkanis (023), and Influence function (Alaa & Van Der Schaar| (019). We refer to such models from these theories as orthogonality-hasee models. These theories rely on stringent assumptions regarding smoothness, Lipschitz continuity. sparsity, convexity, and orthogonality of the true potential outcomes, propensities, and loss functions. While mathematically elegant, their effectiveness typically validated using simulation data.
Despite theoretical guarantees, evaluating CATE models in practice remains challenging. CATE model evaluation methods aim to assess the accuracy of CATE estimation models but encounter the same challenges as the models they evaluate. Ideally, we would compute the MSE for CATE estimates and rank estimators by their MSE relative to the ground truth CATE, a process known as oracle ranking. However, in most real-world datasets, only the factual outcome is observed, not the counterfactual, making the ground truth CATE unknown unless the counterfactual generation process is explicitly known. It is widely accepted that without observing both factual and counterfactual outcomes, ground truth CATE cannot be computed, making it difficult to select the most accurate CATE estimators (Curth & van der Schaar 2021: 2023 Neal et al. [02T MMhajaan al. 2023). Without understanding CATE estimate accuracy, users cannot effectively evaluate the quality of estimators or the risk of inaccurate estimates.
To address challenge of evaluating CATE models without ground truth, two main approaches are used, each with drawbacks. The first approach simulates potential outcomes and ranks CATE estimators by their MSE on semi semi-synthetic datasets, effectively removing the risk of potential outcome estimation (see Hill 2 (Shalit et al. (2017); Künzel et al. (2019); Diemert al. (202T)) But doubts remain about whether promising simulation results translate to equally promising outcomes in real-world cases. The second approach ranks CATE estimators using proxy loss functions (see details in Section 2). particularly those with doubly robust or Neyman orthogonal properties. This approach attempts to manage both potential outcome and propensity estimation risks by exploiting orthogonality properties in the loss. Doubts remain about proxy loss functions, particularly concerning their stringent assumption requirement, finite sample property, and self-serving bias (Curth & van der Schaar 2023). The last bias occurs when CATE estimator is evaluated using loss function sharing common assumptions. For example, if we use R-loss to score CATE estimators and find R-learner (which optimizes R-loss) performs best, we cannot determine whether R-learner has indeed the lowest MSE or simply shares assumptions with R-loss. This situation is analogous to a sports player also acting as the referee. Note that, this situation is unique to causal inference where ground truth is missing in test dataset, making it impossible to compute MSE there like one would do in supervised learning. To summarize, from a risk perspective, current CATE evaluation methods either try to remove potential outcome estimation risk, or manage both outcome estimation and propensity risks. Mcanwhile, few work explores removing the risk of propensity estimation.
Given the limitations of current CATE evaluation methods, we seek new approaches that can assess CATE estimator performance on real-world heterogeneous data while relying on fewer and simpler assumptions. Drawing from the risk discussion, we ask: could eliminating the risk of propensity estimation be the solution? At first glance, this seems counter-intuitive. Observational methods inherently work with unknown propensity, thus the risk cannot be removed. This is where the method
2

---

Published as a conference paper at ICLR 2025
of observational sampling comes in (LaLonde) 1986 Gentzel al. 2021). In observational sampling,
an observational dataset is created by sampling from an Randomized Controlled Trial (RCT) dataset through carefully designed process that introduces selection bias. This approach allows CATE estimators to be trained on the observational sub-sample (with propensity estimation risk) while their performance is evaluated using the full RCT data (without propensity estimation risk).
The history of observational sampling dates back as long as the field of causal inference itself. For instance, LaLonde (1986) used construct the IHDP dataset for evaluating ATE estimates from observational data. Since then, large RCT datasets have become more available in certain domains (Gordon et al., 2019; 2022). However, few researchers (except Gentzel et al.] (202I) explore observational sampling for CATE evaluation, as mainstream research often relies on small semi-synthetic datasets, which have the drawbacks discussed earlier.
Recognizing the untapped potential of observational sampling, we hypothesize that it offers oppor- tunities to develop new statistics for identifying the most accurate CATE estimators and assessing their ability to capture real-world heterogeneity. This forms the central hypothesis of our research. To rigorously test this, we aim to develop new theoretical results and create a benchmark procedure using observational sampling for CATE evaluation.
This paper offers three key contributions to the field of CATE evaluation:
1. Benchmark Findings: Our primary contribution is new findings from the large-scale benchmark study, evaluating sixteen contemporary CATE estimation methods across 43,200 variants sampled from 12 unique datasets with real-world outcomes. As noted in the opening paragraph, these findings reveal current CATE models' limitations in capturing real-world heterogeneity. They highlight the need for further research.
2. New Evaluation Metries: new statistical parameter, Q, which equals MSE minus constant, and develop family of statistics, collectively called Q, that converge Q. We prove that, for RCT data, Q is unbiased estimator to Q and achieves O(1 /VN) asymptotic convergence rate. Additionally, introduce a control- variates-based framework to reduce the variance of Q, showing that common CATE estimation losses, such as R-loss and DR-loss, are special cases of this framework.
Our second contribution new CATE evaluation metrics. We define a
Novel Evaluation Procedure: Our third contribution a novel CATE evaluation procedure based on observational sampling and the newly developed Q This method allows for the training of CATE estimators observational sub-samples and evaluates their performance using Q on the full RCT dataset. Unlike previous benchmarks, our approach does not rely on simulated potential outcomes, addressing concerns about real-world heterogeneity and mitigating the risk of self-serving bias.
2 PRELIMINARY AND RELATED WORK
2.1 PRELIMINARIES
We formalize our problem setting using the potential outcomes framework Rubinn 2005). All
notations can be found in Table 2 in Appendix A Let (X, be tuple of random variables following distribution II, where X E X is the pre-treatment covariates, Y E R the observed outcome, and T {0. 1} is the treatment assignment. Each tuple is associated with two potential outcomes (0) and (1)). However, we observe only the outcome associated to the factual treatment
T E {0. Y (T)). We denote (x) = E[Y (O)( = x] and μ. (. (x) = E = x] as the expected potential outcome functions given the covariate ær and e(x)
x) as the treatment propensity function. The Conditional Average Treatment Effect (CATE) is then defined
as: 7(x) (1) (0))X z] μ (x) (1 (z). The Average Treatment Effect (ATE) is
then TATE = Ex(T(X)
Let D denote dataset with N i.i.d samples {(I,, tn. Vn)) drawn from II. When the treatment assignment is independent of covariates, i.e., T 1 X, we call such dataset a RCT dataset and define the constant treatment propensity E, e(x) = Pr(T = 1|X x) = Pr(T 1) and Eg
E The goal of CATE estimation is to train a CATE estimator f(x) using an observational dataset that approximates 7xx) close as possible. Given trained CATE estimator +(-) : X -> R. the goal
3

---

Published as a conference paper at ICLR 2025
of CATE evaluation is evaluate the quality of f-) by comparing it to T(-). One commonly used evaluation criterion is its MSE, also known as Precision in Estimating Heterogeneous Effects (PEHE) (Hill 2011): P(+()) Ex [((( ), which is functional that maps estimator f to a non-negative real number. Note that, MSE can be calculated only when T(-) is available, which
requires the observation of both factual and counterfactual outcomes. To ensure that effects
identifiable from observational data, we rely on the standard ignorability assumptions (Rosenbaum &
Rubin p983)
Assumption 2.1. (i) Consistency: for a sample with treatment assignment T. we observe the associated potential outcome, i.e. Y = Y(T). (ii) Unconfoundednes:: there are no unobserved confounders, so that (()), (1) 1 T|X. (iii) Overlap: treatment assignment is non-deterministic,
i.e., 0 < Pr(T = 1|X = x) < 1.
2.2 RELATED WORK
CATE estimation. There exist many methods to construct CATE estimator f(r). Here we cover three
most popular strategies. The first outcome prediction strategy predicts potential outcomes μ (0)))
(x) and p (z)) and uses their difference as the CATE estimate, i.e., f(x) = ()) (x) - - (x). This
(yn- (i((t) (zx))) by any regression model. Examples include S-learner, which regress Y on X and T, and T-learnrr, which regress Y on X for T 0 and T = 1 separately (Künzel et al.) 2019). Solving the minimization problem of arg min g ,it) fOp yields the estimator of (z) = ( 70 (x). For observational datasets, learning a shared representation ((z) for both treatment groups can improve CATE estimates. This ap- proach regresses Y on (X) to estimate potential outcomes; Johansson et al. (2018) provides bounds on generalization error. Dragonnet (Shi et al., 2019), variation of this approach, learns the represen- tations of μ ()(x)) (d(x)), ((()) ()(x)), and e(((z)) using three-head neural network. The loss function for dragonnet is Lut. (j(()) (((1)) = N Σn (Vn- - illa, ((x,)))² + ABCE((n, {()(x=))) Solving the problem of arg si ,001, 2000 a LR yield the estimator FRL (x) (o(x) (7(0) (ó(z)).
approach essentially minimizes (o (
The second semi- perimeeraae regression strategy estimates CATE based on transformed outcomes. it can be rewritten as Y mm() (T
When the true model is Y f(X) + (X) +
(X) where m(x) (z) = E|y X = = f(x) ((r)) + e(x)r(z). This reformulation then estimates T(x) by regressing transformed outcome Y mm() on transformed covariate T e(X). Robinson
=x]=
VN-ccosistent. Historically, this approach (9888) and subsequent work show that these estimates partialling-ou out estimators, Double Machine carried different names such as
Learning (Chernozhukov al. 2017). and Neyman orthogonality (Newey) 1994). to name a few.
In its simplest form, the loss function is CR(F) + ((en m)(xx)) (fn (Z.,)
with plug-in estimates m(x) and ē(æ); this is called R-loss in Nie & Wager (2020). Minimizing Ln yields the estimator fre(.). notable extension of this method is causal forests (Athey et al.
2018), which adaptively partition the data to maximize the difference between CATE estimates from different tree partitions, improving the accuracy of the estimates.
The third approach is Inverse- Propensityy Weighting (IPW). IPW is based on the Horvitz-Thompson estimator, defined as 17(2, t, = (ci) - (1-t)
1-e(#) y. When the propensity score e(x) is
known, this estimator is an unbiased estimate of T(x) (Horvitz & Thompson 1952). That
is, El Eln(X, T, Y)JX x] Txx). However, IPW is known to have high variance (Robins)
et al. 1994). To reduce variance, Kennedy (2023) suggests constructing doubly robust loss
LDR(F) = +/ En [m( bn Mn) + >x (x) - 7 (xx,)) 2 where (xx = (1 (())) (()
(1- --(2) (x:()) is shorthand function with plug-in estimates pi()) and ē(x) that we will use later. Solving the problem of arg arg, LDR yields the Doubly Robust estimator #pn(x).
When explaining the strategies mentioned above, we omit the details for sample splitting and regularization to improve readability. Modern implementation of these methods use standard ML
1 'Ltt f be a ground truth function defining the data generation process; f is often unobservable. We use to represent the main estimator, and to represent the plug-in.
4

---

Published as a conference paper at ICLR 2025
regression and classification models components. In different literature components called plug-ins, base learners, or nuisance functions.
CATE model evaluation. There is extensive literature CATE model evaluation. In principle, any score function S(f) can rank and evaluate CATE estimators. However, the key question is whether the ranking helps users identify the best estimators for their needs. While this paper focuses score functions that rank by MSE, it's useful to first explore the broader landscape of score functions.
The first category of score functions includes hypothesis testing statisties. Studies such as Cher nozhukov et al. ) Bartolomeis et al. (2024); Hussain et al. (2023) develop statistics to detect heterogeneity, unobserved confounding, or transportabilityy These statisties can rank CATE esti- mators but do not guarantee finding the estimator with smaller MSE; this makes them unsuitable for general CATE evaluation. For instance, the BLP statistic from Chermozuukov et al. (2023) is
ineffective when the CATE estimator has small heterogeneity. The second category covers rank-based metrics, commonly used in uplift modeling and CATE calibration. Examples include Radcliffe
Dwivedi et al. (2020); Yadlowsky et al. (2023); Imai & Li (202]); Xu & Yadlowsky (2022).
While useful in specific contexts, these metrics lack direct connection to MSE. For example, the
Qini index (Radcliffe. 2007) assigns the same score to two estimators + ((x) and f(x:) + 1, even if
their MSE differs. The third category consists of score functions that compare CATE estimators to ATE estimates from experimental data, as discussed in Gentzel et al. 202TT and related work.
Now, lct us turn to methods designed to find model with smallest MSE; see Curth & van der Schaar (202T: 2023); Mahajan et al. (2023) y Neal et al. (2021) for reviews. The most common approach is simulation using semi-synthetic datasets, such as IHDP and Jobs (Hill) 2011 1 LaLonne 1986), where
simulated potential outcomes make MSE calculation feasible. Extensions of this approach include
generative models for synthetic data (Neal et al. 2021 Athey et al. 2020 Parikh et al., 2022). As
noted before, simulation misrepresents real-world heterogeneity, the precise risk we want to evaluate.
Another strategy is hold-out validation, which constructs CATE estimation loss functions on test datasets. For instance, by estimating potential outcomes pi(0) and (x)), one can compute the proxy f(z) and caleulate the MSE as (f) = 1 (f(:,), ? Other loss functions can also apply; theorem 15.2.1 in Chernozhukov et al. (2024) is an example. The primary risk with this approach still self-serving bias: model may unfairly benefit from being judged by losses that favor its own design. Furthermore, hold-out validation introduces complexity, as there are numerous choices: base regression models, hyperparameters, regularization, sample-spliting, and bias correction techniques (e.g., Neyman orthogonality (Newey 1994), influence functions (Alaa & Van Der Schaar 2019)). These choices further aggravate the self-serving bias.
Recent studies (Curth & van der Schaar 2023 Mahajan et al., 2023 Neal et al. 2021 Athey et al.) 2020; Parikh et al.] (022). largely based on semi semi-synthetic datasets, have evaluated various CATE evaluation criteria, including Lop, LR. L and L However, consensus is yet form.
3 THE PROPOSED EVALUATION METRIC Q
As discussed, existing CATE evaluation criteria fall short of selecting the best models, particularly when it comes to capturing real-world heterogeneity. This makes it difficult for practitioners to choose the most suitable models and prevents the community from making substantial breakthroughs in CATE estimation. We propose to break this dilemma by introducing a new statistical parameter Q, equal to MSE minus a constant, and a family of statistics Q that can be computed from real-world data. We show that, when propensity is known (as in observational sampling). Q is an unbiased estimator Q, thus asymptotically preserving the same order as MSE when ranking different CATE models. We also discuss the generalization property, variance reduction strategies, and ways to use Q to evaluate CATE estimators.
3.1 Q AND ITS STATISTICAL ESTIMATOR Q
For a given CATE estimator f, we refactor MSE P(f) into three parts:
P(f) = Ex ((x)))
=(X)) Exlr Ex]+ (xx((X))((X)
(I)
unobservable constant
approximated from real-world dataset
5

---

Published as a conference paper at ICLR 2025
The first part, P1 = Ex (7) is unobservable but independent from the CATE estimator, thus can be dropped when we evaluate relative performance of models. The two other parts, P2(f) = Ex[+?(XX and (f)
can be approximated with confidence. This yields the statistical parameter that drives oracle model ranking:
Q(f) = P2(f) - 2P P P
Let ((3, f, (x) 27( (z))))(( v). where ry is the shorthand for Horwitz-Thonee estimator, and let (F) = q( tu. Vn: F) for n-th sample. We now define the sample statistic:
1 Σ Un
@(f) = N
( (f( (1 ln Un)]
N
Note that we can compute the value of Q without counterfactual ground truth. follows that: Lemma 3.1. Unbiasedness. When the propensity function P(T = 1|X = x) - e(x) known, =
See Appendix B for all proofs. In particular, Theorem B.2 establish the consistency of Q when propensity needs to be estimated.
Remark 3.2. Relationship with orthogonal MI. Lemma 3.1 is connected orthogonal ML methods
(Foster& & Syrgkanis (203)). For example, Theorem 15.2.1 in @Chernozhukov et al. (2024) discuss similar CATE evaluation techniques based on orthogonality assumptions, requiring the triple product of the propensity error, plug-in outcome estimate error, and the difference between two CATE estimates to converge at an O(1/VN) rate.
While Lemma may initially appear similar to results in orthogonal ML (by zero-ing out propensity risk), it is essential to establish these results without relying on orthogonal ML assumptions. As discussed in Section T CATE evaluation should be based on fewer and simpler assumptions than CATE estimation. Furthermore, as Section 4 will show, orthogonality- -aasod estimators often fail to capture real-world heterogeneity, raising doubts about their reliability in CATE evaluation. This makes it critical to develop results on stronger foundation. To our knowledge, our result is the first to provide asymptotic guarantees for oracle ranking under such general conditions.
Remark 3.3. Local Effect. Lemma 3.I can be easily extended to the case where Q is weighted by function wxx) > O. That is, Q(T; w) = Σn [w(x,) (+2 (xn) - 2m(xn, tu: fn)f( En, w(x,) is an unbiased estimator of Q(;; w) = Ex [mr( (x)((()((
(X))( This result is useful when
some samples more important than others.
Intuitively, Q are relative performance metrics. Meanwhile, they can be used to measure absolute performance of CATE estimators, in three ways below. We will demonstrate their use in Section 4 Remark 3.4. Degeneracy. A CATE estimator is useless if Q(+) 2 O: when this happens, estimator is degenerate. To see that, let f = 0 be a (trivial) CATE estimator that estimates no CATE constantly. If ((F) 2 0, the CATE estimator i has higher MSE than fo. This suggests the model is useless. As result, Q(+) 2 can be used to detect severe errors in CATE estimation.
call the
Remark 3.5. Heterogeneity screening. Secondly, let (f+a) be constant effect (ATE) estimator. CATE estimator with Q(f) > Q(fr) is useless as its MSE is higher than constant effect estimator.
Remark 3.6. Approximate MSE Finally, we can construct an MSE estimate, P(+), ), by decorating P(f) with plug-in estimates of potential outcomes (7(0) (x) and (xx as follows. P(7) helps us understand the error magnitude of CATE estimator and retains same ranking property as Q(+) P(F) (() 1/4 En (z) (x)))
Does CATE evaluation result generalize to new distributions? We answer in the next two theorems:
scientist with access to a dataset generated by one distribution may want to use to find the best CATE estimator for second, different but related distribution, without the cost of second data selection. Theorem 3.7 below shows that we can use data from one distribution to estimate Q on another, via Inverse Propensity Weighting, when the density ratio between two distributions known or can be reliably estimated.
Theorem 3.7. Generalization via Inverse Propensity Weighting. Let II, and Il, be two data dis- tributions sharing the same Txx). Let X, and X₂ be their marginal distribution of X with den- sity pa (x) and p2(x)) respectively. Also assume both distributions share common support and let

---

Published as a conference paper at ICLR 2025
((x) p1 be the density ratio. Let {FIn I, yn) S n Sn be N i.i.d samples drawn from 1T. We have Qff; (2) En, En ((Tn) (f2 - 2n(3n, In Un (xx))]
When density ratio is difficult to estimate, or when the potential outcome distribution changes, a scientist may wonder if the best CATE estimator identified for distribution is also the best for another. Theorem below states that the CATE estimator with smaller Q on one distribution is also the CATE estimator with smaller Q on the second, when the two distributions are close enough:
Theorem 3.8. Ranking Generalization. Let 111 and 1l2 be two different joint distribution of x, x, y(), Let Fn and iz be two deterministic CATE estimators. Let ho(z, yo, 91; f) = =
21())(11 - yo)) be a shorthand function. Let Du (II1, [I2] == suPhem EE, (h(,,,(0),Y)) yY(
En, [XX, K, Y Y())]] < A be the Integral Probability Metric bounded by a f.finte constant A, where Hl is set of real-valued functions such that ho(f)), ho(f2) E Н. When -(() 2 2A.
we have [1) - Q(f2; II2) > O.
3.2 RESULTS FOR OBSERVATIONAL. SAMPLING
In case of observational sampling (to be used Section 4
results can be further improved. In observational sampling, we sample subset from experiment data to train CATE estimators. We then evaluate the CATE estimators on the remaining RCT sub-sample. These results help the benchmark.
First we explore opportunitics of variance reduction. Even with Lemma 3.1. the variance of Q can still be large in finite-sample settings, driven by the high-variance nature of Horwitz-Thompson estimator 17. In this section we provide a general control variates framework to reduce variance of Q while preserving its desirable unbiased property. To start, we introduce the basic concepts of control variates (Glynn & Szechtman 2002). Let be a real-valued random variable and want to estimates its mean E.U]. We can use the sample mean estimator Ú = Σ un/N where un are i..dd samples of U. Suppose that there exists a zero-mean random variable V, V = O. Then, the control variate (()) (", + B/,,)/ = U + 0 Σ,, /N/N is also an unbiased estimator of E(U]. Moreover, the variance- aariance-minimizing minimizini choice is 0
Cov [V].
To apply control variates on Q, note that Q = # En On is the sample mean of (X, Let r(z, t, yt f) be control variates function with zero mean, i.c., E, O. Therefore
((r();))=
Σ N ((Ens (ni + Orlan n, bn Mni
n
has the same expectation Q(f), i.e., E(((r();;) [[}(
Next we show that location invariance and commonly used CATE estimation losses are special cases of this control variates framework.
Proposition 3.9. Location Invariance. Assume X I. T. Let the location invariance control variates 4-" f(x). ((LL)) = + + E., T'I (L bas yns is an
function be (Lr(æ, t, f) unbiased estimator of Q.
Proposition 3.10. Doubly Robust loss. Assume X 1. T. Define the control variates function as TDR(I, F) -2,(1, (x) and ((TDR) Q + En TDR(En, bus Uni where 7(x, 1) is We have [[Q((DD)] = Q and LDn(f) = Q( + 1 En (a tn Un) + (( is equal to @(DRR) plus constant independent from 4.
the shorthand function defined in Section 2
Proposition 3.11. R-loss. Assume X T. Define the control variates function as TR(x, = -4(1- 2x) and Q(rn) = Q N Σn, rn (In, , lns yns Fn; We have [[[((n)) = Q. Moreover when E = Pr(T = 1) = 0.5, CR(F) = @(rr) + N Em m(xx))) is equal to Q(TR)/4 plus constant independent from
Variations of Q are rank- wonk-preserving when used to evaluate CATE models. When the dataset grows large, their difference disappears. Which variant to use depends theoretical and practical consider- ations: the original Q and Q(rI.) are model-free and easy implement. Meanwhile, the variance
7

---

Published as a conference paper at ICLR 2025
reduction variants Q(r), including its special cases Q(rx) and Q(TDR), offers the potential benefit of even lower variance, at the price fitting and saving the extra plug-in estimators. Finally, note that Chernozhukov et al. (2023) proved results similar to Propositions 3.11 and 3.10 based on stringent orthogonality assumptions; their result do not generalize to other control variates.
Finally, we prove that all variants Q achieves O(1/VN) convergence rate: Theorem 3.12. Convergence Rate. Assume Y is bounded random variable, f(x) is bounded function, propensity score is bounded, i.e., O < < e(x) Pr(T= 1|X X = x) < 1 1- < 1, and that f)
the control variate function r(x, t, f) is bounded. We have VN(((r: - Q) + ON where o² (r, f) is the finite variance for g(r; f).
BENCHMARK AND FINDINGS
4.1 BENCHMARK DESIGN VIA OBSERVATIONAL SAMPLING
Now that we have studied the statistical properties of variations of Q, we are ready to use it to evaluate CATE estimation models using real-world datasets. We emphasize that the evaluation is more than a post-mortem examination. Theorems 3.7 and 3.8 suggests that results obtained from one distribution can generalize to new under the right conditions. Performance of CATE estimators on a carefully selected portfolios of observational sampling study are predictive indicators to their future performance on new and similar distributions.
Datasel generation. We use twelve large RCT datasets for this evaluation study. They are listed in Table These datasets were selected to represent diverse real-world data generation processes; the rationale for their inclusion and additional details are in Appendix
generate the estimation Dest with selection bias, and an evaluation RCT dataset Devat' there is no overlap between them. See Appendix for details. We vary three sampling parameters, 4 variations in estimation dataset size, variations in treatment %, and 3 variations in assignment mechanism nonlinearity: this results in 36 settings. For every setting, we sample Dest and D Deval jointly 100 times, yielding 3,600 pairs of Dest and Dewat- Repeating same process for the 12 RCT datasets yields 12 x 3, 600 = 43, 200 benchmark datasets.
Observational sampling, For each RCT dataset D, we sample
Estimation model selection. We evaluate 16 CATE estimation models on Dest- These models include variations of S, R, and T learners (Künzel et al. 2211)) Doubly Robust learners (Kennedy) 2023). Double Machine Learning models (Chernozhukov et al. 2017: Nie & Wager] (020). representation learning-basedd models (Shi et al. 2019), and causal random forest models (Athey et al.. 2018). We use the format <model -name>> <base-bearne
<details> as the model code when
presenting results. Full model details can be found in Table 3 in Appendix C We use code from
Curth & van der Schaar (2023): Curth 2023) Battocchi et al. (2019) for reproducibility.
Estimation and Evaluation. We train 16 models listed in Table on Dest- See Appendix fioor details. We then evaluate the trained models on Devat, using ((DDR).
4.2 FINDINGS
Table 1 summarizes the benchmark findings. For each dataset, we calculate Q for every model. Out of 43,200 datasets, 41,499 (96%) have at least one non non-degenerate model with Q(F) < 0. For these datasets, models are ranked from best (rank for the most negative Q) to worst. A model "wins" if it ranks 1, and its "win share" reflects how often it outperforms other models. We also compute each model's average degenerate rate.
The benchmark reveals critical insights into the current landscape of CATE estimation models:
1. CATE models produce degenerate estimators more than half the time. We found 62% of fitted CATE estimators are degenerate; Among them, 94% are statistically different from zero at 5% significance level. This highlights the need for speccff fine-tuning. It also suggests that using using Q(F) < 0 as guardrail is crucial for avoiding poor performance, when possible.
2. CATE estimators fail to outperform a constant- effect benchmark 80% of the time. We use Double MI. with Lasso base leamerr (dml.. 1 asso) to construct în, constant-effect estimator. Among
8

---

Published as a conference paper at ICLR 2025
Table 1: Model comparison: summary of 43,200 datasets
Model
Wins
Win share
Degenerate
Degenerate rate
Avg rank
S. n..ggb.. CV
10,491
25.5%
2,600
6.3%
4.4
S. s.ridge
5,327
12.9%
12,837
31.2%
4.2
dragon.
4,976
12.1%
18,021
43.8%
5.1
5. b.ext.ridge. cv
4,582
11.1%
20,760
50.4%
5.6
dml. dml.elastic
3,413
8.3%
19,913
48.4%
5.6
3,279
8.0%
19,916
48.4%
5.7
s.ext xxb.cc
2,648
6.4%
21.344
51.8%
6.9
r.ridge.cv
2,532
6.2%
25,209
61.2%
8.7
dr.ridge.
2,499
6. 1
24,384
59.2%
7.1
t. t.ridge.cc
1,780
4.3%
26,383
64.1% %
8.2
dr n..xgbboo
476
1.2%
29,409
71.4%
9.9
cforest
209
0.5%
31,286
76.0%
10.5
t. 1..xbb.vv
187
0.5%
31,561
76.7%
11.4
r. t.xgb.oy
110
0.3%
34,814
84.6%
12.4
dml xgb
0.0%
40,741
99.0%
15.9
dml. linear
0.0%
38,796
94.2%
14.3
25,440 datasets with non-degenerate îB < O, only 20% of CATE estimators (f) outperform řg. This finding is striking, given that these methods are explicitly designed to capture heterogeneity. also highlights the underappreciated value of heterogeneity detection methods (Crump et al. 2008 Chernozhukov et al. 2023), which deserve significantly more attention.
3. Orthogonalit-baasse based learners underperform. Despite their theoretical advantages, these models (model name dml, r, dr, and coorest) have an average degenerate rate of 71%, and win only 30% of the time. This underperformance raises concerns about the self-serving bias inherent in using their proxy losses as CATE evaluation criteria. Their performance, we hypothesize, arises from combination of factors, including the data setttaggnerating process and modeling choices. While the sample-splitting and debiasing mechanisms should. in theory, mitigate risks from poorly spec- ified outcome models, other practical challenges- such violations of assumptions required for orthogonality conditions to hold-may play role. This is a topic for our ongoing research.
4. All learners weak learners. Among the sixteen models evaluated, s xgb. had the highest win share at 25.5%. Unlike prior studies, our findings are based on real-world data rather than simulated outcomes, reinforcing the relevance of these results for practical applications. Detailed performance analysis is available in Appendix
4.3 RESULT VALIDITY AND CONSIDERATIONS
We were surprised by the findings. While anticipated relative accuracy varies, did expect contemporary CATE estimators to provide generally useful estimates. Before concluding that these results reflect fundamental issues with CATE estimation, consider several alternative explanations:
Is Q really performing oracle ranking? We present simulation results the agreement between Q and MSE P when used to select best models. We tested using semi-synthetic datasets based on the Hillstrom dataset (Hillstrom) 2008). Synthetic potential outcomes and treatments were generated, and the dataset was split into an estimation set (Dest) and an evaluation set (Deval) of varying sizes (1,000 64,000 samples). We trained the same 16 CATE models on Dest and evaluated them on Deval using Q variants as the evaluation criteria. To assess the accuracy of Q we compared model rankings from Q with oracle rankings available the simulated data using ranking metrics; see Figure for MRR and Appendix D for more details. As predicted by theory, the agreement between and the oracle improved with larger evaluation datasets, and Q consistently outperformed alternative evaluation metrics. See Appendix D.2 for more results.
Implementation Accuracy. The possibility of implementation errors is a valid concern, but we minimize the risk by re-using the codebase (Curth 2023) that has been used for recent large-scale

---

Published as a conference paper at ICLR 2025
mrr vs test dataset size on hillstrom
oracle
MSE against S-leamer (rgb) proxy loss
0.9
0
against leaemer (ir) praxy less
0.8
0.7
0lru)
MSE against T-leamer (igh) praxy loss
MSE against T-leamer (Ir) proxy loss
((rg) with base learner
Qini
0.5
((%x xgb leamer
MSE against ATE
0.4
O(real with base learner
Calibration Score
0.2
O(rgal with xgb leamer
0.1
Outcome prediction accuracy
0.0
1K
2k
1k
8k
16k
32k
64k
Dataset
Figure 1: Ranking agreement between Q variants and oracle.
benchmarks (Curth & van der Schaar 2023). We relied on existing CATE estimators and evaluation criteria when possible and used EconML for additional implementations. We committed to releasing our code following proper approval to further ensure transparency and reproducibility.
Model Selection. Our evaluation focused 16 widely used CATE models; they span the major strategies in CATE estimation discussed in Section 2.2 While resource constraints limited the number of models could include, this selection offers representative evaluation of contemporary methods. However, we acknowledge that additional models, particularly from deep learning and Gaussian Process approaches (Alaa & van der Schaar 2017). could provide further insights.
Context-Specifee Generalizability. While the datasets used in our benchmark may not cover every researcher's specific needs, they represent a diverse range of real-world data generation processes. Our results expose significant risks in CATE estimation, particularly for practitioners without the deep domain expertise necessary for rigorous model fine-tuning. These findings provide crucial insights into the limitations of widely used CATE methods in capturing real-world heterogeneity.
5 CONCLUSIONS
We introduce a new approach to evaluating CATE estimators using observational sampling, centered around the statistical parameter Q to identify the estimator with the lowest MSE. The Q family of statistics are computable from real-world RCT datasets, allowing us, for the first time, evaluate CATE estimator's 's ability to capture real-worid heterogeneity without counterfactual ground truth. However, the most important contribution of this work is not just the method, but the empirical findings themselves. These findings reveal that contemporary CATE estimators often fail to outperform trivial baselines, raising fundamental concerns about the field's reliance limited benchmarks and simulation-driven validation. They highlight significant challenges in CATE estimation and underscore the need for re-examination of evaluation practices.
Our work brings renewed attention to foundational principles and highlights new opportunities. First, it underscores the central role of RCTs in causal inference and renews interest in observational sampling methods, an underutilized tool for CATE evaluation. Second, it reveals the need for broader, more representative benchmarks. Our study, while large by current standards, is an early step toward this goal. Systematic evaluation across more datasets, including high-sample regimes, may clarify when and why models fail. Third, the observed performance gaps suggest future work should explore alternative modeling assumptions, improved regularization, or new architectures tailored to real- world heterogeneity. Ultimately, our findings do not suggest that progress CATE estimation has stalled, but that evaluation practices must evolve. The reliance small benchmarks and simulated comparisons may have created an artificial sense of model superiority. Moving forward, the field requires both methodological innovation and stronger empirical validation on diverse, high-quality datasets. We hope this work catalyzes broader benchmarking efforts, ensuring that future models and model evaluation better capture real-world heterogeneity.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We sincerely thank Randall Lewis, Lihong Li, Rui Song, Ryan Shyu, Max Farrell, Mathias Cattanco, and Andrew Gelman for their valuable discussions and contributions to this work. We appreciate the constructive feedback from the anonymous reviewers, which helped improve this manuscript.
REFERENCES
Ahmed Alaa and Mihacla Van Der Schaar. Validating causal inference models via influence functions. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research. pp. 201. PMLR, 09-15 Jun 2019. URL httpp:://pppceeeeeeeeeeee a Blaall 9a. html
Ahmed M. Alaa and Mihaela van der Schaar. Bayesian inference of individualized treatment effects using multi-task gaussian processes, 2017.
Susan Athey and Guido Imbens. Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences, 113(27)7733-7336, 2016.
Susan Athey, Julie Tibshirani, and Stefan Wager. Generalized random forests, 2018.
Susan Athey, Guido Imbens, Jonas Metzger, and Evan Munro. Using wasserstein generative adver- sarial networks for the design of monte carlo simulations, 2020.
Philipp Bach, Victor Chernozhukov, Malte Kurz, and Martin Spindler. DoubleML An object- oriented implementation of double machine learning in Python. Journal of Machine Learning Research, 23(53):1-6, 2022. URL http:/tp://// j/mlr htt.t:::/mmlr..og/ppppeeeeeeee wttp://pppppp/////....po//// 08 08 ht.m
Piersilvio De Bartolomeis, Javier Abad, Konstantin Donhauser, and Fanny Yang. Hidden yet quantifiable: A lower bound for confounding strength using randomized trials, 2024.
Keith Battocchi, Eleanor Dillon, Maggie Hei, Greg Lewis, Paul Oka, Miruna Oprescu, and Vasilis Syrgkanis. EconML: A Python Package for ML-BBased Heterogeneous Treatment Effects Estima- 2019. Version 0.x.
tion. https//githbb.compppxwwyyyye
Victor Chermozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and causal parameters, 2017.
Victor Chernozhukov, Mert Demirer, Esther Duflo, and Iván Fernández- Val. Fisher-schultz lecture: Generic machine learning inference on heterogenous treatment effects in randomized experiments, with an application immunization in india, 2023.
Victor Chernozhukov, Christian Hansen, Nathan Kallus, Martin Spindler, and Vasilis Syrgkanis. Applied causal inference powered by ml and ai, 2024. URL https::/aaxiv.. org abs/
2403. 62
Richard K. Crump, V. Joseph Hotz, Guido Imbens, and Oscar A. Mitnik. Nonparametric Tests for Treatment Effect Heterogeneity. The Review of Economics and Statistics, 90(3):389 405, 08 2008. doi: 10.1162/rest900333333
wttp:///...111/eesseeee $0.3. 389
Alicia Curth. Cateselection: Sklearn-style implementatiennnnnnnnssssssss of model selection criteria for cate 2023. Accessed:
estimation. huttaa///gLtaub..cmmAlaacccccac 2023-05-08.
Alicia Curth and Mihaela van der Schaar. Doing great at estimating cate? on the neglected assumptions in benchmark comparisons of treatment effect estimators, 2021.
Alicia Curth and Mihacla van der Schaar. In search of insights, not magic bullets: Towards demystifi- cation of the model selection dilemma in heterogeneous treatment effect estimation, 2023.
11

---

Published as a conference paper at ICLR 2025
Michael Davern, Rene Bautista, Jeremy Freese, Pamela Herd, and Stephen L. Morgan. General social survey 1972-2022. (machine-readable data file]., 2023.
Eustache Diemert, Artem Betlei, Christophe Renaudin, Massih-Rezz Amini, Théophane Gregoir, and Thibaud Rahier. A large scale benchmark for individual treatment effect prediction and uplift modeling, 2021.
Raaz Dwivedi, Yan Shuo Tan, Briton Park, Mian Wei, Kevin Horgan, David Madigan, and Bin Yu. Stable discovery of interpretable subgroups via calibration in causal studies, 2020.
Bruno Ferman. Reading the fine print: Information disclosure in the brazilian credit card market.
Management Science, 62(12):3534-355 3548, 2015.
Dylan J. Foster and Vasilis Syrgkanis. Orthogonal statistical learning, 2023.
Amanda Gentzel, Purva Pruthi, and David Jensen. How and why experimental data to evaluate methods for observational causal inference, 2021.
Peter W. Glynn and Roberto Szechtman. Some new perspectives on the method of control variates. In Monte Carlo and Quasi-Monte Carlo Methods 2000, pp. 27-49, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg.
Brett R. Gordon, Florian Zettelmeyer, Neha Bhargava, and Dan Chapsky. comparison of approaches to advertising measurement: Evidence from big field experiments at facebook. Marketing Science, 38(2):193-225, 2019.
Brett R. Gordon, Robert Moakler, and Florian Zettelmeyer. Close enough? a large-scale exploration of non-experimental approaches to advertising measurement, 2022.
Jennifer Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Stafistics, 20:217-240, 03 2011. doi: 10.1 1998//css.21100081666
Kevin Hillstrom. lenge, 2008.
The minethatdata
e-mail analytics and data mining chal- www.aiiettttattt... httpaatttaaaaaaoaaaoooomomoaoo URL ht
minethatdata--e mail analytics
D. G. Horvitz and D. J. Thompson. A generalization of sampling without replacement from finite universe. Journal of the American Statistical Association, 47(260):663-688 1952. ISSN 01621459.
Zeshan Hussain, Ming-Chieh Shih, Michael Oberst, IIker Demirel, and David Sontag. Falsification of internal and external validity in observational studies via conditional moment restrictions, 2023.
Kosuke Imai and Michael Lingzhi Li. Experimental evaluation of individualized treatment rules. Journal of the American Statistical Association, 118(541):242 256, June 2021. ISSN 1537-274X. doi: 1110000001115555 2900062249922222119233333 202..1951111 URL http: doi. org 10 1 0000/002015000051000111101000000
2021 1923511
IST Collaborative Group and P.A.G. Sandercock. The international stroke trial (ist): A ran- domised trial of aspirin, subcutaneous heparin, both, or neither among 19 435 patients with acute ischaemic stroke. The Lancet, 349(9065):1569--188) May 1997. ISSN 0140-6736. doi: 00.1016690140-6666777000111
Fredrik D. Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual inference, 2018
Edward H. Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects, 2023.
Sören R. Künzel, Jasjeet S. Sekhon, Peter J. Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 116(10)4156 4165, feb 2019. doi: 10.1073/pnas.180444 18045971166 URL https //doi.org/10. wtt.oo///....0.011.... 10.3 10005m99.....8991111111
880559716
12

---

Published as a conference paper at ICLR 2025
Robert J. LaLonde. Evaluating the econometric evaluations of training programs with experimental
data. The American Economic Review, 76(4):604-62 1986. ISSN 00028282. URL http:
//www. jstor. httgg//aaa/////0000ee6e
Divyat Mahajan, Ioannis Mitliagkas, Brady Neal, and Vasilis Syrgkanis. Empirical analysis of model selection for heterogeneous causal effect estimation, 2023.
Brady Neal, Chin-Wa Huang, and Sunand Raghupathi. benchmarking, 2021.
Realcause: Realistic causal inference
Whitney K. Newey, The asymptotic variance of semiparametric estimators. Econometrica, 62(6): 1349-1382, 1994.
Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects, 2020.
Harsh Parikh, Carlos Varjao, Louise Xu, and Eric Tchetgen Tchetgen. Validating causal inference methods. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 17346-17358. PMLR, 17-23 Jul 2022. URL https:: http::ss///rcceeeec ngs. m www.l.ppressssee vl w kh22a .ttll
Nicholas Radeliffe. Using control groups to target on predicted lift: Building and assessing uplift model. 2007. URL
James M. Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Association, 89(427); 846-866, 1994.
M. Robinson. Root-n-consistent semiparametrii regression. Econometrica, 56(4):931-954, 1988. ISSN 00129682, 14680262. URL http::/www// wtt.toor.rrr/raabe///13133/
Paul R. Rosenbaum and Donald B. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55 04 1983.
Donald B. Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Assocciation, 100(469):322-333 2005. ISSN 01621459. URL http 77 www.. jstor httg://aaae///1500555.
Uri Shalit, Fredrik D. Johansson, and David Sontag. Estimating individual treatment effect: general- ization bounds and algorithms,, 2017.
Claudia Shi, David M. Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects, 2019.
Stefan Wager, Wenfei Du, Jonathan Taylor, and Robert Tibshirani. High-dimensional regression adjustments in randomized experiments. Proceedings of the National Academy of Sciences, 113 (45):12673-12678 2016. doi: 1
URL htthhttt:///et http::/wwwwwwwww.o..cc......a org/
doi /01//bs/1ie 10/.///ppa 1614732113
Yizhe Xu and Steve Yadlowsky. Calibration enror for heterogeneous treatment effects, 2022.
Steve Yadlowsky, Scott Fleming, Nigam Shah, Emma Brunskill, and Stefan Wager. Evaluating treatment prioritization rules via rank-weighted average treatment effects, 2023.
13

---

Published as a conference paper at ICLR 2025
A
NOTATION TABLE
Let f be ground truth function defining the data generation process; / is often unobservable. We use represent the main estimator, and f to represent the plug-in.
Notation
Definition
X
Pre-treatment random vector
T
Binary treatment
(()) and (1))
Potential outcomes
Y (T)
Outcome
(1)(x)) and (x) (x)
Expectations of potential outcomes
(x) (z) and (z)
Plug-in in estimates of potential outcomes
T(x) I (x) (x) (z)
ground truth CATE function
7(x)
CATE estimate
c(r) Pr(T = 1|X x)
Propensity function
((x)
Plug-in estimate of e(x:)
Ei
Treatment probability in RCT, i.e., Ej = Pr(T 1). Also Eo 1 Ei A dataset, a list of (X,T,Y) tuples
D
N
Number of samples in dataset n-th sample in
(En, tns yn.)
Dest
The estimation dataset, subset of D The evaluation dataset, a subset D) Mean Squared Error of CATE estimator. Also known PEHE The statistical parameter; also the part of MSE that depends on f
D Deval
P
Q
Q
The sample statistics computed on dataset D
r())
The zero-mean control variate function
(())
The sampled statistics with control variates function r R loss
La
LDR
DR loss
m(x)
E|Y|X z)., used in R loss
m(x)
Plug-in estimate of m(x)
n(x,4,
Shorthand function used for IPW estimator
(x,t))
Shorthand function used in DR learner
Table 2: Notations
B PROOFS
B. PROOF OF LEMMA 3.1
Proof. To see that, notice
Pa(f)
Ex(F(e)<(o)
Ex E N
f (n Vn)
14

---

Published as a conference paper at ICLR 2025
where the second equation is due to unbiasedness of Horwitz-Thompson estimator 1). It follows that
Q
P2
2P'
E N
Σ (f(x)) (f Un))
E
Σ g(En, Erx Bri #)
N
E
Thus E[Q] Q holds as long as the ground truth propensity function e(x) is known, even if it is not constant.
Remark B.I. The proof can be extended to other unbiased CATE estimators.
B.2 PROOF OF THEOREM B.2
Based on Lemma 3.1 We can establish the consistency of Q when propensity needs to be estimated: Theorem B.2. Consistency. Assume propensity e(x) and its estimate é(x) are both bounded away from zero and way on their support: 0 < ē I c(x)),(z) S 1 < 1. Also assume lim, (r (z) e(X)]l c) 0 for all 0. We have for all e 0.
lim Pr(IQn(f) - Q(f) > =0 20
(5)
Proof.
First, notice
-(e(x))-Q(((o)))
- ((E()))
Σ
(((x)) - 2f(xxn)
tn
tu
(6)
N 7
e((n))
- el [mn) )
Σ (2(x)) - 22((xn) e((m))
tn
t
(7)
{(xm) )
Σ (an)))
tn
tn
1 tn {(Tn)
1 tn
(8)
[((un)
e((nn
ε(vT))
follows that
(e((x))
((((x)))
(9)
1
Σ (f
In
In c(xx) + - In
1 In è ê((n)
(10)
N
(((x))
e(xn)
1
Σ 2f(wn)
1 - In In (ē(xm)) -((x)]+ (1- (1-e(xm))( e(xr))
(ē(xn) e(a.10))
N
(((x))((Zn) (((x))((n)) + (1- el==) (l((m))) )
1
(2F()))
tn
- tn
e(xn))
le(xn) e(xu))
(12)
N
< C N
Σ (e(xn)) - E(xx)1
(13)
C (Ex]le(X) é(X)]] + =1)
(14)
s G (Exlee(X e(X)] + (=rl)
(15)
where
1
C = 4 max
max
(16)
52 (1 [[-
constant and
El = Ex(le(X)-eee - [(X)]
N Σ Je(xn) - E(σn))
(17)
15

---

Published as a conference paper at ICLR 2025
is zero-men mean random variable with asymptotic variance the order of o(1/V\) due to Central
Limit Theorem. As a result we have lim,-+0o Pr(E1 > e) = It follows that
lim Pr(E)e(X) - ε(X) > =0
(18)
n 100
Combining the two yields
lim - >))
(19)
Secondly, by Theorem 3.12 have
((((e(())-Q(((()))) + N(O,0²)
(20)
follows that
lim Pr((((e(1)) - Q(e(x)) > =)
(21)
Finally, notice that
(((())) Q(e(x)) (((((=)) Q(e(=)())))))))))))(((())))))))) + (Q(e())))+(()))))))))))))))()))) Q(e(x))
(22)
is the sum of two parts. The convergence for the first part is given by 19) and the second part by follows that
lim Pr((((((=)) Q(e(x)) c) =0
(23)
B.3 PROOF OF THEOREM 3.7
Proof. Recall Q P2 2P3. We have:
Pa X2)
[x,++(X)]
(24)
Ex +x(((x))²(X))
(25)
Em
Σ ((an)) N
1
(26)
where the second equation is is due to inverse propensity weighting and the definition of density ratio, and the third equation due to the unbiasedness nature of sample mean. Similarly,
(a(F:(a))
Ex,, [T(X)f()))
(27)
= Ex (X)
(28)
Ex, ((()))(X)((
(29)
ETI,
Σ ((an)) ((n))
(30)
N
Combining the two yields
[ N
Q(+; = En,
Σ ((xn) (+2(z)) - 2n( tnx Un )o (ax))
(31)
To summarize, if we compute Q(+) on II, and weight by IPW density ratio Ç. we get an unbiased estimator of Q(7) on ll.
16

---

Published as a conference paper at ICLR 2025
B.4 PROOF OF THEOREM 3.8
Proof. First note that, for given CATE estimator F, the difference between Q(f) under II and ll2 bounded by A:
1Q(f:I1) Qf; II)I
Ex)(9(x) 2+(X) - Ex (x)))
|xx1)F(X) 2f - (x -Y(o]]
y(u)]] Ex,1+((X)
En (X, "T,, y(())] - F11, 140 (X, ((0) y
ETT, [f ( y( E11, |n(X Y ( (0), y []]]
sup
NCH
DH (11, I12)
where the third equality is due to the definition of ho and the fifth step is due to the definition of IPM Du (Il1, ll.).
Let us assume we have two CATE estimators, (x) and 2o (z), and
((t: - Qff; (f2) 2 22
(32)
It follows that
(I - A - (Q(f2: II,) + A)
W QI Q(f: II,) 24
0
That is, 72 is also better on IT2.
B.5 PROOF OF PROPOSITION 3.9
Proof. First we show that (X,T, = 0, i.c. it is a control variates function. This is obvious because
(1-T)
i(x)
(X,, = 20E
(33)
Fj
Eo
T
(1 - T)
20E
B(7(X))
(34)
E1
Eu
20(1
1)E((((X))))) (X)]
(35)
0
(36)
where the first step is by definition of (-) and the second step is by property of RCT dataset.
follows that
B[((rL))] = E
Q +
Σ rLJ (Zny fn, Uni +)
(37)
N
[ + (X,,
(38)
E[Q)
(39)
Q
(40)
B.6 PROOF OF PROPOSITION 3. 3.10
Proof. First we prove
= First note that,
E (1
) (1) (X)+(x)
= [ E
(41)
E,
= (1 (x)
(42)
(43)
17

---

Published as a conference paper at ICLR 2025
Similarly
1 T
(x0 =0 1 1 Eo
(44)
Combining the results above yields (Tr = 0. It follows that E[Qna] B[Q] Q
Next we show that LDR a linear function of ((F). Recall
1 Σ (n fn vn) + (( (x) - )2,,))
DDR(F) =
(45)
N
1
Σ [In(zn fn Bn) + (x tn)l (x(x))
2mm(an tn Mn) + (x)
N
1
Σ Min lus
Un) Un) (( (n)
+ +
1
Σ
(47)
N
N
2
Σ N (2 tns vn) + (m))
+ Q(DRR)
(48)
where the first term is independent from and thus can be omitted for ranking purposes.
B.7 PROOF OF PROPOSITION 3.11
Proof. First prove the zero-mean property:
E[(1 - (X)x() = ([ 27)
(49)
E[m(X)(XX)} -0
(50)
O
(51)
It follows that [[ = B[Q]] = Q.
Next, note that when E₁ Eg E(x) 0.5
Q = N
Σ (I2 + 4(1 2t. (xx,)
(52)
n
follows that
Cn(f)
((8n - m)(x)) - (tn, - (((xx)))((xx)
(53)
N
Σ [men 2 + (In - ē(τm) 1) 2 (#x) 2(yn - ((xa))) - =(zx))
(54)
1
(Un m)²
+ N
1
Σ (In {(zx)) ()²) - 2(y,. m(xx)) E(en)) (F
(55)
N
1
Σ
(Un m)²x))) + N Σ
7 (xx)) + (2/ -1
(56)
N
1
Σ
(un m(xx))²² +
4N Σ (f((z))) - 4(m m(Ix))(1 2tt)
(57)
1
(Un m(xx)))
(()))) + 4w, 24, 2l.))(2zm)) - Σ 4ml(x,) (tr
N
4N
Σ In m(xx)))
((Ta)
(59)
N
where the first term is constant without impact ranking.
18

---

Published as a conference paper at ICLR 2025
B.8 PROOF OF THEOREM 3.12
Proof. Based on the assumptions, it is easy to see that the random variable 7) T, Y) Combining this with the boundedness of f(x) bounded. It follow that f(r;f)
YY is bounded.
get ((x,, L, ///7) (x) (f)))(( (. is + r is also bounded and thus have finite expectation and finite variance. Let us denote its variance as o² (r, f). By Lindeberg-Leee CLT, the sample mean (r;; converges in distribution to its expectation Q: (N(((r;F) Q) o2,
C DETAILED CONFIGURATION OF CATE ESTIMATION MODELS
We train sixteen CATE estimation models listed in Table 3 This includes two S-learners, two T-learners, two R-learners, two Doubly Robust learners, four Double MI. learners, causal tree (forest) learner, and one representation learning learner, discussed in Section 2
We hope the selection covers mainstream CATE estimation methods. We include meta-leamer (e.g., S and T learners in Künzel et al. (2119) in empirical study, because they represent the outcome prediction strategy, arguably the most simple and direct methods for causal inference. We include Double ML methods (Chernozhukoy et al. 2017) because they represent econometric/semi parametric view of causal inference, as well as the recent development orthogonal and debiased Machine Learning. We include Doubly Robust learners (Kennedy) (023) because they represent Inverse Propensity Weighting (Robins et al.] 1994). classic causal inference technique. We include causal forest because they we include DragonNet to represent recent trend using deep learning and representation learning for causal inference.
of the earliest ML-based work with theoretical guarantee. Finally,
Our limited resource prevents us from including more CATE estimation methods. As a result we do not claim this list be complete or "optimal". Due resource constraint, we were unable to include more variations of deep learning models following Shalit et al. (2017), causal tree models following Athey et al. (2018). or Gaussian Process models following Alaa & van der Schaar (2017). The list may not be "optimal" because some of the modeling approaches are related, most notably between Double Machine Learning and R learners.
Table CATE estimation Models
CATE estimation Method
Base Learner
Model Code
DR learner
Ridge Regression
dr ridge.ec
DR learner
XGBoost
dr.xbb.
R learner
Ridge Regression
r. r.ridgeccy
R learner
XGBoost
r. x.b.bboo CV
learner
Ridge Regression
8. s.ridge.cv
S learner
XGBoost
s s.xgb.cm
S learner
Ridge Regression
s.ext. www.ridgee... CV
learner
XGBoost
S s.ext. exextxgb.co
T learner
Ridge Regression
t.ridge.cv
T learner
XGBoost
t.xgb.cV
Double ML
Linear Regression
dm1 1ineaar
Double MI
Lasso
dml lasso
Double MI.
Elastic Net
dml.e. elsasii
Double ML
XGBoost
dm1.xgb
Generalized Causal Forest
Random Forest
cforest
Representation Learning
Neural Net
dragon. nnn
Note that, sxext. e.ex..xb. cv and s. ext ridge. ov models are variants learners where the interaction X T are constructed explicitly as model inputs.
We use codebase in Curth 2023) for model estimation and evaluation. Out of the sixteen models listed in Table eight meta-learmers (S-leamnes, T-leamers, R-learners) and two Doubly Robust learners directly come from Curth (2023) implementation. Following Curth & van der Schaar (2023), we use two base learners, linear regression (implemented an sklear RidgeCV object), and XGBoost
19

---

Published as a conference paper at ICLR 2025
(implemented as an XGBoost XCBRegressor object with grid search implemented by sklearn Gr Gride ddeearchCV). See Curth (2023) for details. The five Double Machine Learning learners are implemented using LinearDML and NonParamDul classes in EconML package. using sklearn Gradi ent Boost i RngRegressor as the potential outcome learner and respective base learner as the residual model learner.
All treatment propensity estimators are implemented using sklearn RandomF freer class; clip propensity outputs between 0.05 and 0.95.
20

---

Published as a conference paper at ICLR 2025
D SECTION 4.3 SUPPLEMENTAL DETAILS
D.1
SEMI-SYNTHETIC DATASET GENERATION DETAILS
We follow the steps below transform raw covariates into feature I::
Apply one-hot encoding all categorical covariates
Linearly scale all float covariates between 0 and
Colume stack all covariates to generate a feature vector
If feature vector has more than 100 elements, randomly select 100.
We follow the steps below to generate synthetic outcome:
Generate two random vectors 30 and 31 with discrete values of [0, 2, 4] and discrete probability of [0.5, 0.2, 0.15, 0.1, 0.055
Compute transformed feature and outcome using the one following three approaches:
Linear. First compute zo(z) 2, (z) e" then generate po(x) P 20(2)) and
f1 =f =1(2)
Interaction.
First compute =x(x) = [Igæ] 1, J 12 2 DD and 2 (z)
2113,, .. ID then generate mo(x) B zo (x) and MM 3 21 (x)
[ F1 .... IDo and 21 (z) [od22, 2133, ID then generate (o(x) cos (5) (x)))) and μr sin [5] 21 (x))
Sine.
First compute zo(x)
Scale μo(x) and μi (x) have zero mean and unit standard deviation
generate yo (z) + (O, 1) and yı (z) + (0, 1) + where is now the ATE estimate.
We generate synthetic treatment as follows:
Generate random vector Br
Calculate Pr(T) X = z) 14ePra-1.
Sample using the Pr([)x)
D.2
AGREEMENT BETWEEN MODEL SELECTION CRITERIA AND ORACLE
prec1
spearman
1.0
oracle
0.9
0.9
0.9
0.8
0.8
0.8
Gin) with base learner
((n)) with zgh bane learner
0.7
0.7
0.7
0.6
0.6
0.6
Grea) with ago base icarner prediction
0.5
0.5
0.5
against
(igh)
0.4
0.4
against S-learner (ir MSE against T-eamer (ogb)
MSE againstt T-lemer (tr)
Qtni
Figure 2: Interaction transformation; = 2.0
21

---

Published as a conference paper at ICLR 2025
mr
P02
1psumin
4.5
032
0954
Oltdd w 500 bit norer
MCC net cemer 11 Die Baree
RSC spaint Heareer pig AETWE
MGE SOR ME
OPal wihh weree
Figure 3: Sine transformation; 0.5
PA2:
1.4
4.4
022
63
0009
Opd with bamer
Gtod 10l saree
950 AAARc Maaree Mar NE/WT
Mill 3go 11
Opal with wore
apan
Figure Linear transformation; 0.5
22

---

Published as a conference paper at ICLR 2025
E DATASET INTRODUCTION
We use twelve RCT datasets in Section 4.3 and listed in Table In this section we discuss the rationale of data selection, and provide a brief introduction to individual datasets and data handling.
The goal for the dataset selection is to ensure that, collectively, they better represent real applications of causal inference than those datasets studied by current state of the art (e.g, Curth van der Schaar (2023); Mahajan al. (2023); Neal al. (202I)): including IHDP, ACIC, and LaLonde. We achieve that by applying the following four factors when selecting the datasets:
Real-world heterogeneity. select datasets collected from real-world, and forgo datasets with simulated outcomes, to allow evaluation performance of CATE estimator real-world heterogeneity. In comparison, simulated outcome on IHDP or ACIC do not achieve the same rigor.
Dataset size. we prefer large datasets to allow the asymptotic property of Q to kick in. The smallest the selection sandercocs has 19,000 samples.
Diversity, The datasets are curated cover a diverse sources. They differ domains (c.g.. marketing in eri teo and hils sssstt om, consumer behavior in ferman, medical science in s sandercock, and sociology and political science in GSS datasets), geography (GSS from the United States, fe ferman from Brazil, sande rcock from Europe, and cri triteo from Russia), and form of experiments (traditional RCT in sandercock, online A/B testing in criteo and hillstrom and field survey in GSS)
Prior work. We select datasets previously studied by causal inference and related literature. For example criteo is used for uplift modeling in Diemert et al. (202I); the nat fare is used for regression adjustment Wager et al. (2016)
Table 4: RCT Datasets
Dataset
Samples
Treatment
Features
References
criteo
13,979,592
85%
12
Diemert et al. 2021
ferman
103,116
82%
Ferman (0015)
hillstrom
64,000
67%
12
Hillstrom 2008
sandercock
19,435
50%
24
IST
Collaboratev Group & Sandercock (1997)
nataid
51,957
50%
20
Davern et al.
2023)
natarms
51,987
50%
20
Davern et
2023
natcity
51,915
50%
20
Davern et al.
2023
natcrime
51,977
50%
20
Davern et al.
2023
natdrug
51,961
50%
20
Davern et
20233
nateduc
52,017
50%
20
Davern et al. 2023)
natenvir 52,027
50%
20
Davern et al.
2023)
natfare 51,993
50%
20
Davern et (002))
Criteo dataset Diemert et al. 1021 captures advertising related online shopping behavior for 13,979,592 web users (identified by a browser cookie) RCT. Each user is randomly assigned to either treatment or control group. Pre-assignment user activities before assignment is used to construct covariates.
user treatment, they are subject to ad exposure: they are in control group, they are expose to the ad. The dataset tracks multiple outcomes such as visits and conversion. We use visit as the outcome in the current analysis.
Hillstrom dataset Hillstrom (2008) contains email marketing related activity for 64,000 shoppers who had purchase records within a year. Through randomization, one third of the shoppers receive marketing e-mail campaign featuring Men's merchandise; one third receive email featuring women's s; and the last one third received no marketing email. Covariates include past purchase history, gender, geo location, etc. In the current paper, combine the two groups who receive marketing email into one treatment group; the remaining group (who receives no marketin email) is the control group. We use visit the outcome variable.
23

---

Published as a conference paper at ICLR 2025
Sandercock dataset Collaborative Group & Sandercock (1997) includes data 19,435 patients with acute stroke. Patients in treatment group are treated with aspirin; patients in control group not. Covariates include age. gender, and other medical information. The binary outcome is whether patient is dead or dependent on other people for activities of daily living at six months after randomisation.
Ferman dataset Ferman 2015) includes shopping activity related to credit card payment plan on 103,116 customers of Brazilian credit company. Customers are randomly assigned into three groups: 34,743 customers in the first group were offered menu of payment plans with interest rate equal to 6:39%, 49,573 customers in second group were offered plans with interest rate equal 9:59%, and the third group of 18,800 customers did not receive any payment plan offer. The outcome is whether the customer defaults within 12 months after the offer. We combine the first and second group into one treatment group.
GSS datasets include responses to eight questions from more than 50,000 respondents surveyed by The General Social Survey (GSS) Davern al. (2023) between 1986 and 2022; response to each question consistutes RCT dataset. GSS is an annual sociological survey created 1972 by the National Opinion Research Center (NORC) at the University of Chicago. It collects information biannually and keeps a historical record of the concerns, experiences, attitudes, and practices of residents of the United States. GSS Survey regularly include randomized wording experiments to capture heterogeneity in respondent's s opinion on social issues. For a given randomized question, the question variation forms different treatment arms, the answer the question forms the outcome. GSS also collects hundreds of high-quality demographic variables, capturing demographic, work, family and spouse. household, racial, and region related information. These variables become the pre-treatment covariates. We use eight wording experiments (nataid, d, natarms, natcity, nat er imm,, nat batdrug, nat educ, natenvir, nat fare)) to construct the binary outcome. Its value is equal to 1 if and only if when a respondent answers "too much" to question, and 0 otherwise.
F SECTION 4 ADDITIONAL DETAILS ON OBSERVATIONAL SAMPLING
For every original dataset in Table we vary three parameters when generating Dest: first we set the estimation dataset size to be one of the following value [1000, 2000, 4000, 8000], to test if certain models perform better with more (or less) data. Secondly, we set the expected treatment to one of the following values 10.1, 0.5, 0.9], to test if certain models are sensitive treatment imbalance. Third, we use MLP generating assignment mechanism, and set the number of MLP layers to be 2, or 3. This tests if models are sensitive to nonlincarity in assignment mechanism. We enumerate all parameter combinations, leading to 4 x 3 = 36 settings. For every setting, we sample Dest and Devat jointly 100 times. This gives 3,600 pairs of Dest and Deval- Repeating same process for the 12 datasets yields 12 x 3, 600 = 43, 200 datasets.
We train model on small estimation datasets with selection bias, and evaluate model performance on large unbiased RCT datasets. The starting point is an RCT dataset D. We first randomly split D into Devei and D Denat- We then sample D Devat get estimation dataset: for every sample (x. 4, y), define a random variable K E {0. 1} with Pr( K = 1/7 X x) G(I, We keep the n-th sample in Dest if and only if kn 1. G(t, is the biasing function since it introduces selection bias to the original RCT dataset. This creates the estimation dataset Dests subset of D with selection bias. We apply any CATE estimation method on Dest to obtain an CATE estimator (ar), and use fxx on Deval to compute Q. Fig. illusttrasss the process. Note that, estimation dataset, the treatment is function of covariates; in evaluation dataset, the treatment is randomly generated based on standard binary distribution. For every dataset and evaluation dataset size, we repeat simulation 100 times.
The complete algorithm creating Dest is summarized below:
Trreatment % of 0.9 can be different from 0.1 hecause different potential outcomes may be different in real-world datascts.
24

---

Published as a conference paper at ICLR 2025
Experiment Dataset D
Random split
Dest
Keep subsample based on covariates and treatment
Train Dataset Dobs
Test Dataset Drest
CATE Estimation Algorithms 1,2....A
Training
CATE estimators
Evaluation
estimator a*
Figure 5: Overall approach
Algorithm 1 Creating estimation dataset Dest
Input: RCT dataset D
Input: function G(t, 2), 0 G(t, x) 1
Output: Observational dataset Dest Dest = 0
for every sample in D do
Sample a binary random variable Kn, with Pr(K = 1|X = Ia, T= (.) = G(In, F.))
If K, add sample Dest
end for Return Dest
Note that, the biasing function G(t, function generates the following assignment mechanism for Desti
(60)
=
(61)
= [[[x=ıK==]]
Pr(T'=1,3 =,,,,
(62)
Pr(X =x =x
Pr(T = Pr(X =
(63)
Pr(X x)
x
Pr(T = = =(
(64)
Pr(X -
=.,
0)
= Pr(T 1)G(x, PrT 0)G(z,0)
(65)
(66)
Pr(T=0) G(a.0) 1
Pr(T=1) G(a.1)
As result, T is dependent X, achieving selection bias Dest-
25

---

Published as a conference paper at ICLR 2025
Note that
Pr(X & <x|K=] = Pr(X & XXA.://A........................ 407)
Pr(x S 1,, _(
(68)
Pr(K=177-0)
Sr f())G(z,1)d
S (r,)
(69)
Jo [x))((x 1)dr
Jo (xx) 0)dz
So f())G(z, 1)dr Ex(G(X,1)
1o' (())G(x, 0)dr
(70)
(x(,0
where f is density of X on II. It follows that
G( Ex|C(X,
G(x, 0)
fese()) f(x)
(71)
Ex (x(,(,
The complete CATE estimator evaluation algorithm is summarized below.
Algorithm Selecting best CATE Estimator
Input: list of A CATE estimation models a = 1, 2, ...;
Input: RCT dataset D
Input: Biasing function G(t, x))
Output: a", 1 S a² A the best performing model
Randomly split D into Dtrain and D Deval
Generate Dest from Drroin using G as the biasing function for model a, 1 S a < do
Train CATE estimator fa(x)) using data Dest Compute on every sample (æns tn, Un) E Deval
Compute (faa, Deval)
end for
Return a arg ming Q(fa, Deval)
26

---

Published as a conference paper at ICLR 2025
G
SECTION 4
RESULTS FOR ALL RCT DATASETS COMBINED
Model win share dataset size: all
100%
s.xgb.cV
s.ridge.cv
dragon.nn
80%
s.ext.ridge.cv
dml.elastic
dmi.lasso
s.ext.xgb.cv
60%
Win share
r.ridge.cv
dr.ridge.cv
t.ridge.cv
40%
dr.xgb.cv
cforest
t.xgb.cv
r.xgb.cv
20%
dml.xgb
dmi.linear
0%
2k
4k
Training dataset size
Figure 6: Model win share by training dataset size: all RCT datasets
Model win share vs. mean treatment %: all
100%
s.xgb.cv
s.ridge.cv
dragon.nn
80%
s.ext.ridge.cv
dml.elastic
dmi.lasso
s.ext.xgb.cv
60%
Win share
r.ridge.cv
dr.ridge.cv
t.ridge.cv
40%
dr.xgb.cv
cforest
t.xgb.cv
r.xgb.cv
20%
dml.xgb
dml.linear
0%
0.1
0.5
0.9
Mean treatment %
Figure 7: Model win share by treatment ratio: all RCT datasets
27

---

Published as a conference paper at ICLR 2025
Model win share complexity: all
100%
s.xgb.cv
s.ridge.cv
dragon.nn
80%
s.ext.ridge.cv
dmi.elastic
dmi.lasso
s.ext.xgb.cV
60%
Win share
r.ridge.cv
dr.ridge.cv
t.rigeeccv
40%
dr.xgb.cv
cforest
t.xgb.cv
r.gbbcv
20%
dml.xgb
lml.linear
0%
2
Complexity
Figure 8: Model win share by assignment mechanism complexity: all RCT datasets
Win vs. degenerate rates: all
100%
s.xgb.cv
s.ridge.cv
dragon.nn
s.ext.ridge.cv
80%
dml.elastic
dms.lasso
Degenerate rate
s.ext.xgb.cv
60%
rridge.cv
drridge.cc
t.ridge.cv
dr.xgb.cv
40%
cforest
t.xgb.cv
r.gbb.cv
20%
dml.xgb
dml.linear
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
Win rate
Figure 9: Win share vs. degenerate rate, by model, all RCT datasets
28

---

Published as a conference paper at ICLR 2025
H TABLE WITH STANDARD ERROR AND 95% CONFIDENCE INTERVALS
Table Model comparison: summary of 43,200 datasets. The standard error and 95% confidence interval come from 10,000 bootstrapping
Win Rate
Win Rate
Degeneracy Rate
Degeneracy Rate
Avg Rank Mean (SE)
Avg Rank
Model
Mean (SE)
95% CI
Mean (SE)
95% CI
95% CI
cforest
0.005 0.000)
(0.004, 0.006)
0.760 0.002) )
(0.756, 0.764)
10.5 ( )
(10.4, 10.5)
dml.elastic
0.083 0.001)
(0.080, 0.086)
0.484 (0.002)
(0.479, 0.488)
5.6
(5.6, 5.6)
dml. lasso
0.080 0.001) )
(0.077, 0.082)
0.484 (0.002
(0.479, 0.489)
(0.0)
(5.6, 5.7)
dml.linear
0.000 (0.000)
(0.000, 0.000)
0.942 (0.001)
(0.940, 0.945)
(4.3
(14.3, 14.3)
dmi.xgb
0.000 0.000
(0.000, 0.000)
0.990 0.000)
(0.989, 0.991)
15.9
(15.9, 15.9)
dr.ridge.cv
0.061 0.001)
(0.058, 0.063)
0.592 0.002
(0.588, 0.597)
(0.00
(7.1,
dr.xgb.cv
0.012 (0.001)
(0.011, 0.013)
0.714 (0.002)
(0.710, 0.719)
(0.0)
(9.8, 9.9)
dragon.nn
0.121 (0.002
(0.118, 0.124)
0.438 (0.002)
(0.433, 0.443)
(..0
(5.1, 5.2)
r.ridgecvv
0.062 [0.001]
(0.059, 0.064)
0.612 (.002)
(0.608, 0.617)
(0.0
(8.7, 8.8)
rxgb.cv
0.003 (0.000)
(0.002, 0.003)
0.846 (0.002)
(0.842, 0.849)
12.0
(12.4, 12.4)
s.ext.rdge.cvv
0.111 (0.002
(0.108, 0.114)
0.504 (0.002)
(0.499, 0.509)
(..0)
(5.6, 5,6)
s.ext.ggb.cv
0.064 0.001
(0.062, 0.067)
0.518 (0.002)
(0.514, 0.523)
(..0)
(6.9, 6.9)
s.ridge.cv
0.129 (0.002)
(0.126, 0.133)
0.312 (0.002)
(0.307, 0.316)
(..0)
(4.2, 4.2)
s.xgb.cv
0.255 ( 0.002
(0.251, 0.259)
0.063 (0.001)
(0.061, 0.066)
(..0
4.4)
Lridge.cv
0.043 0.001
(0.041, 0.045)
0.641 (0.002)
(0.636, 0.646)
(..0
(8.2, 8.3)
txgbbev
0.005 (.000)
(0.004, 0.005)
0.767 (0.002)
(0.763, 0.771)
11.4
(11.4, 11.4)
29

---

Published as a conference paper at ICLR 2025
1 SECTION 4 DATASET LEVEL STATISTICS
1.1 SUMMARY OF DATASET SPECIFIC BEHAVIOR
Does models' relative performance vary by amount of training data, treatment imbalance, or level of nonlinearity in assignment mechanism? Looking at the aggregated results (A (Appendix G we found moderate fluctuation of model performance when varying these drivers. This likely because variations averages out. Dataset level statistics shows different picture. We present selected results this section and leave more Appendix !
Training dataset size can have large impact on model performance. On criteo, Win share of R-learner (r. ridge. increase from 7% with Ik cstimation data, to 26% with 8k estimation data. See Figure 10
Model win share vs. dataset size: criteo
100%
s.xgb.cv
r.ridge.cv
dragon.nn
80%
s.ridge.cv
s.ext.ridge.cv
s.ext.xgb.cv
dr.ridge.cv
60%
Win share
dml.elastic
dmi.lasso
cforest
40%
r.xgb.cv
t.ridge.cv
dr.xgb.cv
dml.linear
20%
dml.xgb
t.xgb.cv
0% all
1k
2k
4k
8k
Training dataset size
Figure 10: Training size on criteo
Treatment imbalance can have large impact on model performance, On natcity, win share of DR leamer is 40% with balanced treatment, and 16% when treated ratio is 0.9. See Figure II
Level of nonlinearity in assignment mechanism, found, has limited impact on model performance. This is partly, we think, because our propensity estimator (Random Forest with propensity clipping) flexible enough fit different degrees of nonlinearity. See Figure 8 in Appendix.
30

---

Published as a conference paper at ICLR 2025
Model win share mean treatment %: natcity
100%
dr.ridge.cv
dragon.mn
dml.elastic
80%
s.ext.ridge.co
dmi.lasso
t.ridge.cv
s.xgb.cv
60%
Win share
r.ridge.vv
dr.xgb.cv
s.ridge.cv
40%
t.xgb.cv
s.ext.xgb.cV
cforest
r.gbb.cv
20%
dml.xgb
dml.linear
0%
0.1
0.5
0.9
Mean treatment %
Figure 11: treatment % on natcity
31

---

Published as a conference paper at ICLR 2025
1.2
CRITEO
Win vs. degenerate rates: criteo
100%
s.xgb.cv
r.rideecv
90%
dragon.nn
s.ridge.cv
80%
s.ext.ridge.cv
s.ext.xg..cv
Degenerate rate
dr.ridge.cv
70%
dml.elastic
dml.lasso
60%
cforest
r.xgb.cv
50%
t.ridge.cv
dr.xgb.cv
40%
dml.linear
dml.xgb
30%
t.xgb.cv
20%
0.0%
2.5%
5.0%
7.5%
10.0%
12.5%
15.0%
Win rate
Figure 12: Win share degenerate percentage, by model criteo
Model win share vs. dataset size: criteo
100%
s.xgb.cv
r.ride.cvv
dragon.nn
80%
s.ridge.cv
s.ext.ridge.cv
s.ext.xgb.cv
dr.ridge.cv
60%
Win share
dml.elastic
dml.lasso
cforest
40%
r.xgb.cv
t.ridge.cv
dr.xgb.cv
dmi.linear
20%
dml.xgb
t.xgb.cv
0%
all
2k Training dataset size
Figure 13: Model win share by estimation data, cr criteo
32

---

Published as a conference paper at ICLR 2025
Model win share mean treatment %: criteo
100%
s.xgb.cv
r.ridgecvv
dragon.nn
80%
s.ridge.cv
s.ext.ridge.cv
s.ext.xgb.cV
dr.ridge.cv
60%
Win share
dml.elastic
dml.lasso
cforest
40%
r.xgb.cv
t.ridge.cv
dr.xb.cv
dml.linear
20%
dml.xgb
t.xgb.cv
0%
0.1
0.5
0.9
Mean treatment %
Figure 14: Model win share by treatment ratio, criteo
Model win share vs. complexity: criteo
100%
s.xgb.cv
r.ridgecvv
dragon.nn
80%
s.ridge.cv
s.ext.ridge.cv
s.ext.xgb.cv
dr.ridge.cv
60%
Win share
dml.elastic
dml.lasso
cforest
40%
r.xgb.cv
t.ridge.cv
dr.xgb.cv
dmi.linear
20%
dml.xgb
t.xgb.cv
0%
Complexity
Figure 15: Model win share by assignment mechanism complexity, criteo
33

---

Published as a conference paper at ICLR 2025
1.3
BILLSTROM
Win vs. degenerate rates: hillstrom
100%
dragon.nn
s.ridge.cv
dml.elastic
dml.lasso
80%
s.ext.ridge.cv
s.xgb.cv
Degenerate rate
r.ridg.cvv
60%
s.ext.xgb.cv
dr.ridge.cv
dr.xgb.cv
40%
t.ridge.cv
r.gbb.vv
t.xgb.vv
dml.xgb
20%
cforest
dml.linear
0%
0.0%
5.0%
10.0%
15.0%
20.0%
Win rate
Figure 16: Win share degenerate percentage, by model hillstrom
Model win share
dataset size: hillstrom
100%
dragon.nn
s.ridge.cv
dml.elastic
80%
dml.lasso
s.ext.ridge.cv
s.xgb.cv
r.ridge.cv
60%
Win share
s.ext.xgb.cv
dr.ridge.cv
dr.xgb.cv
40%
t.ridge.cv
r.xgb.cv
t.xgb.cv
dml.xgb
20%
cforest
dml.linear
0%
all
2k Training dataset size
Figure 17: Model win share by estimation data, hi 11 rom
34

---

Published as a conference paper at ICLR 2025
Model win share mean treatment %: hillstrom
100%
dragon.nn
s.ridge.cv
dml.elastic
80%
dml.lasso
s.ext.ridge.cv
s.xgb.cv
r.ridge.cv
60%
Win share
s.ext.xgb.cv
dr.ridge.cv
dr.xgb.cv
40%
t.ridge.cv
r.xgb.cv
t.xgb.cv
dml.xgb
20%
cforest
dml.linear
0%
0.1
0.5
0.9
Mean treatment %
Figure 18: Model win share by treatment ratio, hillstrom
Model win share complexity: hillstrom
100%
dragon.nn
s.ridge.cv
dml.elastic
80%
dml.lasso
s.ext.ridge.cv
s.xgb.cv
r.ridgecvv
60%
Win share
s.ext.xgb.cv
dr.idgge.cv
dr.xgb.cv
40%
t.ridge.cv
r.xgb.cv
t.xgb.cv
dml.xgb
20%
cforest
dml.linear
0%
Complexity
Figure 19: Model win share by assignment mechanism complexity, hillstrom
35

---

Published as a conference paper at ICLR 2025
1.4
FERMAN
Win vs. degenerate rates: ferman
100%
s.xgb.cv
s.ext.xgb.cv
s.ridge.cv
dml.elastic
80%
dml.lasso
dragon.nn
Degenerate rate
r.xgb.cv
60%
dr.ridge.cv
dr.xbb.vv
dml.xgb
cforest
40%
dml.linear
r.ridge.cv
s.ext.ridge.cv
t.ridge.cv
t.xgb.cv
0%
0%
20%
40%
60%
80%
Win rate
Figure 20: Win share degenerate percentage, by model ferman
Model win share vs. dataset size: ferman
100%
s.xgb.cv
s.ext.xgb.cv
s.ridge.cv
80%
dml.elastic
dms.lasso
dragon.nn
r.xgb.cv
60%
Win share
dr.ridge.cv
dr.xgb.cv
dml.xgb
40%
cforest
dmi.linear
r.ridge.cv
s.ext.ridge.cv
20%
t.ridge.cv
t.xgb.cv
0%
all
2k Training dataset size
Figure 21: Model win share by estimation data, ferman
36

---

Published as a conference paper at ICLR 2025
Model win share mean treatment %: ferman
100%
s.xgb.cV
s.ext.xgb.cv
s.ridge.cv
80%
dmi.elastic
dmi.lasso
dragon.nn
r.xgb.cv
60%
Win share
dr.ridge.cv
dr.xgb.cv
dml.xgb
40%
cforest
dml.linear
r.ridgeccv
s.ext.ridge.cv
20%
tridge.cv
t.xgb.cv
0%
0.1
0.5
0.9
Mean treatment %
Figure 22: Model win share by treatment ratio, ferman
Model win share vs. complexity: ferman
100%
s.xgb.cV
s.ext.xgb.cv
s.ridge.cv
80%
dml.elastic
dmi.lasso
dragon.nn
r.xgb.cv
60%
Win share
dr.ridge.cv
dr.xgb.cv
dmi.xgb
40%
cforest
dml.linear
r.ridge.cv
s.ext.ridge.cv
20%
t.ridge.cv
t.xgb.cv
0%
Complexity
Figure 23: Model win share by assignment mechanism complexity, ferman
37

---

Published as a conference paper at ICLR 2025
1,5
SANDERCOCK
Win vs. degenerate rates: sandercock
100%
s.ridge.cv
s.xgb.cv
dms.lasso
dml.elastic
80%
s.ext.xgb.cv
dragon.nn
Degenerate rate
cforest
r.xgb.cv
60%
dr.gb.cv
s.ext.ridge.cv
dr.ridge.cv
r.idge.cv
40%
t.xgb.cv
dml.linear
dml.xgb
t.ridge.cv
20%
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
Win rate
Figure 24: Win share degenerate percentage, by model sandercock
Model win share
dataset size: sandercock
100%
s.ridge.cv
s.xgb.cv
dms.lasso
80%
dml.elastic
s.ext.xgb.cv
dragon.nn
cforest
60%
Win share
r.xgb.cv
dr.xgb.cv
s.ext.ridge.cv
40%
dr.ridge.cv
r.ridge.vv
t.xgb.cv
dmi.linear
20%
dml.xgb
t.ridge.cv
0%
all
2k Training dataset size
Figure 25: Model win share by estimation data, sandercock
38

---

Published as a conference paper at ICLR 2025
Model win share dataset size: sandercock
100%
s.ridge.cv
s.xgb.cv
dml.lasso
80%
dmi.elastic
s.ext.xgb.cv
dragon.nn
cforest
60%
Win share
r.xgb.cv
dr.xgb.cv
s.ext.ridge.cv
40%
dr.ridge.cv
r.ridge.cv
t.xgb.cv
dmi.linear
20%
dml.xgb
t.ridge.cv
0%
2k Training dataset size
4k
Figure 26: Model win share by treatment ratio, sandercock
Model win share complexity: sandercock
100%
s.ridge.cv
s.xgb.cv
dml.lasso
80%
dml.elastic
s.ext.xgb.cv
dragon.
cforest
60%
Win share
r.xbb.cv
dr.xgb.cv
s.ext.ridge.cv
40%
dr.ridge.cv
r.ridge.cv
t.xgb.cv
dmi.linear
20%
dml.xgb
trrige.cv
0%
Complexity
Figure 27: Model win share by assignment mechanism complexity, sandercock
39

---

Published as a conference paper at ICLR 2025
1.6
NATAID
Win vs. degenerate rates: nataid
100%
s.ridge.cv
dragon.nn
dml.lasso
dml.elastic
80%
r.ridge.cv
s.xgb.cv
Degenerate rate
dr.ridge.cv
60%
s.ext.ridge.cv
drxgb.cv
s.ext.xgb.cv
40%
t.ridge.cv
r.gbb.v
dml.xgb
dml.linear
20%
cforest
t.xgb.cv
0%
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
Win rate
Figure 28: Win share degenerate percentage, by model nataid
Model win share
dataset size: nataid
100%
s.ridge.cv
dragon.nn
dml.lasso
80%
dml.elastic
r.ridge.cv
s.xgb.cv
dr.ridge.cv
60%
Win share
s.ext.ridge.cv
dr.xgb.cv
s.ext.xgb.cv
40%
t.ridge.cv
r.xgb.cv
dml.xgb
dmi.linear
20%
cforest
t.xgb.cv
0%
all
2k Training dataset size
Figure 29: Model win share by estimation data, na hataid
40

---

Published as a conference paper at ICLR 2025
Model win share vs. mean treatment %: nataid
100%
s.ridge.cv
dragon.nn
dml.lasso
80%
dmi.elastic
r.ridge.cv
s.xgb.cv
dr.ridge.cv
60%
Win share
s.ext.ridge.cv
dr.xgb.cv
s.ext.xgb.cv
40%
t.ridge.cv
r.xgb.cv
dml.xgb
dmi.linear
20%
cforest
t.xgb.cv
0%
0.1
0.5
0.9
Mean treatment %
Figure 30: Model win share by treatment ratio, nataid
Model win share complexity: nataid
100%
s.ridge.cv
dragon.nn
dml.lasso
80%
dml.elastic
r.ridge.cv
s.xgb.cv
dr.ridge.cv
60%
Win share
s.ext.ridge.cv
dr.xgb.cv
s.ext.xgb.cv
40%
t.ridge.cv
r.xgb.cv
dml.xgb
dml.linear
20%
cforest
t.xgb.cv
0%
Complexity
Figure 31: Model win share by assignment mechanism complexity, nataid
41

---

Published as a conference paper at ICLR 2025
1.7
NATARMS
Win vs. degenerate rates: natarms
100%
s.xgb.cv
s.ridge.cv
dragon.nn
dml.elastic
80%
dml.lasso
r.ridge.cv
Degenerate rate
s.ext.xgb.cv
dr.ridge.cv
60%
drxgb.cv
s.ext.ridge.cv
cforest
dml.linear
dml.xgb
r.gbbcvv
t.ridge.cv
20%
t.xgb.cv
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
Win rate
Figure 32: Win share degenerate percentage, by model natarms
Model win share vs. dataset size: natarms
100%
s.xgb.cv
s.ridge.cv
dragon.nn
80%
dml.elastic
dms.
r.ridge.vv
s.ext.xgb.cv
60%
Win share
dr.ridge.cv
dr.xgb.cv
s.ext.ridge.cv
40%
cforest
dml.linear
dml.xgb
r.xgb.cv
20%
t.ridge.cv
t.xgb.cv
0%
all
2k Training dataset size
Figure 33: Model win share by estimation data, nat arms
42

---

Published as a conference paper at ICLR 2025
Model win share mean treatment %: natarms
100%
s.xgb.cv
s.ridge.cv
dragon.nn
80%
dmi.elastic
dmi.lasso
r.ridge.cv
s.ext.xgb.cV
60%
Win share
dr.ridge.cv
dr.xgb.cv
s.ext.ridge.cv
40%
cforest
dml.linear
dml.xgb
r.xgb.vv
20%
t.rige.ccv
t.xgb.cv
0%
0.1
0.5
0.9
Mean treatment %
Figure 34: Model win share by treatment ratio, natarnn
Model win share vs. complexity: natarms
100%
s.xgb.cv
s.ridge.cv
dragon.nn
80%
dml.elastic
dmi.lasso
r.ridge.cv
s.ext.xgb.cv
60%
Win share
dr.ridge.cv
dr.xgb.cv
s.ext.ridge.cv
40%
cforest
dml.linear
dml.xgb
r.xgb.cv
20%
t.ridge.cv
t.xgb.cv
0%
Complexity
Figure 35: Model win share by assignment mechanism complexity, natarms
43

---

Published as a conference paper at ICLR 2025
1.8
NATCITY
Win vs. degenerate rates: natcity
100%
dr.ridge.cv
dragon.
dml.elastic
80%
s.ext.ridge.co
dml.lasso
tridge.cv
Degenerate rate
s.xgb.cv
60%
r.rige.cv
drxgb.cv
s.ridge.cv
40%
t.xgb.cv
s.ext.xgb.cv
cforest
20%
r.gb.cvv
dml.xgb
dml.linear
0%
0.0%
5.0%
10.0%
15.0%
20.0%
Win rate
Figure 36: Win share degenerate percentage, by model natcity
Model win share vs. dataset size: natcity
100%
dr.ridge.cv
dragon.nn
dml.elastic
80%
s.ext.ridge.cv
das.lasso
t.ridge.cv
s.xgb.cv
60%
Win share
r.ridge.cv
dr.xgb.cv
s.ridge.cv
40%
t.xgb.cv
s.ext.xgb.cv
cforest
r.xgb.cv
20%
dml.xgb
dml.linear
0%
all
2k
4k
Training dataset size
Figure 37: Model win share by estimation data, nat natcity
44

---

Published as a conference paper at ICLR 2025
Model win share mean treatment %: natcity
100%
dr.ridge.cv
dragon.nn
dml.elastic
80%
s.ext.ridge.cv
dmi.lasso
t.ridge.cv
s.xgb.cv
60%
Win share
r.ridge.cv
dr.xgb.cv
s.ridge.cv
40%
t.xgb.cv
s.ext.xgb.cv
cforest
r.gbb.vv
20%
dml.xgb
dml.linear
0%
0.1
0.5
0.9
Mean treatment %
Figure 38: Model win share by treatment ratio, natcity
Model win share vs. complexity: natcity
100%
dr.ridge.cv
dragon.nn
dml.elastic
80%
s.ext.ridge.cv
dmi.lasso
t.ridge.cv
s.xgb.cv
60%
Win share
r.ridge.cv
dr.xgb.cv
s.ridge.cv
40%
t.xgb.cv
s.ext.xgb.cv
cforest
r.xgb.cv
20%
dml.xgb
dmi.linear
0%
Complexity
Figure 39: Model win share by assignment mechanism complexity, natcity
45

---

Published as a conference paper at ICLR 2025
1.9
NATCRIKE
Win vs. degenerate rates: natcrime
100%
t.ridge.cv
dr.ridge.cv
r.ridgecvv
s.ext.ridge.cv
80%
s.xgb.cV
dml.elastic
Degenerate rate
dml.lasso
60%
dragon.nn
drxgb.cv
s.ridge.cv
40%
t.xgb.cv
s.ext.xgb.cv
r.xgb.cv
cforest
20%
dml.xgb
dml.linear
0.0%
5.0%
10.0%
15.0%
20.0%
Win rate
Figure 40: Win share degenerate percentage, by model natcrime
Model win share
dataset size: natcrime
100%
triige.cv
dr.ridge.cv
r.ridge.cv
80%
s.ext.ridge.cv
s.xgb.cv
dml.elastic
dml.lasso
60%
Win share
dragon.nn
dr.xgb.cv
s.ridge.cv
40%
t.xgb.cv
s.ext.xgb.cv
r.xgb.cv
cforest
20%
dml.xgb
dml.linear
0%
all
2k Training dataset size
Figure 41: Model win share by estimation data, natcrin rime
46

---

Published as a conference paper at ICLR 2025
Model win share vs. mean treatment %: natcrime
100%
triige.cv
dr.ridge.cv
r.ridge.cv
80%
s.ext.ridge.cv
s.xgb.cv
dml.elastic
dml.lasso
60%
Win share
dragon.nn
dr.xgb.cv
s.ridge.cv
40%
t.xgb.cv
s.ext.xgb.cv
r.xbb.vv
cforest
20%
dml.xgb
lin.lineer
0%
0.1
0.5
0.9
Mean treatment %
Figure 42: Model win share by treatment ratio, nat naterime
Model win share complexity: natcrime
100%
trridgecvv
dr.ridge.cv
r.ridge.cv
80%
s.ext.ridge.cv
s.xgb.cv
dml.elastic
dml.lasso
60%
Win share
dragon.nn
dr.xgb.cv
s.ridge.cv
40%
t.xgb.cv
s.ext.xgb.cv
r.xgb.cv
cforest
20%
dml.xgb
dmi.linear
0%
Complexity
Figure 43: Model win share by assignment mechanism complexity, natorime
47

---

Published as a conference paper at ICLR 2025
1.10
NATDRUG
Win degenerate rates: natdrug
100%
dragon.nn
s.ext.ridge.cv
s.Xgb.cv
s.ridge.cv
80%
dml.lasso
dml.elastic
Degenerate rate
r.ridg.cvv
60%
dr.ridge.cv
t.ridge.cv
s.ext.xgb.cv
40%
dr.xgb.cv
cforest
dml.xgb
dml.linear
20%
r.xgb.cv
t.xgb.cv
0%
0.0%
5.0%
10.0%
15.0%
20.0%
Win rate
Figure 44: Win share degenerate percentage, by model natdrug
Model win share vs. dataset size: natdrug
100%
dragon.nn
s.ext.ridge.cv
s.xgb.cV
80%
s.ridge.cv
dml.lasso
dml.elastic
r.ridge.cv
60%
Win share
dr.ridge.cv
t.ridge.cv
s.ext.xgb.cv
40%
dr.xgb.cv
cforest
dml.xgb
dmi.linear
20%
r.xgb.cv
t.xgb.cv
0%
all
2k Training dataset size
Figure 45: Model win share by estimation data, nat artu
48

---

Published as a conference paper at ICLR 2025
Model win share vs. mean treatment %: natdrug
100%
dragon.nn
s.ext.ridge.cv
s.xgb.cv
80%
s.ridge.cv
dmi.lasso
dml.elastic
r.ridgecvv
60%
Win share
dr.ridge.cv
t.ridge.cv
s.ext.xgb.cv
40%
dr.xgb.cv
cforest
dml.xgb
dmi.linear
20%
r.xgb.cv
t.xgb.cv
0%
0.1
0.5
0.9
Mean treatment %
Figure 46: Model win share by treatment ratio, nat drug
Model win share vs. complexity: natdrug
100%
dragon.nn
s.ext.ridge.cv
s.xgb.cv
80%
s.ridge.cv
dmi.lasso
dml.elastic
r.ridge.cv
60%
Win share
dr.ridge.cv
t.ridge.cv
s.ext.xgb.cv
40%
dr.xgb.cv
cforest
dml.xgb
dmi.linear
20%
r.xgb.cv
t.xgb.cv
0%
Complexity
Figure 47: Model win share by assignment mechanism complexity, natdrug
49

---

Published as a conference paper at ICLR 2025
1.11
NATEDUC
Model win share vs. complexity: nateduc
100%
s.xgb.cv
s.ext.xgb.cv
s.ridge.cv
80%
dragon.nn
s.ext.ridge.cv
dml.elasten
dmi.lasso
60%
Win share
r.ridge.cv
dr.ridge.cv
dr.xgb.cv
40%
t.ridge.cv
cforest
dml.xgb
dml.linear
20%
r.xgb.cv
t.xgb.cv
0%
all
2
3
Complexity
Figure 48: Win share degenerate percentage, by model nateduc
Win vs. degenerate rates:
nateduc
100%
s.xgb.cv
s.ext.xgb.cv
s.ridge.cv
80%
dragon.nn
s.ext.ridge.cv
dml.elastic
Degenerate rate
dms.lasso
60%
r.ridge.cv
drridge.cc
dr.xgb.cv
40%
t.ridge.cv
cforest
dml.xgb
20%
dml.linear
r.xgb.cv
t.xgb.cv
0%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Win rate
Figure 49: Model win share by estimation data, nat eateduc
50

---

Published as a conference paper at ICLR 2025
Model win share vs. dataset size: nateduc
100%
s.xgb.cV
s.ext.xgb.cv
s.ridge.cv
80%
dragon.nn
s.ext.ridge.cv
dml.elastic
dml.lasso
60%
Win share
r.ridge.cv
dr.ridge.cv
dr.xgb.cv
40%
t.ridge.cv
cforest
dml.xgb
dmi.linear
20%
r.xgb.cv
t.xgb.cv
0%
2k
4k
Training dataset size
Figure 50: Model win share by treatment ratio, nateduc
Model win share vs. mean treatment %: nateduc
100%
s.xgb.cV
s.ext.xgb.cv
s.ridge.cv
80%
dragon.nn
s.ext.ridge.cv
dml.elastic
dml.lasso
60%
Win share
r.ridge.cv
dr.rddge.cv
dr.xgb.cv
40%
t.ridge.cv
cforest
dml.xgb
dmi.linear
20%
r.xgb.cv
t.xgb.cv
0%
0.1
0.5
0.9
Mean treatment %
Figure 51: Model win share by assignment mechanism complexity, nateduc
51

---

Published as a conference paper at ICLR 2025
1.12
NATENVIR
Win vs. degenerate rates: natenvir
100%
s.xgb.cv
s.ridge.cv
dragon.nn
dml.elastic
80%
dml.lasso
s.ext.ggb.cv
Degenerate rate
r.ridgeccv
60%
s.ext.ridge.cv
dr.ridge.cv
t.ridge.cv
40%
dr.xgb.cv
cforest
dml.xgb
dml.linear
20%
r.xgb.cv
t.xgb.cv
0%
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
Win rate
Figure 52: Win share degenerate percentage, by model natenvir
Model win share vs. dataset size: natenvir
100%
s.xgb.cv
s.ridge.cv
dragon.nn
80%
dml.elastic
dms.lasso
s.ext.xgb.cv
r.ridge.cv
60%
Win share
s.ext.ridge.cv
dr.ridge.cv
t.ridge.cv
40%
dr.xgb.cv
cforest
dml.xgb
dmi.linear
20%
r.xgb.cv
t.xgb.cv
0%
all
2k Training dataset size
Figure 53: Model win share by estimation data, natenvic
52

---

Published as a conference paper at ICLR 2025
Model win share mean treatment %: natenvir
100%
s.xgb.cv
s.ridge.cv
dragon.nn
80%
dmi.elastic
dmi.lasso
s.ext.xgb.cv
r.ridge.cv
60%
Win share
s.ext.ridge.cv
dr.ridge.cv
t.rigeeccv
40%
dr.xgb.cv
cforest
dml.xgb
dmi.linear
20%
r.xgb.cv
t.xgb.cv
0%
0.1
0.5
0.9
Mean treatment %
Figure 54: Model win share by treatment ratio, natenvir
Model win share vs. complexity: natenvir
100%
s.xgb.cv
s.ridge.cv
dragon.nn
80%
dml.elastic
dmi.
s.ext.xgb.cv
r.ridgeccv
60%
Win share
s.ext.ridge.cv
dr.iddge.cv
t.ridge.cv
40%
dr.xgb.cv
cforest
dml.xgb
dmi.linear
20%
r.xgb.cv
t.xgb.cv
0%
Complexity
Figure 55: Model win share by assignment mechanism complexity, natenvir
53

---

Published as a conference paper at ICLR 2025
1.13
NATFARE
Win degenerate rates: natfare
s.ext.rdde..cv
t.ridgeccv
80%
dr.ridge.cv
dml.elastic
dml.lasso
60%
s.ext.xgb.cv
Degenerate rate
dr.xgb.cv
t.xgb.cv
rridge.cv
40%
dragon.nn
s.ridge.cv
s.xgb.cv
cforest
20%
r.ggbccv
dml.xgb
dmi.linear
0%
0%
10%
20%
30%
40%
50%
60%
Win rate
Figure 56: Win share degenerate percentage, by model natfare
Model win share vs. dataset size: natfare
100%
s.ext.ridge.cv
t.ridge.cv
dr.ridge.cv
80%
dml.elastic
dms.lasso
s.ext.xgb.cv
dr.xgb.cv
60%
Win share
t.xgb.cv
r.ridge.cv
dragon.nn
40%
s.ridge.cv
s.xgb.cv
cforest
r.xgb.cv
20%
dml.xgb
dml.linear
0%
all
2k Training dataset size
Figure 57: Model win share by estimation data, nat fare
54

---

Published as a conference paper at ICLR 2025
Model win share mean treatment %: natfare
100%
s.ext.ridge.cv
t.ridge.cv
dr.ridge.cv
80%
dmi.elastic
dmi.lasso
s.ext.xgb.cv
dr.xgb.cv
60%
Win share
t.xgb.cv
r.ridge.cv
dragon.nn
40%
s.ridge.cv
s.xgb.cv
cforest
r.gbb.cv
20%
dml.xgb
dml.linear
0%
0.1
0.5
0.9
Mean treatment %
Figure 58: Model win share by treatment ratio, nat fare
Model win share vs. complexity: natfare
100%
s.ext.ridge.cv
t.ridge.cv
dr.ridge.cv
80%
dml.elastic
dmi.
s.ext.xgb.cv
dr.xgb.cv
60%
Win share
t.xgb.cv
r.ridge.cv
dragon.nn
40%
s.ridge.cv
s.xgb.cv
cforest
r.xgb.cv
20%
dml.xgb
dmi.linear
0%
Complexity
Figure 59: Model win share by assignment mechanism complexity, nat fare
55


Reviews:
[{"review_id": "a2bkYIJnul", "reviewer": "Reviewer_tCgC", "summary": "The paper proposes a new statistical parameter to evaluate the performance of a method to estimate CATE. Then the paper shows that this parameter can be consistently estimated. Finally the paper demonstrates that this parameter can be used to select estimation methods for CATE.", "questions": "Please address the weaknesses above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The statistical parameter proposed in the paper is simple to implement and makes intuitive sense. \n- Theoretical guarantees are provided. \n- The paper compares many estimation methods for CATE in the empirical application.", "weaknesses": "- The paper largely overclaims the number of datasets. Essentially, there are only 12 unique datasets and the paper uses a bunch of sampling methods to create many more variants of the original 12 datasets.\n- The findings in the first paragraph of the abstract have been known for a long time, and they are not unexpected, as claimed in the paper. The estimation of CATE is known to be very noisy, and that's why the causal inference literature has primarily focused on the estimation of ATE (but not CATE) for decades. \n- The clarity and rigor of the paper need to be improved. For example, the term \"CATE model\" is a bit awkward and is rarely used in the literature. CATE is an estimand commonly defined based on potential outcomes. When \"CATE model\" is used, it is unclear whether this term refers to an outcome model to define CATE or an estimation method for CATE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t3N4mAWeT2", "reviewer": "Reviewer_x6Rc", "summary": "In this work, the authors propose a new way to estimate treatment effect and perform model selection when the counterfactual is not available. To do this, the authors propose a new metric called Q, which is derived from the Mean Squared Error (MSE) but adjusted to exclude terms that do not depend on the estimator.", "questions": "1. Can you explain why your estimation size is much smaller than the evaluation size?\n\n2. Can you re-conduct the experiments when the estimation size is much larger than the evaluation size? Let's use the more standard ML experimental setup, where the ration between training and test samples is 4:1. You should apply this to all datasets in your Table 4. Then rerun the whole experimental pipeline and report results similar to those shown in Table 1.\n\n3. Also, there is no standard deviation for Table 1. It might be a little bit misleading in using the description \"43,200 datasets\" in Table 1. What you are essentially doing is repeated resampling, similar to bootstrap. In this case, in addition to what I ask for #2 above, could you also report results on the same datasets, with same percentage of treatment and other parameters fixed, with both mean and standard deviation, where the standard deviation solely comes from bootstrap?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Lacking good metrics for evaluation is a big problem in casual inference due to no access to counterfactuals. This paper proposes a new metric, which could be helpful for practitioners when comparing different models.", "weaknesses": "1. For the experiments given in this work, the estimation (training) size is much smaller than the evaluation size. This is not intuitive to me as in typical ML settings, we have training-test ratio to be 4:1. Using a much lower ratio of training vs test samples could potentially lead to the problem of unfitting, which could be a reason why many baselines have too many degenerate cases. I think it is necessary to use a large number to training samples to rule out this hypothetical scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nyytEEJssi", "reviewer": "Reviewer_vxwd", "summary": "This paper presents a large-scale benchmark study evaluating the performance of contemporary Conditional Average Treatment Effect (CATE) estimation models. The authors use a novel application of observational sampling to evaluate 16 modern CATE models across 43,200 datasets.  Their key findings challenge the effectiveness of current CATE models in capturing real-world heterogeneity.", "questions": "The paper discusses the generalization of the Q statistic to new distributions. Could you provide more details about the assumptions underlying these generalization results? How sensitive are these results to violations of these assumptions?\n\nWhat are the main limitations of the current study, and what directions for future research do you suggest based on these findings? What kinds of new CATE models or evaluation methods should be explored? What types of additional data would be useful for further validation?\n\nHow can the findings of this study be used to guide the practical application of CATE models in various domains (medicine, economics, etc.)? What advice would you give to practitioners regarding the selection and interpretation of CATE models?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Large-scale and comprehensive benchmark study.\n\nNovel approach to CATE evaluation using observational sampling and the Q statistic.\n\nRigorous theoretical analysis and proofs supporting the proposed methodology.\n\nUse of real-world datasets provides valuable insights into the limitations of current CATE models.", "weaknesses": "While aiming for diversity, the selection of datasets may still not fully represent the breadth of real-world applications, potentially limiting the generalizability of the findings. More explanation of dataset selection criteria would strengthen the paper.\n\nThe paper focuses on MSE as the primary evaluation metric. While justified by its practical relevance, exploring alternative evaluation metrics could provide additional insights.\n\nWhile innovative, the reliance on the Q statistic is new and requires further validation and adoption by the wider research community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YHYGdKWa7L", "reviewer": "Reviewer_UrBr", "summary": "This paper conducts an empirical study of the accuracy of CATE estimation methods, asking whether existing methods in the literature reliably outperform trivial benchmarks. The main idea is to use RCT datasets, where the propensity is known, to construct benchmarks where a simulated propensity score is used to construct an \"observational\" sample. Since the propensity score is known for the original data, estimators on the observational sample can be benchmarked without having to rely on accurate estimates of propensity or outcome models (effectively using a HT estimator, potentially augmented with other regression models just for variance reduction purposes).", "questions": "Any comments/clarifications, particularly related to points 1 and 3 above, would be helpful.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work is definitely important for the field. Evaluating CATE models is very difficult due to the risk of self-serving bias that the paper discusses, and it is unclear when in practice modern techniques are helpful. Constructing benchmarks and evaluation methods which use only real data, via RCT datasets, is important. Using semi-synthetic datasets with simulated outcomes, as is common in the field, is much less convincing.", "weaknesses": "There are some respects in which the execution of the paper could be improved:\n\n(1) Based on plots in the appendix, there appears to be significant variation in the performance of different methods across datasets (as would be expected). This deserves more discussion, and investigation. E.g. why do double ML methods perform well in some cases and not in many others? One hypothesis, based on benchmarks for ATE estimation, would be that outcome regression methods work best as long as the outcome models are reasonably well estimated, with double-ML style methods providing benefits when the outcomes are not as well captured. \n\nAs is, there is little in the way of takeaways about the conditions under which existing methods do or don't perform well. This sort of diagnosis would be at least as useful as the headline results about % of cases where methods work well or don't, since the headline numbers are very specific to the composition of this specific benchmark. \n\n(2) Relatedly, the benchmark appears to be weighted quite heavily towards a set of multiple tasks all drawn from the same dataset: out of 12 tasks, 8 are different outcomes taken from the general social survey. We might reasonably expect different tasks from the same dataset to share some similar characteristics so the amount of actual diversity in the benchmark is less than what the number of tasks would suggest. \n\n(3) Regarding the degeneracy rate, are the results different if we test for whether Q is significantly larger than 0 (i.e., a CI excludes 0) rather than if the point estimate is > 0? I would guess that most of the datasets are large enough for the estimates of Q to be fairly precise, but it would be helpful to verify this. \n\nWhile I think the paper makes a worthwhile contribution even with these weaknesses, the contribution would be significantly strengthened with a more diverse set of tasks and more analysis of what drives performance variation across those tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cMbdvkUpMF", "reviewer": "Reviewer_jZnM", "summary": "This paper presents a framework to evaluate CATE estimation methods in observational studies. To address challenges in existing approaches that either leverage semi-synthetic data or rely on untestable modeling assumptions, this work proposes to rely on propensity modeling for evaluation. Estimators for the MSE (minus a constant) are proposed and their theoretical properties are discussed under conditions on the propensity models. Then, this paper provides a comprehensive benchmark by deriving observational studies via resampling RCTs, which leads to unbiased evaluation of existing CATE methods. The benchmark reveals surprising findings about the insufficiency of the existing methods for capturing heterogeneity in CATE.", "questions": "1. Why is Theorem 3.4 better than the double robustness results? It seems to require consistency of the propensity model, but double robustness estimation can be consistent if either the outcome model or the propensity model is correct. \n\n2. What are the assumptions for Theorem 3.13 to hold? Does the propensity score need to be estimated? If so, why didn't its estimation error enter the result?\n\n3. It's weird to count randomly split data as a \"new dataset\" so that there are 43200 benchmark datasets. Please consider revise your claim on the number of datasets. \n\n4. In section 4.3, the performance of $\\hat{Q}$ is a bit worse than model-augmented versions of $\\hat{Q}$. It would be nice to provide some suggestions on what estimators to use in practice. Does the model-augmented version has additional bias/modeling concerns?\n\n5. What's special about the Horwitz-Thompson estimator for $\\eta$? It seems that $\\eta$ can be anything whose conditional expectation given $X$ equals $\\tau(X)$. Is it the point that it doesn't use any outcome model? Can it be replaced by other reasonable choices? \n\n6. While addressing the concerns of semi-synthetic evaluation and outcome modeling, here the method requires the propensity model to be well estimated (though I understand that the evaluation part doesn't have this concern due to controlled sampling process). Is this over-simplifying real-world evaluation scenarios where the treatment assignment in observational studies can be complicated as well? What is the method evaluating if an inconsistent propensity model is used? Is it estimating some other quantity that is related to MSE of CATE estimator? \n\n7. I would suggest separating the theoretical results for the most general evaluation case (eg in a real observational study) and the evaluation part in this paper. In real observational studies, consistent evaluation inevitably rely on some good modeling. However, it is important to be clearly stated that the evaluation in this paper doesn't need this because of the controlled sampling process.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper studies the important problem of reliable evaluation of CATE methods. \n2. The paper is comprehensive, containing fruitful results. \n3. The paper contributes new tools and benchmarks to the field, which may benefit other researchers and motivate future works. \n4. The paper is clearly structured.", "weaknesses": "1. The writing quality of the paper can still be improved. Sometimes the advantages of the proposed method is a bit overclaimed. It will be helpful to also discuss the limitation of the proposed approach.\n2. The discussion on the surprising results can be more in depth. Why are existing methods even worse than constant treatment effect estimation? Is this because they are too variable (so that variance in MSE is large)? Is this because the heterogeneity in the datasets is actually small? \n\nPlease see my questions for more comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)